I am a final year CS PhD candidate at Stanford University in the Stanford AI Lab (SAIL), where I'm advised by Jeannette Bohg
and Marco Pavone. My research spans several topics in AI, with a particular focus on systems that operate in the physical world (robots):
Increasing the reliability of AI-driven systems during deployment;
Establishing connections between model behavior and training data;
Expanding full-stack capabilities through planning and reasoning;
Facilitating effective interaction between systems and humans.
Previously, I received a BASc from the University of Toronto (EngSci), where I was advised by Florian Shkurti.
I've also had the pleasure of interning at Meta FAIR, NASA JPL, Microsoft, and Google in past summers.
Jan 2023Our work on sequencing policies for long-horizon manipulation has been accepted to ICRA 2023!
Research
/
I'm broadly interested in building AI systems that reliably solve complex tasks in the physical world.
My research spans learning for planning and decision making, safety and reliability, and data curation and model interpretability (see my dissertation here and defense talk here).
I view these as facets of one system-level problem: toward agents that flexibly reason about and generalize to new tasks, whose behavior can be understood, safeguarded, and improved over time.
Papers and preprints are ordered by recency. Representative works are highlighted in green. Please see my Google Scholar page for a complete and up-to-date list of papers.
Dataset diversity drives success in robot imitation learning, but is hard to quantify over structured, variable-length, and high-dimensional trajectories.
We introduce FAKTUAL, a model-free data curation algorithm that uses signature kernel-based entropy to estimate the relative diversity of individual demonstrations.
LLMs and VLMs are increasingly deployed in robotics but remain vulnerable to jailbreaking attacks that may drive physically harmful behaviors in the real world.
We introduce J-DAPT, a real-time robotics jailbreak detector that adapts to prevent novel threats in data-limited regimes via multimodal domain adaptation.
In robot imitation learning, policy performance is tightly coupled with the quality and composition of the demonstration data.
We present CUPID, a data curation method that uses influence functions to estimate the causal impact of each demonstration on a policy's closed-loop performance.
Vision-Language-Action (VLA) models have demonstrated remarkable capabilities in visuomotor control, yet ensuring their robustness in unstructured real-world environments remains a persistent challenge.
In this paper, we investigate inference-time scaling as means to enhance VLA robustness and generalization.
Foundation models can reason about appropriate safety interventions in hazardous out-of-distribution scenarios beyond a robot's training data.
We present FORTRESS, a framework that generates and reasons about semantically safe fallback strategies in real time to prevent out-of-distribution failures.
How can we plan to solve unseen, long-horizon tasks from a single, partial-view point cloud of the scene, and can we do so without access
to long-horizon training data? Points2Plans leverages transformer-based relational dynamics to learn the symbolic and geometric effects of
robot skills, then compose the skills at test time to generate a long-horizon symbolic and geometric plan.
Robot behavior policies trained via imitation learning are prone to failure under conditions that deviate from their training data.
In this work, we present Sentinel, a runtime monitor that detects unknown failures (requiring no data of failures) of generative robot policies at deployment time.
How can we integrate human preferences into robot plans in a zero-shot manner, i.e., without requiring tens of thousands of data points
of human feedback? We propose Text2Interaction, a planning framework that invokes large language models to generate a task plan, motion preferences as Python code,
and parameters of a safe controller.
How can we mitigate the computational expense and latency of LLMs for real-time anomaly detection and reactive planning?
We propose a two-stage reasoning framework, whereby fast a LLM embedding model flags potential observational anomalies
while a slower generative LLM assesses the safety-criticality of flagged anomalies and selects a safety-preserving fallback plan.
We introduce DROID (Distributed Robot Interaction Dataset), a diverse robot manipulation dataset with
76k demonstration trajectories (or 350 hours of interaction data), collected across 564 scenes and 86
tasks by 50 data collectors in North America, Asia, and Europe over the course of 12 months. We demonstrate
that training with DROID leads to policies with higher performance and improved generalization ability.
Future space exploration missions to unknown worlds will require robust reasoning, planning, and decision-making capabilities,
enabled by the right choice of onboard models. In this work, we aim to understand what onboard models a spacecraft needs for fully autonomous space exploration.
Conventionally, robotic learning methods train a separate model for every application, every robot, and even every environment.
Can we instead train “generalist” X-robot policy that can be adapted efficiently to new robots, tasks, and environments?
Pretrained large language models can be readily used to obtain high-level robot plans from natural lanugage instructions, but should these
plans be executed without verifying them on the geometric-level? We propose Text2Motion, a language-based planner that tests if
LLM-generated plans (a) satisfy user instructions and (b) are geometric feasibility prior to executing them.
System-level failures are not due to failures of any individual component of the autonomy stack but
system-level deficiencies in semantic reasoning. Such edge cases, dubbed semantic anomalies,
are simple for a human to disentangle yet require insightful reasoning. We introduce a runtime monitor based
on large language models to recognize failure-inducing semantic anomalies.
Solving sequential manipulation tasks requires coordinating geometric dependencies between actions.
We develop a scalable framework for training skills independently, and then combine the skills at planning
time to solve unseen long-horizon tasks. Planning is formulated as a maximization problem over the expected
success of the skill sequence, which we demonstrate is well-approximated by the product of Q-values.
3D Scene Graphs (3DSGs) are informative abstractions of our world that unify symbolic, semantic, and metric scene representations.
We present a benchmark for robot task planning over large 3DSGs and evaluate classical and learning-based planners;
showing that real-time planning requires 3DSGs and planners to be jointly adapted to better exploit 3DSG hierarchies.
Pretraining visual representations for robotic reinforcement learning can improve sample efficiency and policy performance.
In this paper, we take an alternate approach and propose to augment the state embeddings of a self-driving agent with
attention in the latent space, accelerating the convergence of Actor-Critic algorithms.
Deep Reinforcement Learning is effective for learning robot navigation policies in rough terrain and cluttered simulated environments.
In this work, we introduce a series of techniques that are applied in the policy learning phase to enhance transferability to real-world domains.
How can semantic information be leveraged to improve localization accuracy in changing environments? We present a robust LiDAR-based localization
algorithm that exploits both semantic and geometric properties of the scene with an adaptive fusion strategy.
Small-scale semantic reconstruction methods have had little success in large outdoor scenes as a result of exponential increases in sparsity,
and a computationally expensive design. We propose a sparse convolutional network architecture based on the Minkowski Engine,
achieving state-of-the-art results for semantic scene completion in 2D/3D space from LiDAR point clouds.
Direct methods are able to track motion with considerable long-term accuracy. However, scale inconsistent estimates arise from random or unit depth initialization.
We integrate dense depth prediction with the Direct Sparse Odometry system to accelerate convergence in the windowed bundle-adjustment and promote estimates with consistent scale.
Theses & Dissertations
My thesis and dissertation research, ordered by recency.
This dissertation investigates how the reliability of learned robot policies can be improved at deployment time through mechanisms that operate around them.
We examine several such mechanisms: (1) runtime failure detection and intervention, (2) data-centric interpretation of policy behavior and performance, and (3) composition and coordination of learned behaviors for real-world, long-horizon tasks.
This thesis tests the suitability of existing embodied AI environments for research at the intersection of robot task planning
and 3D scene graphs and constructs a benchmark for empirical comparison of state-of-the-art classical planners.
Furthermore, we explore the use of graph neural networks to harness invariances in the relational structure of planning domains
and learn representations that afford faster planning.
Patents
Several of my research ideas have been patented alongside conference or journal publication.
Relates to methods and systems for automatically curating robotics datasets for robot imitation learning.
More specifically, relates to filtering or selecting demonstration data to maximize downstream KPIs of an imitation learning system using influence functions.
Relates to processing point clouds for autonomous driving of a vehicle. More specifically, relates to processing a sequence of point
clouds to generate a birds-eye-view (BEV) image of an environment of the vehicle which includes pixels associated with road surface labels.
