Robot Utility Models

General Policies for Zero-Shot Deployment in New Environments

90% success rate in novel environments with 0 additional data or training.

Haritheja Etukuru*, Norihito Naka, Zijin Hu, Seungjae Lee, Julian Mehu, Aaron Edsinger, Chris Paxton, Soumith Chintala, Lerrel Pinto, Nur Muhammad “Mahi” Shafiullah*

Corresponding author: mahi at cs dot nyu dot edu, (*) denotes equal contribution

RUMs, in a nutshell:

Videos

RUMs in action

Our RUMs attempted 5 tasks, each in 5+ environments, on a Hello Robot Stretch. They also attempted a few tasks on an xArm. See sample rollouts below:

RUMs Automatically Retrying Upon Failure

We feed in a summary of robot observations into a multimodal LLM, which determines whether or not the task at hand has succeeded. If the mLLM determines that the task has failed, the robot automatically resets to a new initial state and retries.

Hardware

The Stick V2

We've redesigned the Stick! Addressing some of the previous limitations, Stick V2 is designed to improve on user experience, becoming more ergonomic, more capable, and stronger than before.

Robot Gripper/iPhone Mount

Reorientation Robot
Hello Robot
Drawer Opening with Xarm
xArm 7

We've made it possible to add the Stick gripper onto your own robot arm with a 3D-printed mount and Dynamixel set, allowing for an identical POV. Thus facilitating seemless zero-shot transfer of policies to new robots.

Dataset

5 tasks
180 environments
5509 trajectories

We release the training dataset for our Robot Utility Models, containing 5 tasks, each with on average ~1000 training demonstrations across 36 environments. The dataset contains RGB videos at 30 fps, as well as full action annotations for 6D pose of the gripper and the gripper's opening angle normalized between (0, 1).

Paper

Robot Utility Models: General Policies for Zero-Shot Deployment in New Environments

@article{etukuru2024robot,
    title={Robot Utility Models: General Policies for Zero-Shot Deployment in New Environments}, 
    author={Haritheja Etukuru and Norihito Naka and Zijin Hu and Seungjae Lee and Julian Mehu and Aaron Edsinger and Chris Paxton and Soumith Chintala and Lerrel Pinto and Nur Muhammad Mahi Shafiullah},
    journal={arXiv preprint arXiv:2409.05865},
    year={2024}
}