Universal Manipulation Interface

In-The-Wild Robot Teaching Without In-The-Wild Robots

We present Universal Manipulation Interface (UMI) -- a data collection and policy learning framework that allows direct skill transfer from in-the-wild human demonstrations to deployable robot policies. UMI employs hand-held grippers coupled with careful interface design to enable portable, low-cost, and information-rich data collection for challenging bimanual and dynamic manipulation demonstrations. To facilitate deployable policy learning, UMI incorporates a carefully designed policy interface with inference-time latency matching and a relative-trajectory action representation. The resulting learned policies are hardware-agnostic and deployable across multiple robot platforms. Equipped with these features, UMI framework unlocks new robot manipulation capabilities, allowing zero-shot generalizable dynamic, bimanual, precise, and long-horizon behaviors, by only changing the training data for each task. We demonstrate UMI’s versatility and efficacy with comprehensive real-world experiments, where policies learned via UMI zero-shot generalize to novel environments and objects when trained on diverse human demonstrations.

Paper

Latest version: arXiv or here.

Code and Tutorial

Codebase

Hardware Guide

Data Collection Instruction

3D Printing Tutorial

Assembly Tutorial

Team

Cheng Chi^*1,2

Zhenjia Xu^*1,2

Chuer Pan¹

Eric Cousineau³

Benjamin Burchfiel³

Siyuan Feng³

Russ Tedrake³

Shuran Song^1,2

¹ Stanford University           ² Columbia University           ³ Toyota Research Institute           ^* Indicates equal contribution

BibTeX

@inproceedings{chi2024universal,
	title={Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots},
	author={Chi, Cheng and Xu, Zhenjia and Pan, Chuer and Cousineau, Eric and Burchfiel, Benjamin and Feng, Siyuan and Tedrake, Russ and Song, Shuran},
	booktitle={arXiv},
	year={2024}
}

Hardware Design

Policy Robustness

Capability Experiments

(1) Dynamic Tossing 🤾

(2) Cup Arrangement ☕

(3) Bimanual Cloth Folding 👚

(4) Dish Washing 🍽

In-the-wild Generalization Experiments

Data Collection Methods

Comparison with Other Methods

Human Demonstration on Other Tasks Using UMI Gripper

Acknowledgements

This work was supported in part by the Toyota Research Institute, NSF Award #2037101, and #2132519. We want to thank Google and TRI for the UR5 robots, and IRIS and IPRL lab for the Franka robot hardware. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of the sponsors.

We would like to thank Andy Zeng, Pete Florance, Huy Ha, Yihuai Gao, Samir Gadre, Mandi Zhao, Mengda Xu, Alper Canberk, Kevin Zakka, Zeyi Liu, Dominik Bauer, Tony Zhao, Zipeng Fu and Lucy Shi for their thoughtful discussions. We thank Alex Alspach, Brandan Hathaway, Aimee Goncalves, Phoebe Horgan, and Jarod Wilson for their help on hardware design and prototyping. We thank Naveen Kuppuswamy, Dale McConachie, and Calder Phillips-Graffine for their help on low-level controllers. We thank John Lenard, Frank Michel, Charles Richter, and Xiang Li for their advice on SLAM. We thank Eric Dusel, Nwabisi C., and Letica Priebe Rocha for their help on the MoCap dataset collection. We thank Chen Wang, Zhou Xian, Moo Jin Kim, and Marion Lepert for their assistance with the Franka setup. We especially thank Steffen Urban for his open-source projects on GoPro SLAM and Camera-IMU calibration, and John @ 3D printing world for inspiration of the gripper mechanism.

Contact

If you have any questions, please feel free to contact Cheng Chi and Zhenjia Xu.