RoboCat: A Self-Improving Foundation Agent for Robotic Manipulation

ABSTRACT

Creating general-purpose robots requires models that can rapidly adapt to new tasks and new physical embodiments. We present RoboCat, a self-improving foundation agent for robotic manipulation. RoboCat is an architecture based on a visual-goal-conditioned multi-modal transformer. It is trained on a massive, diverse dataset of trajectories from multiple different robotic arms performing hundreds of tasks. Crucially, RoboCat features a self-improving autonomous loop: when faced with a novel task, it requires only 100 to 1,000 human demonstrations to learn the task, generates its own synthetic practice data to improve its policy, and then adds this new data to its primary training set. RoboCat successfully controls highly distinct robot morphologies zero-shot and demonstrates unprecedented data efficiency in cross-embodiment transfer learning.