Building general purpose robots: Robot learning in the era generative AI

Nicolas Heess

Google DeepMind, UK

Abstract

Just a few years ago, general-purpose robots—machines with human-like motor skills, a deep understanding of the physical world, and the ability to meaningfully interact with humans—seemed out of reach. But recent progress in generative AI, foundation models, and reinforcement and imitation learning has fundamentally changed the field. In this talk, I will review the puzzle pieces that have recently fallen into place and explore how they fit together. Focusing on the work we are doing at Google DeepMind and using key examples from the broader literature, I will trace the developments that have led us to where we are today, highlight recent results, and discuss what I see as the outstanding challenges ahead.