2023: RT-2 Brings AI to Robots

On This Day in AI: July 28, 2023

On July 28, 2023, Google DeepMind introduced RT-2, a vision-language-action model that translated images and natural-language instructions directly into robot actions. The system combined web-scale visual knowledge with robot demonstrations to improve generalization beyond the situations seen during training. Robots traditionally learned narrow tasks from expensive physical demonstrations and struggled when objects, instructions, or environments changed. Vision-language models understood broad concepts from internet data but normally produced words rather than movements in the physical world. The date matters because it turned a fast-moving technical, policy, or commercial development into a visible milestone that people and institutions could evaluate.

The substance of Google DeepMind’s RT-2 vision-language-action model was as important as the headline. RT-2 represented actions as tokens and co-trained on web and robotics data using adapted PaLI-X and PaLM-E backbones. It could follow new commands, reason about object categories, and perform multi-stage tasks such as selecting an improvised tool. Those choices reveal what the people behind the milestone believed was necessary at the time. AI history rarely advances through one breakthrough alone; research, infrastructure, product design, governance, and access all shape what a system can become.

The immediate consequences reached beyond the organization at the center of the event. The model showed a practical path for transferring semantic knowledge from large multimodal systems into robot control. It reduced the need to collect a demonstration for every combination of object and task while revealing new safety demands for embodied AI. The moment also invited scrutiny, because new capabilities create new dependencies. Reliability, safety, cost, privacy, and human control determine whether an impressive demonstration becomes a trustworthy part of everyday life. Those practical limits are essential to understanding both the achievement and the response it received.

In retrospect, Vision-language-action models became a major direction for general-purpose robotics and physical agents. RT-2 stands as an early milestone in treating robot movements as another language a foundation model could learn to generate. Looking back from today’s rapidly changing AI landscape, this event belongs to a larger transition from specialized machine-learning experiments to systems embedded in ordinary products, workplaces, institutions, and public decisions. Its enduring lesson is that technical progress becomes historically significant when it changes what people can do and what society expects next. It also reminds us that every celebrated capability is shaped by the data, institutions, hardware, and human judgment surrounding it. The anniversary therefore records a turning point, not simply an announcement.