About This Talk
Physical AI is still largely built one application at a time. A robot policy, an autonomous-driving system, and a world simulator each get their own model, data pipeline, and limitations. NVIDIA’s Cosmos 3 starts from the premise that, as with language models, scaling a shared foundation can bring many narrow capabilities together.
In this talk, Max Li will explain how Cosmos 3 combines language, images, video, audio, and action in a single omnimodal world model, and why building it became as much a data-systems challenge as a modeling problem. Cosmos has grown to roughly a million hours of video and tens of billions of multimodal training candidates. At that scale, local storage and conventional robotics formats break down. Data must be streamed from object storage, filtered and enriched continuously, inspected at the sample level, and delivered fast enough to keep large GPU clusters busy.
Max will share how NVIDIA’s SILA platform uses Lance as a common substrate for media, metadata, curation signals, embeddings, and processing state; how the next generation of Cosmos is moving from curating in Lance to training from it directly; and what that architecture changes for iteration speed, observability, and scale.
The larger bet is that physical AI will follow the trajectory of language models, with: many specialized systems giving way to a general foundation that can be adapted, distilled, and deployed across domains. This talk is a look at the infrastructure (and the research conviction) required to reach that inflection point.
About The Speaker
Transcript
Show transcriptHide transcript
00:00
Hello, everyone. It's my pleasure to be here. I realize I didn't dress as pink as Brian, which suits the theme better today. But yeah, I'm here to talk to you about something about Cosmos, about NVIDIA, and particularly about physical AI, what we are envisioning, and what we think the future will look like. So you are probably familiar with NVIDIA's tech stack already, but we think that AI is going to revolutionize our daily life. It's going to diffuse into every layer of our society and fundamentally change how we think and how we work. And that depends on NVIDIA's hardware, and many of you may have already worked with, such as GPUs, and also the software, such as CUDA libraries. In the era of AI, there's one more layer, which is models. And we think that these models will be specialized for all kinds of different applications from chatbots, AI coding agents, robotics, robotaxi, and many more. And to serve this vision that we want to diffuse the technology into our daily life, we think software and hardware needs to be co-designed in a way that is optimal for the solution. For example, we think software needs to be agentic. It needs to act towards the goal, leverage other resources that it can use, and collaborate with its peers and also potentially humans to finish the task. It needs to be specialized. For example, a best model for chatbot will not be the most optimal model for driving a car. So there needs to be specialization in terms of the model design, and it needs to continuously improve. That's very important. Just like humans, given the past experience it has curated, it needs to become more adept at the task it's given.
02:05
And at the same time, the hardware needs to be productive, like we want every dollar of our customer count. We want to maximize the return of investment that our customer have and then really turn it into the more you buy, the more you save. And at the same time, the hardware needs to be durable. And in fact, very long-lasting, such as, for example, our other series GPUs are still on demand in the market for tasks such as data processing. And last but not least, fungible. You need to be able to take the GPUs anywhere on Earth and then do any of the task loads. It needs to be programmable, just like the islands that we were hearing about. So combining software and hardware extreme co-design, we hope that we'll be able to support the vision that AI will revolutionize our life. And when we look around, we see physical AI as very critical application because a lot of our GDP is actually associated with physical activities. Just throwing some stats here, we have 10 million factories around the world, four million industrial robots already installed, more than 500 million farms, 12,000 cities, 1.5 billion vehicles driving on the road, and 2 billion commercial cameras installed for surveillance purpose. And these robots and hardwares span all the application layers and domains such as agriculture, energy, healthcare, and many others that we are all familiar with. So we think physical AI is the next frontier. But what do we need? How do we make physical AI possible and actually working? So we've been thinking about this idea of physical agents, like an agent that's really smart, intelligent, just like human, that is able to work in the physical world.
04:21
Fundamentally, we think there are three key capabilities. For example, world understanding. It needs to be able to see the world, understand what has happened in the past, and reason about what needs to happen in the future. It needs to be able to simulate the outcome of certain actions and decide what is the best move next, because certain actions are very costly, like driving around pedestrians and et cetera. Last but not least, it needs to be able to take the actions, materialize that actions into the interactions with the physical world, and that compose into a physical agent that we want. And we really hope that Cosmos is this foundation layer of these physical agents. And during last launch, Cosmos 3, we introduced this architecture using mixture of transformer. Basically, it has two towers. One is autoregressive reasoner, and the other one is diffusion generator, which has shared latent space. It supports all many modality inputs such as text, image, video, audio, and action, and at the same time, being able to generate all these modalities. We released three sets of the models from Super, which is 64B, designed for maximum fidelity and works well with frontier compute, and also the smallest set, which is Edge, 4B model that works for real-time applications and run on Edge devices. And then the Nano is a blend between the two extremes and is often the best starting point for any post-training adaptations. So this is what we have released And what it enables, including things like data curation and search and annotation, because it is fundamentally the reasoner can reason about the video content and generates what's the meaning of the videos.
06:26
It can also be a planner that plans what the robot needs to do given the current environment observation and the task at hand, and then think about the best motion the robot needs to take. It can also be the backbone of any cameras installed for surveillance, doing real-time analysis of the environments, and then summarize the findings. Because we have the two towers, the generator tower can become the simulation foundation of physical AI. You can use the text-to-image modality or capability to sketch out what the scenario you want, and then turn that into a dynamic simulation with the image-to-video capability, and because physical AI requires multi-view, we've extended Kosmos to generate coherent views across different camera views for applications such as autonomous vehicle and robotics. And action is actually the very important layer of physical AI. But when we talk about action, a lot of people think about generating the actions and interacting with the world, but actually, it's more than that. For example, we support forward dynamics, where the model takes an input image and the actions and generate what could be the possible video frames from these actions. And many refer to this as a world model. On the other hand, we can do, given an input video and generating what's the underlying actions leading to these video observations. And that can turn Kosmos into a data generation engine for things like SLAM, camera pose estimation, human pose estimation. They can all be broadly categorized into this capability.
08:34
And last but not least, video action policy. So the model thinks, reasons, and generates what could be the future visual look like, and also generating the corresponding actions. And this becomes really the currency of physical AI, turning intelligence into tokens that can actually lead to revenues for a lot of the applications. Just to recap a little bit, Kosmos is a single architecture that supports many modalities and many applications. Just by configuring different input and output space, we're able to support applications such as vision language reasoning, image generation, audio-visual generation, robot policy, forward dynamics, and inverse dynamics. So all we need to do is toggle the input and output space. So how does this eventually work with one single architecture? We actually spend a lot of time thinking about how to make the model be aware of the physical time and understanding the different tokens coming from different modalities. Therefore, we actually came up with this idea with a unified mROPE that is FPS modulated. For example, the language will share the indexes tokens across different dimensions. Video will be temporal and spatial. Audio and action will just be temporal because they are 1D signals. And we modulate them with FPS, where faster FPS meaning that the tokens are actually closer in terms of distance, and slower FPS pushes the token away so that the model has a sense of time in the physical world. At the same time, we've been questioning ourselves, do we need this unified architecture between understanding and generation?
10:44
And we actually found great evidence that better understanding helps generation. So we did a really simple experiment by comparing different reasoners that is trained for general-purpose computer use agents and web searching, et cetera, versus a physical AI specialized reasoner. We actually find that the reasoner is able to improve the generation quality in terms of physics fidelity, which is very critical for physical AI. And that trend gives us a lot of confidence, further specialize the reasoner into physical AI tasks so that the generation tower can become better in terms of the physical AI-related generations. And coming back to action a little bit, I've beautified and simplified a lot of things when we talk about action, because action is actually a very hard modality to deal with. When we think about, for example, robots, different robots could have different form factors and morphologies, et cetera. So it's almost impossible to unify them into a single space unless we do some abstractions. And one thing that we have realized is that despite all these robots have different joints and different grippers, as long as you mount a camera on them, they are They can be unified into this camera motion space, and that becomes the foundation of our scaling for actions in physical world. We try to use the camera pose as the proxy layer for any of the actions that we're taking. And once we switch to this camera pose as proxy layer of action, we find it's actually pretty trivial to unify the action across many things. For example, a camera moving in space. For a SLAM, that is just a camera pose.
12:59
For autonomous vehicle, that's just a dash cam installed on the vehicle, and then for single-arm robot, dual-arm robot, and humanoid robot, they all have cameras mounted at various locations that we can use them to backward derive the joint positions for them. And in fact, it's even possible to unify robots with human, because once human wears a head camera, the head motion becomes the same action space. And then to augment this visual observation with the capability for application that involves contacts with surroundings such as robots and human, we further augment the representation with grasp states so that we can model the interaction with objects. And another question that we had during the development is that does different modality work, and particularly action? Because this was never done before, and we are really eager to find out if physical AI foundation models actually should put action into pre-training, just the same as text, same as pixels. Action is another layer that we can further scale up. And this is one of the experiments that we studied where we find increasing the amount of action data and mixing that with training with video for generation does not hurt video training quality at all. But at the same time, the action metric, the lower the better, is continuously improving as we add more action data. So that gives us a lot of confidence for later development, and really putting action as the first-class citizen during our modeling. And we're pretty pleased to see the trend that was followed by many other companies, including Black Forest Labs, Flux Model, and et cetera, and that really defines a new era for physical AI action into pre-training modality.
15:08
Another advantage of the mixture of transformer architecture is the multimodal chain of thought, because naturally, given a user input, the reasoner can ingest it and start to reason about what it needs to do and what it observes as state, what's the underlying causality will cause the task to succeed or not, and what it needs to avoid and what it needs to complete. And all this latent reasoning actually can be heavy cached and sent to the generator to generate the final output. And the finding is pretty interesting. On a set of benchmark, which is the Robolab in this case, we actually find that turning on multimodal reasoning improves the task success rate, which is what most robot manipulation companies care about, by a large margin. It's not small, by a large margin. And it really shows that the intelligence of the reasoner can be passed through to the generation side via this chain of thought process, and steer the model to do the right thing. And what's even more exciting is that during inference time, it needs a lot fewer environment interactions with the multimodal chain of thought, and in fact, 2X fewer steps, as shown on the bottom left. And during training, we see also a great improvement in terms of convergence speed. We saw up to 4X compute saved to reach the same level of performance with this kind of training paradigm. So that was all about Cosmos and what we've been seeing and observing up to the generation three. And for upcoming release, we're releasing the next generation of Cosmos, 3.5, and we're designing two architectures. One is Ultra.
17:17
Ultra will be the first half trillion parameter model that we've ever built, finally bringing physical AI into the trillion parameter era so that we can scale up the intelligence of physical AI, and as well as a lightning model. I know a lot of people care about real-time performance. A lot of applications require a small model to succeed. We'll try our best to build the smallest model ever built that retaining the reasonable quality. We'll push very hard for agentic, we'll push hard for multi-view, and we'll try to make generalization possible across multiple embodiments and make the model learn in context and adapt, and take actions in the physical world, just like what we envision a physical agent would do. And a lot of the scaling for the Ultra model actually needs a good infrastructure, and that's what we've been collaborating with Lance together, bringing everything up into the Lance database From the raw data ingestion stage to the filtering stage, to the annotation stage, everything was done on this layer. And the most importantly, first ever, which win on thousands and tens of thousands GPU scale with Lance table directly. And the difficulty of a heterogeneous data access pattern, for example, VLM training, video training, and also action training, are all converted into this single data infrastructure layer so that we can continue to scale, continue to build our next generation model. For a lot of the things, we have open sourced on GitHub, under the repo Cosmos Framework. We are hoping that all the skills that we have defined in the agentic era, so that no one risk code anymore, and all the models that we have released can help people to train and adapt Cosmos into their applications, and then optimize the model performance, both in terms of the training and the inference at different scales, from edge devices all the way to cloud compute.
19:52
So it's been an amazing journey, and it's our pleasure to have collaborated with many different companies, research labs, and startups on fronts, both in terms of the capability definition, data curation, and application. And we have found evidence that Cosmos can serve as a better data engine, a better environment for a lot of the simulation cases for visual reality, and a better checkpoint for any post-training adaptation because the amount of data that we have seen. And internally at NVIDIA, we've been supporting applications such as visual search, NVIDIA Isaac simulation, closed loop simulation for autonomous driving, and also self-driving applications like AutoMayu and the robotics applications like Groot. So with this single architecture, we are hoping it will bring up the foundation layer of the physical AI and make everyone build their applications much easier with all our learnings and findings. So with that, I'll welcome Brian back to the stage, and thank you for the time.












