About This Talk
Multimodal AI is moving quickly from models that understand individual modalities toward systems that can reason across images, video, audio, language, and the physical world. At the same time, the way these models are built is changing as datasets grow richer, research becomes more iterative, and the boundaries between data, models, and infrastructure continue to shift.
This opening keynote will look at where multimodal research and model development stand today, the ideas shaping the next generation of systems, and the challenges researchers face in turning new ideas into working models. It will explore how these shifts are changing the infrastructure beneath AI, particularly as teams work with larger multimodal datasets, run more experiments, and need tighter loops between data curation, feature engineering, retrieval, and training.
The session will also lay out LanceDB’s vision for that future and where the platform is headed next. Focusing on the infrastructure that makes datasets easier to search, transform, curate, and train from, the keynote will show how LanceDB is evolving alongside the next generation of model development and set the stage for the conversations ahead at Reverie.
About The Speaker
Transcript
Show transcriptHide transcript
00:00
In 2018, two researchers, David Ha and Jürgen Schmidhuber, asked a question that sounded like science fiction: can agents learn inside of their own dreams? So they gave an agent a small world model of a video game, one where monsters throw fireballs and the only goal is to dodge them. This agent trained entirely inside the model's dream of that game. Then they put it back into the real game. It dodged the fireballs well enough to solve the task. In that setting, the answer was yes. An agent could learn inside its own dream, and that dream was 64 by 64 pixels. Pretty small dreams back then. That's what Reverie means. Reverie means it's a daydream. What are those dreams made of, and what do we need to do and build to make them better? Now it's eight years later. The dreams look like this. Genie-04, for example, generates worlds you can explore and reshape in real time, with sound and characters that respond as the scene unfolds. World models are also helping robots plan actions. V-JEPA 2 watched more than a million hours of internet video. It's a lot of AI cat videos. Then with less than 62 hours of robot videos, it planned pick-and-place tasks on robot arms in two labs without collecting any data in those labs. We've moved from learning inside a simple game to using models of the physical world to plan actions. The next challenge is making that work reliably. Vision, sound, language, and actions are all coming together in unified systems. The world itself is now the interface, and the training, the signal. One super interesting recent development, many of you I'm sure have seen, is the evaluations of Astra that showed really strong robotics benchmark results from a general purpose model.
02:16
Some have called it the ChatGPT moment for robotics or bitter lesson playing out in robotics. I will be supremely disappointed if there aren't lots of spicy takes throughout the day that you're here. Regardless, that progress gives us a stronger starting point, and the next question is how to build on it for tasks that are more challenging and require more precise controls. So let's say you have a robot. It can pick up ceramic mugs, but struggles with transparent plastic ones. What would help it improve? Well, you need to compare its successful and failed attempts, connect what it saw, what it did, to what happens next. That gives you evidence to form a hypothesis. Then you can test whether changing the training signals actually makes a difference and improves the behavior. Which experiences should it learn from? How quickly can you turn those into an experiment? So tonight when you fall asleep, your brain will replay pieces of today. Hopefully, it will be good things you've learned at Reverie. That replay is how new experiences become memory. If you've seen the recent nature study, when researchers actually interrupted replay like this in sleeping mice, the mice actually had trouble remembering what they had just learned. So experience alone doesn't teach, but what gets replayed actually does. In our world, that is data curation. So models benefit from really carefully chosen experiences, too. It's not just sleeping mice. If you were at ICLR this year, researchers presented a method called DataMill. The goal is to teach robots specific tasks, like putting bread in a bowl or placing a napkin in a drawer. For each experiment, DataMill selected about 1% or less of a large open robot dataset. Researchers then combined those examples with a few demonstrations of the larger tasks when they fine-tuned the model.
04:28
Across four real-world tests, average success reached about 61%. The best previous record was about 40%, and they both use roughly the same number of training steps. So choosing better examples really help robots perform those actions more reliably, and that makes which examples do we actually train on a serious engineering question worth prioritizing. Now, if you think about these examples, if we were to do a full performance review of your day every night before bed, that sounds miserable, but fortunately, models don't feel misery. We can keep asking which examples would teach them something useful, then test the answer. And then choosing what gets replayed becomes just a new engineering discipline. So the best models all now come with a data engine for that purpose. For SAM 2, Meta put the model inside the annotation loop, and labeling ran about 8.4 times faster. Each failure tells you what to collect next. Now agents are joining the loop for research. In that AutoResearch famous report, an agent edits the training code, trains for five minutes, keeps or discards that change, and then repeats. About 100 experiments while you sleep, and you wake up to a log. For a researcher, that's a dream. That's the training run you checked on this morning, maybe. But for data infrastructure, that can be a nightmare. Your research agent can queue up 100 experiments in parallel, but If each experiment requires manual babysitting of data pipelines, you're still bottlenecked on human attention. This is, I guess, Amdahl's law applied to research experiments. So let's take a robot that keeps missing these transparent plastic cups. Find transparent cups is an image search problem, but finding the seconds before the failed grasps with the camera frames, gripper state, action, and outcome all aligned is a training data problem.
06:37
So a perfectly clear image of a successful grasp might be easy to find, but that actually might be the wrong example to add. You need the episode around the failure, then you need to freeze the selection so you can reproduce the experiment. And you want to be able to keep neighboring frames from the same episode out of your held-out evaluation for contamination. Otherwise, you can keep benchmaxing without getting a better robot. So that's the loop that I care about. You observe a failure, you assemble the right evidence, you run a controlled experiment autonomously, and you measure what happened and what changed. The critical metric here is how long this loop actually takes. I think today, a lot of these sit in different systems, and a poor infra engineer has to write pipelines to stitch them together by hand, and babysit data syncs and pipelines, and be there to deal with failures. While all your GPUs are waiting to be fed for the next run. So I guess this is my obligatory section to shill about LanceDB. So we think that the data infrastructure should support the way research actually happens. You or your agents find a failure, form a hypothesis, we curate the next training dataset, and run the next experiment, and that should be a painless and fast workflow. That's what we're building here. It's a multimodal lakehouse built on open-source Lance format. You have your raw data, embeddings, metadata together in a unified and versioned table, and you can find the examples that matter and keep track of which data each experiment used. Several teams you'll hear from today are already building on this. So for every research idea you have, your agents can come up with 10 variants. You should be able to run them all in parallel without babysitting 10 data pipelines. You can point LanceDB at raw robot logs in an MCAP, and they become searchable episodes with camera sensor streams all aligned on one timeline.
08:47
We give you interactive search and curation at petabyte scale on the same table you're training from. With LanceDB feature engineering, your summarizer or label judge just becomes distributed resumable backfills. Your recovery strategy when you're launching your agents overnight can finally be something other than, "Please don't fail." So I think what we do is we're not trying to compete with your agents. What we're doing is we're giving your research agents reliable primitives. Fast reads, incremental updates, checkpointed execution. It means a lot less pipeline code and complexity for your agents to have to reinvent and debug, and more experiments they can carry on autonomously while you're asleep. So that's the direction we're building towards. All the relevant data in one place. People and research agents work well together on the same data with a record of which examples each experiment used. The test is whether the robot handles cups it hasn't seen before, and then we can say, like the knight in "Indiana Jones," "You grasp wisely." I wanted one of those robots to melt like the Nazi face, but yeah. So in 2018, an agent learned to dodge fireballs. Today, models dream whole worlds, but need to act in this one. So what will AI dream next? Well, that depends on what we give it to learn from. As you listen to all the talks today, think of the questions you'd leave running with your agent tonight and what you'd need to trust the answer come tomorrow morning. Models have learned to perceive the world, and the next era belongs to the team that turns our raw experiences from all over the world into the right training dataset. But we can build that world together. Thank you.












