About This Talk
The last 10 years have shown that simply scaling FLOPS (Floating Point Operations Per Second) is an unbeatable recipe for rapidly improving AI. In spite of modern scale, FLOPS are like a gas. They fill up the volume that they occupy. Since available compute is always bounded, quality emerges as a complementary, and often orthogonal, improvement axis to quantity.
At Runway, the team recently built a system for scaling both the quantity and quality of its multimodal training data. The system uses multimodal embeddings, semantic deduplication, and large-scale similarity search to identify high-quality examples, reduce redundancy, and curate diverse training pools with strong cross-modal alignment. Central to this system is LanceDB, which Runway uses as both a storage and query engine for managing and searching large multimodal datasets.
This talk covers the architecture behind the system, how Runway uses embeddings and deduplication to improve dataset quality without sacrificing scale, and the engineering challenges encountered along the way. It also shares lessons from operating the system in production, where the current bottlenecks are, and what Runway sees as the next frontier for scalable multimodal data curation.
About The Speaker
Transcript
Show transcriptHide transcript
00:01
Hi there. So today I'm going to talk about scalable multimodal data curation, which is a giant pile of buzzwords that you will probably hear many more times before today is over. Specifically, I'll be talking about how we curate data at Runway, where we train our own foundation models from scratch to simulate the world. So historically, these models have been just generative video models, classical generative video models, so to speak. But nowadays, they cover all sorts of applications from generative user interfaces to controllable interactive worlds, all the way to becoming policy models for actual robots that interact in the real world. But specifically what I'll talk about today is how we curate our training data for training the core video foundation models. And if you look at some papers from the past two years, maybe we don't actually need to curate our data. From both diffusion models to LM papers, if you're bitter lesson pilled, then arguably it is better to train on more noisy data than to curate your data. But of course, from the bitter lesson, this is assuming that we have sufficient compute, and unfortunately I do not, and I doubt that any of you have fully sufficient compute. And one of the reasons is for video models, if you look at any paper for a generative video model over the past couple of years, you will often see a Sankey diagram like this that shows how the data gets curated through different stages from pre-training to mid-training to fine-tuning. And the reason this data gets curated is because of math. See, the cost to compute a single data sample is proportional to the number of pixels, let's say, in an image or a video. And as we move up in resolution, pixels scale as the square of the resolution. But it's even worse than that. So transformers scale as the square of the number of pixels in terms of the compute cost or the flops.
02:11
And so we have a mathematical cost that scales as a power of four, which is pretty bad. To put it in another way, your throughput for a 1080p video is 1.2% your throughput for 360p videos. And so it's just entirely intractable to train fully high-resolution video models nowadays. And so we need to curate our data. Oftentimes, when people are thinking about these different stages of training, they have different approaches to how they handle this. So you might have pre-training at the top, which is more mass production. We just need to get things out the door. Maybe quality control is lacking and things like that. We can move down to small batch manufacturing for mid-training, and then maybe for fine-tuning, we are composing artisanal, locally sourced data points for that. And usually people have these different approaches to this, but we have a little thing that we've been sipping on the side, which kind of allows us to use some of the same processes up and down the stack. And that is what we all love here, which is a little Lance elixir. And so what I'll talk about today is how we use Lance up and down the stack at every stage from pre-training to mid-training to fine-tuning, and how this actually helps to accelerate what we do a bit. But before we do that, I need to give you a crash course in how to build a generative video model. At the core, all you need are examples of videos and then captions or prompts that describe what go on in the video. If you have these two data points, then you can train a generative video model. And so nowadays, the easy way to do this is if you have a bunch of videos, you can feed these videos to an LLM, and you can ask the LLM to write a caption for your videos. Once you have these data points of videos and captions, you can then invert this process by training a diffusion transformer to go from caption or prompt back to video.
04:21
There's a lot of other data that's involved with this, but this is the core of what we're dealing with. And so when we do this for pre-training, we store pointers to all of our videos, and we store captions in one giant Lance table. Very big Lance table. And for pre-training, we like this because we need to train on often almost all of our data. And so during pre-training, we might be covering a large portion of this table, and we need fast random access because we need to distribute videos to different ranks in our training process. And we need to randomize our data as well. And maybe one thing to call out is that we data load directly from the Lance tables. As we move into mid-training, we do this by adding columns onto our Lance tables. Maybe we want to now train on only high-resolution video. We can add a column that indicates whether or not a given row is HD. And then we can now run basically a SQL WHERE clause in order to filter on this table and just send relevant rows to our different ranks for mid-training. And then finally, during fine-tuning, perhaps we have a bunch of other quality filters that we want to add. We can again add these as columns to our tables. We can again run our filters and use the exact same process for data loading that we were using at pre-training scale all the way down at our artisanal fine-tuning scale. It might seem kind of silly for us to use this big, heavyweight pre-training technology, even down at the fine-tuning scale. You could argue that if we really care about efficient data loading, then we should build a very tightly optimized frozen data set that will data load very quickly, and that will be very strong and robust and scalable and everything else.
06:28
And while that's true, I work on a research engineering team, and I think of my team as like the Federal Reserve. So it's like we have a dual mandate, and our dual mandate is that, yes, we should build very strong, robust, scalable systems, but we also need to provide maximum flexibility to the researchers. And the reason I think that is that I want to lower the barrier to the researchers iterating on data because the best gains that we see are from iterating on data rather than futzing with model parameters and things like that. And by trying to use the same system for pre-training and fine-tuning, we're trying to navigate that balance of scale and flexibility. So what does this look like in practice? How do we build these columns that I'm talking about in order to be able to filter our data? To do this, we build the world's simplest DAG that you've ever seen. There's three nodes. It's a line. And so we start with a source of data. This is the data that we want to read. We can then pass each row of this source through a user-defined function. Maybe we are generating a caption, maybe we're calculating a quality score or something like that. And then we want to write these results back to some sync. What this ends up looking like for a Lance table is that our source is just a Lance table. We pass every row of the table through a UDF, and then finally, we write the results back as a column in the Lance table. Fairly straightforward. In practice, we have very large tables. We might have very slow UDFs that we want to run. We want to run at very large scale with very high concurrency. And so in order to do that, we need to paralyze everything that we do. And so we end up having this notion of chunks where we chunk up our data in order to partition it. A chunk for us is a subset of a Lance fragment.
08:33
We then send each chunk to a Ray actor. So we actually use Ray for orchestrating this whole system. We use actors so that we can amortize the cost of instantiating classes. So it takes a while to download a large LLM and load it into memory. And then we need to send the results of the actors to partitions where we will write the data. Sometimes our chunks are not perfectly aligned with our partitions, and so we need a coordinator or a head node to manage this shuffling. So fine. Again, still a pretty straightforward DAG. You might look at it and wonder, "Isn't this two lines in Spark? Why am I up here talking about this?" And it might be two lines in Spark. I've been avoiding Spark for 10 years, and I don't plan to stop. But I think that what we have here is maybe a little bit more complicated, and one of the reasons is that at the scale that we operate at, a lot of things fail. So we discover quotas in clouds that we didn't realize existed, and we start getting rate limited and things like that. We also run a lot of our compute on spot instances, which gets preempted, but it's cheap. And so we have to build a system that is very idempotent and robust to failure. And in order to do that, we end up having to unfortunately reach for one of computer science's hardest problems, which is the cache. And this cache ends up being pretty complicated and means that we can't really use off-the-shelf tools. The reason the cache is complicated is that it has to talk to every part of our system. Our chunks need to talk to the cache that they know what data has already been processed so that they don't bother to load that data. Our actors need to talk to the cache so that they can checkpoint their progress against it. And then finally, our partitions need to talk to the cache so that they can actually get the data that they need to write. I would just like to stress that this cache ends up being pretty complicated.
10:37
Because everything is coupled, you have all sorts of configuration that can really affect the throughput of your system. If you cache too frequently, then you actually pay a large cost in reading from the cache upfront. If you don't cache frequently enough, then you lose progress. And it turns out that the rates that you need to do these things are all dependent on each other. So a UDF that is very slow will need to be cached more frequently than a UDF that is very fast. This all depends on the rate that your instances get preempted and everything else. Thankfully, if you kind of probe claw it a little bit and give it a hint that spot instance preemption is like a Poisson process, you can get out a nice closed-form solution that allows you to optimize all of these different parameters, and that's what we've done, and it ends up having a very large impact on our throughput. Okay. Let's zoom back out a little bit. What was this talk about again? Scalable multimodal data curation. I feel like I've maybe beaten it to death that our system is scalable. We spent a lot of time thinking about that. I think I maybe glossed over the multimodality. Everything I said was just talking about video models, but actually nowadays, we train these any-to-any models, so we can have all sorts of modalities as our inputs. We can have text, image, video, audio, other sorts of things, depth maps, et cetera. And we can also generate all of these modalities as outputs. And so in fact, everything I'm talking about here, you should consider it to be modality agnostic or even mixtures of modalities. And again, having heterogeneous data, pretty nice fit for Lance. And then finally, there's curation. I've just kind of had these like is high quality columns that I've labeled in this talk. That's all pretty vague, and so how do we actually decide what data we want to train on at these different stages? So let's say we want to move to mid-training, and we now need to filter our data set down a bit. What are we going to do?
12:44
Like everything in programming, this is a compromise that we have to make. And so for us, this is a balance between diversity and quality. What I mean by diversity is that our models only know what they have seen in the data. And so if there are certain concepts that our models have never seen, then they're not going to be able to generate it. Like for example, we once had a model that had never seen a crab walking on the beach, and when you would prompt for a crab walking on the beach, it was hideous and terrifying. And it turned out that we needed to provide examples of this. But it's hard to know what are all the possible concepts of the universe that could be relevant for the model. Quality is fairly straightforward. There's high-quality videos, low-quality videos. You can kind of see it when you look at it. Maybe the range of concepts that are covered by high-quality videos might be fairly narrow, and so we need to balance between these two. How do we balance? We try to strike a Pareto frontier of what we're doing. Is what we're doing actually the Pareto frontier? I don't know. It's as good as any AI paper that you read and whatever they end up claiming. But we think it's working pretty well for us. And so how do we do it? We draw inspiration from this fantastic paper from FAIR from a number of years ago called SemDup. This stands for semantic deduplication. It's fairly straightforward. You first need to calculate embeddings for all of your data points. Nowadays, we have all sorts of embeddings. We have multimodal embeddings. We have embeddings of joint video and audio and text and everything else. And so kind of whatever modality you have, you can probably find an embedding for it. Once you have your embeddings, you then want to throw out any data points that are too close to each other in embedding space. If they're too close to each other, then they probably represent some duplicative concept. In practice, it is totally intractable to do this for very large data sets if you want to do pairwise comparisons. And so what we end up doing is we run K-means over our data.
14:49
We calculate K-centroids. You can think of this like clustering your data. And then for each cluster, we only consider the pairwise comparisons within that cluster. That makes the problem tractable. It does make our data processing more complicated, but I'll get into that. I find it's very good to actually look at your data. Sometimes researchers are allergic to this, but it's a good way to make sure that what you're doing actually makes sense. And so here are some stock video footage from videos that were in the same cluster and were deemed to be semantic duplicates. And I feel like this gives us two things. One, it shows us that our embeddings are capturing actual semantic meaning. I feel like these look pretty similar to each other. It also tells us that our deduping algorithm is working, and so that's good. But now imagine that we can only keep three videos from this cluster. Maybe that's our limit of data that we can handle. We can try to see what other samples from this cluster might look like. And if you imagine that we could only keep three samples, I would argue that keeping the three samples that I've highlighted here would be a better usage of our compute budget than randomly selecting three samples and maybe accidentally grabbing three very duplicative samples. And so this is again kind of a sign that things are working, our algorithm makes sense, and we're capturing this semantic meaning. But let's say that we still have too much data. We've semantically deduped the data. We need to curate even more. What do we do? One thing you can do is you can first run your deduplication algorithm, and then for whatever samples are left over, we can rank these samples based on some quality score. So there's simple things you can do, like the alignment between caption embeddings and the modality embedding, and then there's more complicated things that we can do, too. Once we have ranked, we can then choose what samples we want to keep. Perhaps we do a weighted sampling. Perhaps we apply a threshold or something else like that.
16:52
But all of this becomes available to us. What does this end up looking like in practice for our actual DAG? The first step is that we fit K-means on our data, and then we want to run one of these pipelines in order to assign every sample to a centroid as a column. This is just a single column in Lance. We are going to need to group our data by the centroids. And for that, we're big fans of scalar indices, so we can add a scalar index so it's faster to run these sorts of filters. This is where my framework or abstractions come into play. We now have a different type of source that we want to read from, but we can reuse the same pipeline. We now have a notion of being able to read data from different centroids and just calculating UDFs on those. And so for each centroid, we can run deduplication and ranking on it. In practice, the population of centroids follows a power law, and so you end up with some centroids that are very large, some that are very small. We end up using heterogeneous compute for this, where large centroids we want to send out to GPUs, small centroids we send out to CPUs, and that ends up kind of navigating that for us. And then there ends up being a bit of shuffling so that we can finally end up writing back columns to our table where we have a column that indicates whether or not a sample would have been deduped, and then another column that indicates kind of the rank of that sample If we then apply scalar indices to these columns, what's really nice is that we now have a dynamically sizable mid-training dataset. So by just applying different thresholds to our rank filter, we can size our dataset up and down while ensuring this balance of quality versus diversity. And this ends up being basically available at training time and available for ablations. And then finally, we have fine-tuning.
18:58
For fine-tuning, we find it's really, really good to look at your data. It's one thing I would like to stress. And so we built our own little UI called Lens, which allows us to run SQL queries, vector search, full text search over Lance tables, and actually view the results of what we're seeing. And from there, we can manually go in and we can curate data, we can compose different collections, and then we can even automate the process of sending those curations out and writing them back as columns in Lance tables in order to indicate them. And so I feel like at the beginning of this talk, I tried to propose that there's these different levels of processing techniques that we use and everything else like that. But I feel like it was maybe a bit disingenuous. I feel like maybe what we're doing a bit more of is vertical integration. If you'll deal with me straining an analogy. So I feel like by using Lance in the same processes up and down the stack, we enjoy the economies of scale while still ensuring a bit of quality control on every last artisanal data point. And we do this through these principles that I've stressed a bit in the talk now. We like our data to stay put. Videos are very, very heavy. They take up a lot of space in object stores. They're very expensive to store, and we don't want to duplicate them. We don't want to shuffle them around in order to create different datasets. That's just not efficient. Instead, our different datasets are just views on top of the tables. We should maybe take a page from databases that have existed for a long time. There's a reason they have views. Maybe we should use that too. This data pipeline that I'm describing, this was not the first pipeline we built. This was probably the third we've built in the three years that I've been at the company. The first couple pipelines solved one problem very well. We ended up with a lot of configuration, a lot of if-else statements. But by refactoring things into a bit more of a composable framework, we're now able to solve these multi-chain problems while using the machinery that we've already built up.
21:04
And then finally, I painted this dual mandate of scale versus flexibility, but I would argue that by having the right engineering primitives and really solving for scale early on, this gives us the flexibility to basically be able to operate on whatever data we want, which I think is true flexibility. And so maybe these two things go hand-in-hand. And that's all I have for you today. Of course, we're hiring. And thank you for your time.











