From Raw Delivery to the Training Loop: Luma's Data Platform on Lance

Multimodal Data

Data Curation

Model Training

duration
25
min

Haoran Cai explains how Luma AI uses Lance to normalize, enrich, and stream multimodal data into large-scale model training.

From Raw Delivery to the Training Loop: Luma's Data Platform on Lance

From Raw Delivery to the Training Loop: Luma's Data Platform on Lance

duration
25
min

Multimodal Data

Data Curation

Model Training

Haoran Cai explains how Luma AI uses Lance to normalize, enrich, and stream multimodal data into large-scale model training.

Thank you! Your recording is ready.

Your submission was successful. The video will appear in a moment.

Haoran Cai
Tech Lead, Data Infra
Luma AI

About This Talk

Luma trains multimodal and physics AI foundation models, and Lance is the backbone of the data platform that feeds them.

This talk walks that path and what we had to build at each hop.

Curation at scale: deliveries arrive in every shape imaginable and get normalized into a canonical schema. Distributed pipelines then enrich them, and results are written back as new columns instead of rewritten datasets, so a curated view becomes a query rather than a copy.

End-to-end automation: schema contracts, validation and orchestration turn work that used to take weeks of hand-holding into a job you launch and walk away from.

Performance: the things that actually break at scale, including layout and metadata overhead, cross-region reads against object storage, filtered reprocessing without materializing subsets, and a caching layer that keeps remote training clusters reading at near local-disk speed.

Dataloading: streaming samples straight out of Lance and keeping thousands of GPUs saturated.

About The Speaker

Haoran Cai

Haoran Cai leads Research Data Infrastructure at Luma AI, where he architects the unified data platform powering state-of-the-art multimodal foundation and physical AI models. His team handles the end-to-end data lifecycle at scale — from processing and enrichment to high-throughput streaming to thousands of GPUs. Previously, he led the AI data platform team for Adobe Firefly.

Transcript

Show transcriptHide transcript

00:02

Good afternoon, everyone. This is Haoran from Luma. I'm the tech lead for data infra team at Luma AI. So today, I'm going to talk about the data platform we have been building at Luma AI, that we will touch on the data from the moment where we get data from the vendor and all the way to how data is efficiently feeding into the GPU on each of the rack. Let's get started. So the road for today is we will talk about the overall platform, and then I will follow the data from ingestion to enrichment to data loading, which is training, and then we touch base a lot on the tooling side of the things. The mission at Luma to scale up the data platform is in different folds. Number one, we need to scale up the volume. This means that every day we need to ingest new data coming either from our vendor or from ourselves. Furthermore, we need to scale up the feature enrichment. The researchers need to constantly test different new features, captionings, embeddings, and scores. Secondly, we need to scale up the modality. So at Luma, we are building multimodal foundation models. This means that the model is seeing image, video, robotics actions, and all kinds of text data. Lastly, for both batch inference and training, we need to make sure the performance is good. So the GPUs need to focus on the training and the inference instead of waiting for data. So all the platforms and optimizations I will cover in this talk is following these missions. For the platform, as I mentioned, the data follows through ingestion, enrichment and data loading.

02:14

And Lance is our unified lakehouse. It's the foundation for everything we do at Luma data platform. Across ingestion, enrichment and data loading, we also build a set of toolings to help our researchers to self-serve and inspect both the data and the training. Now starts with ingestion. So roughly speaking, we have three different data sources. The first one is we have data coming from our contracted vendors. They come at a scheduled sequence, sometimes daily, sometimes weekly. And also we have our in-house data, such as simulation data and teleop data. Lastly, we also use some open source datasets. And the open source dataset, as we all know, is very messy. It contains different file formats and different qualities. So in order to ingest all this different data from different sources into our training ready table, we have to ingest all this data in. So in Lance, we have defined a unified Lance table and a schema to put all this data into a unified schema. And depending on the size, sometimes the images and audios are directly embedded into the Lance table, and for videos, we put it on the cloud storage and have a pointer pointed to the videos. But the advantage of this unified schema and a unified table means that all of our data processing toolings and data loadings can refer to the same schema, no matter what data source it's coming from. That's a good design, but as we all know, it's hard to execute in real. So we have been working really hard to coordinate with different pieces. So the data ingestion is not a single directed path. It's actually involved three different parties.

04:22

Firstly, the data providers. Secondly, the training teams. And lastly, the data infra platforms. So as the center of this stage, for each new data coming in, we need to run data quality checks. Not only check the missing values, the range of the data, but we dive in and check the real quality of the data to make sure the data satisfy our requirement. And then we notify our data providers to find what's the gap between the expected data and the delivered data. Furthermore, we also get requirements coming from the training teams. They will inform us what's the next best action for us to talk to data providers. And we have run this automated pipeline through optimization and a lot of cross-org communication that now we can optimize that only after 44 minutes, a new batch of data when we receive from the vendor, we can get into a training-ready state. After the data is ready, ingested, the next big thing is from the researchers is that they want to enrich the data. In this sense, when you have a new vision language model, you want to generate a new caption on the existing data. And this is definitely the core piece of our data platform. And our in-house built offline data processing platform is the bridge to connect between users and our data compute So our users can use the unified command line interface as well as APIs from backend surveys or from the Dexter data pipelines. They will connect into our data platform and the front door will admit the jobs based on quota priority and the load for each cluster.

06:28

And generally speaking, our data computer clusters are splitted into different cloud providers and regions. And we split that into different piece. And high level speaking, it has some CPU job, we typically use for video transformation, filtering or coordination with inference providers. We have some small GPU queue that we typically use for computer embeddings and do short and boundary or span detection. And lastly, we have large GPU queues that is reserved for VRM inference, for captionings, stuff like that. And each data job inside the cluster is a dedicated Ray cluster. So each job has a dedicated Ray head node, as well as the Ray workers. And depending on the cluster capacity, the Ray job can auto scale. Last but not least, we have a centralized job status service that monitor all the jobs that gives feedback to the user, to the tooling, or to the agent. Now, let's dive in one step further to look at what actually happens in the data processing job. I think that is one of the things that Lance format really shines, because it enables us to quickly add new columns and new features into the existing large-scale datasets. For example, the researchers wants to select a subset of data, and to add a new caption column, caption V1. And in our data processing Ray job, the worker will just select a subset of rows and compute and run the VRM inference and get the caption. It will write down intermediate Lance table catered by the row ID. And then we will join back with the original source with just the new columns. As you can see here, the columns that are not touched by this specific processing job are not copied.

08:38

And for rows that are not in this job, we will just return new for the new caption column. So it's very lightweight and easy for our users to experiment different captioning and different enrichment signals. Prior to that, our researchers will think twice about, is it worth the time to do a big operation to try a new vision language model? But now they can just try it, and they can self-serve it with their agent. We all know that during a large-scale model training run, pre-training or mid-training, the data curation signals and captionings are co-located with training. We may need to update it and the new signals from the data, but we always need for more GPUs. So in Luma, in order to improve the utilization for all the GPUs, we also have internal spot market to run data inference jobs inside the training clusters. For example, if sometimes the training job does not use all the training clusters and we have some idle GPUs, our inference job can actually run on it. And then when the training job starts or there is another training run, the inference job has the lowest priority. The inference job will get preempted. But because we build a durable queue, so the progress is saved. And when the training jobs grow back, and then we can run more inference jobs. So in this way, our data inference jobs and training jobs can always fill up the hole and use all the training GPUs, even though the training job is not always running. So now we have the data ingested, and we have the features, captionings, and embeddings, and the data curated.

10:47

The next thing is how to load the data into the GPUs. And the key thing is efficiently, because we all know that during a large-scale training run, even one worker in one rank, if it's slow, it will slow down the whole training, and it will make the MFU and the GPU utilization much worse. So we definitely need a more performance data loader. And in Luma, we use data loading directly streaming data from Lance, which is stored in the cloud storage. So the input for the data is a mix of dataset. It will have the source table, along with the pinned version as well as the row ID. Basically, it's a subset of the table. So a typical data loading pipeline, we will directly stream the data from Lance, and then we will run video decoding or image decoding from frames to the bytes. We will run a lot of data processing and VAE encoding. And lastly, we will pack the data which contain a mix of data modalities in a pack and send it to the training. As we can see, a lot of times, because data loading demands for high performance and for video decoding and transformation, and especially for VAE encoding or text conditioning, it's very expensive, and it is competing for the resources for the real training GPU run. So what we do here at Luma is that we offload all this expensive computation during the training, offload that into a separated disaggregated compute pool, so that in the disaggregated inference pool, we run all these processing steps just in time, and we will return back the data into the pack.

12:56

And of course, when we build up this processing pool with scalability and stability, it does not stop with just video decoding and VAE. We can actually run any frozen forward model during the training on this disaggregated compute pool. So here is some results. So we have run this data loading and compute pool setup over a lot of large-scale runs at Luma for the past few months. And most of the time, the step time waiting for data on our training run is near to zero, and is consistently to be stable. So for some of our largest training runs, we have thousands of ranks fed by one remote data processing pool. And we have seen consistently that our data processing pool has been running for 10K plus steps without stability issues. And over the time of the day, we have thousands of data processing workers running in cluster and serve for multiple training jobs. Okay. So now with our scale and the complexity of the multi-model data in the pack, we really need to see exactly what happens inside the training. And this is super important for researchers as well, because the researchers will, of course, debug some of the training run during pre-training. They will see the loss curve is unusual or not aligned with the expectation. The first question they will ask is, what's actually fed into the GPUs?

15:02

Throughout the way, we have built a lot of tools to help researchers and also us to understand what's inside the GPU. For example, the users can see for each step and each rank what's exactly in the pack, so we can see the pack composition. If you click into that, you can also see the exact data inside the pack. It will see both the source table, the source videos, as well as the transformed videos, because sometimes in the data loading, we will resize or crop or doing some random augmentation for the video. And you can also see all the captioning and action data. Furthermore, you can also see for a specific step, for all the ranks, all samples for that step inside the pack. So this will help our researchers to understand if there are any unusual signals in the data, so we can go back to double-check the data. So all this, we can enable these features is because that in our data loading, we can do deterministically randomness. So we can fast and deterministically replay for a specific rank and a specific step what's actually inside the pack. So here is a real example. We all know that one of the worst things that are coming to large-scale run is you see a GradNorm spike, and mostly it comes with some bad data. And typically, what we do is that we just change the data seed and hope that the GradNorm spike is not coming back. But we can do more scientifically. So when we see in a certain step, we have a GradNorm spike, because we can faster replay deterministically, we can replay it quickly and look at all the samples and all the ranks for that specific step.

17:09

And we see some metrics is significantly higher than others. And then we know for the specific sample and a specific token, it may have some issue. Of course, we can see the specific videos and the frames and all the action data. And we find, oh, maybe this is a bug in the data quality, and we can go back to fix the data systematically. I think this is a more scientific way to debug the training instability issues. As we can see that at Luma, Lance is right in the center and the core foundation for everything we do in data platform. We are heavy Lance users. But besides that, our small team of engineers also actively contribute back to open source communities. Just for the last few months that we have contributed more than 40 PRs and involving all aspects of Lance format. For example, the stable row ID, and also the index build performance. We love Lance and love to work with the open source community as well as Lance, to further advance data infer for multi-model. Thank you very much.

Related recordings

Cosmos: Omnimodal World Models for Physical AI

Max Li
•
November 5, 2026
cosmos-omnimodal-world-models-for-physical-ai

Scalable, Multimodal Data Curation

Ethan Rosenthal
•
November 5, 2026
scalable-multimodal-data-curation

Welcome: What Will AI Dream Next?

Chang She
•
Keynote
opening-keynote-what-will-ai-dream-next

AI Research

Multimodal Data

AI Infrastructure

Cosmos: Omnimodal World Models for Physical AI

Max Li
•
Session
cosmos-omnimodal-world-models-for-physical-ai

Physical AI

Multimodal Data

Model Training

Scalable, Multimodal Data Curation

Ethan Rosenthal
•
Session
scalable-multimodal-data-curation

Data Curation

Multimodal Data

Search & Retrieval

Building the Data Flywheel for Frontier AI Model

Suqiang (Jack) Song
•
Session
building-the-data-flywheel-for-frontier-ai-model

Data Curation

Model Training

Multimodal Data

From Raw Driving Logs to RL Training: A Petabyte-Scale Lance Pipeline in Production

Shiyan Xu
•
Session
from-raw-driving-logs-to-rl-training-a-petabyte-scale-lance-pipeline-in-production

Autonomous Driving

Data Curation

Model Training

From Raw Delivery to the Training Loop: Luma's Data Platform on Lance

Haoran Cai
•
Session
from-raw-delivery-to-the-training-loop-luma-s-data-platform-on-lance

Multimodal Data

Data Curation

Model Training

How Adobe Built a Multimodal Lakehouse for AI-Native Experiences

Miao Wang
Jiayao Hu
•
Session
how-adobe-built-a-multimodal-lakehouse-for-ai-native-experiences

Multimodal Data

AI Infrastructure

Search & Retrieval

Data Infrastructure for Robotics Foundation Models

Zhipeng Yan
•
Session
data-infrastructure-for-robotics-foundation-models

Physical AI

AI Infrastructure

Model Training

What Does "Now" Mean to Your Model? Time Alignment for Multimodal AI Data

Vasundra Srinivasan
•
Session
what-does-now-mean-to-your-model

Multimodal Data

Physical AI

Model Training

Scaling the Data Foundation for XPENG's VLA 2.0

Bo Tian
•
Session
scaling-the-data-foundation-for-xpengs-vla-2-0

Autonomous Driving

Model Training

Data Curation

Keynote | Rethinking AI Infrastructure

Gerred Dillon
•
Keynote
rethinking-ai-infrastructure

AI Infrastructure

Model Training