From Raw Driving Logs to RL Training: A Petabyte-Scale Lance Pipeline in Production

Autonomous Driving

Data Curation

Model Training

Follow raw driving logs through Applied Intuition’s petabyte-scale pipeline as Shiyan Xu traces their path into searchable, versioned data for hard-example mining and RL training.

From Raw Driving Logs to RL Training: A Petabyte-Scale Lance Pipeline in Production

From Raw Driving Logs to RL Training: A Petabyte-Scale Lance Pipeline in Production

Autonomous Driving

Data Curation

Model Training

Follow raw driving logs through Applied Intuition’s petabyte-scale pipeline as Shiyan Xu traces their path into searchable, versioned data for hard-example mining and RL training.

Thank you! Your recording is ready.

Your submission was successful. The video will appear in a moment.

Shiyan Xu
Software Engineer, Data Engine Team
Applied Intuition

About This Talk

Training autonomous-driving models means continuously turning raw driving logs into high-quality training data as new data arrives, quality filters evolve, features are added, and hard cases are discovered. At Applied Intuition, Lance is the backbone of that production loop. Raw multimodal driving logs flow from Applied Intuition’s internal lakehouse into curated, versioned Lance datasets that can be updated as the definition of useful training data changes. Indices make those datasets queryable for hard-example mining and targeted scenario discovery, while training dataloaders random-sample directly from the same tables, including large multimodal payloads, without creating separate copies for each stage.

This talk walks through the pipeline end-to-end and shares lessons from running it in production. It covers the architecture behind the system, why Applied Intuition chose Lance for petabyte-scale multimodal data, and what it takes to keep the loop fast as datasets and teams grow. It also covers the production lessons that mattered most, including the design decisions that paid off, the bottlenecks that surprised the team, and what could be built differently today.

About The Speaker

Shiyan Xu

Shiyan Xu is a software engineer at Applied Intuition, where he architects data platforms and pipelines for training AI models on multimodal data at scale. Previously, he was a founding engineer and engineering lead at Onehouse and led the team behind Zendesk's petabyte-scale data lake platform. Shiyan is an Apache Hudi PMC member and lead author of O'Reilly's Apache Hudi: The Definitive Guide. His expertise spans data platform architecture, distributed data processing, and engineering leadership.

Transcript

Show transcriptHide transcript

00:01

Good morning. Welcome to my talk. How do we transform raw logs into something that self-driving model can train on and get improved? And how can we keep improving this, repeating this process so that model gets better and better? In this session, I'm going to walk through and share some real-world implementation of how we process raw log into something that makes the better model, and share some high-level design considerations and learnings, challenges, when we deal at a large scale. My name is Shiyi, and I work at Applied Intuition. At Applied, we're building autonomy softwares for cars, trucks, mining machines, robotics, and all the simulation tools and stacks. In short form, we are building physical AIs that moves the world. A quick introduction about me. I work at the data engine team at Applied. My responsibility is across architecting data platforms, tuning data pipelines, building tools and services for users for quick, efficient access of the data. I've been also doing a lot of open source contribution. I've been working as Apache Hudi project management committee member for a few years. I also published a O'Reilly book, "Deep Dive into Hudi" last year. A quick flash out of the agenda. We're going to look at the high-level data as an overview, and also going through those stages of the journey about the training for the data and the training. Then we will cover some challenges in the end. First off, let's look at the data, the scale. At Applied, we store several hundred petabytes of data across multiple verticals, as I mentioned previously, and we are accumulating more than one petabytes every day, and the data is multi-model.

02:09

As you can imagine, there are camera recordings, there are lidar, 3D point clouds, and there are radar detections, vehicle signals and poses. So all these are very valuable assets that we want to make the best use of for our models. So how do we do that? We build something called Data Flywheel. A Data Flywheel is a loop. We take the raw logs from the vehicles, the driving fleet, we process them, we transform them, and do enrichment and prepare them for training. After the training loop, it produce a better model. A better model that can pass evaluation and may still struggle with some real cases, like hard cases. We put them on the road and then collect more data, and especially for those real hard cases, and we feed them back into the loop so that improves the training for the next time. So we keep this Data Flywheel spinning, and then we keep improving the model over time. So I've broken down this flywheel, this loop, into seven stages. First stage is about ingestion. The raw logs keeps getting appended into the vehicles, the local drives, hard drives, as they comes in. So you can see they are all interleaved, like cameras, lidar, radar. They all just append to the disk without any extra manipulation. So when we ingest this, the principle number one is think about how do we consume this data when we design the storage layer. So the further step is really about consuming the logs in some way that we satisfy the further steps. So one key aspect of using this data is about playback, streaming read. So the simulator would play back, it's essentially a sequential read of all this data.

04:20

And there's also a need of a point lookup or range lookup about this data. So given a point of time or a range of time, give me all the data that across all these sensors. So this requires some efficient lookup components. So what we need to do is we need to split them based on the category. You don't want to mix camera data with lidar data, for example. They should be laid out nicely based on their own category. And we need to build a time index component on top so that satisfy the lookup needs. So that's not all about transformation. So we still need to do some time alignment. As you can imagine, all these sensors, they have different clocks. They all run their own frequencies. So they all just keep appending with their own timestamps. But think about, again, it's the same principle. Think about when we want to reason about this data, when the model, the training program, when they need to load this data, the program will just need to look at the timestamp and say, "Give me everything that happens at this moment or during this time window." So, I don't want to stitch all this data from all these sources together at the training time. I want everything being lined up well for me. So that's what needs to be done at training time. So that kind of alignment needs to happen before the training starts. So we need to align them based on timestamp. We're using some concept called frame. So this frame has its own frequency, has its own clock. This is something that we built as a virtual concept of model to have all this data to be snapped together and then ... to the nearest data points. So they can be processed in a unified way. And we can store them in some table, and then we can still keep the raw bytes image in the log store that we ingest into, so that we don't need to copy the data twice.

06:25

So that's the end of the boring part of the data processing. Now let's move on to something more interesting. It's about enrichment. So we have the raw data. It's a bunch of pixels from the camera, and there's not much interesting thing happening to it, so we need to enrich them. The two main categories, first one on the first line in the diagram is about the metrics. So the metric is something that you can derive using some functions. For example, you can detect if this is a nighttime drive or if it's something about the quality. For example, for some intervals of the data frame, or intervals of the training data range, how many frames it has, whether there's enough number of frames, that indicates whether there's some data get lost during the upload. This is some quality indicators about these metrics. So that's something we want to capture when we ingest the data. So that's very intuitive. We could keep adding more metrics, but there's another big aspect of enrichment is about labels. This is not something you can easily derive based on writing some functions. You need some recognition model like Auto-Labeler, using some Auto-Labeler model to keep scanning the data and then be able to recognize some objects from the frame, for example. So this by itself is actually some mini data flywheel inside the big data flywheel where we are going through. So the model itself needs to be improved over time, and because of the vast amount of data, we cannot do human labeling for this amount of data. But we need to also improve, get the better Auto-Labelers that gives us better results. So we need to build some kind of a small data flywheel in the lower part of this diagram. It runs on the loop, and we need to have a human label to help adjust and improve its quality. So with this kind of input enrichment, additional data, we should be able to capture into somewhere, I call the curation table. So the curation table, the name suggests its purpose about the next stage we're talking about.

08:53

It's about curation. Essentially, it's like a manifest of everything so that we can look at the table and find what we want. So now on to curation. So curation sounds very simple, right? It's just like, given these criterias and given these conditions, I want this kind of data to be fetched out, and so that I can send for training. So you can do some basic curation with some criterion defined, and then when it's time for training, you trigger this, and it prepares the data for training. But there's some caveats that you will always end up finding some real scenarios that is hard to be described and to be defined by a fixed set of labels or metrics. Those real conditions or real scenarios that usually you need to set up some natural language search that can describe the scenarios more specifically, and you need to develop this kind of natural language search between text and video so that you can mine the videos out from your curation table and then be able to get a specific set. This is very critical and very helpful for model fine-tuning. Now, with the curated data set we have, we call something that curated segment. So the segment is essentially some time intervals indicating that these are the data that I want to prepare for the training. It does not have the data yet. So at this stage, scenarios, it's about assembling this data for training. Because scenarios will correspond to the same time segment that we have been created. The difference is scenario data will contain all the data that we have collected so far, like the metrics, the labels, and all kind of map information should be all assembled together and then batched and be prepared for training.

11:09

So there's something I want to deep dive a bit on how we design this layout for efficient training process. Remember, the principle that we talked about earlier is about when you design something for storage layout, you want to think about the consumers. Design for your consumers. Design for how you want to query this table efficiently. So it's about the data pattern, the query pattern that you want to perform. So here we choose lines as the table format for scenario tables. So there are a bunch of advantages. We found it's very valuable for making this scenario table easy to consume. Of course, it also helps with writing the scenarios to the table. There are lake house primitives like insert, update, delete, and have versioning be able to do time travel and pin to the specific version of the table so that you can trace back to the training result, tied back to the version of the data that you trained with. And you have primitives like compactions to help optimize the layout. You have cleanups, reclaim the space. These are all primitive operations in a modern lakehouse format. What's more shining about using Lance here is, first is the blob API support. The blob V2 API from Lance helps us to manage the blob files nicely and easily. You don't need to worry about having a lot of small blobs need to be packed into some bigger files to reduce the small files number. The blob API from Lance can easily pack those files for you by simple setting a config. And of course, if you have external bytes that stores outside of the table, it can also be easily configured as external blob kind that you can point into that. That works out-of-box, just by setting some configs. The second advantage I've seen using Lance at this stage is the data evolution support. So from time to time, we feel like we need to add more data, more columns to the table, like the metrics. We develop some new useful metrics, and we want to just add to the column, and it happens from time to time. And we want this process to be very efficient and easy to work with. So Lance has this columnar support that you can simply add a new column, and it does not require you to rewrite a lot of files because the new columns data can exist on its own. And this makes the adding a column and backfill a really smooth process.

14:00

There's a third biggest advantage of using Lance here is about reading part, which is the next stage about data loading. So before diving into that, I want to start introducing what the data loading is. So the data loading at this stage is essentially the reading side of the training. So before training starts, it needs to convert the data that humans can understand, like the ints, the strings, and all these Booleans values into something called tensors. So we want to basically transform the scenario table into something like tensorized data. So that is ready for input, fit into the GPUs for training. So what actually happens in this workflow is, with scenario data set, the training actually just keeps loading random sample, take random samples from the table. There can be hundreds of millions of rows in the table. And the random sample just happen in a loop so that you can imagine there are hundreds of millions of read processes, read operations happen on this table. So the key is to be able to really do this efficiently. So that's the biggest advantage of using Lance here is Lance has designed a row ID that is used for sampling. With the row ID, it also comes with a row address concept. The row address concept allow you to identify a specific fragment, which is a logical grouping of the Lance table on storage, and identify the fragment and then identify the row offset in that Lance file. So the Lance file has the row offset. The Lance file format also be able to identify based on the row offset to translate to the row address of that bytes that you need to fetch. So the advantage of doing this is really about fetching only the data you need. So instead of fetching a data block for typical storage layout, you would just need to fetch the bytes that actually correspond to the row.

16:21

So this actually saves a lot of bandwidth and also a lot of time of identifying the data that you want to fetch. So as you can see, this is sitting on the hot path of doing data loading for training. Now move on to the training. So we have two stages. So first stage is about imitation. The imitation is about teaching the model to drive like a human. So we have all the human driving recordings, and they record all the sensors and the trajectory of the path, the driving logs, and we just want to play back and feed to the model for the training to see how would the model drive for a small fragment comparing to how the human drive, and keep comparing some episode after another episode. So with this kind of teaching, so model will be able to learn drive like a human. But that's the limitation of it. It only goes as far as how you collect the data from the human drivers. So that's like teaching some basic foundation for the model to learn to drive. So then it comes to reinforcement learning. So I can think of it like I teach the model some basic stuff, now let's show me what you have learned. So reinforcement learning happens in a simulator. So we want to see how the model behaves in the real world, but it's not time yet. We can't just let it out and drive in the real world, so the simulator comes into play. The simulator will feedback to the model's output and then simulate the trajectory, all based on the model's own driving, own controller output. So either when the model drives and crashed into some other objects, or it passes some passes. So that rewards or penalty goes back to the model so that it will converge and give the model more generalized capability of driving.

18:35

After these training stages, it produce a candidate which will be ready for evaluation. And the evaluation, I think of it as exam, right? You teach the model something, you give the model something to practice, and then now it's ready for exam. You want to make sure you don't leak the exam question into their practice question, right? So that's why evaluation set not leaked into the training set. So the evaluation, we will replay and be able to produce some results. Once it passes some baseline comparing with the candidate versus baseline, once it's passed, we are ready to deploy. Usually, we want to deploy with some staged rollout, like have some test drive limited and then be able to catch any accidents before you roll out to bigger fleet. So once the rollout happens to a bigger fleet, it goes out and collect more data. And again, when we see when there's some struggled cases, we want to specifically collect those data and feed back into the loop, and then that completes the loop and keeps improving the data. It keeps improving the model behavior. Some challenges highlights. As you can see, the whole Data Flywheel is a big system. They are multiple stages. They all coupled and depends on each other. For example, we talk about auto labeling loop itself is like a small Data Flywheel that you need to keep improving it so that the labeler model is accurate, and then that feeds into the driving model that can recognize the objects better. Obviously, there are some long tail coverage issue that you always find some real scenarios that there's so many real scenarios that you don't have enough data. And you need to be able to identify those gaps and go back to the road and then collect more data, and then you'll be able to improve on all kinds of scenarios for the model. A simulator's fidelity obviously is also important when you do the RAL. You want to have as close as to the real world as possible.

20:47

Evaluation coverage, like being comprehensive and be able to guard the release, probably be honest about the evaluation results. So you want to cover as many as possible as you expand the training data set coverage. Another big part is about efficiency. Obviously, we want to do faster iteration, get the result faster, get improved in a shorter time. All these components, all these stages that we have been touched with, all contributed to the overall efficiency. Labeling, obviously, we want to have the labeler to be faster and better. And write versus read performance, this is a typical trade-off. When you want to process the data faster, you would fan out to more writers, but more writers means you write more small files. And small files hurts read performance. You want to kind of balance this as a trade-off, and running compaction for a Lance table asynchronously is obviously a very effective approach to address this trade-off. Data loading throughput, we talked about this with random sampling. We should do it really fast because it's on the hot path of data loading. And some key takeaway is also after data loading, you send the GPU to kick off the training process, but you don't want the GPU to be waiting for long because that's just burning money. So you want to kind of improve the overall throughput and also at the same time consider the training programs. You want to do some asynchronous arrangement for the data loading and the training program so that you just keep GPU running, not idle for long. And that is the overall challenge highlights. To recap, what we built is a Data Flywheel. The Data Flywheel will keeps capturing new data from new drives.

22:52

The drives gives us more coverage, especially for those failing cases and disengagements. We also label those raw data. We want more better quality labels, and the labeling loop itself improves and give us all this useful information, additional enrichment information. And the evaluation should be comprehensive. When we found some failures that keeps giving us disengagements, we want to expand the evaluation suite so to be comprehensive, and we using some sharper criterion to let the candidate to pass to release. And of course, when there are disengagements or failure cases, we want to collect those and target those specifically to collect more data and do more model fine-tuning. With that, we will be able to keep improving the data, keep the Data Flywheel spinning, and improving the model. Thank you. Back to you.

Related recordings

Cosmos: Omnimodal World Models for Physical AI

Max Li
•
Session
cosmos-omnimodal-world-models-for-physical-ai

Physical AI

Multimodal Data

Model Training

Scalable, Multimodal Data Curation

Ethan Rosenthal
•
Session
scalable-multimodal-data-curation

Data Curation

Multimodal Data

Search & Retrieval

Building the Data Flywheel for Frontier AI Model

Suqiang (Jack) Song
•
Session
building-the-data-flywheel-for-frontier-ai-model

Data Curation

Model Training

Multimodal Data

From Raw Driving Logs to RL Training: A Petabyte-Scale Lance Pipeline in Production

Shiyan Xu
•
Session
from-raw-driving-logs-to-rl-training-a-petabyte-scale-lance-pipeline-in-production

Autonomous Driving

Data Curation

Model Training

From Raw Delivery to the Training Loop: Luma's Data Platform on Lance

Haoran Cai
•
Session
from-raw-delivery-to-the-training-loop-luma-s-data-platform-on-lance

Multimodal Data

Data Curation

Model Training

Data Infrastructure for Robotics Foundation Models

Zhipeng Yan
•
Session
data-infrastructure-for-robotics-foundation-models

Physical AI

AI Infrastructure

Model Training

What Does "Now" Mean to Your Model? Time Alignment for Multimodal AI Data

Vasundra Srinivasan
•
Session
what-does-now-mean-to-your-model

Multimodal Data

Physical AI

Model Training

Scaling the Data Foundation for XPENG's VLA 2.0

Bo Tian
•
Session
scaling-the-data-foundation-for-xpengs-vla-2-0

Autonomous Driving

Model Training

Data Curation

Keynote | Rethinking AI Infrastructure

Gerred Dillon
•
Keynote
rethinking-ai-infrastructure

AI Infrastructure

Model Training