About This Talk
AI training goes beyond one process. The same multimodal data moves through repeated cycles of curation, deduplication, feature generation, search, pretraining, and iteration, with each stage having distinct compute and query needs.
In this session, Jack Song of Microsoft AI will share how Lance serves as a common data foundation across the flywheel, keeping large-scale training data in a single durable, evolvable layer while different workloads operate on it in ways that best fit the job.
The session will look at real workflows spanning multimodal dataset preparation, deduplication and decontamination, feature pipelines, generation of pretraining-ready data, high-throughput training reads, and low-latency search. Rather than coupling the data to a single processing or query engine, teams can bring the right compute to the same underlying data as requirements change.
The result is a tighter data flywheel to curate, enrich, search, train, learn, and feed those outputs back into the next model iteration without rebuilding the data layer at every step.












