How Adobe Built a Multimodal Lakehouse for AI-Native Experiences

Multimodal Data

AI Infrastructure

Search & Retrieval

Adobe evolved its enterprise lakehouse for multimodal and agentic AI, and Jiayao Hu and Miao Wang share how the resulting vector-native platform preserves the governance enterprises depend on.

How Adobe Built a Multimodal Lakehouse for AI-Native Experiences

How Adobe Built a Multimodal Lakehouse for AI-Native Experiences

Multimodal Data

AI Infrastructure

Search & Retrieval

Adobe evolved its enterprise lakehouse for multimodal and agentic AI, and Jiayao Hu and Miao Wang share how the resulting vector-native platform preserves the governance enterprises depend on.

Thank you! Your recording is ready.

Your submission was successful. The video will appear in a moment.

Miao Wang
Director of Engineering, Head of AEP Lakehouse
Adobe

About This Talk

Enterprise data platforms were built for structured analytics, dashboards, and reporting. Conversational, multimodal, and agentic AI introduce different requirements: embedding backfills, semantic retrieval, free-form text search, multimodal assets, and interactive latency.

At Adobe, building Customer Journey Analytics and Brand Concierge required rethinking how the data platform supported these workloads. The existing lakehouse provided strong governance, change data capture, and mature data operations, but was not designed for vector-native storage or semantic interaction across large volumes of unstructured data.

Rather than add more disconnected services, Adobe evolved the architecture into a unified multimodal platform powered by LanceDB. Structured data, conversations, embeddings, and multimodal assets can coexist within a shared data model while preserving existing governance and operational foundations.

This talk covers how Adobe:
- Identified where traditional lakehouse architectures break down for AI
- Unified analytics, semantic retrieval, and multimodal data
- Evaluated latency, cost, governance, and data residency tradeoffs
- Integrated vector-native infrastructure into an enterprise platform
- Contributed production-driven improvements upstream to lance-core and lance-spark

It is the story of how Adobe is helping make AI infrastructure ready for enterprise-scale AI.

About The Speaker

Miao Wang

Miao Wang is a Director of Engineering at Adobe, where he leads teams building large-scale lakehouse and data infrastructure for Adobe Experience Platform. He has spent more than a decade designing and building distributed data systems, with deep expertise in Apache Spark, large-scale data processing, and modern data platform architecture.

His recent work focuses on evolving data platforms to make multimodal data readily accessible and enable applications to work across data systems with minimal data movement. Miao has led major data platform transformations at Adobe and contributes to open-source projects. He holds a Ph.D. in Computer Science and an MBA from Wharton.

Jiayao Hu

JiaYao Hu is a Lead Architect at Adobe, leading Adobe Experience Platform’s Lakehouse. JiaYao specializes in Lakehouse architectures and distributed systems, with extensive experience building and operating data infrastructure at massive scale using Spark, Databricks, Ray, Kubernetes, and cloud technologies across Azure and AWS. JiaYao leads Adobe’s multimodal Lakehouse on Ray and LanceDB, building scalable data foundations for analytics, AI agents, and personalization.

Transcript

Show transcriptHide transcript

00:01

Thanks everyone for joining us. We're talking about Adobe multi-modal lakehouse. Let me introduce myself briefly. I'm Miao. I joined Adobe in 2019, leading the data engineering and data platform for Adobe Experience Platform. Over the years, we evolved Adobe platform from a data dump to a lakehouse and a multi-modal lakehouse. So in this talk, this talk will be divided in two parts. I will cover mainly three questions, who we are, what we do, and why we need to evolve to a multi-modal lakehouse, and why we need less. So, then the second part of my colleague, Jiayao, will cover how we build, how we evolve to multi-modal lakehouse from traditional lakehouse. So first of all, everyone knows Adobe has Photoshop, has PDF. Actually, we have another cloud called Experience Cloud. That is for advertisement or marketing technology. So mission of our organization in Vala is providing personalization at scale. So to achieve that mission, we have quite a few applications. So I'll just name three flagship applications. One called Customer Journey Analytics. That we're generating the analytics and the insights in real time to helping the marketers or CMOs make a decision from days to some hours. Second one is the Adobe Journey Optimizer, or AJO for short. That help our customer to tracking the customer journey in their ecosystem and how to optimize their journey to achieve their business goals, for example, maximize revenue. The third application is our flagship real-time CDP, customer data platform. So it is providing the real-time activation and the personalization. So to support these applications, we need a strong, resilient, scalable data foundation.

02:15

That is what my team is doing. So in one statement, our team's goal is to making customer data actionable for the personalization task. So over the years, we do the data ingestion, data transformation, and make it actionable for the downstream. Recently, we added a data sharing functionality, so we don't have to move data and can directly activate the journey using your own data on your own lakehouse. So that is by mid-2025. The whole business is still on the behavior log clickstreams. But since ChatGPT and all bots come into the market, more and more things are changing from clickstreams into the chat streams or conversation streams. So then from that time on, we are still thinking how to extract a signal from the conversations to enrich what we have already offered to provide a better personalization. That's when we start a journey using a new application called Brand Concierge. That help people to enrich their brand using the chatbots on their own website. So from that time on, beside conversation, we believe conversation also have multimedia data like image, video, audio, and then we want to see how we can manage and extract signal from this one at scale. So that is why we want to evolve to a multi-modal lakehouse. Then we evaluate quite a few technology, find out Lance is the one providing us the separation between compute and the storage, so we can achieve the scalability at some efficiency and affordable cost, and we can scale up even further to support hundred millions of users. Just give you some sense. In our data lakehouse, we have about 40, 50 petabytes of data under management, and we have more than 220 billion events collected every day.

04:30

So at that scale, so how to providing a scalable solution is a challenge. Now I will introduce my colleague, Jiayao, who joined with me the whole journey, is the most senior IC in our team, and do the implement of this lakehouse, to multi-lakehouse, in all details. Welcome, Jiayao. Thank you, Miao. Hi, everyone. I'm Jiayao. I'm the lead architect in Adobe's Lakehouse team. Miao just shared you some new AI experiences in Adobe. All of those new AI experiences, they create a lot of new datas like prompts, text, images, and videos, and embeddings. So we had a very basic and simple question. Where should this new AI data live? Let me start with what we already have. Today, in Adobe's Exchange Platform, we run one data lakehouse. In this lakehouse, we are managing more than 800,000 data tables. We store more than 40 petabytes of active data in the lake, and every day, more than 800 terabytes of data are ingested into the lake. The scale is significant, but these six current here I think are more important. Every table in our lake gets these six guarantees. It doesn't matter which product team owns them. First one is data with contents through managed data pipeline, and they will be either they come from the batch ingestion or stream ingestion, and they will land into governed data set. And every insert, update, and delete, we send the notifications to their downstream consumers, so they will know what happened to this table. And also, the platform will do their table maintenance such as optimization and vacuum, so product team don't need to worry about them. The platform will also do their guarantee the GDPR and other compliance.

06:42

For example, for this GDPR delete, we will make sure that the delete request will propagate to every single copy in the table, and we will remove the data from the disk, physically in time. And also, on every read pass, we need to apply the access control. This no matter it's a field-level access control or object-level access control. And we don't want to be locked into one single vendor. So we have the open multi-engine mode, which means that currently, both open source Spark and Databricks runtime can do the read and write in our lakehouse. Together, these six guarantees are a contract. This contract is what makes our lakehouse valuable. And any new data, I think including the AI-native data, ideally, they also need to be under the same contract. Let's keep in mind these six card. I will come back to them as end of this talk. So architect served us very well for traditional tabular structured data. However, those AI products arrived. They brought data that our tabular tables are never built for. I will show you how these three AI product that forced us to expand our lakehouse layout. The first one is the CX Enterprise Co-worker. This is currently the agentic level in our Adobe Exchange platform. Its agentic skills need very fast and governed access to structured data and embeddings at the same place. The second one is Customer Journey Analytics. CJ team, they analyze their conversation at scale, not only by keyword, but also by meaning, such as intent and sentiment. And the data are not fixed. So where a new model is developed, they need to add their output as a column, or they need to recompute their existing columns.

08:52

The third one is the Brand Concierge. This application need to replay a conversation, search conversation, and do semantic search on the conversation also. And those conversations can include much more data, such as images and videos. Although those three products look different, their underlying storage requirement is very similar. They all need a fast point access to a specific record. They need a semantic and a full-text search. They frequently add new columns for AI-derived data. And also they have the requirement to store long text and media. And also, they want all the data still within our lakehouse and governed by our system. The next question is whether our existing Delta lake can meet this requirement. When we evaluated if our existing Delta and Parquet store stack could support this workload, we found a gap. Particularly, this is not a criticism of Delta. Delta is still the default format of our lakehouse. The workload requirement is just different. The first gap is fast point lookup. So Parquet and Delta, they are optimized for large retrieval scans. So retrieval on a single record in Delta means that we need to decode and read in their entire data pages. The second is the search. Parquet doesn't have a native vector or indexes, so semantic search requires us to copy the data into an external table for the lookup. The third gap, which we also found very important to our setup, adding a column in Delta is a metadata operation, but backfill the data into the table in that column means that we need to rewrite the whole table. Some of our tables are extremely large, and this can be very expensive for us. The fourth one is media. Image and video long text can fit into the Delta columns as binaries, but retrieving those individual objects is very slow.

11:08

The fifth is the latency. AI application and AI agents need low latency retrievals. While Delta and Parquet were designed for batch or micro-batch analytical throughput. We realized that we need a lakehouse format that speaks to AI, but not another new database. So when we chose Lakehouse, we compared three options. The first one is to stay with Delta. Delta is a default lakehouse format. It works well with our open-source Spark and Databricks Runtime. But it's not built for AI. There is no native indexes, and then adding a new AI column, as I said, is so expensive. The third option we looked at is the standalone vector database. It is good at vector search, and it runs very fast, but it has its own storage and compute engines. So we must copy our data into the vector database and keep it in sync with the lake. This becomes an isolated island. It's very difficult for us to guarantee the GDPR and do the access control there. Last is the third option we evaluated. Last is, I think it's the only option that gives us the strengths of both. It stores table, media, and vectors together in the same table. It keeps using our object storage store. It supports fast vector full text search and hybrid search. When adding a column, we don't need to rewrite the whole table. It's a very cheap operation, and it has low latency. And because we don't need to copy the data out for the lookup, everything still stays in our AP lakehouse, the ecosystem. And also, very good thing that Lakehouse support is their open engine support. We can use the Databricks Runtime, open-source Spark, and Ray to do the read and write. This is why Lakehouse won.

13:15

So it brings vector search into our lakehouse, and the data still stays within our lakehouse. I think Lakehouse is a real lakehouse format for AI. Like Delta, it has ACID transactions and versioning, it has time travel, it has schema revolution. It supports table maintenance job like compaction and version cleanup, just like Delta vector and optimization. It also adds what the IT needs, such as fast random low access, efficient encoding for the media, and it has the native Python and Ray APIs. Now let me show you what we built. In short, it's one governed lakehouse, Delta for tables and Lakehouse for AI-native data. This diagram shows the end-to-end flow. Data can come from many sources. It can be a Delta table, it can be different Adobe products, external storage connectors, and other external data feeds. And the data format can be in text, conversations, image, and videos. We use Spark and Ray to do the data ingestion. Depends on the source and the type of the data. And all the data lands in the last table on our objective storage store. Structure, data field, vectors, and media, they stay together in the same table. Then our distributed Ray jobs will run model inference. They write their embeddings and other model output as new columns. At the same time, indexes were being built asynchronously, and also other table maintenance job will also run automatically. Finally, applications and AI agent can query the table through LastDB. They can either do this from open source LastDB or LastDB Enterprise. I will explain that choice later. As we see, the data always stays in with our lakehouse.

15:28

Let me zoom into one table. The dark columns belongs to the data engineering. They use open-source Spark, Databricks Runtime, and Ray. They handle the data ingestion, CDC merges, GDPR delete, and table maintenance based on the base columns of the table. The purple columns belongs to the feature engineering team. They use Ray and Python. They do model inference, add the new model outcomes as a new column, such as a new embedding version or a new internal text. So one table, two teams, and different engines. They work together in the same last table. No one blocks each other. When new columns come, we don't rebuild the table. We just add a new column. The existing columns and the file were never touched. So we can keep different versions of embedding side by side. And at the same time, every different engine, like Spark, Databricks, Ray, Python, they can still read the table. Beyond conversations, the same approach also works for multi-modal data like images, videos. We build a declarative data processing pipeline based on Ray and LastDB. So here, users can specify what is our data source, and they can choose their models they want to use from our model registry and provide their customized UDF. And they also need to specify what is our output columns. And then our data pipeline will run the models automatically for the user. So the product team don't need to worry when, where, and how they run the data pipeline. They only need to describe what they need. Our pipeline is also tuned for CPUs and GPUs. Our CPU stages handle the decoding, parsing, and the data preprocessing. Our GPU stages mainly do the model inferences.

17:40

We split this because GPU is the most expensive part. While CPU is doing the data processing jobs, the GPU don't need to be idle and wait for the output. All the output lands into the last tables. From there, different team can do the search, retrieval, and also training and the analytics using the same last table. So this pipeline here is not a separate AI platform. It's just one more new writer in our lakehouse. How we serve LakeDB? We use both open source Last and Enterprise Last. For our pipeline jobs, we use the open source Last. We use Spark, Databricks, and Ray for ingestion and most of their feature engineering jobs. Because we use Last as a library, so we don't need to maintain separate new services. For critical customer-facing jobs, we use Last Enterprise. We get three things from this. First thing is the speed. Last has a very good NVMe cache. It keeps the whole data close to the compute, and also it can provide us very good distributed query engines. Together, they can provide us a very low-latency query experience. The second thing is the reliability. Last has their high availability support. It can also do the background indexing and compaction automatically, without downtime. Also, the third thing is their control. We want to control the compute and storage we own. So we deploy Last Enterprise in our own Kubernetes clusters with bring your own cloud model. So we can also support RBAC and audit logs. And the most important part is that both sides here, they use the same Last format. So converting one workflow to another side is just as simple as a configuration change.

19:49

We don't need data migration. Remember the six cards from the beginning. Here they are again, this time they are for Last. For their ingestion pipelines, we use the same data pipelines that ingest data into Last. And Last change data feed tracks every insert, update, and delete. They change propagate to downstream consumers. And we also do the table maintenance job, like compaction, version cleanup, and asynchronous index build for our users. With the platform, we also guarantee the GDPR and compliance, and we do the same access control as Delta for less. And teams can keep using their existing engines, like open source Spark, Databricks Runtime, and they can use our new engine Ray also. All these six cards still hold, so all the AI data gets the same contract as every other data set in our lakehouse. Of course, running Last at our scale is not always easy. We found real gaps, and we choose to fix them upstream. Let me give one example, the stable row ID. If a row ID changes after a compaction or update operation, the CDC won't know which row really changed. We found this issue in the Last Spark connector, and we fixed this issue upstream by preserving the row ID during a Spark write and update pass. We did the same for the change data feed. And for native update pass, we also fixed issues, the conflict issues when two writers write to the same column at the same time. And also we add some Azure storage support and tune-ins for Adobe's use cases. In addition, we also contribute to the Apache Spark. For example, the nearest by join PR brings a native top K vector search in Spark SQL.

21:59

So data engineers can do vector search as a simple join operation. In total, since this February, we contributed more than 20 PRs to the community, and every one of them comes from a real production need. To wrap up, let me go back to the question at the beginning: Where should the AI native data live? Our answer is in the same lakehouse, under the same contract as everything else. So we not only add a vector search to our lakehouse platform, we also make the AI data a first-class citizen in our lakehouse. Miao and I will be around after the session. We are happy to share. Thank you.

Related recordings

Welcome: What Will AI Dream Next?

Chang She
•
Keynote
opening-keynote-what-will-ai-dream-next

AI Research

Multimodal Data

AI Infrastructure

Cosmos: Omnimodal World Models for Physical AI

Max Li
•
Session
cosmos-omnimodal-world-models-for-physical-ai

Physical AI

Multimodal Data

Model Training

Scalable, Multimodal Data Curation

Ethan Rosenthal
•
Session
scalable-multimodal-data-curation

Data Curation

Multimodal Data

Search & Retrieval

Building the Data Flywheel for Frontier AI Model

Suqiang (Jack) Song
•
Session
building-the-data-flywheel-for-frontier-ai-model

Data Curation

Model Training

Multimodal Data

From Raw Delivery to the Training Loop: Luma's Data Platform on Lance

Haoran Cai
•
Session
from-raw-delivery-to-the-training-loop-luma-s-data-platform-on-lance

Multimodal Data

Data Curation

Model Training

How Adobe Built a Multimodal Lakehouse for AI-Native Experiences

Miao Wang
Jiayao Hu
•
Session
how-adobe-built-a-multimodal-lakehouse-for-ai-native-experiences

Multimodal Data

AI Infrastructure

Search & Retrieval

Data Infrastructure for Robotics Foundation Models

Zhipeng Yan
•
Session
data-infrastructure-for-robotics-foundation-models

Physical AI

AI Infrastructure

Model Training

What Does "Now" Mean to Your Model? Time Alignment for Multimodal AI Data

Vasundra Srinivasan
•
Session
what-does-now-mean-to-your-model

Multimodal Data

Physical AI

Model Training

Managing an Index of 100s of Billions of Pages With Exa Skeleton

Jan van der Vegt
•
Session
managing-an-index-of-100s-of-billions-of-pages-with-exa-skeleton

Search & Retrieval

AI Infrastructure

Keynote | Rethinking AI Infrastructure

Gerred Dillon
•
Keynote
rethinking-ai-infrastructure

AI Infrastructure

Model Training