Managing an Index of 100s of Billions of Pages With Exa Skeleton

Search & Retrieval

AI Infrastructure

Behind Exa’s index of hundreds of billions of pages is Skeleton, the distributed streaming engine Jan van der Vegt traces from pipeline definition through massive production backfills.

Managing an Index of 100s of Billions of Pages With Exa Skeleton

Managing an Index of 100s of Billions of Pages With Exa Skeleton

Search & Retrieval

AI Infrastructure

Behind Exa’s index of hundreds of billions of pages is Skeleton, the distributed streaming engine Jan van der Vegt traces from pipeline definition through massive production backfills.

Thank you! Your recording is ready.

Your submission was successful. The video will appear in a moment.

Jan van der Vegt
Machine Learning Engineer
Exa

About This Talk

Indexing billions of crawls a day across an index spanning hundreds of billions of pages creates a unique set of systems challenges, including maintaining global state, coordinating heterogeneous compute, moving data efficiently between distributed workers, and writing continuously into S3-backed tables without sacrificing freshness or reliability.

To solve this, Exa built Exa Skeleton, a highly distributed streaming engine built on Lance, queues, S3, ScyllaDB, and Kubernetes. Exa Skeleton powers the ingestion and transformation pipelines behind Exa’s search indices, supporting low-latency, high-throughput writes while giving the team a framework for evolving production data at massive scale.

This talk explains how Exa Skeleton works from the interface down to the infrastructure it generates. It covers how pipelines compile into Kubernetes resources, how data flows through the system, how Exa Skeleton tracks where data and work live across a distributed environment, and how Exa handles massive column backfills to productionize new models and ranking innovations without rebuilding entire datasets. It also shares the architectural tradeoffs involved in operating this system at the scale of hundreds of billions of pages.

About The Speaker

Jan van der Vegt

Jan is the tech lead for two of the index teams at Exa, building out the data infrastructure to manage web scale data as well as building intelligence into the index with modern algorithms. Before Exa Jan worked at Databricks on coding agents, had an MLOps startup and worked on large data systems in the aviation industry. In a previous career Jan was a Magic: the Gathering streamer.

Transcript

Show transcriptHide transcript

00:00

Hi, everyone. My name is Jan van der Vegt. I lead some of the index teams at Exa, and we heavily leverage Lance to get everything out of the data that we need. So let me tell you more today about Exa Skeleton, which is our data processing framework that we built, which allows us to reach web scale. Before Exa, initially, I started in the aviation industry working on big data and machine learning systems. I ended up founding an MLOps company which got acquired by Databricks. And at Databricks, I was one of the tech leads on Genie Codes, where I worked on retrieval for tables, notebooks, code snippets, et cetera, to help these coding agents get the right context. And at Exa, I manage the team that does our index management as well as the content intelligence teams, where we have a lot of models and algorithms that we use to better parse, better decision-making on what makes it into the index. But yeah, today I'm going to talk about Exa Skeleton, which is our system that constantly feeds our index and allows us to constantly make changes. So just to give you some numbers. So Exa is a web search engine. It's a very generic web search engine. So think of it like Google for agents. There's a lot of different types of agents with very different use cases. Think of legal, coding agents. Personal agents are hot and happening right now. Yeah, so all of these agents need very different things from the web, and for that, we need a very wide coverage, but we also need the index to be very up to date so that things like the latest security issue with CVEs, that those are available to coding agents, the latest regulation is available for these legal companies. So, to do that, we built Exa Skeleton, and right now we have around 100 billion pages in our index.

02:07

This is a semantic index, and we have around 700 billion crawls historically that we've been processing. We've managed to get our latency down from... it's actually less than five minutes now. From the moment it's crawled to it being retrievable to our customers is around three minutes, so the P50. And just to give you a sense of the scale, we have around 35 petabytes of data in Lance tables. So yeah, the main goal is, as the data comes into our systems, we need to make decisions around that. We need to feed it through GPUs for embedding models, for other types of extraction, and we need to make sure that this happens very quickly. But at the same time, we have these research teams that constantly build the next generation of embedding models. So whenever they have a new document embedding model, we need to be able to re-embed the whole index. And similarly, we have all these custom parsers that extract things like publish dates, that extract the main content that's relevant for agents while getting rid of all the noise. So all the teams at Exa are constantly improving the way that we go from this really messy web data to a really clean index that best serves our customers. So we also need to be able to reapply all these improvements to this massive amount of data and not just change the ingestion pipeline. And to achieve that, we ended up iterating on a few mechanisms. So when I joined Exa, which was 11 months ago now, there was a platform called ExaD, which was heavily tailored around Ray and Lance. And Lance was really used as a data protocol both for data that was in transit as well as the data that would be served to our customers as well as stored for future use. But it had a few flaws. It wasn't able to scale to the size we needed, and it wasn't reliable enough.

04:11

So I joined, I didn't really understand what we were doing with ExaD, and it was in hindsight more of an issue on me than on the framework. But we ended up moving to using Spark with structured streaming, which allowed us to get to the next scale that we needed to hit. And at the time, we were crawling around a billion pages a day. And, yeah, so we were using Spark with Lance, and we did all kinds of joins to get state about previous crawls, et cetera. But in the end, this didn't get us to the latency that we needed, and it didn't scale linearly with the amount of data we were processing. It scaled worse, and it was clear that we needed to go up another level to be able to process 10 billion pages a day. And second to that, Spark is notoriously rough when you're dealing with GPUs because Spark is very batchy, even with micro batches and GPUs require you to constantly keep them filled so that you use these expensive assets in a very useful way. And then lastly, the Lance support was kind of weak, and that was partially an issue with Databricks that we were using at the time. So all the things that we learned, both what we needed out of the system and the flaws of previous systems, we ended up designing Exa Skeleton, which is a little bit more like ExaD, but with a few key differences. And today, I'm going to talk about Exa Skeleton, how that allows us to do these massive operations using these low-level primitives that we built on top of Lance and Kubernetes and a bunch of other things. So One of the things that's made Spark less convenient is the fact that we have these very different types of compute needs. So you have, in some cases, you need large memory machines to build out these big fragments. In some cases, you need very CPU-heavy workloads. And then in other cases, you need GPUs.

06:16

So we ended up building this very heterogeneous compute framework, and we model it as a computational graph. So the way that this graph works is you can define different types of transformations and filters. You can specify sources and sinks, and then data kind of streams through these nodes. And the nodes themselves are Kubernetes deployments that are highly scalable. And then the communication between these nodes is using Lance files, so not the tables, together with SQS to kind of communicate between the nodes. So yeah, there's sources, processors, and sinks. There's these edges, and because Lance files are columnar files, it allows you to specify only the columns that you need to minimize the IO needed to get the data back from S3. And because instead of doing these joins to get this state from previous crawls or from domain information, instead we replace that with a key value store that allows us to have very high throughput and low latency. It allows us to eliminate these joins, which means that all the data can kind of stream independently through the system. So even if there's some really poorly behaved page that is blocking something, it doesn't block the rest of the system, and things can just flow. So how did we achieve this? First of all, we've defined this kind of Python API that allows you to model these nodes as a Python stream. So you can see here there's a graph, which is kind of the job. You can see we have a source, so in this case, there's a static Lance table. You can define transformations, and you can be like, "Hey, I need to run this on a specific GPU." You can specify what type, and then you can sync it. So in this case, this would be a simple batch job that goes from one table into another, and it adds some GPU transformations and other things to the data set.

08:23

But under the hood, even though this is Python, under the hood, what happens when you run this, it compiles into a number of different types of assets. So we spin up these Kubernetes deployments. We spin up these SQS queues. We have these S3 buffers. And so we go from this Python graph to an artifact, which is a kind of a compiled specification of the job. There's a Kubernetes controller that picks that up and kind of reconciles all the things that are needed to execute this job successfully. And then eventually, there's these Kubernetes deployments that can scale to pretty absurd sizes. So we had a big backfill job recently. We had 35,000 Kubernetes pods running just for that single job. So yeah, these workloads are Kubernetes deployments. They can be on CPU or GPU. Because it's queue-based, we have a very clean auto-scaling signal, so we use Kata to scale based on how much pressure there is on the input side. And then we have a dead letter queue system that allows us to capture kind of non-retriable errors in a way that we can inspect them, we can redrive them. Maybe we need to change the code or the configuration, and that way we can deal with the stragglers, because if you're dealing with hundreds of billions of pages, there's a lot of really weird s**t that you don't expect. And yeah. So this allows us to keep things running, and we get Slack notifications when this happens. Agents look at it, tell you what to do. So it's really nicely integrated. So this is a very dense slide, but basically what happens under the hood is that we want to operate on batches of data. So a batch is typically between 1,000 and 5,000 rows, and the reason for that is that you want to balance how much operations are you doing on SQS and S3 versus how much data are you processing in one pass, because that adds memory pressure to your GPUs. So in this example, we actually go from a Lance table into a different Lance table.

10:31

And what we start with are these fragments. So these Lance fragments are big files that represent hundreds of thousands of rows sometimes, but that's too big of a unit to work on for a lot of these smaller pods. So the first step is to kind of batch these into what are called atto-batches. Atto is 10 to the power of -18, which is excess 10 to the power of 18. I thought it was really funny, but nobody agreed with me. So we first started with cutting them up into smaller pieces, and then these get written to S3 somewhere, and there's an SQS message that goes down to the next node, which when it gets picked up, it has a pointer to, "Oh, here's the data. You can find these columns in this file and this column in this file." And then it flows through these different nodes. And yeah, depending on how heavy it is, we have different amount of pods to process these. And then at the end, if we want to write into a new Lance table, we need to accumulate them back into big fragments because if you deal with hundreds of billions of pages, too many fragments makes it impossible to deal with these tables. So these accumulators basically build up all these batches of data into big files, pushes them back into S3, and then sends them to a committer. There's a shared committer service so that even multiple jobs can commit into the same table. And this committer will constantly push new commits with new data back into the table. So let's talk a little bit about the sources. So, it's both kind of a batch framework, which can both be used for just normal batch jobs that produce a new table, or they can be used for backfilling Lance tables, which is a very nice feature about Lance that we heavily leverage. But I'll talk about that in a bit. As well as, I guess, a constantly streaming engine where you can point to, for example, S3 location where files keep getting dropped by our crawling system, and then we will convert these into these Etl batches while they stream into the sink-- or, sorry, into the graph.

12:46

And then we'll have different sinks so we can write into multiple tables for future use, or these tables are also being used by our database to then serve them to our customers. So a lot of the magic happens in between the sources and the sinks. So an obvious thing are the transformations. So you can think of it like, oh, we want to add these additional columns that are derived from the inputs, as well as maybe from some global state that is, at the moment, DynamoDB based, but it could be other external systems. We have a feature store now that has all kinds of scores about domains and URLs. And yeah, you can define these transformations in either Python or in Rust, and we use actually Polars as a way to define these transformations, so we're always vectorizing things in Rust. And examples are okay, we need to pre-process the data before it goes into the GPUs. We can extract all kinds of signals, classifiers for moderation scores, et cetera. So yeah, you define these, and then in this graph definition, you can say, "Hey, here's a CPU-based node that does these transformations, or a GPU-based node." And then second, we also need to do a lot of filtering. Well, there's a lot of nasty things on the web, but also maybe there's crawling failures. So we have models that can detect like, hey, we got bot detected or there's, I don't know, it's a 404, maybe it's a soft 404. So there's all these different reasons why we don't actually want to process it, or we just don't think it's a valuable page for our index. There's a lot of pages on the web. We only want to serve maybe like 100 billion, but we're crawling way more. So we want to have some kind of gates to make sure that we're maintaining a high-quality index. To support that in the framework, we actually have these batches, and every time a batch makes it further into the system, we don't rewrite the file, but we just add new files that only encapsulate the new columns. And then to support these filters in SQS, we actually specify which rows are still part of the batch after that stage.

14:58

So it's a little similar to how deletion vectors work in Lance, in the sense that we just null out the rows that are no longer part of it. And then when a reader comes in, we only give the rows that are still part of the batch, so it gets sparser and sparser as it makes it through the nodes. Yeah, and then eventually we need to persist this data somewhere. So all these intermediate Lance files are just a buffer as the data makes it through the process. And we actually have a garbage collection process that knows when, oh, this batch is no longer needed by any of the sinks, so we can clear it from S3. But yeah, we have different types of sinks. The main ones are Lance because we heavily rely on Lance, even for the interface between the index system and our databases. But we have very low latency ingestion that is more API based, and we have all these other kinds of sinks. Yeah, so-- Actually, sorry. Thought I had one more slide. Yeah. So, as I mentioned, these accumulators, they basically are responsible for turning them into big batches, and this allows us to actually write at 20 to 50 million rows a second into these Lance tables. It's just purely horizontally scalable. And, yeah. So relying on SQS for the communication between nodes and S3 for the actual physical data is a very nice property because sometimes we're handling hundreds of gigabytes a second, and pushing that over the wire as opposed to over S3 is extremely expensive. So, at the cost of maybe 20 seconds of latency as you go through the whole graph by relying on S3, it makes it a lot more affordable to handle this kind of scale. So yeah, if you think about what are the types of data we handle. So S3 is both for the in-flight data as well as the persistent data in the Lance tables. S3 is the main target. We also use S3 Express for very low latency use cases.

17:05

SQS is really the metadata that flows through the system that explains where is the data now, what is still part of the data. And other things, we use Dynamo both as an index on top of our Lance tables, as well as all kinds of metadata that we use to make decisions. So things like Simhashes from a previous crawl. So let's say we crawl the page, we embedded it, we keep a Simhash around, which is kind of a binary hash that allows you to approximate how much it has changed based on the number of bits that have flipped since the previous one. So that allows us to decide whether we want to re-embed this page or whether we can just skip it or use the old embedding. And then Volky is kind of like Redis that we use to coordinate in-flight data. So an example is tracking if a batch has been processed by every sink so that we can garbage collect it and other kinds of caching mechanisms. And the whole system is set up that we have at least once delivery, and we can use Volky and other mechanisms to do item potency checks. So we're- We're close to 100% exactly once, and once in very specific scenarios, we end up duplicating the data, but there's guarantees that we can never lose data. That was not true the first few weeks when we launched this system, but since July, we haven't lost any data. Yeah, so we actually managed a number of indices. So Cosmos is our crawl history. Right now it's almost 700 billion rows. So every page that gets ingested by the framework, even if we filter it out halfway, still ends up being persisted in Cosmos. So this allows us to, in the future, make different choices, or it allows us to go back to a previous version. So we're starting to prune some of this for the really low-value things, but this is basically our entire crawl history since around April.

19:09

And then Orion and Andromeda are mutually exclusive indices that have a different value, and they have a different serving stack with different kind of cost and quality attributes. And then Nebula is our content cache, so this is when a user requests specific content from our cache or when our re-rankers need access to the content to decide out of the 50 candidates that made it all the way to the end, which 10 to serve to the user. All of that content that goes into that comes from our content cache, which is also a Lance table. And then we have a lot of different types of other use cases that contain all kinds of signal data that is useful for, for example, the crawling stack or for training data, things like that. So I think the biggest unlock compared to other systems or other table formats available is the notion of being able to do column backfills without having to rewrite the whole table. And yeah, so I used to work at Databricks, I used Delta a lot, but the fact that if you want to add a new embedding model, for example, you need to re-derive that from the inputs. With other, like Iceberg, Delta, it basically means you have to rewrite the whole table. And at our scale, that's a really expensive operation. But the way that Lance works is that you can actually write new files and in the manifest say, "Hey, this new file belongs to the same fragment and this column overrides the old one, and these are two new columns." And this allows you to only write what has actually changed and just have the reader combine everything. And this means that these jobs go from, oh, this would've cost like half a million dollars, maybe it's only $50,000. So yeah. Similar to how I showed how this batch job works, we basically do the same thing. So we go from a fragment, we cut it up into smaller batches, we process all these batches, but now we keep track of like, oh, these batches all belong to this original fragment. And then at the end when we know all the batches that belong to a specific fragment have been done, we end up realigning them in the original order. We write them to S3 similar to the normal accumulators, and then we tell the committer, like, "Hey, this fragment, the backfill is done." And this allows us to basically rip it even over Cosmos. So we did a massive backfill.

21:39

It took less than a week the first time we did this. I'm pretty confident we can do it in less than two days the next time we're going to do it. So yeah. And the nice thing about these backfills, appends can still keep going. They're not conflicting at all. So we can do these massive operations while we're still having this few-minute latency ingestion into all our different indices. Row addresses stay valid. We can do partial overrides to only update the things that need to change. We have a way to, if a fragment ends up failing, we can notice that and we can do a repair if that's needed. And we do all kinds of additional checks before we actually commit things to make sure we don't have any data corruption. Yeah, and I need to speed up a little bit, but we regularly have new document embedding models, so that means that we then need to dual write the embeddings for a while, as well as do this massive backfill. So we've been able to scale up to 4,000 GPUs to rip the new embedding model over our whole index while dual writing into the new column. And then once it's validated, we can switch the production models to now use the new document embedding space, and we can stop writing the old one, and then we can retire the old model. And the next month or two, I want to make this a single-button process. The last time it was a little bit more involved. Yeah, so Lance has really enabled us to do these massive backfills in a way that's easy to manage and it's economically feasible. And being able to use this storage layer for historical data as a stream of data that flows into our systems, as well as serve content directly from it with these blob columns is a really powerful thing that wouldn't have been possible without Lance. So yeah. Columnar storage allows us to only read what we need.

23:44

These column backfills allow us to keep the old data in place and only write what is new. And random access allows us to actually serve parsed content directly from S3 or S3 Express without having to duplicate the data into a different storage system. So yeah, two lessons that I've learned as I've been working on this system for the past 11 months. Getting to a great architecture that can scale to the actual Exa scale takes a while. Both learning what kind of systems are going to break at this scale, as well as just wrapping your head around what is needed, yeah, just takes some time to build up. But the current system is in a really good shape. I'm very happy with where we ended up. And Lance is a really great technology. At Exa, we've really consolidated everything related to storing data and processing data around Lance. And yeah, it has really helped us accelerate delivering all these great models and index to our customers. Thank you.

Related recordings

Welcome: What Will AI Dream Next?

Chang She
•
Keynote
opening-keynote-what-will-ai-dream-next

AI Research

Multimodal Data

AI Infrastructure

Scalable, Multimodal Data Curation

Ethan Rosenthal
•
Session
scalable-multimodal-data-curation

Data Curation

Multimodal Data

Search & Retrieval

How Adobe Built a Multimodal Lakehouse for AI-Native Experiences

Miao Wang
Jiayao Hu
•
Session
how-adobe-built-a-multimodal-lakehouse-for-ai-native-experiences

Multimodal Data

AI Infrastructure

Search & Retrieval

Data Infrastructure for Robotics Foundation Models

Zhipeng Yan
•
Session
data-infrastructure-for-robotics-foundation-models

Physical AI

AI Infrastructure

Model Training

Managing an Index of 100s of Billions of Pages With Exa Skeleton

Jan van der Vegt
•
Session
managing-an-index-of-100s-of-billions-of-pages-with-exa-skeleton

Search & Retrieval

AI Infrastructure

Keynote | Rethinking AI Infrastructure

Gerred Dillon
•
Keynote
rethinking-ai-infrastructure

AI Infrastructure

Model Training