Today, we're introducing the LanceDB Realtime Engine, which brings streaming writes and live scalar, full-text and vector indexes to LanceDB Enterprise. New rows can be searched within milliseconds, and every acknowledged write is already durable in object storage.
For AI workloads, getting a row into storage is only part of the problem. Applications need to look it up, filter it and search its text and embeddings while more data keeps arriving. Take a robotics engineer replaying a grasp that just failed, or an RL researcher pulling up a rollout that finished a second ago:
- Robotics and physical AI training data. We are working with a growing number of robotics and physical AI labs. Their fleets continuously produce images and sensor readings, while annotation and feature pipelines add labels and embeddings. These labs need to find new frames, inspect annotations and assemble training datasets as the data arrives. A corrected label should affect the next filter, and a recomputed embedding should show up in the next vector search, without waiting for the next batch commit or index refresh.
- Reinforcement learning traces and rollouts. Across the AI labs we work with, reinforcement learning workloads produce a continuous stream of model responses, tool calls and artifacts. Generation workers write trajectories, graders add rewards, and learners consume the resulting training examples. These pipelines need many concurrent writers, fast reads of token and training-signal columns, and a reproducible record of which rollouts trained each checkpoint, all while new rollouts keep arriving.
In both cases, reads have to keep up with writes. Lakehouses, however, were built around large batch commits that collect data, write efficient columnar files to object storage, then publish a new table version. That design makes large datasets economical to store and scan, but it creates a tradeoff for streaming workloads. Commit more often, and small files and metadata updates pile up. Wait for a larger batch, and readers wait too.
The LanceDB Realtime Engine removes this tradeoff. Writes don't wait for a table commit, and readers don't wait for a batch. Full-text and vector indexes stay live as rows arrive, so applications can read and search new data without waiting for a separate refresh job or a merge into the base table.
To make this possible, we developed the MemWAL format as part of the Lance table format. It gives Lance a log-structured merge tree (LSM tree) on object storage, with a write-ahead log, in-memory indexed buffers, and indexed data files. LanceDB Enterprise is the best engine for this new capability; it executes reads and writes across the entire LSM tree and handles data compaction and index maintenance in the background.
Comparing lakehouse ingestion solutions
Over the past several years, the industry has worked on this problem at two layers. At the storage layer, Iceberg and Delta Lake continue to improve merge efficiency with deletion vectors, while Apache Paimon builds an LSM tree into its table format to support streaming updates.
At the ingestion layer, cloud platforms and compute vendors typically add a write-buffering layer on top of a lakehouse format. New data can be acknowledged or served from that layer well before it becomes available in the underlying table. These paths also bring constraints such as append-only ingestion and limited native vector and full-text index support, often requiring another synchronization into a search service.
Access to proprietary write buffers adds another layer of vendor lock-in until the data is flushed into underlying lakehouse tables. During this buffering window, other lakehouse engines cannot read the buffered data through the open table format, preventing cross-engine real-time reads. Durability also often depends on vendor-managed buffers and replication rather than the backing object store, making data loss prevention and recovery more complicated across the system.
The figure below follows one write through each approach.
The LanceDB Realtime Engine removes these tradeoffs by combining streaming writes, fully indexed live search and object-storage durability in one open Lance table. With no separate buffer or search-sync layer outside the table, it is the most native ingestion solution for the lakehouse.
Horizontally scalable and lightning fast
In our Realtime Engine benchmark on Amazon EKS, the median time from a durable write in S3 to a readable row was 52 ms. We ran the tests with three public datasets spanning web text, search passages, and image-text pairs: FineWeb-Edu, MS MARCO, and LAION.
Each single-node run used one m8gd.8xlarge node (30 vCPU, 112 GiB). A write counted only after it was durable in S3, and every run checked that the table ended up with exactly the acknowledged rows.
With live primary-key indexing, durable write throughput grew from 79-83 MB/s on one shard to 312-334 MB/s on four shards, all on a single node. Across four nodes, each serving its own four-shard table, combined throughput reached 946 MB/s.
A newly acknowledged row became visible to another client after 52 ms at the median. During live ingestion, median latency for queries on fresh rows was 12-14 ms for key lookups, 15-28 ms for full-text search, and 26-30 ms for vector search.
Writing into a base table that already held the entire FineWeb-Edu or MS MARCO corpus was at least as fast as writing into a small one. Existing data stays searchable while new rows arrive.
The engine can write each incoming image straight to object storage and keep only a small reference in memory. On LAION with only the primary-key index live, this cut peak node memory from 74 GiB to 18 GiB, while throughput stayed within 3%.
Keeping an index live costs CPU on the node, and the cost depends on the data. On LAION, where images make up most of each row, keeping every index live costs little (289 MB/s with every index live, vs. 314 MB/s with only the primary-key index). Long text documents need more full-text indexing work for each byte written. Each table may optionally choose which of its indexes the engine keeps live, balancing write throughput against query speed on the newest rows.
The throughput numbers above are the rates the nodes accepted. Holding a high write rate for hours also needs maintenance to merge and index data as fast as it arrives. Maintenance runs on its own workers and scales separately from the nodes, so a table with heavy sustained writes gets more workers.
How the Lance MemWAL format works
These results rest on two layers: the open Lance MemWAL format and the engine that runs it. The MemWAL format gives Lance an LSM-tree structure for streaming data. New rows are logged in object storage and indexed in an in-memory MemTable. Full MemTables are then written out as indexed Lance files called SSTables, which are merged into the base table in the background. The log, metadata, files and indexes are all part of the open Lance table format.
Each shard writes independently, so ingestion can scale across nodes without a table-wide commit for every write. The table below compares what each format provides on the streaming write path:
More shards and nodes let Lance scale toward object-storage write throughput. The Lance format specification describes the storage layout for compatible readers and writers.
How the LanceDB Realtime Engine works
The Realtime Engine turns the open Lance format into a managed service for streaming writes and live indexed reads. Each write goes to a Realtime Engine node, which persists it in the table's write-ahead log in object storage and adds it to an indexed MemTable. The write returns once the data is durable, so if a node goes away, nothing it acknowledged is lost. Queries search the MemTables alongside the SSTables and the base table.
The engine serves vector search, full-text search, point lookups and scalar filters on MemTables with low latency. It keeps enabled indexes current as new rows arrive, so applications can search recent and historical data through the same table. LanceDB Enterprise manages shard placement, writer recovery, memory limits and backpressure as the workload grows.
LanceDB Enterprise also runs distributed maintenance on these tables, including data compaction, index updates and garbage collection. Dedicated workers merge SSTables into the base table, update its indexes and reclaim obsolete files. Maintenance scales independently of ingestion and query serving.
While this work runs, queries keep using the SSTable indexes that cover each row until the base table's indexes catch up, so new rows never drop out of search:
Query the data you just wrote
With the Realtime Engine, LanceDB Enterprise brings streaming writes, live search and automated maintenance together on open Lance tables. Robotics teams can find new frames and corrected labels as soon as they land, and RL pipelines can train on fresh rollouts while new ones keep arriving. Every acknowledged write is already durable in object storage, and the data stays in the open Lance format from the first write to long-term storage.
Talk to our team about running the Realtime Engine on your robotics streams, RL rollouts or other live AI data, at the ingestion rates you need.

.avif)





