lancedb-real-time-engine
Xuanyu Zhan
Jack Ye
Dan Rammer
lancedb-series-b-funding
Chang She
newsletter-october-2026
ChanChan Mao
table-formats-in-depth-what-is-a-table-format
Weston Pace
augmetec-finds-critical-evidence-faster
Augmetec
newsletter-september-2026
ChanChan Mao
how-jev-compares-to-other-rerankers
Taylor Smith
Ayush Chaurasia
giving-back-to-open-source
Xuanwo
Taylor Smith
10-billion-vector-search
Yang Cen
practical-llm-pretraining
Ayush Chaurasia
newsletter-august-2026
ChanChan Mao
data-mining-challenge-in-physical-ai
Lei Xu
feature-engineering-examples
Justin Miller
announcing-reverie-summit-2026
LanceDB
newsletter-july-2026
ChanChan Mao
crewai-rebuilt-agent-memory-on-lancedb
CrewAI
data-loading-guide
Weston Pace
volcano-engine-lance-agent-memory
Bytedance
make-handwritten-notes-searchable-optimizing-an-ocr-pipeline-with-lancedb
Prashanth Rao
china-merchants-lancedb-story
China Merchants Lion Rock AI Lab
rabitq-gets-faster-higher-recall-lower-latency-query-time-control
Yang Cen
newsletter-june-2026
ChanChan Mao
from-messy-pdfs-to-verifiable-answers-with-liteparse-and-lancedb
Prashanth Rao
Clelia Astra Bertelli
faster-vlm-fine-tuning-with-materialized-model-features-in-lancedb
Prashanth Rao
Ayush Chaurasia
lance-blob-v2-late-materialization-for-large-binary-data-in-spark
Drew Gallardo
semantic-memory-for-hermes-agent-with-lancedb
Prashanth Rao
a-metadata-benchmark-of-lance-delta-lake-and-iceberg-on-s3
Jack Ye
scalable-feature-engineering-on-multimodal-datasets
Prashanth Rao
stable-worldmodel-a-high-performance-platform-for-reproducible-world-model-research
Ayush Chaurasia
Quentin Lhoest
Lucas Maes
Quentin Le Lidec
reproducible-data-curation-in-the-multimodal-lakehouse
Prashanth Rao
newsletter-may-2026
ChanChan Mao
newsletter-april-2026
ChanChan Mao
how-lancedb-accelerates-vector-search-at-10-billion-scale
Yang Cen
opensearch-vs-lancedb-for-vector-search-query-cost-and-infrastructure
Justin Miller
volcano-engine-autonomous-driving-data-lake-solution
Kejian Ju
unifying-the-av-ml-stack-lancedb
Ayush Chaurasia
lance-json-support-why-you-might-not-really-need-variant
Jack Ye
building-a-storage-format-for-the-next-era-of-biology
Pavan Ramkumar
newsletter-march-2026
ChanChan Mao
smart-parsing-meets-sharp-retrieval-combining-liteparse-and-lancedb
Clelia Astra Bertelli
Prashanth Rao
lance-format-v2-2-benchmarks-half-the-storage-none-of-the-slowdown
Xuanwo
make-your-sql-workflows-multimodal-with-lancedb-x-duckdb
Prashanth Rao
agentic-coding-as-community-stewardship
Xuanwo
what-we-mean-by-multimodal
Prashanth Rao
ai-native-development-local-continue-lancedb
Ty Dunn
lance-file-format-2-2-taming-complex-data
Xuanwo
lance-blob-v2
Xuanwo
Jack Ye
openclaw-lancedb-memory-layer
Xuanwo
Prashanth Rao
openclaw-lancedb-seed2
LanceDB
openclaw-memory-from-zero-to-lancedb-pro
Prashanth Rao
upload-lance-datasets-to-hf-hub
Prashanth Rao
zero-shot-image-classification-with-vector-search
Vipul Maheshwari
werides-data-platform-transformation-how-lancedb-fuels-model-development-velocity
Qian Zhu
Fei Chen
training-a-variational-autoencoder-from-scratch-with-the-lance-file-format
LanceDB
track-ai-trends-crewai-agents-rag
LanceDB
tokens-per-second-is-not-all-you-need
Mingran Wang
Tan Li
the-future-of-open-source-table-formats-iceberg-and-lance
Jack Ye
the-case-for-random-access-i-o
LanceDB
series-a-funding
Chang She
semanticdotart
Ayush Chaurasia
second-dinners-secret-weapon-lancedb-powered-rag-for-faster-smarter-game-development
Qian Zhu
search-within-an-image-331b54e4285e
Kaushal Choudhary
scalable-computer-vision-with-lancedb-voxel51-d8b65066d5f6
LanceDB
rethinking-table-file-paths-lance-multi-base-layout
Jack Ye
rag-isnt-one-size-fits-all
Leonard Marcq
python-package-to-convert-image-datasets-to-lance-type
Vipul Maheshwari
one-million-iops
Weston Pace
november-feature-roundup
Will Jones
newsletter-september-2025
Jasmine Wang
newsletter-october-2025
Jasmine Wang
newsletter-november-2025
ChanChan Mao
newsletter-june-2025
David Myriel
newsletter-july-2025
Jasmine Wang
newsletter-january-2026
ChanChan Mao
newsletter-february-2026
ChanChan Mao
newsletter-december-2025
ChanChan Mao
newsletter-august-2025
Jasmine Wang
my-summer-internship-experience-at-lancedb-2
Raunak Sinha
my-simd-is-faster-than-yours-fb2989bf25e7
LanceDB
multimodal-myntra-fashion-search-engine-using-lancedb
LanceDB
multimodal-lakehouse
David Myriel
multi-document-agentic-rag-a-walkthrough
Vipul Maheshwari
modified-rag-parent-document-bigger-chunk-retriever-62b3d1e79bc6
Mahesh Deshwal
memgpt-os-inspired-llms-that-manage-their-own-memory-793d6eed417e
Ayush Chaurasia
late-interaction-efficient-multi-modal-retrievers-need-more-than-just-a-vector-index
Ayush Chaurasia
lancedb-x-continue
LanceDB
lance-x-huggingface-a-new-era-of-sharing-multimodal-data
Prashanth Rao
Quentin Lhoest
Xuanwo
Ayush Chaurasia
lance-x-duckdb-sql-retrieval-on-the-multimodal-lakehouse-format
Xuanwo
lance-windows-windows-lance
Chang She
lance-v2
Weston Pace
lance-namespace-lancedb-and-ray
Jack Ye
lance-file-2-1-stable
Weston Pace
lance-file-2-1-smaller-and-simpler
Weston Pace
lance-data-viewer
Gordon Murray
lance-community-governance
Jack Ye
introducing-lance-namespace-spark-integration
Jack Ye
implementing-corrective-rag-in-the-easiest-way-2
LanceDB
hybrid-search-rag-for-real-life-production-grade-applications-e1e727b3965a
Mahesh Deshwal
hybrid-search-combining-bm25-and-semantic-search-for-better-results-with-lan-1358038fe7e6
LanceDB
hybrid-search-and-custom-reranking-with-lancedb-4c10a6a3447e
LanceDB
how-to-reduce-hallucinations-from-llm-powered-agents-using-long-term-memory-72f262c3cc1f
Tevin Wang
guide-to-use-contextual-retrieval-and-prompt-caching-with-lancedb
LanceDB
grpo-understanding-and-fine-tuning-the-next-gen-reasoning-model-2
Mahesh Deshwal
graphrag-hierarchical-approach-to-retrieval-augmented-generation
Akash Desai
gpu-accelerated-indexing-in-lancedb-27558fa7eee5
LanceDB
geo-support
Jack Ye
geneva-twelvelabs
David Myriel
geneva-feature-engineering
Jonathan Hsieh
from-bi-to-ai-lance-and-iceberg
Jack Ye
Prashanth Rao
fluss-integration
Wayne Wang
file-readers-in-depth-parallelism-without-row-groups
Weston Pace
feature-rabitq-quantization
David Myriel
Yang Cen
feature-full-text-search
David Myriel
enhance-rag-integrate-contextual-compression-and-filtering-for-precision-a29d4a810301
Kaushal Choudhary
effortlessly-loading-and-processing-images-with-lance-a-code-walkthrough
LanceDB
designing-a-table-format-for-ml-workloads
Weston Pace
custom-dataset-for-llm-training-using-lance
LanceDB
creating-a-fintech-agent
Vipul Maheshwari
convert-any-image-dataset-to-lance
LanceDB
columnar-file-readers-in-depth-structural-encoding
Weston Pace
columnar-file-readers-in-depth-repetition-definition-levels
Weston Pace
columnar-file-readers-in-depth-compression-transparency
Weston Pace
columnar-file-readers-in-depth-column-shredding
Weston Pace
columnar-file-readers-in-depth-backpressure
Weston Pace
columnar-file-readers-in-depth-apis-and-fusion
Weston Pace
chunking-techniques-with-langchain-and-llamaindex
Prashant Kumar
chunking-analysis-which-is-the-right-chunking-approach-for-your-language
Shresth Shukla
chat-with-csv-excel-using-lancedb
LanceDB
case-study-netflix
David Myriel
case-study-dosu
Qian Zhu
Michael Ludden
case-study-cognee
David Myriel
Vasilije Markovic
case-study-coderabbit
Qian Zhu
building-rag-on-codebases-part-2
Sankalp Shubham
building-rag-on-codebases-part-1
Sankalp Shubham
branching-and-shallow-clone
Jack Ye
better-rag-with-active-retrieval-augmented-generation-flare-3b66646e2a9f
LanceDB
benchmarking-random-access-in-lance
Chang She
benchmarking-lancedb-92b01032874a-2
LanceDB
benchmarking-cohere-reranker-with-lancedb
LanceDB
anythingllms-competitive-edge-lancedb-for-seamless-rag-and-agent-workflows
Ayush Chaurasia
announcing-lance-sdk
Weston Pace
agentic-rag-using-langgraph-building-a-simple-customer-support-autonomous-agent
LanceDB
advanced-rag-precise-zero-shot-dense-retrieval-with-hyde-0946c54dfdcb
LanceDB
accelerate-vector-search-applications-using-openvino-lancedb
LanceDB
a-primer-on-text-chunking-and-its-types-a420efc96a13
Prashant Kumar
a-practical-guide-to-training-custom-rerankers
Ayush Chaurasia
a-practical-guide-to-fine-tuning-embedding-models
Ayush Chaurasia
keep-your-data-fresh-with-cocoindex-and-lancedb
Prashanth Rao
Linghua Jin

Introducing LanceDB Realtime Engine: Streaming Writes You Can Query in Milliseconds

•
November 18, 2026
•
AnnouncementEngineering

Today, we're introducing the LanceDB Realtime Engine, which brings streaming writes and live scalar, full-text and vector indexes to LanceDB Enterprise. New rows can be searched within milliseconds, and every acknowledged write is already durable in object storage.

For AI workloads, getting a row into storage is only part of the problem. Applications need to look it up, filter it and search its text and embeddings while more data keeps arriving. Take a robotics engineer replaying a grasp that just failed, or an RL researcher pulling up a rollout that finished a second ago:

  • Robotics and physical AI training data. We are working with a growing number of robotics and physical AI labs. Their fleets continuously produce images and sensor readings, while annotation and feature pipelines add labels and embeddings. These labs need to find new frames, inspect annotations and assemble training datasets as the data arrives. A corrected label should affect the next filter, and a recomputed embedding should show up in the next vector search, without waiting for the next batch commit or index refresh.
  • Reinforcement learning traces and rollouts. Across the AI labs we work with, reinforcement learning workloads produce a continuous stream of model responses, tool calls and artifacts. Generation workers write trajectories, graders add rewards, and learners consume the resulting training examples. These pipelines need many concurrent writers, fast reads of token and training-signal columns, and a reproducible record of which rollouts trained each checkpoint, all while new rollouts keep arriving.

In both cases, reads have to keep up with writes. Lakehouses, however, were built around large batch commits that collect data, write efficient columnar files to object storage, then publish a new table version. That design makes large datasets economical to store and scan, but it creates a tradeoff for streaming workloads. Commit more often, and small files and metadata updates pile up. Wait for a larger batch, and readers wait too.

The LanceDB Realtime Engine removes this tradeoff. Writes don't wait for a table commit, and readers don't wait for a batch. Full-text and vector indexes stay live as rows arrive, so applications can read and search new data without waiting for a separate refresh job or a merge into the base table.

To make this possible, we developed the MemWAL format as part of the Lance table format. It gives Lance a log-structured merge tree (LSM tree) on object storage, with a write-ahead log, in-memory indexed buffers, and indexed data files. LanceDB Enterprise is the best engine for this new capability; it executes reads and writes across the entire LSM tree and handles data compaction and index maintenance in the background.

Comparing lakehouse ingestion solutions

Over the past several years, the industry has worked on this problem at two layers. At the storage layer, Iceberg and Delta Lake continue to improve merge efficiency with deletion vectors, while Apache Paimon builds an LSM tree into its table format to support streaming updates.

At the ingestion layer, cloud platforms and compute vendors typically add a write-buffering layer on top of a lakehouse format. New data can be acknowledged or served from that layer well before it becomes available in the underlying table. These paths also bring constraints such as append-only ingestion and limited native vector and full-text index support, often requiring another synchronization into a search service.

Access to proprietary write buffers adds another layer of vendor lock-in until the data is flushed into underlying lakehouse tables. During this buffering window, other lakehouse engines cannot read the buffered data through the open table format, preventing cross-engine real-time reads. Durability also often depends on vendor-managed buffers and replication rather than the backing object store, making data loss prevention and recovery more complicated across the system.

The figure below follows one write through each approach.

How soon a new row can be read and searched Three ways to stream rows into a table, on one time axis. Readable means lookups and scans return the row; searchable means full-text and vector searches find it through an index. Direct writes to other lakehouse formats make a row durable and readable at the next commit, and it becomes searchable once a sync job adds it to a separate search index. Streaming ingestion into other lakehouse formats makes it durable in the ingestion service's own log and readable through the service while the row exists only outside the table, then copies it into the table. The LanceDB Realtime Engine makes it durable at about the same time, but in the table's own write-ahead log, and readable and searchable through the table's own indexes milliseconds later. Direct writes to other lakehouse formats Readers see new rows after the next commit. Search runs on a separate index that a sync job keeps up to date. readable, but not yet in the search index durable and readable after the next commit searchable in a separate index, after a sync job Streaming ingestion into other lakehouse formats New rows are readable through the service before they are copied into the table. rows live only in the service, outside the table durable in the service's log readable through the service in the table after the copy LanceDB Realtime Engine The row goes into the table's own log (Lance's MemWAL format) and is indexed in memory, so nothing waits on a copy or an index job. durable in the table in its own write-ahead log readable and searchable milliseconds later time after the write Readable means lookups and scans return the row. Searchable means full-text and vector searches find it through an index.

The LanceDB Realtime Engine removes these tradeoffs by combining streaming writes, fully indexed live search and object-storage durability in one open Lance table. With no separate buffer or search-sync layer outside the table, it is the most native ingestion solution for the lakehouse.

Horizontally scalable and lightning fast

In our Realtime Engine benchmark on Amazon EKS, the median time from a durable write in S3 to a readable row was 52 ms. We ran the tests with three public datasets spanning web text, search passages, and image-text pairs: FineWeb-Edu, MS MARCO, and LAION.

Each single-node run used one m8gd.8xlarge node (30 vCPU, 112 GiB). A write counted only after it was durable in S3, and every run checked that the table ended up with exactly the acknowledged rows.

With live primary-key indexing, durable write throughput grew from 79-83 MB/s on one shard to 312-334 MB/s on four shards, all on a single node. Across four nodes, each serving its own four-shard table, combined throughput reached 946 MB/s.

Throughput with more shards Grouped bars of acknowledged MB/s on one node with 1, 2 and 4 shards: FineWeb-Edu 83, 169, 334; MS MARCO 81, 162, 312; LAION 79, 160, 314. Throughput grows with shards One node. Acknowledged MB/s, durable in S3. FineWeb-Edu MS MARCO LAION 0 100 200 300 400 MB/s 83 81 79 1 shard 169 162 160 2 shards 334 312 314 4 shards Primary-key index live. Medians of three runs.

A newly acknowledged row became visible to another client after 52 ms at the median. During live ingestion, median latency for queries on fresh rows was 12-14 ms for key lookups, 15-28 ms for full-text search, and 26-30 ms for vector search.

Fresh-data read latency A just-acknowledged row is found by another client after 52 ms at the median, and no row was ever missing. Median latency on new rows: key lookup 12-14 ms, full-text search 15-28 ms, vector search 26-30 ms. Fresh rows, read in milliseconds While writes stream in, with every index kept up to date in memory. Write returns, then another client finds the row by key 52ms median No row ever missing every lookup found the row that was written Median read latency on new rows range across the three datasets 0 10 20 30 40 ms Key lookup 12-14 ms Full-text search 15-28 ms Vector search 26-30 ms 476K reads, each checked against the row that was written.

Writing into a base table that already held the entire FineWeb-Edu or MS MARCO corpus was at least as fast as writing into a small one. Existing data stays searchable while new rows arrive.

The engine can write each incoming image straight to object storage and keep only a small reference in memory. On LAION with only the primary-key index live, this cut peak node memory from 74 GiB to 18 GiB, while throughput stayed within 3%.

Keeping an index live costs CPU on the node, and the cost depends on the data. On LAION, where images make up most of each row, keeping every index live costs little (289 MB/s with every index live, vs. 314 MB/s with only the primary-key index). Long text documents need more full-text indexing work for each byte written. Each table may optionally choose which of its indexes the engine keeps live, balancing write throughput against query speed on the newest rows.

The throughput numbers above are the rates the nodes accepted. Holding a high write rate for hours also needs maintenance to merge and index data as fast as it arrives. Maintenance runs on its own workers and scales separately from the nodes, so a table with heavy sustained writes gets more workers.

How the Lance MemWAL format works

These results rest on two layers: the open Lance MemWAL format and the engine that runs it. The MemWAL format gives Lance an LSM-tree structure for streaming data. New rows are logged in object storage and indexed in an in-memory MemTable. Full MemTables are then written out as indexed Lance files called SSTables, which are merged into the base table in the background. The log, metadata, files and indexes are all part of the open Lance table format.

Lance MemWAL architecture Writes are routed to shards. Each shard keeps a MemTable in memory with live indexes, appends every write to a WAL file in object storage before the write returns, and flushes the MemTable as an indexed SSTable when it is full. Compaction merges SSTables into the base table, which keeps its own indexes. The WAL, SSTables and base table all live in the Lance table in object storage. Writes In memory Lance table in object storage SSTables Shard A MemTable indexes kept live 3 2 1 WAL, one file per write Shard B MemTable indexes kept live 2 1 WAL, one file per write flush when full every write, before it returns compaction Base table, with its own indexes Each shard has one writer, and shards are written in parallel. SSTables are numbered by generation; higher is newer.

Each shard writes independently, so ingestion can scale across nodes without a table-wide commit for every write. The table below compares what each format provides on the streaming write path:

CapabilityLanceIcebergDelta LakePaimon
LSM structure✓✗✗✓
No table-wide commit bottleneck✓✗✗✗
Indexing across LSM tree layers✓✗✗✗

More shards and nodes let Lance scale toward object-storage write throughput. The Lance format specification describes the storage layout for compatible readers and writers.

How the LanceDB Realtime Engine works

The Realtime Engine turns the open Lance format into a managed service for streaming writes and live indexed reads. Each write goes to a Realtime Engine node, which persists it in the table's write-ahead log in object storage and adds it to an indexed MemTable. The write returns once the data is durable, so if a node goes away, nothing it acknowledged is lost. Queries search the MemTables alongside the SSTables and the base table.

The Realtime Engine in LanceDB Enterprise Agents, apps and pipelines send writes and queries to Realtime Engine nodes. Each node holds MemTables for several shards, possibly of different tables, and keeps their indexes live. A write is stored in the table's write-ahead log in object storage before it returns. Full MemTables are flushed as indexed SSTables. Maintenance workers, planned and scheduled by the engine, merge SSTables into the base table and update its indexes, and more workers can be added for bursts. Everything in object storage is part of the Lance table. The Realtime Engine in LanceDB Enterprise Nodes take writes and serve the newest rows. Workers merge them into the table. Agents Apps Pipelines writes and queries Realtime Engine nodes Node 1 MemTables with live indexes table A, shard 1 table A, shard 2 Node 2 MemTables with live indexes table B table C Maintenance workers Worker 1 Worker 2 more workers for bursts planned and scheduled by the engine Object storage all of it part of the Lance table Write-ahead log SSTables, with indexes Base table with its indexes + rows durable before the write returns flush merge and index

The engine serves vector search, full-text search, point lookups and scalar filters on MemTables with low latency. It keeps enabled indexes current as new rows arrive, so applications can search recent and historical data through the same table. LanceDB Enterprise manages shard placement, writer recovery, memory limits and backpressure as the workload grows.

LanceDB Enterprise also runs distributed maintenance on these tables, including data compaction, index updates and garbage collection. Dedicated workers merge SSTables into the base table, update its indexes and reclaim obsolete files. Maintenance scales independently of ingestion and query serving.

While this work runs, queries keep using the SSTable indexes that cover each row until the base table's indexes catch up, so new rows never drop out of search:

Moving rows into the base table Shard A has SSTables gen 1 to gen 4. Compaction merges gen 1 into the base table, and the same commit moves the merged watermark past gen 1. Queries keep reading gen 1 through its own indexes. The base table's index then adds the gen 1 rows, and that commit moves the indexed watermark past gen 1. Once both watermarks are past gen 1, queries read those rows from the base table, and gen 1's files are safe to delete. Moving rows into the base table Two watermarks track how far each shard's SSTables have been merged and indexed. Shard A SSTables, oldest first gen 1 gen 2 gen 3 gen 4 gen 1 merged indexed Queries keep reading an SSTable, through its own indexes, until both watermarks are past it. Base table rows gen 1 rows index 1. Compaction merges gen 1 into the base table. The same commit moves the merged watermark. 2. The base table's index adds the gen 1 rows. That commit moves the indexed watermark. 3. Both watermarks are past gen 1. Queries stop reading it, and its files are safe to delete.

Query the data you just wrote

With the Realtime Engine, LanceDB Enterprise brings streaming writes, live search and automated maintenance together on open Lance tables. Robotics teams can find new frames and corrected labels as soon as they land, and RL pipelines can train on fresh rollouts while new ones keep arriving. Every acknowledged write is already durable in object storage, and the data stays in the open Lance format from the first write to long-term storage.

Talk to our team about running the Realtime Engine on your robotics streams, RL rollouts or other live AI data, at the ingestion rates you need.

Xuanyu Zhan
Software Engineer
Jack Ye
Data engineer and table format specialist, working on distributed systems and modern data lake architectures.
Dan Rammer
Senior Software Engineer

LanceDB Raises $57.5M as AI Research Outgrows the Modern Data Stack

Chang She
•
November 16, 2026
lancedb-series-b-funding

Lance Table Format in Depth: What is a Table Format?

Weston Pace
•
November 2, 2026
table-formats-in-depth-what-is-a-table-format

🤖 LeRobot v0.7.0 on Lance, 🤝 ASOF Joins in DataFusion, 🔄 LLM Pretraining from One LanceDB Table

ChanChan Mao
•
November 11, 2026
newsletter-october-2026