📊 Lance vs Delta vs Iceberg, 🔗 Lance Blob V2 Late Materialization, 🤖 Stable-Worldmodel Research Platform

ChanChan Mao

•

July 9, 2026

•

Newsletter

Table of Contents

This is a title

This is a subtitle

‍_{📊 A Metadata Benchmark of Lance, Delta Lake, and Iceberg on S3}

In a 10,000-commit benchmark on S3, Lance averaged 140ms commit latency versus 534ms for Delta Lake and 457ms for Iceberg. Active metadata footprint stayed at 0.78 MiB compared to 42+ MiB for Iceberg.

At 200 concurrent writers, Lance saw a 16% failure rate versus 88% for Delta Lake and 89–94% for Iceberg. The gap comes from publishing compact manifests directly to storage instead of relying on log replay or a catalog service.

Updating a label on a row with a 50MB video shouldn’t require Spark to read that video — Lance Blob V2 keeps blob columns as lightweight descriptors through the query plan, materializing bytes only at write time.

UPDATE, MERGE, INSERT…SELECT, and joins all carry copy tokens that resolve during the write path, so mixed tables with small thumbnails and large clips can use different layouts row by row without schema changes. The SQL stays the same — what changes is that Spark moves references instead of payloads.

stable-worldmodel’s Lance data layer delivers ~4,800 samples/sec on Push-T versus ~1,400 for HDF5 locally, and ~3,200 samples/sec reading directly from S3 — training from object storage without syncing to local disk first.

The loader is URI-agnostic — s3://, gs://, HF Buckets, or local paths all run the same code — and Lance’s zero-copy data evolution adds vector, FTS, or hybrid indexes to training data at no performance cost. LeWorldModel trains end-to-end from pixels on a single H200 in hours, hitting 94% success on Push-T with 50× less planning latency than DINO-WM.

🎤 Talks & Recordings

Less Tokio is More Tokio: Strategies for Accelerating a Rust Native Search Database by Weston Pace, Lu Qiu

Weston Pace (LanceDB) · Lu Qiu (LanceDB)

Weston Pace and Lu Qiu present how LanceDB achieved storage roofline performance by redesigning their Rust I/O loop to use fewer Tokio tasks, eagerly poll futures during I/O submission, and adopt a push/pull scheduling API—along with profiling techniques for complex async applications.

Watch the recording →

Agents Will Break Your Stack

Chang She (LanceDB) · Ben Lorica (Gradient Flow)

Chang She, co-founder and CEO of LanceDB, discusses with Ben Lorica why traditional analytical tools like Pandas and Parquet fall short for AI workloads, the concept of a “multimodal lakehouse,” and the unique data storage and retrieval challenges that AI agents introduce.

Watch the recording →

Trillion is the New Billion: Managing Really Large Multimodal Datasets for AI

Lei Xu (LanceDB)

This talk covers the data infrastructure challenges of managing trillion-row multimodal datasets for AI workloads, explaining why existing systems fall short for storing large blobs, supporting search/curation/training directly from datasets, and handling distributed pipelines across clouds—with a look under the hood at how Lance format and LanceDB address these problems and fit alongside Iceberg.

Watch the recording →

📸 AI Engineer World's Fair in SF

We had a great time at AI Engineer World's Fair in San Francisco last week! Thanks to everyone who stopped by the booth to say hi and dive into LanceDB internals.

It was great meeting everyone who came out to our events last week: SPIN SF with Theory Ventures & Ollama and USA World Cup Watch Party with Exa, Extend, and Fastino Labs. Both were packed, both were a good time, and we'd absolutely do them again.

📅 Upcoming Events

AI Lakehouse Meetup — July 15, 2026 · 5-8pm PT · San Jose, CA

ChanChan Mao (LanceDB) · Lu Qiu (LanceDB)

Covers building multimodal lakehouse with Lance and LanceDB—unified vector, full-text, and SQL search across text, images, and embeddings without forcing separate systems for each modality. Hosted at Cloudera’s San Jose office with talks by ChanChan Mao and Lu Qiu from LanceDB.

Ceph: Object Storage Meets Vector Search — July 16, 2026 · 10am PT / 1pm ET · Virtual

Prashanth Rao (LanceDB) · Kyle Bader (IBM) · Christine Bassani (Seagate)

Ceph is integrating LanceDB libraries to deliver vector search through S3 Vectors API actions, enabling billion-scale, multi-tenant nearest neighbor search directly within existing object storage infrastructure. The implementation targets RAG workloads without requiring a separate vector database deployment.

🏗️ LanceDB Enterprise Updates

Performance

Faster query routing — Caching per-query routing tables instead of rebuilding them on every request removed a CPU bottleneck that had cut throughput by roughly 13x under random-take (point-lookup) workloads.
Smarter cluster load placement — A new two-choice consistent-hashing algorithm spreads index segments and query routing more evenly across nodes, with the underlying hash function now pinned so placement doesn't reshuffle if the Rust standard library's hash implementation changes.
Faster scalar index scans — On billion-row indexes, scalar index results now serialize directly from the selected-row bitmap instead of rebuilding a merged copy, cutting CPU usage on this path by roughly 10% in steady state.‍
Lower-overhead index operations at scale — Index status checks now read directly from stored manifests instead of disk, and index cache prewarming for multi-segment indexes drops from O(n²) to O(n) in segment count, reducing overhead as clusters grow.
‍Faster reads on recently written data — Row-limited queries used to scan every generation of the not-yet-compacted layer in full before applying the limit; bounding the scan to the requested range instead cut a test query against an 18-generation backlog to roughly 27 requests and under 3 MB.‍
Reliability — Index cache prewarming now covers every replica group, preventing cold-cache latency spikes on groups that weren’t previously warmed.

Features

Feature	Description
Branching for tables	Tables now support git-like branches — create, delete, checkout — with indexing, caching, and writes isolated per branch, so teams can experiment without touching production data.
Expanded distributed indexing	The distributed indexing system now builds bitmap, label-list, and FM-index types, and distributes full-text and scalar indexing (including nested vector fields) across workers to keep pace with larger, more varied datasets.
Faster access to newly written data	The low-latency layer serving data before compaction now supports full-text search, row-level deletes, and partial-column updates — freshly written data is searchable and editable without waiting on compaction.
New job management system	A new job coordination system tracks background job runs end-to-end — discovery, event logging, and run-status APIs — improving visibility and reliability for long-running work.
Cache placement controls	New admin APIs and CLI commands control where index and table caches are placed and prewarmed across a cluster, reducing cold-cache query latency at scale.

🌟 Open Source Releases

Project	Description
Lance v8.0.0 Release notes	• FM-Index scalar index for exact substring search with configurable multi-segment builds (#7026, #7123); ngram index now accelerates regex and infix `LIKE` queries (#7139) • RaBitQ vector search gains approx mode (#7179), SIMD reranking kernels (#7205), and vectorized distance-table quantization (#7241) for faster ANN • Full-text search gets block-max WAND pruning (#7089), shared top-k thresholds across partitions (#7062), and deferred posting-list loading (#6983) • Write-ahead tier now matches hnswlib throughput via AVX-512 distance kernels (#7009) and closes the FTS gap with Lucene (#7029)
LanceDB v0.34.0 Release notes	• Table branching — create, checkout, and manage branches for local and remote tables, enabling git-like version control (#3490, #3504, #3540) • FM-Index scalar index enables efficient `LIKE '%substring%'` queries (#3532) • Native Polars DataFrame integration alongside PyArrow and Pandas (#3584) • Breaking: repeated `.where()` filters now combine with AND instead of silently replacing the prior one (#3585)
lance-namespace v0.8.0 – v0.8.6 Release notes	• Table branching — create, delete, switch — via new API endpoints and a branch parameter (#350, #352) • `IndexContent` now includes `describe_indices` metadata, exposing index config through the API (#349) • Materialized view refresh now accepts `source_task_size` to control task granularity, with Java client support (#355, #356)
lance-context v0.3.0 – v0.5.0 Release notes	• Post-training pipeline: `export_training()` curates context records into SFT, preference (DPO/SimPO/ORPO paired, KTO unpaired, judge-ranked N-way), and RL-rollout datasets with manifest-backed exports, group-disjoint train/eval splits, semantic dedup, and decontamination (#96, #103, #111) • Retrieval evaluation harness: `evaluate()` and `evaluate_versions()` compute recall@k, precision@k, MRR, nDCG@k, and hit-rate against labeled query sets, with cross-version A/B comparison and version-pinned reports (#98, #110) • Write-path additions: MemWAL integration (#36), async Python API via `AsyncContext` (#61), REST API server with Rust client SDK (#66), hybrid retrieval API (#76), upsert/partial-update APIs (#84), and bulk `upsert_many()` with indexed, sub-linear uniqueness validation replacing full O(n²) dataset scans on writes (#100, #102)
lance-trino v0.3.2 Release notes	• Predicate filters on nested/struct columns now push down to the Lance format layer (#164) • Filtered `LIMIT` queries skip full fragment enumeration and coalesce splits, cutting overhead for selective queries (#121, #159)
lance-spark v0.5.0 – v0.5.1 Release notes	• `SEARCH`, `VECTOR_SEARCH`, and `HYBRID_SEARCH` table functions bring vector and hybrid search into Spark SQL (#582) • Zonemap-based fragment pruning and storage-partitioned join (SPJ) support improve query planning with Lance indices (#396, #425) • Float16 vector, Map type, and blob v2 read/write support expand data type coverage (#378, #379, #548, #560) • Native update path preserves stable row IDs (#407); `use_large_var_types` prevents 2GB Arrow vector overflow on large string/binary columns (#413)

🫶 Community Contributions

Thank you to contributors from Netflix, Uber, ByteDance, Microsoft, Pinterest, and Adobe for improvements across storage, indexing, query execution, distributed processing, and ecosystem integrations in LanceDB, Lance, and the broader ecosystem.

Notable contributions this month:

@LuciferYang — Added Spark 4.1 TimeType support, VectorUDT writes, and automatic UDT column round-tripping via __udt field metadata in lance-spark
@beinan — Implemented zonemap-based fragment pruning with storage-partitioned join (SPJ) support and distributed zonemap index builds in lance-spark
@wombatu-kun — Added float16 vector type support, table rename, and ALTER TABLE SET/UNSET properties to lance-spark
@summaryzb — Added Map type support, custom Lance scan metrics, and non-microsecond Arrow timestamp column reading in lance-spark
@ivscheianu — Implemented native update path with stable row-id preservation and compression config via Spark TBLPROPERTIES in lance-spark
@atakanyenel — Fixed UInt4 to BIGINT mapping and optimized split coalescing for LIMIT queries in lance-trino
@xuzha — Added --file-format-version option to TPC-DS benchmark data generator and exposed state metadata in lance-context Python API
@puchengy — Made Hive2Namespace and Hive3Namespace Closeable in lance-namespace-impls and fixed string literal parsing in lance-spark DDL
@ykozxy — Added LOCATION support to CREATE TABLE for custom paths and existing datasets in lance-spark
@fangbo — Added configurable rows_per_range for range-based btree index builds and optimized vector data export to Rust for OOM prevention in lance-spark

A heartfelt thank you to our community contributors of Lance and LanceDB this past month:

@a-agmon • @aheev • @aimanmalib • @ali2arslan • @alowator • @ar-maan05 • @atakanyenel • @beinan • @bryanck • @burlacio • @butnaruandrei • @chunxutang • @claydugo • @cwj0bzxg • @danielmao1 • @dcfocus • @ddupg • @devteamaegis • @dhruvgarg111 • @everysympathy • @fangbo • @fanng1 • @geserdugarov • @gstamatakis95 • @haochengliu • @hashwnath • @hfutatzhanghb • @huahuay • @ilya-zlobintsev • @ivscheianu • @jerryjch • @jiaoew1991 • @jja725 • @jo-migo • @jsap0914 • @jtuglu1 • @julianyg • @k • @kushudai • @lalitx17 • @liuzemei • @lixmgl • @luciferyang • @luizfonseca • @majin1102 • @mansiverma897993 • @mehulbatra • @missing-identity • @mohit-twelvelabs • @mr-neutr0n • @neo-x7 • @noethix55555 • @nuthalapativarun • @paramt • @pjdurden • @plotor • @puchengy • @ritwijparmar • @rtmalikian • @sarahnasser576 • @sezruby • @shiwk • @shiyan-xu-ai • @sinianluoye • @siriapps • @skyshineb • @solarcloud7 • @stumpylog • @summaryzb • @touch-of-grey • @valkum • @vinaysurtani • @vitaliy-pikalo • @wangxiaobao1222 • @wending-y • @whitewooood • @wirybeaver • @wombatu-kun • @wulansari999 • @xiaguanglei • @xloya • @xuqianjin-stars • @xushiyan • @xuzha • @yanghua • @yesunbmh • @ykozxy • @yuanggao • @yuju-huang • @yyzhao2025 • @zhangyang0418 • @zhangyue19921010 • @zouhuajian • @ztorchan

🤝 Lance Community Sync Recap

This month’s community syncs covered governance and format evolution for the Lance ecosystem. Format governance changes were discussed, including a requirement for votes before merging format changes and improved design review processes. A new shuffle format proposal was presented aimed at reducing IOPS and better supporting GPU workloads, alongside discussions of new indexes and core Lance architecture.

The next Lance Community Sync will take place on Thursday, July 16 @ 9am PT.

ChanChan Mao

Developer Relations @ LanceDB

GitHub