Why Multimodal Data Needs a Better Lakehouse? — Download the Research Study
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.
Searching in Blog Posts
Reproducible Data Curation In The Multimodal Lakehouse
Learn how LanceDB turns raw multimodal data into reproducible, training-ready datasets with search, filtering, deduplication, sampling, and versioned curation workflows.
🌍 Lance-Backed World Model Platform, 🦆 Multimodal SQL with Lance DuckDB Extension, 💰 LanceDB vs OpenSearch Cost Breakdown
stable-worldmodel standardizes world model pipelines on Lance, DuckDB Lance extension adds native multimodal SQL, and LanceDB benchmarks 100M vectors at ~$779/month, plus upcoming events, enterprise updates, and community updates.
⚡Vector Search at 10B Scale, 📊 Lance Format Benchmarks, 🚗 AV Pipelines at Scale
Distributed vector search at 10B scale, more efficient storage with Lance format v2.2, and production AV pipelines simplified, plus upcoming events and community updates.
How LanceDB Accelerates Vector Search at 10 Billion Scale
How LanceDB applies distributed indexing, distributed query execution, HNSW centroid routing, and fast RaBitQ rotation to scale search to 10B vectors and beyond.
OpenSearch vs LanceDB for Vector Search: Query Cost and Infrastructure
Choosing a vector database usually comes down to a tradeoff between a full search service and an in-process library. This post showcases benchmarks that compare OpenSearch and LanceDB on the COCO 2017 images embedded with SigLIP. We measure ingestion throughput, query cost, storage layout, and overall infra cost.
Volcano Engine LAS's Lance-Based PB-Scale Autonomous Driving Data Lake Solution
How Bytedance Volcano Engine LAS (Lake for AI Service) leverages Lance as the core storage format, rapidly constructing a next-gen AI data lake to efficiently store, manage, and process multimodal data (text, images, audio/video).
Unifying the AV ML Stack: From Raw Data to Trained Model with LanceDB
A complete walkthrough of building an autonomous vehicle perception model training pipeline on top of LanceDB and the Multimodal Lakehouse.
Lance JSON Support: Why You Might Not Really Need Variant
Lance's JSONB storage, scalar indexing, data evolution, and full-text search already deliver what most users want from Variant — with explicit control, schema consistency, and no vendor lock-in.
Building A Storage Format For The Next Era of Biology
How Lance can serve as the foundation for AI on single-cell genomics atlases and a new generation for modeling in biology.
📄 Lance Blob V2, 🤗 Upload Lance Datasets to HF Hub, 🦞 LanceDB for OpenClaw's Memory
Lance Blob V2 introduces adaptive storage semantics, easily upload Lance datasets to Hugging Face Hub, and OpenClaw establishes LanceDB as a default memory layer for agents, plus community and enterprise updates.
Smart Parsing Meets Sharp Retrieval: Combining LiteParse and LanceDB
Build a structure-aware PDF QA agent with LiteParse, LanceDB, and Claude to answer complex questions over visually rich documents.
Lance Format v2.2 Benchmarks: Half the Storage, None of the Slowdown
Benchmarks that show how Lance format v2.2 cuts storage by 50%+, beats Parquet on compression, and delivers up to 68x faster blob reads — while preserving the scan and random access patterns that multimodal training depends on.
Make your SQL Workflows Multimodal With LanceDB × DuckDB
A hands-on walkthrough showing how LanceDB and DuckDB work together to query multimodal data in SQL, join against multiple tables, and materialize results back into LanceDB.
Agentic Coding as Community Stewardship
Agentic coding lets upstream maintainers turn ecosystem-wide mechanical updates into fast, contextual, low-friction contributions for downstream users.
One System, Many Workloads: Rethinking What "Multimodal" Means for AI
A practical definition of multimodal complexity, and how LanceDB’s Multimodal Lakehouse is built to address these challenges.
The Future of AI-Native Development is Local: Inside Continue's LanceDB-Powered Evolution
Discover how Continue revolutionized AI-native development with LanceDB's embedded TypeScript library, enabling lightning-fast semantic code search while maintaining complete developer privacy and offline capability.
Lance File Format 2.2: Taming Complex Data
Lance file format 2.2 introduces Blob V2, nested schema evolution, native Map type support, and additional compression and performance improvements for AI/ML data workloads.
Lance Blob V2: Making Multimodal Data a First-Class Citizen in the Lakehouse
How we redesigned blob storage in Lance to make multimodal data a first-class citizen, with four storage semantics (Inline, Packed, Dedicated, External) that automatically adapt to your workload.
Why LanceDB Is the Most Natural Memory Layer for OpenClaw
OpenClaw and similar personal autonomous agents need a local-first long-term memory layer. LanceDB fits that role with embedded deployment, filesystem-native storage, and multimodal retrieval.
OpenClaw + LanceDB + Seed 2.0: Turn Visual Ideas into Reality, Fast!
OpenClaw and similar personal autonomous agents need a local-first long-term memory layer. LanceDB fits that role with embedded deployment, filesystem-native storage, and multimodal retrieval.
Memory for OpenClaw: From Zero to LanceDB Pro
Benchmarking three OpenClaw memory plugins on the LOCOMO dataset
A Guide to Uploading Lance Datasets on the Hugging Face Hub
Build a multimodal Lance dataset, publish it to the Hub, and query the precomputed vector + FTS indexes in LanceDB, without needing to download the dataset locally.
Zero Shot Image Classification with Vector Search
Get about zero shot image classification with vector search. Get practical steps, examples, and best practices you can use now.
WeRide's Data Platform Transformation: How LanceDB Fuels Model Development Velocity
Discover how WeRide, a leading autonomous driving company, leveraged LanceDB to revolutionize their data platform, achieving 90x improvement in ML developer productivity and reducing data mining time from 1 week to 1 hour.
Training a Variational AutoEncoder from Scratch with Lance File Format
Train a Variational Autoencoder end‑to‑end using Lance for fast, scalable data handling. You’ll set up the dataset, build the VAE in PyTorch, and run training, sampling, and reconstructions.
Track AI Trends: CrewAI Agents & RAG
This article will teach us how to make an AI Trends Searcher using CrewAI Agents and their Tasks. But before diving into that, let's first understand what CrewAI is and how we can use it for these applications.
Tokens per Second Is NOT All You Need
Explore about tokens per second is not all you need. Get practical steps, examples, and best practices you can use now.
The Future of Open Source Table Formats: Apache Iceberg and Lance
Explore the future of open source table formats: apache iceberg and lance with practical insights and expert guidance from the LanceDB team.
The Case for Random Access I/O
One of the reasons we started the Lance file format and have been investigating new encodings is because we wanted a format with better support for random access.
LanceDB Raises $30M Series A to Build the Multimodal Lakehouse
We have closed another funding round to accelerate development of the Multimodal Lakehouse - a unified platform for AI data infrastructure.
SemanticDotArt: Rethinking Art Discovery with LanceDB
SemanticDotArt turns art discovery into a multimodal search experience, matching feelings, phrases, and images with LanceDB's hybrid retrieval.
Second Dinner's Secret Weapon: LanceDB-Powered RAG for Faster, Smarter Game Development
Discover how Second Dinner, creators of Marvel Snap, leveraged LanceDB Cloud to transform game development workflows, reducing prototyping time from months to hours and automating QA test generation with 81% better results.
Search Within an Image with Segment Anything
Get about search within an image with segment anything. Get practical steps, examples, and best practices you can use now.
Scalable Computer Vision with LanceDB & Voxel51
Explore about scalable computer vision with lancedb & voxel51. Get practical steps, examples, and best practices you can use now.
Rethinking Table File Paths with Uber: Lance’s Multi-Base Layout
A tour of Lance's file path design, and how Lance’s new multi-base layout enables multi-location datasets (such as Uber’s multi-bucket setup) with minimal metadata rewrites.
RAG Isn't One-Size-Fits-All: Here's How to Tune It for Your Use Case
Great RAG comes from a tight iteration loop. Learn how to systematically improve each layer of your RAG system using Kiln and LanceDB.
Python Package to convert image datasets to Lance type
Explore python package to convert image datasets to lance type with practical insights and expert guidance from the LanceDB team.
The Quest for One Million IOPS: Benchmarking Storage at LanceDB
Learn how LanceDB benchmarks storage and how we achieved one million disk reads per second.
November Feature Roundup
Explore November feature roundup with practical insights and expert guidance from the LanceDB team.
🛡️ Newly Knighted Lancelot, ▶️ TwelveLabs Semantic Video Recommendations, 🧠 Cognee's AI Memory Layer with LanceDB
Our September newsletter welcomes new Lancelot members, highlights TwelveLabs semantic video recommendations, Cognee’s AI memory layer, and shares the latest product and community updates.
🎨 Semantic.Art, 💾 Stable Lance 2.1, 🎥 Ray+LanceDB powers Netflix
Our October newsletter highlights Semantic.Art, Lance File 2.1, RaBitQ Quantization, upcoming events, latest product and community updates.
🛡️Lance Community Governance, Lance + Iceberg 🧊, Netflix’s Multimodal Search Demo 🔍
Our November newsletter highlights Lance community governance, a deep dive on Lance and Iceberg, a demo of Netflix's multimodal search, previous talk recordings, and the latest product and community updates.
June 2025: $30M Series A, Multimodal Lakehouse Launch & Product Updates
LanceDB's June 2025 newsletter covering latest company news, product updates, open source releases, and community highlights.
⚖️ Harvey’s Enterprise-Grade RAG on LanceDB, 💼 Dosu Case Study, Minimax&LumaLabs❤️Lance-Ray
Our August newsletter features a new case study with Dosu, recaps from events with Harvey and Databricks, and the latest product and community updates.
🦆 Lance x DuckDB SQL Retrieval, 🚗 Uber-Scale Storage, ⚡ 1.5M IOPS
Kicking off 2026 with Lance-native SQL retrieval via DuckDB, Uber-scale multi-bucket storage, 1.5M IOPS benchmarks, and continued OSS momentum across the Lance ecosystem.
🤗 Lance x Hugging Face, 🪾Git-Style Branching, 🏔️ Geospatial in Lance
Native Lance support on Hugging Face Hub, Git-style branching and shallow clone for AI data, and Arrow-native geospatial with R-Tree indexing, plus steady OSS and community momentum.
💾 Lance SDK v1.0.0, 🗓️ 1st Lance Community Sync, 🔍 Wikisearch
Our December newsletter highlights Lance SDK v1.0.0, our upcoming Lance community sync, Wikisearch demo, and the latest product and community updates.
Netflix’s Media Data Lake ❤️ LanceDB, CodeRabbit 💼 Case Study, Lance Namespace
Our August newsletter highlights LanceDB powering Netflix's Media Data Lake, a case study on CodeRabbit's AI-powered code reviews, and updates on Lance Namespace and Spark integration.
My Summer Internship Experience at LanceDB
I'm Raunak, a master's student at the University of Illinois, Urbana-Champaign. This summer, I had the opportunity to intern as a Software Engineer at LanceDB, an early-stage startup based in San Francisco.
My SIMD Is Faster than Yours
An untold story about how we make LanceDB vector search fast. Get practical steps and examples from 'My SIMD is faster than Yours'.
Multimodal Myntra Fashion Search Engine Using LanceDB
Build a multimodal fashion search engine with LanceDB and CLIP embeddings. Follow a step‑by‑step workflow to register embeddings, create the table, query by text or image, and ship a Streamlit UI.
What is the LanceDB Multimodal Lakehouse?
Introducing the Multimodal Lakehouse - a unified platform for managing AI data from raw files to production-ready features, now part of LanceDB Enterprise.
Multi Document Agentic RAG: a Walkthrough
Unlock about multi document agentic rag: a walkthrough. Get practical steps, examples, and best practices you can use now.
Modified RAG: Parent Document & Bigger Chunk Retriever
Get about modified rag: parent document & bigger chunk retriever. Get practical steps, examples, and best practices you can use now.
MemGPT: OS Inspired LLMs That Manage Their Own Memory
Explore about memgpt: os inspired llms that manage their own memory. Get practical steps, examples, and best practices you can use now.
Late Interaction & Efficient Multi-modal Retrievers Need More Than a Vector Index
Explore late interaction & efficient multi-modal retrievers need more than a vector index with practical insights and expert guidance from the LanceDB team.
Developers, Ditch the Black Box: Welcome to Continue
Remember flipping through coding manuals? Those quickly became relics with the rise of Google and Stack Overflow, a one-stop shop for developer queries.
Lance × Hugging Face: A New Era of Sharing Multimodal Data on the Hub
Announcing native read support for Lance format on Hugging Face Hub. You can now distribute your large multimodal datasets as a single, searchable artifact (including blobs, embeddings and indexes) all in one place!
Lance × DuckDB: SQL for Retrieval on the Multimodal Lakehouse Format
Use the Lance format as your lakehouse layer for retrieval, RAG and more, with the native Lance extension for DuckDB
Lance, Windows. Windows, Lance
It was Spring of 2012. After being an avid user for 2+ years, I finally decided to join Wes Mckinney and work on pandas full time.
Lance v2: A New Columnar Container Format
Explore lance v2: a new columnar container format with practical insights and expert guidance from the LanceDB team.
Productionalize AI Workloads with Lance Namespace, LanceDB, and Ray
Learn how to productionalize AI workloads with Lance Namespace's enterprise stack integration and the scalability of LanceDB and Ray for end-to-end ML pipelines.
Lance File 2.1 is Now Stable
The 2.1 file version is now stable, learn what that means for you and what's coming next.
Lance File 2.1: Smaller and Simpler
Explore lance file 2.1: smaller and simpler with practical insights and expert guidance from the LanceDB team.
Introducing Lance Data Viewer: A Simple Way to Explore Lance Tables
A lightweight open source web UI for exploring Lance datasets, viewing schemas, and browsing table data with vector visualization support.
Building the Future Together: Introducing Lance Community Governance
Announcing the formal governance structure for the Lance community, establishing clear pathways for contribution and leadership with a three-tier system of contributors, maintainers and PMC.
Manage Lance Tables in Any Catalog using Lance Namespace and Spark
Access and manage your Lance tables in Hive, Glue, Unity Catalog, or any catalog service using Lance Namespace with the latest Lance Spark connector.
Implementing Corrective RAG in the Easiest Way
Even though text-generation models are good at generating content, they sometimes need to improve in returning facts. This happens because of the way they are trained.
Hybrid Search: RAG for Real-Life Production-Grade Applications
Get about hybrid search: rag for real-life production-grade applications. Get practical steps, examples, and best practices you can use now.
Hybrid Search: Combining BM25 and Semantic Search for Better Results with Langchain
Have you ever thought about how search engines find exactly what you're looking for? They usually use a mix of matching specific words and understanding the meaning behind them.
Hybrid Search and Custom Reranking with LanceDB
Combine keyword and vector search for higher‑quality results with LanceDB. This post shows how to run hybrid search and compare rerankers (linear combination, Cohere, ColBERT) with code and benchmarks.
Reduce Hallucinations from LLM-Powered Agents Using Long-Term Memory
Understand about reduce hallucinations from llm-powered agents using long-term memory. Get practical steps, examples, and best practices you can use now.
Implement Contextual Retrieval and Prompt Caching with LanceDB
Unlock about implement contextual retrieval and prompt caching with lancedb. Get practical steps, examples, and best practices you can use now.
RAG with GRPO Fine-Tuned Reasoning Model
Explore rag with grpo fine-tuned reasoning model with practical insights and expert guidance from the LanceDB team.
GraphRAG: Hierarchical Approach to Retrieval-Augmented Generation
Explore GraphRAG: hierarchical approach to retrieval-augmented-generation with practical insights and expert guidance from the LanceDB team.
GPU-Accelerated Indexing in LanceDB
Speed up vector index training in LanceDB with CUDA or Apple Silicon (MPS). See how GPU‑accelerated IVF/PQ training compares to CPU and how to enable it in code.
How We Added Geospatial Support To Lance With No New Code
How Lance's Arrow-native architecture enables first-class geospatial support through extension types, GeoDataFusion integration, and R-Tree indexing.
Building Semantic Video Recommendations with TwelveLabs and LanceDB
Build semantic video recommendations using TwelveLabs embeddings, LanceDB storage, and Geneva pipelines with Ray.
LanceDB's Geneva: Scalable Feature Engineering
Learn how to build scalable feature engineering pipelines with Geneva and LanceDB. This demo transforms image data into rich features including captions, embeddings, and metadata using distributed Ray clusters.
From BI to AI: A Modern Lakehouse Stack with Lance and Iceberg
A comparison of where Iceberg and Lance sit in the modern lakehouse stack. We highlight emerging architectures that are bridging the worlds of analytics and AI/ML workloads using these two formats, while being built on the same data foundation.
Setup Real-Time Multimodal AI Analytics with Apache Fluss (incubating) and Lance
Learn how to build real-time multimodal AI analytics by integrating Apache Fluss streaming storage with Lance's AI-optimized lakehouse. This guide demonstrates streaming multimodal data processing for RAG systems and ML workflows.
Columnar File Readers in Depth: Parallelism without Row Groups
Explore columnar file readers in depth: column shredding with practical insights and expert guidance from the LanceDB team.
LanceDB's RaBitQ Quantization for Blazing Fast Vector Search
Introducing RaBitQ quantization in LanceDB for higher compression, faster indexing, and better recall on high‑dimensional embeddings.
LanceDB WikiSearch: Native Full-Text Search on 41M Wikipedia Docs
No more Tantivy! We stress-tested native full-text search in our latest massive-scale search demo. Let's break down how it works and what we did to scale it.
Efficient RAG with Compression and Filtering
Discover about efficient rag with compression and filtering. Get practical steps, examples, and best practices you can use now.
Effortlessly Loading and Processing Images with Lance: a Code Walkthrough
Working with large image datasets in machine learning can be challenging, often requiring significant computational resources and efficient data-handling techniques.
Designing a Table Format for ML Workloads
Explore designing a table format for ML workloads with practical insights and expert guidance from the LanceDB team.
Custom Datasets for Efficient LLM Training Using Lance
See about custom datasets for efficient llm training using lance. Get practical steps, examples, and best practices you can use now.
Creating a FinTech AI Agent From Scratch
Explore fintech ai agent from scratch with practical insights and expert guidance from the LanceDB team.
Convert Any Image Dataset to Lance
In our article, we explored the remarkable capabilities of the Lance format, a modern, columnar data storage solution designed to revolutionize the way we work with large image datasets in machine learning.
Columnar File Readers in Depth: Structural Encoding
Deep dive into LanceDB's dual structural encoding approach - mini-block for small data types and full-zip for large multimodal data. Learn how this optimizes compression and random access performance compared to Parquet.
Columnar File Readers in Depth: Repetition & Definition Levels
Explore columnar file readers in depth: repetition & definition levels with practical insights and expert guidance from the LanceDB team.
Columnar File Readers in Depth: Compression Transparency
Explore columnar file readers in depth: compression transparency with practical insights and expert guidance from the LanceDB team.
Columnar File Readers in Depth: Column Shredding
Explore columnar file readers in depth: column shredding with practical insights and expert guidance from the LanceDB team.
Columnar File Readers in Depth: Backpressure
Streaming data applications can be tricky. When you can read data faster than you can process the data then bad things tend to happen. The various solutions to this problem are largely classified as backpressure.
Columnar File Readers in Depth: APIs and Fusion
The API used to read files has evolved over time, from simple full table reads to batch reads and eventually to iterative record batch readers. Lance takes this a step further to return a stream of read tasks.
Chunking Techniques with Langchain and LlamaIndex
In our last blog, we talked about chunking and why it is necessary for processing data through LLMs. We covered some simple techniques to perform text chunking.
Chunking Analysis: Which is the right chunking approach for your language?
Explore chunking analysis: which is the right chunking approach for your language? with practical insights and expert guidance from the LanceDB team.
Chat with Your Stats Using Langchain Dataframe Agent & LanceDB Hybrid Search
In this blog, we’ll explore how to build a chat application that interacts with CSV and Excel files using LanceDB’s hybrid search capabilities.
Netflix's Media Data Lake and the Rise of the Multimodal Lakehouse
How Netflix built a Media Data Lake powered by LanceDB and the Multimodal Lakehouse to unify petabytes of media assets for machine learning pipelines.
No results found in blog posts
Existing Privacy Settings Found
Submitted On
Mar 26, 2025 – 14:37:22 UTC
Your Consent ID
AMC89MQ329407NSADRFA09MA7SD
Required
Essential Consent
Off
On
Marketing Consent
Off
On
Analytics Consent
Off
On
Personalization Consent
LanceDB uses cookies and similar technologies to improve your experience, analyze traffic, and to show you relevant content and advertising.
  • You can accept all, reject all, or customize your privacy settings.
  • Non-essential cookies are disabled by default.
  • Closing this banner does not confirm any choice.
  • See our Privacy Policy for more information.
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.
LanceDB uses cookies and similar technologies to improve your experience, analyze traffic, and to show you relevant content and advertising.

🌍 Lance-Backed World Model Platform, 🦆 Multimodal SQL with Lance DuckDB Extension, 💰 LanceDB vs OpenSearch Cost Breakdown

ChanChan Mao
•
•
Newsletter
Table of Contents
This is a title
This is a subtitle

🌍 stable-worldmodel: A Platform for Reproducible World Modeling Research and Evaluation

World model training requires small-batch random access across large sequence buffers—a pattern that performs poorly on HDF5 or video formats, especially over the network. stable-worldmodel is an open-source platform with a Lance-based data layer that streams directly from S3 several times faster than alternatives, making ephemeral GPU training practical without local sync.

The platform includes reference implementations of DINO-WM, LeWorldModel, PLDM, and TD-MPC2, plus ~150 environments with controllable visual and physical factors for standardized zero-shot generalization benchmarks. Details and benchmark numbers are in the paper.

Paper → · Code →

🦆 Make Your SQL Workflows Multimodal with LanceDB x DuckDB

database for embeddings, and a SQL warehouse for metadata. The Lance core extension for DuckDB collapses all three—image bytes live as a BLOB column alongside embeddings and structured fields in a single table.

lance_vector_search() is a SQL table function that returns ranked results with raw JPEG bytes inline. Standard SQL handles the rest: joins, filters, aggregations. The blog post walks through setup and query examples.

Read more →

💰 OpenSearch vs LanceDB for Vector Search: Query Cost and Infrastructure

Vector search costs scale with RAM when the index lives in memory — bigger dataset means bigger instance. LanceDB stores the index in S3 and memory-maps it, so RAM scales with QPS rather than dataset size. At 100M docs (1152-dim, SQ8): ~$779/month total. At 10M: ~$148. At 1M: ~$65.

Benchmarked on 287K COCO images with SigLIP 2 embeddings using IVF_HNSW_SQ — above 0.95 recall@10, sub-50ms p95 on a single node. Full cost breakdown and OpenSearch comparison in the post.

Read more →

📚 Also Published

🎤 Talks & Recordings

Managing Data at Exabyte Scale for AI Model Training | Chang She (LanceDB) | OpenXdata 2026

Chang She (LanceDB) discusses how current data infrastructure forces ML teams to copy data across disconnected tools throughout the training workflow, and presents LanceDB’s approach of managing multimodal training data in a unified table to streamline the path from exploration to GPU loading.

Watch the recording →

Q&A with Chang She, CEO, LanceDB: Why the 20-year-old data stack is breaking under AI workloads

Chang She (CEO, LanceDB) discusses why traditional database architectures that separate data storage from file storage via pointers are failing under AI workloads, particularly the throughput challenges posed by agentic data access at scale.

Watch the recording →

📅 Upcoming Events

Microsoft Build — June 2-3, 2026 · San Francisco, CA

Compares multi-system AI data architectures on Azure with unified approaches that use Blob Storage as the foundation for raw data, embeddings, and features. Covers the practical tradeoffs when curation, feature engineering, training, search, and analytics start spreading data across systems.

Register →

Snowflake Summit — June 1-4, 2026 · San Francisco, CA

Apache Polaris is expanding beyond Iceberg to support Delta and Lance formats as an open catalog layer, with new capabilities including S3-compatible on-prem storage, generic tables, a built-in policy store, and catalog federation. Engineers from Snowflake and partners will demo real-world usage and walk through the evolving Polaris REST catalog spec.

Register →

🏗️ LanceDB Enterprise Updates

Performance

  • Page-Boundary Read Coalescing — The storage caching layer now widens reads to page boundaries instead of issuing one request per page, significantly reducing read fan-out and improving throughput for cached access patterns.
  • KNN Pushdown for Filtered Queries — K-nearest-neighbor queries over remotely filtered result sets now push execution closer to the data, reducing data movement and improving latency for hybrid filtered vector searches.
  • IVF_RQ Index in Distributed Indexer — The distributed indexer now supports IVF with residual quantization (IVF_RQ), offering a better compression/speed tradeoff for large-scale vector indexes built across multiple nodes.
  • Vector Index Row Count Persistence — Row counts for vector indexes are now persisted at build time, eliminating expensive full-table scans that were previously required to determine index coverage.‍
  • Distributed Indexer Training Optimization — Single-segment index training in the distributed indexer was optimized to reduce unnecessary work, improving build throughput for tables that fit within one segment.

Features

Feature Description
Distributed Full-Text Search Full-text and BTree index queries can now execute across distributed query nodes using a two-phase RPC coordinator, bringing distributed search on par with distributed vector search.
Segmented Distributed Indexing The distributed indexer now supports fragment-scoped and segmented builds for vector, scalar, FTS, and BTree indexes, enabling finer-grained parallelism and faster incremental index updates at scale.
Two-Tier Index Cache & Prewarm A two-tier index cache (in-memory + NVMe disk) is now available, with new API and CLI commands to prewarm the cache ahead of traffic, significantly reducing cold-start query latency.
Write-Ahead Log A new write-ahead log (WAL) writer built on the Lance OSS memory WAL is now integrated, alongside a unified developer CLI with fuzzy testing support, laying the foundation for stronger durability guarantees.
Feature Engineering Views & Schema Evolution Materialized views and user-defined table-valued function (UDTF) views are now supported in the query engine, along with add_columns and alter_columns schema evolution APIs.

🌟 Open Source Releases

Project Description
Lance v6.0.0 – v7.0.0
Release notes		 • New MemWAL (memory write-ahead log) system with ShardWriter, memtable-based HNSW index, manual compaction APIs, and Java bindings—enables low-latency ingestion pipelines with durable replay (#6669, #6675, #6795, #6833)
• Segmented and distributed index builds: segmented inverted index (#6305), segmented btree (#6605), zonemap index segments (#6593), distributed bitmap index (#6598), and FTS segment merging (#6790)
• SIMD distance kernels for scalar-quantized vectors: u8 dot/L2/cosine (#6506, #6517), bf16 (#6510), f64 (#6540), and 16× faster RaBitQ 4-bit LUT on ARM (#6537)
LanceDB v0.32.0 – v0.33.0
Release notes		 • New IVF_HNSW_FLAT vector index type available in Python (#3366); native FTS now supports model-backed tokenizers (#3289)
• Nested namespace operations for organizing tables hierarchically, with manifest-enabled directory mode (#3265, #3332); Node.js SDK gains connectNamespace and namespace management methods (#3371, #3383)
• PyTorch DataLoader compatibility: Permutation is now fork-safe and picklable for multiprocessing workers (#3339, #3335)
• Vector indexes on nested fields now work correctly—paths are discovered automatically and canonicalized for remote tables (#3408, #3423, #3430)
lance-ray v0.3.0 – v0.4.2
Release notes		 • New dataset maintenance operations: optimize_indices (#2152) and compact_database (#2250) for managing Lance datasets at scale via Ray
• Index creation now supports namespaces via create_index() (#2594) and configurable vector index sample rates (#4631)
• Blob v2 support with legacy compatibility and multi-base config for read/write operations (#1614, #4516)
lance-trino v0.3.0
Release notes		 • Added support for Struct type mapping to Trino RowType, including Substrait schema support for nested field filtering (#96, #125)
• Upgraded to lance-core 6.0.0 and lance-namespace 0.7.6 (#124)
lance-namespace v0.7.3 – v0.7.7
Release notes		 • Added backfill columns and refresh materialized view operations (#335)
• Extended virtual column entries with UDF metadata support (#337)
• API changes for materialized views and UDTFs (#344)

🫶 Community Contributions

Announcing 5 New Lance Maintainers

Thank you to contributors from Bytedance, Adobe, Baidu, Tencent, Hugging Face, Google, Red Hat, Roblox, Uber for improvements across storage, indexing, query execution, distributed processing, and ecosystem integrations in LanceDB, Lance, and the broader ecosystem.

Notable contributions this month:

  • @zhangyue19921010 — Exposed LSM API to Python and Java with distributed bitmap index build support, enabling scalable incremental indexing workflows
  • @beinan — Added segmented btree indices and unenforced clustering key support to the format spec, improving query optimization for large-scale datasets
  • @touch-of-grey — Built write-ahead log appender/tailer primitives and MemWAL HNSW integration, enabling real-time vector indexing for streaming workloads
  • @yanghua — Added Rust support for update by _row_id, enabling efficient row-level mutations in analytical pipelines
  • @alex766 — Added Dataset.takeRows() Java binding for physical row ID access, unlocking low-level row retrieval for JVM applications
  • @Its-Tanay — Added Scannable primitive for Node.js streaming ingestion, enabling efficient large-dataset loading in JavaScript workflows
  • @shenganzhang — Added IVF_HNSW_FLAT vector index support to Python SDK, expanding hybrid search options for production deployments
  • @atakanyenel — Added Struct type mapping to Trino RowType, enabling nested data querying through the Trino connector
  • @devteamaegis — Fixed inverted scores and missing-FTS penalty in LinearCombinationReranker, correcting hybrid search ranking accuracy
  • @yuvalif — Added DataFusion Expr support for table row deletions in Rust SDK, enabling programmatic delete operations

A heartfelt thank you to our community contributors of Lance and LanceDB this past month:

@aayushbaluni • @adaworldapi • @alex766 • @ali2arslan • @ar-maan05 • @atakanyenel • @beinan • @chenghao-guo • @ddupg • @dentiny • @devteamaegis • @dhruvgarg111 • @fangbo • @farmerchillax • @geserdugarov • @gezi-lzq • @ghx5t-sol • @guillesd • @guinik • @haochengliu • @hfutatzhanghb • @hushengquan • @its-tanay • @ivscheianu • @jay-ju • @jerryjch • @jiaoew1991 • @jiengup • @jja725 • @johnchak • @kaan-simbe • @keepromise • @kushudai • @lennylxx • @leoreeyang • @lhoestq • @luciferyang • @majin1102 • @martji • @mesut-doner • @mikewhb • @myandpr • @n1teshy • @omkar-334 • @pengw0048 • @plotor • @pragnyanramtha • @qingfeng-occ • @ragnorc • @sezruby • @shenganzhang • @shiwk • @siddiqueahmad • @singhvishalkr • @sinianluoye • @snigenigmatic • @summaryzb • @touch-of-grey • @vip892766gma • @wangxiaobao1222 • @wending-y • @wojiaodoubao • @wombatu-kun • @xiaguanglei • @xodn348 • @xuqianjin-stars • @yanghua • @yuqi1129 • @yuvalif • @zelys-dfkh • @zhangyue19921010 • @zouhuajian

🤝 Lance Community Sync Recap

Community syncs this month covered the upcoming Lance 2.1.0 release, which will make the Lance 2.1 file format the default, along with indexing improvements and DataFusion enhancements. Early GPU acceleration work through a CuVS integration was also discussed. On the SDK side, Lance 6.0.1 shipped with bug fixes, and Lance 7.0.0 moved toward a release candidate with a community vote planned. The Lance DuckDB extension gained visibility with an official blog post published on the DuckDB blog.

The next Lance Community Sync will take place on Thursday, June 4 @ 9am PT.

• 📬 Subscribe to the Lance mailing list to receive the meeting invite →

• 📄 Add discussion topics to the meeting notes →

• 📺 Watch previous recordings →

ChanChan Mao
Developer Relations @ LanceDB
GitHub
Linkedin

Reproducible Data Curation In The Multimodal Lakehouse

Prashanth Rao
•
May 29, 2026
reproducible-data-curation-in-the-multimodal-lakehouse

Make your SQL Workflows Multimodal With LanceDB × DuckDB

Prashanth Rao
•
April 2, 2026
make-your-sql-workflows-multimodal-with-lancedb-x-duckdb

OpenSearch vs LanceDB for Vector Search: Query Cost and Infrastructure

Justin Miller
•
April 15, 2026
opensearch-vs-lancedb-for-vector-search-query-cost-and-infrastructure