At a glance
| Feature | LanceDB | Turbopuffer |
|---|---|---|
| What it is | Multimodal lakehouse for AI | Object-storage-native search engine |
| Scope | Curation, feature engineering, training and search on one table | First-stage retrieval |
| Images, video, audio | Raw bytes in binary columns; lazy-loading Blob API | bytes type up to 8 MiB, not indexed; multimodal is work-in-progress |
| Training data loading | PyTorch loading with row filters, column selection, deterministic shuffle and resumable state | Search/query APIs; equivalent training loader not documented |
| Feature engineering | Embedding functions on insert; Python UDFs with distributed backfills (Enterprise) | Native embedding |
| New embedding model | New column; raw media columns aren't rewritten | Re-ingest the data; vector columns fixed at creation |
| Transactions | ACID (Lance table format) | No general read-write transactions; Serializable conditional writes |
| Versioning | Versioned tables, time travel, tags and branches; retain training snapshots with tags | Copy-on-write branches |
| Data format | Open Lance format, read by Spark, Ray, Trino, DuckDB, Polars, PyTorch | Export via paged query API (JSON) |
| Query interface | Python, TypeScript, Rust SDKs; Java (Enterprise); SQL via DuckDB or FlightSQL (Enterprise) | JSON HTTP API and client libraries |
| Vector search | IVF family (PQ, RQ, SQ, FLAT, HNSW variants), managed automatically in Enterprise; dense, binary, multivector | SPFresh; dense, sparse, multivector |
| Full-text and hybrid | BM25; hybrid with RRF by default | BM25; multi-query with server-side RRF |
| Filtering | SQL filters on vector, full-text, hybrid or plain queries; LIKE, regex, nested fields; scalar indexes; pre- or post-filter | Inverted-index filters, regex, glob, fuzzy |
| Storage | OSS uses local disk or object storage. Enterprise uses object storage and an NVMe cache | Object storage + NVMe SSD and memory cache |
| Read consistency | Enterprise is strong by default. OSS offers configurable refresh | Strong by default (~10ms floor); eventual optional |
| Deployment | OSS runs as an embedded library. Enterprise offers Managed or BYOC on AWS, GCP and Azure | Multi-tenant SaaS; single-tenant and BYOC on Enterprise |
| Published search scale | Published benchmark over 10B vectors on 10 nodes; separate from training throughput | Reports 1T+ documents across its service; not a single-dataset training result |
| Pricing | OSS is open source (Apache 2.0). For Enterprise pricing, contact LanceDB | Usage-based; $16 / $256 / ≥$4,096 monthly minimums |
| Compliance | SOC 2 Type II, HIPAA, GDPR (Enterprise) | SOC 2 Type 2, GDPR DPA, HIPAA BAA |
Training-related claims reviewed November 8, 2026. “Not documented” means no equivalent was found in the public docs reviewed; it is not proof that a capability cannot be built. Enterprise-specific features are identified separately.
Why choose LanceDB for AI training workloads?
LanceDB is designed for the work that determines what a model learns: selecting examples, preparing features and delivering training batches. For vision, language and multimodal workloads, that means keeping source data and derived columns connected as experiments evolve.
Feed PyTorch from your training tables. StreamingDataset supports row filters, column selection, deterministic shuffling, prefetching and transforms in both LanceDB OSS and Enterprise. Its saved loader state can be restored alongside model and optimizer checkpoints so interrupted runs can resume.
Make each experiment traceable. Record the table version, selected columns, split or filter, and preprocessing configuration with the model checkpoint. Retain that version with a tag so later label or feature updates do not change the dataset used by the run. Dataset versioning supports reproducibility; it does not by itself make model training deterministic.
Turn model failures into better training sets. Combine metadata filters, vector search and full-text search to find similar examples, mine hard negatives and inspect underrepresented scenarios. Use the resulting selection to define the next training subset. This connection between curation and training is the main reason to choose LanceDB over a retrieval-focused service.
Sources: PyTorch data loading, checkpointing and resuming, table versioning, and data curation.
What is the difference between LanceDB and Turbopuffer?
LanceDB brings curation, feature engineering, retrieval and training access together around versioned multimodal tables. That can reduce the exports and synchronization work between data preparation and training. Turbopuffer focuses on first-stage retrieval, with native reranking available for the resulting candidates.
Both use object storage as the durable layer. Turbopuffer serves queries from NVMe SSD and memory caches, and LanceDB Enterprise serves them from a distributed NVMe cache. LanceDB OSS is an Apache 2.0 library that runs inside the application, and both LanceDB editions use the same open Lance format. Turbopuffer offers no open source version and no free tier.
Terminology
Turbopuffer stores documents with attributes in namespaces. A Turbopuffer namespace is an isolated container for documents and vectors with its own object storage prefix. The closest LanceDB terms are table (namespace), row (document) and column (attribute). LanceDB also uses the word namespace, for the catalog path that groups tables.
Can LanceDB and Turbopuffer store images, video and training data?
LanceDB keeps images, audio, video and PDFs alongside labels, vectors and metadata. Binary and blob columns let teams manage raw media and derived features in the same table, with on-demand access to large blobs. A training job can select cached features or tokenized fields without also reading the original media columns.
Turbopuffer's bytes type holds arbitrary binary data up to 8 MiB per value, with no indexing or filtering. Turbopuffer's docs say multi-modal support is a work-in-progress and that there is no dedicated type for image or video data.
Sources: LanceDB multimodal storage, Turbopuffer data types, and native embedding limitations.
How do feature engineering and embedding model changes compare?
LanceDB makes feature iteration a column-level operation. Add a new model's embeddings or precomputed features beside the existing columns without rewriting raw media. This is useful when repeatedly comparing feature sets or training on a frozen encoder's outputs. New features still require computation and storage, and must be refreshed when their inputs or generating model change.
LanceDB Enterprise feature engineering
LanceDB Enterprise adds managed computed columns and materialized views for feature preparation. Python Functions require a deployment with Functions enabled. LanceDB OSS also supports ordinary column additions and application-managed preprocessing; the managed Enterprise workflow is a separate capability.
- Enterprise refresh jobs populate computed columns on the cluster and commit completed batches as they run, so partial results can survive a failed job.
- Computed columns and materialized views can refresh on a schedule when automatic refresh is enabled for the deployment and the relevant object.
- Chunkers expand one row into many, for example splitting videos into clips or chunking documents.
- Materialized views store filtered or transformed results as tables that can be queried, indexed and used as prepared training datasets.
- Branches let teams run experiments and backfills without disturbing production reads on main.
Turbopuffer
Turbopuffer offers native embedding, which turns documents and queries into vectors on write and query, and the vectors can be exported like any other field. Changing the embedding model does not re-embed existing data, so the data in the namespaces has to be re-ingested.
- Vector attributes cannot currently be patched; the whole document has to be retrieved and upserted.
- A namespace has at most 8 vector columns, and the number is fixed when the namespace is created.
- Changing the type of an existing attribute is currently an error.
- Turbopuffer's roadmap lists easier embedding model migrations as a planned item.
Sources: LanceDB computed columns, materialized views, materialized features for VLM training, and Turbopuffer model changes.
How do transactions and versioning compare?
LanceDB's table versions give training teams a concrete record of the data used by an experiment. Writes create new versions that can be inspected, checked out or restored; tags retain important snapshots. Preserve the version and training configuration together. The Lance table format uses MVCC for table-level ACID transactions.
Turbopuffer branches are instant copy-on-write clones of a namespace that stay independent after creation. Turbopuffer does not support general-purpose read-write transactions. Its conditional writes behave as if run under Serializable isolation, and patch_by_filter and delete_by_filter behave as if run under Read Committed.
Can other tools read LanceDB and Turbopuffer data?
Lance tables can be read directly by other engines, while Turbopuffer data is read through its API. Lance tables used by LanceDB can be read by PyTorch, Ray, Spark, Trino, DuckDB and Polars. Lance is an open-source format with its own file, table and catalog specs. LanceDB tables can be queried with SQL, including joins and search, from DuckDB.
Turbopuffer is queried through a JSON HTTP API. Its documented way to export a namespace is to page through it with the query API. In a September 30, 2026 post, Turbopuffer said its v3 storage engine lays the foundation to move many more SQL queries to Turbopuffer, ahead of a production rollout.
How do search features compare?
LanceDB and Turbopuffer cover the same core search features, including vector search, BM25 full-text search, hybrid search fused with Reciprocal Rank Fusion, and reranking.
- Turbopuffer supports sparse vectors alongside dense f32, f16 and i8 vectors.
- Turbopuffer supports regex, glob and fuzzy filters on string attributes, and warns that regex is not recommended on large namespaces without other selective filters.
- LanceDB filters are SQL expressions run by Apache DataFusion, including LIKE pattern matching and regexp_match regular expressions, with pre- or post-filtering.
- LanceDB has scalar indexes (BTree, Bitmap, LabelList, FM) for filtering and substring search.
LanceDB full-text search takes AND, OR and NOT as boolean query objects, not as operators typed into the search string. Multivector search in LanceDB supports only the cosine metric.
How do LanceDB and Turbopuffer handle scale and latency?
For AI training, evaluate time to a usable dataset, samples per second, GPU idle time and recovery after interruptions. LanceDB's random access, projected column reads and training loader address these parts of the pipeline. Search latency alone does not measure how effectively a system supports training.
LanceDB Enterprise has published a search benchmark over 10 billion random vectors with 1,536 float32 dimensions. Across 10 nodes, top-100 queries reached 21 ms p99 latency with the reported IVF RaBitQ configuration. This demonstrates large-scale retrieval on LanceDB; it is not a training-throughput result.
Turbopuffer reports serving more than 1 trillion documents across its service. That aggregate figure is not a single-dataset training benchmark or a demonstrated limit for LanceDB. Both systems use caching for search; compare warm and cold behavior on the same data, hardware, recall target and query mix.
Published search benchmarks use different conditions and do not establish a head-to-head training winner. Measure the full training pipeline, including decoding, transforms and storage access, to assess throughput and cost.
Sources: LanceDB search benchmark, Turbopuffer service scale, and training pipeline performance.
What does LanceDB add beyond search?
Turbopuffer's managed retrieval can fit an application with a separate training data platform. LanceDB is compelling when your team wants the following capabilities connected to the data it searches:
- Training access to selected rows and columns through PyTorch-compatible data loading.
- Raw media, annotations, embeddings and feature values in the same table model.
- Versioned datasets that let experiments reference a retained snapshot.
- New feature columns without rewriting the existing media columns.
- An open Lance format and an Apache 2.0 OSS edition, with Enterprise services for managed operation.
Compare the documented scope: LanceDB multimodal lakehouse and Turbopuffer design tradeoffs.
Can you self-host LanceDB or Turbopuffer?
LanceDB OSS can be self-hosted as an embedded library, while Turbopuffer has no open-source version and offers BYOC on its Enterprise plan. LanceDB OSS runs in-process inside a binary, container or serverless function, and can keep tables on S3, GCS or Azure Blob Storage. LanceDB Enterprise can be installed into the customer's own cloud account on AWS, GCP or Azure.
How do pricing and compliance compare?
Turbopuffer bills $0.33 per GB-month of indexed storage and $2 per GB written on its Launch and Scale plans. Its monthly minimums are $16 (Launch), $256 (Scale) and at least $4,096 (Enterprise). LanceDB OSS is open source under Apache 2.0.
LanceDB Enterprise and Turbopuffer both cover SOC 2 Type II, HIPAA and GDPR. Turbopuffer lists its HIPAA BAA on the Scale and Enterprise plans. Turbopuffer encrypts data at rest with AES-256 and offers customer-managed keys on Enterprise, and LanceDB Enterprise encrypts data at rest in object storage and its cache. LanceDB can encrypt S3 data at rest with the customer's own AWS KMS key, and LanceDB Enterprise on Azure uses Azure Private Link.
Choose LanceDB when training is part of the workload
LanceDB: a data foundation for AI teams
- You train models on the same multimodal data you search, and want one table for raw media, labels, embeddings and features.
- You want to feed training jobs directly from the table, with shuffling, filtering and resumable loading.
- You add features or try new embedding models often, and don't want to rewrite your raw media to do it.
- You need to tie each model checkpoint back to an exact data version.
- You want the data in an open format that Spark, Ray, DuckDB and PyTorch can read, with an open-source edition you can run anywhere.
Turbopuffer: a focused search option
- Your requirement is managed retrieval, and your training datasets and feature pipelines are handled elsewhere.
- You need search across many isolated tenant namespaces and prefer a commercial search service.
- Your retrieval relies on sparse vectors.
Frequently asked questions
Is Turbopuffer open source?
No. Turbopuffer states that it does not offer an open source version or a free tier. LanceDB OSS is open source under the Apache License 2.0.
Is there an open-source alternative to Turbopuffer?
Yes. LanceDB OSS is an Apache 2.0 embedded library with vector, full-text and hybrid search, plus a path into multimodal training workflows. It can use local storage or supported object stores. LanceDB Enterprise adds managed services and distributed infrastructure beyond the OSS library.
Can Turbopuffer store images or video?
Turbopuffer has a bytes type for binary data up to 8 MiB per value, which cannot be indexed or filtered. Its docs say multi-modal support is a work-in-progress, with no dedicated type for image or video data. LanceDB stores raw images, audio and video in binary columns next to their vectors.
Can I use LanceDB as a PyTorch data loader?
Yes. LanceDB provides PyTorch-compatible access to tables and a StreamingDataset for filtered, shuffled and prefetched reads. Save its loader state with your training checkpoint to resume loading. Your pipeline still handles model-specific decoding, transforms and batch collation.
What happens when I change embedding models in Turbopuffer?
Turbopuffer's docs say existing data is not re-embedded, so the data in the namespaces has to be re-ingested. Easier embedding model migrations are on Turbopuffer's roadmap. In LanceDB, a new model's embeddings go in a new column without rewriting the raw media.
Does Turbopuffer support SQL?
Turbopuffer is queried through a JSON HTTP API. Turbopuffer said in September 2026 that its v3 storage engine lays the foundation to move many more SQL queries to Turbopuffer. LanceDB tables can be queried with SQL from DuckDB, and LanceDB Enterprise has a FlightSQL endpoint.
Does Turbopuffer support transactions?
Turbopuffer provides atomicity, consistency and durability, but not general-purpose read-write transactions. Its conditional writes behave as if run under Serializable isolation. The Lance table format behind LanceDB supports ACID transactions.
Which scales further, LanceDB or Turbopuffer?
The published numbers do not establish a winner: Turbopuffer's service-wide document count and LanceDB's dataset and search benchmarks measure different things. For training workloads, LanceDB's more relevant advantages are versioned data, feature evolution and direct training access. Benchmark loading throughput and end-to-end training on your own data.
Build your training data layer on LanceDB
Bring your dataset and training workload. Explore PyTorch loading with LanceDB OSS, or talk to us about Enterprise for multimodal curation, feature engineering and training data at scale.
