Volcano Engine, ByteDance's cloud and AI service platform adopted Lance for multiple use cases. Over the past two years, its AI data infrastructure was rebuilt around Lance. It later used LanceDB as the memory backbone of ArkClaw, its managed cloud version of the OpenClaw agent framework, at 100,000+ queries per second in production.
TL;DR
- Model training pipeline: 7 days → 1 day
- GPU utilization rose to 95%
- Model training efficiency: 1.5x improvement
- Enterprise agent memory (ArkClaw): 100K+ QPS in production
- 27+ Lance features contributed back to open source
Who They Are
Volcano Engine is ByteDance's cloud and AI service platform, serving internal product teams and enterprise customers across China. Its data and AI infrastructure team is one of Lance's most active open source contributors, with 27+ features shipped back to the community, including distributed IVFPQ index training, binary copy compaction, and the Lance branching API that later became the foundation for GitforMemory.
Built for analytics. Broken by AI.
AI data pipelines have different demands from analytics pipelines. Parquet is designed for columnar batch scanning. It was not built for frequent schema changes, multimodal data types, random access, or vector search on the same table. Volcano Engine's enterprise customers were hitting all of these limits.
Index training was the most acute pain. IVFPQ index training had to run on a single node. As customer datasets grew, so did the hardware requirements. "When customers' tables got very large, single-node IVFPQ became a real bottleneck," said Zhang Yue, a software engineer on the Volcano Engine AI Data Lake Service (LAS) team. "We were buying nodes with 500GB+ of memory just to train the index. Training usually took several hours, sometimes days."
The memory layer problem arrived separately. The team building ArkClaw, Volcano Engine's managed cloud version of the OpenClaw agent framework, needed a production-grade memory backend. They evaluated two options: a Markdown-based, local-first backend with limited scalability, and open source LanceDB. LanceDB was the clear architectural winner. The question was how much to build on top of it for enterprise production.
Two problems. One answer
The data platform team and the ArkClaw team arrived at the same answer through separate evaluations.
The platform team needed a single format that could handle every data type (images, video, embeddings, structured metadata, point cloud) without forcing copies into a separate vector database or rewriting data for schema changes. Lance, the open lakehouse format for AI, has native multimodal support, zero-cost column addition, and random access performance that Parquet-based alternatives couldn't match.
The ArkClaw team ran a structured comparison across eight dimensions, evaluating the Markdown-based backend against open-source LanceDB. "For scale, for multimodal memory, for versioning, LanceDB has a better foundation," said Yang Hua, a software engineer on the Volcano Engine AI Data Lake Service (LAS) team. The Markdown-based alternative used local files with no version history; LanceDB's native multimodal format and branching API were decisive.
With the architectural decision made, the team built memory-lancedb-ultra on top of OpenClaw's standard memory-lancedb plugin to meet enterprise production requirements, contributing features back to open source as they went.
“In 2024, we were introducing Lance to customers. By 2025, our team was building on it ourselves. In 2026, Lance is the core of our agent cloud lake architecture.”
Ding Yuanpu, Founder of Lance Chinese Community at Volcano Engine
From data lake to agent memory, powered by LanceDB
The Volcano Engine AI Data Lake Service (LAS)
The platform is four layers deep, including Lance as the storage core, Spark/Daft/Ray for compute, a library of pre-built AI processing algorithms (PDF parsing, video ingestion, audio transcription, auto-labeling), and an agent/skill layer on top. Every data type lives in Lance tables and is queried directly by compute engines. No export step, no ETL pipeline between storage and compute.
Three capabilities were built and contributed back to the community:
Distributed IVFPQ
Volcano Engine replaced the single-node setup with a coordinator-based architecture. The coordinator handles sampling and calculates codebook and centroid data. It then distributes vector-to-centroid assignment across Lance's fragment units, each running as an independent parallel task. The 500GB+ single-node memory requirement is gone. Index training scales across available compute instead of being limited to what fits on one machine.
Binary Copy Compaction
Normal compaction deserializes all data, rebuilds it, and reserializes. It is CPU-intensive at scale. Binary copy compaction works directly on the binary buffers, skipping the decode/re-encode cycle entirely. On a representative workload with a complex schema and 5 million rows, compaction time dropped from 418.6 seconds to 15.3 seconds.
Lance Insight
Observability tooling built into the platform:
- Show: data distribution, manifest overview, fragment statistics
- Analyze: health checks covering small files, page size, index coverage, and fragmentation
- Optimize: intelligent diagnosis that recommends and executes compaction, cleanup, and index rebuilds to keep tables healthy
One Volcano Engine customer replaced an Argo-based data pipeline (CPU-only, no GPU support) with a Ray-based pipeline on Lance. Processing time fell from 7 days to 1 day. GPU utilization, previously chronically low, rose to 95%. Model training efficiency improved 1.5x.
A second customer replaced a Parquet setup that relied on cascading, filter-generated intermediate tables and lacked lineage and history. The new architecture uses Lance branches and zero-copy column addition. Every data state is now tracked through Lance's native versioning; historical queries no longer require full re-scans.
"Lance is the new generation of data storage for AI. That's why we built the entire Volcano Engine AI data platform on it."
Zhang Yue, Senior Software Engineer, Volcano Engine Team
memory-lancedb-ultra for ArkClaw
ArkClaw is Volcano Engine's managed cloud version of OpenClaw, the open-source AI agent framework. OpenClaw ships with memory-lancedb , plugin that uses LanceDB as its memory layer, with embedded deployment, filesystem-native storage, and multimodal retrieval making it a natural fit for long-term agent memory. Volcano Engine built memory-lancedb-ultra on top of that plugin, adding enterprise-grade capabilities and contributing improvements back upstream. ArkClaw has been running memory-lancedb-ultra since early this year.
The production layer added:
- Hybrid search: full-text plus vector search using Lance's native FTS alongside IVFPQ
- Embedding flexibility: Doubao support (ByteDance's own LLM family) plus an offline small-model option for air-gapped deployments
- Three-tier memory: session memory, daily summary, long-term storage
- AutoDream: a memory consolidation pipeline that periodically samples memory contents, identifies drift and contradictions, consolidates entries, and prunes low-value memories, modeled on how the brain consolidates memory during sleep.
GitforMemory is the most novel feature in the stack. Lance's branching API, built for data management, turned out to be exactly the right primitive for versioned agent memory.
“We used Lance’s native branching to build GitforMemory. Just as a developer branches code, an agent can branch its memory, explore a direction, and merge back.”
Yang Hua, Software Engineer, Volcano Engine Team
Users can snapshot memory state using Lance's CreateTag, roll back to any prior version, or launch parallel agent branches that each run in isolated memory. When one branch produces a useful result (a story direction that worked, a research thread worth keeping), its memory can be merged back to main or discarded. Support teams can isolate memory by context using branches, keeping sensitive operational data separate from public knowledge without losing access to either.
Faster training and production-scale agent memory
On the data platform, a 7-day model training pipeline dropped to 1 day, GPU utilization reached 95%, and training efficiency improved by 1.5x. No separate vector database infrastructure.
On ArkClaw, LanceDB is serving enterprise agent memory at 100,000+ QPS in production. "We needed something that could handle real 100,000+ QPS in production," said Yang Hua. "Open source LanceDB gave us the right foundation. We built the enterprise layer on top of it."
What’s Next
Volcano Engine is working on native agent-data patterns (Lance for Agent), large-scale Spark partition support for cross-data-lake joins, and BlobV2 for direct URL-addressable multimodal value management. The longer-term architecture, which they call CloudLake, is a three-layer system (memory lake, knowledge lake, multimodal data lake) built entirely on Lance and running on the customer's own object storage. Their view is that memory, knowledge, and training data are going to converge into a single data layer, and that's where they're building.
