TL;DR
- CrewAI’s new 'cognitive memory' system that encodes selectively, resolves contradictions, and gates recall on confidence.
- CrewAI’s memory is built on LanceDB, consolidating agent memory into a lean, single-table architecture designed for multimodal data.
- By consolidating its previous two-system memory stack. CrewAI reduced dependencies, improved on-disk query performance, and delivered a smoother developer experience.
- LanceDB is now part of every CrewAI open-source install as the default vector backend.
- CrewAI runs at massive open-source scale, including 6M monthly package downloads and 2B+ billion agent executions, turning LanceDB into the memory layer behind one of the most widely adopted agent frameworks.
"I had absolutely zero hiccups setting up the entire thing."
- Joao Moura, CEO, CrewAI
Who Is CrewAI
CrewAI is where enterprise agents are built, owned, and run. In one environment, any builder takes an agent from idea to trusted production, and owns the result as real, exportable code, on any model and their own infrastructure, with no black box and no lock-in. It's why 63% of the Fortune 500 build on CrewAI, including PepsiCo, RBC, AB InBev, and the U.S. Department of Defense, alongside a global open-source community. Not starting from a blank page, every build draws on 700,000+ proven patterns from over 2 billion executions, with 6 million package downloads a month, so teams ship the agents that move the business, not another stalled pilot.
The Before: Two Systems, One Problem
CrewAI's memory stack ran on two pieces: a vector store for embeddings and a separate database for structured memory state. Long-term memory, short-term memory, entity memory, and embedding-backed tools all routed through this combination. It worked until the costs started compounding.
The two-system approach brought a chain of transitive dependencies and steady operational friction: compatibility issues, key management, and version tracking the team couldn't ignore indefinitely. Setup and performance friction showed up for users too, surfacing across Discord, Reddit, and GitHub.
The deeper issue was architectural. Memory was treated as a storage problem, with one simple job: save the embedding, retrieve by similarity, done. Agents accumulated contradictory facts with no mechanism to resolve them. The team described the failure mode: "Your agent learns one thing on Monday and another conflicting on Friday. Now it remembers both."
From LanceDB's perspective, this is where the traditional vector-store-plus-metadata-database pattern starts to break down. Agent memory is not just nearest-neighbor search. It also depends on scope, recency, importance, contradictions, and updates. Splitting embeddings and structured state across systems turns every recall and consolidation step into a sync problem, when it should be a single query over vectors and metadata together.
The Turn: From Storage To Cognition
CrewAI chose to rebuild memory from scratch, starting from a different premise that memory isn't a storage problem, it's a cognition problem. What to remember, how to resolve conflicts, and when to forget. Those decisions belong inside the memory system, not delegated to the application.
"We rebuilt memory from being focused on storage to actually being an agentic system on itself."
- Joao Moura, CEO, CrewAI
Building on that meant finding a single backend that could handle active encode and recall flows without reintroducing operational overhead. The team ran a survey of existing vector stores through Claude Opus to map the tradeoffs. LanceDB won on the merits that surfaced first:
- Native multimodal: LanceDB handles text, images, video, and audio in the same table; the previous vector database was limited to text and embeddings
- No server: LanceDB stores everything as files on disk; no process to manage
- Low dependency footprint: a single library install, unlike the transitive dependency chain their previous stack brought
- Strictly typed table schema: LanceDB's columns store scope keys, importance weights, timestamps, and vectors in the same row, B-tree indexed for fast metadata filtering. The previous stack had no equivalent.
"It felt more scalable, and the fact that it was disk-based was a big plus. It just felt like the right thing for an open-source framework that runs locally."
- Joao Moura, CEO, CrewAI
The implementation was simply a library install, a local directory path, and no new infrastructure to stand up.
The Solution: One Table, Two Flows, Zero Sync
The new system is what CrewAI calls cognitive memory. Built as a CrewAI Flow, it runs on the same agentic architecture it serves.
LanceDB is the multimodal lakehouse for AI, with one table for vectors, structured metadata, and multimodal data, with no server to run and no sync layer between systems. It's the default backend for all memory and embedding features in CrewAI OSS, installed automatically at ./.crewai/memory when a user enables memory. CrewAI removed the need for two separate systems for structured data and embeddings, along with the sync layer between them.
There are five cognitive operations: encode, consolidate, recall, extract, and forget.
Encode and consolidate
Every write goes through the encode flow first. The system analyzes the content, assigns a scope and category, and weights importance on a 0–1 scale before anything is written. When a new memory conflicts with a stored one above a 0.85 threshold, consolidation handles it: the old record is updated, its context preserved, and the stale fact removed.
When "We migrated to MySQL last week" arrives after "We use PostgreSQL" is already stored, the system detects the conflict, resolves it, and updates the record. That decision, which fact is current, which is stale, is exactly what gets pushed to the application layer when memory is split across multiple systems, or not made at all.
Recall
"It's not about retrieving, it's more about discovery."
- Joao Moura, CEO, CrewAI
The recall flow replaces static similarity lookup with dynamic, iterative retrieval. It scores every result set using a composite:
score = (similarity × w_sim) + (recency × w_rec) + (importance × w_imp)A critical architecture decision from six months ago outranks a trivial mention from yesterday. When confidence is insufficient, the system searches deeper before returning, filling evidence gaps rather than surfacing the nearest results.
Extract and forget
Extract decomposes large agent outputs into independent atomic facts before encoding. A 500-word infrastructure review becomes four discrete, separately-scoped memories, each with its own importance weight.
The system supports explicit memory removal by scope and age. An agent working across multiple projects can retire the memory of a deprecated one without touching anything else.
memory = Memory(storage="./.crewai/memory")
# encode → consolidate → write
memory.remember("PostgreSQL is our primary database")
# composite scoring, confidence-gated
memory.recall("what database are we using?")
# atomic decomposition before encode
memory.extract_memories(long_report)
# purposeful, scoped deletion
memory.forget(scope="/project/old", older_than=cutoff)One table, scoped with metadata
Rather than splitting memory across separate tables by type (one for long-term, one for short-term, one for entity memory), everything lives in a single LanceDB table. Scoping happens via SQL-style WHERE clause filtering. B-tree indexing on scope keys means filtered queries skip full table scans entirely. What had been a cross-system coordination problem became a WHERE clause.
The typed schema also matters for what agents do with retrieved memory. Fields like scope, importance, and timestamp are explicit typed columns. Agents get structured data they can act on directly, not unstructured blobs to parse.
Writes are non-blocking, where remember_many() submits to background threads and returns immediately. Reads drain pending writes first, so queries always see the latest state. Multiple memory instances pointing at the same database, which includes agent memory and crew memory running concurrently, use serialized operations with automatic conflict retry; the application doesn't manage locking.
For a deeper walkthrough of the implementation, including how the five cognitive operations compose, see CrewAI's technical writeup: How We Built Cognitive Memory for Agentic Systems.
The After: Simpler Stack, Faster Recall
Stack simplification
Two systems became one. LanceDB is now the only storage dependency for developers enabling memory in CrewAI OSS. Every developer who enables memory across 2B+ executions and 6 million monthly downloads gets it automatically, along with fast memory retrieval at ~200ms latency and no LLM call required. After years of managing that dependency overhead, the simplicity was the first concrete payoff.
What it unlocked
CrewAI is now building toward organizational memory, a graph of everything an organization's agents have learned across every run, queryable by scope and agent. They also shipped a terminal UI (crewai memory --storage-path ./my_memory) for browsing and auditing stored memories. The value showed up in the scoped memory inspection. Instead of scanning every stored memory across every agent and project, CrewAI could query a specific scope, such as a project, task, or agent context, using LanceDB metadata filters. What would have been a cross-system reconciliation step in the old stack became a filtered query over vectors and typed metadata in one table.
"With LanceDB, memory went from something that was a little fluffy to now something that actually works."
- Joao Moura, CEO, CrewAI
