LanceDB depends on open source projects across our stack, from storage access to query execution. We benefit every day from the engineering, review, and maintenance behind those projects. That lets us spend more time on the problems that are specific to LanceDB.
When we run into a missing capability or find a useful optimization, we try to contribute it upstream rather than keep it inside LanceDB. Our engineers have contributed to LeRobot, Apache OpenDAL, CocoIndex, ICU4X, and other projects we depend on. Nearly 90% of LanceDB's members on the Lance PMC also contribute to or help govern other major open-source projects.
A recent example is adding ASOF join support to Apache DataFusion, the query engine behind SQL execution in LanceDB. We needed a better way to match records captured at different timestamps, a common pattern in robotics and other time-series workloads. Contributing the feature upstream makes it available to other DataFusion users too, with the benefit of review from the wider community.
Why DataFusion matters to LanceDB
Apache DataFusion is an extensible query engine written in Rust and built around Apache Arrow. It provides SQL and DataFrame APIs, query planning and optimization, and a columnar execution engine. Developers can extend it with their own data sources, functions, and operators.
For LanceDB, that means we do not have to build a SQL query engine from scratch. We can focus on storage, indexing, and the access patterns our users need, while DataFusion handles a large part of query planning and execution.
That also gives us a natural way to contribute back. When a capability we need could be useful beyond LanceDB, we would rather improve the shared engine than keep the solution private. ASOF joins are one example.
When timestamps do not line up
Imagine a robotics dataset where camera frames and robot states are recorded at different frequencies. For each frame, you may want the most recent state recorded at or before the frame timestamp.
An equality join only works when the timestamps match exactly. Joining against every earlier state creates the opposite problem, leaving multiple candidates for each frame. What we actually want is the latest eligible state for the same robot and recording episode. Differently sampled sensor data is one area where DataFusion’s ASOF feature helps.
An ASOF join lets you write that relationship directly. A DataFusion query over the camera and state tables could look like this.
SELECT
f.robot_id,
f.episode_id,
f.frame_id,
f.ts AS frame_time,
s.ts AS state_time,
s.joint_positions
FROM camera_frames AS f
ASOF JOIN robot_states AS s
MATCH_CONDITION (f.ts >= s.ts)
ON f.robot_id = s.robot_id
AND f.episode_id = s.episode_id;For each frame, this finds the latest state at or before the frame timestamp for the same robot and episode. Frames with no matching state stay in the result with null state fields. Other comparison operators can be used to pick the next eligible record instead.
This is useful beyond robotics too. In financial data, for example, an ASOF join can match a trade to the most recent quote. In both cases, records need to be matched by time even when their timestamps do not line up exactly.
Why we contribute upstream
Contributing upstream means a feature built for one LanceDB use case can become useful to teams we may never meet. It also puts the implementation in front of maintainers and users with different workloads and assumptions.
That exchange improves our work, too. Review from the DataFusion community brings in perspectives we would not get from LanceDB’s requirements alone.
We depend on open source, so contributing back is part of how we want to build LanceDB. DataFusion is a good example of that relationship, and we are glad to contribute when the problems we solve can help the broader community.
You can read the ASOF join contribution or explore the Apache DataFusion documentation for more detail.
