Physical AI systems generate an enormous amount of data.
Every vehicle, robot, and sensor-equipped machine is continuously producing camera frames, LiDAR, telemetry, control signals, trajectories, model outputs, annotations, and other state. At any meaningful fleet size, collecting data is not really the hard problem anymore.
The hard problem is finding the small fraction of that data that can make the model better.
A robot fails to grasp an object under unusual lighting. An autonomous vehicle brakes harder than expected at an intersection. A perception model behaves strangely around a particular combination of rain, pedestrians, and road geometry.
You may have already captured hundreds of similar examples. But if they are buried inside petabytes of fleet logs, they are not very useful.
This is why I think data mining is becoming one of the most important infrastructure problems in physical AI. The data system needs to turn raw fleet experience into searchable, reproducible datasets for training and evaluation.
The better your model gets, the harder the data problem becomes
Early in model development, almost any additional data can help.
That changes as the model improves.
Most of the fleet eventually represents situations the model already handles well. Driving straight on a clear road for the ten-millionth time probably has much less training value than the first examples did.
The useful examples move further into the long tail.
You start asking questions like:
- Show me every gripper slip where force variance increased immediately before failure.
- Find hard decelerations in Boston.
- Find left turns between 6 PM and 8 PM in this geofence where two or three pedestrians crossed from the left side of the vehicle.
- Find clips visually similar to this failure, even if nobody previously labeled them.
The difficulty is that the rarer the event becomes, the more data you have to search to find it. This is a slightly unusual infrastructure problem. A two-second query instead of a 200 ms query might not matter very much to the researcher. Being able to search 100 billion examples instead of 100 million matters enormously.
That distinction comes up repeatedly when we talk with robotics and autonomous-vehicle teams. The challenge is not just low-latency retrieval. It is making very large multimodal datasets searchable enough to mine the long tail. This is fundamentally a scale problem where supporting tens or hundreds of billions of rows matters much more than shaving small amounts of interactive latency.
Data mining is the center of the Physical AI data flywheel
A new failure happens in the field. You find similar examples across the historical fleet. You enrich and curate those examples into a dataset. You train or evaluate a new model. You deploy it. Then you mine the next generation of failures.
The faster this loop runs, the faster real-world experience turns into model quality.
But there is an important detail here. Mining does not simply mean writing SQL over a table.
Physical AI data starts as video, images, point clouds, ROS bags, MCAP files, trajectories, sensor streams, and other complex structures. The thing you want to search for often does not exist as a column yet. You have to create it.
A researcher might be mining grasp failures and realize the useful signal is not an existing label but a pattern in joint position, velocity, force, and torque over the two seconds before failure. An AV researcher might define a new metric from ego kinematics, object tracks, and planner outputs to capture a long-tail interaction that existing metrics miss. Or a team might run a video-language model over stored clips to generate scene descriptions, direct video embeddings, or model-error features, then use those outputs to retrieve similar failures. In practice, the features needed for mining are often discovered during mining itself, which makes feature engineering part of the loop.
What this workflow looks like today
In practice, a lot of Physical AI stacks evolved one workload at a time.
The raw data lands in object storage. ROS bags or MCAP files are decoded with Spark, Ray, Dataflow, or custom pipelines. Some subset is loaded into a visualization system such as Foxglove or Rerun. Metadata goes into a warehouse or analytical database. Embeddings go into a vector database or search engine. Training data gets rewritten again into WebDataset, LeRobot, or another training-optimized representation.
Then somebody builds an API that tries to make all of those systems look like one thing.
The problem is that you have to synchronize copies. You have to maintain lineage between the raw event, its derived features, the search index, and the eventual training example. Adding a feature can mean another large backfill and sometimes another physical representation of the dataset.
This gets particularly ugly with multimodal data because the feature computation itself can be substantial. A new column might require downloading an MP4, extracting a time window, decoding frames, converting them into tensors, running a PyTorch model, and writing an embedding back into the dataset.
Feature engineering changes what is searchable
This is the part of data mining that I think is sometimes underestimated.
Imagine that you want every left turn between 6 PM and 8 PM inside this geofence where two or three pedestrians cross from the left side of the vehicle.
Time is structured metadata. Location can be indexed geospatially. But “left turn” may need to be derived from trajectory data. “Two or three pedestrians” may require a perception model. “Crossing from the left side” may require temporal reasoning over a video clip.
The researcher cannot query these concepts until the system turns them into something computationally searchable. For example, leveraging yaw rate and speed, add a column is_left_turn to find left turns in the data set:
@geneva.udf(
data_type=pa.bool_(),
input_columns=[
"telemetry_time_s",
"yaw_rate_down_rps",
"steering_angle_deg",
"car_speed_mps",
],
num_cpus=1.0,
num_gpus=0.0,
version="left-turn-v1",
)
def detect_left_turn(
telemetry_time_s: list[float],
yaw_rate_down_rps: list[float],
steering_angle_deg: list[float],
car_speed_mps: list[float],
) -> bool:
import numpy as np
t = np.asarray(telemetry_time_s, dtype=np.float64)
yaw = np.asarray(yaw_rate_down_rps, dtype=np.float64)
steering = np.asarray(steering_angle_deg, dtype=np.float64)
speed = np.asarray(car_speed_mps, dtype=np.float64)
n = min(len(t), len(yaw), len(steering), len(speed))
if n < 2:
return False
t, yaw, steering, speed = t[:n], yaw[:n], steering[:n], speed[:n]
valid = np.isfinite(t) & np.isfinite(yaw) & np.isfinite(steering) & np.isfinite(speed)
t, yaw, steering, speed = t[valid], yaw[valid], steering[valid], speed[valid]
if len(t) < 2:
return False
moving = speed >= 2.0
candidate = moving & (yaw <= -0.08) # negative down-axis yaw is left
# Learn whether steering sign agrees or disagrees with yaw sign.
informative = moving & (np.abs(yaw) >= 0.03) & (np.abs(steering) >= 1.0)
if informative.sum() >= 20:
corr = np.corrcoef(yaw[informative], steering[informative])[0, 1]
if np.isfinite(corr) and abs(corr) >= 0.20:
steering_in_yaw_sign = steering * (1.0 if corr > 0 else -1.0)
candidate &= steering_in_yaw_sign <= -5.0
typical_dt = float(np.median(np.diff(t)))
max_gap = max(0.20, 2.5 * typical_dt)
run_start = None
previous_t = None
for timestamp, is_candidate in zip(t, candidate, strict=True):
if is_candidate and (previous_t is None or timestamp - previous_t <= max_gap):
run_start = timestamp if run_start is None else run_start
if timestamp - run_start >= 0.75:
return True
elif is_candidate:
run_start = timestamp
else:
run_start = None
previous_t = timestamp
return False
detect_left_turn
if "is_left_turn" not in table.schema.names:
table.add_columns({"is_left_turn": detect_left_turn})
with db.local_ray_context():
left_turn_job = table.backfill(
"is_left_turn",
concurrency=1,
task_size=1,
max_checkpoint_size=1,
)
table.checkout_latest()
table.search().select(["segment_id", "duration_s", "is_left_turn"]).to_pandas()And we can leverage a pretrained, COCO-trained Faster R-CNN MobileNet detector once per worker and reuse it across rows. COCO's `person` class can be used to find pedestrians.
@geneva.udf(
data_type=pa.int32(),
input_columns=["video_hevc"],
num_cpus=2.0,
num_gpus=0.0,
memory=2_000_000_000,
version="pedestrian-count-v3-resnet50",
)
class PedestrianCounter(Callable):
def __init__(
self,
score_threshold: float = 0.60,
frame_stride: int = 50,
max_frames: int = 12,
inference_batch_size: int = 2,
) -> None:
self.score_threshold = score_threshold
self.frame_stride = frame_stride
self.max_frames = max_frames
self.inference_batch_size = inference_batch_size
self._loaded = False
def setup(self) -> None:
import torch
from torchvision.models.detection import (
FasterRCNN_ResNet50_FPN_V2_Weights,
fasterrcnn_resnet50_fpn_v2,
)
weights = FasterRCNN_ResNet50_FPN_V2_Weights.DEFAULT
self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
self.model = fasterrcnn_resnet50_fpn_v2(weights=weights)
self.model.to(self.device).eval()
self._loaded = True
def _max_people_in_batch(self, images: list) -> int:
import torch
if not images:
return 0
with torch.inference_mode():
predictions = self.model([image.to(self.device) for image in images])
return max(
int(((pred["labels"] == 1) & (pred["scores"] >= self.score_threshold)).sum().item())
for pred in predictions
)
def __call__(self, video_hevc: bytes) -> int:
import io
import av
from torchvision.transforms.functional import pil_to_tensor
if not self._loaded:
self.setup()
scene_max = 0
batch = []
sampled = 0
# Auto-detect raw HEVC (comma2k19) or MP4 (JAAD).
with av.open(io.BytesIO(video_hevc)) as container:
for frame_index, frame in enumerate(container.decode(video=0)):
if frame_index % self.frame_stride != 0:
continue
tensor = pil_to_tensor(frame.to_image()).float().div_(255.0)
batch.append(tensor)
sampled += 1
if len(batch) == self.inference_batch_size:
scene_max = max(scene_max, self._max_people_in_batch(batch))
batch.clear()
if sampled >= self.max_frames:
break
if batch:
scene_max = max(scene_max, self._max_people_in_batch(batch))
return int(scene_max)
pedestrian_counter = PedestrianCounter()
pedestrian_counter
if "pedestrian_count" not in table.schema.names:
table.add_columns({"pedestrian_count": pedestrian_counter})
with db.local_ray_context():
pedestrian_job = table.backfill(
"pedestrian_count",
concurrency=1,
task_size=1,
max_checkpoint_size=1,
)
table.checkout_latest()Then we can search for scenes similar to a reference clip while filtering on the structured features we just created:
results = (
table.search(
example_clip_embedding,
vector_column_name="left_camera_embedding",
)
.where("""
city = 'boston'
AND hour BETWEEN 18 AND 20
AND is_left_turn = true
AND pedestrian_count BETWEEN 2 AND 3
""")
.limit(100)
.to_pandas()
)This is why treating feature engineering, data management, and retrieval as completely separate systems becomes increasingly awkward. Mining is iterative. Researchers discover that they need a new signal, compute it, inspect the results, change it, backfill history, and query again. Raw sensor logs become useful for mining only after they are transformed into discrete attributes, booleans, numerical values, or semantic embeddings.
A useful data system should keep the raw experience and the searchable representation together
This is the architectural idea behind Lance and LanceDB. Instead of treating the raw multimodal data, analytical representation, embeddings, and training dataset as unrelated copies, we want them to behave like different views of the same underlying dataset.
The raw media and sensor data stay addressable. New features and model outputs can be added over time. The same dataset can support structured filters, full-text search, vector search, and other retrieval patterns. Historical data can be backfilled when researchers define a new feature.
And importantly, the output of mining does not have to become another disconnected data silo. It can become a versioned dataset or materialized subset that is used directly for training and evaluation.
LanceDB's direction here is to combine the pieces required for this loop, with multimodal storage, iterative feature engineering, historical backfills, indexing, hybrid retrieval, dataset creation, and training access over the same underlying data. The original data-mining design specifically calls out feature enrichment, automatic backfilling, hybrid search across SQL/full-text/vector/geospatial signals, and mining directly against source data.
Mining is also how you build better evaluation sets
There is another benefit to this architecture. Every new failure mode gives you a candidate regression test. If a vehicle encounters a new scenario, the team can mine historically similar incidents and turn them into a versioned evaluation set. The next model should not only solve the new failure. It should continue to solve the previous ones.
Over time, the fleet itself becomes the source of increasingly difficult “golden” evaluation sets. This is especially useful in safety-critical systems. When something unusual occurs, engineers need to answer two questions quickly.
This is fundamentally a data retrieval and dataset-management problem. The original workflow similarly connects incident investigation with building regression sets from real-world failures.
The data advantage in Physical AI will come from using experience better
The companies that win in Physical AI will not just be the ones that collect the most data. They will be the ones that can turn fleet experience into better training data faster by finding rare failures, computing new signals over historical data, and turning those results into reproducible datasets for training and evaluation.
As models improve, the valuable examples become harder to find, and the data system becomes part of the model-development loop itself. Data mining is what connects what happened in the real world to what the model learns next, and the teams that can run that loop efficiently will improve faster.
