About This Talk
A robot learning to make coffee and a computer-use agent learning to build a pivot table face the same problem: a video shows what happened, but it doesn’t come with a reliable record of each action and when it occurred.
Figure reported that pre-training on human behavior raised a humanoid’s zero-shot success in 30 unseen homes from 9% to 56%. VideoAgentTrek turned 39,000 screen-recorded tutorials into 1.5 million training steps for computer-use agents. In both settings, turning observation into training data requires inferring actions and aligning them with the video.
Get that timing wrong, and a training example can pair an action with frames that already show its result. That gives the model information it won’t have when choosing what to do next.
Drawing on patterns from my book, I’ll walk through two practical decisions: how to store inferred actions alongside their timestamps, confidence scores, and labeler versions, and how to measure the delay between an action and its visible effect before training. Attendees will leave with a concrete approach to checking temporal alignment before these errors become part of what a model learns.












