Brain waves may be the next frontier for training physical AI, Encord and Zander Labs experiment

In a warehouse in San Leandro, California, a small but growing number of startups are betting that the next big unlock for physical AI will not come from better model architecture, but from solving a more fundamental problem: the scarcity of real-world training data. Encord, a company that builds data tooling for AI models, is running a trial with Zander Labs, a German neuroscience startup, to see whether measuring brain activity during physical tasks can create a richer, more useful dataset for training robots.

The data bottleneck in physical AI

While large language models (LLMs) were built on the text of the entire internet, finding equivalent raw materials to teach neural networks about physical manipulation is far more challenging. Self-driving car companies collect their own data, but that approach is hard to scale. Training from video can work, but it lacks the fidelity of real-world data. Vineeth Velmurugan, Encord’s head of robot learning and a veteran of OpenAI’s robot lab and Berkshire Grey, estimates that breaking through will require a dataset roughly five times the size of YouTube’s video corpus. That scale helps explain why data-generation itself has become a business, not just a research problem.

How brain waves could help

The headset worn by Encord’s pilot, Andrew Ceja, includes sensors that measure his brain waves as he carefully disassembles a Jenga tower. Lucas Gehrke, a Zander neuroscientist supervising the work, says that the amount of brain activity used at any point during a given task offers clues for model builders trying to figure out when they need to deploy their highest-effort models. The goal is to build an initial brain-wave-tagged dataset, run it through customer robotics models, and evaluate whether it actually improves performance before deciding whether to scale up.

Beyond brain waves: other new data modalities

Encord is also experimenting with sensors strapped to the forearm to detect electrical signals in muscles. Video of human hands manipulating objects typically does not capture the entire hand, but Velmurugan hopes to build a 3D depiction of hand position based on arm sensors, creating a more robust understanding for models. The company’s datasets are annotated with physical descriptions like ‘right hand tightens bolt’ to aid LLM-based models in understanding what is happening. Velmurugan estimates this kind of dense annotation is worth 100 times as much as ‘junky ego data’ for training specific tasks, though it costs about 20 times more to produce.

Why this matters for the robotics industry

The bet that generative AI can do for robots what it has done for chatbots keeps running into the same wall: physical training data has to be manufactured, not just collected. That changes the economics of building these models. Encord’s vantage point — sitting between many robotics companies at once — allows it to spot which data techniques are gaining traction industry-wide before any single customer can. As humanoid and warehouse robotics companies race to solve manipulation tasks, the ability to generate high-quality training data efficiently may become a key competitive advantage.

Conclusion

Encord’s trial with Zander Labs represents the ‘bleeding edge’ of efforts to solve the robotics data bottleneck, according to Velmurugan. While still experimental, the approach of using brain activity to tag physical training data could provide model builders with valuable signals about when to deploy high-effort models. For now, the work continues in a San Leandro warehouse, where pilots like Ceja and Sofia Infante are building the building blocks for neural networks — one Jenga block at a time.

FAQs

Q1: What is Encord testing with brain-wave headsets?
Encord is running a trial with Zander Labs to see if measuring brain activity during physical tasks — like pulling blocks from a Jenga tower — can create richer datasets for training AI models. The brain activity data is used to deduce mental states like error, intent, and surprise.

Q2: Why is physical AI training data so scarce?
Unlike LLMs, which were trained on text from the entire internet, physical AI requires real-world data about manipulation and movement. This data is expensive and difficult to collect at scale. Self-driving car companies collect it themselves, but that approach is hard to replicate for general robotics.

Q3: What other methods is Encord using to generate training data?
Encord uses egocentric video collected by workers wearing cameras, leader-follower robotic rigs for tasks like pouring coffee, and forearm sensors that detect electrical signals in muscles. All data is densely annotated with physical descriptions to aid model understanding.