I wrote this piece for Menlo Research, translating the Kepler v0.1 paper by Ishneet Sukhvinder Singh, Dhanoosh Pooranakumaran, Alex Nguyen, and Jia Qi Yip into a version a general audience can follow. The full article is on Menlo's research blog. This is the short version and why I think it matters. The research is the authors'; the framing here is mine.
What it covers
Most robots treat the camera as the primary sense and everything else, joint angles, force, torque, as a secondary signal. That choice discards the information a camera cannot see: contact, slip, and weight. Kepler is a multimodal encoder that fuses vision, proprioception, and force/torque into a single 256-dimension representation.
The training idea is what makes it work. Each sense is trained to predict the others. One modality is hidden, the rest are fused, and the encoder has to reconstruct the missing one, a Joint Embedding Predictive Architecture objective. To do that well, the senses have to share a common latent space rather than sit in separate silos.
Two results stood out. A single encoder trained this way transfers across robot bodies: it matches robot-specific models on their own hardware and beats them on robots it has never seen. And because the model predicts each sense from the others, a large prediction error is a reliable sign that the robot is in an invalid state, so anomaly detection comes for free with no extra training.
Why it matters
The pattern generalises past robotics. When several noisy sources describe the same situation, training them to predict each other tends to use the signal better than naming one source primary and attaching the rest. The prediction error doubling as a health check is a property worth designing for on purpose, not treating as a side effect.
Read the complete piece on Menlo Research's research blog.