I wrote this piece for Menlo Research, translating Ariel Hu and the Menlo Research team's work on training a humanoid to flip cartons into a version a general audience can follow. The full article is on Menlo's research blog. Short version and takeaways below. The research is the team's; the framing here is mine.
What it covers
The task: Asimov, a dual-arm humanoid, flipping cartons on a moving conveyor. Each carton has to rotate 90 degrees from standing on its edge to lying on its face while the belt keeps moving.
Training a vision policy end to end with reinforcement learning is slow, because you have to render camera images inside the training loop. The team split the problem in two. A teacher policy learns with PPO in simulation, with privileged access to exact object poses and contact states, information a real robot would never have. A student policy then learns by imitation, using DAgger, to copy the teacher's actions from RGB-D camera input and joint sensors. Only the student is meant to reach hardware.
A few choices make it practical. 16,384 simulated robots train in parallel on one GPU. A single teleoperated demonstration shapes exploration through an Adversarial Motion Prior. Rewards are staged milestones rather than dense shaping, which keeps the policy from gaming them. The result is a two-arm pinch-and-rotate strategy that completes the flip in simulation. Sim-to-real is the next step.
Why it matters
Teacher-student distillation is a recurring move in applied reinforcement learning: let the teacher use information that only exists in simulation, then force the student to solve the same task with only what a real robot can sense. The milestone reward is a concrete answer to reward hacking, and it is easier to reason about than a hand-tuned dense reward.
Read the complete piece on Menlo Research's research blog.