EMOS
Planning & Manipulation Overview#
Bridge the gap between high-level reasoning and physical actuation. These recipes cover Cortex – the EMOS planner-executor that decomposes goals into ordered tool calls, dispatches them, and replans on failure, all on top of capability components you wrote in isolation – alongside VLM-based planning and VLA-based end-to-end manipulation for direct motor control.
A component that sits on top of the rest of your recipe and turns it into a self-directing agent. Cortex auto-discovers every component’s capabilities and uses them to achieve a high-level goal.
Cortex paired with graph-backed spatio-temporal memory. It recalls past observations, reasons about internal state via interoception, and wraps action tasks in episodes that consolidate into long-term memory.
The full stack. Cortex orchestrates Vision, VLM, Memory, the Kompass navigation stack, and TTS end-to-end. Compound goals fulfilled by a single agent – no behavior trees, no state machines.
An agent that extends itself. Give Cortex a standing instruction the recipe has no tool for, such as “email me whenever you see a person”, and the planner writes the missing action in Python, has it checked against the action contract, and wires it into an event, all while the robot keeps running.
Navigation guided by sight, not maps. A planning VLM grounds free-form descriptions like “the yellow chair” in the live camera frame and projects them into goal points the navigation stack acts on.
End-to-end neural manipulation. Use one of the latest VLA foundation models – SmolVLA, Pi0 and Pi0.5, GR00T, ACT, Diffusion or any other policy LeRobot serves – and go straight from camera frames + text to joint commands.
Closed-loop manipulation from an open-loop policy. A VLM watches the camera during execution and stops the VLA the moment it sees the task complete – or sees it going wrong.
Classical, collision-aware manipulation through MoveIt 2 as one component: pose, joint, named and Cartesian goals, pick and place, a gripper, and a planning scene filled with the objects the camera sees in 3D. Every bit of it a tool for Cortex.