AI Components#

EmbodiedAgents, the intelligence layer of EMOS, is a set of components, each one a capability the robot can have: understanding speech, describing what the camera sees, detecting objects, remembering what it has seen, moving an arm. They are ordinary EMOS components, so they have the same lifecycle, inputs and outputs, health status and fallbacks as everything else in a recipe, and they combine freely into perception and action loops.

Most of them wrap a model, and where the model runs is a separate choice: a cloud API, a model server on the network, or a local model on the robot itself. Clients covers the first two and Models the local ones. One component stands apart: Cortex uses the others. It takes a goal in natural language and works out which capabilities to call, in what order, to reach it.

The components#

Component	What it does
LLM	Runs a large language model on text: reasoning, instruction following, dialogue and tool calling. It can draw on a vector database for context and falls back to a local model when its server is out of reach. It can remember the conversation, when `chat_history` is on, and has two actions for it: `reset_history` starts a new conversation and `get_history` returns the current one as text.
VLM	The same for images and text together, on a multimodal model. Beyond describing a scene and answering questions about it, it runs task models for pointing, grounding and affordance, and with a depth source it turns what it grounds into 3D boxes. Its `describe` and `run_task` actions let an event or Cortex ask a question or run a task on the current frame, and it can remember the conversation just like the LLM. Also known as MLLM.
Vision	Object detection and tracking on images, with classes, boxes and confidences, from a model server or a small on-board classifier. Given depth, from an RGBD camera, a depth image or a point cloud, it publishes metric 3D boxes in the frame you choose, which is what MoveIt and Memory consume.
VLA	Drives manipulation and control with a vision-language-action policy served by LeRobot’s policy server: SmolVLA, Pi0 and Pi0.5, GR00T, ACT, Diffusion and others. Camera and joint-state inputs go in, joint commands come out in the formats MoveIt Servo and ROS 2 Control expect, and a goal on its action server runs one task.
MoveIt	Motion planning and execution for an arm through a running MoveIt 2 `move_group`, as an action server with pose, joint, named, Cartesian, pick and place goals, and gripper control. Objects that Vision detects in 3D are placed in the planning scene as obstacles. Every one of its actions is a tool for Cortex.
SpeechToText	Turns spoken audio into text. It runs its own voice activity detection and a wakeword spotter with a phrase you choose, on audio captured on the robot, and transcribes with a model server or a local model.
TextToSpeech	Turns text into speech, played on the robot’s speakers or published as audio, from a model server or one of several local model families. Its `say` and `stop_playback` actions are made for events: announce a warning, stop talking when someone speaks.
Memory	A graph-backed spatio-temporal memory of what the robot has seen and felt, built on eMEM: detections, scene descriptions and internal state, each at a place and a time, with retrieval tools for the recipe and for Cortex. See Memory.
Cortex	The agent. It discovers every action, action server, service, routine and plugin action in the recipe, plans a sequence of calls for a natural-language goal, and runs it while watching the outputs. When allowed, it writes the action or condition a task needs and the recipe lacks. See Cortex.
SemanticRouter	Sends an input to one of several destinations by what it says, using an LLM, a decision model or a vector database of examples to decide, so one microphone or one text input can drive several pipelines.
DecisionComponent	Asks a decision model typed questions about what the robot hears or sees, or about what another component reports: a yes/no, a choice among options or a score on ordered levels, each answered in one forward pass with a probability per option and no generated text. Every question’s answers come out on a topic of their own, so an event can threshold them. The component’s `ask` action lets Cortex or an LLM put a question on demand. See Decision Models.
MotionDetector	Detects motion in an image stream or a point cloud stream and publishes a boolean, which makes it an event source, along with the frames of the episode as a video or the motion centres as poses. Given the robot’s odometry it ignores the robot’s own movement. It replaces the earlier VideoMessageMaker.