# EMOS Documentation -- Context for AI Agents You are an expert EMOS recipe developer. EMOS (The Embodied Operating System) is a unified orchestration layer for Physical AI that combines EmbodiedAgents (intelligence) and Kompass (navigation) into a single framework. ## How to Write an EMOS Recipe An EMOS Recipe is a pure Python script that defines a robot behavior. When writing recipes, follow these principles: 1. **Define Topics** -- Declare ROS2 topics as `Topic(name=..., msg_type=...)` for inter-component communication. Match `msg_type` to your data (String, Image, Audio, Detections, etc.). 2. **Configure Clients & Models** -- Create a model client (OllamaClient, GenericHTTPClient, LeRobotClient, etc.) with a model wrapper. Clients are interchangeable -- swap inference backends without changing component logic. 3. **Build Components** -- Instantiate components (LLM, VLM, VLA, SpeechToText, TextToSpeech, Vision, Memory, Cortex, SemanticRouter) with inputs, outputs, and a model_client. Set `trigger` to control when the component executes. Use `Memory` for spatio-temporal memory and `Cortex` as an agentic harness that auto-discovers the rest of the graph as LLM tools. 4. **Wire Navigation** -- For mobile robots, configure a `RobotConfig` and instantiate Kompass components (Planner, Controller, DriveManager) with appropriate algorithms (DWA, PurePursuit, etc.). 5. **Add Events & Fallbacks** -- Use `on_fail()` per component for restart-style recovery, `launcher.on_process_fail()` for process-level crash recovery, and custom event/action pairs for runtime adaptivity. Events can trigger model swaps, component restarts, or arbitrary callbacks. 6. **Launch** -- Use `Launcher()` to add component packages with `launcher.add_pkg(components=[...], multiprocessing=True)` and call `bringup()`. The `multiprocessing` flag goes on `add_pkg`, not on the Launcher constructor. The documentation below is ordered as a curriculum: architecture first, then components and APIs, then example recipes of increasing complexity. --- ## File: overview.md ```markdown # EMOS -- The Embodied Operating System **The open-source unified orchestration layer for Physical AI.** EMOS transforms robots into Physical AI Agents. It provides a hardware-agnostic runtime that lets robots **see**, **think**, **move**, and **adapt** -- all orchestrated from pure Python scripts called Recipes. Write a _Recipe_ once, deploy it on any robot -- from wheeled AMRs to humanoids -- without rewriting code. :::{image} _static/images/diagrams/emos_robot_stack_light.png :align: center :width: 70% :class: light-only ::: :::{image} _static/images/diagrams/emos_robot_stack_dark.png :align: center :width: 70% :class: dark-only :::

Get Started • Why EMOS? • View on GitHub

--- ## What You Can Build ::::{grid} 1 2 2 2 :gutter: 3 :::{grid-item-card} {material-regular}`psychology;1.2em;sd-text-primary` Intelligent Agents Wire together vision, language, speech, and memory components into **agentic workflows**. Route queries by intent, answer questions about the environment, or build a semantic map -- all from a single Python script. [See cognition recipes](recipes/foundation/index) {material-regular}`arrow_forward;0.9em` ::: :::{grid-item-card} {material-regular}`route;1.2em;sd-text-primary` Autonomous Navigation **GPU-accelerated** planning and control for real-world mobility. Point-to-point navigation, path recording, and vision-based target following -- across differential drive, Ackermann, and omnidirectional platforms. [See navigation recipes](recipes/navigation/index) {material-regular}`arrow_forward;0.9em` ::: :::{grid-item-card} {material-regular}`sync_alt;1.2em;sd-text-primary` Runtime Adaptivity **Event-driven** architecture lets agents reconfigure themselves at runtime. Hot-swap ML models on network failure, switch navigation algorithms when stuck, trigger recovery maneuvers from sensor events, or compose complex behaviors with logic gates. [See adaptivity recipes](recipes/events-and-resilience/index) {material-regular}`arrow_forward;0.9em` ::: :::{grid-item-card} {material-regular}`precision_manufacturing;1.2em;sd-text-primary` Planning & Manipulation Use **VLMs** for high-level task decomposition and **VLAs** for end-to-end manipulation. Closed-loop control where a VLM referee stops actions on visual task completion. [See manipulation recipes](recipes/planning-and-manipulation/index) {material-regular}`arrow_forward;0.9em` ::: :::: --- ## What's Inside EMOS is built on three open-source components: | Component | Role | | :--- | :--- | | **[EmbodiedAgents](https://github.com/automatika-robotics/embodied-agents)** | Intelligence layer -- agentic graphs of ML models with semantic memory and event-driven reconfiguration | | **[Kompass](https://github.com/automatika-robotics/kompass)** | Navigation layer -- GPU-powered planning and control for real-world mobility | | **[Sugarcoat](https://github.com/automatika-robotics/sugarcoat)** | Architecture layer -- event-driven system primitives and imperative launch API | --- ::::{grid} 1 2 3 3 :gutter: 3 :::{grid-item-card} {material-regular}`lightbulb;1.2em;sd-text-primary` Why EMOS :link: why-emos :link-type: doc The problem EMOS solves -- from custom R&D projects to universal, adaptive robot apps. ::: :::{grid-item-card} {material-regular}`rocket_launch;1.2em;sd-text-primary` Getting Started :link: getting-started/installation :link-type: doc Install EMOS and run your first Recipe in minutes. ::: :::{grid-item-card} {material-regular}`menu_book;1.2em;sd-text-primary` Recipes & Tutorials :link: recipes/overview :link-type: doc Build intelligent robot behaviors with step-by-step guides. ::: :::{grid-item-card} {material-regular}`architecture;1.2em;sd-text-primary` Core Concepts :link: concepts/architecture :link-type: doc Understand the architecture, components, events, and fallbacks. ::: :::{grid-item-card} {material-regular}`dashboard;1.2em;sd-text-primary` Dashboard :link: getting-started/dashboard :link-type: doc Zero-touch web console on the robot for browsing and running recipes ::: :::{grid-item-card} {material-regular}`terminal;1.2em;sd-text-primary` CLI & Deployment :link: getting-started/cli :link-type: doc Package and run Recipes with the `emos` CLI. ::: :::{grid-item-card} {material-regular}`smart_toy;1.2em;sd-text-primary` AI-Assisted Coding :link: llms.txt Get the `llms.txt` for your coding agent and let it write recipes for you. ::: :::: ``` ## File: why-emos.md ```markdown # Why EMOS The robotics industry is undergoing a structural shift. Robots are transitioning from **single-purpose tools** -- hard-coded for fixed tasks -- to **general-purpose platforms** that must perform different jobs in different environments. While the AI industry races to build foundation models, a critical vacuum remains in the infrastructure required to actually ground these models on robots usable in the field. EMOS fills that vacuum. It is the missing orchestration layer between capable hardware and capable AI. --- ## The Problem Modern robot hardware ships with stable locomotion controllers and basic SDKs, but little else. Getting a robot to actually *do something useful* -- navigate a cluttered warehouse, respond to voice commands, recover from failures -- requires stitching together a fragile patchwork of ROS packages, custom launch files, and one-off scripts. Every new deployment becomes a bespoke R&D project. This approach has three fatal flaws: 1. **It doesn't scale.** Every new robot, environment, or task requires months of custom engineering. 2. **It doesn't adapt.** Rigid state machines and declarative graphs cannot handle the chaos of the real world -- sensor failures, dynamic obstacles, network drops. 3. **It doesn't transfer.** Software written for one robot rarely works on another, even if the task is identical. --- ## What EMOS Changes ### From Custom Projects to Universal Recipes EMOS replaces brittle, robot-specific software projects with **Recipes**: reusable, hardware-agnostic application packages written in pure Python. A Recipe is a complete agentic workflow -- perception, reasoning, navigation, memory, and interaction -- defined in a single script and launched with one command. - {material-regular}`smart_toy;1.2em;sd-text-primary` **One Robot, Many Tasks:** The same robot can run different Recipes for different jobs -- inspection in the morning, delivery at noon, security patrol at night. - {material-regular}`devices;1.2em;sd-text-primary` **One Recipe, Many Robots:** A Recipe written for a wheeled AMR runs identically on a quadruped. EMOS handles the kinematic translation beneath the surface. ### From Rigid Graphs to Adaptive Agents Legacy stacks treat failure as a system crash. EMOS treats it as a **control flow state**. Its event-driven architecture lets robots reconfigure themselves at runtime: - {material-regular}`sync;1.2em;sd-text-primary` Hot-swap ML models when the network drops - {material-regular}`swap_horiz;1.2em;sd-text-primary` Switch navigation algorithms when the robot gets stuck - {material-regular}`flash_on;1.2em;sd-text-primary` Trigger recovery maneuvers based on sensor events - {material-regular}`hub;1.2em;sd-text-primary` Compose complex behaviors with logic gates (AND, OR, NOT) across multiple data streams This isn't bolted-on error handling -- adaptivity is a **first-class primitive** in the system design. ### From Stateless Tools to Embodied Agents Current robots have logs, not memory. They record data for post-facto analysis but cannot recall it at runtime. EMOS introduces **embodiment primitives** that give robots a sense of self and history: - {material-regular}`map;1.2em;sd-text-primary` **Spatio-Temporal Semantic Memory:** A queryable world-state backed by vector databases that persists across tasks. - {material-regular}`self_improvement;1.2em;sd-text-primary` **Self-Referential State:** Components can inspect and modify each other's configuration, enabling system-level awareness rather than isolated self-repair. ### From CPU Bottlenecks to GPU-Accelerated Navigation While other stacks use GPUs only for vision, EMOS moves the entire navigation control stack to the GPU. Kompass, the EMOS navigation engine, provides **GPGPU-accelerated kernels** for motion planning and control: - {material-regular}`speed;1.2em;sd-text-primary` **Up to 3,106x speedup** over CPU-bound stacks for trajectory evaluation - {material-regular}`grid_on;1.2em;sd-text-primary` **1,850x speedup** for dense occupancy grid mapping - {material-regular}`memory;1.2em;sd-text-primary` **Vendor-neutral** -- works on NVIDIA, AMD, Intel, and integrated GPUs via SYCL - {material-regular}`developer_board;1.2em;sd-text-primary` Falls back to optimized process-level parallelism on CPU-only platforms This enables reactive autonomy in dynamic, unstructured environments where traditional CPU-bound stacks like Nav2 simply cannot keep up. ### From Separate Backends to Auto-Generated Interaction In traditional robotics, the automation logic is "backend" and the user interface is a separate custom project. EMOS treats the **Recipe as the single source of truth** -- defining the logic automatically generates a bespoke Web UI for real-time monitoring, configuration, and control. No separate frontend development required. --- ## The Architecture EMOS is built on three open-source components that work in tandem: :::{image} _static/images/diagrams/emos_diagram_light.png :align: center :width: 50% :class: light-only ::: :::{image} _static/images/diagrams/emos_diagram_dark.png :align: center :width: 50% :class: dark-only ::: | Component | Layer | What It Does | |:---|:---|:---| | [**EmbodiedAgents**](https://github.com/automatika-robotics/embodied-agents) | Intelligence | Agentic graphs of ML models with semantic memory, information routing, and adaptive reconfiguration | | [**Kompass**](https://github.com/automatika-robotics/kompass) | Navigation | GPU-powered planning and control for real-world mobility across all motion models | | [**Sugarcoat**](https://github.com/automatika-robotics/sugarcoat) | Architecture | Event-driven system primitives, lifecycle management, and the imperative launch API that underpins both layers | Together, they provide a complete runtime: from raw sensor data to intelligent action, with adaptivity and resilience built in at every level. --- ## Who Is EMOS For ### 1. Robot Managers & End-Users Use pre-built Recipes or write your own with the high-level Python API. Focus on your business logic -- EMOS handles the robotics complexity. ### 2. Integrators & Solution Providers EMOS is your SDK for the physical world. Connect robot events to ERPs, building management systems, or fleet software using the event-action architecture. Spend your time on enterprise integration, not low-level robotics plumbing. ### 3. OEM Teams Write a single Hardware Abstraction Layer plugin and instantly unlock the entire EMOS ecosystem for your chassis. Every Recipe written by any developer runs on your hardware without custom code. --- ## EMOS is Built for the Real World EMOS is not a research prototype. It is shaped by the demands of production deployments -- autonomous inspection patrols, security operations, and field robotics on quadruped and wheeled platforms. Every feature in the stack exists because a real-world deployment needed it. --- ## Get Started ::::{grid} 1 2 2 2 :gutter: 3 :::{grid-item-card} {material-regular}`rocket_launch;1.2em;sd-text-primary` Install EMOS :link: getting-started/installation :link-type: doc Get up and running in minutes. ::: :::{grid-item-card} {material-regular}`menu_book;1.2em;sd-text-primary` Browse Recipes :link: recipes/overview :link-type: doc Step-by-step tutorials from simple to production-grade. ::: :::: ``` ## File: getting-started/installation.md ```markdown # Installation ## EMOS CLI The fastest way to get started with EMOS is through the CLI. Download the latest release: ```bash curl -sSL https://raw.githubusercontent.com/automatika-robotics/emos/main/stack/emos-cli/scripts/install.sh | sudo bash ``` Or build from source (requires Go 1.25+): ```bash git clone https://github.com/automatika-robotics/emos.git cd emos/stack/emos-cli make build sudo make install ``` ```{tip} The CLI is a single static binary with no runtime dependencies, copy `/usr/local/bin/emos` to another machine on the same architecture and it just works. ``` ## Deployment Modes EMOS supports four deployment modes. Run `emos install` without arguments for an interactive menu, or use the `--mode` flag directly. ::::{tab-set} :::{tab-item} Container No ROS2 installation required. Runs EMOS inside a Docker container using the public image. ```bash emos install --mode container ``` You will be prompted to select a ROS2 distribution (Jazzy, Humble, or Kilted). The CLI pulls the image, creates the container, and sets up the `~/emos/` directory structure. **Requirements:** Docker installed and running. ::: :::{tab-item} Native Builds EMOS packages from source and installs them directly into your ROS2 installation at `/opt/ros/{distro}/`. No container needed. ```bash emos install --mode native ``` The CLI will: 1. Detect your ROS2 installation 2. Clone the EMOS source and dependencies into a build workspace (`~/emos/ros_ws/`) 3. Install system packages (portaudio, GeographicLib, rmw-zenoh) 4. Install Python dependencies 5. Install kompass-core with GPU acceleration support 6. Build all packages with colcon and install them into `/opt/ros/{distro}/` After installation, EMOS packages are available whenever you source `/opt/ros//setup.bash`. See [Running Recipes](running-recipes.md) for how to launch a recipe -- directly with `python` or via the `emos run` flow. **Requirements:** A working ROS2 installation (Humble, Jazzy, or Kilted). ::: :::{tab-item} pixi ```{note} Currently pinned to **ROS 2 Jazzy**. ``` Installs ROS2 and all EMOS dependencies into an isolated userspace environment using [pixi](https://pixi.sh). No root privileges, no Docker, no pre-installed ROS2 required. Works on any Linux distribution. ```bash # Install pixi first (emos install --mode pixi tells you if it's missing) curl -fsSL https://pixi.sh/install.sh | bash # Install EMOS in pixi mode emos install --mode pixi ``` The CLI clones the EMOS workspace into `~/.local/share/emos`, pulls ROS 2 Jazzy and all dependencies as pre-built packages from [RoboStack](https://robostack.github.io/) and conda-forge, installs kompass-core with GPU acceleration, then builds the EMOS packages with colcon — independent of any system ROS 2. See [Running Recipes](running-recipes.md) for how to launch a recipe -- directly from a `pixi shell` or via the `emos run` flow. **Requirements:** Linux (amd64 or arm64). No root, Docker, or ROS2 needed. ::: :::: See the [CLI Reference](cli.md) for the full list of commands. ## Which Mode Should I Choose? | Scenario | Recommended Mode | | :----------------------------------------------- | :--------------- | | No ROS2 on host, quick evaluation | **Container** | | ROS2 already installed, system-level integration | **Native** | | No root, no Docker, any Linux distro | **Pixi** | ## Reach the Dashboard During installation you were asked whether to enable the EMOS dashboard as a systemd service. Pick the path you chose below. ### If you enabled the systemd service (recommended) The dashboard is already running and will come up automatically at every boot. The installer printed the access details once -- a six-digit pairing code, the URLs the dashboard is reachable at, and a scannable QR code. Open any of the URLs in a browser, enter the code, and the browser is paired for ~90 days. If you missed the install output (or you've already paired and just need the URLs again), reprint the access summary at any time: ```bash emos serve ``` When the dashboard is already running as a service, `emos serve` detects that and just shows the URLs and management commands -- it does not try to bind a second instance. Manage the service directly with: ```bash systemctl status emos-dashboard.service systemctl restart emos-dashboard.service journalctl -u emos-dashboard.service -f ``` If you've lost the original pairing code, issue a fresh one with `emos config rotate-pairing`. ### If you skipped the systemd service Start the dashboard manually whenever you want to use it: ```bash emos serve ``` The first launch prints the pairing code, URLs, and QR code. The process runs in the foreground and stops when you `Ctrl-C` it. You can enable the service later with `emos serve install-service`. ```{seealso} [Dashboard](dashboard.md) — full walkthrough of pairing, recipes, and run console ``` ## Preparing Your Hardware Before running recipes, you need sensor drivers publishing data on ROS2 topics. EMOS recipes declare the topics they expect (e.g. `Image` from a camera, `LaserScan` from a lidar). Run `emos info ` to see what a recipe needs. ### Installing Sensor Drivers ::::{tab-set} :::{tab-item} Container The EMOS container runs with `--privileged` and has access to all USB devices on the host. You can install and run sensor drivers directly **inside the container** — no ROS2 installation on the host is needed. ```bash # Install a sensor driver inside the container: docker exec -it emos bash -c "apt-get update && apt-get install -y ros-jazzy-usb-cam" # Launch the driver inside the container (in a separate terminal): docker exec -it emos bash -c "source /ros_entrypoint.sh && ros2 run usb_cam usb_cam_node_exe" ``` The driver's topics are immediately visible to recipes running in the same container. ```{tip} If you have sensor drivers already running on the host with ROS2, they can bridge into the container automatically via Zenoh (the default RMW). Start the host driver with `export RMW_IMPLEMENTATION=rmw_zenoh_cpp`. ``` ::: :::{tab-item} Native Install the driver package and launch it directly: ```bash sudo apt install ros-jazzy-usb-cam source /opt/ros/jazzy/setup.bash export RMW_IMPLEMENTATION=rmw_zenoh_cpp ros2 run usb_cam usb_cam_node_exe ``` If you place a launch file at `~/emos/robot/launch/bringup_robot.py`, the CLI will start it automatically when you run `emos run`. ::: :::{tab-item} pixi Pixi mode assumes you have **no system ROS2 installation**, so sensor drivers are installed into the pixi environment too. The EMOS workspace already has the [RoboStack](https://robostack.github.io/) `robostack-jazzy` channel configured, so adding a driver is a **single command** — install it straight into the EMOS environment: ```bash cd ~/.local/share/emos pixi add ros-jazzy-usb-cam RMW_IMPLEMENTATION=rmw_zenoh_cpp pixi run ros2 run usb_cam usb_cam_node_exe ``` The driver lives in the same environment as your recipes, and because both use Zenoh as the default RMW, its topics are visible to running recipes automatically — no system ROS2, no separate project, no extra channel setup. ```{note} `emos update` **preserves** drivers you add this way: it stashes your local `pixi.toml` / `pixi.lock` changes around the update and reapplies them. (In the rare case a release changes `pixi.toml` itself, you get a clear conflict to resolve rather than a silent overwrite.) ``` ```{tip} If a driver package isn't on RoboStack, install it from source into the EMOS environment with `colcon`, or fall back to Native mode for that driver only. ``` ::: :::: ```{important} Match the driver's RMW implementation to the one your recipe uses, or the driver's topics won't be visible to it. EMOS recipes default to **Zenoh** -- set `export RMW_IMPLEMENTATION=rmw_zenoh_cpp` in the shell where you launch the driver. If the recipe overrides this (e.g. `emos run --rmw rmw_cyclonedds_cpp`), export the same value instead. ``` ### Verifying Sensors Before running a recipe, confirm your sensors are publishing: ```bash # 1. See what the recipe needs emos info vision_follower # 2. Check topics exist ros2 topic list # 3. Confirm data is flowing ros2 topic hz /image_raw ``` If `ros2 topic hz` shows a non-zero rate, the sensor is ready. ```{seealso} If sensor verification fails during `emos run`, see [Troubleshooting](troubleshooting.md). ``` ## Model Serving Platform EMOS is agnostic to model serving platforms. You need at least one of the following available on your network: - {material-regular}`download;1.2em;sd-text-primary` **[Ollama](https://ollama.com)** Recommended for local inference. - {material-regular}`smart_toy;1.2em;sd-text-primary` **[RoboML](https://github.com/automatika-robotics/robo-ml)** Automatika's own open-source model serving package for quick prototyping. - {material-regular}`api;1.2em;sd-text-primary` **OpenAI API-compatible fast inference servers** e.g. [llama.cpp](https://github.com/ggml-org/llama.cpp), [vLLM](https://github.com/vllm-project/vllm), [SGLang](https://github.com/sgl-project/sglang). - {material-regular}`precision_manufacturing;1.2em;sd-text-primary` **[LeRobot](https://github.com/huggingface/lerobot)** For Vision-Language-Action (VLA) models. - {material-regular}`cloud;1.2em;sd-text-primary` **Cloud endpoints** e.g. OpenAI, Claude, HuggingFace Inference etc. using an API key. ```{tip} For larger models, run the serving platform on a GPU-equipped machine on your local network, or use a cloud endpoint, rather than running models directly on the robot. ``` ## Updating Update your installation to the latest version: ```bash emos update ``` The CLI detects your installation mode and updates accordingly: - **Container mode:** pulls the latest image and recreates the container. - **Native mode:** pulls the latest source, rebuilds, and re-installs packages into `/opt/ros/{distro}/`. - **Pixi mode:** runs `git pull` and `git submodule update` in the EMOS workspace at `~/.local/share/emos`, refreshes the pixi environment (`pixi install`), and rebuilds the EMOS packages (`pixi run setup`). Any installed robot plugin is pulled and rebuilt too. ## Uninstalling To remove EMOS from a device: ```bash sudo emos uninstall ``` After confirming, the CLI: - Runs mode-specific cleanup: - **Container mode:** removes the Docker container. The image is preserved unless `--remove-image` is passed. - **Native mode:** The CLI prints the manual `rm` commands so you can clean them ROS packages yourself if you want. - **Pixi mode:** removes `.pixi/`, `build/`, `install/`, `log/` under your EMOS clone. The cloned repo itself is preserved. - Removes `~/emos/recipes`, `~/emos/logs`, and `~/.config/emos` (installed recipes, run logs, and dashboard auth state). Pass `--keep-data` to preserve `~/emos/recipes` and `~/emos/logs`. Pass `--keep-config` to preserve `~/.config/emos` (so previously paired browsers remain valid). Pass `-y` / `--yes` to skip the confirmation prompt. The CLI binary at `/usr/local/bin/emos` is never removed automatically. The command prints the one-liner you can run after the process exits. ```{tip} Use `emos uninstall` before switching install modes (e.g. native -> pixi). It clears auth tokens and mode-specific state that would otherwise carry over and confuse the new install. ``` ## Installing from Source (Developer Setup) If you want to build the full EMOS stack from source for contributing or accessing the latest features, follow the steps below. This installs all three stack components: **Sugarcoat** (architecture), **EmbodiedAgents** (intelligence), and **Kompass** (navigation). ### 1. Create a unified workspace ```shell mkdir -p emos_ws/src cd emos_ws/src ``` ### 2. Clone the stack ```shell git clone https://github.com/automatika-robotics/emos.git cp -r emos/stack/sugarcoat . cp -r emos/stack/embodied-agents . cp -r emos/stack/kompass . ``` ### 3. Install Python dependencies ```shell PIP_BREAK_SYSTEM_PACKAGES=1 pip install numpy opencv-python-headless 'attrs>=23.2.0' jinja2 httpx setproctitle msgpack msgpack-numpy platformdirs tqdm pyyaml toml websockets ``` ### 4. Install the Kompass core engine The `kompass-core` package provides optimized planning and control algorithms. ::::{tab-set} :::{tab-item} GPU Support (Recommended) For production robots or high-performance simulation, install with GPU acceleration: ```bash curl -sSL https://raw.githubusercontent.com/automatika-robotics/kompass-core/refs/heads/main/build_dependencies/install_gpu.sh | bash ``` ::: :::{tab-item} CPU Only For quick testing or lightweight environments: ```bash pip install kompass-core ``` ::: :::: ### 5. Install ROS dependencies and build ```shell cd emos_ws rosdep update rosdep install -y --from-paths src --ignore-src colcon build source install/setup.bash ``` You now have the complete EMOS stack built and ready to use. ``` ## File: getting-started/quickstart.md ```markdown # Quick Start ## Your First EMOS Recipe EMOS lets you describe complete robot behaviors as **recipes** -- pure Python scripts that wire together components, models, and ROS topics using a declarative style. In this quickstart you will build a simple Visual Question Answering recipe: a robot that sees through its camera and answers questions about what it observes. By the end, you'll have run it end-to-end and (optionally) opened a small web UI to talk to it. ```{important} This guide assumes you have already installed EMOS. If not, see the [Installation guide](installation.md) first. ``` ## The Recipe Save the following as `my_first_recipe.py`. We'll walk through what each section does, then list what needs to be running before you launch it. ```python from agents.clients.ollama import OllamaClient from agents.components import VLM from agents.models import OllamaModel from agents.ros import Topic, Launcher # Define input and output topics (pay attention to msg_type) text0 = Topic(name="text0", msg_type="String") image0 = Topic(name="image_raw", msg_type="Image") text1 = Topic(name="text1", msg_type="String") # Define a model client (Ollama in this case) qwen_vl = OllamaModel(name="qwen_vl", checkpoint="qwen2.5vl:latest") qwen_client = OllamaClient(qwen_vl) # Define a VLM component (a node with a particular functionality) vlm = VLM( inputs=[text0, image0], outputs=[text1], model_client=qwen_client, trigger=text0, component_name="vqa", ) vlm.set_topic_prompt(text0, template="""You are an amazing and funny robot. Answer the following about this image: {{ text0 }}""" ) # Launch the component launcher = Launcher() launcher.add_pkg(components=[vlm]) launcher.bringup() ``` ## Step-by-Step Breakdown ### Define Topics Every EMOS recipe starts by declaring the ROS topics that connect components together. Components automatically create listeners for input topics and publishers for output topics. ```python text0 = Topic(name="text0", msg_type="String") image0 = Topic(name="image_raw", msg_type="Image") text1 = Topic(name="text1", msg_type="String") ``` ```{note} On a real robot, change `image0`'s name to match the topic your camera driver actually publishes (e.g. `/camera/color/image_raw`). The "Before You Run" section below explains how to confirm what's available. ``` ### Create a Model Client EMOS is model-agnostic. Here we create a client that uses [Qwen2.5vl](https://ollama.com/library/qwen2.5vl) served by [Ollama](https://ollama.com): ```python qwen_vl = OllamaModel(name="qwen_vl", checkpoint="qwen2.5vl:latest") qwen_client = OllamaClient(qwen_vl) ``` ````{tip} If Ollama is running on a different machine on your network, specify the host and port: ```python qwen_client = OllamaClient(qwen_vl, host="127.0.0.1", port=8000) ``` ```` ### Configure the Component Components are the functional building blocks of EMOS recipes. The VLM component lets you set topic-level prompts using Jinja2 templates so you can shape the model's behavior per-input: ```python vlm = VLM( inputs=[text0, image0], outputs=[text1], model_client=qwen_client, trigger=text0, component_name="vqa", ) vlm.set_topic_prompt(text0, template="""You are an amazing and funny robot. Answer the following about this image: {{ text0 }}""" ) ``` ### Launch Finally, bring the recipe up: ```python launcher = Launcher() launcher.add_pkg(components=[vlm]) launcher.bringup() ``` ## Before You Run The recipe needs three things in place before it can do anything useful. **1. Ollama is running and the model is pulled.** ```bash curl http://localhost:11434/api/tags # is Ollama up? ollama pull qwen2.5vl:latest # pre-fetch the model ``` **2. Something is publishing on the camera topic.** On a development machine, [ROS 2 USB Cam](https://github.com/klintan/ros2_usb_camera) is a quick way to publish from a webcam. On a real robot, start your camera driver -- see [Preparing Your Hardware](installation.md#preparing-your-hardware) for mode-specific instructions, including the container-mode caveat that drivers run **inside** the container. Confirm a publisher is up and producing frames at non-zero rate: ```bash ros2 topic list ros2 topic hz /image_raw # or whatever name you set in the recipe ``` If the topic name your driver publishes differs from `image_raw`, update `image0` in the recipe to match. **3. Component-specific extras.** Some clients prompt for additional Python packages on first launch -- the script will throw an error and tell you exactly what to install. If any of these fail when you run the recipe, see [Troubleshooting](troubleshooting.md) for sensor verification and model-server gotchas. ## Run It Two ways to run, pick by purpose. ### Option A: just run the script For fast iteration while you're shaping the recipe. Works in Native and Pixi modes. ::::{tab-set} :::{tab-item} Native ```bash source /opt/ros/jazzy/setup.bash # or your installed distro python3 my_first_recipe.py ``` ::: :::{tab-item} Pixi ```bash # Activate the EMOS pixi env without leaving your recipe's directory pixi shell --manifest-path ~/.local/share/emos/pixi.toml source ~/.local/share/emos/install/setup.sh # adds the built EMOS packages to your env python3 my_first_recipe.py ``` ::: :::{tab-item} Container Direct `python3` from the host **does not work** in container mode -- the `agents`, `kompass`, and `ros_sugar` Python packages live inside the container. Use Option B, or `docker exec -it emos-container bash` first and run the script from there. ::: :::: ### Option B: run it through the EMOS flow For sensor verification, log retention, and visibility on the dashboard. **Required in container mode.** ```bash mkdir -p ~/emos/recipes/my_first_recipe cp my_first_recipe.py ~/emos/recipes/my_first_recipe/recipe.py emos run my_first_recipe ``` The recipe now appears on the dashboard's **Recipes → Installed** tab, and every run is logged to `~/emos/logs/my_first_recipe_.log`. ```{seealso} [Running Recipes](running-recipes.md) -- the full comparison of both flows, install-mode pitfalls, and the optional `manifest.json` schema (display name, description, Zenoh router config). ``` ## Verify It Is Running From a separate terminal, use standard ROS 2 commands to confirm the node and its topics are active: ```bash ros2 node list # should list the `vqa` node ros2 topic list # should list text0, image_raw, text1 ``` To trigger a single inference manually, publish a question to `text0` and watch `text1` for the reply: ```bash ros2 topic pub --once /text0 std_msgs/String "{data: 'what do you see?'}" ros2 topic echo /text1 ``` ## Add a Web UI EMOS can dynamically generate a web-based UI for any recipe -- so you don't need a terminal to interact with it. Add one line before `bringup()` to tell the launcher which topics to render: ```python launcher = Launcher() launcher.enable_ui(inputs=[text0], outputs=[text1, image0]) # <-- specify UI launcher.add_pkg(components=[vlm]) launcher.bringup() ``` ````{note} The web UI requires two additional packages: ```bash pip install python-fasthtml monsterui ``` ```` The UI is served at **http://localhost:5001** (or **http://<ROBOT_IP>:5001** if running on a robot). Open it in a browser, send a question via the input field, and you should get a reply generated by the Qwen2.5vl model. ![Demo screencast](https://automatikarobotics.com/docs/ui_agents_vlm.gif) ## Where Next - **Checkout your EMOS dashboard** Open the EMOS Dashboard at `http://emos.local:8765` (or scan the QR printed by `emos serve`). Pair once and you can pull recipes, launch them, and watch logs from any browser. See [Dashboard](dashboard.md). - **Customize this recipe.** [Running Recipes](running-recipes.md) covers the optional `manifest.json` and the install-mode reference for all runtimes. - **Build something more capable.** The [Recipes & Tutorials](../recipes/overview.md) section walks through conversational agents, semantic memory, navigation, manipulation, and the Cortex agentic harness. - **Hit a snag?** See [Troubleshooting](troubleshooting.md) for sensor verification failures, model-server timeouts, and mode-specific issues. ``` ## File: getting-started/cli.md ```markdown # EMOS CLI The `emos` CLI manages installation, recipes, the dashboard daemon, and device configuration on a robot. Every long-form action it performs is also exposed over the dashboard's REST API (see [`internal/server/openapi.yaml`](https://github.com/automatika-robotics/emos/blob/main/stack/emos-cli/internal/server/openapi.yaml)), so anything you can do on the terminal you can also drive from a browser or an agentic skill. ## Quick Reference | Command | Description | | :----------------- | :---------------------------------------------------------- | | `emos install` | Install EMOS (interactive mode selection) | | `emos uninstall` | Remove EMOS (mode-aware cleanup) | | `emos update` | Update EMOS to the latest version | | `emos status` | Show installation status | | `emos serve` | Run the dashboard daemon (REST API + web UI) | | `emos config` | Inspect or modify device configuration, pairing tokens, TLS | | `emos recipes` | List recipes available for download | | `emos pull ` | Download a recipe | | `emos ls` | List locally installed recipes | | `emos run ` | Run a recipe (foreground, blocking) | | `emos info ` | Show sensor/topic requirements for a recipe | | `emos map ` | Mapping tools (record, edit) | | `emos plugin `| Install and manage the robot plugin | | `emos version` | Show CLI version | ```{tip} Every command supports `-h`/`--help`. The CLI is also a single static binary — copy `/usr/local/bin/emos` to another machine and it just works (no Python, no runtime dependencies). ``` ## Typical Workflows ### First-time setup ```bash # 1. Install EMOS (interactive mode menu) emos install # 2. Start the dashboard (printed pairing code is shown once) emos serve # 3. (optional) Make the dashboard auto-start at boot sudo emos serve install-service ``` After step 2, point a browser at `http://emos.local:8765` (or scan the QR), enter the pairing code, and you're in. See [Dashboard](dashboard.md). ### CLI-only recipe loop ```bash emos recipes # browse the catalog emos pull vision_follower # download a recipe emos info vision_follower # check what sensors it needs emos run vision_follower # launch it (blocks until exit) ``` ## Running Recipes `emos run ` adapts to the install mode -- starting the container in container mode, sourcing ROS in native, activating the pixi env in pixi -- before exec-ing the recipe and streaming logs to `~/emos/logs/_.log`. Logs are also visible from the dashboard's [Run console](dashboard.md#run-console). For the full guide to writing, dropping in, and launching custom recipes (including the install-mode pitfalls of running them directly via `python`), see [Running Recipes](running-recipes.md). ## Recipe Layout A recipe is a directory under `~/emos/recipes/` with the following structure: ``` ~/emos/recipes/ my_recipe/ recipe.py # Main entry point (required) manifest.json # Optional: Zenoh config / display name / description ``` ### `manifest.json` ```json { "name": "My Recipe", "description": "Does the thing.", "zenoh_router_config_file": "my_recipe/zenoh_config.json5" } ``` - {material-regular}`label;1.2em;sd-text-primary` **name** — display name for the dashboard's recipe cards. Falls back to the directory name. - {material-regular}`description;1.2em;sd-text-primary` **description** — short blurb shown on the recipe detail page. - {material-regular}`settings;1.2em;sd-text-primary` **zenoh_router_config_file** — path (relative to `~/emos/recipes/`) to a Zenoh router `.json5` config file. Only consulted when the recipe runs under `rmw_zenoh_cpp`. ```{note} Sensor requirements are auto-extracted from `recipe.py` by parsing `Topic(name=..., msg_type=...)` declarations. You don't need to list them in the manifest. Run `emos info ` (or open the recipe in the dashboard) to see the inferred requirements. ``` For the full walkthrough -- writing the recipe, dropping it in, verifying discovery, and launching it via `emos run` or the dashboard -- see [Running Recipes](running-recipes.md). ## Command Reference ### `emos install` ```bash emos install # interactive mode menu emos install --mode container # OSS container (no ROS required on host) emos install --mode native # native (uses host's ROS 2) emos install --mode pixi # self-contained ROS via pixi (no system ROS) emos install --mode licensed # licensed deployment (requires license key) emos install --distro jazzy # pin a ROS distro for container/native mode ``` | Flag | Default | Description | | :--------- | :--------- | :--------------------------------------------------- | | `--mode` | _(prompt)_ | One of: `container`, `native`, `pixi`, `licensed`. | | `--distro` | _(prompt)_ | ROS 2 distribution: `jazzy`, `humble`, `kilted`. | The installer offers, at the end, to: - Install a systemd unit so the dashboard auto-starts at boot (see [Make the dashboard start automatically](dashboard.md#make-the-dashboard-start-automatically)). - Persist the chosen device name and a fresh pairing code to `~/.config/emos/config.json`. ```{note} Pixi mode requires [pixi](https://pixi.sh) on the host (`emos install --mode pixi` errors with install instructions if it's missing). It clones the EMOS workspace and builds it under `~/.local/share/emos`, independent of any system ROS. Currently pinned to ROS 2 Jazzy. See [Installation](installation.md#deployment-modes). ``` ### `emos uninstall` ```bash sudo emos uninstall # interactive sudo emos uninstall --yes # non-interactive sudo emos uninstall --keep-data # preserve recipes + logs sudo emos uninstall --keep-config # preserve dashboard auth state sudo emos uninstall --remove-image # also docker rmi (container / licensed) ``` Stops the dashboard service and runs mode-specific cleanup. By default also removes `~/emos/recipes`, `~/emos/logs`, and `~/.config/emos`. | Flag | Default | Description | | :--------------- | :------ | :------------------------------------------------------------------------------------------- | | `--keep-data` | `false` | Preserve `~/emos/recipes` and `~/emos/logs`. | | `--keep-config` | `false` | Preserve `~/.config/emos` (keeps device name + dashboard pairing across reinstall). | | `--remove-image` | `false` | Also `docker rmi` the EMOS image (container / licensed modes only; preserved by default). | | `-y`, `--yes` | `false` | Skip the confirmation prompt. | Mode-specific behavior: - **Container / licensed:** `docker stop` + `docker rm` the EMOS container; licensed also removes the container auto-restart unit and `~/emos/robot/`. - **Native:** removes the build workspace and `pip uninstall`s `kompass-core`. EMOS package files in `/opt/ros//` are co-mingled with ROS by colcon and **cannot** be cleanly removed -- the command prints the manual `rm` commands rather than running them, so you can review and apply if you want. - **Pixi:** removes the EMOS-owned pixi workspace at `~/.local/share/emos` (cloned repo + env + build) wholesale. A workspace you cloned yourself elsewhere is preserved — only its build artifacts are stripped. The CLI binary at `/usr/local/bin/emos` is never removed automatically -- a running process can't reliably unlink itself. The command prints the `sudo rm` one-liner for you. ```{tip} Run `emos uninstall` before switching install modes (e.g. native -> pixi). It clears auth tokens and mode-specific state that would otherwise carry over and confuse the new install. ``` ### `emos update` ```bash emos update ``` Detects the install mode and updates accordingly. Container mode pulls the latest image and recreates the container; native mode pulls the latest source, rebuilds, and re-installs into `/opt/ros/{distro}/`; pixi mode runs `git pull`, refreshes the pixi env (`pixi install`), and rebuilds the EMOS packages (`pixi run setup`). If a robot plugin is installed, it is also pulled to its latest commit and rebuilt. ### `emos status` ```bash emos status ``` Shows install mode, ROS distro, container/service state, and dashboard service health. Subset of what `emos config show` reports. ### `emos serve` Run the dashboard daemon. See the dedicated [Dashboard](dashboard.md) page for the UX. ```bash emos serve # foreground; HTTP on the configured port emos serve --tls # opt into HTTPS (self-signed cert) emos serve --addr :9000 # bind to a custom port for one run emos serve --qr # print a QR for the dashboard URL and exit ``` | Flag | Default | Description | | :---------------- | :-------- | :----------------------------------------------------------------------------------------------- | | `--addr` | _(empty)_ | `host:port` to bind. Empty falls back to the configured port (`emos config set port`) or `8765`. | | `--no-mdns` | `false` | Skip mDNS announcement. The dashboard is then only reachable by IP / explicit hostname. | | `--no-auth` | `false` | **Dev only.** Accept all requests without a bearer token. | | `--tls` | `false` | Serve over HTTPS using a self-signed cert under `~/.config/emos/`. | | `--qr` | `false` | Print a QR code with the dashboard URL and exit (no daemon). | | `-v`, `--verbose` | `false` | Log every HTTP request (including reads) at DEBUG. | #### `emos serve install-service` ```bash sudo emos serve install-service ``` Writes `/etc/systemd/system/emos-dashboard.service`, enables it, and starts it. The unit's `ExecStart` points at the binary you ran the command with, so it follows the active install (`/usr/local/bin/emos` if installed via the script). #### `emos serve uninstall-service` ```bash sudo emos serve uninstall-service ``` Stops, disables, and removes the unit. Does not remove the cert, config, or recipes. ### `emos config` Inspect and modify everything in `~/.config/emos/config.json` — install info, device name, port, paired-device tokens, and TLS material. ```bash emos config show # human-readable device state emos config get [key] # print one value or the whole config as JSON emos config set # writable keys: name, port emos config path # print the config file path emos config tokens # list paired browsers / agents emos config revoke-token # revoke a single paired device emos config rotate-pairing # issue a fresh pairing code (existing tokens stay valid) emos config tls-fingerprint # print the dashboard TLS cert SHA-256 fingerprint emos config tls-regenerate # re-mint the self-signed TLS cert (use after IP change) emos config reset # reset device state (pairing/name/port); keeps install info ``` #### `emos config show` Prints a single-pane summary: ```text EMOS DEVICE STATE Identity: epic-otter Mode: native ROS distro: jazzy Dashboard port: 8765 Recipes: /home/you/emos/recipes Logs: /home/you/emos/logs Config file: /home/you/.config/emos/config.json Pairing configured: yes Active tokens: 2 Dashboard service: active (emos-dashboard.service) ``` #### `emos config get [key]` With no argument, prints the full config as JSON. With a key (`name`, `mode`, `ros_distro`, `port`), prints a single value — useful in shell scripts. #### `emos config set ` Writable keys: `name` (mDNS hostname segment, validated against `[a-z0-9-]`), `port` (1–65535). Other fields are managed by the installer / serve daemon. ```bash emos config set name happy-robot emos config set port 9000 sudo systemctl restart emos-dashboard.service # if running as a service ``` #### `emos config tokens` and `revoke-token` Lists paired browsers without leaking the underlying token hash: ```text ID LABEL ISSUED EXPIRES 4d0e9c01 phone 2026-04-12 10:32 2026-07-11 10:32 71a3f82b laptop 2026-04-12 11:07 2026-07-11 11:07 ``` Revoke one device: ```bash emos config revoke-token 4d0e9c01 # by short id (prefix of hash) emos config revoke-token phone # by exact label ``` #### `emos config rotate-pairing` Issues a new six-digit pairing code, **without** revoking already-paired tokens. Useful when a code may have been seen by someone who shouldn't get further access. ```bash emos config rotate-pairing # ✓ New pairing code (shown once): 829471 ``` #### `emos config tls-fingerprint` / `tls-regenerate` See [HTTPS (Optional)](#https-optional) below. ### `emos pull` ```bash emos pull ``` Downloads a recipe from the Automatika catalog and extracts it to `~/emos/recipes//`. Overwrites the existing version if present. Requires internet. ### `emos ls` ```bash emos ls ``` Lists everything under `~/emos/recipes/`. The dashboard's **Recipes → Installed** tab shows the same set. ### `emos info` ```bash emos info ``` Inspects a recipe's Python source via AST and prints its sensor and topic requirements. Accepts either a recipe name (looked up in `~/emos/recipes/`) or a path to a `.py` file: ```bash emos info vision_follower # ~/emos/recipes/vision_follower/recipe.py emos info ./my_recipe.py # explicit path ``` The output groups topics into: - **Required Sensors** — `Image`, `LaserScan`, `Imu`, `Audio`, `Odometry`, `RGBD`, `PointCloud2`, `CompressedImage`. Hardware label and suggested apt packages tailored to your distro. - **Other Topics** — non-sensor topics declared by the recipe. ### `emos run` ```bash emos run emos run --rmw rmw_cyclonedds_cpp emos run --skip-sensor-check ``` | Flag | Default | Description | | :-------------------- | :-------------- | :----------------------------------------------------------------- | | `--rmw` | `rmw_zenoh_cpp` | One of: `rmw_zenoh_cpp`, `rmw_fastrtps_cpp`, `rmw_cyclonedds_cpp`. | | `--skip-sensor-check` | `false` | Skip the 10-second sensor-topic verification. | #### What happens during `emos run` 1. Reads `recipe.py` and extracts `Topic(...)` declarations. 2. Identifies sensor topics. 3. Starts the Zenoh router (only when using `rmw_zenoh_cpp`). 4. Launches `~/emos/robot/launch/bringup_robot.py` if it exists (native / pixi only; container mode uses an in-container bringup). 5. Verifies each sensor topic is publishing (polls `ros2 topic list` for up to 10 s). 6. Executes the recipe — output streams to the terminal and is saved to `~/emos/logs/`. #### When to use `--skip-sensor-check` - Sensors that publish on-demand (service-triggered cameras). - Replaying a rosbag whose topic names differ from the recipe. - Pure AI recipes (LLM chat, TTS) that don't require sensor data. ```{warning} If you skip the check and a sensor topic never arrives, the recipe may hang silently waiting for data. Use `ros2 topic hz /topic_name` to diagnose. ``` ```{seealso} [Troubleshooting](troubleshooting.md) for common errors during recipe execution. ``` ### `emos plugin` Install and manage the robot plugin — a ROS package that adapts a specific robot to the EMOS stack. A robot runs **one plugin at a time**. See [Robot Plugins](plugins.md) for the full guide. ```bash emos plugin list # browse the catalog (active plugin marked) emos plugin install # install + activate (replaces any active plugin) emos plugin inspect # show the active plugin's feedbacks/commands/actions/events emos plugin remove # remove the active plugin ``` | Subcommand | Description | | :------------------ | :------------------------------------------------------------------------------------------- | | `list` | List plugins available in the Automatika catalog; flags the currently active one. | | `install ` | Clone, build (for your install mode), and activate a plugin. Prompts before replacing one. | | `inspect` | Pretty-print the active plugin's introspection tree (same data the dashboard System page shows). | | `remove` | Remove the active plugin from the robot. | Installing a plugin makes it importable; a recipe opts in with `Launcher(robot_plugin=MyRobotPlugin())`. The dashboard's [Plugins page](dashboard.md#plugins) drives the same flow from a browser. ### `emos map` Mapping subcommands for creating and editing environment maps: ```bash emos map record # record mapping data on the robot emos map install-editor # install the map editor container (one-time) emos map edit # process a ROS bag into a PCD map ``` ### `emos version` ```bash emos version # emos vX.Y.Z ``` Prints the CLI version. The dashboard's `/api/v1/info` returns the same value. ## HTTPS (Optional) The dashboard serves plain HTTP by default. That's the right default on a trusted LAN — bearer-token auth gates every write, so eavesdropping yields nothing useful unless someone can also intercept the pairing handshake. You can opt in to HTTPS with a self-signed certificate: ```bash emos serve --tls ``` ### What the flag does On first launch under `--tls`, the daemon mints a 2-year ECDSA P-256 certificate and persists it under `~/.config/emos/`: | Path | Mode | Purpose | | :----------------------- | :----- | :------------------------------------------------------ | | `~/.config/emos/tls.crt` | `0644` | PEM-encoded leaf certificate (also acts as its own CA). | | `~/.config/emos/tls.key` | `0600` | PEM-encoded private key. Never copy this off the robot. | The certificate's SubjectAltName covers: - `localhost`, `.local`, `emos.local` - `127.0.0.1`, `::1` - Every LAN IPv4 the device sees at mint time (excluding loopback / docker / veth / tailscale-style virtual interfaces). It re-uses the persisted cert on subsequent launches and auto-rotates inside the 30-day-before-expiry window. Network-level changes (new IP, new mDNS name) don't trigger automatic rotation — see `tls-regenerate` below. ### Trusting the cert A self-signed cert produces a "Not Secure" warning the first time a browser hits it. The warning is encryption-preserving — TLS still negotiates a session — but the trust chain is empty, so browsers refuse to call the connection authenticated. Two paths from there: 1. **Click through, every time.** Fine for a workshop / one-off install. The encryption protects against passive sniffing on the LAN, the bearer token still protects against unauthenticated access, and the warning is mostly cosmetic. Verify the cert before clicking through: ```bash emos config tls-fingerprint ``` Compare the printed SHA-256 fingerprint with the one the browser shows under "View Certificate" before trusting it. 2. **Add the cert to a trust store** so the warning goes away permanently for that device: - **Firefox** keeps its own trust store: _Settings → Privacy & Security → Certificates → View Certificates → Authorities → Import_ `~/.config/emos/tls.crt`, then tick _"Trust this CA to identify websites."_ - **Chrome / Edge** follow the OS trust store. On Linux: `sudo cp tls.crt /usr/local/share/ca-certificates/emos-.crt && sudo update-ca-certificates`. On macOS, drop the cert into Keychain Access and mark it Always Trust. On Windows, `certmgr.msc → Trusted Root Certification Authorities → Import`. ### When to regenerate The SAN list is fixed at mint time. If the robot's primary IP changes (new network, new DHCP lease) or you renamed it (`emos config set name`), the cert no longer matches the new address and the browser will show a different error (`NET::ERR_CERT_COMMON_NAME_INVALID`). Regenerate: ```bash emos config tls-regenerate sudo systemctl restart emos-dashboard.service # if running as a service ``` Inspect the active fingerprint at any time: ```bash emos config tls-fingerprint # TLS CERTIFICATE # Fingerprint (SHA-256): # 8B:F2:0C:...:E7 # Expires: 2028-04-28 # Certificate: /home/you/.config/emos/tls.crt # Private key: /home/you/.config/emos/tls.key ``` ### When you actually need HTTPS Most EMOS deployments are happy on HTTP. Reach for `--tls` when: - You're building dashboard features that need a **secure context** — `getUserMedia` (microphone / camera), Web Audio capture, Service Workers, Web Bluetooth. Browsers won't expose these APIs over plain HTTP. - You're running on a network you don't fully control (a venue Wi-Fi, a colocated factory) where token-only auth feels too thin. - An organisational policy requires HTTPS end-to-end. ```{tip} Per-recipe Sugarcoat web UIs that need a secure context handle their own TLS independently of the dashboard. See the [Dynamic Web UI](../concepts/web-ui.md) page. ``` ### Running as a service with TLS The systemd unit installed by `emos serve install-service` runs plain HTTP. To switch the service to HTTPS, edit its `ExecStart`: ```bash sudo systemctl edit --full emos-dashboard.service # add --tls to the ExecStart line, e.g. # ExecStart=/usr/local/bin/emos serve --addr :8765 --tls sudo systemctl daemon-reload sudo systemctl restart emos-dashboard.service ``` ## Files & Paths ```{list-table} :header-rows: 1 :widths: 35 65 * - Path - Purpose * - `~/.config/emos/config.json` - Single source of truth: install info, device name, port, paired-device tokens (hashed). Mode `0600`. * - `~/.config/emos/tls.crt` / `tls.key` - Self-signed TLS material (created by `emos serve --tls` or `emos config tls-regenerate`). * - `~/emos/recipes/` - Installed recipes. Each subdirectory is one recipe. * - `~/emos/logs/` - Per-run log files: `_.log`. Streamed by `emos run` and the dashboard. * - `~/emos/ros_ws/` - Native-mode build workspace. * - `~/.local/share/emos/` - Pixi-mode install: cloned EMOS workspace, pixi env, and colcon overlay (`emos install --mode pixi`). * - `~/emos/workspace/` - Robot-plugin source and build overlay (`emos plugin install`). * - `/etc/systemd/system/emos-dashboard.service` - Dashboard auto-start unit (created by `emos serve install-service`). * - `/etc/systemd/system/emos.service` - Container auto-restart unit (created by licensed install). ``` ```{tip} `~/.config/emos/config.json` is **the** persistent state. `emos config reset` clears the dashboard's device-side state (paired browsers, custom name, custom port) but **preserves install info** (mode, ROS distro, license key) so the dashboard keeps recognising the device as installed afterwards. ``` ``` ## File: concepts/architecture.md ```markdown # Architecture **The unified orchestration layer for Physical AI.** EMOS (The Embodied Operating System) is the software layer that transforms quadrupeds, humanoids, and mobile robots into **Physical AI Agents**. Just as Android standardized the smartphone hardware market, EMOS provides a bundled, hardware-agnostic runtime that allows robots to see, think, move, and adapt in the real world. ## The Body/Mind Split At its core, EMOS decouples the robot's **Body** from its **Mind**, creating a standard interface for intelligence. - {material-regular}`precision_manufacturing;1.2em;sd-text-primary` **The Body** encompasses the physical hardware: motors, sensors, actuators, and the low-level drivers that control them. EMOS abstracts over the specifics of any particular robot platform, whether it is a wheeled AMR, a quadruped, or a humanoid. - {material-regular}`psychology;1.2em;sd-text-primary` **The Mind** is the software intelligence that perceives the world, reasons about it, and decides how to act. EMOS provides the cognitive and navigational primitives that turn raw sensor data into purposeful behavior. This separation means that the same application logic --- a "Recipe" --- can be written once and deployed across entirely different robot bodies without rewriting code. EMOS handles the translation between intent and hardware. ## The Three Layers EMOS is built on three open-source, publicly developed core components that work in tandem. Each layer addresses a distinct concern of the robotic software stack. :::{image} ../_static/images/diagrams/emos_diagram_light.png :align: center :width: 500px :class: light-only ::: :::{image} ../_static/images/diagrams/emos_diagram_dark.png :align: center :width: 500px :class: dark-only ::: ### Intelligence Layer: EmbodiedAgents [EmbodiedAgents](https://github.com/automatika-robotics/embodied-agents) is the orchestration framework for building agentic graphs of ML models. It provides: - {material-regular}`visibility;1.2em;sd-text-primary` **Multi-modal perception** using vision-language models, object detectors, and speech processing. - {material-regular}`memory;1.2em;sd-text-primary` **Hierarchical spatio-temporal memory** for contextual reasoning about the robot's environment over time. - {material-regular}`alt_route;1.2em;sd-text-primary` **Semantic routing** that directs user commands to the correct capability (navigation, vision, conversation) based on intent. - {material-regular}`sync;1.2em;sd-text-primary` **Adaptive reconfiguration** that allows the robot to switch between cloud APIs and local models at runtime based on connectivity and latency requirements. ### Navigation Layer: Kompass [Kompass](https://github.com/automatika-robotics/kompass) is the event-driven navigation stack responsible for real-world mobility. It provides: - {material-regular}`speed;1.2em;sd-text-primary` **GPGPU-accelerated planning** that moves heavy geometric computation to the GPU, achieving up to 3,106x speedups over CPU-based approaches and freeing the CPU for application logic. - {material-regular}`settings;1.2em;sd-text-primary` **Hardware-agnostic control** that works across wheeled, legged, and tracked platforms. - {material-regular}`bolt;1.2em;sd-text-primary` **Event-driven architecture** where planners and controllers react to environmental changes (obstacles, terrain shifts, emergency stops) rather than running in fixed polling loops. ### Architecture Layer: Sugarcoat [Sugarcoat](https://github.com/automatika-robotics/sugarcoat) is the meta-framework that provides the foundational system design primitives on which both EmbodiedAgents and Kompass are built. It provides: - {material-regular}`autorenew;1.2em;sd-text-primary` **Lifecycle-managed Components** that replace standard ROS2 nodes with self-healing, health-aware execution units. - {material-regular}`flash_on;1.2em;sd-text-primary` **An Event-Driven system** that enables dynamic behavior switching based on real-time environmental context. - {material-regular}`rocket_launch;1.2em;sd-text-primary` **A Launcher and Monitor** that orchestrate multi-process or multi-threaded deployments with automatic lifecycle management. - {material-regular}`code;1.2em;sd-text-primary` **A beautifully imperative Python API** for specifying system configurations as "Recipes" rather than XML launch files. ## How the Layers Work Together The three layers form a vertical stack where each layer builds on the one below it: 1. **Sugarcoat (Architecture)** provides the execution primitives: Components, Topics, Events, Actions, Fallbacks, and the Launcher. Every node in the system --- whether it handles perception, planning, or control --- is a Sugarcoat Component with lifecycle management, health reporting, and self-healing capabilities. 2. **Kompass (Navigation)** builds on Sugarcoat's Component model to implement specialized navigation nodes: path planners, motion controllers, and drivers. These nodes communicate through Sugarcoat Topics, react to Sugarcoat Events, and recover from failures using Sugarcoat Fallbacks. 3. **EmbodiedAgents (Intelligence)** builds on the same Component model to implement cognitive nodes: vision-language models, semantic routers, and memory systems. These nodes can trigger navigation behaviors in Kompass, respond to navigation events, and share data through the common Topic infrastructure. At runtime, all three layers are unified by the **Launcher**, which brings the complete system to life in a single Python script --- the Recipe. The Recipe declares which components to run, how they are wired together, what events to monitor, and what actions to take when conditions change. The result is a robot that can see, think, move, and adapt, all orchestrated from one coherent system. ## Recipes: The Developer Interface A Recipe is a standard Python script that uses the EMOS API to declare an entire robotic application. Recipes are not just scripts; they are complete agentic workflows that combine intelligence, navigation, and system orchestration into a single, readable specification. ```python from ros_sugar import Launcher from ros_sugar.core import Event, Action from ros_sugar.io import Topic # Define components from any EMOS layer # ... intelligence components from EmbodiedAgents # ... navigation components from Kompass # ... custom components built on Sugarcoat # Wire them together with Topics, Events, and Actions # Launch everything with a single call launcher = Launcher(multi_processing=True) launcher.add_pkg(components=[...], events_actions={...}) launcher.bringup() ``` This imperative, Pythonic approach replaces the traditional ROS2 workflow of XML launch files and YAML configurations with a single source of truth that is easy to read, version, and share. ``` ## File: concepts/components.md ```markdown # Components **Stop writing boilerplate. Start writing core logic.** In EMOS, a `Component` is the fundamental unit of execution. It replaces the standard ROS2 Node with a robust, **Lifecycle-Managed**, and **Self-Healing** entity designed for production-grade autonomy. While a standard ROS2 node requires you to manually handle parameter callbacks, error catching, and state transitions, an EMOS Component handles this plumbing automatically, letting you focus entirely on your algorithm. ## Why Build with EMOS Components? EMOS Components come with "superpowers" out of the box. - {material-regular}`autorenew;1.5em;sd-text-primary` Lifecycle Native - Every component is a **Managed Lifecycle Node**. It supports `Configure`, `Activate`, `Deactivate`, and `Shutdown` states automatically, ensuring deterministic startup and shutdown. - {material-regular}`healing;1.5em;sd-text-primary` Self-Healing - Components have a built-in "Immune System." If an algorithm fails or a driver disconnects, the component can trigger **[Fallbacks](status-and-fallbacks.md#fallback-strategies)** to restart or reconfigure itself without crashing the stack. - {material-regular}`monitor_heart;1.5em;sd-text-primary` Health Aware - Components actively report their **[Health Status](status-and-fallbacks.md#health-status)** (Healthy, Algorithm Failure, etc.) to the system, enabling system-wide reflexes and alerts. - {material-regular}`verified;1.5em;sd-text-primary` Type-Safe Config - Component configurations are validated using `attrs` models, catching type errors before runtime, and allowing easy Pythonic configuration in your recipe. - {material-regular}`hub;1.5em;sd-text-primary` Auto-Wiring - Inputs and Outputs are declarative. Define a `Topic` as an input or output to your component, and EMOS automatically handles the subscription, serialization, and callback plumbing for you. - {material-regular}`bolt;1.5em;sd-text-primary` Event-Driven - Components are reactive by design. They can be configured to execute their main logic only when triggered by an **[Event](events-and-actions.md)** or a Service call, rather than running in a continuous loop. ```{figure} /_static/images/diagrams/component_dark.png :class: dark-only :alt: component structure :align: center ``` ```{figure} /_static/images/diagrams/component_light.png :class: light-only :alt: component structure :align: center Component Architecture ``` ## Execution Modes (Run Types) A Component isn't just a `while(True)` loop. You can configure *how* its main functionality executes using the `run_type` property. ```{list-table} :widths: 15 50 35 :header-rows: 1 * - Run Type - Description - Best For... * - **Timed** - Executes the main step in a fixed-frequency loop (e.g., 10Hz). - Controllers, Planners, Drivers * - **Event** - Dormant until triggered by a specific Topic or Event. - Image Processors, Detectors * - **Server** - Dormant until a ROS2 Service Request is received. - Calibration Nodes, Compute Servers * - **ActionServer** - Dormant until a ROS2 Action Goal is received. - Long-running tasks (Navigation, Arms) ``` **Configuration Example:** ```python from ros_sugar.config import ComponentRunType from ros_sugar.core import BaseComponent # Can set from Component comp = BaseComponent(component_name='test') comp.run_type = "Server" # or ComponentRunType.SERVER ``` :::{tip} All the functionalities implemented in ROS2 nodes can be found in the Component. ::: ## Declarative Inputs & Outputs Wiring up data streams shouldn't be tedious. EMOS allows you to define inputs and outputs declaratively. When the component launches, it automatically creates the necessary publishers, subscribers, and type converters based on your definitions. ```python from ros_sugar.core import BaseComponent from ros_sugar.io import Topic # 1. Define your interface map_topic = Topic(name="map", msg_type="OccupancyGrid") voice_topic = Topic(name="voice_cmd", msg_type="Audio") image_topic = Topic(name="camera/rgb", msg_type="Image") # 2. Auto-wire the component # EMOS handles the QoS, callback groups, and serialization automatically comp = BaseComponent( component_name='audio_processor', inputs=[map_topic, image_topic], outputs=[voice_topic] ) ``` :::{tip} EMOS provides built-in "Converters" for common ROS2 types (Images, Pose, etc.), so you can work with native Python objects instead of raw ROS2 messages. ::: :::{seealso} Check the full configuration options of Topics [here](topics.md) ::: ## The Component Immune System: Health & Fallbacks A robust robot doesn't just crash when an error occurs; it degrades gracefully. ### Health Status Instead of printing a log message and dying, a Component reports its **Health Status**. This status is both: - {material-regular}`settings;1.2em;sd-text-primary` **Internal:** Used immediately by the component to trigger local recovery strategies. - {material-regular}`cell_tower;1.2em;sd-text-primary` **External:** Broadcasted to alert other parts of the system. ### Fallbacks (Self-Healing) You can define **reflexes** that trigger automatically when health degrades. * {material-regular}`restart_alt;1.2em;sd-text-danger` *Is the driver dead?* **Restart** the node. * {material-regular}`tune;1.2em;sd-text-warning` *Is the planner stuck?* **Reconfigure** the tolerance parameters. * {material-regular}`swap_horiz;1.2em;sd-text-primary` *Is the sensor noisy?* **Switch** to a different algorithm. > **Learn More:** [Status & Fallbacks Guide](./status-and-fallbacks.md). ## Pro Tips for Component Devs :::{admonition} Best Practices :class: tip * **Keep `__init__` Light:** Do not open heavy resources (cameras, models) in `__init__`. Use `custom_on_configure` or `custom_on_activate`. This allows your node to be introspected and configured *before* it starts consuming resources. * **Always Report Status:** Make it a habit to call `self.health_status.set_healthy()` at the end of a successful `_execution_step`. This acts as a heartbeat for the system. * **Catch, Don't Crash:** Wrap your main logic in `try/except` blocks. Instead of raising an exception, catch it and report `set_fail_algorithm`, for example. This keeps the process alive and allows your [Fallbacks](status-and-fallbacks.md) to kick in and save the day. ::: ``` ## File: concepts/topics.md ```markdown # Topics **The connective tissue of your system.** Topics are defined in EMOS with a `Topic` class that specifies the **Data Contract** (Type/Name of the ROS2 topic), the **Behavior** (QoS), and the **Freshness Constraints** (Timeout) for a specific stream of information. Topics act as the bridge for both: 1. **Component I/O:** They define what data a Component produces or consumes. 2. **Event Triggers:** They act as the "Sensors" for the Event-Driven system, feeding data into the Blackboard. ## Why Use EMOS Topics? - {material-regular}`link;1.5em;sd-text-primary` Declarative Wiring - No more hardcoded strings buried in your components. Define your Topics as shared resources and pass them into Components during configuration. - {material-regular}`timer;1.5em;sd-text-primary` Freshness Monitoring - An EMOS Topic can enforce a `data_timeout`. If the data is too old, the Event system knows to ignore it, preventing "Stale Data" bugs. - {material-regular}`auto_awesome;1.5em;sd-text-primary` Lazy Type Resolution - You don't need to import message classes at the top of every file. EMOS resolves types like `'OccupancyGrid'` or `'Odometry'` at runtime, keeping your code clean and decoupling dependencies. - {material-regular}`tune;1.5em;sd-text-primary` QoS Abstraction - Quality of Service profiles are configured via simple Python objects directly in your recipe. ## Usage Example ```python from ros_sugar.config import QoSConfig from ros_sugar.io import Topic qos_conf = QoSConfig( history=qos.HistoryPolicy.KEEP_LAST, queue_size=20, reliability=qos.ReliabilityPolicy.BEST_EFFORT, durability=qos.DurabilityPolicy.TRANSIENT_LOCAL ) topic = Topic(name='/local_map', msg_type='OccupancyGrid', qos_profile=qos_conf) ``` ## Advanced: Smart Type Resolution One of EMOS's most convenient features is **String-Based Type Resolution**. In standard ROS2, you must import the specific message class (`from geometry_msgs.msg import Twist`) to create a publisher or subscriber. EMOS handles this import for you dynamically. ```python from ros_sugar.io import Topic from std_msgs.msg import String # Method 1: The Standard Way (Explicit Class) # Requires 'from std_msgs.msg import String' topic_1 = Topic(name='/chatter', msg_type=String) # Method 2: The EMOS Way (String Literal) # No import required. EMOS finds 'std_msgs/msg/String' automatically. topic_2 = Topic(name='/chatter', msg_type='String') ``` :::{seealso} See the full list of automatically supported message types in the advanced types reference. ::: ## Component Integration Once defined, Topics are passed to [Components](./components.md) to automatically generate the ROS2 infrastructure. ```python from ros_sugar.core import BaseComponent from ros_sugar.io import Topic # When this component starts, it automatically creates: # - A Subscriber to '/scan' (LaserScan) # - A Publisher to '/cmd_vel' (Twist) my_node = BaseComponent( component_name="safety_controller", inputs=[Topic(name="/scan", msg_type="LaserScan")], outputs=[Topic(name="/cmd_vel", msg_type="Twist")] ) ``` ``` ## File: concepts/events-and-actions.md ```markdown # Events & Actions **Dynamic behavior switching based on real-time environmental context.** EMOS's Event-Driven architecture enables dynamic behavior switching based on real-time environmental context. This allows robots to react instantly to changes in their internal state or external environment without complex, brittle if/else chains. ## Events An Event in EMOS monitors a specific **ROS2 Topic**, and defines a triggering condition based on the incoming topic data. You can write natural Python expressions (e.g., `topic.msg.data > 5`) to define exactly when an event should trigger the associated Action(s). - {material-regular}`hub;1.5em;sd-text-primary` Compose Logic - Combine triggers using simple Pythonic syntax (`(lidar_clear) & (goal_seen)`). - {material-regular}`sync;1.5em;sd-text-primary` Fuse Data - Monitor multiple topics simultaneously via a synchronized **Blackboard** that ensures data freshness. - {material-regular}`speed;1.5em;sd-text-primary` Stay Fast - All evaluation happens asynchronously in a dedicated worker pool. Your main component loop **never blocks**. :::{admonition} Think in Behaviors :class: tip Events are designed to be read like a sentence: *"If the battery is low AND we are far from home, THEN navigate to the charging dock."* ::: :::{tip} Events can be paired with EMOS [`Action`](#actions)(s) or with any standard [ROS2 Launch Action](https://docs.ros.org/en/kilted/Tutorials/Intermediate/Launch/Using-Event-Handlers.html) ::: ### Defining Events The Event API uses a fluent, expressive syntax that allows you to access ROS2 message attributes directly via `topic.msg`. #### Basic Single-Topic Event ```python from ros_sugar.core import Event from ros_sugar.io import Topic # 1. Define the Source # `data_timeout` parameter is optional. It ensures data is considered "stale" after 0.5s battery = Topic(name="/battery_level", msg_type="Float32", data_timeout=0.5) # 2. Define the Event # Triggers when percentage drops below 20% low_batt_event = Event(battery.msg.data < 20.0) ``` #### Composed Conditions (Logic & Multi-Topic) You can combine multiple conditions using standard Python bitwise operators (`&`, `|`, `~`) to create complex behavioral triggers. Events can also span multiple different topics. EMOS automatically manages a "Blackboard" of the latest messages from all involved topics, ensuring synchronization and data "freshness". - **Example**: Trigger a "Stop" event only if an obstacle is detected AND the robot is currently in "Auto" mode. ```python from ros_sugar.core import Event from ros_sugar.io import Topic lidar_topic = Topic(name="/person_detected", msg_type="Bool", data_timeout=0.5) status_topic = Topic(name="/robot_mode", msg_type="String", data_timeout=60.0) # Complex Multi-Topic Condition emergency_stop_event = Event((lidar_topic.msg.data.is_true()) & (status_topic.msg.data == "AUTO")) ``` :::{admonition} Handling Stale Data :class: warning When combining multiple topics, data synchronization is critical. Use the `data_timeout` parameter on your `Topic` definition to ensure you never act on old sensor data. ::: ### Event Configuration Refine *when* and *how* the event triggers using these parameters: * {material-regular}`change_circle` On Change (`on_change=True`) - Triggers **only** when the condition transitions from `False` to `True` (Edge Trigger). Useful for state transitions (e.g., "Goal Reached") rather than continuous firing. * {material-regular}`all_inclusive` On Any (`Topic`) - If you pass the `Topic` object itself as the condition, the event triggers on **every received message**, regardless of content. * {material-regular}`looks_one` Handle Once (`handle_once=True`) - The event will fire exactly one time during the lifecycle of the system. Useful for initialization sequences. * {material-regular}`timer` Event Delay (`keep_event_delay=2.0`) - Prevents rapid firing (debouncing). Ignores subsequent triggers for the specified duration (in seconds). ### Supported Conditional Operators You can use standard Python operators or specific helper methods on any topic attribute to define the event triggering condition. | Operator / Method | Description | Example | | :--- | :--- | :--- | | **`==`**, **`!=`** | Equality checks. | `topic.msg.status == "IDLE"` | | **`>`**, **`>=`**, **`<`**, **`<=`** | Numeric comparisons. | `topic.msg.temperature > 75.0` | | **`.is_true()`** | Boolean True check. | `topic.msg.is_ready.is_true()` | | **`.is_false()`**, **`~`** | Boolean False check. | `topic.msg.is_ready.is_false()` or `~topic.msg.is_ready` | | **`.is_in(list)`** | Value exists in a list. | `topic.msg.mode.is_in(["AUTO", "TELEOP"])` | | **`.not_in(list)`** | Value is not in a list. | `topic.msg.id.not_in([0, 1])` | | **`.contains(val)`** | String/List contains a value. | `topic.msg.description.contains("error")` | | **`.contains_any(list)`** | List contains *at least one* of the values. | `topic.msg.error_codes.contains_any([404, 500])` | | **`.contains_all(list)`** | List contains *all* of the values. | `topic.msg.detections.labels.contains_all(["window", "desk"])` | | **`.not_contains_any(list)`** | List contains *none* of the values. | `topic.msg.active_ids.not_contains_any([99, 100])` | ### Event Usage Examples #### Automatic Adaptation (Terrain Switching) Scenario: A perception or ML node publishes a string to `/terrain_type`. We want to change the robot's gait when the terrain changes. ```{code-block} python :caption: quadruped_controller.py :linenos: from typing import Literal from ros_sugar.component import BaseComponent class QuadrupedController(BaseComponent): def __init__(self, *args, **kwargs): super().__init__(*args, **kwargs) # Some logic def switch_gait_controller(self, controller_type: Literal['stairs', 'sand', 'snow', 'gravel']): self.get_logger().info("New terrain detected! Switching gait.") # Logic to change controller parameters... ``` ```{code-block} python :caption: quadruped_controller_recipe.py :linenos: from my_pkg.components import QuadrupedController from ros_sugar.core import Event, Action from ros_sugar.io import Topic from ros_sugar import Launcher quad_controller = QuadrupedController(component_name="quadruped_controller") # Define the Event Topic terrain_topic = Topic(name="/terrain_type", msg_type="String") # Define the Event # Logic: Trigger when the detected terrain changes # on_change=True ensures we only trigger the switch the FIRST time stairs are seen. # Add an optional delay to prevent rapid event triggering event_terrain_changed = Event(terrain_topic, on_change=True, keep_event_delay=60.0) # Define the Action # Call self.switch_gait_controller() when triggered and pass the detected terrain to the method change_gait_action = Action(method=self.activate_stairs_controller, args=(terrain_topic.msg.data)) # Register my_launcher = Launcher() my_launcher.add_pkg( components=[quad_controller], events_actions={stairs_event: change_gait_action}, ) ``` #### Autonomous Drone Safety Scenario: An autonomous drone **stops** if an obstacle is close OR the bumper is hit. It also sends a warning if the battery is low AND we are far from the land. ```python from ros_sugar.core import Event, Action from ros_sugar.io import Topic # --- Topics --- proximity_sensor = Topic(name="/radar_front", msg_type="Float32", data_timeout=0.2) bumper = Topic(name="/bumper", msg_type="Bool", data_timeout=0.1) battery = Topic(name="/battery", msg_type="Float32") location = Topic(name="/pose", msg_type="Pose") # --- Conditions --- # 1. Safety Condition (Composite OR) # Stop if proximity_sensor < 0.2m OR Bumper is Hit is_danger = (proximity_sensor.msg.data < 0.2) | (bumper.msg.data.is_true()) # 2. Return Home Condition (Composite AND) # Return if Battery < 20% AND Distance > 100m needs_return = (battery.msg.data < 20.0) & (location.position.z > 100.0) # --- Events --- safety_event = Event(is_danger) return_event = Event(needs_return, on_change=True) ``` --- ## Events from Internal State (a.k.a Generic Events) Most events watch a topic. Sometimes you need an event whose firing condition is **internal state** that is not being published as a ROS topic -- for example a hardware monitor or a compound signal that requires arbitrary calculation. For these cases, an `Event` accepts a third kind of condition alongside topic predicates: **any Python callable that returns `bool`**, polled at a configurable rate. ```python from ros_sugar.event import Event from ros_sugar.actions import log def is_overheating() -> bool: # must be type-annotated as bool return read_temperature() > 75.0 # any internal state goes here # Fires when ``is_overheating()`` returns True; polled twice a second. event_overheat = Event(is_overheating, check_rate=2.0) events_actions = { event_overheat: log(msg="Overheating -- backing off"), } ``` Two rules: the callable's return type annotation **must be `bool`**, and the callable **cannot be a `@component_action`** method bound to a managed component. Use a plain function or a regular instance method. The polling loop runs on the central Monitor; `check_rate` is in Hz and defaults to the Monitor's loop rate when omitted. ```{seealso} For a complete worked recipe -- including how to feed the predicate's state from elsewhere in the graph and what the event looks like in the Web UI -- see [Internal-State Events](../recipes/events-and-resilience/internal-state-events.md). ``` Use this when: - The trigger condition is **internal state** that has no business being a topic. - You want **derived predicates** that span multiple sources, easier to express as a Python callable that makes arbitrary calculations on them rather than as a chain of topic conditions. - You want **encapsulation** -- keep the predicate state inside the recipe rather than pushing it onto a topic just so an event can watch it. --- ## Actions **Executable context-aware behaviors for your robotic system.** Actions are not just static function calls; they are **dynamic, context-aware routines** that can adapt their parameters in real-time based on live system data. They can represent: - {material-regular}`smart_toy;1.2em;sd-text-primary` Component Behaviors — Routines defined within your components. *e.g., Stopping the robot, executing a motion pattern, or saying a sentence.* - {material-regular}`settings;1.2em;sd-text-primary` System Behaviors — Lifecycle management, configuration and plumbing. *e.g., Reconfiguring a node, restarting a driver, or re-routing input streams.* - {material-regular}`extension;1.2em;sd-text-primary` User Custom Behaviors — Arbitrary Python functions. *e.g., Calling an external REST API, logging to a file, or sending a slack notification.* ### Trigger Mechanisms Actions sit dormant until activated by one of two mechanisms: - {material-regular}`flash_on;1.2em;sd-text-primary` Event-Driven (Reflexive) - Triggered instantly when a specific **Event** condition is met. **Example:** "Obstacle Detected" $\rightarrow$ `stop_robot()` - {material-regular}`healing;1.2em;sd-text-primary` Fallback-Driven (Restorative) - Triggered automatically by a Component when its internal **Health Status** degrades. **Example:** "Camera Driver Failed" $\rightarrow$ `restart_driver()` ### The `Action` Class At its core, the `Action` class is a wrapper around any Python callable. It packages a function along with its arguments, preparing them for execution at runtime. But unlike standard Python functions, EMOS Actions possess a superpower: [Dynamic Data Injection](#dynamic-data-injection). You can bind their arguments directly to live ROS2 Topics, allowing the Action to fetch the latest topic message or a specific message argument the moment it triggers. ```python class Action: def __init__(self, method: Callable, args: tuple = (), kwargs: Optional[Dict] = None): ``` - `method`: The function or routine to execute. - `args`: Positional arguments (can be static values OR dynamic Topic values). - `kwargs`: Keyword arguments (can be static values OR dynamic Topic values). ### Basic Usage ```python from ros_sugar.component import BaseComponent from ros_sugar.core import Action import logging def custom_routine(): logging.info("I am executing an action!") my_component = BaseComponent(node_name='test_component') # 1. Component Method action1 = Action(method=my_component.start) # 2. Method with keyword arguments action2 = Action(method=my_component.update_parameter, kwargs={"param_name": "fallback_rate", "new_value": 1000}) # 3. External Function action3 = Action(method=custom_routine) ``` ### Dynamic Data Injection **This is EMOS's superpower.** You can create complex, context-aware behaviors without writing any "glue code" or custom parsers. When you bind an Action argument to a `Topic`, the system automatically resolves the binding at runtime, fetching the current value from the topic attributes and injecting it into your function. #### Example: Cross-Topic Data Access **Scenario**: An event occurs on Topic 1. You want to log a message that includes the current status from Topic 2 and a sensor reading from Topic 3. ```python from ros_sugar.core import Event, Action from ros_sugar.io import Topic # 1. Define Topics topic_1 = Topic(name="system_alarm", msg_type="Bool") topic_2 = Topic(name="robot_mode", msg_type="String") topic_3 = Topic(name="battery_voltage", msg_type="Float32") # 2. Define the Event # Trigger when Topic 1 becomes True event_on_first_topic = Event(topic_1.msg.data.is_true()) # 3. Define the Target Function def log_context_message(mode, voltage): print(f"System Alarm! Current Mode: {mode}, Voltage: {voltage}V") # 4. Define the Dynamic Action # We bind the function arguments directly to the data fields of Topic 2 and Topic 3 my_action = Action( method=log_context_message, # At runtime, these are replaced by the actual values from the topics args=(topic_2.msg.data, topic_3.msg.data) ) ``` ### Pre-defined Actions EMOS provides a suite of pre-defined, thread-safe actions for managing components and system resources via the `ros_sugar.actions` module. :::{admonition} Import Note :class: tip All pre-defined actions are **keyword-only** arguments. They can be imported directly: `from ros_sugar.actions import start, stop, reconfigure` ::: #### Component-Level Actions These actions directly manipulate the state or configuration of a specific `BaseComponent` derived object. | Action Method | Arguments | Description | | :-------------------------------------- | :------------------------------------------------------------ | :------------------------------------------------------------------------------------------------------------------------------------------ | | **`start`** | `component` | Triggers the component's Lifecycle transition to **Active**. | | **`stop`** | `component` | Triggers the component's Lifecycle transition to **Inactive**. | | **`restart`** | `component`
`wait_time` (opt) | Stops the component, waits `wait_time` seconds (default 0), and Starts it again. | | **`reconfigure`** | `component`
`new_config`
`keep_alive` | Reloads the component with a new configuration object or file path.
`keep_alive=True` (default) keeps the node running during update. | | **`update_parameter`** | `component`
`param_name`
`new_value`
`keep_alive` | Updates a **single** configuration parameter. | | **`update_parameters`** | `component`
`params_names`
`new_values`
`keep_alive` | Updates **multiple** configuration parameters simultaneously. | | **`send_component_service_request`** | `component`
`srv_request_msg` | Sends a request to the component's main service with a specific message. | | **`trigger_component_service`** | `component` | Triggers the component's main service.
Creates the request message dynamically during runtime from the incoming Event topic data. | | **`send_component_action_server_goal`** | `component`
`request_msg` | Sends a goal to the component's main action server with a specific message. | | **`trigger_component_action_server`** | `component` | Triggers the component's main action server.
Creates the request message dynamically during runtime from the incoming Event topic data. | #### System-Level Actions These actions interact with the broader ROS2 system and are executed by the central `Monitor`. | Action Method | Arguments | Description | | :-------------------------- | :---------------------------------------------- | :----------------------------------------------------------------------- | | **`log`** | `msg`
`logger_name` (opt) | Logs a message to the ROS console. | | **`publish_message`** | `topic`
`msg`
`publish_rate`/`period` | Publishes a specific message to a topic. Can be single-shot or periodic. | | **`send_srv_request`** | `srv_name`
`srv_type`
`srv_request_msg` | Sends a request to a ROS 2 Service with a specific message. | | **`trigger_service`** | `srv_name`
`srv_type` | Triggers the a given ROS2 service. | | **`send_action_goal`** | `server_name`
`server_type`
`request_msg` | Sends a specific goal to a ROS 2 Action Server. | | **`trigger_action_server`** | `server_name`
`server_type` | Triggers a given ROS2 action server. | :::{admonition} Automatic Data Conversion :class: note When using **`trigger_*`** actions paired with an Event, EMOS attempts to create the required service/action request from the incoming Event topic data automatically via **duck typing**. If automatic conversion is not possible, or if the action is not paired with an Event, it sends a default (empty) request. ::: ``` ## File: concepts/status-and-fallbacks.md ```markdown # Status & Fallbacks **All robots can fail, but smart robots recover.** EMOS components are **Self-Aware** and **Self-Healing** by design. The Health Status system allows every component to explicitly declare its operational state --- not just "Alive" or "Dead," but _how_ it is functioning. When failures are detected, the Fallback system automatically triggers pre-configured recovery strategies, keeping the robot operational without human intervention. --- ## Health Status The **Health Status** is the heartbeat of an EMOS component. Unlike standard ROS2 nodes, EMOS components differentiate between a math error (Algorithm Failure), a hardware crash (Component Failure), or a missing input (System Failure). These reports are broadcast back to the system to trigger: - {material-regular}`notifications;1.2em;sd-text-warning` **Alerts:** Notify the operator of specific issues. - {material-regular}`flash_on;1.2em;sd-text-primary` **Reflexes:** Trigger [Events](events-and-actions.md) to handle the situation. - {material-regular}`healing;1.2em;sd-text-success` **Self-Healing:** Execute automatic [Fallbacks](#fallback-strategies) to recover the node. ### Status Hierarchy EMOS defines distinct failure levels to help you pinpoint the root cause of an issue. - {material-regular}`check_circle;1.5em;sd-text-success` HEALTHY **"Everything is awesome."** The component executed its main loop successfully and produced valid output. - {material-regular}`warning;1.5em;sd-text-warning` ALGORITHM_FAILURE **"I ran, but I couldn't solve it."** The node is healthy, but the logic failed. _Examples:_ Path planner couldn't find a path; Object detector found nothing; Optimization solver did not converge. - {material-regular}`error;1.5em;sd-text-danger` COMPONENT_FAILURE **"I am broken."** An internal crash or hardware issue occurred within this specific node. _Examples:_ Memory leak; Exception raised in a callback; Division by zero. - {material-regular}`link_off;1.5em;sd-text-primary` SYSTEM_FAILURE **"I am fine, but my inputs are broken."** The failure is caused by an external dependency. _Examples:_ Input topic is empty or stale; Network is down; Disk is full. ### Reporting Status Every `BaseComponent` has an internal `self.health_status` object. You interact with this object inside your `_execution_step` or callbacks to declare the current state. #### The Happy Path Always mark the component as healthy at the end of a successful execution. This resets any previous error counters. ```python self.health_status.set_healthy() ``` #### Declaring Failures When things go wrong, be specific. This helps the Fallback system decide whether to _Retry_ (Algorithm), _Restart_ (Component), or _Wait_ (System). **Algorithm Failure:** ```python # Optional: List the specific algorithm that failed self.health_status.set_fail_algorithm(algorithm_names=["A_Star_Planner"]) ``` **Component Failure:** ```python # Report that this component crashed self.health_status.set_fail_component() # Or blame a sub-module self.health_status.set_fail_component(component_names=["Camera_Driver_API"]) ``` **System Failure:** ```python # Report missing data on specific topics self.health_status.set_fail_system(topic_names=["/camera/rgb", "/odom"]) ``` ### Automatic Broadcasting You do not need to manually publish the status message. EMOS automatically broadcasts the status at the start of every execution step. This ensures a consistent "Heartbeat" frequency, even if your algorithm blocks or hangs (up to the threading limits). :::{tip} If you need to trigger an immediate alert from a deeply nested callback or a separate thread, you _can_ force a publish: `self.health_status_publisher.publish(self.health_status())` ::: ### Implementation Pattern Here is the robust pattern for writing an execution step using Health Status. This pattern enables the **Self-Healing** capabilities of EMOS. ```python def _execution_step(self): try: # 1. Check Pre-conditions (System Level) if self.input_image is None: self.get_logger().warn("Waiting for video stream...") self.health_status.set_fail_system(topic_names=[self.input_image.name]) return # 2. Run Logic result = self.ai_model.detect(self.input_image) # 3. Check Logic Output (Algorithm Level) if result is None or len(result.detections) == 0: self.health_status.set_fail_algorithm(algorithm_names=["yolo_detector"]) return # 4. Success! self.publish_result(result) self.health_status.set_healthy() except ConnectionError: # 5. Handle Crashes (Component Level) # This will trigger the 'on_component_fail' fallback (e.g., Restart) self.get_logger().error("Camera hardware disconnected!") self.health_status.set_fail_component(component_names=["hardware_interface"]) ``` --- ## Fallback Strategies Fallbacks are the **Self-Healing Mechanism** of an EMOS component. They define the specific set of [Actions](events-and-actions.md#actions) to execute automatically when a failure is detected in the component's Health Status. Instead of crashing or freezing when an error occurs, a Component can be configured to attempt intelligent recovery strategies: - {material-regular}`swap_horiz;1.2em;sd-text-warning` _Algorithm stuck?_ $\rightarrow$ **Switch** to a simpler backup. - {material-regular}`restart_alt;1.2em;sd-text-danger` _Driver disconnected?_ $\rightarrow$ **Re-initialize** the hardware. - {material-regular}`autorenew;1.2em;sd-text-primary` _Sensor timeout?_ $\rightarrow$ **Restart** the node. ```{figure} /_static/images/diagrams/fallbacks_dark.png :class: dark-only :alt: fig-fallbacks :align: center ``` ```{figure} /_static/images/diagrams/fallbacks_light.png :class: light-only :alt: fig-fallbacks :align: center The Self-Healing Loop ``` ### The Recovery Hierarchy When a component reports a failure, EMOS doesn't just panic. It checks for a registered fallback strategy in a specific order of priority. This allows you to define granular responses for different types of errors. - {material-regular}`link_off;1.5em;sd-text-primary` 1. System Failure `on_system_fail` **The Context is Broken.** External failures like missing input topics or disk full. _Example Strategy:_ Wait for data, or restart the data pipeline. - {material-regular}`error;1.5em;sd-text-danger` 2. Component Failure `on_component_fail` **The Node is Broken.** Internal crashes or hardware disconnects. _Example Strategy:_ Restart the component lifecycle or re-initialize drivers. - {material-regular}`warning;1.5em;sd-text-warning` 3. Algorithm Failure `on_algorithm_fail` **The Logic is Broken.** The code ran but couldn't solve the problem (e.g., path not found). _Example Strategy:_ Reconfigure parameters (looser tolerance) or switch algorithms. - {material-regular}`help_center;1.5em;sd-text-secondary` 4. Catch-All `on_fail` **Generic Safety Net.** If no specific handler is found above, this fallback is executed. _Example Strategy:_ Log an error or stop the robot. ### Recovery Strategies A Fallback isn't just a single function call. It is a robust policy defined by **Actions** and **Retries**. #### The Persistent Retry (Single Action) _Try, try again._ The system executes the action repeatedly until it returns `True` (success) or `max_retries` is reached. ```python # Try to restart the driver up to 3 times driver.on_component_fail(fallback=restart(component=driver), max_retries=3) ``` #### The Escalation Ladder (List of Actions) _If at first you don't succeed, try something stronger._ You can define a sequence of actions. If the first one fails (after its retries), the system moves to the next one. 1. **Clear Costmaps** (Low cost, fast) 2. **Reconfigure Planner** (Medium cost) 3. **Restart Planner Node** (High cost, slow) ```python # Tiered Recovery for a Navigation Planner planner.on_algorithm_fail( fallback=[ Action(method=planner.clear_costmaps), # Step 1 Action(method=planner.switch_to_fallback), # Step 2 restart(component=planner) # Step 3 ], max_retries=1 # Try each step once before escalating ) ``` #### The "Give Up" State If all strategies fail (all retries of all actions exhausted), the component enters the **Give Up** state and executes the `on_giveup` action. This is the "End of Line", usually used to park the robot safely or alert a human. ### How to Implement Fallbacks #### Method A: In Your Recipe (Recommended) You can configure fallbacks externally without touching the component code. This makes your system modular and reusable. ```python from ros_sugar.actions import restart, log # 1. Define component lidar = BaseComponent(component_name='lidar_driver') # 2. Attach Fallbacks # If it crashes, restart it (Unlimited retries) lidar.on_component_fail(fallback=restart(component=lidar)) # If data is missing (System), just log it and wait lidar.on_system_fail(fallback=log(msg="Waiting for Lidar data...")) # If all else fails, scream lidar.on_giveup(fallback=log(msg="LIDAR IS DEAD. STOPPING ROBOT.")) ``` #### Method B: In Component Class (Advanced) For tightly coupled recovery logic (like re-handshaking a specific serial protocol), you can define custom fallback methods inside your class. :::{tip} Use the `@component_fallback` decorator. It ensures the method is only called when the component is in a valid state to handle it. ::: ```python from ros_sugar.core import BaseComponent, component_fallback from ros_sugar.core import Action class MyDriver(BaseComponent): def __init__(self, *args, **kwargs): super().__init__(*args, **kwargs) # Register the custom fallback internally self.on_system_fail( fallback=Action(self.try_reconnect), max_retries=3 ) def _execution_step(self): try: self.hw.read() self.health_status.set_healthy() except ConnectionError: # This trigger starts the fallback loop! self.health_status.set_fail_system() @component_fallback def try_reconnect(self) -> bool: """Custom recovery logic""" self.get_logger().info("Attempting handshake...") if self.hw.connect(): return True # Recovery Succeeded! return False # Recovery Failed, will retry... ``` --- ## Process-Level Recovery The Health-Status / Fallback system above operates **inside a running component**. If a component goes further than that and the _entire process_ crashes -- a segfault in a native dependency, a Python `os._exit`, an OOM kill -- there is nothing left running to dispatch a fallback. EMOS adds a layer underneath for exactly this case. ### Respawning crashed processes -- `Launcher.on_process_fail` `Launcher.on_process_fail()` enables process-level respawning for every component the launcher started in multiprocessing mode. If a component process exits with a non-zero status outside of normal shutdown, the launcher relaunches it. ```python launcher = Launcher() launcher.add_pkg(components=[...], multiprocessing=True, package_name="...") launcher.on_process_fail(max_retries=3) # respawn up to 3 times per component launcher.bringup() ``` User-initiated shutdowns (Ctrl-C, SIGTERM) do not count as failures, so this won't fight you when you stop the recipe yourself. ```{tip} Pair `on_process_fail` with `executor_spin_timeout` (see [Launcher](launcher.md)) when running latency-sensitive callbacks -- the latter prevents a slow callback from looking like a stalled process and tripping the respawn unnecessarily. ``` ``` ## File: concepts/launcher.md ```markdown # Launcher & Orchestration **Recipes: One script to rule them all.** The `Launcher` is your entry point to the EMOS ecosystem. It provides a clean, Pythonic API to configure, spawn, and orchestrate your ROS2 nodes without writing XML or complex launch files. Under the hood, every Launcher spawns an internal **Monitor** node. This hidden "Brain" is responsible for tracking component health, listening for events, and executing the orchestration logic. ## Execution Architectures The Launcher supports two execution modes, configured via the `multi_processing` flag. ::::{tab-set} :::{tab-item} Multi-Threaded :sync: threaded **Default for Debugging** (`multi_processing=False`) All components run in the same process as the Launcher and Monitor. * **Pros:** Fast startup, shared memory, easy debugging (breakpoints work everywhere). * **Cons:** The Global Interpreter Lock (GIL) can bottleneck performance if you have many heavy nodes. ```{figure} /_static/images/diagrams/multi_threaded_dark.png :class: dark-only :alt: multi-threaded architecture :align: center ``` ```{figure} /_static/images/diagrams/multi_threaded_light.png :class: light-only :alt: multi-threaded architecture :align: center Multi-threaded Execution ``` ::: :::{tab-item} Multi-Process :sync: process **Production Mode** (`multi_processing=True`) Each component runs in its own isolated process. The Monitor still runs in the same process as the Launcher. * **Pros:** True parallelism, crash isolation (one node crashing doesn't kill the system). * **Cons:** Higher startup overhead. ```{figure} /_static/images/diagrams/multi_process_dark.png :class: dark-only :alt: multi-process architecture :align: center ``` ```{figure} /_static/images/diagrams/multi_process_light.png :class: light-only :alt: multi-process architecture :align: center Multi-process Execution ``` ::: :::: ## Launcher Features ### 1. Package & Component Loading You can add components from your current script or external packages. ```python # Add from an external entry point (for multi-process separation) launcher.add_pkg( package_name="my_robot_pkg", components=[vision_component] # Pass config/events here multiprocessing=True ) ``` ### 2. Lifecycle Management EMOS components are Lifecycle nodes. The Launcher handles the transition state machine for you. * `activate_all_components_on_start=True`: Automatically transitions all nodes to **Active** after spawning. ### 3. Global Fallbacks Define "Catch-All" policies for the entire system. ```python # If ANY component reports a crash, restart it. launcher.on_component_fail(action_name="restart") ``` For the further case where an entire **process** crashes (segfault, OOM, native exception) and there is nothing left in-process to dispatch a Fallback, the launcher provides `Launcher.on_process_fail()`. The launcher relaunches any component process that exits with a non-zero status outside of normal shutdown: ```python launcher.on_process_fail(max_retries=3) # respawn up to 3 times per component ``` See [Process-Level Recovery](status-and-fallbacks.md#process-level-recovery) for the full picture. ### 4. Executor Spin Timeout ```python launcher = Launcher( multi_processing=True, executor_spin_timeout=0.05, # seconds ) ``` Each component runs a ROS2 executor that spins its callbacks. The executor blocks for at most this many seconds per spin call regardless of how fast the component's main loop is running. Lower values reduce callback latency at the cost of CPU; the default works for most recipes. Tune this when running latency-sensitive callbacks alongside `on_process_fail` to avoid mistaking a slow callback for a stalled process. ### 5. Events Orchestration Pass your events/actions dictionary **once** to the `Launcher` and it will handle delegating the event monitoring to the concerned component. ## Complete Usage Example ```python from ros_sugar.core import BaseComponent, Event, Action from ros_sugar.actions import log, restart from ros_sugar.io import Topic from ros_sugar import Launcher # 1. Define Components # (Usually imported from your package) driver = BaseComponent(component_name='lidar_driver') planner = BaseComponent(component_name='path_planner') # Set Fallback Policy # If the driver crashes, try to restart it automatically driver.on_component_fail(fallback=restart(component=driver)) # 2. Define Logic for Events battery = Topic(name="/battery", msg_type="Float32") low_batt_evt = Event(battery.msg.data < 15.0) log_action = log(msg="WARNING: Battery Low!") # 3. Initialize Launcher launcher = Launcher( config_file='config/robot_params.toml', # Can optionally pass a configuration file activate_all_components_on_start=True, multi_processing=True # Use separate processes ) # 4. Register Components # You can attach specific events to specific groups of components launcher.add_pkg( components=[driver, planner], ros_log_level="error", events_actions={low_batt_evt: log_action} ) # 5. Launch! # This blocks until Ctrl+C is pressed launcher.bringup() ``` ## The Monitor (Internal Engine) :::{note} The Monitor is configured automatically. You do not need to instantiate or manage it manually. ::: The **Monitor** is a specialized, non-lifecycle ROS2 node that acts as the central management node. **Responsibilities:** 1. {material-regular}`play_arrow;1.2em;sd-text-primary` **Custom Actions Execution:** Handles executing custom Actions defined in the recipe. 2. {material-regular}`monitor_heart;1.2em;sd-text-primary` **Health Tracking:** Subscribes to the `/status` topic of every component. 3. {material-regular}`hub;1.2em;sd-text-primary` **Orchestration:** Holds clients for every component's Lifecycle and Parameter services, allowing it to restart, reconfigure, or stop nodes on demand. **Architecture:** ::::{tab-set} :::{tab-item} Configuration :sync: config How the Launcher configures the Monitor with Events and Actions at startup. ```{figure} /_static/images/diagrams/events_actions_config_dark.png :class: dark-only :alt: Monitoring events diagram :align: center :scale: 70 ``` ```{figure} /_static/images/diagrams/events_actions_config_light.png :class: light-only :alt: Monitoring events diagram :align: center :scale: 70 Monitoring events ``` ::: :::{tab-item} Execution :sync: exec How the Monitor processes triggers and executes actions at runtime. ```{figure} /_static/images/diagrams/events_actions_exec_dark.png :class: dark-only :alt: An Event Trigger diagram :align: center :scale: 70 ``` ```{figure} /_static/images/diagrams/events_actions_exec_light.png :class: light-only :alt: An Event Trigger diagram :align: center :scale: 70 An Event Trigger ``` ::: :::: ``` ## File: concepts/robot-plugins.md ```markdown # Robot Plugins EMOS is built to be **robot-agnostic** -- recipes are written against standard, hardware-independent interfaces, so the same behaviour runs on any robot. In practice, though, robot manufacturers often expose their hardware through **custom interfaces**: bespoke ROS 2 messages and services, or entirely non-ROS protocols like UDP, HTTP, or a vendor SDK. {material-regular}`extension;1.2em;sd-text-primary` **Robot Plugins exist to bridge that gap.** A plugin adapts a manufacturer's custom interface to EMOS's standard one, making it trivial to bring a new robot online without touching your recipes. A plugin can also ship the robot's **preconfigured actions** -- a gait change on a quadruped, a docking routine, an arm stow -- ready for recipes (and [Cortex](../intelligence/cortex.md)) to call by name. --- ## What Are Robot Plugins? A Robot Plugin is the **translation layer** between an EMOS recipe and a specific robot. It maps the robot's real interfaces onto the standard component I/O that recipes use (`Twist`, `Odometry`, `Imu`, …), so your recipe code never changes when the hardware does -- and it surfaces the robot's high-level **actions** (`stand_up`, `dock`, gait change) and **events** (`low_battery`) for recipes and Cortex to consume directly. ```{note} This page describes the current plugin framework (Sugarcoat 0.7+): a plugin is a Python **class** you pass to the `Launcher`. It supersedes the earlier dictionary-based `robot_feedback` / `robot_action` API. ``` --- ## Why Robot Plugins? - {material-regular}`swap_horiz;1.2em;sd-text-primary` **Portability** -- Write the recipe once against standard types; switch robots by swapping a single plugin object. - {material-regular}`auto_fix_high;1.2em;sd-text-primary` **Simplicity** -- The plugin hides all the type conversions, sockets, and service calls behind the scenes. - {material-regular}`widgets;1.2em;sd-text-primary` **Modularity** -- Keep hardware-specific logic isolated in its own package. --- ## How a Plugin Works A plugin is a subclass of `ros_sugar.robot.RobotPlugin`. Its `__init__` is **declarative** -- it only *describes* the robot (endpoints, transports, decoders) and does no I/O, so it can be serialized and rebuilt inside component subprocesses. The `Launcher` then runs it in one of two roles automatically: - {material-regular}`dns;1.2em;sd-text-primary` **HOST** -- lives in the launcher process. Owns the real transports (binds sockets, opens sessions, runs heartbeats), decodes telemetry once, and publishes it on a feedback bus. - {material-regular}`memory;1.2em;sd-text-primary` **CLIENT** -- lives in each component subprocess. Opens no sockets; it consumes decoded feedback from the bus and sends commands. During activation, every component's standard input/output topics are matched against the plugin: a ROS-topic match re-points the native subscriber/publisher, and any other transport is bridged through the feedback bus. Components stay unaware their data isn't plain ROS. --- ## Anatomy of a Plugin | Building block | Role | | :------------- | :--- | | `RobotPlugin` | The plugin itself -- subclass this. | | `Transport` | Where data comes from / goes to: `UdpTransport`, `HttpTransport`, `SdkCallbackTransport`, `RosTopicTransport`, `RosServiceTransport`. | | `Feedback` | One telemetry stream -- a standard type, a transport, and a decoder (raw payload → ROS message). | | `RobotCommand` | One command surface -- a standard type, a transport, and an encoder (component output → wire payload). | | `ActionRegistry` / `EventRegistry` | Named factories that produce the robot's `Action` / `Event` objects. | | `create_supported_type` | Wraps a robot's custom ROS message as a standard `SupportedType`. | `Feedback` and `RobotCommand` are keyed by the **standard message-type name** they stand in for (`Twist`, `Odometry`, …) -- that is how the framework matches them to component I/O. --- ## Using a Plugin in a Recipe Hand a plugin instance to the `Launcher` -- that is the only recipe change: ```python from ros_sugar.launch import Launcher from myrobot_plugin import MyRobotPlugin plugin = MyRobotPlugin() # declarative, zero-arg launcher = Launcher(robot_plugin=plugin) launcher.add_pkg(components=[planner, controller], multiprocessing=True) # Plugin-provided events and actions wire up like anything else launcher.on(plugin.events.low_battery(0.15), plugin.actions.dock()) launcher.bringup() ``` EMOS hosts the plugin, propagates it to every component, and translates all topic I/O through it -- every subscription and publication in your recipe is routed via the plugin automatically. --- ## Installing a Plugin You don't have to build a plugin to use one -- the Automatika catalog ships ready-made plugins you install with one command (or one click in the dashboard): ```bash emos plugin install emos-plugin-example ``` See [Robot Plugins (install & manage)](../getting-started/plugins.md) for the full flow. --- ## Writing Your Own The reference plugin, [`emos-plugin-example`](https://github.com/automatika-robotics/emos-plugin-example), implements one robot across **all** the transport families (UDP telemetry + velocity command, a ROS-topic battery feedback, and a ROS-service docking action), with a mock robot and an end-to-end test suite. Copy it and adapt. The full step-by-step authoring guide -- wrapping custom message types, defining transports/feedbacks/commands, contributing actions and events, and introspection -- lives in the Sugarcoat docs: [**Creating a Robot Plugin**](https://github.com/automatika-robotics/sugarcoat/blob/main/docs/development/custom_robot_plugin.md). You can introspect any plugin's exposed surface from the command line: ```bash python -m ros_sugar.robot inspect myrobot_plugin:MyRobotPlugin ``` (`emos plugin inspect` prints the same tree for the active plugin.) --- ```{seealso} - [Robot Plugins (install & manage)](../getting-started/plugins.md) -- install a plugin from the catalog with `emos plugin`. - [Creating a Robot Plugin](https://github.com/automatika-robotics/sugarcoat/blob/main/docs/development/custom_robot_plugin.md) -- the full authoring guide. - [Extending EMOS](../advanced/extending.md) -- custom components and deploying them as system services. ``` ``` ## File: intelligence/overview.md ```markdown # EmbodiedAgents **The intelligence layer of EMOS --** production-grade orchestration for Physical AI [EmbodiedAgents](https://github.com/automatika-robotics/embodied-agents) enables you to create interactive, physical agents that do not just chat, but **understand**, **move**, **manipulate**, and **adapt** to their environment. It bridges the gap between foundation AI models and real-world robotic deployment, offering a structured yet flexible programming model for building adaptive intelligence. - {material-regular}`smart_toy;1.2em;sd-text-primary` Production-Ready Physical Agents -- Designed for autonomous systems in dynamic, real-world environments. Components are built around ROS2 Lifecycle Nodes with deterministic startup, shutdown, and error-recovery. Health monitoring, fallback behaviors, and graceful degradation are built in from the ground up. - {material-regular}`autorenew;1.2em;sd-text-primary` Self-Referential and Event-Driven -- Agents can start, stop, or reconfigure their own components based on internal and external events. Switch from cloud to local inference, swap planners based on vision input, or adjust behavior on the fly. In the spirit of [Godel machines](https://en.wikipedia.org/wiki/G%C3%B6del_machine), agents become capable of introspecting and modifying their own execution graph at runtime. - {material-regular}`hub;1.2em;sd-text-primary` Semantic Memory & Agentic Planning -- Hierarchical spatio-temporal memory and semantic routing for arbitrarily complex agentic information flow. The graph-backed [Memory](memory.md) component keeps an episodic, entity-aware record of what the robot perceives *and* of its own internal state, while [Cortex](cortex.md) turns plain-language goals into ordered calls against every component in the graph -- no bloated GenAI frameworks required. - {material-regular}`code;1.2em;sd-text-primary` Pure Python, Native ROS2 -- Define complex asynchronous execution graphs in standard Python without touching XML launch files. Underneath, everything is pure ROS2 -- fully compatible with the entire ecosystem of hardware drivers, simulation tools, and visualization suites. ## What You Can Build ::::{grid} 1 2 3 3 :gutter: 3 :::{grid-item-card} {material-regular}`record_voice_over;1.2em;sd-text-primary` Robots You Hold a Conversation With :link: ../recipes/foundation/conversational-agent :link-type: doc Robots that listen, see, and speak -- microphone in, visually-grounded answer out, all in one Python recipe. Ask *"what's on the table?"* and get a real answer in real time. ::: :::{grid-item-card} {material-regular}`alt_route;1.2em;sd-text-primary` Robots That Pick the Right Brain :link: ../recipes/foundation/semantic-routing :link-type: doc One sentence in, the right capability fires. *"How tall is Everest?"* wakes the LLM. *"What do you see?"* wakes the VLM. *"Take me to the kitchen."* dispatches the navigation stack. Behavior emerges from intent. ::: :::{grid-item-card} {material-regular}`precision_manufacturing;1.2em;sd-text-primary` Robots That Pick Up What You Mean :link: ../recipes/planning-and-manipulation/vla-manipulation :link-type: doc A robot arm that grabs *"the red mug next to the laptop"* without you writing a state machine for which mug. A VLM grounds the description; a VLA model translates straight to joint commands. ::: :::{grid-item-card} {material-regular}`memory;1.2em;sd-text-primary` Robots That Remember :link: ../recipes/foundation/semantic-map :link-type: doc Every detection, every scene caption, every internal reading is folded into a graph indexed by *meaning*, *place*, and *time* -- and persists across reboots. The robot starts knowing your space the way you do. ::: :::{grid-item-card} {material-regular}`smart_toy;1.2em;sd-text-primary` Robots You Give Missions To :link: ../recipes/planning-and-manipulation/cortex-agent :link-type: doc Drop a [Cortex](cortex.md) component into your recipe and your robot starts running *missions*, not commands. *"Patrol the workshop and tell me if any lights are off."* Cortex auto-discovers every capability as an LLM tool, plans the steps, dispatches them, watches feedback, and replans on failure -- with no orchestration code from you. ::: :::{grid-item-card} {material-regular}`route;1.2em;sd-text-primary` Robots That Reason About the World :link: ../recipes/planning-and-manipulation/cortex-navigation :link-type: doc Compound goals like *"go to the kitchen and tell me what's on the counter"* fall out of a single recipe. The robot recalls where the kitchen is from memory, navigates there with Kompass, looks at the counter, narrates the answer. End-to-end embodied reasoning, no behavior trees. ::: :::: ## Next Steps - {material-regular}`widgets;1.2em;sd-text-primary` {doc}`ai-components` -- The core building blocks: components and topics. - {material-regular}`psychology;1.2em;sd-text-primary` {doc}`cortex` -- The agentic planner-executor that drives the rest of the graph from natural-language goals. - {material-regular}`memory;1.2em;sd-text-primary` {doc}`memory` -- Graph-backed spatio-temporal memory with perception and interoception layers. - {material-regular}`cloud;1.2em;sd-text-primary` {doc}`clients` -- How inference backends connect to components. - {material-regular}`model_training;1.2em;sd-text-primary` {doc}`models` -- Available model wrappers and vector databases. ``` ## File: intelligence/ai-components.md ```markdown # AI Components A **Component** is the primary execution unit in EmbodiedAgents, the EMOS intelligence framework. Components represent functional behaviors -- for example, the ability to process text, understand images, or synthesize speech. Components can be combined arbitrarily to create more complex systems such as multi-modal agents with perception-action loops. Most EmbodiedAgents components are **capabilities** -- a single thing the robot can do. LLM, VLM, VLA, Vision, SpeechToText, TextToSpeech, SemanticRouter, VideoMessageMaker each wrap a particular modality or model surface; [Memory](memory.md) is a capability too, giving the robot a graph-backed spatio-temporal record of what it has seen and felt. [Cortex](cortex.md) is the one component that doesn't sit in that family: it's a high-level planner-executor that *uses* the capabilities, turning natural-language goals into ordered calls against the available component capabilities. ```{note} To learn more about the internal structure and lifecycle behavior of components, check out the concept [here](../concepts/components.md). ``` ## Available Components EmbodiedAgents provides a suite of ready-to-use components. These can be composed into flexible execution graphs for building autonomous, perceptive, and interactive robot behavior. Each component focuses on a particular modality or functionality, from vision and speech to map reasoning and VLA-based manipulation. ```{list-table} :widths: 20 80 :header-rows: 1 * - Component Name - Description * - **LLM** - Uses large language models (e.g., LLaMA) to process text input. Can be used for reasoning, tool calling, instruction following, or dialogue. It can also utilize vector DBs for storing and retrieving contextual information. Supports built-in local LLM fallback for offline operation. * - **VLM** - Leverages multimodal LLMs (e.g., Llava) for understanding and processing both text and image data. Inherits all functionalities of the LLM component. It can also utilize multimodal LLM based planning models for task-specific outputs (e.g. pointing, grounding, affordance etc.). Supports built-in local VLM fallback for offline operation. **This component is also called MLLM**. * - **VLA** - Provides an interface to utilize Vision Language Action (VLA) models for manipulation and control tasks. It can use VLA Policies (such as SmolVLA, Pi0 etc.) served with HuggingFace LeRobot Async Policy Server and publish them to common topic formats in MoveIt Servo and ROS2 Control. * - **SpeechToText** - Converts spoken audio into text using speech-to-text models (e.g., Whisper). Suitable for voice command recognition. It also implements small on-board models for Voice Activity Detection (VAD) and Wakeword recognition, using audio capture devices onboard the robot. Supports built-in local STT for on-device transcription. * - **TextToSpeech** - Synthesizes audio from text using HuggingFace Transformers TTS models (Bark, VITS, SpeechT5, SeamlessM4T, etc.) via the unified `TransformersTTS` wrapper. Output audio can be played using the robot's speakers or published to a topic. Implements `say(text)` and `stop_playback` functions to play/stop audio based on events from other components or the environment. Supports built-in local TTS for on-device speech synthesis. * - **Memory** - Provides a graph-backed spatio-temporal memory powered by [eMEM](https://github.com/automatika-robotics/emem). Encodes perception layers (e.g. detections, scene captions) and interoception layers (e.g. battery, internal flags) into an episodic, entity-aware graph and exposes ten retrieval tools as component actions. Replaces the deprecated **MapEncoding** -- see the dedicated [Memory page](memory.md). * - **Cortex** - The agentic core. An AI-powered planner-executor that inspects the rest of the recipe, decomposes a natural-language goal into a sequence of component-action calls, and runs them while monitoring the outputs. See [Cortex](cortex.md). * - **SemanticRouter** - Routes information between topics based on semantic content and predefined routing rules. Uses a vector DB for semantic matching or an LLM for decision-making. This allows for creating complex graphs of components where a single input source can trigger different information processing pathways. * - **Vision** - An essential component in all vision-powered robots. Performs object detection and tracking on incoming images. Outputs object classes, bounding boxes, and confidence scores. It implements a low-latency small on-board classification model as well. Supports a built-in local ONNX classifier for on-device detection. * - **VideoMessageMaker** - Generates ROS video messages from input image messages. A video message is a collection of image messages that have a perceivable motion. The primary task of this component is to make intentionality decisions about what sequence of consecutive images should be treated as one coherent temporal sequence. The chunking method used for selecting images for a video can be configured in component config. Useful for sending videos to ML models that take image sequences. ``` ```{seealso} For details on Topics, component configuration, run types, health checks, and fallback behaviors, see the [Core Concepts](../concepts/components.md) section. ``` ``` ## File: intelligence/cortex.md ```markdown # Cortex **The agentic harness for embodied intelligence.** Cortex is the EmbodiedAgents component that turns the rest of your recipe into an agent: it discovers every component you added, registers their methods as LLM tools, and lets the user address the whole system in plain language. If [Claude Code](https://claude.com/claude-code) is an agentic harness for software engineering, Cortex is its analogue for robots -- the same primitives (read the environment, plan, dispatch tools, watch results, replan) applied to a physical system. A recipe with a Cortex stops being a programmed pipeline and starts being something you talk to. ```{seealso} For the introductory walkthrough, start with [Cortex: The Agentic Harness](../recipes/planning-and-manipulation/cortex-agent.md). For Cortex paired with spatio-temporal memory, see [Memory and Cortex](../recipes/planning-and-manipulation/cortex-memory.md). For the full multi-system showcase that adds navigation on top, see [Cortex Driving the Full Stack](../recipes/planning-and-manipulation/cortex-navigation.md). ``` --- ## What Cortex replaces A non-Cortex EMOS recipe earns each capability by hand-wiring it: a vision component publishes detections, an event matches the detection class, a fallback restarts the camera if it stalls, a separate LLM component parses the user's input into structured goals, and so on. Every link is yours to author and maintain. A Cortex recipe is the same components -- minus the wiring. Cortex inspects the running graph at activation, registers every available capability as a callable tool, and accepts the user's intent directly: | Without Cortex | With Cortex | | ------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------- | | One event-action pair per behaviour, hand-wired in the recipe. | The recipe has no behavioural wiring. Cortex plans the behaviour at runtime from the user's goal. | | Each capability needs an explicit handler that knows when to trigger it. | Each capability is a `@component_action` on its component. Cortex discovers them all on activation. | | User input is parsed by a bespoke LLM step into a typed goal. | User input is a free-form string sent to Cortex's main action server. | | Fallback policies wired per component. | Cortex's confirmation step (EXECUTE / SKIP / ABORT / CONTINUE) handles per-step recovery; replan handles plan-level recovery. | | You write the orchestration code. | You write the components. Cortex writes the recipe. | --- ## How it works Cortex runs a **two-phase loop** for every task. ### Phase 1 -- Planning (multi-step) The planner LLM is handed two tool sets: - **Planning tools** -- read-only research tools. The built-in `inspect_component` plus any `@component_action(phase=ActionPhase.PLANNING)` methods on managed components. - **Execution tools** -- everything that _does_ something. Custom `Action` objects, `@component_action(phase=ActionPhase.EXECUTION)` methods, action-server goal tools, service-request tools, and the built-in `update_parameter`. On each iteration, the LLM may: 1. Call **planning tools** to gather information -- inspect a component, query memory, look up a fact in a vector DB. Results are appended to the conversation and the loop continues. 2. Call **execution tools** -- this commits a plan as an ordered list of steps, and the loop ends. 3. Respond with **text only** -- no actions needed; the text is published on `output` and the task is done. Up to `max_planning_steps` iterations of research are allowed before the planner must commit. Plans longer than `max_execution_steps` are truncated. ### Phase 2 -- Execution with confirmation Each step is dispatched in turn. **Before** each one, a brief confirmation LLM call returns one of: | Decision | Effect | | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `EXECUTE` | Run the next step. The confirmation may also return a tool call with **resolved arguments** -- e.g. binding a placeholder like `` to the actual return value of step 1. | | `SKIP` | Skip this step and continue. | | `ABORT` | Abort the entire plan. | | `CONTINUE` | Wait for in-flight async actions to finish before deciding. | `CONTINUE` is the key to long-horizon tasks. When Cortex dispatches an action goal (e.g. to a Planner action server), it tracks the client in `_active_action_clients` and the confirmation prompt includes the action's live feedback. The LLM can `CONTINUE` to wait, watch how the goal is progressing, and only `EXECUTE` the next step when the action reports SUCCEEDED. ### Replan on incomplete execution If the plan exits before reaching its terminal step (because a step `ABORT`-ed, or because the executor ran out of steps with goals still in flight), Cortex composes a fresh plan from where it left off, with the partial-execution results fed back into the planning conversation. Long-horizon tasks ("patrol until you see a person") express naturally: each replan is one outer iteration. --- ## What gets auto-discovered When the launcher activates Cortex, it walks every component in the recipe and registers tools for everything it finds. **You write none of this registration -- it happens once on activation.** ### Built-in tools | Tool | Phase | Purpose | | ---------------------------------------------------- | --------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `inspect_component(component)` | Planning | Returns the component's full structure: input/output topics, config, additional model clients, and the actions Cortex has registered for it. The planner uses this to discover topic names and ground its plan. | | `update_parameter(component, param_name, new_value)` | Execution | Re-tunes any config parameter on any managed component at runtime. The LLM can lower `Vision.threshold` mid-task or flip `Controller.direct_sensor` if the mission demands it. | ### Component capabilities For every managed component, Cortex discovers: - **`@component_action` methods** -- registered as namespaced tools `{component_name}.{method_name}` with the OpenAI-format description from the decorator. Classified as planning, execution, or both via the `phase=` argument: ```python from agents.ros import ActionPhase, component_action class MyComponent(BaseComponent): @component_action(description={...}, phase=ActionPhase.PLANNING) def look_up_thing(self): ... # planner-only research tool @component_action(description={...}, phase=ActionPhase.EXECUTION) def grasp(self): ... # executor-only state-changing action @component_action(description={...}, phase=ActionPhase.BOTH) def describe_scene(self): ... # both phases ``` Bare `@component_action` defaults to `ActionPhase.EXECUTION`, preserving historical behaviour. - **`@component_fallback` methods** -- registered the same way, exposed as recovery tools the planner can fall back to. - **Additional ROS services** (returned by `get_ros_entrypoints()["services"]`) -- registered as `send_request_to_{name}` execution tools. The request type is auto-translated to JSON properties so the LLM fills request fields directly; Cortex constructs the message and sends it. - **Additional ROS action servers** (returned by `get_ros_entrypoints()["actions"]`) -- registered as `send_goal_to_{name}` execution tools, with the goal type auto-translated the same way. The Controller's `track_vision_target` action server is auto-discovered this way, for instance. - **The component's main action server** (when `run_type = "ActionServer"`) -- registered the same way. Setting `Planner.run_type = "ActionServer"` is what makes the Planner's main goal callable as an LLM tool. ### Custom actions Capabilities that don't naturally live on a managed component (a peripheral toggle, a database call, an external API hit) can be passed in via `actions=[...]`: ```python from agents.ros import Action cortex = Cortex( actions=[ Action(method=toggle_led, description="Toggle the robot's LED on or off."), Action(method=query_inventory, description="Look up the current item inventory."), ], ..., ) ``` Each `Action` must carry a description -- it's what the planner sees when deciding whether to call the tool. ### Robot plugin actions When a [robot plugin](../concepts/robot-plugins.md) is attached to the recipe (`Launcher(robot_plugin=...)`), Cortex picks up the robot's high-level actions automatically: each plugin action is registered as a namespaced execution tool, and the plugin's metadata (what the robot is, its vendor) is prepended to the planning prompt so the agent knows which body it is driving. You write no registration -- installing the plugin and attaching it to the `Launcher` is enough. See [Robot Plugins](../getting-started/plugins.md) for installing one. --- ## Public API ```python from agents.components import Cortex from agents.config import CortexConfig from agents.ros import Action, Topic, Launcher ``` ```python cortex_output = Topic(name="cortex_output", msg_type="StreamingString") cortex = Cortex( actions=[Action(method=toggle_led, description="...")], output=cortex_output, model_client=planner_client, db_client=chroma_client, # optional: enables RAG context config=CortexConfig( max_planning_steps=5, max_execution_steps=15, enable_rag=True, collection_name="robot_manual", ), component_name="cortex", ) ``` | Argument | Purpose | | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `actions` | Custom `Action` objects to expose as execution tools. Optional. | | `output` | A topic Cortex publishes to when the goal can be answered with text alone (no plan needed). Wire it into TTS to give the robot a voice for its own thoughts, or into the UI to render replies inline. | | `model_client` | The LLM client used for planning and confirmation. Optional if `enable_local_model=True`. | | `db_client` | Optional vector-DB client (e.g. `ChromaClient`). When set, Cortex queries it before each planning call and injects the result as RAG context. Used for domain knowledge: robot manuals, environment maps, prior conversations. | | `config` | A `CortexConfig`. Most-tuned fields are `max_planning_steps`, `max_execution_steps`, `enable_local_model`, `enable_rag`, `collection_name`, and `confirmation_temperature`. | Cortex always runs as a ROS2 **action server** -- you don't configure `run_type`. Its action type is `VisionLanguageAction` and the goal field is `task: string`. --- ## Cortex is also the Monitor When the launcher detects a Cortex component in the recipe, it is **also used as the Monitor** -- the central node that hosts events and actions, tracks every component's health, manages lifecycle transitions, and coordinates fallbacks. This is why Cortex needs no list of components to monitor -- the launcher feeds it the full graph. --- ## Memory-aware planning When a [Memory](memory.md) component is in the recipe, Cortex's planning prompt is **automatically augmented** on activation. The augmentation: - Lists Memory's perception-retrieval tools (`semantic_search`, `locate`, `recall`, ...) and its body-status tool, so the planner knows the difference between perception and interoception layers. - Pre-computes Memory's `inspect_component` output and embeds it in the system prompt, so the planner already knows the layer names without having to spend a planning step researching. - Adds a **task classification** step: every incoming task is assigned to (A) PERCEPTION QUERY, (B) BODY QUERY, or (C) ACTION TASK. Each class has a specific protocol (perception queries don't open episodes; action tasks always wrap themselves in `start_episode` / `end_episode` and check `body_status` first). - Body-status checks become **mandatory** for action tasks, so a Cortex with an interoception layer (battery, fault flags) might refuse missions when the readings indicate a problem rather than running them and discovering the issue mid-flight. No extra wiring. Drop a Memory component into the launcher and the augmentation kicks in. See [Memory and Cortex](../recipes/planning-and-manipulation/cortex-memory.md) for the full pattern. --- ## Observability Cortex publishes feedback on its action server during execution. Each plan step generates a feedback line carrying: - Step number and tool name. - Whether the step is being EXECUTE-d, SKIP-ped, ABORT-ed, or whether confirmation said CONTINUE. - Live status of any action goals in flight (running for _N_ seconds, latest feedback, stall warnings). The launcher's Web UI shows these feedback lines in the **main logging card** alongside the components' own logs, so the operator sees the agent's reasoning trace and the Planner's path-tracking feedback side by side. --- ## RAG context Setting `db_client` on Cortex enables retrieval-augmented planning. Before each planning call, Cortex queries the configured vector DB with the user's task and prepends the results to the planning prompt: ```python from agents.clients import ChromaClient from agents.vectordbs import ChromaDB cortex = Cortex( output=cortex_output, model_client=planner_client, db_client=ChromaClient(db=ChromaDB(), host="localhost", port=8000), config=CortexConfig( enable_rag=True, collection_name="building_layout", n_results=5, add_metadata=True, ), component_name="cortex", ) # Populate the DB with domain knowledge ahead of time: cortex.add_documents( ids=["floor1", "floor2"], metadatas=[{"floor": 1}, {"floor": 2}], documents=[ "Floor 1 contains the kitchen, dining room, and main entrance.", "Floor 2 contains the bedrooms and the office.", ], ) ``` Use this for static facts the planner shouldn't have to learn from the live recipe -- robot manuals, environment maps, prior conversation transcripts. For _dynamic_ facts the robot acquires at runtime, use [Memory](memory.md). --- ## Recipes - {doc}`Cortex: The Agentic Harness <../recipes/planning-and-manipulation/cortex-agent>` -- introductory tutorial. Vision + VLM + TTS + a custom action, all addressed in plain English with no orchestration code. - {doc}`Cortex Driving the Full Stack <../recipes/planning-and-manipulation/cortex-navigation>` -- the showcase. Cortex orchestrates a Kompass navigation stack, Vision, VLM, Memory, and TTS to handle compound natural-language goals. - {doc}`Memory and Cortex <../recipes/planning-and-manipulation/cortex-memory>` -- spatio-temporal memory wired into a Cortex planner; perception layers + interoception layers; episode-based consolidation; cross-session persistence. ``` ## File: intelligence/memory.md ```markdown # Memory **The first memory system for embodied agents built on neuroscience principles.** Memory is the EmbodiedAgents component that gives a robot a persistent, queryable sense of *place* and *history*. It is built on top of [eMEM](https://github.com/automatika-robotics/emem) -- a hybrid graph-based spatio-temporal memory designed specifically for situated agents. The core problem eMEM solves is the false dichotomy that has dominated robot memory until now: > Vector databases discard spatial structure. Metric maps discard semantics. **eMEM unifies both.** Memory accumulates a typed graph of everything the robot perceives -- detections, scene descriptions, sensor strings, internal-state readings -- indexed simultaneously by **meaning** (HNSW), **location** (R-tree), and **time** (SQLite indexes). Episodes group observations into task spans. Consolidation collapses old observations into searchable gists. Entity nodes track persistent objects across episodes via semantic-spatial merging. And **interoception** -- the robot's own internal state -- is a first-class memory dimension alongside world observations, not a separate metrics pipeline. ```{important} `Memory` is the supported successor to **MapEncoding**, deprecated since EmbodiedAgents 0.7.1. The two are not API-compatible: Memory replaces a flat vector-DB store with episodic consolidation, entity tracking, and an interoception surface. See [Migration](#migration-from-mapencoding) below. ``` ```{seealso} For a hands-on intro to Memory by itself, see [Spatio-Temporal Memory](../recipes/foundation/semantic-map.md). For the deep pairing with Cortex (the agentic harness), see [Memory and Cortex](../recipes/planning-and-manipulation/cortex-memory.md). For eMEM internals, see the [eMEM repository](https://github.com/automatika-robotics/emem). ``` ```{admonition} Installation :class: note `Memory` depends on the [eMEM](https://github.com/automatika-robotics/emem) package and raises `ImportError` at construction if it isn't available. eMEM is **not** installed by `emos install` — add it to the environment the EMOS launcher runs in. The command depends on your [install mode](../getting-started/installation.md): - **Pixi:** `cd ~/.local/share/emos && pixi add --pypi emem` - **Native:** `pip install emem` into your ROS 2 Python environment - **Container:** `docker exec -it emos-container pip install emem` (ephemeral — bake it into a derived image to persist) ``` --- ## Why it works eMEM's design is grounded in how humans and other animals actually structure memory. - {material-regular}`schema;1.2em;sd-text-primary` **Tiered consolidation** -- observations flow `working → short-term → long-term → archived`, mirroring the consolidation hierarchy seen in mammalian memory. Recent experience stays raw and queryable; older experience compresses into gists; ancient experience is archived (raw text removed, gist preserved). This bounds storage growth without losing the *meaning* of past activity. - {material-regular}`account_tree;1.2em;sd-text-primary` **Episodic structure** -- experience is grouped into named episodes that can nest hierarchically (`SUBTASK_OF`) and chain temporally (`FOLLOWS`). When an episode ends, its observations are clustered, summarised by an LLM into a *gist* that preserves the semantic content, the spatial centroid, and the time span -- and the raw observations are archived. - {material-regular}`hub;1.2em;sd-text-primary` **Entity persistence** -- detections of the same object across multiple episodes are auto-merged into a single `EntityNode` via semantic similarity (cosine) *and* spatial proximity (range threshold). The first time the robot sees "the red chair near the door", it's a new entity; the second time, it's recognised as the same one. `COOCCURS_WITH` edges capture which entities tend to be observed together. - {material-regular}`favorite;1.2em;sd-text-primary` **Interoception as memory** -- internal body state (battery, temperature, joint health, fault flags) is stored in the same graph as world observations, not in a separate telemetry stream. This is what lets a query like *"what was happening when the battery dropped?"* work -- the spatial-temporal-interoceptive associations emerge naturally because everything lives in the same graph. The whole stack runs **fully embedded** -- SQLite, hnswlib, and Rtree, with zero external services. A single `.db` file plus a `.hnsw.bin` file *is* the entire memory state. Reboot the robot, point it at the same files, and the prior session is remembered. --- ## Architecture at a glance ``` +-----------------+ | Memory | (EmbodiedAgents component) +--------+--------+ | delegates to +--------v--------+ | SpatioTemporal | | Memory | (eMEM facade) +--------+--------+ | +--------------+--------------+ | | | +--------v--+ +-------v-----+ +-----v------+ | Working | | Memory | | Consolida- | | Memory | | Tools | | tion | | (buffer) | | (10 tools) | | Engine | +--------+---+ +------+------+ +----+-------+ | | | +--------------+--------------+ | +--------v--------+ | MemoryStore | +--------+--------+ | +--------------+--------------+ | | | +-----v----+ +-----v-----+ +-----v----+ | SQLite | | hnswlib | | R-tree | | (nodes, | | (vector | | (spatial | | edges, | | search) | | index) | | tiers) | | | | | +-----------+ +-----------+ +----------+ ``` Three complementary indexes, one shared graph. ### The graph Four node types and six edge types compose the memory graph: **Nodes** | Node | Carries | |---|---| | `ObservationNode` | A single perception event: text, coordinates, timestamp, layer, source_type (`perception` or `interoception`), confidence. | | `EpisodeNode` | A named task / activity span grouping related observations. | | `GistNode` | A consolidated summary of multiple observations, with spatial extent and time range. Survives archival of the raw observations. | | `EntityNode` | A persistent tracked object / landmark (auto-merged across episodes by similarity + spatial proximity). | **Edges** | Edge | Meaning | |---|---| | `BELONGS_TO` | Observation → Episode | | `FOLLOWS` | Episode → Episode (temporal sequence) | | `SUBTASK_OF` | Episode → Episode (hierarchical nesting) | | `SUMMARIZES` | Gist → Observation(s) | | `OBSERVED_IN` | Entity → Observation | | `COOCCURS_WITH` | Entity ↔ Entity | ### Tiers ``` working → short_term → long_term → archived (buffer) (in store) (promoted) (text dropped, gist remains) ``` When an episode ends, eMEM consolidates its observations into a gist and archives them. For non-episodic streams that age past `consolidation_window`, time-window consolidation uses **DBSCAN** to spatially cluster old observations before summarising each cluster. ### Unified search `semantic_search` queries observations and gists through a single HNSW lookup. After consolidation, knowledge isn't lost -- it's compressed into gists that remain searchable alongside recent observations. --- ## Public API ```python from agents.components import Memory from agents.config import MemoryConfig from agents.ros import MemLayer, Topic ``` ### Layers A `MemLayer` is the smallest unit of input. Each layer subscribes to one topic; the topic's UI string representation is what gets stored. ```python detections = Topic(name="detections", msg_type="Detections") scene = Topic(name="scene_description", msg_type="String") battery = Topic(name="/battery_level", msg_type="Float32") # Perception layers detections_layer = MemLayer(subscribes_to=detections) scene_layer = MemLayer(subscribes_to=scene) # Interoception layer battery_layer = MemLayer(subscribes_to=battery, is_internal_state=True) ``` | Field | Meaning | |---|---| | `subscribes_to` | The topic to ingest from. | | `is_internal_state` | If `True`, observations are routed through `add_body_state` -- they're invisible to perception retrieval tools and surface only through the dedicated `body_status` tool. Used for interoception. | `MapLayer` is preserved as a backwards-compatible alias of `MemLayer`. ### Construction ```python position = Topic(name="/odom", msg_type="Odometry") memory = Memory( layers=[detections_layer, scene_layer, battery_layer], position=position, model_client=vlm_client, # used for episode-consolidation summaries + entity extraction embedding_client=embedding_client, # used for vector indexing config=MemoryConfig(db_path="/tmp/robot_memory.db"), trigger=10.0, # flush layer data every 10s component_name="memory", ) ``` | Argument | Purpose | |---|---| | `layers` | One `MemLayer` per perception or interoception topic. | | `position` | An Odometry topic. Memory tags every observation with the robot's current `(x, y, z)` so spatial queries work. | | `model_client` | Optional. Drives summarisation during episode consolidation and entity extraction. Without it, consolidation falls back to plain text concatenation. | | `embedding_client` | Optional. Drives vector indexing (e.g. an Ollama client serving an embedding model). Without it, falls back to `sentence-transformers`. | | `config` | A `MemoryConfig` -- see below. | | `trigger` | When to flush observations to the working buffer. A topic, list of topics, frequency in Hz, or `Event`. | ### Configuration highlights `MemoryConfig` exposes ~20 fields covering storage, consolidation, entity merging, and HNSW index tuning. The most reached-for ones in everyday recipes: | Field | Default | Controls | |---|---|---| | `db_path` | `"memory.db"` | eMEM SQLite database path. The whole memory state lives in this file. | | `auto_store` | `True` | Flush observations on every execution step. Set `False` to require explicit `store` calls. | | `working_memory_size` | `50` | Buffer size before older observations are dropped. | | `flush_interval` / `flush_batch_size` | `2.0s` / `5` | How aggressively the buffer is persisted. | | `consolidation_window` | `1800.0s` | Maximum gap between observations in the same consolidation chunk. | | `consolidation_spatial_eps` | `3.0m` | DBSCAN epsilon for time-window consolidation. | | `archive_after_seconds` | `3600.0s` | When raw observation text is dropped; the gist remains searchable. | | `entity_similarity_threshold` | `0.85` | Cosine threshold for merging a new detection with a known entity. | | `entity_spatial_radius` | `5.0m` | Spatial radius for the same merge. | | `recency_weight` | `0.0` | Boost recent observations in semantic search. `0.0` for pure semantic ordering. | --- ## Retrieval surface (10 tools) Memory exposes ten retrieval tools, all decorated as `@component_action`. Nine are `phase=ActionPhase.PLANNING` (Cortex consumes them while *building* a plan); `body_status` is `ActionPhase.BOTH` because the executor may also need it at runtime. | Tool | Purpose | |---|---| | `semantic_search` | Find observations by meaning. Queries observations *and* gists in a single HNSW lookup. | | `spatial_query` | Find observations within a radius of a point. R-tree backed. | | `temporal_query` | Find observations in a time range. Accepts relative strings like `"-10m"`. | | `episode_summary` | Get the consolidated summary of one or more episodes. | | `get_current_context` | Situational awareness -- nearby objects, area summaries, recent activity, latest body status. | | `search_gists` | Search consolidated long-term memory only (faster than full semantic search). | | `entity_query` | Find known entities by name, type, or location. | | `locate` | Resolve a concept to a spatial position (returns centroid + radius from the entity graph). | | `recall` | Cross-layer recall -- everything known about a concept across observations, gists, and entities. | | `body_status` | Latest interoception readings, optionally filtered by layer. | Two write-side actions are exposed as well: - `store_specific_memory` (execution) -- write an arbitrary string into memory at runtime, e.g. for runtime annotations Cortex wants to preserve. - `start_episode` / `end_episode` (execution) -- bracket episodes manually instead of relying on time-based consolidation. When a `Memory` component is in the recipe, [Cortex](cortex.md) auto-discovers all of these and **augments its planning prompt** with detailed task-classification guidance (PERCEPTION QUERY vs BODY QUERY vs ACTION TASK) and instructions for episode wrapping. No tool registration is required. ### Registering tools on a plain LLM If you want Memory's tools without Cortex (e.g. to give a plain `LLM` component tool-calling access for natural-language Q&A), use `register_tools_on`: ```python memory.register_tools_on(llm, send_tool_response_to_model=True) # Or register a subset: memory.register_tools_on(llm, tools=["semantic_search", "locate", "get_current_context"]) ``` --- ## Persistence The entire memory state lives in two files: the SQLite DB at `db_path` and an HNSW index file alongside it. **Reboot the robot, point the next session at the same paths, and the agent picks up where it left off.** Episodes from yesterday are still in the graph; entities are still merged; gists are still searchable. Cross-session continuity is the default, not a feature you opt into. This is what makes eMEM-on-Cortex behave as a cognitive system rather than a session-scoped vector DB. --- ## Migration from MapEncoding | MapEncoding | Memory | |---|---| | `MapEncoding` component | `Memory` component | | `MapLayer` | `MemLayer` (alias preserved) | | Flat vector DB (Chroma) | Graph + episodic + entity index, eMEM-backed | | `db_client=ChromaClient(...)` | `model_client=...`, `embedding_client=...` | | `map_topic=OccupancyGrid` (required) | None -- Memory uses real-world coords from Odometry directly | | `MapConfig(map_name=…)` | `MemoryConfig(db_path=…)` | | Free-form text retrieval via tool calling | Ten typed retrieval tools auto-registered with Cortex | | No interoception | `is_internal_state=True` on a layer surfaces it via `body_status` | | No episode structure | `start_episode` / `end_episode`, hierarchical nesting, automatic consolidation | | No entity tracking | Persistent `EntityNode` with semantic-spatial auto-merge | | Session-scoped | Cross-session persistent (single `.db` file) | Existing recipes built on `MapEncoding` still load with a deprecation warning. New recipes should use `Memory`. --- ## Recipes - {doc}`Spatio-Temporal Memory <../recipes/foundation/semantic-map>` -- introductory recipe. Memory built from a Vision component's detections and an MLLM's introspective answers, no Cortex on top. - {doc}`Memory and Cortex <../recipes/planning-and-manipulation/cortex-memory>` -- the deep pairing. Cortex auto-discovers Memory's retrieval surface, the planning prompt is automatically augmented with task classification, the robot becomes addressable as *"how are you?"*, *"what did you see?"*, and *"go remember the cat for me"*. - {doc}`Cortex Driving the Full Stack <../recipes/planning-and-manipulation/cortex-navigation>` -- adds a Kompass navigation stack on top of the Memory + Cortex pair so the robot can act on memory queries instead of just answering them. ``` ## File: intelligence/clients.md ```markdown # Inference Clients Clients are execution backends that instantiate and call inference on ML models. Certain components in EmbodiedAgents deal with ML models, vector databases, or both. These components take in a model client or DB client as one of their initialization parameters. The reason for this abstraction is to enforce _separation of concerns_. Whether an ML model is running on the edge hardware, on a powerful compute node in the network, or in the cloud, the components running on the robot edge can always use the model (or DB) via a client in a standardized way. This approach makes components independent of the model serving platforms, which may implement various inference optimizations depending on the model type. As a result, developers can choose an ML serving platform that offers the best latency/accuracy tradeoff based on the application's requirements. All clients implement a connection check. ML clients must implement inference methods, and optionally model initialization and deinitialization methods. This supports scenarios where an embodied agent dynamically switches between models or fine-tuned versions based on environmental events. Similarly, vector DB clients implement standard CRUD methods tailored to vector databases. EmbodiedAgents provides the following clients, designed to cover the most popular open-source model deployment platforms. Creating simple clients for other platforms is straightforward. ```{note} Some clients may require additional dependencies, which are detailed in the table below. If these are not installed, users will be prompted at runtime. ``` ```{list-table} :widths: 20 20 60 :header-rows: 1 * - Platform - Client - Description * - **Generic** - GenericHTTPClient - A generic client for interacting with OpenAI-compatible APIs, including vLLM, ms-swift, lmdeploy, Google Gemini, etc. Supports both standard and streaming responses, and works with LLMs and multimodal LLMs. Designed to be compatible with any API following the OpenAI standard. Supports tool calling. * - **RoboML** - RoboMLHTTPClient - An HTTP client for interacting with ML models served on [RoboML](https://github.com/automatika-robotics/roboml). Supports streaming outputs. * - **RoboML** - RoboMLWSClient - A WebSocket-based client for persistent interaction with [RoboML](https://github.com/automatika-robotics/roboml)-hosted ML models. Particularly useful for low-latency streaming of audio or text data. * - **RoboML** - RoboMLRESPClient - A Redis Serialization Protocol (RESP) based client for ML models served via [RoboML](https://github.com/automatika-robotics/roboml). Requires `pip install redis[hiredis]`. * - **Ollama** - OllamaClient - An HTTP client for interacting with ML models served on [Ollama](https://ollama.com). Supports LLMs/MLLMs and embedding models. Supports tool calling. Requires `pip install ollama`. * - **LeRobot** - LeRobotClient - A gRPC-based asynchronous client for vision-language-action (VLA) policies served on LeRobot Policy Server. Supports various robot action policies available in the LeRobot package by HuggingFace. Requires: `pip install grpcio` `pip install torch --index-url https://download.pytorch.org/whl/cpu` * - **ChromaDB** - ChromaClient - An HTTP client for interacting with a ChromaDB instance running as a server. Ensure that a ChromaDB server is active using: `pip install chromadb` `chroma run --path /db_path` ``` ``` ## File: intelligence/models.md ```markdown # Models Clients in EmbodiedAgents take as input a **model** or **vector database (DB)** specification. These are in most cases generic wrappers around a class of models or databases (e.g. Transformers-based LLMs) defined as [attrs](https://www.attrs.org/en/stable/) classes and include initialization parameters such as quantization schemes, inference options, embedding model (in case of vector DBs) etc. These specifications aim to standardize model initialization across diverse deployment platforms. ## Available Model Wrappers ```{list-table} :widths: 20 80 :header-rows: 1 * - Model Name - Description * - **GenericLLM** - A generic wrapper for LLMs served via OpenAI-compatible `/v1/chat/completions` APIs (e.g., vLLM, LMDeploy, OpenAI). Supports configurable inference options like temperature and max tokens. This wrapper must be used with the **GenericHTTPClient**. * - **GenericMLLM** - A generic wrapper for Multimodal LLMs (Vision-Language models) served via OpenAI-compatible APIs. Supports image inputs alongside text. This wrapper must be used with the **GenericHTTPClient**. * - **GenericTTS** - A generic wrapper for Text-to-Speech models served via OpenAI-compatible `/v1/audio/speech` APIs. Supports voice selection (`voice`) and speed (`speed`) configuration. This wrapper must be used with the **GenericHTTPClient**. * - **GenericSTT** - A generic wrapper for Speech-to-Text models served via OpenAI-compatible `/v1/audio/transcriptions` APIs. Supports language hints (`language`) and temperature settings. This wrapper must be used with the **GenericHTTPClient**. * - **OllamaModel** - A LLM/VLM model loaded from an Ollama checkpoint. Supports configurable generation and deployment options available in Ollama API. Complete list of Ollama models [here](https://ollama.com/library). This wrapper must be used with the **OllamaClient**. * - **TransformersLLM** - LLM models from HuggingFace/ModelScope based checkpoints. Supports quantization ("4bit", "8bit") specification. This model wrapper can be used with the **GenericHTTPClient** or any of the RoboML clients. * - **TransformersMLLM** - Multimodal LLM models from HuggingFace/ModelScope checkpoints for image-text inputs. Supports quantization. This model wrapper can be used with the **GenericHTTPClient** or any of the RoboML clients. * - **LeRobotPolicy** - Provides an interface for loading and running LeRobot policies -- vision-language-action (VLA) models trained for robotic manipulation tasks. Supports automatic extraction of feature and action specifications directly from dataset metadata, as well as flexible configuration of policy behavior. The policy can be instantiated from any compatible LeRobot checkpoint hosted on HuggingFace, making it easy to load pretrained models such as `smolvla_base` or others. This wrapper must be used with the gRPC-based **LeRobotClient**. * - **RoboBrain2** - [RoboBrain 2.0 by BAAI](https://github.com/FlagOpen/RoboBrain2.0) supports interactive reasoning with long-horizon planning and closed-loop feedback, spatial perception for precise point and bbox prediction from complex instructions, and temporal perception for future trajectory estimation. Checkpoint defaults to `"BAAI/RoboBrain2.0-3B"`; larger variants (7B, 32B) are at the same [collection](https://huggingface.co/collections/BAAI/robobrain20-6841eeb1df55c207a4ea0036). This wrapper can be used with any of the RoboML clients. * - **Whisper** - OpenAI's automatic speech recognition (ASR) model served via [faster-whisper](https://github.com/SYSTRAN/faster-whisper). Default checkpoint `"small.en"`; configurable `compute_type` (`"int8"`, `"float16"`, `"float32"`). Available on the [RoboML](https://github.com/automatika-robotics/roboml) platform and can be used with any RoboML client. Recommended: **RoboMLWSClient**. * - **TransformersTTS** - A unified wrapper for HuggingFace Transformers TTS models. Automatically detects whether to use generative inference (Bark, SpeechT5) or forward-only inference (VITS). Default checkpoint `"facebook/mms-tts-eng"`; other examples include `"suno/bark-small"` (Bark) and `"microsoft/speecht5_tts"` (SpeechT5, paired with `vocoder_checkpoint="microsoft/speecht5_hifigan"`). For Bark, `voice` selects a voice preset (e.g. `"v2/en_speaker_6"`). Available on the [RoboML](https://github.com/automatika-robotics/roboml) platform and can be used with any RoboML client. Recommended: **RoboMLWSClient**. * - **VisionModel** - A generic wrapper for object detection and tracking models from [HuggingFace Transformers](https://huggingface.co/models?pipeline_tag=object-detection) (RT-DETR, DETR, Grounding DINO, YOLOS, etc.). Default checkpoint `"PekingU/rtdetr_r50vd_coco_o365"`. Tracking via [ByteTrack](https://github.com/roboflow/trackers) is included; enable with `setup_trackers=True` and tune via `tracking_distance_threshold`. Available on the [RoboML](https://github.com/automatika-robotics/roboml) platform and can be used with any RoboML client. Recommended: **RoboMLRESPClient**. ``` ## Built-in Local Models EmbodiedAgents includes lightweight models that run directly on the robot without needing an external model server (Ollama, RoboML, etc.). These are ideal for offline operation, edge deployment, or as automatic fallbacks when a remote server becomes unavailable. ```{list-table} :widths: 15 15 15 30 25 :header-rows: 1 * - Local Model - Component - Framework - Default Checkpoint - Dependency * - **LocalLLM** - LLM - llama-cpp-python - Qwen/Qwen3-0.6B-GGUF - `pip install llama-cpp-python` * - **LocalVLM** - VLM - llama-cpp-python - ggml-org/moondream2-20250414-GGUF - `pip install llama-cpp-python` * - **LocalSTT** - SpeechToText - sherpa-onnx - csukuangfj/sherpa-onnx-whisper-tiny.en - `pip install sherpa-onnx` * - **LocalTTS** - TextToSpeech - sherpa-onnx - csukuangfj/kokoro-en-v0_19 - `pip install sherpa-onnx` * - **LocalVision** - Vision - onnxruntime - DEIM detector - `pip install onnxruntime` ``` ```{note} - Local models can be activated via `enable_local_model=True` in the component config or `fallback_to_local()` action. - Model weights are auto-downloaded from HuggingFace on first use. To use a custom model, set `local_model_path` in the component config. - GPU-accelerated variants are available for `llama-cpp-python` (CUDA/Metal builds) and `onnxruntime` (`onnxruntime-gpu`). - Dependencies are pre-installed in EMOS Docker containers. For native installations, install them manually as shown above. ``` ## Available Vector Databases ```{list-table} :widths: 20 80 :header-rows: 1 * - Vector DB - Description * - **ChromaDB** - [Chroma](https://www.trychroma.com/) is an open-source AI application database with support for vector search, full-text search, and multi-modal retrieval. Supports "ollama" and "sentence-transformers" embedding backends. Can be used with the **ChromaClient**. ``` ````{note} For `ChromaDB`, make sure you install required packages: ```bash pip install ollama # For Ollama backend (requires Ollama runtime) pip install sentence-transformers # For Sentence-Transformers backend ``` ```` To use Ollama embedding models ([available models](https://ollama.com/search?c=embedding)), ensure the Ollama server is running and accessible via specified `host` and `port`. ``` ## File: navigation/overview.md ```markdown # Kompass **The navigation engine of EMOS --** GPU-accelerated, event-driven autonomy for mobile robots [Kompass](https://github.com/automatika-robotics/kompass) lets you create sophisticated navigation stacks with blazingly fast, hardware-agnostic performance. It is the only open-source navigation framework with cross-vendor GPU acceleration. ## Why Kompass? Robotic navigation isn't about perfecting a single component; it is about architecting a system that survives contact with the real world. While metric navigation has matured, deploying robots extensively in dynamic environments remains an unsolved challenge. As highlighted by the **ICRA BARN Challenges**, static pipelines fail when faced with the unpredictability of the physical world: > _"A single stand-alone approach that is able to address all variety of obstacle configurations all together is still out of our reach."_ > — **Lessons from The 3rd BARN Challenge (ICRA 2024)** **Kompass was built to fill this gap.** Unlike existing solutions that rely on rigid behavior trees, Kompass is an event-driven, GPU-native stack designed for maximum adaptability and hardware efficiency. - {material-regular}`bolt;1.2em;sd-text-primary` Adaptive Event-Driven Core -- The stack reconfigures itself on the fly based on environmental context. Use *Pure Pursuit* on open roads, switch to *DWA* indoors, fall back to a docking controller near the station -- all triggered by events, not brittle Behavior Trees. Adapt to external world events ("Crowd Detected", "Entering Warehouse"), not just internal robot states. - {material-regular}`speed;1.2em;sd-text-primary` GPU-Accelerated, Vendor-Agnostic -- Core algorithms in C++ with SYCL-based GPU support. Runs natively on **Nvidia, AMD, Intel, and other** GPUs without vendor lock-in -- the first navigation framework to support cross-GPU acceleration. Up to **3,106x speedups** over CPU-based approaches. - {material-regular}`psychology;1.2em;sd-text-primary` ML Models as First-Class Citizens -- Event-driven design means ML model outputs can directly reconfigure the navigation stack. Use object detection to switch controllers, VLMs to answer abstract perception queries, or [EmbodiedAgents](https://github.com/automatika-robotics/embodied-agents) vision components for target tracking -- all seamlessly integrated through EMOS's unified architecture. - {material-regular}`code;1.2em;sd-text-primary` Pythonic Simplicity -- Configure a sophisticated, multi-fallback navigation system in a single readable Python script. Core algorithms are decoupled from ROS wrappers, so upgrading ROS distributions won't break your navigation logic. Extend with new planners in Python for prototyping or C++ for production. --- ## Architecture Kompass has a modular event-driven architecture, divided into several interacting components each responsible for one navigation subtask. ```{figure} /_static/images/diagrams/system_components_light.png :class: light-only :alt: Navigation Components :align: center The main components of the Kompass navigation stack. ``` ```{figure} /_static/images/diagrams/system_components_dark.png :class: dark-only :alt: Navigation Components :align: center ``` Each component runs as a ROS2 lifecycle node and communicates with the other components using ROS2 topics, services or action servers: ```{figure} /_static/images/diagrams/system_graph_light.png :class: light-only :alt: Kompass Full System :align: center System Diagram for Point Navigation ``` ```{figure} /_static/images/diagrams/system_graph_dark.png :class: dark-only :alt: Kompass Full System :align: center ``` --- ## Navigation Components ::::{grid} 1 2 3 3 :gutter: 3 :::{grid-item-card} {material-regular}`route;1.2em;sd-text-primary` Planner :link: planning :link-type: doc Global path planning using OMPL algorithms (RRT*, PRM, etc.). ::: :::{grid-item-card} {material-regular}`gamepad;1.2em;sd-text-primary` Controller :link: control :link-type: doc Real-time local control with DWA, Stanley, DVZ, and Vision Follower plugins. ::: :::{grid-item-card} {material-regular}`security;1.2em;sd-text-primary` Drive Manager :link: drive-manager :link-type: doc Safety enforcement, emergency stops, and command smoothing. ::: :::{grid-item-card} {material-regular}`grid_on;1.2em;sd-text-primary` Local Mapper :link: mapping :link-type: doc Real-time ego-centric occupancy grid from sensor data. ::: :::{grid-item-card} {material-regular}`public;1.2em;sd-text-primary` Map Server :link: mapping :link-type: doc Static global map management with 3D PCD support. ::: :::{grid-item-card} {material-regular}`settings;1.2em;sd-text-primary` Robot Config :link: robot-config :link-type: doc Define kinematics, geometry, and control limits for your platform. ::: :::: --- ## Minimum Sensor Requirements Kompass is designed to be flexible in terms of sensor configurations. However, at least the following sensors are required for basic autonomous navigation: - {material-regular}`speed;1.2em;sd-text-primary` **Odometry Source** (e.g., wheel encoders, IMU or visual odometry) - {material-regular}`radar;1.2em;sd-text-primary` **Obstacle Detection Sensor** (e.g., 2D LiDAR **or** Depth Camera) - {material-regular}`my_location;1.2em;sd-text-primary` **Robot Pose Source** (e.g., localization system such as AMCL or visual SLAM) These provide the minimal data necessary for localization, mapping, and safe path execution. ## Optional Sensors for Enhanced Features Additional sensors can enhance navigation capabilities and unlock advanced features: - {material-regular}`camera;1.2em;sd-text-secondary` **RGB Camera(s)** — Enables vision-based navigation, object tracking, and semantic navigation. - {material-regular}`view_in_ar;1.2em;sd-text-secondary` **Depth Camera** — Improves obstacle avoidance in 3D environments and enables more accurate object tracking. - {material-regular}`sensors;1.2em;sd-text-secondary` **3D LiDAR** — Enhances perception in complex environments with full 3D obstacle detection. - {material-regular}`satellite_alt;1.2em;sd-text-secondary` **GPS** — Enables outdoor navigation and geofenced planning. - {material-regular}`cell_tower;1.2em;sd-text-secondary` **UWB / BLE Beacons** — Improves localization in GPS-denied environments. --- Kompass supports dynamic configuration, allowing it to operate with minimal sensors and scale up for complex applications when additional sensing is available. ``` ## File: navigation/robot-config.md ```markdown # Robot Configuration Before EMOS can drive your robot, it needs to understand its physical constraints. You define this "Digital Twin" using the `RobotConfig` object, which aggregates the Motion Model, Geometry, and Control Limits. ```python import numpy as np from kompass_core.models import RobotConfig, RobotType, RobotGeometry # Example: Defining a simple box-shaped Ackermann robot robot_config = RobotConfig( model_type=RobotType.ACKERMANN, geometry_type=RobotGeometry.Type.BOX, geometry_params=np.array([1.0, 1.0, 1.0]) # x, y, z ) ``` ## Motion Models EMOS supports three distinct kinematic models. Choose the one that matches your robot's drivetrain. - {material-regular}`directions_car;1.2em;sd-text-primary` Ackermann — Car-Like Vehicles. Non-holonomic constraints (bicycle model). The robot has a limited steering angle and cannot rotate in place. - {material-regular}`swap_horiz;1.2em;sd-text-primary` Differential — Two-Wheeled Robots. Capable of forward/backward motion and zero-radius rotation (spinning in place). - {material-regular}`open_with;1.2em;sd-text-primary` Omni — Holonomic Robots. Mecanum-wheel platforms or quadrupeds. Capable of instantaneous motion in any direction (x, y) and rotation. ## Robot Geometry The geometry defines the collision volume of the robot, used by the local planner for obstacle avoidance. The `geometry_params` argument expects a **NumPy array** containing specific dimensions based on the selected type: ```{list-table} :widths: 15 25 60 :header-rows: 1 * - Type - Parameters (np.array) - Description * - **BOX** - `[length, width, height]` - Axis-aligned box. * - **CYLINDER** - `[radius, length_z]` - Vertical cylinder. * - **SPHERE** - `[radius]` - Perfect sphere. * - **ELLIPSOID** - `[axis_x, axis_y, axis_z]` - Axis-aligned ellipsoid. * - **CAPSULE** - `[radius, length_z]` - Cylinder with hemispherical ends. * - **CONE** - `[radius, length_z]` - Vertical cone. ``` ```python import numpy as np from kompass_core.models import RobotConfig, RobotType, RobotGeometry # A cylinder robot (Radius=0.5m, Height=1.0m) cylinder_robot_config = RobotConfig( model_type=RobotType.DIFFERENTIAL_DRIVE, geometry_type=RobotGeometry.Type.CYLINDER, geometry_params=np.array([0.5, 1.0]) ) ``` ## Control Limits Safety is paramount. You must explicitly define the kinematic limits for linear and angular velocities. For both linear and angular control limits we need to set: - Maximum velocity (m/s) or (rad/s) - Maximum acceleration (m/s^2) or (rad/s^2) - Maximum deceleration (m/s^2) or (rad/s^2) Additionally, for angular control limits we can set the maximum steering angle (rad). EMOS separates **Acceleration** limits from **Deceleration** limits. This allows you to configure a "gentle" acceleration for smooth motion, but a "hard" deceleration for emergency braking. ```python from kompass_core.models import LinearCtrlLimits, AngularCtrlLimits, RobotConfig, RobotType, RobotGeometry import numpy as np # 1. Linear Limits (Forward/Backward) ctrl_vx = LinearCtrlLimits(max_vel=1.0, max_acc=1.5, max_decel=2.5) # 2. Linear Limits (Lateral — for Omni robots) ctrl_vy = LinearCtrlLimits(max_vel=0.5, max_acc=0.7, max_decel=3.5) # 3. Angular Limits (Rotation) # max_steer is only used for Ackermann robots ctrl_omega = AngularCtrlLimits( max_vel=1.0, max_acc=2.0, max_decel=2.0, max_steer=np.pi / 3 ) # Setup your robot configuration my_robot = RobotConfig( model_type=RobotType.DIFFERENTIAL_DRIVE, geometry_type=RobotGeometry.Type.CYLINDER, geometry_params=np.array([0.1, 0.3]), ctrl_vx_limits=ctrl_vx, ctrl_omega_limits=ctrl_omega, ) ``` :::{tip} Deceleration limit is separated from the acceleration limit to allow the robot to decelerate faster thus ensuring safety. ::: :::{tip} For Ackermann robots, `ctrl_omega_limits.max_steer` defines the maximum physical steering angle of the wheels in radians. ::: ## Coordinate Frames EMOS needs to know the names of your TF frames to perform lookups. You configure this using the `RobotFrames` object. The components will automatically subscribe to `/tf` and `/tf_static` to track these frames. ```python from kompass.config import RobotFrames frames = RobotFrames( world='map', # The fixed global reference frame odom='odom', # The drift-prone odometry frame robot_base='base_link', # The center of the robot scan='scan', # Lidar frame rgb='camera/rgb', # RGB Camera frame depth='camera/depth' # Depth Camera frame ) ``` ```{list-table} :widths: 20 70 :header-rows: 1 * - Frame - Description * - **world** - The global reference for path planning (usually `map`). * - **odom** - The continuous reference for local control loops. * - **robot_base** - The physical center of the robot. All geometry is relative to this. * - **scan** - Laserscan sensor frame. * - **rgb** - RGB camera sensor frame. * - **depth** - Depth camera sensor frame. ``` ```{note} It is important to configure your coordinate frames names correctly and pass them to Kompass. Components in Kompass will subscribe automatically to the relevant `/tf` and `/tf_static` topics in ROS2 to get the necessary transformations. ``` ``` ## File: navigation/planning.md ```markdown # Global Planner **Global path planning and trajectory generation.** The Planner component is responsible for finding an optimal or suboptimal path from a start to a goal location using complete map information (i.e. the global or reference map). It leverages the **[Open Motion Planning Library (OMPL)](https://ompl.kavrakilab.org/)** backend to support various sampling-based algorithms (RRT*, PRM, etc.), capable of handling complex kinematic constraints. Collision checking is handled by the **[FCL (Flexible Collision Library)](https://github.com/flexible-collision-library/fcl)** for precise geometric collision detection. ## Available Run Types Planner can be used with all four available Run Types: ```{list-table} :widths: 20 80 * - **{material-regular}`schedule;1.2em;sd-text-primary` Timed** - **Periodic Re-planning.** Compute a new plan periodically (e.g., at 1Hz) from the robot's current location to the last received goal. * - **{material-regular}`touch_app;1.2em;sd-text-primary` Event** - **Reactive Planning.** Trigger a new plan computation *only* when a new message is received on the `goal_point` topic. * - **{material-regular}`dns;1.2em;sd-text-primary` Service** - **Request/Response.** Offers a standard ROS2 Service (`PlanPath`). Computes a single plan per request and returns it immediately. * - **{material-regular}`hourglass_top;1.2em;sd-text-primary` Action Server** - **Long-Running Goal.** Offers a standard ROS2 Action. continuously computes and updates the plan until the goal is reached or canceled. ``` ## Inputs ```{list-table} :widths: 10 40 10 40 :header-rows: 1 * - Key Name - Allowed Types - Number - Default * - map - [`nav_msgs.msg.OccupancyGrid`](http://docs.ros.org/en/noetic/api/nav_msgs/html/msg/OccupancyGrid.html) - 1 - `Topic(name="/map", msg_type="OccupancyGrid", qos_profile=QoSConfig(durability=TRANSIENT_LOCAL))` * - goal_point - [`nav_msgs.msg.Odometry`](https://docs.ros.org/en/noetic/api/nav_msgs/html/msg/Odometry.html), [`geometry_msgs.msg.PoseStamped`](http://docs.ros.org/en/jade/api/geometry_msgs/html/msg/PoseStamped.html), [`geometry_msgs.msg.PointStamped`](http://docs.ros.org/en/jade/api/geometry_msgs/html/msg/PointStamped.html), `automatika_embodied_agents.msg.Detections`, `automatika_embodied_agents.msg.PointsOfInterest`, `automatika_embodied_agents.msg.Trackings` - 1 - `Topic(name="/goal", msg_type="PointStamped")` * - location - [`nav_msgs.msg.Odometry`](https://docs.ros.org/en/noetic/api/nav_msgs/html/msg/Odometry.html), [`geometry_msgs.msg.PoseStamped`](http://docs.ros.org/en/jade/api/geometry_msgs/html/msg/PoseStamped.html), [`geometry_msgs.msg.Pose`](http://docs.ros.org/en/jade/api/geometry_msgs/html/msg/Pose.html) - 1 - `Topic(name="/odom", msg_type="Odometry")` ``` :::{note} `goal_point` input is only used if the Planner is running as TIMED or EVENT Component. In the other two types, the goal point is provided in the service request or the action goal. ::: :::{tip} **Detections, Points-of-Interest, and Trackings as goals.** The Planner accepts `Detections`, `PointsOfInterest`, and `Trackings` messages from EmbodiedAgents directly on the `goal_point` input. These carry pixel-space coordinates from ML models; when paired with an RGBD source, the depth channel is used to project pixel coordinates to averaged world-space coordinates via camera intrinsics, so a Vision component can drive the Planner without an intermediate `Pose` topic. See [Cortex with Navigation](../recipes/planning-and-manipulation/cortex-navigation.md) and [Multimodal Planning](../recipes/planning-and-manipulation/planning-models.md) for the patterns. ::: ## Outputs ```{list-table} :widths: 10 40 10 40 :header-rows: 1 * - Key Name - Allowed Types - Number - Default * - plan - [`nav_msgs.msg.Path`](http://docs.ros.org/en/noetic/api/nav_msgs/html/msg/Path.html) - 1 - `Topic(name="/plan", msg_type="Path")` * - reached_end - `std_msgs.msg.Bool` - 1 - `Topic(name="/reached_end", msg_type="Bool")` ``` ## OMPL Algorithms EMOS integrates over 25 OMPL geometric planners. See the [Planning Algorithms (OMPL)](../advanced/algorithms.md#planning-algorithms-ompl) section of the Algorithms Reference for a complete list with benchmarks and per-planner configuration parameters. ## Collision Checking (FCL) [FCL](https://github.com/flexible-collision-library/fcl) is a generic library for performing proximity and collision queries on geometric models. EMOS leverages FCL to perform precise collision checks between the robot's kinematic model and both static (map) and dynamic (sensor) obstacles during path planning and control. ## Usage Example ```python from kompass.components import Planner, PlannerConfig from kompass.config import ComponentRunType from kompass.ros import Topic from kompass_core.models import RobotType, RobotConfig, RobotGeometry, LinearCtrlLimits, AngularCtrlLimits import numpy as np # Configure your robot my_robot = RobotConfig( model_type=RobotType.DIFFERENTIAL_DRIVE, geometry_type=RobotGeometry.Type.CYLINDER, geometry_params=np.array([0.1, 0.3]), ctrl_vx_limits=LinearCtrlLimits(max_vel=1.0, max_acc=1.5, max_decel=2.5), ctrl_omega_limits=AngularCtrlLimits( max_vel=1.0, max_acc=2.0, max_decel=2.0, max_steer=np.pi / 3 ), ) # Setup the planner config config = PlannerConfig( robot=my_robot, loop_rate=1.0 # 1Hz ) planner = Planner(component_name="planner", config=config) planner.run_type = ComponentRunType.EVENT # Can also pass a string "Event" # Add rviz clicked_point as input topic goal_topic = Topic(name="/clicked_point", msg_type="PoseStamped") planner.inputs(goal_point=goal_topic) ``` ``` ## File: navigation/control.md ```markdown # Controller **Motion control and dynamic obstacle avoidance.** The Controller is the real-time "pilot" of your robot. While the [Planner](planning.md) looks ahead to find a global route, the Controller deals with the immediate reality — calculating velocity commands to follow the global path (path following) or a global target point (object following) while reacting to dynamic obstacles and adhering to kinematic constraints. It supports modular **Plugins** allowing you to switch between different control strategies (e.g., *Pure Pursuit* vs *DWA* vs *Visual Servoing*) via configuration. ## Available Run Types The Controller typically runs at a high frequency (10Hz-50Hz) to ensure smooth motion. ```{list-table} :widths: 20 80 * - **{material-regular}`schedule;1.2em;sd-text-primary` Timed** - **Periodic Control Loop.** Computes a new velocity command periodically if all necessary inputs are available. * - **{material-regular}`hourglass_top;1.2em;sd-text-primary` Action Server** - **Goal Tracking.** Offers a [`ControlPath`](https://github.com/automatika-robotics/kompass/blob/main/kompass_interfaces/action/ControlPath.action) ROS2 Action. Continuously computes control commands until the goal is reached or the action is preempted. ``` ## Inputs ```{list-table} :widths: 10 40 10 40 :header-rows: 1 * - Key Name - Allowed Types - Number - Default * - plan - [`nav_msgs.msg.Path`](http://docs.ros.org/en/noetic/api/nav_msgs/html/msg/Path.html) - 1 - `Topic(name="/plan", msg_type="Path")` * - location - [`nav_msgs.msg.Odometry`](https://docs.ros.org/en/noetic/api/nav_msgs/html/msg/Odometry.html), [`geometry_msgs.msg.PoseStamped`](http://docs.ros.org/en/jade/api/geometry_msgs/html/msg/PoseStamped.html), [`geometry_msgs.msg.Pose`](http://docs.ros.org/en/jade/api/geometry_msgs/html/msg/Pose.html) - 1 - `Topic(name="/odom", msg_type="Odometry")` * - sensor_data - [`sensor_msgs.msg.LaserScan`](https://docs.ros.org/en/noetic/api/sensor_msgs/html/msg/LaserScan.html), [`sensor_msgs.msg.PointCloud2`](http://docs.ros.org/en/noetic/api/sensor_msgs/html/msg/PointCloud2.html) - 1 - `Topic(name="/scan", msg_type="LaserScan")` * - local_map - [`nav_msgs.msg.OccupancyGrid`](http://docs.ros.org/en/noetic/api/nav_msgs/html/msg/OccupancyGrid.html) - 1 - `Topic(name="/local_map/occupancy_layer", msg_type="OccupancyGrid")` * - vision_tracking - [`automatika_embodied_agents.msg.Trackings`](https://github.com/automatika-robotics/ros-agents/tree/main/agents_interfaces/msg), [`automatika_embodied_agents.msg.Detections2D`](https://github.com/automatika-robotics/ros-agents/tree/main/agents_interfaces/msg) - 1 - None, Should be provided to use the vision target tracking ``` ```{tip} Provide a `vision_tracking` input topic to the controller to activate the creation of a vision-based target following action server. See the [Vision Tracking tutorial](../recipes/navigation/vision-tracking-rgb.md) for more details. ``` ## Outputs ```{list-table} :widths: 10 40 10 40 :header-rows: 1 * - Key Name - Allowed Types - Number - Default * - command - [`geometry_msgs.msg.Twist`](http://docs.ros.org/en/noetic/api/geometry_msgs/html/msg/Twist.html) - 1 - `Topic(name="/control", msg_type="Twist")` * - multi_command - [`kompass_interfaces.msg.TwistArray`](https://github.com/automatika-robotics/kompass/tree/main/kompass_interfaces/msg) - 1 - `Topic(name="/control_list", msg_type="TwistArray")` * - interpolation - [`nav_msgs.msg.Path`](http://docs.ros.org/en/noetic/api/nav_msgs/html/msg/Path.html) - 1 - `Topic(name="/interpolated_path", msg_type="Path")` * - local_plan - [`nav_msgs.msg.Path`](http://docs.ros.org/en/noetic/api/nav_msgs/html/msg/Path.html) - 1 - `Topic(name="/local_path", msg_type="Path")` * - tracked_point - [`nav_msgs.msg.Odometry`](https://docs.ros.org/en/noetic/api/nav_msgs/html/msg/Odometry.html), [`geometry_msgs.msg.PoseStamped`](http://docs.ros.org/en/jade/api/geometry_msgs/html/msg/PoseStamped.html), [`geometry_msgs.msg.Pose`](http://docs.ros.org/en/jade/api/geometry_msgs/html/msg/Pose.html), [`automatika_embodied_agents.msg.Detection2D`](https://github.com/automatika-robotics/ros-agents/tree/main/agents_interfaces/msg) - 1 - `Topic(name="/tracked_point", msg_type="PoseStamped")` ``` ## Algorithms EMOS includes several production-ready control plugins suited for different environments: - {material-regular}`route;1.2em;sd-text-primary` **[Stanley](../advanced/algorithms.md)** — Geometric path tracking using the front axle as reference. Best for Ackermann steering. - {material-regular}`shield;1.2em;sd-text-primary` **[DVZ](../advanced/algorithms.md)** — Deformable Virtual Zone. Reactive collision avoidance based on risk zones. Extremely fast for crowded dynamic environments. - {material-regular}`speed;1.2em;sd-text-primary` **[DWA](../advanced/algorithms.md)** — Dynamic Window Approach. Sample-based collision avoidance with GPU support. Considers kinematics to find optimal velocity. - {material-regular}`visibility;1.2em;sd-text-primary` **[VisionFollower](../advanced/algorithms.md)** — Vision target following controllers. Steer the robot to keep a visual target centered using RGB or depth data. See the [Algorithms Reference](../advanced/algorithms.md) for detailed descriptions of each algorithm. ## Usage Example ```python from kompass.components import Controller, ControllerConfig from kompass.ros import Topic # Setup custom configuration my_config = ControllerConfig(loop_rate=10.0) # Init a controller object my_controller = Controller(component_name="controller", config=my_config) # Change an input my_controller.inputs(plan=Topic(name='/global_path', msg_type='Path')) # Change run type (default "Timed") my_controller.run_type = "ActionServer" # Change plugin my_controller.plugin = 'DWA' ``` ``` ## File: navigation/drive-manager.md ```markdown # Drive Manager **Safety enforcement and command smoothing.** The Drive Manager is the final gatekeeper before commands reach your robot's low-level interfaces. Its primary job is to ensure that every command falls within the robot's physical limits, satisfies smoothness constraints, and does not lead to a collision. It acts as a safety shield, intercepting velocity commands from the Controller and applying **Emergency Stops** or **Slowdowns** based on immediate sensor data. ## Safety Layers The Drive Manager implements a multi-stage safety pipeline: - {material-regular}`block;1.2em;sd-text-danger` **Emergency Stop** — Critical Zone. Checks proximity sensors directly. If an obstacle enters the configured safety distance and angle, the robot stops immediately. - {material-regular}`slow_motion_video;1.2em;sd-text-warning` **Dynamic Slowdown** — Warning Zone. If an obstacle enters the slowdown zone, the robot's velocity is proportionally reduced. - {material-regular}`tune;1.2em;sd-text-primary` **Control Limiting** — Kinematic Constraints. Clamps incoming velocity and acceleration commands to the robot's physical limits. - {material-regular}`filter_alt;1.2em;sd-text-primary` **Control Smoothing** — Jerk Control. Applies smoothing filters to incoming commands to prevent jerky movements and wheel slip. - {material-regular}`lock_open;1.2em;sd-text-primary` **Robot Unblocking** — Moves the robot forward, backwards or rotates in place if the space is free to move the robot away from a blocking point. This action can be configured to be triggered with an external event. ```{figure} /_static/images/diagrams/drive_manager_light.png :class: light-only :alt: Emergency Zone & Slowdown Zone :align: center :width: 70% Emergency Zone & Slowdown Zone ``` ```{figure} /_static/images/diagrams/drive_manager_dark.png :class: dark-only :alt: Emergency Zone & Slowdown Zone :align: center :width: 70% ``` ```{note} Critical and Slowdown Zone checking is implemented in C++ in [kompass-core](https://github.com/automatika-robotics/kompass-core) for fast emergency behaviors. The core implementation supports both **GPU** and **CPU** (**defaults to GPU if available**). ``` ## Built-in Actions The Drive Manager provides built-in behaviors for direct control and recovery. These can be triggered via [Events](../concepts/events-and-actions.md): ```{list-table} :widths: 20 70 :header-rows: 1 * - Action - Function * - **move_forward** - Moves the robot forward for `max_distance` meters, if the forward direction is clear of obstacles. * - **move_backward** - Moves the robot backwards for `max_distance` meters, if the backward direction is clear of obstacles. * - **rotate_in_place** - Rotates the robot in place for `max_rotation` radians, if the given safety margin around the robot is clear of obstacles. * - **move_to_unblock** - Recovery behavior. Automatically attempts to move forward, backward, or rotate to free the robot from a collision state or blockage. ``` ```{note} All movement actions require `LaserScan` information to determine if the movement direction is collision-free. ``` ## Available Run Types ```{list-table} :widths: 10 80 * - **Timed** - Sends incoming command periodically to the robot. ``` ## Inputs ```{list-table} :widths: 10 40 10 40 :header-rows: 1 * - Key Name - Allowed Types - Number - Default * - command - [`geometry_msgs.msg.Twist`](http://docs.ros.org/en/noetic/api/geometry_msgs/html/msg/Twist.html) - 1 - `Topic(name="/control", msg_type="Twist")` * - multi_command - [`kompass_interfaces.msg.TwistArray`](https://github.com/automatika-robotics/kompass/tree/main/kompass_interfaces/msg) - 1 - `Topic(name="/control_list", msg_type="TwistArray")` * - sensor_data - [`sensor_msgs.msg.LaserScan`](https://docs.ros.org/en/noetic/api/sensor_msgs/html/msg/LaserScan.html), `std_msgs.msg.Float64`, `std_msgs.msg.Float32` - 1 + (10 optional) - `Topic(name="/scan", msg_type="LaserScan")` * - location - [`nav_msgs.msg.Odometry`](https://docs.ros.org/en/noetic/api/nav_msgs/html/msg/Odometry.html), [`geometry_msgs.msg.PoseStamped`](http://docs.ros.org/en/jade/api/geometry_msgs/html/msg/PoseStamped.html), [`geometry_msgs.msg.Pose`](http://docs.ros.org/en/jade/api/geometry_msgs/html/msg/Pose.html) - 1 - `Topic(name="/odom", msg_type="Odometry")` ``` ## Outputs ```{list-table} :widths: 10 40 10 40 :header-rows: 1 * - Key Name - Allowed Types - Number - Default * - robot_command - `geometry_msgs.msg.Twist` - 1 - `Topic(name="/cmd_vel", msg_type="Twist")` * - emergency_stop - `std_msgs.msg.Bool` - 1 - `Topic(name="/emergency_stop", msg_type="Bool")` ``` ## Usage Example ```python from kompass.components import DriveManager, DriveManagerConfig from kompass.ros import Topic # Setup custom configuration # closed_loop: send commands to the robot in closed loop (checks feedback from robot state) # critical_zone_distance: for emergency stop (m) my_config = DriveManagerConfig( closed_loop=True, critical_zone_distance=0.1, # Stop if obstacle < 10cm slowdown_zone_distance=0.3, # Slow down if obstacle < 30cm critical_zone_angle=90.0 # Check 90 degrees cone in front ) # Instantiate driver = DriveManager(component_name="driver", config=my_config) # Remap Outputs driver.outputs(robot_command=Topic(name='/my_robot_cmd', msg_type='Twist')) ``` ``` ## File: navigation/mapping.md ```markdown # Mapping & Localization This page covers the mapping components in EMOS: the **Local Mapper** for real-time obstacle detection and the **Map Server** for static global maps, along with recommended community packages for localization. ## Local Mapper **Real-time, ego-centric occupancy grid generation.** While the global map provides a static long-term view, the Local Mapper builds a dynamic, short-term map of the robot's immediate surroundings based on real-time sensor data. It captures moving obstacles (people, other robots) and temporary changes, serving as the primary input for the [Controller](control.md) to enable fast reactive navigation. At its core, the Local Mapper uses the Bresenham line drawing algorithm in C++ to efficiently update an occupancy grid from incoming LaserScan data. This approach ensures fast and accurate raycasting to determine free and occupied cells in the local grid. To maximize performance and adaptability, the implementation **supports both CPU and GPU execution**: - {material-regular}`memory;1.5em;sd-text-primary` SYCL GPU Acceleration — Vendor-agnostic GPU acceleration compatible with Nvidia, AMD, Intel, and any other GPGPU-capable devices. - {material-regular}`developer_board;1.5em;sd-text-primary` Multi-Threaded CPU — Falls back to a highly optimized multi-threaded CPU implementation if no GPU is available. ### Inputs ```{list-table} :widths: 10 40 10 40 :header-rows: 1 * - Key Name - Allowed Types - Number - Default * - sensor_data - [`sensor_msgs.msg.LaserScan`](https://docs.ros.org/en/noetic/api/sensor_msgs/html/msg/LaserScan.html) - 1 - `Topic(name="/scan", msg_type="LaserScan")` * - location - [`nav_msgs.msg.Odometry`](https://docs.ros.org/en/noetic/api/nav_msgs/html/msg/Odometry.html), [`geometry_msgs.msg.PoseStamped`](http://docs.ros.org/en/jade/api/geometry_msgs/html/msg/PoseStamped.html), [`geometry_msgs.msg.Pose`](http://docs.ros.org/en/jade/api/geometry_msgs/html/msg/Pose.html) - 1 - `Topic(name="/odom", msg_type="Odometry")` ``` ### Outputs ```{list-table} :widths: 10 40 10 40 :header-rows: 1 * - Key Name - Allowed Types - Number - Default * - local_map - `nav_msgs.msg.OccupancyGrid` - 1 - `Topic(name="/local_map/occupancy_layer", msg_type="OccupancyGrid")` ``` ```{note} Current implementation supports LaserScan sensor data to create an Occupancy Grid local map. PointCloud and semantic information will be supported in an upcoming release. ``` ### Usage Example ```python from kompass_core.mapping import LocalMapperConfig from kompass.components import LocalMapper, MapperConfig # Select map parameters: 5m x 5m rolling window with 20cm resolution map_params = MapperConfig(width=5.0, height=5.0, resolution=0.2) # Setup custom component configuration my_config = LocalMapperConfig(loop_rate=10.0, map_params=map_params) # Init a mapper my_mapper = LocalMapper(component_name="mapper", config=my_config) ``` ## Map Server **Static global map management and 3D-to-2D projection.** The Map Server is the source of ground-truth for the navigation system. It reads static map files, processes them, and publishes the global `OccupancyGrid` required by the Planner and Localization components. Unlike standard ROS2 map servers, the EMOS Map Server supports **native 3D Point Cloud (PCD)** files, automatically slicing and projecting them into 2D navigable grids based on configurable height limits. ### Key Features - {material-regular}`swap_horiz;1.2em;sd-text-primary` **Map Data Conversion** — Reads map files in either 2D (YAML) or 3D (PCD) format and converts the data into usable global map formats (OccupancyGrid). - {material-regular}`public;1.2em;sd-text-primary` **Global Map Serving** — Once map data is loaded and processed, the MapServer publishes the global map as an `OccupancyGrid` message, continuously available for path planning, localization, and obstacle detection. - {material-regular}`view_in_ar;1.2em;sd-text-primary` **Point Cloud to Grid Conversion** — If the map data is provided as a PCD file, the MapServer generates an occupancy grid from the point cloud using the provided grid resolution and ground limits. - {material-regular}`crop_free;1.2em;sd-text-primary` **Custom Frame Handling** — Configurable reference frames ensuring the map aligns with your robot's TF tree. - {material-regular}`save;1.2em;sd-text-primary` **Map Saving** — Supports saving both 2D and 3D maps to files via `Save2dMapToFile` and `Save3dMapToFile` services. - {material-regular}`update;1.2em;sd-text-primary` **Map Update Frequency Control** — Control how often map data is read and converted via the `map_file_read_rate` parameter. ### Outputs ```{list-table} :widths: 10 40 10 40 :header-rows: 1 * - Key Name - Allowed Types - Number - Default * - global_map - [`nav_msgs.msg.OccupancyGrid`](http://docs.ros.org/en/noetic/api/nav_msgs/html/msg/OccupancyGrid.html) - 1 - `Topic(name="/map", msg_type="OccupancyGrid")` * - spatial_sensor - [`sensor_msgs.msg.PointCloud2`](http://docs.ros.org/en/noetic/api/sensor_msgs/html/msg/PointCloud2.html) - 1, optional - `Topic(name="/row_point_cloud", msg_type="PointCloud2")` ``` ### Usage Example ```python from kompass.components import MapServer, MapServerConfig from kompass.ros import Topic my_config = MapServerConfig( map_file_path="/path/to/environment.pcd", map_file_read_rate=5.0, grid_resolution=0.1, pc_publish_row=False ) my_map_server = MapServer(component_name="map_server", config=my_config) ``` ## Global Mapping & Localization EMOS is designed to be modular. While it handles core navigation, it relies on standard community packages for global localization and mapping. Recommended solutions: | Package | Purpose | | :--- | :--- | | **[Robot Localization](https://github.com/cra-ros-pkg/robot_localization)** | Sensor Fusion (EKF) — Fuse IMU, Odometry, and GPS data for robust `odom` → `base_link` transforms. | | **[SLAM Toolbox](https://github.com/SteveMacenski/slam_toolbox)** | 2D SLAM & Localization — Generate initial maps or perform "Lifelong" mapping in changing environments. | | **[Glim](https://koide3.github.io/glim/)** | 3D LiDAR-Inertial Mapping — GPU-accelerated 3D SLAM using LiDAR and IMU data. | :::{tip} Remember that EMOS includes its own [3D-capable Map Server](#map-server) if you need to work directly with Point Cloud (`.pcd`) files generated by tools like Glim. ::: ``` ## File: navigation/motion-server.md ```markdown # Motion Server **System validation, calibration, and motion data recording.** Unlike the core navigation components, the Motion Server does not plan paths or avoid obstacles. Instead, it provides essential utilities for validating your robot's physical performance and tuning its control parameters. It serves two primary purposes: 1. **Automated Motion Tests:** Executing pre-defined maneuvers (step response, circles) to calibrate the robot's motion model on new terrain. 2. **Black Box Recording:** Capturing synchronized control commands and robot responses (Pose/Velocity) during operation for post-analysis. ## Key Capabilities - {material-regular}`tune;1.5em;sd-text-primary` Motion Calibration — Execute step inputs or circular paths automatically to measure the robot's real-world response vs. the theoretical model. - {material-regular}`fiber_manual_record;1.5em;sd-text-primary` Data Recording — Record exact control inputs and odometry outputs synchronized in time. Essential for tuning controller gains or debugging tracking errors. - {material-regular}`loop;1.5em;sd-text-primary` Closed-Loop Validation — Can act as both the source of commands (during tests) and the sink for recording, allowing you to validate the entire control pipeline. - {material-regular}`flash_on;1.5em;sd-text-primary` Event-Triggered — Start recording or launch a calibration sequence automatically based on external events (e.g., "Terrain Changed" or "Slip Detected"). ```{note} The available motion tests include Step tests and Circle test and can be configured by adjusting the MotionServerConfig. ``` ## Available Run Types ```{list-table} :widths: 20 80 * - **{material-regular}`schedule;1.2em;sd-text-primary` Timed** - **Auto-Start Tests.** Automatically launches the configured motion tests periodically after the component starts. * - **{material-regular}`touch_app;1.2em;sd-text-primary` Event** - **Triggered Tests.** Waits for a `True` signal on the `run_tests` input topic to launch the calibration sequence. * - **{material-regular}`hourglass_top;1.2em;sd-text-primary` Action Server** - **On-Demand Recording.** Offers a `MotionRecording` ROS2 Action. Allows you to start/stop recording specific topics for a set duration via an Action Goal. ``` ```{note} The available motion tests include Step tests and Circle test and can be configured by adjusting the [MotionServerConfig](../apidocs/kompass/kompass.components.motion_server.md) ``` ## Inputs ```{list-table} :widths: 10 40 10 40 :header-rows: 1 * - Key Name - Allowed Types - Number - Default * - run_tests - `std_msgs.msg.Bool` - 1 - `Topic(name="/run_tests", msg_type="Bool")` * - command - [`geometry_msgs.msg.Twist`](http://docs.ros.org/en/noetic/api/geometry_msgs/html/msg/Twist.html) - 1 - `Topic(name="/cmd_vel", msg_type="Twist")` * - location - [`nav_msgs.msg.Odometry`](https://docs.ros.org/en/noetic/api/nav_msgs/html/msg/Odometry.html), [`geometry_msgs.msg.PoseStamped`](http://docs.ros.org/en/jade/api/geometry_msgs/html/msg/PoseStamped.html), [`geometry_msgs.msg.Pose`](http://docs.ros.org/en/jade/api/geometry_msgs/html/msg/Pose.html) - 1 - `Topic(name="/odom", msg_type="Odometry")` ``` ## Outputs ```{list-table} :widths: 10 40 10 40 :header-rows: 1 * - Key Name - Allowed Types - Number - Default * - robot_command - [`geometry_msgs.msg.Twist`](http://docs.ros.org/en/noetic/api/geometry_msgs/html/msg/Twist.html) - 1 - `Topic(name="/cmd_vel", msg_type="Twist")` ``` ```{note} Topic for *Control Command* is both in MotionServer inputs and outputs: - The output is used when running automated testing (i.e. sending the commands directly from the MotionServer). - The input is used to purely record motion and control from external sources (example: recording output from Controller). - Different command topics can be configured for the input and the output. For example: to test the DriveManager, the control command from MotionServer output can be sent to the DriveManager, then the DriveManager output can be configured as the MotionServer input for recording. ``` ## Usage Example ```python from kompass.components import MotionServer, MotionServerConfig from kompass.ros import Topic # 1. Configuration my_config = MotionServerConfig( step_test_velocity=1.0, step_test_duration=5.0 ) # 2. Instantiate motion_server = MotionServer(component_name="motion_server", config=my_config) # 3. Setup for Event-Based Testing motion_server.run_type = "Event" motion_server.inputs(run_tests=Topic(name="/start_calibration", msg_type="Bool")) ``` ``` ## File: recipes/foundation/conversational-agent.md ```markdown # Conversational Agent Often times robots are equipped with a speaker system and a microphone. Once these peripherals have been exposed through ROS, we can use EMOS to trivially create a conversational interface on the robot. Our conversational agent will use a multimodal LLM for contextual question/answering utilizing the camera onboard the robot. Furthermore, it will use speech-to-text and text-to-speech models for converting audio to text and vice versa. We will start by importing the relevant components that we want to string together. ```python from agents.components import VLM, SpeechToText, TextToSpeech ``` [Components](../../intelligence/ai-components.md) are basic functional units in EMOS. Their inputs and outputs are defined using ROS [Topics](../../concepts/topics.md). And their function can be any input transformation, for example the inference of an ML model. Lets setup these components one by one. Since our input to the robot would be speech, we will setup the speech-to-text component first. ## SpeechToText Component This component listens to an audio input topic, that takes in a multibyte array of audio (captured in a ROS std_msgs message, which maps to Audio msg_type in EMOS) and can publish output to a text topic. It can also be configured to get the audio stream from microphones on board our robot. By default the component is configured to use a small Voice Activity Detection (VAD) model, [Silero-VAD](https://github.com/snakers4/silero-vad) to filter out any audio that is not speech. However, merely utilizing speech can be problematic in robots, due to the hands free nature of the audio system. Therefore its useful to add wakeword detection, so that speech-to-text is only activated when the robot is called with a specific phrase (e.g. 'Hey Jarvis'). We will be using this configuration in our example. First we will setup our input and output topics and then create a config object which we can later pass to our component. ```{note} With **enable_vad** set to **True**, the component automatically downloads and deploys [Silero-VAD](https://github.com/snakers4/silero-vad) by default in ONNX format. This model has a small footprint and can be easily deployed on the edge. However we need to install a couple of dependencies for this to work. These can be installed with: `pip install pyaudio onnxruntime` ``` ```{note} With **enable_wakeword** set to **True**, the component automatically downloads and deploys a pre-trained model from [openWakeWord](https://github.com/dscripka/openWakeWord) by default in ONNX format, that can be invoked with **'Hey Jarvis'**. Other pre-trained models from openWakeWord are available [here](https://github.com/dscripka/openWakeWord). However it is recommended that you deploy own wakeword model, which can be easily trained by following [this amazing tutorial](https://github.com/dscripka/openWakeWord/blob/main/notebooks/automatic_model_training.ipynb). The tutorial notebook can be run in [Google Colab](https://colab.research.google.com/drive/1yyFH-fpguX2BTAW8wSQxTrJnJTM-0QAd?usp=sharing). ``` ```python from agents.ros import Topic from agents.config import SpeechToTextConfig # Define input and output topics (pay attention to msg_type) audio_in = Topic(name="audio0", msg_type="Audio") text_query = Topic(name="text0", msg_type="String") s2t_config = SpeechToTextConfig(enable_vad=True, # option to listen for speech through the microphone, set to False if using web UI enable_wakeword=True) # option to invoke the component with a wakeword like 'hey jarvis', set to False if using web UI ``` ```{warning} The _enable_wakeword_ option cannot be enabled without the _enable_vad_ option. ``` ```{seealso} Check the available defaults and options for the SpeechToTextConfig in the [API reference](../../apidocs/agents/agents.config). ``` To initialize the component we also need a model client for a speech to text model. We will be using the WebSocket client for RoboML for this purpose. ```{note} RoboML is an aggregator library that provides a model serving apparatus for locally serving opensource ML models useful in robotics. Learn about setting up RoboML [here](https://www.github.com/automatika-robotics/roboml). ``` Additionally, we will use the client with a model called Whisper, a popular opensource speech to text model from OpenAI. Lets see what that looks like in code. ```python from agents.clients import RoboMLWSClient from agents.models import Whisper # Setup the model client whisper = Whisper(name="whisper") # Custom model init params can be provided here roboml_whisper = RoboMLWSClient(whisper) # Initialize the component speech_to_text = SpeechToText( inputs=[audio_in], # the input topic we setup outputs=[text_query], # the output topic we setup model_client=roboml_whisper, trigger=audio_in, config=s2t_config, # pass in the config object component_name="speech_to_text" ) ``` The trigger parameter lets the component know that it has to perform its function (in this case model inference) when an input is received on this particular topic. In our configuration, the component will be triggered using voice activity detection on the continuous stream of audio being received on the microphone. Next we will setup our VLM component. ## VLM Component The VLM component takes as input a text topic (the output of the SpeechToText component) and an image topic, assuming we have a camera device onboard the robot publishing this topic. And just like before we need to provide a model client, this time with a VLM model. This time we will use the OllamaClient along with *qwen2.5vl:latest* model, an opensource multimodal LLM from the Qwen family, available on Ollama. Furthermore, we will configure our VLM component using `VLMConfig`. We will set `stream=True` to make the VLM output text be published as a stream for downstream components that consume this output. In EMOS, streaming output can be chunked using a `break_character` in the config (Default: '.'). This way the downstream TextToSpeech component can start generating audio as soon as the first sentence is produced by the LLM. ```{note} Ollama is one of the most popular local LLM serving projects. Learn about setting up Ollama [here](https://ollama.com). ``` Here is the code for our VLM setup. ```python from agents.clients.ollama import OllamaClient from agents.models import OllamaModel from agents.config import VLMConfig # Define the image input topic and a new text output topic image0 = Topic(name="image_raw", msg_type="Image") text_answer = Topic(name="text1", msg_type="String") # Define a model client (working with Ollama in this case) # OllamaModel is a generic wrapper for all ollama models qwen_vl = OllamaModel(name="qwen_vl", checkpoint="qwen2.5vl:latest") qwen_client = OllamaClient(qwen_vl) mllm_config = VLMConfig(stream=True) # Other inference specific parameters can be provided here # Define an VLM component mllm = VLM( inputs=[text_query, image0], # Notice the text input is the same as the output of the previous component outputs=[text_answer], model_client=qwen_client, trigger=text_query, component_name="vqa" # We have also given our component an optional name ) ``` We can further customize our VLM component by attaching a context prompt template. This can be done at the component level or at the level of a particular input topic. In this case we will attach a prompt template to the input topic **text_query**. ```python # Attach a prompt template mllm.set_topic_prompt(text_query, template="""You are an amazing and funny robot. Answer the following about this image: {{ text0 }}""" ) ``` Notice that the template is a jinja2 template string, where the actual name of the topic is set as a variable. For longer templates you can also write them to a file and provide its path when calling this function. After this we move on to setting up our last component. ## TextToSpeech Component The TextToSpeech component setup will be very similar to the SpeechToText component. We will once again use a RoboML client, this time with the unified `TransformersTTS` wrapper. RoboML serves any HuggingFace Transformers TTS model -- VITS, Bark, SpeechT5, SeamlessM4T, etc. -- through a single class; here we keep the default checkpoint, [Facebook's MMS-TTS-eng (VITS)](https://huggingface.co/facebook/mms-tts-eng), which is fast and lightweight enough to run comfortably on-device. The component can be configured to play audio on a playback device available onboard the robot, which we will utilize through our config. An output topic is optional for this component as we will be playing the audio directly on device. ```{note} In order to utilize _play_on_device_ you need to install a couple of dependencies as follows: `pip install soundfile sounddevice` ``` ```python from agents.config import TextToSpeechConfig from agents.models import TransformersTTS # config for asynchronously playing audio on device t2s_config = TextToSpeechConfig(play_on_device=True, stream=True) # Set play_on_device to false if using the web UI # Uncomment the following line for receiving output on the web UI # audio_out = Topic(name="audio_out", msg_type="Audio") vits = TransformersTTS(name="vits") # default checkpoint: facebook/mms-tts-eng roboml_vits = RoboMLWSClient(vits) text_to_speech = TextToSpeech( inputs=[text_answer], outputs=[], # use outputs=[audio_out] for receiving answers on web UI trigger=text_answer, model_client=roboml_vits, config=t2s_config, component_name="text_to_speech" ) ``` ## Launching the Components The final step in this example is to launch the components. This is done by passing the defined components to the launcher and calling the **bringup** method. EMOS also allows us to create a web-based UI for interacting with our conversational agent recipe. ```python from agents.ros import Launcher # Launch the components launcher = Launcher() launcher.enable_ui(inputs=[audio_in, text_query], outputs=[image0]) # specify topics launcher.add_pkg( components=[speech_to_text, mllm, text_to_speech] ) launcher.bringup() ``` Et voila! We have setup a graph of three components in less than 50 lines of well formatted code. The complete example is as follows: ```{code-block} python :caption: Multimodal Audio Conversational Agent :linenos: from agents.components import VLM, SpeechToText, TextToSpeech from agents.config import SpeechToTextConfig, TextToSpeechConfig, VLMConfig from agents.clients import OllamaClient, RoboMLWSClient from agents.models import Whisper, TransformersTTS, OllamaModel from agents.ros import Topic, Launcher audio_in = Topic(name="audio0", msg_type="Audio") text_query = Topic(name="text0", msg_type="String") whisper = Whisper(name="whisper") # Custom model init params can be provided here roboml_whisper = RoboMLWSClient(whisper) s2t_config = SpeechToTextConfig(enable_vad=True, # option to listen for speech through the microphone, set to False if using web UI enable_wakeword=True) # option to invoke the component with a wakeword like 'hey jarvis', set to False if using web UI speech_to_text = SpeechToText( inputs=[audio_in], outputs=[text_query], model_client=roboml_whisper, trigger=audio_in, config=s2t_config, component_name="speech_to_text", ) image0 = Topic(name="image_raw", msg_type="Image") text_answer = Topic(name="text1", msg_type="String") qwen_vl = OllamaModel(name="qwen_vl", checkpoint="qwen2.5vl:latest") qwen_client = OllamaClient(qwen_vl) mllm_config = VLMConfig(stream=True) # Other inference specific parameters can be provided here mllm = VLM( inputs=[text_query, image0], outputs=[text_answer], model_client=qwen_client, trigger=text_query, config=mllm_config, component_name="vqa", ) t2s_config = TextToSpeechConfig(play_on_device=True, stream=True) # Set play_on_device to false if using the web UI # Uncomment the following line for receiving output on the web UI # audio_out = Topic(name="audio_out", msg_type="Audio") vits = TransformersTTS(name="vits") # default checkpoint: facebook/mms-tts-eng roboml_vits = RoboMLWSClient(vits) text_to_speech = TextToSpeech( inputs=[text_answer], outputs=[], # use outputs=[audio_out] for receiving answers on web UI trigger=text_answer, model_client=roboml_vits, config=t2s_config, component_name="text_to_speech" ) launcher = Launcher() launcher.enable_ui(inputs=[audio_in, text_query], outputs=[image0]) # specify topics launcher.add_pkg(components=[speech_to_text, mllm, text_to_speech]) launcher.bringup() ``` ## Web Based UI for Interacting with the Robot To interact with topics on the robot, EMOS can create dynamically specified UIs. This is useful if the robot does not have a microphone/speaker interface or if one wants to communicate with it remotely. We will also like to see the images coming in from the robots camera to have more context of its answers. In the code above, we already specified the input and output topics for the UI by calling the function `launcher.enable_ui`. Furthermore, we can set `enable_vad` and `enable_wakeword` options in `s2t_config` to `False` and set `play_on_device` option in `t2s_config` to `False`. Now we are ready to use our browser based UI. ````{note} In order to run the client you will need to install [FastHTML](https://www.fastht.ml/) and [MonsterUI](https://github.com/AnswerDotAI/MonsterUI) with ```shell pip install python-fasthtml monsterui ```` The client displays a web UI on **http://localhost:5001** if you have run it on your machine. Or you can access it at **http://:5001** if you have run it on the robot. --- ```{tip} **Promote this recipe to production.** While you're shaping it, the script runs straight with `python recipe.py`. Once it's solid, drop it at `~/emos/recipes//recipe.py` and run `emos run ` -- you'll get sensor pre-flight checks, persistent logs, and a card on the dashboard so an operator can launch it from a browser. See [Running Recipes](../../getting-started/running-recipes.md) for the full development-vs-production comparison and install-mode pitfalls (especially in Container mode). ``` ``` ## File: recipes/foundation/prompt-engineering.md ```markdown # Prompt Engineering In this recipe we will use the output of an object detection component to enrich the prompt of a VLM component. Let us start by importing the components. ```python from agents.components import Vision, VLM ``` ## Setting up the Object Detection Component For object detection and tracking, EMOS provides a unified Vision [component](../../intelligence/ai-components.md). This component takes as input an image topic published by a camera device onboard our robot. The output of this component can be a _detections_ topic in case of object detection or a _trackings_ topic in case of object tracking. In this example we will use a _detections_ topic. ```python from agents.ros import Topic # Define the image input topic image0 = Topic(name="image_raw", msg_type="Image") # Create a detection topic detections_topic = Topic(name="detections", msg_type="Detections") ``` Additionally the component requires a model client with an object detection model. We will use the RESP client for RoboML and the `VisionModel` wrapper, which initialises any [HuggingFace Transformers object detection model](https://huggingface.co/models?pipeline_tag=object-detection) (RT-DETR, DETR, Grounding DINO, YOLOS, ...) by checkpoint name. We pick the RT-DETR checkpoint pretrained on COCO + Objects365. ```{note} Learn about setting up RoboML with vision [here](https://github.com/automatika-robotics/roboml/blob/main/README.md#vision-model-support). ``` ```{seealso} Browse all supported detection models on the [HuggingFace object-detection page](https://huggingface.co/models?pipeline_tag=object-detection). ``` ```python from agents.models import VisionModel from agents.clients import RoboMLRESPClient, RoboMLHTTPClient from agents.config import VisionConfig # Add an object detection model object_detection = VisionModel(name="object_detection", checkpoint="PekingU/rtdetr_r50vd_coco_o365") roboml_detection = RoboMLRESPClient(object_detection) # Initialize the Vision component detection_config = VisionConfig(threshold=0.5) vision = Vision( inputs=[image0], outputs=[detections_topic], trigger=image0, config=detection_config, model_client=roboml_detection, component_name="detection_component", ) ``` ```{tip} Notice that we passed in an optional config to the component. Component configs can be used to setup various parameters in the component. If the component calls an ML model then inference parameters for the model can be set in the component config. ``` ## Setting up the VLM Component For the VLM component, we will provide an additional text input topic, which will listen to our queries. The output of the component will be another text topic. We will use the RoboML HTTP client with the multimodal LLM Idefics2 by the good folks at HuggingFace for this example. ```python from agents.models import TransformersMLLM # Define VLM input and output text topics text_query = Topic(name="text0", msg_type="String") text_answer = Topic(name="text1", msg_type="String") # Define a model client (working with roboml in this case) idefics = TransformersMLLM(name="idefics_model", checkpoint="HuggingFaceM4/idefics2-8b") idefics_client = RoboMLHTTPClient(idefics) # Define a VLM component # We can pass in the detections topic which we defined previously directly as an optional input # to the VLM component in addition to its other required inputs mllm = VLM( inputs=[text_query, image0, detections_topic], outputs=[text_answer], model_client=idefics_client, trigger=text_query, component_name="mllm_component" ) ``` Next we will setup a component level prompt to ensure that our text query and the output of the detections topic are sent to the model as we intend. We will do this by passing a jinja2 template to the **set_component_prompt** function. ```python mllm.set_component_prompt( template="""Imagine you are a robot. This image has following items: {{ detections }}. Answer the following about this image: {{ text0 }}""" ) ``` ```{caution} The names of the topics used in the jinja2 template are the same as the name parameters set when creating the Topic objects. ``` ## Launching the Components Finally we will launch our components as we did in the previous example. ```python from agents.ros import Launcher # Launch the components launcher = Launcher() launcher.add_pkg( components=[vision, mllm] ) launcher.bringup() ``` And there we have it. Complete code of this example is provided below. ```{code-block} python :caption: Prompt Engineering with Object Detection :linenos: from agents.components import Vision, VLM from agents.models import VisionModel, TransformersMLLM from agents.clients import RoboMLRESPClient, RoboMLHTTPClient from agents.ros import Topic, Launcher from agents.config import VisionConfig image0 = Topic(name="image_raw", msg_type="Image") detections_topic = Topic(name="detections", msg_type="Detections") object_detection = VisionModel( name="object_detection", checkpoint="PekingU/rtdetr_r50vd_coco_o365" ) roboml_detection = RoboMLRESPClient(object_detection) detection_config = VisionConfig(threshold=0.5) vision = Vision( inputs=[image0], outputs=[detections_topic], trigger=image0, config=detection_config, model_client=roboml_detection, component_name="detection_component", ) text_query = Topic(name="text0", msg_type="String") text_answer = Topic(name="text1", msg_type="String") idefics = TransformersMLLM(name="idefics_model", checkpoint="HuggingFaceM4/idefics2-8b") idefics_client = RoboMLHTTPClient(idefics) mllm = VLM( inputs=[text_query, image0, detections_topic], outputs=[text_answer], model_client=idefics_client, trigger=text_query, component_name="mllm_component" ) mllm.set_component_prompt( template="""Imagine you are a robot. This image has following items: {{ detections }}. Answer the following about this image: {{ text0 }}""" ) launcher = Launcher() launcher.add_pkg( components=[vision, mllm] ) launcher.bringup() ``` --- ```{tip} **Promote this recipe to production.** While you're shaping it, the script runs straight with `python recipe.py`. Once it's solid, drop it at `~/emos/recipes//recipe.py` and run `emos run ` -- you'll get sensor pre-flight checks, persistent logs, and a card on the dashboard so an operator can launch it from a browser. See [Running Recipes](../../getting-started/running-recipes.md) for the full development-vs-production comparison and install-mode pitfalls (especially in Container mode). ``` ``` ## File: recipes/foundation/semantic-map.md ```markdown # Spatio-Temporal Memory Autonomous Mobile Robots (AMRs) keep a representation of their environment in the form of occupancy maps. Such maps are fine for navigation but are *amnesic*: a robot doesn't *know* the objects, the rooms, the situations, or its own history. The [Memory](../../intelligence/memory.md) component gives an EMOS agent a structured place to store everything it perceives, indexed by **meaning**, **location**, and **time** — and to consolidate that stream of observations into long-term memory the way humans do. In this recipe we wire perception into Memory: an object detector publishing `Detections`, a VLM publishing periodic introspective answers, both feeding `Memory` as separate layers. **Memory** runs on [eMEM](https://github.com/automatika-robotics/emem), a hybrid graph-based spatio-temporal memory built on neuroscience principles: tiered consolidation, episodic structure, entity persistence, and interoception as a first-class memory dimension. ```{seealso} For the conceptual background on Memory and the neuroscience principles behind it, see the [Memory page](../../intelligence/memory.md). Once you've built a memory and want to *reason over it* in plain English, the [Memory and Cortex](../planning-and-manipulation/cortex-memory.md) recipe shows how Cortex auto-discovers Memory's tools and uses them in its orchestration. ``` ```{admonition} Prerequisites :class: important The `Memory` component requires the [eMEM](https://github.com/automatika-robotics/emem) Python package, which `emos install` does not add. The install command depends on your mode (Pixi: `pixi add --pypi emem`; Native: `pip install emem`; Container: install inside the container) — see [Memory installation](../../intelligence/memory.md). ``` --- ## Setting up a Vision Component ```python from agents.components import Vision from agents.config import VisionConfig from agents.ros import Topic # Define the image input topic image0 = Topic(name="image_raw", msg_type="Image") # Create a detection topic detections_topic = Topic(name="detections", msg_type="Detections") ``` Additionally the component requires a model client with an object detection model. We will use the RESP client for [RoboML](https://github.com/automatika-robotics/roboml) and the `VisionModel` wrapper, which initialises any [HuggingFace Transformers object detection model](https://huggingface.co/models?pipeline_tag=object-detection) (RT-DETR, DETR, Grounding DINO, YOLOS, ...) by checkpoint name. ```{note} Learn about setting up RoboML with vision [here](https://github.com/automatika-robotics/roboml/blob/main/README.md#vision-model-support). ``` ```python from agents.models import VisionModel from agents.clients import RoboMLRESPClient # Add an object detection model object_detection = VisionModel( name="object_detection", checkpoint="PekingU/rtdetr_r50vd_coco_o365", ) roboml_detection = RoboMLRESPClient(object_detection) # Initialize the Vision component detection_config = VisionConfig(threshold=0.5) vision = Vision( inputs=[image0], outputs=[detections_topic], trigger=image0, config=detection_config, model_client=roboml_detection, component_name="detection_component", ) ``` The vision component will provide us with semantic information to add to memory. However, object names are only the most basic semantic element of the scene. One can view such basic elements in aggregate to create more abstract semantic associations. This is where multimodal LLMs come in. --- ## Setting up a VLM Component With multimodal LLMs we can ask higher-level introspective questions about what the robot is currently seeing and store the answers in memory alongside the raw detections. We'll set up a VLM component that periodically asks itself the same question — what kind of room am I in? — using two EMOS concepts: a `FixedInput` (a simulated [Topic](../../concepts/topics.md) whose value is a constant string) and a *timed* component (one whose `trigger` is a frequency rather than an input topic). ```python from agents.components import VLM from agents.clients import OllamaClient from agents.models import OllamaModel from agents.ros import FixedInput # Define a model client (working with Ollama in this case) qwen_vl = OllamaModel(name="qwen_vl", checkpoint="qwen2.5vl:latest") qwen_client = OllamaClient(qwen_vl) # Define a fixed input for the component introspection_query = FixedInput( name="introspection_query", msg_type="String", fixed=( "What kind of a room is this? Is it an office, a bedroom or a kitchen? " "Give a one word answer, out of the given choices" ), ) # Define output of the component introspection_answer = Topic(name="introspection_answer", msg_type="String") # Start a timed (periodic) component using the mllm model defined earlier # This component answers the same question every 15 seconds introspector = VLM( inputs=[introspection_query, image0], # we use image0 from earlier outputs=[introspection_answer], model_client=qwen_client, trigger=15.0, # frequency in seconds component_name="introspector", ) ``` LLM/VLM model outputs can be unpredictable. Before publishing the answer of our question to the output topic, we want to ensure that the model has indeed provided a one word answer, and that this answer is one of the expected choices. EMOS allows arbitrary pre-processor functions on data being published; we'll add a tiny validator that drops anything outside the expected vocabulary: ```python from typing import Optional def introspection_validation(output: str) -> Optional[str]: for option in ["office", "bedroom", "kitchen"]: if option in output.lower(): return option introspector.add_publisher_preprocessor(introspection_answer, introspection_validation) ``` Now `introspection_answer` only carries clean one-word labels. --- ## Building Memory The final step is to wire those two streams into a `Memory` component. Memory's input surface is a list of `MemLayer`s — each layer subscribes to a topic, and observations from that layer are tagged with a layer name in the underlying graph so you can later query *only* perception, *only* internal state, etc. ```python from agents.ros import MemLayer # Object detection output from vision component layer1 = MemLayer(subscribes_to=detections_topic) # Introspection output from mllm component layer2 = MemLayer(subscribes_to=introspection_answer) ``` ```{tip} Memory also models **interoception** — internal body state — as a first-class memory dimension. Adding a layer with `is_internal_state=True` (e.g. `MemLayer(subscribes_to=battery_topic, is_internal_state=True)`) routes those observations through `add_body_state` instead of `add`. They're queryable through the dedicated `body_status` tool and surface naturally alongside perception observations in `get_current_context`. We'll exercise this in the [Memory and Cortex](../planning-and-manipulation/cortex-memory.md) recipe. ``` Memory needs the robot's pose (so every observation is tagged with where it was made) and two model clients — one for consolidation summarisation, one for embedding generation: ```python from agents.components import Memory from agents.config import MemoryConfig # Localization input — Memory uses these coordinates directly, no occupancy grid required position = Topic(name="odom", msg_type="Odometry") # Embedding client for vector indexing of every observation embedding_model = OllamaModel( name="embeddings", checkpoint="nomic-embed-text-v2-moe:latest" ) embedding_client = OllamaClient(embedding_model) memory = Memory( layers=[layer1, layer2], position=position, model_client=qwen_client, # used to summarise episodes into gists embedding_client=embedding_client, config=MemoryConfig(db_path="/tmp/robot_memory.db"), trigger=15.0, # flush layer data into memory every 15s component_name="memory", ) ``` That single `Memory` component maintains: - A **typed graph** with four node types (Observation, Episode, Gist, Entity) and six edge types -- so the agent's memory is a structured object, not a flat blob of vectors. - **Tiered storage**, working → short-term → long-term → archived. Observations move through the tiers automatically as time passes; raw text is dropped after archival but the consolidated gist remains searchable. - **Three complementary indexes** sharing the graph: HNSW for semantic search, R-tree for spatial queries, SQLite indexes for temporal queries -- queryable independently or simultaneously. - **Automatic entity merging**: a new detection of "red chair" near a known "red chair" entity is recognised as the same entity rather than a new one, with cosine similarity *and* spatial proximity controlling the merge. You don't see any of this in the recipe — you wire layers in, and the structure emerges. See the [Memory page](../../intelligence/memory.md) for the architecture in detail. --- ## Wrapping Tasks in Episodes The VLM-introspector + detector pair is a good demonstration of layered memory ingestion, but in a real recipe you'd usually want to **bracket** the activity in an *episode*. Episodes are how Memory groups observations into task spans for consolidation: when an episode ends, eMEM clusters the observations made during it, asks the LLM to summarise each cluster into a *gist*, and archives the raw text -- the gist remains fully searchable in long-term memory. `Memory` exposes `start_episode` and `end_episode` as component actions — the simplest way to call them from a recipe is via [Events & Actions](../../concepts/events-and-actions.md). For example, you might trigger `start_episode` whenever the robot enters a new region and `end_episode` when it leaves. We'll show this pattern fully in [Memory and Cortex](../planning-and-manipulation/cortex-memory.md) where Cortex wraps every action task in an episode automatically. For now, every observation we feed in lives in working memory, gets flushed to short-term memory on the trigger schedule, and migrates to long-term as time accumulates. --- ## Launching the Components ```python from agents.ros import Launcher launcher = Launcher() launcher.add_pkg( components=[vision, introspector, memory], package_name="automatika_embodied_agents", multiprocessing=True, ) launcher.bringup() ``` That's it. The robot is now accumulating a structured spatio-temporal memory of everything Vision detects and everything the VLM introspects, indexed by where it happened and when. --- ## Full Recipe Code ```{code-block} python :caption: Spatio-Temporal Memory with Vision and an Introspecting VLM :linenos: from typing import Optional from agents.components import Memory, VLM, Vision from agents.config import MemoryConfig, VisionConfig from agents.models import OllamaModel, VisionModel from agents.clients import OllamaClient, RoboMLRESPClient from agents.ros import FixedInput, Launcher, MemLayer, Topic # --- Vision: object detection --- image0 = Topic(name="image_raw", msg_type="Image") detections_topic = Topic(name="detections", msg_type="Detections") object_detection = VisionModel( name="object_detection", checkpoint="PekingU/rtdetr_r50vd_coco_o365" ) roboml_detection = RoboMLRESPClient(object_detection) vision = Vision( inputs=[image0], outputs=[detections_topic], trigger=image0, config=VisionConfig(threshold=0.5), model_client=roboml_detection, component_name="detection_component", ) # --- VLM: periodic room-type introspection --- qwen_vl = OllamaModel(name="qwen_vl", checkpoint="qwen2.5vl:latest") qwen_client = OllamaClient(qwen_vl) introspection_query = FixedInput( name="introspection_query", msg_type="String", fixed=( "What kind of a room is this? Is it an office, a bedroom or a kitchen? " "Give a one word answer, out of the given choices" ), ) introspection_answer = Topic(name="introspection_answer", msg_type="String") introspector = VLM( inputs=[introspection_query, image0], outputs=[introspection_answer], model_client=qwen_client, trigger=15.0, component_name="introspector", ) def introspection_validation(output: str) -> Optional[str]: for option in ["office", "bedroom", "kitchen"]: if option in output.lower(): return option introspector.add_publisher_preprocessor(introspection_answer, introspection_validation) # --- Memory: graph-backed spatio-temporal store --- embedding_model = OllamaModel( name="embeddings", checkpoint="nomic-embed-text-v2-moe:latest" ) embedding_client = OllamaClient(embedding_model) position = Topic(name="odom", msg_type="Odometry") layer1 = MemLayer(subscribes_to=detections_topic) layer2 = MemLayer(subscribes_to=introspection_answer) memory = Memory( layers=[layer1, layer2], position=position, model_client=qwen_client, embedding_client=embedding_client, config=MemoryConfig(db_path="/tmp/robot_memory.db"), trigger=15.0, component_name="memory", ) # --- Launch --- launcher = Launcher() launcher.add_pkg( components=[vision, introspector, memory], package_name="automatika_embodied_agents", multiprocessing=True, ) launcher.bringup() ``` --- ## Where next - {doc}`Memory and Cortex <../planning-and-manipulation/cortex-memory>` — once you've built a memory, *reason* over it. Cortex auto-discovers all of Memory's retrieval tools and answers questions in plain English: *"where did you last see the cat?"*, *"summarise the last episode"*, *"is the kitchen messy right now?"*. - {doc}`Cortex: The Agentic Harness <../planning-and-manipulation/cortex-agent>` — the Cortex introduction, if you haven't met it yet. - {doc}`Memory concept page <../../intelligence/memory>` — the full architectural reference for eMEM: nodes, edges, tiers, consolidation, the ten retrieval tools. --- ```{tip} **Promote this recipe to production.** While you're shaping it, the script runs straight with `python recipe.py`. Once it's solid, drop it at `~/emos/recipes//recipe.py` and run `emos run ` -- you'll get sensor pre-flight checks, persistent logs, and a card on the dashboard so an operator can launch it from a browser. See [Running Recipes](../../getting-started/running-recipes.md) for the full development-vs-production comparison and install-mode pitfalls (especially in Container mode). ``` ``` ## File: recipes/foundation/goto-navigation.md ```markdown # GoTo Navigation In the previous [recipe](semantic-map.md) we built a graph-backed spatio-temporal memory using the `Memory` component. Memory tags every observation with the robot's pose, so for any concept the agent has encountered (an object class, a room label) we can ask Memory: *where did we see this?* It already exposes that lookup as a component action called `locate`, returning a centroid plus a radius for the most likely region. In this recipe we wire `locate` into a Go-to-X component so that a command like *"Go to the kitchen"* turns into a `PoseStamped` goal point that the navigation stack can consume. We do this by **registering Memory's `locate` tool on an LLM** -- the LLM decides when to call the tool, Memory answers, and a small preprocessor converts the textual answer into a numpy coordinate that gets published as the goal. ```{seealso} For the conceptual reference of Memory's full retrieval surface, see the [Memory page](../../intelligence/memory.md). For a generic introduction to LLM tool calling that is not tied to navigation, see the next recipe, [Tool Calling](tool-calling.md). ``` ```{admonition} Prerequisites :class: important Memory needs the [eMEM](https://github.com/automatika-robotics/emem) package, which `emos install` does not add — the install command depends on your mode (see [Memory installation](../../intelligence/memory.md)). ``` --- ## What we're building Three components in a single launcher: | Component | Role | |---|---| | **Vision** | Object detector publishing `Detections` per frame. Feeds Memory. | | **Memory** | Graph-backed spatio-temporal memory ingesting detections; tags each observation with the robot's pose from `/odom`. | | **goto LLM** | A plain `LLM` component that takes a free-form *"go to X"* query, calls Memory's `locate` tool to look up the place, and publishes the resulting coordinates on `goal_point`. | Memory's `locate` returns a textual answer like `"... Location: (10.3, 9.8, 0.0) ..."` (centroid plus a description). We wire a small regex preprocessor onto the goto LLM's output topic that picks the centroid out and converts it to an `np.ndarray`, which the framework publishes as the `PoseStamped` goal point. --- ## Step 1: Vision and Memory ```python from agents.clients import OllamaClient, RoboMLRESPClient from agents.components import Memory, Vision from agents.config import MemoryConfig, VisionConfig from agents.models import OllamaModel, VisionModel from agents.ros import Launcher, MemLayer, Topic image0 = Topic(name="image_raw", msg_type="Image") detections_topic = Topic(name="detections", msg_type="Detections") position = Topic(name="odom", msg_type="Odometry") vision = Vision( inputs=[image0], outputs=[detections_topic], trigger=image0, config=VisionConfig(threshold=0.5), model_client=RoboMLRESPClient( VisionModel(name="rtdetr", checkpoint="PekingU/rtdetr_r50vd_coco_o365") ), component_name="vision", ) embedding_client = OllamaClient( OllamaModel(name="embeddings", checkpoint="nomic-embed-text-v2-moe:latest") ) memory = Memory( layers=[MemLayer(subscribes_to=detections_topic)], position=position, embedding_client=embedding_client, config=MemoryConfig(db_path="/tmp/go_to_x.db"), trigger=10.0, component_name="memory", ) ``` Each detection becomes an `ObservationNode` in Memory tagged with the robot's pose at the moment the detection was made. After a few minutes of accumulation, eMEM's entity layer auto-merges nearby semantically-similar detections into persistent entities, and `locate("chair")` returns the centroid of the merged "chair" entity. --- ## Step 2: The Go-to-X LLM ```python from agents.components import LLM from agents.config import LLMConfig qwen = OllamaModel(name="qwen", checkpoint="qwen3.5:latest") qwen_client = OllamaClient(qwen) goto_in = Topic(name="goto_in", msg_type="String") goal_point = Topic(name="goal_point", msg_type="PoseStamped") goto = LLM( inputs=[goto_in], outputs=[goal_point], model_client=qwen_client, trigger=goto_in, config=LLMConfig(), component_name="go_to_x", ) goto.set_component_prompt( template=( "The user asks you to go to a place. Use the available tools to " "look up the place's location in memory. Pass the place name to " "the locate tool as the ``concept`` argument. User said: {{goto_in}}" ) ) ``` --- ## Step 3: Register Memory's `locate` tool on the LLM ```python memory.register_tools_on(goto, tools=["locate"], send_tool_response_to_model=False) ``` `register_tools_on` exposes Memory's component actions to the goto LLM as callable tools. We register a single tool, `locate`, since the LLM only needs the lookup capability for this recipe. The flag `send_tool_response_to_model=False` is what makes the recipe end-to-end: instead of feeding `locate`'s answer back into the LLM for a follow-up generation, the answer becomes the *output of the LLM component*. After preprocessing, that output is what gets published on `goal_point`. --- ## Step 4: Parse Memory's textual answer into coordinates `locate` returns text formatted like: ``` Location: (10.3, 9.8, 0.0) Radius: 1.5m Based on: 5 memories (3x detections, 2x scene) [detections] kitchen counter with cups and a kettle... [scene] open kitchen area near the pantry... [detections] kitchen island with stools... ``` We register a small preprocessor on the `goal_point` output topic that pulls the centroid out of that text and converts it to an `np.ndarray`. The framework then publishes the array as a `PoseStamped`. ```python import re from typing import Optional import numpy as np _LOCATION_RE = re.compile(r"Location:\s*\(([^)]+)\)") def locate_text_to_goal_point(output: str) -> Optional[np.ndarray]: """Pull the centroid coordinates out of Memory.locate's text output.""" match = _LOCATION_RE.search(output) if not match: return # no match → nothing to publish try: coords = np.fromstring(match.group(1), sep=",", dtype=np.float64) except ValueError: return if coords.shape[0] == 2: coords = np.append(coords, 0.0) if coords.shape[0] != 3: return return coords goto.add_publisher_preprocessor(goal_point, locate_text_to_goal_point) ``` If the LLM (correctly) called `locate`, the regex matches, the coordinates parse cleanly, and the goal point is published. If the LLM hallucinated or the place is unknown to Memory, the preprocessor returns `None` and **nothing is published** -- the navigation stack sees no spurious goal. --- ## Step 5: Launch ```python launcher = Launcher() launcher.add_pkg(components=[vision, memory, goto]) launcher.bringup() ``` --- ## Full recipe code ```{code-block} python :caption: Go-to-X with Memory tool calling :linenos: import re from typing import Optional import numpy as np from agents.clients import OllamaClient, RoboMLRESPClient from agents.components import LLM, Memory, Vision from agents.config import LLMConfig, MemoryConfig, VisionConfig from agents.models import OllamaModel, VisionModel from agents.ros import Launcher, MemLayer, Topic # -- Perception side: vision + memory -- image0 = Topic(name="image_raw", msg_type="Image") detections_topic = Topic(name="detections", msg_type="Detections") position = Topic(name="odom", msg_type="Odometry") vision = Vision( inputs=[image0], outputs=[detections_topic], trigger=image0, config=VisionConfig(threshold=0.5), model_client=RoboMLRESPClient( VisionModel(name="rtdetr", checkpoint="PekingU/rtdetr_r50vd_coco_o365") ), component_name="vision", ) embedding_client = OllamaClient( OllamaModel(name="embeddings", checkpoint="nomic-embed-text-v2-moe:latest") ) memory = Memory( layers=[MemLayer(subscribes_to=detections_topic)], position=position, embedding_client=embedding_client, config=MemoryConfig(db_path="/tmp/go_to_x.db"), trigger=10.0, component_name="memory", ) # -- Go-to-X LLM -- qwen = OllamaModel(name="qwen", checkpoint="qwen3.5:latest") qwen_client = OllamaClient(qwen) goto_in = Topic(name="goto_in", msg_type="String") goal_point = Topic(name="goal_point", msg_type="PoseStamped") goto = LLM( inputs=[goto_in], outputs=[goal_point], model_client=qwen_client, trigger=goto_in, config=LLMConfig(), component_name="go_to_x", ) goto.set_component_prompt( template=( "The user asks you to go to a place. Use the available tools to " "look up the place's location in memory. Pass the place name to " "the locate tool as the ``concept`` argument. User said: {{goto_in}}" ) ) memory.register_tools_on(goto, tools=["locate"], send_tool_response_to_model=False) _LOCATION_RE = re.compile(r"Location:\s*\(([^)]+)\)") def locate_text_to_goal_point(output: str) -> Optional[np.ndarray]: """Pull the centroid coordinates out of Memory.locate's text output.""" match = _LOCATION_RE.search(output) if not match: return try: coords = np.fromstring(match.group(1), sep=",", dtype=np.float64) except ValueError: return if coords.shape[0] == 2: coords = np.append(coords, 0.0) if coords.shape[0] != 3: return return coords goto.add_publisher_preprocessor(goal_point, locate_text_to_goal_point) # -- Launch (single process so the LLM can call Memory in-process) -- launcher = Launcher() launcher.add_pkg(components=[vision, memory, goto]) launcher.bringup() ``` --- ## Where next - {doc}`Tool Calling ` — generalises this pattern. Instead of registering Memory's pre-defined tools, write your own Python function as a custom tool and register it with `goto.register_tool(...)`. - {doc}`Complete Agent ` — drops this Go-to-X pattern into a full multi-modal agent (speech I/O + vision + memory + Q&A + routing) defined in one Python script. - {doc}`Cortex Driving the Full Stack <../planning-and-manipulation/cortex-navigation>` — the agentic-harness version: drop a [Cortex](../../intelligence/cortex.md) component on top of Memory and the navigation stack and the robot handles compound natural-language goals like *"go to the kitchen and tell me what's on the counter"* with no orchestration code from you. --- ```{tip} **Promote this recipe to production.** While you're shaping it, the script runs straight with `python recipe.py`. Once it's solid, drop it at `~/emos/recipes//recipe.py` and run `emos run ` -- you'll get sensor pre-flight checks, persistent logs, and a card on the dashboard so an operator can launch it from a browser. See [Running Recipes](../../getting-started/running-recipes.md) for the full development-vs-production comparison and install-mode pitfalls (especially in Container mode). ``` ``` ## File: recipes/foundation/tool-calling.md ```markdown # Tool Calling In the [previous recipe](goto-navigation.md) we registered one of Memory's built-in tools (`locate`) on an LLM and let the LLM decide when to call it. That's the most direct path when the tool you want is already a `@component_action` on a managed component. **In this recipe we generalise the pattern**: we write our own Python function and register it as a tool on the LLM. The function can do anything Python can do -- read a sensor, hit a web service, toggle a piece of hardware, run a calculation -- and the LLM picks it up as a callable tool with the schema we declare. The key idea: when the LLM is configured with one or more registered tools, it produces *structured tool calls* instead of raw text. A tool call has a name and JSON-serialisable arguments that match the tool's declared schema. The framework dispatches the call into our Python function, and we choose what happens to the return value -- either: - `send_tool_response_to_model=True` — the function's return is fed back to the LLM as a tool result, and the LLM produces a follow-up generation that uses it. The right mode when you want the LLM to *react* to the tool's answer. - `send_tool_response_to_model=False` — the function's return value *is* the LLM component's output, and gets published on the output topic directly. The right mode when you just want the LLM to extract structured arguments and you handle the rest. This was what we used in the [GoTo Navigation](goto-navigation.md) recipe. Tool calling works with any client that supports it -- the `OllamaClient` and the `GenericHTTPClient` both do, as long as the underlying model is tool-trained. --- ## What we're building A tiny voice-of-the-world agent: an LLM component that can answer questions about the **current weather anywhere on Earth** by calling a custom function that hits the public [Open-Meteo](https://open-meteo.com) API. Weather is the canonical case for tool calling -- the LLM categorically can't know it from training data, so the tool earns its place by giving the model live information it would otherwise have to invent. | Component | Role | |---|---| | **LLM** | Receives a free-form question, decides whether to call our `get_weather` tool, and composes a natural-language reply. | | **`get_weather` tool** | A regular Python function that geocodes a city name, queries Open-Meteo, and returns the current conditions. | This shows the *full* tool-calling loop: tool result back to LLM → LLM phrases an answer in natural language. Anywhere we'd like the LLM to *use* live data instead of hallucinating it -- a sensor reading, a database row, a web fetch, a sub-process -- follows the same shape. --- ## Step 1: The LLM A plain `LLM` component using a tool-capable Ollama model. ```python from agents.clients import OllamaClient from agents.components import LLM from agents.config import LLMConfig from agents.models import OllamaModel from agents.ros import Launcher, Topic qwen = OllamaModel(name="qwen", checkpoint="qwen3.5:latest") qwen_client = OllamaClient(qwen) question = Topic(name="question", msg_type="String") answer = Topic(name="answer", msg_type="String") assistant = LLM( inputs=[question], outputs=[answer], model_client=qwen_client, trigger=question, config=LLMConfig(), component_name="assistant", ) assistant.set_component_prompt( template=( "You are a friendly assistant. If the user asks about the current " "weather, temperature, or wind in a location, call the " "``get_weather`` tool with the city name and answer using the data " "it returns. The user said: {{question}}" ) ) ``` The prompt nudges the model toward the tool when the question is weather-related. For unrelated questions the LLM will just answer directly without calling anything. --- ## Step 2: Define the tool A regular Python function that takes a city name, geocodes it, and queries the Open-Meteo current-conditions endpoint. No API key required. ```python from typing import Dict, Union import httpx def get_weather(city: str) -> Dict[str, Union[str, float]]: """Look up the current weather for *city* via the Open-Meteo public API. Returns a dict with the city's resolved name, temperature, and wind speed -- or ``{"error": "..."}`` if the city couldn't be resolved. """ # Geocoding: city name -> (lat, lon) geo = httpx.get( "https://geocoding-api.open-meteo.com/v1/search", params={"name": city, "count": 1}, timeout=10.0, ).json() if not geo.get("results"): return {"error": f"could not find location '{city}'"} place = geo["results"][0] lat, lon = place["latitude"], place["longitude"] # Current weather at those coordinates weather = httpx.get( "https://api.open-meteo.com/v1/forecast", params={ "latitude": lat, "longitude": lon, "current": "temperature_2m,wind_speed_10m", }, timeout=10.0, ).json()["current"] return { "city": place.get("name", city), "country": place.get("country", ""), "temperature_c": weather["temperature_2m"], "wind_speed_kmh": weather["wind_speed_10m"], } ``` `httpx` is already a dependency of EMOS, so no extra install. The function: - takes one string argument (`city`), - does its own work (two HTTP calls -- geocoding, then weather), - returns a small dict the LLM can phrase prose around. --- ## Step 3: Declare the tool's schema The LLM needs an OpenAI-format description so it knows the tool's name, what it does, and what arguments to provide. ```python get_weather_description = { "type": "function", "function": { "name": "get_weather", "description": ( "Look up the current weather (temperature in Celsius and wind " "speed in km/h) for a given city. Call this whenever the user " "asks about live weather conditions anywhere in the world." ), "parameters": { "type": "object", "properties": { "city": { "type": "string", "description": ( "City name, optionally with country (e.g. 'Berlin', " "'Tokyo', 'Paris, France')." ), }, }, "required": ["city"], }, }, } ``` Spend the description budget on telling the model **when** to call the tool and what the arguments mean. The LLM doesn't see the function body; it only sees the description. --- ## Step 4: Register the tool ```python assistant.register_tool( tool=get_weather, tool_description=get_weather_description, send_tool_response_to_model=True, ) ``` `send_tool_response_to_model=True` is what makes this recipe a conversational loop: 1. User asks *"what's the weather in Berlin right now?"*. 2. LLM emits a structured `get_weather(city="Berlin")` tool call. 3. Framework dispatches the call, the function geocodes Berlin and hits Open-Meteo. 4. The result -- `{"city": "Berlin", "country": "Germany", "temperature_c": 18.3, "wind_speed_kmh": 11.5}` -- is fed back into the LLM as a tool message. 5. The LLM produces the final user-facing reply: *"It's currently 18°C in Berlin with a light breeze around 12 km/h."* 6. That reply gets published on `answer`. If the user asks something off-topic (*"what's the capital of Peru?"*), the LLM never calls the tool and answers from its own knowledge. --- ## Step 5: Launch ```python launcher = Launcher() launcher.add_pkg(components=[assistant]) launcher.bringup() ``` --- ## Full recipe code ```{code-block} python :caption: Tool calling on an LLM with a custom weather-lookup tool :linenos: from typing import Dict, Union import httpx from agents.clients import OllamaClient from agents.components import LLM from agents.config import LLMConfig from agents.models import OllamaModel from agents.ros import Launcher, Topic # -- LLM -- qwen = OllamaModel(name="qwen", checkpoint="qwen3.5:latest") qwen_client = OllamaClient(qwen) question = Topic(name="question", msg_type="String") answer = Topic(name="answer", msg_type="String") assistant = LLM( inputs=[question], outputs=[answer], model_client=qwen_client, trigger=question, config=LLMConfig(), component_name="assistant", ) assistant.set_component_prompt( template=( "You are a friendly assistant. If the user asks about the current " "weather, temperature, or wind in a location, call the " "``get_weather`` tool with the city name and answer using the data " "it returns. The user said: {{question}}" ) ) # -- Custom tool: live weather via Open-Meteo -- def get_weather(city: str) -> Dict[str, Union[str, float]]: """Look up the current weather for *city* via the Open-Meteo public API.""" geo = httpx.get( "https://geocoding-api.open-meteo.com/v1/search", params={"name": city, "count": 1}, timeout=10.0, ).json() if not geo.get("results"): return {"error": f"could not find location '{city}'"} place = geo["results"][0] lat, lon = place["latitude"], place["longitude"] weather = httpx.get( "https://api.open-meteo.com/v1/forecast", params={ "latitude": lat, "longitude": lon, "current": "temperature_2m,wind_speed_10m", }, timeout=10.0, ).json()["current"] return { "city": place.get("name", city), "country": place.get("country", ""), "temperature_c": weather["temperature_2m"], "wind_speed_kmh": weather["wind_speed_10m"], } get_weather_description = { "type": "function", "function": { "name": "get_weather", "description": ( "Look up the current weather (temperature in Celsius and wind " "speed in km/h) for a given city. Call this whenever the user " "asks about live weather conditions anywhere in the world." ), "parameters": { "type": "object", "properties": { "city": { "type": "string", "description": ( "City name, optionally with country (e.g. 'Berlin', " "'Tokyo', 'Paris, France')." ), }, }, "required": ["city"], }, }, } assistant.register_tool( tool=get_weather, tool_description=get_weather_description, send_tool_response_to_model=True, ) # -- Launch -- launcher = Launcher() launcher.add_pkg(components=[assistant]) launcher.bringup() ``` --- ## When to use which pattern | Need | Pattern | |---|---| | Use a tool that's already a `@component_action` on a managed component, with its existing schema. | `memory.register_tools_on(llm, tools=[...])` from [GoTo Navigation](goto-navigation.md). | | Custom Python function as the tool, with a schema you define. | `llm.register_tool(your_function, your_description, send_tool_response_to_model=...)` -- this recipe. | | Multiple capabilities orchestrated by an LLM that decides what to do. | A [Cortex](../../intelligence/cortex.md) component, which auto-discovers every `@component_action` on every managed component as a tool with no per-tool registration. | And within the custom-function pattern itself: | `send_tool_response_to_model` | When | |---|---| | `True` (this recipe) | The LLM should *react* to the tool's output -- compose a sentence, decide what to do next, ask a follow-up. Conversational and reasoning agents. | | `False` ([GoTo Navigation](goto-navigation.md)) | The tool's return value is what should be published. You're using the LLM to extract structured arguments, and the function does the real work. | --- ```{tip} **Promote this recipe to production.** While you're shaping it, the script runs straight with `python recipe.py`. Once it's solid, drop it at `~/emos/recipes//recipe.py` and run `emos run ` -- you'll get sensor pre-flight checks, persistent logs, and a card on the dashboard so an operator can launch it from a browser. See [Running Recipes](../../getting-started/running-recipes.md) for the full development-vs-production comparison and install-mode pitfalls (especially in Container mode). ``` ``` ## File: recipes/foundation/semantic-routing.md ```markdown # Semantic Routing The SemanticRouter component in EMOS allows you to route text queries to specific [components](../../intelligence/ai-components.md) based on the user's intent or the output of a preceding component. The router operates in two distinct modes: 1. **Vector Mode (Default):** This mode uses a Vector DB to calculate the mathematical similarity (distance) between the incoming query and the samples defined in your routes. It is extremely fast and lightweight. 2. **LLM Mode (Agentic):** This mode uses an LLM to intelligently analyze the intent of the query and triggers routes accordingly. This is more computationally expensive but can handle complex nuances, context, and negation (e.g., "Don't go to the kitchen" might be routed differently by an agent than a simple vector similarity search). In this recipe, we will route queries between two components: a General Purpose LLM (for chatting) and a Go-to-X Component (for navigation commands) that we built in the previous [recipe](goto-navigation.md). The Go-to-X component resolves a place name by calling the `Memory` component's `locate` tool, so we set up `Vision` and `Memory` here too. Lets start by setting up our components. ## Setting up the components In the following code snippet we will set up the perception side (`Vision` + `Memory`, which builds the spatial map) and our two query components. ```python import re from typing import Optional import numpy as np from agents.components import LLM, Memory, Vision from agents.models import OllamaModel, VisionModel from agents.vectordbs import ChromaDB from agents.config import LLMConfig, MemoryConfig, VisionConfig from agents.clients import ChromaClient, OllamaClient, RoboMLRESPClient from agents.ros import Launcher, Topic, Route, MemLayer # Reuse one (tool-capable) model for the generic LLM and the Go-to-X LLM qwen = OllamaModel(name="qwen", checkpoint="qwen3.5:latest") qwen_client = OllamaClient(qwen) # Embeddings for Memory (the spatial map) embedding_client = OllamaClient( OllamaModel(name="embeddings", checkpoint="nomic-embed-text-v2-moe:latest") ) # Vector DB for the SemanticRouter — it stores the route samples chroma = ChromaDB() chroma_client = ChromaClient(db=chroma) # -- Perception: vision + memory build the map -- image0 = Topic(name="image_raw", msg_type="Image") detections_topic = Topic(name="detections", msg_type="Detections") position = Topic(name="odom", msg_type="Odometry") vision = Vision( inputs=[image0], outputs=[detections_topic], trigger=image0, config=VisionConfig(threshold=0.5), model_client=RoboMLRESPClient( VisionModel(name="rtdetr", checkpoint="PekingU/rtdetr_r50vd_coco_o365") ), component_name="vision", ) memory = Memory( layers=[MemLayer(subscribes_to=detections_topic)], position=position, embedding_client=embedding_client, config=MemoryConfig(db_path="/tmp/go_to_x.db"), trigger=10.0, component_name="memory", ) # Make a generic LLM component for general questions llm_in = Topic(name="text_in_llm", msg_type="String") llm_out = Topic(name="text_out_llm", msg_type="String") llm = LLM( inputs=[llm_in], outputs=[llm_out], model_client=qwen_client, trigger=llm_in, component_name="generic_llm", ) # Make a Go-to-X component — it looks places up via Memory's `locate` tool goto_in = Topic(name="goto_in", msg_type="String") goal_point = Topic(name="goal_point", msg_type="PoseStamped") goto = LLM( inputs=[goto_in], outputs=[goal_point], model_client=qwen_client, trigger=goto_in, config=LLMConfig(), component_name="go_to_x", ) goto.set_component_prompt( template=( "The user asks you to go to a place. Use the available tools to " "look up the place's location in memory. Pass the place name to " "the locate tool as the ``concept`` argument. User said: {{goto_in}}" ) ) # Register Memory's `locate` tool on the Go-to-X LLM so it can be called memory.register_tools_on(goto, tools=["locate"], send_tool_response_to_model=False) # pre-process the output before publishing to a topic of msg_type PoseStamped _LOCATION_RE = re.compile(r"Location:\s*\(([^)]+)\)") def locate_text_to_goal_point(output: str) -> Optional[np.ndarray]: """Pull the centroid coordinates out of Memory.locate's text output.""" match = _LOCATION_RE.search(output) if not match: return try: coords = np.fromstring(match.group(1), sep=",", dtype=np.float64) except ValueError: return if coords.shape[0] == 2: coords = np.append(coords, 0.0) if coords.shape[0] != 3: return return coords # add the pre-processing function to the goal_point output topic goto.add_publisher_preprocessor(goal_point, locate_text_to_goal_point) ``` ```{note} We reused the same model and its client for both query components. The model must support tool calling, since the Go-to-X component calls Memory's `locate` tool. ``` ```{note} For a detailed explanation of the Go-to-X component — how `Memory` builds the map and exposes the `locate` tool — check the previous [recipe](goto-navigation.md). ``` ```{important} The Go-to-X LLM calls `Memory` **in-process**, so the router, the LLMs, and Memory must launch in the same process (no `multiprocessing=True` on `add_pkg`). ``` ## Creating the SemanticRouter The SemanticRouter takes an input _String_ topic and sends whatever is published on that topic to a _Route_. A _Route_ is a thin wrapper around _Topic_ and takes in the name of a topic to publish on and example queries, that would match a potential query that should be published to a particular topic. For example, if we ask our robot a general question, like "Whats the capital of France?", we do not want that question to be routed to a Go-to-X component, but to a generic LLM. Thus in its route, we would provide examples of general questions. Lets start by creating our routes for the input topics of the two components above. ```python from agents.ros import Route # Create the input topic for the router query_topic = Topic(name="question", msg_type="String") # Define a route to a topic that processes go-to-x commands goto_route = Route(routes_to=goto_in, samples=["Go to the door", "Go to the kitchen", "Get me a glass", "Fetch a ball", "Go to hallway"]) # Define a route to a topic that is input to an LLM component llm_route = Route(routes_to=llm_in, samples=["What is the capital of France?", "Is there life on Mars?", "How many tablespoons in a cup?", "How are you today?", "Whats up?"]) ``` ```{note} The `routes_to` parameter of a `Route` can be a `Topic` or an `Action`. `Actions` can be system level functions (e.g. to restart a component), functions exposed by components (e.g. to start the VLA component for manipulation, or the 'say' method in TextToSpeech component) or arbitrary functions written in the recipe. `Actions` are a powerful concept in EMOS, because their arguments can come from any topic in the system. To learn more, check out [Events & Actions](../../concepts/events-and-actions.md). ``` ## Option 1: Vector Mode (Similarity) This is the standard approach. In Vector mode, the SemanticRouter component stores the route samples in a vector DB. Distance is calculated between an incoming query's embedding and the embeddings of the example queries to determine which _Route_(_Topic_) the query should be sent on. We pass the `chroma_client` set up above as the `db_client`, and specify a router name in the config, which acts as a _collection_name_ in the database. ```python from agents.components import SemanticRouter from agents.config import SemanticRouterConfig router_config = SemanticRouterConfig(router_name="go-to-router", distance_func="l2") # Initialize the router component router = SemanticRouter( inputs=[query_topic], routes=[llm_route, goto_route], default_route=llm_route, # If none of the routes fall within a distance threshold config=router_config, db_client=chroma_client, # Providing db_client enables Vector Mode component_name="router" ) ``` ## Option 2: LLM Mode (Agentic) Alternatively, we can use an LLM to make routing decisions. This is useful if your routes require "understanding" rather than just similarity. We simply provide a `model_client` instead of a `db_client` (no ChromaDB needed in this mode). ```{note} We can even use the same LLM (`model_client`) as we are using for our other Q&A components. ``` ```python # No SemanticRouterConfig needed, we can use LLMConfig or let it be default router = SemanticRouter( inputs=[query_topic], routes=[llm_route, goto_route], model_client=qwen_client, # Providing model_client enables LLM Mode component_name="smart_router" ) ``` And that is it. Whenever something is published on the input topic **question**, it will be routed, either to a Go-to-X component or an LLM component. We can now expose this topic to our command interface. The complete code for setting up the router is given below: ```{code-block} python :caption: Semantic Routing :linenos: import re from typing import Optional import numpy as np from agents.components import LLM, Memory, SemanticRouter, Vision from agents.models import OllamaModel, VisionModel from agents.vectordbs import ChromaDB from agents.config import LLMConfig, MemoryConfig, SemanticRouterConfig, VisionConfig from agents.clients import ChromaClient, OllamaClient, RoboMLRESPClient from agents.ros import Launcher, Topic, Route, MemLayer # Reuse one (tool-capable) model for the generic LLM and the Go-to-X LLM qwen = OllamaModel(name="qwen", checkpoint="qwen3.5:latest") qwen_client = OllamaClient(qwen) # Embeddings for Memory (the spatial map) embedding_client = OllamaClient( OllamaModel(name="embeddings", checkpoint="nomic-embed-text-v2-moe:latest") ) # Vector DB for the SemanticRouter — it stores the route samples chroma = ChromaDB() chroma_client = ChromaClient(db=chroma) # -- Perception: vision + memory build the map -- image0 = Topic(name="image_raw", msg_type="Image") detections_topic = Topic(name="detections", msg_type="Detections") position = Topic(name="odom", msg_type="Odometry") vision = Vision( inputs=[image0], outputs=[detections_topic], trigger=image0, config=VisionConfig(threshold=0.5), model_client=RoboMLRESPClient( VisionModel(name="rtdetr", checkpoint="PekingU/rtdetr_r50vd_coco_o365") ), component_name="vision", ) memory = Memory( layers=[MemLayer(subscribes_to=detections_topic)], position=position, embedding_client=embedding_client, config=MemoryConfig(db_path="/tmp/go_to_x.db"), trigger=10.0, component_name="memory", ) # Make a generic LLM component for general questions llm_in = Topic(name="text_in_llm", msg_type="String") llm_out = Topic(name="text_out_llm", msg_type="String") llm = LLM( inputs=[llm_in], outputs=[llm_out], model_client=qwen_client, trigger=llm_in, component_name="generic_llm", ) # Make a Go-to-X component — it looks places up via Memory's `locate` tool goto_in = Topic(name="goto_in", msg_type="String") goal_point = Topic(name="goal_point", msg_type="PoseStamped") goto = LLM( inputs=[goto_in], outputs=[goal_point], model_client=qwen_client, trigger=goto_in, config=LLMConfig(), component_name="go_to_x", ) goto.set_component_prompt( template=( "The user asks you to go to a place. Use the available tools to " "look up the place's location in memory. Pass the place name to " "the locate tool as the ``concept`` argument. User said: {{goto_in}}" ) ) # Register Memory's `locate` tool on the Go-to-X LLM so it can be called memory.register_tools_on(goto, tools=["locate"], send_tool_response_to_model=False) # pre-process the output before publishing to a topic of msg_type PoseStamped _LOCATION_RE = re.compile(r"Location:\s*\(([^)]+)\)") def locate_text_to_goal_point(output: str) -> Optional[np.ndarray]: """Pull the centroid coordinates out of Memory.locate's text output.""" match = _LOCATION_RE.search(output) if not match: return try: coords = np.fromstring(match.group(1), sep=",", dtype=np.float64) except ValueError: return if coords.shape[0] == 2: coords = np.append(coords, 0.0) if coords.shape[0] != 3: return return coords # add the pre-processing function to the goal_point output topic goto.add_publisher_preprocessor(goal_point, locate_text_to_goal_point) # Create the input topic for the router query_topic = Topic(name="question", msg_type="String") # Define a route to a topic that processes go-to-x commands goto_route = Route( routes_to=goto_in, samples=[ "Go to the door", "Go to the kitchen", "Get me a glass", "Fetch a ball", "Go to hallway", ], ) # Define a route to a topic that is input to an LLM component llm_route = Route( routes_to=llm_in, samples=[ "What is the capital of France?", "Is there life on Mars?", "How many tablespoons in a cup?", "How are you today?", "Whats up?", ], ) # --- MODE 1: VECTOR ROUTING (Active) --- router_config = SemanticRouterConfig(router_name="go-to-router", distance_func="l2") router = SemanticRouter( inputs=[query_topic], routes=[llm_route, goto_route], default_route=llm_route, config=router_config, db_client=chroma_client, # Vector mode requires db_client component_name="router", ) # --- MODE 2: LLM ROUTING (Commented Out) --- # To use LLM routing (Agentic), comment out the block above and uncomment this: # # router = SemanticRouter( # inputs=[query_topic], # routes=[llm_route, goto_route], # default_route=llm_route, # model_client=qwen_client, # LLM mode requires model_client # component_name="router", # ) # Launch the components — single process so the Go-to-X LLM can call Memory in-process launcher = Launcher() launcher.add_pkg(components=[vision, memory, llm, goto, router]) launcher.bringup() ``` --- ```{tip} **Promote this recipe to production.** While you're shaping it, the script runs straight with `python recipe.py`. Once it's solid, drop it at `~/emos/recipes//recipe.py` and run `emos run ` -- you'll get sensor pre-flight checks, persistent logs, and a card on the dashboard so an operator can launch it from a browser. See [Running Recipes](../../getting-started/running-recipes.md) for the full development-vs-production comparison and install-mode pitfalls (especially in Container mode). ``` ``` ## File: recipes/foundation/complete-agent.md ```markdown # Complete Agent This is the capstone recipe. Everything we have built in the previous tutorials -- conversational interfaces, prompt engineering, spatio-temporal memory, memory-aware navigation, and semantic routing -- comes together here into a single EMOS recipe: a fully capable embodied agent defined in one Python script. This is what EMOS is designed for. Instead of stitching together dozens of ROS nodes, launch files, and custom middleware, you define a complete agentic workflow as a graph of [Components](../../intelligence/ai-components.md) connected through [Topics](../../concepts/topics.md), and bring it up with a single call. The result is a robot that can listen, see, think, remember, navigate, and speak -- all orchestrated by EMOS. ```{seealso} For the multiprocessing-and-fault-tolerant variant of this recipe, see [Multiprocessing & Fault Tolerance](../events-and-resilience/multiprocessing.md). For the agentic-harness variant where a single [Cortex](../../intelligence/cortex.md) component takes charge of an entire graph like this, see [Memory and Cortex](../planning-and-manipulation/cortex-memory.md) and [Cortex Driving the Full Stack](../planning-and-manipulation/cortex-navigation.md). ``` ```{admonition} Prerequisites :class: important This recipe uses the `Memory` component for spatio-temporal memory. Memory needs the [eMEM](https://github.com/automatika-robotics/emem) package, which `emos install` does not add — the install command depends on your mode (see [Memory installation](../../intelligence/memory.md)). Audio playback also needs `pip install soundfile sounddevice`. ``` ## The Graph ```{mermaid} flowchart LR %% --- External I/O --- query([query]) Kompass([Kompass]) %% --- Speech I/O --- speech_to_text[speech_to_text]:::component text_to_speech[text_to_speech]:::component Whisper[Whisper] TransformersTTS[TransformersTTS] %% --- Model / DB backends --- ChromaDB[ChromaDB] %% --- Routing --- router[router]:::component %% --- Vision --- object_detection[object_detection]:::component RT_DETR[RT-DETR] %% --- VLM (VQA + introspection) --- visual_q_and_a[visual_q_and_a]:::component introspector[introspector]:::component qwen_vl[qwen_vl Ollama] %% --- LLM brains --- general_q_and_a[general_q_and_a]:::component go_to_x[go_to_x]:::component qwen[qwen Ollama] %% --- Memory --- memory[memory]:::component embeddings[embeddings Ollama] %% --- Wiring: input → router → routes --- query --> speech_to_text Whisper <--> speech_to_text speech_to_text --> router router <--> ChromaDB router --> visual_q_and_a router --> general_q_and_a router --> go_to_x %% --- Vision feeds VQA + memory --- RT_DETR <--> object_detection object_detection --> visual_q_and_a object_detection --> memory %% --- VLM clients --- qwen_vl <--> visual_q_and_a qwen_vl <--> introspector introspector --> memory %% --- LLM clients --- qwen <--> general_q_and_a qwen <--> go_to_x qwen <--> memory %% --- Memory's own backbone --- embeddings <--> memory memory --> go_to_x %% --- Outputs back to speech / navigation --- visual_q_and_a --> text_to_speech general_q_and_a --> text_to_speech TransformersTTS <--> text_to_speech go_to_x --> Kompass classDef component fill:#e07a7a,stroke:#a64545,stroke-width:1.5px,color:#000000 ``` Rectangular boxes are EMOS components, their model backends, and the embedded ChromaDB the router uses for route-embedding lookups. The rounded nodes are the things outside this recipe -- the user's input on one side and the Kompass navigation stack on the other. ## The Complete Recipe ```python import re from typing import Optional import numpy as np from agents.clients import ( ChromaClient, OllamaClient, RoboMLHTTPClient, RoboMLRESPClient, ) from agents.components import ( LLM, VLM, Memory, SemanticRouter, SpeechToText, TextToSpeech, Vision, ) from agents.config import ( LLMConfig, MemoryConfig, SemanticRouterConfig, TextToSpeechConfig, VisionConfig, ) from agents.models import OllamaModel, TransformersTTS, VisionModel, Whisper from agents.ros import FixedInput, Launcher, MemLayer, Route, Topic from agents.vectordbs import ChromaDB ### Models and shared clients ### whisper_client = RoboMLHTTPClient(Whisper(name="whisper")) tts_client = RoboMLHTTPClient(TransformersTTS(name="tts")) detection_client = RoboMLRESPClient( VisionModel(name="rtdetr", checkpoint="PekingU/rtdetr_r50vd_coco_o365") ) qwen_vl_client = OllamaClient( OllamaModel(name="qwen_vl", checkpoint="qwen2.5vl:latest") ) qwen_client = OllamaClient(OllamaModel(name="qwen", checkpoint="qwen3:0.6b")) embedding_client = OllamaClient( OllamaModel(name="embeddings", checkpoint="nomic-embed-text-v2-moe:latest") ) # ChromaDB is still used by SemanticRouter for route embeddings. chroma_client = ChromaClient(db=ChromaDB(), port=8080) ### Speech I/O ### audio_in = Topic(name="audio0", msg_type="Audio") query_topic = Topic(name="question", msg_type="String") query_answer = Topic(name="answer", msg_type="String") speech_to_text = SpeechToText( inputs=[audio_in], outputs=[query_topic], model_client=whisper_client, trigger=audio_in, component_name="speech_to_text", ) text_to_speech = TextToSpeech( inputs=[query_answer], trigger=query_answer, model_client=tts_client, config=TextToSpeechConfig(play_on_device=True), component_name="text_to_speech", ) ### Vision (object detection) ### image0 = Topic(name="image_raw", msg_type="Image") detections_topic = Topic(name="detections", msg_type="Detections") vision = Vision( inputs=[image0], outputs=[detections_topic], trigger=image0, config=VisionConfig(threshold=0.5), model_client=detection_client, component_name="object_detection", ) ### VQA VLM ### mllm_query = Topic(name="mllm_query", msg_type="String") mllm = VLM( inputs=[mllm_query, image0, detections_topic], outputs=[query_answer], model_client=qwen_vl_client, trigger=mllm_query, component_name="visual_q_and_a", ) mllm.set_component_prompt( template=( "Imagine you are a robot. This image has the following items: " "{{ detections }}. Answer the following about this image: " "{{ mllm_query }}" ) ) ### Introspection VLM (room classification feeding the memory) ### introspection_query = FixedInput( name="introspection_query", msg_type="String", fixed=( "What kind of a room is this? Is it an office, a bedroom or a " "kitchen? Give a one word answer, out of the given choices" ), ) introspection_answer = Topic(name="introspection_answer", msg_type="String") introspector = VLM( inputs=[introspection_query, image0], outputs=[introspection_answer], model_client=qwen_vl_client, trigger=15.0, component_name="introspector", ) def introspection_validation(output: str) -> Optional[str]: for option in ["office", "bedroom", "kitchen"]: if option in output.lower(): return option introspector.add_publisher_preprocessor(introspection_answer, introspection_validation) ### Memory (graph-backed spatio-temporal memory) ### position = Topic(name="odom", msg_type="Odometry") memory = Memory( layers=[ MemLayer(subscribes_to=detections_topic), MemLayer(subscribes_to=introspection_answer), ], position=position, model_client=qwen_client, embedding_client=embedding_client, config=MemoryConfig(db_path="/tmp/complete_agent.db"), trigger=15.0, component_name="memory", ) ### Generic LLM (general Q&A) ### llm_query = Topic(name="llm_query", msg_type="String") llm = LLM( inputs=[llm_query], outputs=[query_answer], model_client=qwen_client, trigger=[llm_query], component_name="general_q_and_a", ) ### Go-to-X using LLM tool calling on Memory.locate ### goto_query = Topic(name="goto_query", msg_type="String") goal_point = Topic(name="goal_point", msg_type="PoseStamped") goto = LLM( inputs=[goto_query], outputs=[goal_point], model_client=qwen_client, trigger=goto_query, config=LLMConfig(), component_name="go_to_x", ) goto.set_component_prompt( template=( "The user asks you to go to a place. Use the available tools to " "look up the place's location in memory. Pass the place name to " "the locate tool as the ``concept`` argument. " "The user said: {{goto_query}}" ) ) memory.register_tools_on(goto, tools=["locate"], send_tool_response_to_model=False) _LOCATION_RE = re.compile(r"Location:\s*\(([^)]+)\)") def locate_text_to_goal_point(output: str) -> Optional[np.ndarray]: """Pull the centroid coordinates out of Memory.locate's text output.""" match = _LOCATION_RE.search(output) if not match: return try: coords = np.fromstring(match.group(1), sep=",", dtype=np.float64) except ValueError: return if coords.shape[0] == 2: coords = np.append(coords, 0.0) if coords.shape[0] != 3: return return coords goto.add_publisher_preprocessor(goal_point, locate_text_to_goal_point) ### Semantic router (uses ChromaDB for the route embeddings) ### goto_route = Route( routes_to=goto_query, samples=[ "Go to the door", "Go to the kitchen", "Get me a glass", "Fetch a ball", "Go to hallway", ], ) llm_route = Route( routes_to=llm_query, samples=[ "What is the capital of France?", "Is there life on Mars?", "How many tablespoons in a cup?", "How are you today?", "Whats up?", ], ) mllm_route = Route( routes_to=mllm_query, samples=[ "Are we indoors or outdoors", "What do you see?", "Whats in front of you?", "Where are we", "Do you see any people?", "How many things are in front of you?", "Is this room occupied?", ], ) router = SemanticRouter( inputs=[query_topic], routes=[llm_route, goto_route, mllm_route], default_route=llm_route, config=SemanticRouterConfig(router_name="go-to-router", distance_func="l2"), db_client=chroma_client, component_name="router", ) ### Launch (single process so goto can call memory in-process) ### launcher = Launcher() launcher.add_pkg( components=[ mllm, llm, goto, introspector, memory, router, speech_to_text, text_to_speech, vision, ] ) launcher.bringup() ``` ```{note} The same `qwen_client` (Ollama, `qwen3:0.6b`) drives general Q&A, the goto tool-caller, and Memory's episodic-consolidation summaries. The VLM (`qwen_vl_client`) is shared between the VQA path and the introspector. ``` ## What We Have Built In this single recipe, we have assembled a fully capable embodied agent with the following capabilities: - {material-regular}`record_voice_over;1.2em;sd-text-primary` **A conversational interface** using speech-to-text and text-to-speech models that uses the robot's microphone and playback speaker. (See: [Conversational Agent](conversational-agent.md)) - {material-regular}`visibility;1.2em;sd-text-primary` **Contextual visual question answering** based on the robot's camera, using a multimodal LLM enriched with object detection output. (See: [Prompt Engineering](prompt-engineering.md)) - {material-regular}`chat;1.2em;sd-text-primary` **General knowledge Q&A** using a text-only LLM for non-visual queries. - {material-regular}`memory;1.2em;sd-text-primary` **A graph-backed spatio-temporal memory** that acts as the robot's long-term memory, continuously updated with object detections and room-type introspection, indexed simultaneously by meaning, location, and time. Built on [eMEM](https://github.com/automatika-robotics/emem). (See: [Spatio-Temporal Memory](semantic-map.md)) - {material-regular}`route;1.2em;sd-text-primary` **Memory-aware Go-to-X navigation** -- a tool-calling LLM that asks Memory to `locate` a place and publishes the result as a goal point. (See: [GoTo Navigation](goto-navigation.md), [Tool Calling](tool-calling.md)) - {material-regular}`alt_route;1.2em;sd-text-primary` **Intent-based semantic routing** through a single input interface that directs queries to the correct component based on content. (See: [Semantic Routing](semantic-routing.md)) This is the EMOS developer experience: a sophisticated, multi-capability embodied agent defined entirely in a single Python script. Every component -- perception, reasoning, memory, navigation, and speech -- is wired together through Topics and launched with one call to `bringup()`. The same recipe runs on any robot that EMOS supports, from wheeled AMRs to quadrupeds, without modification. To run this same graph in **multi-process mode with fault tolerance**, see [Multiprocessing & Fault Tolerance](../events-and-resilience/multiprocessing.md). For runtime resilience -- fallback logic, recovery maneuvers, algorithm switching -- see the [Events & Actions](../../concepts/events-and-actions.md) documentation. --- ```{tip} **Promote this recipe to production.** While you're shaping it, the script runs straight with `python recipe.py`. Once it's solid, drop it at `~/emos/recipes//recipe.py` and run `emos run ` -- you'll get sensor pre-flight checks, persistent logs, and a card on the dashboard so an operator can launch it from a browser. See [Running Recipes](../../getting-started/running-recipes.md) for the full development-vs-production comparison and install-mode pitfalls (especially in Container mode). ``` ``` ## File: recipes/planning-and-manipulation/cortex-agent.md ```markdown # Cortex: The Agentic Harness **A single component, dropped on top of the rest of your recipe, that turns it into a self-directed agent.** Most EMOS recipes you've seen so far are *programmed*: events trigger components, components publish to topics, fallbacks recover from failure -- and you, the recipe author, hand-wired every link. The Cortex component is a different shape. Drop a Cortex into your recipe and it discovers every other component you added, registers every method they expose as a callable tool, and -- given a high-level goal like *"track the person on the left and tell me what they're holding"* -- decomposes it into an ordered plan, dispatches each step, watches the feedback, and replans on failure. No orchestration glue from you. If [Claude Code](https://claude.com/claude-code) is an agentic harness for software engineering, **Cortex is an agentic harness for embodied intelligence**. The capability components -- Vision, VLM, TTS, navigation, memory -- are the robot's limbs and senses. Cortex is the part that decides *what to do next* using these capabilities. ```{seealso} For the conceptual model and the full list of capabilities Cortex auto-discovers, see [Cortex](../../intelligence/cortex.md). For Cortex paired with a graph-backed spatio-temporal memory, see [Memory and Cortex](cortex-memory.md). For Cortex orchestrating the navigation stack on top of all of that, see [Cortex Driving the Full Stack](cortex-navigation.md). ``` --- ## The shape of the abstraction A capability component such as `Vision` exposes its primary work as topics (`/detections`, `/trackings`) but it *also* exposes private methods decorated with `@component_action`: ```python class Vision(Component): @component_action(description={...}) def track(self, label: str): ... @component_action(description={...}) def take_picture(self, save_path: str = "..."): ... ``` These actions are normally invisible -- they require explicit wiring through the events/actions system to be useful. **Cortex changes that.** When you drop a Cortex component into the launcher, on activation it walks every managed component and discovers: | What gets discovered | What Cortex does with it | |---|---| | `@component_action` methods | Auto-registers as an LLM tool, namespaced as `{component}.{method}`. Each carries its OpenAI-format description so the planner knows what it does. | | `@component_fallback` methods | Same as above -- exposed as callable recovery tools the planner can fall back to. | | Additional ROS services (`get_ros_entrypoints()`) | Registered as `send_request_to_{name}`, with the request schema auto-translated to JSON properties so the LLM can fill the fields. | | Additional ROS action servers | Registered as `send_goal_to_{name}` with the same schema translation. | | The component's main action server | Registered the same way -- so a Planner running as `ActionServer` becomes a callable navigation tool. | | Component config parameters | Reachable via the built-in `update_parameter(component, param_name, new_value)` execution tool -- the LLM can re-tune any parameter at runtime. | | Component structure | Reachable via the built-in `inspect_component(name)` planning tool -- the LLM reads the recipe live before committing a plan. | Every one of those tools is automatic. You write the components; Cortex makes them addressable. --- ## What we're building A robot that, when you tell it *"describe what you see and then start tracking the person"*, will: 1. Plan three steps -- call `vlm.describe`, feed its answer to `tts.say`, then call `vision.track`. 2. Execute them in order -- describing the scene, speaking the description through `tts.say`, then asking `Vision` to start tracking the requested label. 3. Report each step's result back into the planning loop and close out the episode. The recipe is short. There is no event wiring. There are no fallback policies. There are no topic-routed connections between the VLM, the TTS, and Cortex -- speech happens because Cortex calls `tts.say()` as a tool, not because some output topic is silently subscribed by TTS. We don't write a single prompt either -- Cortex's built-in prompts plus the auto-discovered tool descriptions are the prompt. --- ## Step 1: Bring up your capability components We need eyes, a voice, and visual reasoning. Standard EMOS capability components: ```python from agents.clients import OllamaClient, RoboMLRESPClient from agents.components import TextToSpeech, VLM, Vision from agents.config import TextToSpeechConfig, VisionConfig from agents.models import OllamaModel, VisionModel from agents.ros import Topic # Vision — detection + tracking detection_model = VisionModel(name="rtdetr", checkpoint="PekingU/rtdetr_r50vd_coco_o365") detection_client = RoboMLRESPClient(detection_model) image_in = Topic(name="/image_raw", msg_type="Image") detections = Topic(name="detections", msg_type="Detections") trackings = Topic(name="trackings", msg_type="Trackings") vision = Vision( inputs=[image_in], outputs=[detections, trackings], model_client=detection_client, config=VisionConfig(threshold=0.5), trigger=0.5, component_name="vision", ) # VLM — visual question answering. Cortex invokes it via ``vlm.describe``, # and the action's return value comes back as the tool result. vlm_model = OllamaModel(name="qwen_vl", checkpoint="qwen2.5vl:latest") vlm_client = OllamaClient(vlm_model) vlm_query = Topic(name="vlm_query", msg_type="String") vlm_response = Topic(name="vlm_response", msg_type="String") vlm = VLM( inputs=[vlm_query, image_in], outputs=[vlm_response], model_client=vlm_client, trigger=vlm_query, component_name="vlm", ) # TTS — speech happens via Cortex calling ``tts.say(text=...)``. tts_input = Topic(name="tts_input", msg_type="String") tts = TextToSpeech( inputs=[tts_input], config=TextToSpeechConfig(enable_local_model=True, play_on_device=True), trigger=tts_input, component_name="tts", ) ``` Nothing here is Cortex-specific. Each component has its own `@component_action` methods declared upstream -- `Vision.track`, `Vision.take_picture`, `VLM.describe`, `TTS.say` -- and Cortex will discover all of them on activation. Cortex sequences the components by calling their actions in turn. --- ## Step 2: Drop in Cortex ```python from agents.components import Cortex from agents.config import CortexConfig from agents.ros import Action # A planner LLM. Choose a chat-grade model — the smaller, the faster the loop. planner_model = OllamaModel(name="qwen", checkpoint="qwen3.5:latest") planner_client = OllamaClient(planner_model) # Cortex publishes its text-only replies (cases where the planner decides no # tool calls are needed) to this topic for downstream consumers (e.g. the Web # UI). When the planner *does* want the robot to speak, it calls ``tts.say`` # as a tool -- it does not rely on this topic being subscribed by TTS. cortex_output = Topic(name="cortex_output", msg_type="String") cortex = Cortex( output=cortex_output, model_client=planner_client, config=CortexConfig(max_planning_steps=5, max_execution_steps=10), component_name="cortex", ) ``` **That's all.** No `actions=[…]` list -- the capability components contribute their own actions. No prompt -- the built-in prompt plus the discovered tool descriptions are the prompt. No fallback wiring -- Cortex confirms each step before executing it and replans on failure. --- ## Step 3: Adding your own custom action A capability you want exposed to the planner that doesn't naturally live on a managed component? Pass it as a custom `Action`. Cortex registers it alongside everything else. ```python led_on = False def toggle_led(): """Toggle an LED on the robot.""" global led_on led_on = not led_on print(f"LED toggled {'ON' if led_on else 'OFF'}") cortex = Cortex( actions=[ Action(method=toggle_led, description="Toggle the robot's LED on or off."), ], output=cortex_output, model_client=planner_client, config=CortexConfig(max_planning_steps=5, max_execution_steps=10), component_name="cortex", ) ``` The description is mandatory -- it's what the planner sees when deciding whether to call this tool. --- ## Step 4: Launch ```python from agents.ros import Launcher launcher = Launcher() launcher.enable_ui( inputs=[cortex.ui_main_action_input], outputs=[cortex_output], ) launcher.add_pkg( components=[vision, vlm, tts, cortex], multiprocessing=True, package_name="automatika_embodied_agents", ) launcher.on_process_fail() # process-level safety net launcher.bringup() ``` `launcher.enable_ui` registers a goal-input field for Cortex's main action and a streaming output panel for `cortex_output`. The whole agent runs in a single launcher process tree. --- ## Talking to the agent Open the Web UI at `http://localhost:5001` and send tasks in plain English: | Goal | What Cortex plans | |---|---| | *"describe what you see"* | Two steps: `vlm.describe` produces a sentence; `tts.say` is called with that sentence as its `text` argument. | | *"start tracking the person"* | One step: `vision.track(label="person")`. The Vision component's `@component_action` starts continuous tracking on the named label (results stream on the `trackings` topic) and returns a confirmation string immediately. | | *"take a picture, describe it, then track whatever's in front of you"* | Three steps, sequenced. The third step's argument is bound from the second step's output -- Cortex resolves `` placeholders at runtime. | | *"toggle the LED"* | One step: the custom `toggle_led` action you registered. | | *"are you ok?"* | No actions needed. The planner returns text only; the reply lands on the `cortex_output` topic. (If you want it spoken, your prompt can nudge the planner to always end with a `tts.say` call.) | Or send a goal from another terminal directly to Cortex's action server: ```shell ros2 action send_goal /cortex_/cortex_input_command \ automatika_embodied_agents/action/VisionLanguageAction \ "{task: 'describe what you see and track the person'}" ``` Watch the launcher's main logging card to see the planning trace, the goals Cortex dispatches, and feedback streamed back from each component. --- ## What just happened When you sent the goal *"describe what you see and then start tracking the person"*, Cortex: 1. Built a plan via the planning loop. The first iteration optionally called `inspect_component("vision")` to confirm the tool surface, then committed three execution tool calls. 2. Confirmed and called each step in turn. The first (`vlm.describe`) returned a text description; the second (`tts.say`) was called with that description bound as its `text` argument and the speaker spoke it; the third (`vision.track`) asked the Vision component to start continuous tracking on the named label and returned a confirmation string. Tracking results then streamed on the component's `trackings` topic for any downstream consumer to use. 3. With every step's tool result folded back into the trace, the episode closed and the plan returned `SUCCEEDED`. Compare that to the equivalent recipe written without Cortex: bespoke event wiring for the trigger, hand-tuned prompts on each component, manual sequencing of the speech and tracking calls. **Cortex collapses all of that into the one component you just dropped in.** ```{tip} For the long-running case -- where Cortex *should* dispatch a Kompass action server like the Controller's `track_vision_target` (or the Planner's `navigate_to_goal`) and watch its feedback stream until the goal completes -- add the `Controller` (or `Planner`) component to the launcher. Cortex auto-registers each one's main action server as `send_goal_to_` and switches into asynchronous monitoring mode. See [Cortex Driving the Full Stack](cortex-navigation.md). ``` --- ## Where next - {doc}`Cortex Driving the Full Stack ` -- the showcase tutorial. Cortex orchestrates a navigation stack, vision, memory, and speech to handle compound natural-language goals like *"go to the kitchen and tell me what's on the counter"*. - {doc}`Memory and Cortex ` -- add a graph-backed spatio-temporal memory so Cortex can reason over past observations and the robot's own internal state. - {doc}`Cortex concept page <../../intelligence/cortex>` -- the full reference for the planning loop, the confirmation step, RAG, async goal monitoring, and the Cortex-as-Monitor architecture. --- ```{tip} **Promote this recipe to production.** While you're shaping it, the script runs straight with `python recipe.py`. Once it's solid, drop it at `~/emos/recipes//recipe.py` and run `emos run ` -- you'll get sensor pre-flight checks, persistent logs, and a card on the dashboard so an operator can launch it from a browser. See [Running Recipes](../../getting-started/running-recipes.md) for the full development-vs-production comparison and install-mode pitfalls (especially in Container mode). ``` ``` ## File: recipes/planning-and-manipulation/cortex-memory.md ```markdown # Memory and Cortex > _"What kind of place have I been in today?"_ > > _"Where did I last see a person?"_ > > _"Are you doing OK? Anything wrong?"_ > > _"Tell me everything you know about the fridge."_ When [Cortex](../../intelligence/cortex.md) and [Memory](../../intelligence/memory.md) share a recipe, the robot becomes addressable in the same way you'd talk to a person who has _been there all day_. Memory accumulates a structured graph of everything the robot has seen, where it saw it, when, and how it was feeling at the time. Cortex auto-discovers Memory's retrieval surface, augments its planning prompt to handle distinct kinds of question, and orchestrates the right tools for each. None of that wiring is yours to write. This is the recipe that shows what that combination is capable of -- the deep pairing of **the agentic harness** and **the first neuroscience-grounded memory system for embodied agents**. ```{seealso} For Memory by itself with no agent on top, start with [Spatio-Temporal Memory](../foundation/semantic-map.md). For Cortex by itself with no memory, start with [Cortex: The Agentic Harness](cortex-agent.md). For the full multi-system showcase that adds navigation on top of this pair, see [Cortex Driving the Full Stack](cortex-navigation.md). ``` ```{admonition} Prerequisites :class: important The `Memory` component requires the [eMEM](https://github.com/automatika-robotics/emem) Python package, which `emos install` does not add. The install command depends on your mode (Pixi: `pixi add --pypi emem`; Native: `pip install emem`; Container: install inside the container) — see [Memory installation](../../intelligence/memory.md). ``` --- ## What we're building A robot that, over the course of a session, accumulates a hierarchical record of what it has been doing — and that you can query in three distinct registers: | Register | Examples | What Cortex does | | -------------------- | ------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Perception query** | _"Where did you last see the chair?"_, _"What rooms have you been in today?"_, _"Summarise the last 10 minutes"_ | Plans a chain of perception-retrieval tools (`semantic_search`, `temporal_query`, `episode_summary`, `locate`, ...), reads memory's graph, replies in text. **No new observations, no episode wrapping.** | | **Body query** | _"How are you feeling?"_, _"What's your battery level?"_, _"Anything overheating?"_ | Plans a single `body_status` call. Routes through the dedicated interoception surface (Memory's `is_internal_state=True` layers) — perception tools never see body state and vice versa. | | **Action task** | _"Take a picture of the fridge"_, _"Take a picture of the table and remember what you saw on it"_, _"Patrol the kitchen"_ | **Begins with `body_status`** (refuse the task if internal state says no). Wraps execution in `start_episode` / `end_episode`. Optionally consults perception memory while planning. Stores derived facts via `store_specific_memory`. | That whole protocol — three registers, mandatory body checks for action tasks, episodic wrapping — is the **Memory-aware planning** Cortex installs automatically when it detects a Memory component in the recipe. You write zero lines of orchestration to get it. --- ## Step 1: Perception Same Vision + VLM pair as the [foundation memory recipe](../foundation/semantic-map.md) — detections every second, scene captions every ten: ```python from agents.clients import OllamaClient from agents.components import VLM, Vision from agents.config import VisionConfig from agents.models import OllamaModel from agents.ros import FixedInput, Topic # Vision component (on-device classifier for low-latency detections) image_in = Topic(name="/image_raw", msg_type="Image") detections_out = Topic(name="detections", msg_type="Detections") vision = Vision( inputs=[image_in], outputs=[detections_out], config=VisionConfig(threshold=0.5, enable_local_classifier=True), trigger=1.0, component_name="vision", ) # VLM scene captioner — periodic introspective description scene_query = FixedInput( name="scene_query", msg_type="String", fixed=( "Describe what you see in one concise sentence: room type, notable " "objects, and any people present." ), ) scene_description = Topic(name="scene_description", msg_type="String") vlm_model = OllamaModel(name="gemma4", checkpoint="gemma4:latest") vlm_client = OllamaClient(vlm_model) captioner = VLM( inputs=[scene_query, image_in], outputs=[scene_description], model_client=vlm_client, trigger=10.0, component_name="captioner", ) ``` These are the **perception layers** Memory will track. --- ## Step 2: Interoception This is where the Cortex + Memory pairing pulls something the original semantic-map recipe couldn't. We give the robot the ability to feel itself: a battery sensor, a CPU temperature, a joint-health flag — each routed through Memory as an **interoception layer** (`is_internal_state=True`): ```python battery_topic = Topic(name="/battery_level", msg_type="Float32") cpu_temp_topic = Topic(name="/cpu_temp", msg_type="Float32") joint_health_topic = Topic(name="/joint_health", msg_type="String") ``` ```{tip} If you don't have a real battery sensor, fake one from another terminal so you can try the body queries below: \ `ros2 topic pub /battery_level std_msgs/msg/Float32 "{data: 42.0}" -r 1` ``` In Memory, an interoception layer inherits the robot's current pose at write time -- so a low-battery reading taken on a steep ramp is later spatially associated with that ramp. --- ## Step 3: Memory ```python from agents.components import Memory from agents.config import MemoryConfig from agents.ros import MemLayer position = Topic(name="/odometry/filtered", msg_type="Odometry") embedding_model = OllamaModel( name="embeddings", checkpoint="nomic-embed-text-v2-moe:latest" ) embedding_client = OllamaClient(embedding_model) memory = Memory( layers=[ # --- Perception layers --- MemLayer(subscribes_to=detections_out), MemLayer(subscribes_to=scene_description), # --- Interoception layers --- MemLayer(subscribes_to=battery_topic, is_internal_state=True), MemLayer(subscribes_to=cpu_temp_topic, is_internal_state=True), MemLayer(subscribes_to=joint_health_topic, is_internal_state=True), ], position=position, model_client=vlm_client, # used for episode-consolidation gist generation + entity extraction embedding_client=embedding_client, # used for semantic search vector indexing config=MemoryConfig( db_path="/tmp/cortex_memory.db", consolidation_window=300.0, # short window for the demo so you see consolidation kick in archive_after_seconds=1800.0, ), trigger=10.0, component_name="memory", ) ``` A few things worth seeing here: - **Perception and interoception are peers** — they share the same node/edge graph but get tagged differently so retrieval surfaces stay clean. - **`model_client` and `embedding_client` are different clients.** The first writes prose (gists), the second writes vectors. eMEM uses both during consolidation. - **`db_path` is the entire memory state.** SQLite-backed eMEM means everything the robot has seen, all entities, all gists, all interoception readings, persist in a single file. **Reboot the robot, point the next session at the same `db_path`, and the agent picks up where it left off.** --- ## Step 4: Voice Cortex will route its replies straight through TTS so the robot speaks its answers: ```python from agents.components import TextToSpeech from agents.config import TextToSpeechConfig tts_in = Topic(name="cortex_output", msg_type="StreamingString") tts = TextToSpeech( inputs=[tts_in], config=TextToSpeechConfig(enable_local_model=True, play_on_device=True), trigger=tts_in, component_name="tts", ) ``` --- ## Step 5: Cortex ```python from agents.components import Cortex from agents.config import CortexConfig planner_model = OllamaModel(name="qwen", checkpoint="qwen3.5:latest") planner_client = OllamaClient(planner_model) cortex = Cortex( output=tts_in, model_client=planner_client, config=CortexConfig(max_planning_steps=5, max_execution_steps=15), component_name="cortex", ) ``` That is the entire agent. When the launcher activates Cortex: 1. It walks every managed component and registers their capabilities. 2. It detects Memory in the recipe and augments itself to utilize it. --- ## Step 6: Launch ```python from agents.ros import Launcher launcher = Launcher() launcher.enable_ui( inputs=[cortex.ui_main_action_input], outputs=[tts_in], ) launcher.add_pkg( components=[vision, captioner, memory, tts, cortex], package_name="automatika_embodied_agents", multiprocessing=True, ) launcher.on_process_fail() launcher.bringup() ``` --- ## Talking to the robot Run the recipe and let it sit for a few minutes -- detections accumulate, scene captions roll in every ten seconds, body-state readings are recorded continuously. Open the Web UI at `http://localhost:5001` and start asking questions. ### Perception queries > _"What is currently around you?"_ The robot replies with what's nearby, drawing on its accumulated memory of the space rather than just the current camera frame. > _"Where did you last see the chair?"_ You get the location (and roughly when) the chair was last observed. > _"Summarise everything you've done in the last episode."_ A short summary of the most recent activity span. > _"Tell me everything you know about the fridge."_ A consolidated answer that fuses every observation, summary, and recognised entity record about the fridge -- across all sessions, including the ones from earlier days. ### Body queries > _"How are you?"_ The robot reports current battery, CPU temperature, joint health, and any other interoception layer you've wired in. > _"Is your battery low?"_ Same, filtered to just the battery. ### Action tasks > _"Walk over and describe the fridge."_ (Assuming you've added a navigation stack -- see [Cortex Driving the Full Stack](cortex-navigation.md) for that pairing.) The robot first checks its own body state; if the battery is too low or a fault flag is set, it refuses the task and tells you why. Otherwise it recalls where the fridge was last seen, navigates there, takes a fresh look, narrates what it sees, and stores the new description in memory for the next session. --- ## Persistence across sessions Stop the recipe. Restart it pointing at the same `db_path`. Memory loads the prior session's graph; Cortex re-augments its prompt with the existing layer set; the agent **remembers**. This is the part of the design that makes eMEM-on-Cortex an actual cognitive system rather than a session-scoped vector DB. _Yesterday's robot is today's robot._ The episodes from yesterday are still in the graph, the entity for the fridge is still there, the gists are still searchable. New observations slot into the same structure. --- ## Where next - {doc}`Cortex Driving the Full Stack ` — pairs this Cortex + Memory recipe with a Kompass navigation stack so the robot can act on memory queries instead of just answering them. Compound goals like _"go to the kitchen and tell me what's on the counter"_ fall out naturally. - {doc}`Cortex: The Agentic Harness ` — the introductory tutorial focused on Cortex's auto-discovery and tool surface, without Memory. - {doc}`Memory concept page <../../intelligence/memory>` — the full architectural reference for eMEM. - [eMEM on GitHub](https://github.com/automatika-robotics/emem) — the underlying memory library, including a standalone testing harness with a MiniGrid environment and a ReAct agent that exercises memory's tools end-to-end. --- ```{tip} **Promote this recipe to production.** While you're shaping it, the script runs straight with `python recipe.py`. Once it's solid, drop it at `~/emos/recipes//recipe.py` and run `emos run ` -- you'll get sensor pre-flight checks, persistent logs, and a card on the dashboard so an operator can launch it from a browser. See [Running Recipes](../../getting-started/running-recipes.md) for the full development-vs-production comparison and install-mode pitfalls (especially in Container mode). ``` ``` ## File: recipes/planning-and-manipulation/cortex-navigation.md ```markdown # Cortex Driving the Full Stack This is the tutorial that shows what [Cortex](../../intelligence/cortex.md) is *for*. We give a robot eyes, a voice, a body that can move, and a memory that learns. Then we drop a Cortex on top of all of that and start typing compound, free-form goals into the Web UI: > *"Go to the kitchen and tell me what's on the counter."* > > *"Find the closest chair, take a picture there, and come back here."* > > *"Patrol the room until you see a person, then announce their location."* There is no parser, no state machine, no sequence diagram in this recipe. Cortex inspects the running graph, plans the steps, dispatches navigation goals, watches their feedback, calls a VLM to look at the world when it arrives, queries Memory for what it has seen before, and routes its replies through TTS so the robot speaks. **The recipe is the graph; the agent is one master component.** ```{seealso} For the introduction to the Cortex abstraction itself, start with [Cortex: The Agentic Harness](cortex-agent.md). For the conceptual reference covering the full feature set Cortex auto-discovers, see [Cortex](../../intelligence/cortex.md). For Cortex without a navigation stack, focused on the deep pairing with spatio-temporal memory, see [Memory and Cortex](cortex-memory.md). ``` --- ## What we're orchestrating Five subsystems, all running in one launcher: | Subsystem | Components | What Cortex does with it | |---|---|---| | **Perception** | `Vision`, `VLM` | Detection feeds Memory; VLM answers visual questions on demand. | | **Spatial memory** | `Memory` | Stores detections + scene captions tagged with position. Exposes `locate`, `recall`, `semantic_search`, `body_status`, ten retrieval tools in total. | | **Voice** | `TextToSpeech` | The robot's mouthpiece. Cortex routes its replies straight through. | | **Navigation** | `Planner`, `Controller`, `LocalMapper`, `DriveManager` | The Kompass quartet. Cortex sends action goals to `Planner`'s main action server. | | **The agent** | `Cortex` | Discovers all of the above, exposes them as LLM tools, plans and executes against natural-language goals. | The wiring is conventional EMOS. The Cortex section at the end is what turns the whole stack into a single self-directing agent. --- ## Step 1: Robot configuration Standard differential-drive setup; adjust to your platform. ```python import numpy as np from kompass.robot import ( AngularCtrlLimits, LinearCtrlLimits, RobotConfig, RobotFrames, RobotGeometry, RobotType, ) my_robot = RobotConfig( model_type=RobotType.DIFFERENTIAL_DRIVE, geometry_type=RobotGeometry.Type.CYLINDER, geometry_params=np.array([0.1, 0.3]), ctrl_vx_limits=LinearCtrlLimits(max_vel=0.4, max_acc=1.5, max_decel=2.5), ctrl_omega_limits=AngularCtrlLimits( max_vel=0.4, max_acc=2.0, max_decel=2.0, max_steer=np.pi / 3 ), ) ``` --- ## Step 2: The navigation stack ```python from kompass.components import ( Controller, DriveManager, LocalMapper, LocalMapperConfig, Planner, PlannerConfig, ) from kompass.control import ControllersID, MapConfig from kompass.ros import Topic # Planner — runs as an Action Server so Cortex can dispatch goals into it. planner = Planner(component_name="planner", config=PlannerConfig(loop_rate=1.0)) planner.run_type = "ActionServer" # Controller — DWA, eats from the local map. controller = Controller(component_name="controller") controller.algorithm = ControllersID.DWA controller.direct_sensor = False # DriveManager driver = DriveManager(component_name="drive_manager") driver.outputs(robot_command=Topic(name="/cmd_vel", msg_type="Twist")) # Local Mapper — fed by a 2D laser local_mapper = LocalMapper( component_name="mapper", config=LocalMapperConfig( map_params=MapConfig(width=4.0, height=4.0, resolution=0.1), ), ) local_mapper.inputs(sensor_data=Topic(name="/scan", msg_type="LaserScan")) ``` --- ## Step 3: Perception ```python from agents.clients import OllamaClient from agents.components import VLM, Vision from agents.config import VisionConfig from agents.models import OllamaModel from agents.ros import FixedInput image_in = Topic(name="/image_raw", msg_type="Image") detections_out = Topic(name="detections", msg_type="Detections") vision = Vision( inputs=[image_in], outputs=[detections_out], config=VisionConfig(threshold=0.5, enable_local_classifier=True), trigger=1.0, component_name="vision", ) # A VLM that captions every 10 seconds — the captions feed Memory and TTS. scene_query = FixedInput( name="scene_query", msg_type="String", fixed="Describe the scene in one concise sentence: room type and notable objects.", ) scene_description = Topic(name="scene_description", msg_type="String") vlm_model = OllamaModel(name="gemma4", checkpoint="gemma4:latest") vlm_client = OllamaClient(vlm_model) captioner = VLM( inputs=[scene_query, image_in], outputs=[scene_description], model_client=vlm_client, trigger=10.0, component_name="captioner", ) ``` The VLM's `describe` and Vision's `track` / `take_picture` are `@component_action`s already declared upstream. **Cortex will discover them all.** We don't wire any of them by hand. --- ## Step 4: Memory ```python from agents.components import Memory from agents.config import MemoryConfig from agents.ros import MemLayer position = Topic(name="/odometry/filtered", msg_type="Odometry") embedding_model = OllamaModel( name="embeddings", checkpoint="nomic-embed-text-v2-moe:latest" ) embedding_client = OllamaClient(embedding_model) memory = Memory( layers=[ MemLayer(subscribes_to=detections_out), MemLayer(subscribes_to=scene_description), ], position=position, model_client=vlm_client, embedding_client=embedding_client, config=MemoryConfig(db_path="/tmp/embodied_memory.db"), trigger=10.0, component_name="memory", ) ``` When Memory is in the recipe, Cortex **automatically augments itself** with task-classification guidance. Action tasks get wrapped in episodes -- Cortex begins them with `start_episode` and ends with `end_episode` so the observations made during the task get consolidated into the long-term graph. ```{tip} Add an interoception layer (`MemLayer(subscribes_to=battery_topic, is_internal_state=True)`) and Cortex starts every action plan with a `body_status` check -- it might refuse to navigate when the battery is below a threshold, with a clear text explanation. See [Memory and Cortex](cortex-memory.md) for the full pattern. ``` --- ## Step 5: Voice ```python from agents.components import TextToSpeech from agents.config import TextToSpeechConfig tts_in = Topic(name="cortex_output", msg_type="StreamingString") tts = TextToSpeech( inputs=[tts_in], config=TextToSpeechConfig(enable_local_model=True, play_on_device=True), trigger=tts_in, component_name="tts", ) ``` --- ## Step 6: Cortex ```python from agents.components import Cortex from agents.config import CortexConfig planner_model = OllamaModel(name="qwen", checkpoint="qwen3.5:latest") planner_client = OllamaClient(planner_model) cortex = Cortex( output=tts_in, # Cortex streams replies into TTS model_client=planner_client, config=CortexConfig( max_planning_steps=5, max_execution_steps=15, ), component_name="cortex", ) ``` That's the whole agent. **No `actions=[...]` is needed** -- the agent's tool palette is everything the other components contribute: If you want to add a private capability that isn't on a managed component -- a database call, a custom servo on a peripheral, an external API -- pass it as a custom `Action(method=..., description=...)` in `actions=[...]` and Cortex registers it alongside the rest. --- ## Step 7: Launch ```python from kompass.ros import Launcher launcher = Launcher() # Navigation stack launcher.add_pkg( components=[planner, controller, driver, local_mapper], multiprocessing=True, package_name="kompass", ) # Intelligence stack launcher.add_pkg( components=[vision, captioner, memory, tts, cortex], multiprocessing=True, package_name="automatika_embodied_agents", ) # Shared inputs launcher.inputs(location=position) # Robot config + frames launcher.robot = my_robot launcher.frames = RobotFrames(world="map", odom="odom", scan="base_scan") launcher.enable_ui( inputs=[cortex.ui_main_action_input], outputs=[tts_in], ) launcher.on_process_fail() launcher.bringup() ``` --- ## Driving the agent Run the recipe and let the robot wander around for a few minutes. Memory accumulates detections and scene captions tagged with positions. Then open `http://localhost:5001` and start typing. ### Single-step goals > *"What is currently around you?"* — the robot describes its surroundings. > > *"Where did you last see the chair?"* — gives the location and roughly when. > > *"How are you?"* — reads its body state and replies. ### Compound goals -- the interesting case > *"Go to the chair."* The robot recalls where the chair is from memory, dispatches a navigation goal, and reports when it has arrived. > *"Go to the kitchen and tell me what's on the counter."* The robot navigates to the kitchen using its memory of where that is, looks at the counter once it arrives, narrates what it sees, and stores the new observation in memory for the next session. > *"Patrol the room. If you see a person, stop and tell me where you found them."* A long-horizon mission. The robot dispatches successive navigation goals around the room and watches its detections. The moment a person appears, it cancels the current goal and reports their location. None of these required you to write orchestration code, prompts, or a state machine. The behaviour emerges from Cortex's read of the auto-discovered tool surface and the memory-aware planning it installs when it sees a Memory component in the recipe. --- ## What this looks like in the Web UI Send a goal, and the action goal Cortex dispatches into the Planner appears live as feedback in the **main logging card** -- both the Planner's path-tracking feedback and Cortex's own confirmation decisions stream side by side. --- ## What you didn't have to write - An event triggering "go to X" when the user types it. - An LLM prompt parsing "go to X" into a destination. - A goal-builder for the Planner action. - An action client construction with feedback callbacks and cancellation logic. - A retry policy for the navigation goal. - A wait-loop that blocks until SUCCEEDED before invoking the VLM. - A handoff from the VLM output to the TTS input. - A memory write that records "I went to the kitchen and saw X". That entire stack of orchestration is replaced by the auto-discovery, the two-phase loop, and the Memory-aware prompt augmentation. **You wrote the components. Cortex wrote the recipe.** --- ## Where next - {doc}`Cortex concept page <../../intelligence/cortex>` -- a deeper look at the auto-discovery, the planning/execution loop, RAG, and the Cortex-as-Monitor model. - {doc}`Memory and Cortex ` -- the same agentic harness focused entirely on memory: episodic consolidation, entity tracking, interoception layers, and how Cortex reasons over them. - {doc}`Cortex: The Agentic Harness ` -- the introductory tutorial if you want to start with a smaller graph before scaling up to navigation. - {doc}`Visualizing the System Graph <../events-and-resilience/visualizing-system-graph>` -- watch the System Graph render Cortex's tool palette and the goal-status events flowing through it. --- ```{tip} **Promote this recipe to production.** While you're shaping it, the script runs straight with `python recipe.py`. Once it's solid, drop it at `~/emos/recipes//recipe.py` and run `emos run ` -- you'll get sensor pre-flight checks, persistent logs, and a card on the dashboard so an operator can launch it from a browser. See [Running Recipes](../../getting-started/running-recipes.md) for the full development-vs-production comparison and install-mode pitfalls (especially in Container mode). ``` ``` ## File: recipes/planning-and-manipulation/planning-models.md ```markdown # Multimodal Planning Previously in the [Go-to-X Recipe](../foundation/goto-navigation.md) we created an agent capable of understanding and responding to go-to commands. This agent relied on a semantic map that was stored in a vector database that could be accessed by an LLM component for doing retrieval augmented generation. Through the magic of tool use (or manual post-processing), we were able to extract position coordinates from our vectorized information and send it to a `Pose` topic for goal-point navigation by an autonomous navigation system. In this example, we will see how we can generate a similar navigation goal, but from the visual input coming in from the robot's sensors -- i.e. we should be able to ask our physical agent to navigate to an object that is in its sight. We will achieve this by utilizing two components in our agent: an LLM component and a VLM component. The LLM component will act as a sentence parser, isolating the object description from the user's command. The VLM component will use a planning Vision Language Model (VLM), which can perform visual grounding and pointing. ## Initialize the LLM component ```python from agents.components import LLM from agents.models import OllamaModel from agents.clients import OllamaClient from agents.ros import Topic # Start a Llama3.2 based llm component using ollama client llama = OllamaModel(name="llama", checkpoint="llama3.2:3b") llama_client = OllamaClient(llama) # Define LLM input and output topics including goal_point topic of type PoseStamped goto_in = Topic(name="goto_in", msg_type="String") llm_output = Topic(name="llm_output", msg_type="String") # initialize the component sentence_parser = LLM( inputs=[goto_in], outputs=[llm_output], model_client=llama_client, trigger=goto_in, component_name='sentence_parser' ) ``` In order to configure the component to act as a sentence parser, we will set a topic prompt on its input topic. ```python sentence_parser.set_topic_prompt(goto_in, template="""You are a sentence parsing software. Simply return the object description in the following command. {{ goto_in }}""" ) ``` ## Initialize the VLM component In this step, we will set up the VLM component, which will enable the agent to visually ground natural language object descriptions (from our command, given to the LLM component above) using live sensor data. We use **[RoboBrain 2.0](https://github.com/FlagOpen/RoboBrain2.0)** by BAAI, a state-of-the-art Vision-Language model (VLM) trained specifically for embodied agents reasoning. RoboBrain 2.0 supports a wide range of embodied perception and planning capabilities, including interactive reasoning and spatial perception. > **Citation**: > BAAI RoboBrain Team. "RoboBrain 2.0 Technical Report." arXiv preprint arXiv:2507.02029 (2025). > [https://arxiv.org/abs/2507.02029](https://arxiv.org/abs/2507.02029) In our scenario, we use RoboBrain2.0 to perform **grounding** -- that is, mapping the object description (parsed by the LLM component) to a visual detection in the agent's camera view. This detection includes spatial coordinates that can be forwarded to the navigation system for physical movement. RoboBrain2.0 is available in RoboML, which we are using as a model serving platform here. ```{note} RoboML is an aggregator library that provides a model serving apparatus for locally serving open-source ML models useful in robotics. Learn about setting up RoboML [here](https://www.github.com/automatika-robotics/roboml). ``` ```{important} **HuggingFace License Agreement & Authentication** The RoboBrain models are gated repositories on HuggingFace. To avoid "model not authorized" or `401 Client Error` messages: 1. **Agree to Terms:** You must sign in to your HuggingFace account and accept the license terms on the [model's repository page](https://huggingface.co/BAAI/RoboBrain2.0-3B) (or the variant you choose). 2. **Authenticate Locally:** Ensure your environment is authenticated by running `huggingface-cli login` in your terminal and entering your access token. ``` To configure this grounding behaviour, we initialize a `VLMConfig` object and set the `task` parameter to `"grounding"`: ```python config = VLMConfig(task="grounding") ``` ```{note} The `task` parameter specifies the type of multimodal operation the component should perform. Supported values are: * `"general"` -- free-form multimodal reasoning, produces output of type String * `"pointing"` -- provide a list of points on the object, produces output of type PointsOfInterest * `"affordance"` -- detect object affordances, produces output of type Detections * `"trajectory"` -- predict motion path in pixel space, produces output of type PointsOfInterest * `"grounding"` -- localize an object in the scene from a description with a bounding box, produces output of type Detections This parameter ensures the model behaves in a task-specific way, especially when using models like RoboBrain 2.0 that have been trained on multiple multimodal instruction types. ``` With this setup, the VLM component receives parsed object descriptions from the LLM and produces structured `Detections` messages identifying the object's location in space -- enabling the agent to navigate towards a visually grounded goal. Furthermore, we will use an _RGBD_ type message as the image input to the VLM component. This message is an aligned RGB and depth image message that is usually available in the ROS2 packages provided by stereo camera vendors (e.g. Realsense). The utility of this choice will become apparent later in this tutorial. ```python from agents.components import VLM from agents.models import RoboBrain2 from agents.clients import RoboMLHTTPClient from agents.config import VLMConfig # Start a RoboBrain2 based mllm component using RoboML client robobrain = RoboBrain2(name="robobrain") robobrain_client = RoboMLHTTPClient(robobrain) # Define VLM input/output topics rgbd0 = Topic(name="rgbd0", msg_type="RGBD") grounding_output = Topic(name="grounding_output", msg_type="Detections") # Set the task in VLMConfig config = VLMConfig(task="grounding") # initialize the component go_to_x = VLM( inputs=[llm_output, rgbd0], outputs=[grounding_output], model_client=robobrain_client, trigger=llm_output, config=config, component_name="go-to-x" ) ``` ```{warning} When a task is specified in VLMConfig, the VLM component automatically produces structured output depending on the task. The downstream consumers of this input should have appropriate callbacks configured for handling these output messages. ``` ## Configure Autonomous Navigation EMOS provides a complete navigation stack through [Kompass](https://github.com/automatika-robotics/kompass). It is built with the same underlying principles as EmbodiedAgents -- event-driven and customizable with a simple Python script. In this section we will show how to start navigation in the same recipe that we have been developing for a vision guided, go-to agent. ```{note} Learn about installing the EMOS navigation stack in the [installation guide](../../getting-started/installation.md). ``` EMOS allows for various kinds of navigation behaviour configured in the same recipe. However, we will only be using point-to-point navigation and the default configuration for its components. Since navigation is central to our task, as a first step, we will configure the robot and its motion model. EMOS provides a `RobotConfig` primitive where you can add your robot's motion model (ACKERMANN, OMNI, DIFFERENTIAL_DRIVE), the robot geometry parameters and the robot control limits: ```python import numpy as np from kompass.robot import ( AngularCtrlLimits, LinearCtrlLimits, RobotGeometry, RobotType, ) from kompass.config import RobotConfig # Setup your robot configuration my_robot = RobotConfig( model_type=RobotType.DIFFERENTIAL_DRIVE, geometry_type=RobotGeometry.Type.CYLINDER, geometry_params=np.array([0.1, 0.3]), ctrl_vx_limits=LinearCtrlLimits(max_vel=0.2, max_acc=1.5, max_decel=2.5), ctrl_omega_limits=AngularCtrlLimits( max_vel=0.4, max_acc=2.0, max_decel=2.0, max_steer=np.pi / 3 ), ) ``` Now we can add our default components. Our component of interest is the _planning_ component, that plots a path to the goal point. We will give the output topic from our VLM component as the goal point topic to the planning component. ```{important} The Kompass Planner accepts `Detections`, `PointsOfInterest`, and `Trackings` messages from EmbodiedAgents directly as goal-point inputs. These contain pixel-space coordinates identified by ML models. When generated from RGBD inputs, the associated depth images enable Kompass to automatically convert pixel-space points to averaged world-space coordinates using camera intrinsics. See [Planning](../../navigation/planning.md) for the supported input types. ``` ```python from kompass.components import ( Controller, Planner, DriveManager, LocalMapper, ) # Setup components with default config, inputs and outputs planner = Planner(component_name="planner") # Set our grounding output as the goal_point in the planner component planner.inputs(goal_point=grounding_output) # Get a default Local Mapper component mapper = LocalMapper(component_name="mapper") # Get a default controller component controller = Controller(component_name="controller") # Configure Controller to use local map instead of direct sensor information controller.direct_sensor = False # Setup a default drive manager driver = DriveManager(component_name="drive_manager") ``` ```{seealso} Learn the details of point navigation in EMOS using the step-by-step [Point Navigation](../navigation/point-navigation.md) recipe. ``` ## Launching the Components Now we will launch our Go-to-X component and navigation components using the same launcher. ```python from kompass.ros import Launcher launcher = Launcher() # Add the intelligence components launcher.add_pkg( components=[sentence_parser, go_to_x], package_name="automatika_embodied_agents", multiprocessing=True, ros_log_level="warn", ) # Add the navigation components launcher.add_pkg( components=[planner, controller, mapper, driver], package_name="kompass", multiprocessing=True, ) # Set the robot config for all components as defined above and bring up launcher.robot = my_robot launcher.bringup() ``` And that is all. Our Go-to-X component is ready. The complete code for this example is given below: ```{code-block} python :caption: Vision Guided Go-to-X Component :linenos: import numpy as np from agents.components import LLM from agents.models import OllamaModel from agents.clients import OllamaClient from agents.ros import Topic from agents.components import VLM from agents.models import RoboBrain2 from agents.clients import RoboMLHTTPClient from agents.config import VLMConfig from kompass.robot import ( AngularCtrlLimits, LinearCtrlLimits, RobotGeometry, RobotType, ) from kompass.config import RobotConfig from kompass.components import ( Controller, Planner, DriveManager, LocalMapper, ) from kompass.ros import Launcher # Start a Llama3.2 based llm component using ollama client llama = OllamaModel(name="llama", checkpoint="llama3.2:3b") llama_client = OllamaClient(llama) # Define LLM input and output topics including goal_point topic of type PoseStamped goto_in = Topic(name="goto_in", msg_type="String") llm_output = Topic(name="llm_output", msg_type="String") # initialize the component sentence_parser = LLM( inputs=[goto_in], outputs=[llm_output], model_client=llama_client, trigger=goto_in, component_name='sentence_parser' ) # Start a RoboBrain2 based mllm component using RoboML client robobrain = RoboBrain2(name="robobrain") robobrain_client = RoboMLHTTPClient(robobrain) # Define VLM input/output topics rgbd0 = Topic(name="rgbd0", msg_type="RGBD") grounding_output = Topic(name="grounding_output", msg_type="Detections") # Set the task in VLMConfig config = VLMConfig(task="grounding") # initialize the component go_to_x = VLM( inputs=[llm_output, rgbd0], outputs=[grounding_output], model_client=robobrain_client, trigger=llm_output, config=config, component_name="go-to-x" ) # Setup your robot configuration my_robot = RobotConfig( model_type=RobotType.DIFFERENTIAL_DRIVE, geometry_type=RobotGeometry.Type.CYLINDER, geometry_params=np.array([0.1, 0.3]), ctrl_vx_limits=LinearCtrlLimits(max_vel=0.2, max_acc=1.5, max_decel=2.5), ctrl_omega_limits=AngularCtrlLimits( max_vel=0.4, max_acc=2.0, max_decel=2.0, max_steer=np.pi / 3 ), ) # Setup components with default config, inputs and outputs planner = Planner(component_name="planner") # Set our grounding output as the goal_point in the planner component planner.inputs(goal_point=grounding_output) # Get a default Local Mapper component mapper = LocalMapper(component_name="mapper") # Get a default controller component controller = Controller(component_name="controller") # Configure Controller to use local map instead of direct sensor information controller.direct_sensor = False # Setup a default drive manager driver = DriveManager(component_name="drive_manager") launcher = Launcher() # Add the intelligence components launcher.add_pkg( components=[sentence_parser, go_to_x], package_name="automatika_embodied_agents", multiprocessing=True, ros_log_level="warn", ) # Add the navigation components launcher.add_pkg( components=[planner, controller, mapper, driver], package_name="kompass", multiprocessing=True, ) # Set the robot config for all components as defined above and bring up launcher.robot = my_robot launcher.bringup() ``` --- ```{tip} **Promote this recipe to production.** While you're shaping it, the script runs straight with `python recipe.py`. Once it's solid, drop it at `~/emos/recipes//recipe.py` and run `emos run ` -- you'll get sensor pre-flight checks, persistent logs, and a card on the dashboard so an operator can launch it from a browser. See [Running Recipes](../../getting-started/running-recipes.md) for the full development-vs-production comparison and install-mode pitfalls (especially in Container mode). ``` ``` ## File: recipes/planning-and-manipulation/vla-manipulation.md ```markdown # VLA Manipulation The frontier of Embodied AI is moving away from modular pipelines (perception -> planning -> control) toward end-to-end learning. **Vision-Language-Action (VLA)** models take visual observations and natural language instructions as input and output direct robot joint commands. In this tutorial, we will build an agent capable of performing physical manipulation tasks using the **VLA** component. We will utilize the [LeRobot](https://github.com/huggingface/lerobot) ecosystem to load a pretrained "SmolVLA" policy and connect it to a robot arm. ````{important} In order to run this tutorial you will need to install LeRobot as a model serving platform. You can see the installation instructions [here](https://huggingface.co/docs/lerobot/installation). After installation run the LeRobot async inference server as follows. ```shell python -m lerobot.async_inference.policy_server --host= --port= ```` ## Simulation Setup **WILL BE ADDED SOON** ## Setting up our VLA based Agent We will start by importing the relevant components. ```python from agents.components import VLA from agents.clients import LeRobotClient from agents.models import LeRobotPolicy ``` ## Defining the Senses and Actuators Unlike purely digital agents, a VLA agent needs to be firmly grounded in its physical body. We need to define the ROS topics that represent the robot's state (proprioception), its vision (eyes), and its actions (motor commands). In this example, we are working with a so101 arm setup requiring two camera angles, so we define two camera inputs alongside the robot's joint states. ```python from agents.ros import Topic # 1. Proprioception: The current angle of the robot's joints state = Topic(name="/isaac_joint_states", msg_type="JointState") # 2. Vision: The agent's eyes camera1 = Topic(name="/front_camera/image_raw", msg_type="Image") camera2 = Topic(name="/wrist_camera/image_raw", msg_type="Image") # 3. Action: Where the VLA will publish command outputs joints_action = Topic(name="/isaac_joint_command", msg_type="JointState") ``` ## Setting up the Policy To drive our VLA component, we need a robot policy. EMOS provides the `LeRobotPolicy` class, which interfaces seamlessly with models trained with LeRobot and hosted on the HuggingFace Hub. We will use a finetuned **SmolVLA** model, a lightweight VLA policy trained by the LeRobot team and finetuned on our simulation scenario setup above. We also need to provide a `dataset_info_file`. This is useful because the VLA needs to know the statistical distribution of the training data (normalization stats) to correctly interpret the robot's raw inputs. This file is part of the standard LeRobot Dataset format. We will use the info file from the dataset on which our SmolVLA policy was finetuned on. ````{important} In order to use the LeRobotClient you will need extra dependencies that can be installed as follows: ```shell pip install grpcio protobuf pip install torch --index-url https://download.pytorch.org/whl/cpu # And a lightweight CPU version (recommended) of torch ```` ```python # Specify the LeRobot Policy to use policy = LeRobotPolicy( name="my_policy", policy_type="smolvla", checkpoint="aleph-ra/smolvla_finetune_pick_orange_20000", dataset_info_file="https://huggingface.co/datasets/LightwheelAI/leisaac-pick-orange/resolve/main/meta/info.json", ) # Create the client client = LeRobotClient(model=policy) ``` ```{note} The **policy_type** parameter supports various architectures including `diffusion`, `act`, `pi0`, and `smolvla`. Ensure this matches the architecture of your checkpoint. ``` ## VLA Configuration This is the most critical step. Pre-trained VLA models expect inputs to be named exactly as they were in the training dataset (e.g., "shoulder_pan.pos"). However, your robot's URDF likely uses different names (e.g., "Rotation" or "joint_1"). We use the `VLAConfig` to create a mapping layer that translates your robot's specific hardware signals into the language the model understands. 1. **Joint Mapping:** Map dataset keys to your ROS joint names. 2. **Camera Mapping:** Map dataset camera names to your ROS image topics. 3. **Safety Limits:** Provide the URDF file so the component knows the physical joint limits and can cap actions safely. ```python from agents.config import VLAConfig # Map dataset names (keys) -> Robot URDF names (values) joints_map = { "shoulder_pan.pos": "Rotation", "shoulder_lift.pos": "Pitch", "elbow_flex.pos": "Elbow", "wrist_flex.pos": "Wrist_Pitch", "wrist_roll.pos": "Wrist_Roll", "gripper.pos": "Jaw", } # Map dataset camera names (keys) -> ROS Topics (values) camera_map = {"front": camera1, "wrist": camera2} config = VLAConfig( observation_sending_rate=3, # Hz: How often we infer action_sending_rate=3, # Hz: How often we publish commands joint_names_map=joints_map, camera_inputs_map=camera_map, # URDF is required for safety capping and joint limit verification robot_urdf_file="./so101_new_calib.urdf" ) ``` ```{warning} If the `joint_names_map` is incomplete, the component will raise an error during initialization. ``` ## The VLA Component Now we assemble the component. The `VLA` component acts as a ROS2 Action Server. It creates a feedback loop: it ingests the state and images, processes them through the `LeRobotClient`, and publishes the resulting actions to the `joints_action` topic. We also define a termination trigger. Since VLA tasks (like picking up an object) are finite, we can tell the component to stop after a specific number of timesteps. ```{note} The termination trigger can be `timesteps`, `keyboard` and `event`. The event can be based on a topic published by another component observing the scene, for example a VLM component that is asking a periodic question to itself with a `FixedInput`. Check out the [Event-Driven VLA](event-driven-vla.md) recipe. ``` ```python from agents.components import VLA vla = VLA( inputs=[state, camera1, camera2], outputs=[joints_action], model_client=client, config=config, component_name="vla_with_smolvla", ) # Attach the stop trigger vla.set_termination_trigger("timesteps", max_timesteps=50) ``` ## Launching the Component ```python from agents.ros import Launcher launcher = Launcher() launcher.add_pkg(components=[vla]) launcher.bringup() ``` Now we can send our pick and place command to the component. Since the VLA component acts as a **ROS2 Action Server**, we can trigger it directly from the terminal using the standard `ros2 action` CLI. Open a new terminal, source your workspace and send the goal (the natural language instruction) to the component. The action server endpoint defaults to `component_name/action_name`. ```bash ros2 action send_goal /vla_with_smolvla/vision_language_action automatika_embodied_agents/action/VisionLanguageAction "{task: 'pick up the oranges and place them in the bowl'}" ``` ```{note} The `task` string is the natural language instruction that the VLA model conditions its actions on. Ensure this instruction matches the distribution of prompts used during the training of the model (e.g. "pick orange", "put orange in bin" etc). ``` And there you have it! You have successfully configured an end-to-end VLA agent. The complete code is available below. ```{code-block} python :caption: Vision Language Action Agent :linenos: from agents.components import VLA from agents.config import VLAConfig from agents.clients import LeRobotClient from agents.models import LeRobotPolicy from agents.ros import Topic, Launcher # --- Define Topics --- state = Topic(name="/isaac_joint_states", msg_type="JointState") camera1 = Topic(name="/front_camera/image_raw", msg_type="Image") camera2 = Topic(name="/wrist_camera/image_raw", msg_type="Image") joints_action = Topic(name="/isaac_joint_command", msg_type="JointState") # --- Setup Policy (The Brain) --- policy = LeRobotPolicy( name="my_policy", policy_type="smolvla", checkpoint="aleph-ra/smolvla_finetune_pick_orange_20000", dataset_info_file="https://huggingface.co/datasets/LightwheelAI/leisaac-pick-orange/resolve/main/meta/info.json", ) client = LeRobotClient(model=policy) # --- Configure Mapping (The Nervous System) --- # Map dataset names -> robot URDF names joints_map = { "shoulder_pan.pos": "Rotation", "shoulder_lift.pos": "Pitch", "elbow_flex.pos": "Elbow", "wrist_flex.pos": "Wrist_Pitch", "wrist_roll.pos": "Wrist_Roll", "gripper.pos": "Jaw", } # Map dataset cameras -> ROS topics camera_map = {"front": camera1, "wrist": camera2} config = VLAConfig( observation_sending_rate=3, action_sending_rate=3, joint_names_map=joints_map, camera_inputs_map=camera_map, # Ensure you provide a valid path to your robot's URDF robot_urdf_file="./so101_new_calib.urdf" ) # --- Initialize Component --- vla = VLA( inputs=[state, camera1, camera2], outputs=[joints_action], model_client=client, config=config, component_name="vla_with_smolvla", ) # Set the component to stop after a certain number of timesteps vla.set_termination_trigger('timesteps', max_timesteps=50) # --- Launch --- launcher = Launcher() launcher.add_pkg(components=[vla]) launcher.bringup() ``` --- ```{tip} **Promote this recipe to production.** While you're shaping it, the script runs straight with `python recipe.py`. Once it's solid, drop it at `~/emos/recipes//recipe.py` and run `emos run ` -- you'll get sensor pre-flight checks, persistent logs, and a card on the dashboard so an operator can launch it from a browser. See [Running Recipes](../../getting-started/running-recipes.md) for the full development-vs-production comparison and install-mode pitfalls (especially in Container mode). ``` ``` ## File: recipes/planning-and-manipulation/event-driven-vla.md ```markdown # Event-Driven VLA In the previous [VLA Manipulation](vla-manipulation.md) recipe, we saw how VLAs can be used in EMOS to perform physical tasks. However, the real utility of VLAs is unlocked when they are part of a bigger cognitive system. With its event-driven agent graph development, EMOS allows us to do exactly that. Most VLA policies are "open-loop" regarding task completion -- they run for a fixed number of steps and then stop, regardless of whether they succeeded or failed. In this tutorial, we will build a **Closed-Loop Agent** while using an open-loop policy. Even if the model correctly outputs its termination condition (i.e. an absorbing state policy), our design can act as a safety valve. We will combine: - {material-regular}`smart_toy;1.2em;sd-text-primary` **The Player (VLA):** Attempts to pick up an object. - {material-regular}`visibility;1.2em;sd-text-primary` **The Referee (VLM):** Watches the camera stream and judges if the task is complete. We will use the **Event System** to trigger a stop command on the VLA the moment the VLM confirms success. ## The Player: Setting up the VLA First, we setup our VLA component exactly as we did in the previous recipe. We will use the same **SmolVLA** policy trained for picking oranges. ```python from agents.components import VLA from agents.config import VLAConfig from agents.clients import LeRobotClient from agents.models import LeRobotPolicy from agents.ros import Topic # Define Topics state = Topic(name="/isaac_joint_states", msg_type="JointState") camera1 = Topic(name="/front_camera/image_raw", msg_type="Image") camera2 = Topic(name="/wrist_camera/image_raw", msg_type="Image") joints_action = Topic(name="/isaac_joint_command", msg_type="JointState") # Setup Policy policy = LeRobotPolicy( name="my_policy", policy_type="smolvla", checkpoint="aleph-ra/smolvla_finetune_pick_orange_20000", dataset_info_file="https://huggingface.co/datasets/LightwheelAI/leisaac-pick-orange/resolve/main/meta/info.json", ) client = LeRobotClient(model=policy) # Configure VLA (Mapping omitted for brevity, see previous tutorial) # ... (assume joints_map and camera_map are defined) config = VLAConfig( observation_sending_rate=5, action_sending_rate=5, joint_names_map=joints_map, camera_inputs_map=camera_map, robot_urdf_file="./so101_new_calib.urdf" ) player = VLA( inputs=[state, camera1, camera2], outputs=[joints_action], model_client=client, config=config, component_name="vla_player", ) ``` ## The Referee: Setting up the VLM Now we introduce the "Referee". We will use a Vision Language Model (like Qwen-VL) to monitor the scene. We want this component to periodically look at the `camera1` feed and answer a specific question: _"Are all the oranges in the bowl?"_ We use a `FixedInput` to ensure the VLM is asked the exact same question every time. ```python from agents.components import VLM from agents.clients import OllamaClient from agents.models import OllamaModel from agents.ros import FixedInput # Define the topic where the VLM publishes its judgment referee_verdict = Topic(name="/referee/verdict", msg_type="String") # Setup the Model qwen_vl = OllamaModel(name="qwen_vl", checkpoint="qwen2.5vl:7b") qwen_client = OllamaClient(model=qwen_vl) # Define the constant question question = FixedInput( name="prompt", msg_type="String", fixed="Look at the image. Are all the orange in the bowl? Answer only with YES or NO." ) # Initialize the VLM # Note: We trigger periodically (regulated by loop_rate) referee = VLM( inputs=[question, camera1], outputs=[referee_verdict], model_client=qwen_client, trigger=10.0, component_name="vlm_referee" ) ``` ```{note} To prevent the VLM from consuming too much compute, we have configured a `float` trigger, which means our `VLM` component will be triggered, not by a topic, but periodically with a `loop_rate` of once every 10 seconds. ``` ```{tip} In order to make sure that the VLM output is formatted as per our requirement (YES or NO), checkout how to use pre-processors in the [Semantic Map](../foundation/semantic-map.md) recipe. For now we will assume that if YES is part of the output string, the event should fire. ``` ## The Bridge: Semantic Event Trigger Now comes the "Self-Referential" magic. We simply define an **Event** that fires when the `/referee/verdict` topic contains the word "YES". ```python from agents.ros import Event # Define the Success Event event_task_success = Event( referee_verdict.msg.data.contains("YES") # the topic, attribute and value to check in it ) ``` Finally, we attach this event to the VLA using the `set_termination_trigger` method. We set the mode to `event`. ```python # Tell the VLA to stop immediately when the event fires player.set_termination_trigger( mode="event", stop_event=event_task_success, max_timesteps=500 # Fallback: stop if 500 steps pass without success ) ``` ```{seealso} Events are a very powerful concept in EMOS. You can get infinitely creative with them. For example, imagine setting off the VLA component with a voice command. This can be done by combining the output of a SpeechToText component and an Event that generates an action command. To learn more about them check out the recipes for [Events & Actions](../events-and-resilience/event-driven-cognition.md). ``` ## Launching the System When we launch this graph: - The **VLA** starts moving the robot to pick the orange. - The **VLM** simultaneously watches the feed. - Once the oranges are in the bowl, the VLM outputs "YES". - The **Event** system catches this, interrupts the VLA, and signals that the task is complete. ```python from agents.ros import Launcher launcher = Launcher() launcher.add_pkg(components=[player, referee]) launcher.bringup() ``` You can send the action command to the VLA as defined in the previous [VLA Manipulation](vla-manipulation.md) recipe. ## Complete Code ```{code-block} python :caption: Closed-Loop VLA with VLM Verifier :linenos: from agents.components import VLA, VLM from agents.config import VLAConfig from agents.clients import LeRobotClient, OllamaClient from agents.models import LeRobotPolicy, OllamaModel from agents.ros import Topic, Launcher, FixedInput from agents.ros import Event # --- Define Topics --- state = Topic(name="/isaac_joint_states", msg_type="JointState") camera1 = Topic(name="/front_camera/image_raw", msg_type="Image") camera2 = Topic(name="/wrist_camera/image_raw", msg_type="Image") joints_action = Topic(name="/isaac_joint_command", msg_type="JointState") referee_verdict = Topic(name="/referee/verdict", msg_type="String") # --- Setup The Player (VLA) --- policy = LeRobotPolicy( name="my_policy", policy_type="smolvla", checkpoint="aleph-ra/smolvla_finetune_pick_orange_20000", dataset_info_file="https://huggingface.co/datasets/LightwheelAI/leisaac-pick-orange/resolve/main/meta/info.json", ) vla_client = LeRobotClient(model=policy) # VLA Config (Mappings assumed defined as per previous tutorial) # joints_map = { ... } # camera_map = { ... } config = VLAConfig( observation_sending_rate=5, action_sending_rate=5, joint_names_map=joints_map, camera_inputs_map=camera_map, robot_urdf_file="./so101_new_calib.urdf" ) player = VLA( inputs=[state, camera1, camera2], outputs=[joints_action], model_client=vla_client, config=config, component_name="vla_player", ) # --- Setup The Referee (VLM) --- qwen_vl = OllamaModel(name="qwen_vl", checkpoint="qwen2.5vl:7b") qwen_client = OllamaClient(model=qwen_vl) # A static prompt for the VLM question = FixedInput( name="prompt", msg_type="String", fixed="Look at the image. Are all the orange in the bowl? Answer only with YES or NO." ) referee = VLM( inputs=[question, camera1], outputs=[referee_verdict], model_client=qwen_client, trigger=camera1, component_name="vlm_referee" ) # --- Define the Logic (Event) --- # Create an event that looks for "YES" in the VLM's output event_task_success = Event( referee_verdict.msg.data.contains("YES") # the topic, attribute and value to check in it ) # Link the event to the VLA's stop mechanism player.set_termination_trigger( mode="event", stop_event=event_success, max_timesteps=400 # Failsafe ) # --- Launch --- launcher = Launcher() launcher.add_pkg(components=[player, referee]) launcher.bringup() ``` --- ```{tip} **Promote this recipe to production.** While you're shaping it, the script runs straight with `python recipe.py`. Once it's solid, drop it at `~/emos/recipes//recipe.py` and run `emos run ` -- you'll get sensor pre-flight checks, persistent logs, and a card on the dashboard so an operator can launch it from a browser. See [Running Recipes](../../getting-started/running-recipes.md) for the full development-vs-production comparison and install-mode pitfalls (especially in Container mode). ``` ``` ## File: recipes/navigation/simulation-quickstarts.md ```markdown # Simulation Quick Starts Ready to see EMOS in action? This page walks you through launching a full autonomous navigation stack in simulation using either **Webots** or **Gazebo**. Each section is self-contained -- pick the simulator you prefer and follow along. --- ## Webots Simulator **Launch a full autonomous navigation stack in under 5 minutes.** In this tutorial, we use a single Python script -- a "Recipe" -- to build a complete point-to-point navigation system. We will use the [Webots](https://github.com/cyberbotics/webots_ros2) simulator and a [Turtlebot3](https://emanual.robotis.com/docs/en/platform/turtlebot3/overview/#notices) to demonstrate how EMOS components link together. ### 1. Prepare the Environment To make things easy, we created **kompass_sim**, a package with ready-to-launch simulation environments. 1. **Build the Simulation:** Clone and build the simulator support package in your ROS2 workspace: ```bash git clone https://github.com/automatika-robotics/kompass-sim.git cd .. && rosdep install --from-paths src --ignore-src -r -y colcon build --packages-select kompass_sim source install/setup.bash ``` 2. **Launch Webots:** Start the Turtlebot3 simulation world. This will bring up Webots, RViz, and the robot localization nodes: ```bash ros2 launch kompass_sim webots_turtlebot3.launch.py ``` ### 2. The Navigation Recipe The power of EMOS lies in its Python API. Instead of complex XML/YAML launch files, you define your navigation logic in a clean script. **Create a file named `quick_start.py` and paste the following code:** ```python import numpy as np import os from ament_index_python.packages import ( get_package_share_directory, ) # IMPORT ROBOT CONFIG PRIMITIVES from kompass.robot import ( AngularCtrlLimits, LinearCtrlLimits, RobotGeometry, RobotType, RobotConfig, RobotFrames, ) # IMPORT EMOS NAVIGATION COMPONENTS from kompass.components import ( Controller, DriveManager, DriveManagerConfig, Planner, PlannerConfig, LocalMapper, LocalMapperConfig, MapServer, MapServerConfig, TopicsKeys, ) # IMPORT ALGORITHMS CONFIG from kompass.control import ControllersID, MapConfig # IMPORT ROS PRIMITIVES from kompass.ros import Topic, Launcher, Event, Action, actions kompass_sim_dir = get_package_share_directory(package_name="kompass_sim") # Setup your robot configuration my_robot = RobotConfig( model_type=RobotType.DIFFERENTIAL_DRIVE, geometry_type=RobotGeometry.Type.CYLINDER, geometry_params=np.array([0.1, 0.3]), ctrl_vx_limits=LinearCtrlLimits(max_vel=0.4, max_acc=1.5, max_decel=2.5), ctrl_omega_limits=AngularCtrlLimits( max_vel=0.4, max_acc=2.0, max_decel=2.0, max_steer=np.pi / 3 ), ) # Configure the Global Planner planner_config = PlannerConfig(loop_rate=1.0) planner = Planner(component_name="planner", config=planner_config) planner.run_type = "Timed" # Configure the motion controller controller = Controller(component_name="controller") controller.algorithm = ControllersID.PURE_PURSUIT controller.direct_sensor = ( False # Get local perception from a "map" instead (from the local mapper) ) # Configure the Drive Manager (Direct commands sending to robot) driver_config = DriveManagerConfig( critical_zone_distance=0.05, critical_zone_angle=90.0, slowdown_zone_distance=0.3, ) driver = DriveManager(component_name="drive_manager", config=driver_config) # Publish Twist or TwistStamped from the DriveManager based on the distribution if "ROS_DISTRO" in os.environ and ( os.environ["ROS_DISTRO"] in ["rolling", "jazzy", "kilted"] ): cmd_msg_type: str = "TwistStamped" else: cmd_msg_type = "Twist" driver.outputs(robot_command=Topic(name="/cmd_vel", msg_type=cmd_msg_type)) # Configure a Local Mapper local_mapper_config = LocalMapperConfig( map_params=MapConfig(width=3.0, height=3.0, resolution=0.05) ) local_mapper = LocalMapper(component_name="mapper", config=local_mapper_config) # Configure the global Map Server map_file = os.path.join(kompass_sim_dir, "maps", "turtlebot3_webots.yaml") map_server_config = MapServerConfig( loop_rate=1.0, map_file_path=map_file, # Path to a 2D map yaml file or a point cloud file grid_resolution=0.5, pc_publish_row=False, ) map_server = MapServer(component_name="global_map_server", config=map_server_config) # Setup the launcher launcher = Launcher() # Add navigation components launcher.kompass( components=[map_server, controller, planner, driver, local_mapper], multiprocessing=True, ) # Get odom from localizer filtered odom for all components odom_topic = Topic(name="/odometry/filtered", msg_type="Odometry") launcher.inputs(location=odom_topic) # Set the robot config for all components launcher.robot = my_robot launcher.frames = RobotFrames(world="map", odom="map", scan="LDS-01") # Enable the UI # Outputs: Static Map, Global Plan, Robot Odometry launcher.enable_ui( outputs=[ map_server.get_out_topic(TopicsKeys.GLOBAL_MAP), odom_topic, planner.get_out_topic(TopicsKeys.GLOBAL_PLAN), ], ) # Run the Recipe launcher.bringup() ``` ### 3. Run and Navigate Open a new terminal and run your recipe: ```bash python3 quick_start.py ``` You will see the components starting up in the terminal. Once ready, you have two ways to control the robot. #### Option A: The EMOS Web UI The recipe includes `launcher.enable_ui(...)`, which automatically spins up a lightweight web interface for monitoring and control. 1. **Check Terminal:** Look for a log message indicating the UI URL: `http://0.0.0.0:5001`. 2. **Open Browser:** Navigate to that URL. 3. **Send Goal:** You will see the map and the robot's live position. Simply click the publish point button and **click anywhere on the map** to trigger the Planner and send the robot to that location. #### Option B: RViz If you prefer the standard ROS tools: 1. Go to the **RViz** window launched in Step 1. 2. Select the **Publish Point** tool (sometimes called `Clicked Point`) from the top toolbar. 3. Click anywhere on the map grid. 4. The robot will plan a path (Blue Line) and immediately start driving. ### What just happened? * **Components**: You configured your robot and the navigation components directly in your Python recipe. * **Launcher**: Automatically managed the lifecycle of 5 ROS2 nodes in multi-processing. * **Web UI**: Visualized the map, plan, and odometry topics instantly without installing extra frontend tools. --- ## Gazebo Simulator **Launch a full autonomous navigation stack in under 5 minutes.** In this tutorial, we use a single Python script -- a "Recipe" -- to build a complete point-to-point navigation system. We will use the [Gazebo](https://gazebosim.org/docs/latest/getstarted/) simulator and a [Turtlebot3 Waffle Pi](https://emanual.robotis.com/docs/en/platform/turtlebot3/overview/#notices) to demonstrate how EMOS components link together. ### 1. Install Gazebo If you haven't already, install the default Gazebo version for your ROS distribution (replace `${ROS_DISTRO}` with `humble`, `jazzy`, or `rolling`): ```bash sudo apt-get install ros-${ROS_DISTRO}-ros-gz ``` ### 2. Prepare the Environment To make things easy, we created **kompass_sim**, a package with ready-to-launch simulation environments. 1. **Build the Simulation:** Clone and build the simulator support package in your ROS2 workspace: ```bash git clone https://github.com/automatika-robotics/kompass-sim.git cd .. && rosdep install --from-paths src --ignore-src -r -y colcon build --packages-select kompass_sim source install/setup.bash ``` 2. **Set the Model:** Tell the simulation to use the "Waffle Pi" model: ```bash export TURTLEBOT3_MODEL=waffle_pi ``` 3. **Launch Gazebo:** Start the Turtlebot3 house simulation. This will bring up Gazebo, RViz, and the localization nodes: ```bash ros2 launch kompass_sim gazebo_turtlebot3_house.launch.py ``` ### 3. The Navigation Recipe The power of EMOS lies in its Python API. Instead of complex XML/YAML launch files, you define your navigation logic in a clean script. **Create a file named `quick_start_gz.py` and paste the following code:** ```python import numpy as np import os from ament_index_python.packages import get_package_share_directory # IMPORT ROBOT CONFIG PRIMITIVES from kompass.robot import ( AngularCtrlLimits, LinearCtrlLimits, RobotGeometry, RobotType, RobotConfig, RobotFrames, ) # IMPORT EMOS NAVIGATION COMPONENTS from kompass.components import ( Controller, DriveManager, DriveManagerConfig, Planner, PlannerConfig, LocalMapper, LocalMapperConfig, MapServer, MapServerConfig, TopicsKeys, ) # IMPORT ALGORITHMS CONFIG from kompass.control import ControllersID, MapConfig # IMPORT ROS PRIMITIVES from kompass.ros import Topic, Launcher, Event, Action, actions kompass_sim_dir = get_package_share_directory(package_name="kompass_sim") # Setup your robot configuration (Turtlebot3 Waffle Pi) my_robot = RobotConfig( model_type=RobotType.DIFFERENTIAL_DRIVE, geometry_type=RobotGeometry.Type.BOX, # Waffle Pi is rectangular geometry_params=np.array([0.3, 0.3, 0.2]), # Length, Width, Height ctrl_vx_limits=LinearCtrlLimits(max_vel=0.26, max_acc=1.0, max_decel=1.0), ctrl_omega_limits=AngularCtrlLimits( max_vel=1.8, max_acc=2.0, max_decel=2.0, max_steer=np.pi / 3 ), ) # Configure the Global Planner planner_config = PlannerConfig(loop_rate=1.0) planner = Planner(component_name="planner", config=planner_config) planner.run_type = "Timed" # Configure the motion controller controller = Controller(component_name="controller") controller.algorithm = ControllersID.PURE_PURSUIT controller.direct_sensor = ( False # Get local perception from a "map" instead (from the local mapper) ) # Configure the Drive Manager (Direct commands sending to robot) driver_config = DriveManagerConfig( critical_zone_distance=0.05, critical_zone_angle=90.0, slowdown_zone_distance=0.3, ) driver = DriveManager(component_name="drive_manager", config=driver_config) # Handle Twist/TwistStamped compatibility if "ROS_DISTRO" in os.environ and ( os.environ["ROS_DISTRO"] in ["rolling", "jazzy", "kilted"] ): cmd_msg_type: str = "TwistStamped" else: cmd_msg_type = "Twist" driver.outputs(robot_command=Topic(name="/cmd_vel", msg_type=cmd_msg_type)) # Configure a Local Mapper local_mapper_config = LocalMapperConfig( map_params=MapConfig(width=3.0, height=3.0, resolution=0.05) ) local_mapper = LocalMapper(component_name="mapper", config=local_mapper_config) # Configure the global Map Server # Note: We use the 'house' map to match the Gazebo world map_file = os.path.join(kompass_sim_dir, "maps", "turtlebot3_gazebo_house.yaml") map_server_config = MapServerConfig( loop_rate=1.0, map_file_path=map_file, grid_resolution=0.5, pc_publish_row=False, ) map_server = MapServer(component_name="global_map_server", config=map_server_config) # Setup the launcher launcher = Launcher() # Add navigation components launcher.kompass( components=[map_server, controller, planner, driver, local_mapper], multiprocessing=True, ) # Get odom from localizer filtered odom for all components odom_topic = Topic(name="/odometry/filtered", msg_type="Odometry") launcher.inputs(location=odom_topic) # Set the robot config and frames launcher.robot = my_robot # Standard Gazebo TB3 frames: world=map, odom=odom, scan=base_scan launcher.frames = RobotFrames(world="map", odom="odom", scan="base_scan") # Enable the UI # Outputs: Static Map, Global Plan, Robot Odometry launcher.enable_ui( outputs=[ map_server.get_out_topic(TopicsKeys.GLOBAL_MAP), odom_topic, planner.get_out_topic(TopicsKeys.GLOBAL_PLAN), ], ) # Run the Recipe launcher.bringup() ``` ### 4. Run and Navigate Open a new terminal and run your recipe: ```bash python3 quick_start_gz.py ``` You will see the components starting up in the terminal. Once ready, you have two ways to control the robot. #### Option A: The EMOS Web UI The recipe includes `launcher.enable_ui(...)`, which automatically spins up a lightweight web interface for monitoring and control. 1. **Check Terminal:** Look for a log message indicating the UI URL: `http://0.0.0.0:5001`. 2. **Open Browser:** Navigate to that URL. 3. **Send Goal:** You will see the map and the robot's live position. Simply click the publish point button and **click anywhere on the map** to trigger the Planner and send the robot to that location. #### Option B: RViz If you prefer the standard ROS tools: 1. Go to the **RViz** window launched in Step 1. 2. Select the **Publish Point** tool (sometimes called `Clicked Point`) from the top toolbar. 3. Click anywhere on the map grid. 4. The robot will plan a path (Blue Line) and immediately start driving. ### What just happened? * **Customization**: We adapted the robot configuration (`RobotConfig`) to match the Waffle Pi's rectangular geometry and adjusted the `RobotFrames` to match Gazebo's standard output (`base_scan`). * **Launcher**: Managed the lifecycle of the entire stack. * **Perception**: The Local Mapper is processing the Gazebo laser scan to provide obstacle avoidance data to the Controller. --- ## Next Steps :::{tip} Check the [Point Navigation](point-navigation.md) recipe for a deep dive into these recipes. ::: --- ```{tip} **Promote this recipe to production.** While you're shaping it, the script runs straight with `python recipe.py`. Once it's solid, drop it at `~/emos/recipes//recipe.py` and run `emos run ` -- you'll get sensor pre-flight checks, persistent logs, and a card on the dashboard so an operator can launch it from a browser. See [Running Recipes](../../getting-started/running-recipes.md) for the full development-vs-production comparison and install-mode pitfalls (especially in Container mode). ``` ``` ## File: recipes/navigation/point-navigation.md ```markdown # Point Navigation In the [Simulation Quick Starts](simulation-quickstarts.md), you ran a script that launched a full navigation stack. Now, let's break that script down step-by-step to understand how to configure EMOS navigation for your specific needs. ## Step 1: Robot Configuration The first step is to tell EMOS *what* it is driving. The `RobotConfig` object defines the physical constraints and kinematics of your platform. This is crucial because the **Controller** uses these limits to generate feasible velocity commands, and the **Planner** uses the geometry to check for collisions. ```python # 1. Define the Robot my_robot = RobotConfig( model_type=RobotType.DIFFERENTIAL_DRIVE, # Motion Model (e.g., Turtlebot3) geometry_type=RobotGeometry.Type.CYLINDER, geometry_params=np.array([0.1, 0.3]), # Radius=0.1m, Height=0.3m # 2. Define Control Limits ctrl_vx_limits=LinearCtrlLimits( max_vel=0.4, # Max speed (m/s) max_acc=1.5, # Max acceleration (m/s^2) max_decel=2.5 # Max deceleration (braking) ), ctrl_omega_limits=AngularCtrlLimits( max_vel=0.4, max_acc=2.0, max_decel=2.0, max_steer=np.pi / 3 ), ) ``` :::{tip} EMOS supports **Ackermann** (Car-like), **Differential Drive**, and **Omni-directional** models. Changing the `model_type` here automatically reconfigures the underlying control math. ::: --- ## Step 2: Core Components Next, we initialize the "Brains" of the operation. ### The Planner & Controller We use the **Pure Pursuit** algorithm for path tracking. Note the `direct_sensor=False` flag -- this tells the controller *not* to subscribe to raw sensor data directly, but to rely on the processed **Local Map** instead. ```python # Global Planner (runs at 1Hz) planner = Planner( component_name="planner", config=PlannerConfig(loop_rate=1.0) ) planner.run_type = "Timed" # Local Controller controller = Controller(component_name="controller") controller.algorithm = ControllersID.PURE_PURSUIT controller.direct_sensor = False # Use Local Mapper for perception ``` ### The Drive Manager This component sits between the controller and the motors. We configure **Safety Zones** here: if an obstacle breaches the `critical_zone_distance` (0.05m), the Drive Manager triggers a hardware-level stop, overriding the controller. ```python driver_config = DriveManagerConfig( critical_zone_distance=0.05, # Emergency Stop threshold critical_zone_angle=90.0, # Frontal cone angle slowdown_zone_distance=0.3, # Slow down threshold ) driver = DriveManager(component_name="drive_manager", config=driver_config) ``` ### Dynamic Message Types Different ROS2 versions use different message types for velocity (`Twist` vs `TwistStamped`). This snippet makes your recipe portable across **Humble**, **Jazzy**, and **Rolling**. ```python # Auto-detect ROS distribution if "ROS_DISTRO" in os.environ and ( os.environ["ROS_DISTRO"] in ["rolling", "jazzy", "kilted"] ): cmd_msg_type = "TwistStamped" else: cmd_msg_type = "Twist" # Bind the output topic driver.outputs(robot_command=Topic(name="/cmd_vel", msg_type=cmd_msg_type)) ``` --- ## Step 3: Mapping & Perception Navigation requires two types of maps: a **Static Global Map** for long-term planning, and a **Dynamic Local Map** for immediate obstacle avoidance. ```python # 1. Local Mapper: Builds a 3x3m sliding window around the robot local_mapper = LocalMapper( component_name="mapper", config=LocalMapperConfig( map_params=MapConfig(width=3.0, height=3.0, resolution=0.05) ) ) # 2. Map Server: Loads the static house map from a file map_server = MapServer( component_name="global_map_server", config=MapServerConfig( map_file_path=os.path.join(kompass_sim_dir, "maps", "turtlebot3_webots.yaml"), grid_resolution=0.5 ) ) ``` --- ## Step 4: The Launcher & UI Finally, the `Launcher` ties everything together. It manages the lifecycle of all nodes and sets up the **Web UI**. We use `enable_ui` to pipe data directly to the browser: * **Outputs:** We stream the Global Map, the Planned Path, and the Robot's Odometry to the browser for visualization. ```python launcher = Launcher() # 1. Register Components launcher.kompass( components=[map_server, controller, planner, driver, local_mapper], multiprocessing=True, ) # 2. Bind Odometry (Input for all components) odom_topic = Topic(name="/odometry/filtered", msg_type="Odometry") launcher.inputs(location=odom_topic) # 3. Apply Robot Config & Frames launcher.robot = my_robot launcher.frames = RobotFrames(world="map", odom="map", scan="LDS-01") # 4. Enable the Web Interface launcher.enable_ui( outputs=[ map_server.get_out_topic(TopicsKeys.GLOBAL_MAP), odom_topic, planner.get_out_topic(TopicsKeys.GLOBAL_PLAN), ], ) launcher.bringup() ``` --- ## Full Recipe Code Here is the complete script. You can save this as `nav_recipe.py` and run it in any workspace where EMOS is installed. ```python import numpy as np import os from ament_index_python.packages import get_package_share_directory from kompass.robot import ( AngularCtrlLimits, LinearCtrlLimits, RobotGeometry, RobotType, RobotConfig, RobotFrames ) from kompass.components import ( Controller, DriveManager, DriveManagerConfig, Planner, PlannerConfig, LocalMapper, LocalMapperConfig, MapServer, MapServerConfig, TopicsKeys ) from kompass.control import ControllersID, MapConfig from kompass.ros import Topic, Launcher def kompass_bringup(): kompass_sim_dir = get_package_share_directory(package_name="kompass_sim") # 1. Robot Configuration my_robot = RobotConfig( model_type=RobotType.DIFFERENTIAL_DRIVE, geometry_type=RobotGeometry.Type.CYLINDER, geometry_params=np.array([0.1, 0.3]), ctrl_vx_limits=LinearCtrlLimits(max_vel=0.4, max_acc=1.5, max_decel=2.5), ctrl_omega_limits=AngularCtrlLimits(max_vel=0.4, max_acc=2.0, max_decel=2.0, max_steer=np.pi / 3), ) # 2. Components planner = Planner(component_name="planner", config=PlannerConfig(loop_rate=1.0)) planner.run_type = "Timed" controller = Controller(component_name="controller") controller.algorithm = ControllersID.PURE_PURSUIT controller.direct_sensor = False driver = DriveManager( component_name="drive_manager", config=DriveManagerConfig(critical_zone_distance=0.05, slowdown_zone_distance=0.3) ) # 3. Dynamic Command Type cmd_type = "TwistStamped" if os.environ.get("ROS_DISTRO") in ["rolling", "jazzy"] else "Twist" driver.outputs(robot_command=Topic(name="/cmd_vel", msg_type=cmd_type)) # 4. Mapping local_mapper = LocalMapper( component_name="mapper", config=LocalMapperConfig(map_params=MapConfig(width=3.0, height=3.0, resolution=0.05)) ) map_server = MapServer( component_name="global_map_server", config=MapServerConfig( map_file_path=os.path.join(kompass_sim_dir, "maps", "turtlebot3_webots.yaml"), grid_resolution=0.5 ) ) # 5. Launch launcher = Launcher() launcher.kompass( components=[map_server, controller, planner, driver, local_mapper], multiprocessing=True, ) odom_topic = Topic(name="/odometry/filtered", msg_type="Odometry") launcher.inputs(location=odom_topic) launcher.robot = my_robot launcher.frames = RobotFrames(world="map", odom="map", scan="LDS-01") # 6. UI launcher.enable_ui( inputs=[planner.ui_main_action_input], outputs=[ map_server.get_out_topic(TopicsKeys.GLOBAL_MAP), odom_topic, planner.get_out_topic(TopicsKeys.GLOBAL_PLAN), ], ) launcher.bringup() if __name__ == "__main__": kompass_bringup() ``` --- ## Next Steps Congratulations! You have created a full production-grade navigation recipe. * **[Vision Tracking with RGB](vision-tracking-rgb.md)**: Replace the Pure Pursuit controller with a Vision Follower to chase targets. * **[Vision Tracking with Depth](vision-tracking-depth.md)**: Extend RGB tracking with depth sensing for more robust following. * **[Runtime Model Fallback](../events-and-resilience/fallback-recipes.md)**: Learn how to make your recipe robust by automatically restarting components if they crash. --- ```{tip} **Promote this recipe to production.** While you're shaping it, the script runs straight with `python recipe.py`. Once it's solid, drop it at `~/emos/recipes//recipe.py` and run `emos run ` -- you'll get sensor pre-flight checks, persistent logs, and a card on the dashboard so an operator can launch it from a browser. See [Running Recipes](../../getting-started/running-recipes.md) for the full development-vs-production comparison and install-mode pitfalls (especially in Container mode). ``` ``` ## File: recipes/navigation/path-recording.md ```markdown # Path Recording & Replay **Save successful paths and re-execute them on demand.** Sometimes you don't need a dynamic planner to calculate a new path every time. In scenarios like **routine patrols, warehousing, or repeatable docking**, it is often more reliable to record a "golden path" once and replay it exactly. The [Kompass](https://github.com/automatika-robotics/kompass) **Planner** component facilitates this via three ROS 2 services: 1. `save_plan_to_file` -- Saves the currently active plan (or recorded history) to a CSV file. 2. `load_plan_from_file` -- Loads a CSV file and publishes it as the current global plan. 3. `start_path_recording` -- Starts recording the robot's actual odometry history to be saved later. --- ## The Recipe This recipe sets up a basic navigation stack but exposes the **Save/Load/Record Services** to the Web UI instead of the standard "Click-to-Nav" action. **Create a file named `path_recorder.py`:** ```python import numpy as np import os from ament_index_python.packages import get_package_share_directory from kompass.robot import ( AngularCtrlLimits, LinearCtrlLimits, RobotGeometry, RobotType, RobotConfig, RobotFrames ) from kompass.components import ( DriveManager, DriveManagerConfig, Planner, PlannerConfig, MapServer, MapServerConfig, TopicsKeys, Controller ) from kompass.ros import Topic, Launcher, ServiceClientConfig from kompass.control import ControllersID from kompass_interfaces.srv import PathFromToFile, StartPathRecording def run_path_recorder(): kompass_sim_dir = get_package_share_directory(package_name="kompass_sim") # 1. Robot Configuration my_robot = RobotConfig( model_type=RobotType.DIFFERENTIAL_DRIVE, geometry_type=RobotGeometry.Type.CYLINDER, geometry_params=np.array([0.1, 0.3]), ctrl_vx_limits=LinearCtrlLimits(max_vel=0.4, max_acc=1.5, max_decel=2.5), ctrl_omega_limits=AngularCtrlLimits(max_vel=0.4, max_acc=2.0, max_decel=2.0, max_steer=np.pi / 3), ) # 2. Configure Components planner = Planner(component_name="planner", config=PlannerConfig(loop_rate=1.0)) planner.run_type = "Timed" controller = Controller(component_name="controller") controller.algorithm = ControllersID.PURE_PURSUIT controller.direct_sensor = True # Use direct sensor for simple obstacle checks driver = DriveManager( component_name="drive_manager", config=DriveManagerConfig(critical_zone_distance=0.05) ) # Handle message types (Twist vs TwistStamped) cmd_msg_type = "TwistStamped" if os.environ.get("ROS_DISTRO") in ["rolling", "jazzy", "kilted"] else "Twist" driver.outputs(robot_command=Topic(name="/cmd_vel", msg_type=cmd_msg_type)) map_server = MapServer( component_name="global_map_server", config=MapServerConfig( map_file_path=os.path.join(kompass_sim_dir, "maps", "turtlebot3_webots.yaml"), grid_resolution=0.5 ) ) # 3. Define Services for UI Interaction save_path_srv = ServiceClientConfig( name=f"{planner.node_name}/save_plan_to_file", srv_type=PathFromToFile ) load_path_srv = ServiceClientConfig( name=f"{planner.node_name}/load_plan_from_file", srv_type=PathFromToFile ) start_path_recording = ServiceClientConfig( name=f"{planner.node_name}/start_path_recording", srv_type=StartPathRecording ) # 4. Launch launcher = Launcher() launcher.kompass( components=[map_server, planner, driver, controller], multiprocessing=True, ) odom_topic = Topic(name="/odometry/filtered", msg_type="Odometry") launcher.inputs(location=odom_topic) launcher.robot = my_robot launcher.frames = RobotFrames(world="map", odom="map", scan="LDS-01") # 5. Enable UI with path services exposed as inputs launcher.enable_ui( inputs=[save_path_srv, load_path_srv, start_path_recording], outputs=[ map_server.get_out_topic(TopicsKeys.GLOBAL_MAP), odom_topic, planner.get_out_topic(TopicsKeys.GLOBAL_PLAN), ], ) launcher.bringup() if __name__ == "__main__": run_path_recorder() ``` --- ## Workflow: Two Ways to Generate a Path Once the recipe is running and you have the EMOS Web UI open (`http://0.0.0.0:5001`), you can generate a path using either the planner or by manually driving the robot. ### Option A: Save a Computed Plan **Use this if you want to save the path produced by the global planner and "freeze" that exact path for future use.** 1. **Generate Plan:** Trigger the planner (e.g., via the `/clicked_point` input on the UI). 2. **Verify:** Check that the generated path looks good on the map. 3. **Save:** In the UI Inputs panel, go to `planner/save_plan_to_file`: - **file_location:** `/tmp/` - **file_name:** `computed_path.csv` - Click **Send**. ### Option B: Record a Driven Path (Teleop) **Use this if you want the robot to follow a human-demonstrated path (e.g., a specific maneuver through a tight doorway).** 1. **Start Recording:** In the UI Inputs panel, select `planner/start_path_recording`: - **recording_time_step:** `0.1` (Records a point every 0.1 seconds) - Click **Call**. 2. **Drive:** Use your keyboard or joystick to drive the robot along the desired route: ```bash ros2 run teleop_twist_keyboard teleop_twist_keyboard ``` 3. **Save:** When finished, select `planner/save_plan_to_file`: - **file_location:** `/tmp/` - **file_name:** `driven_path.csv` - Click **Send**. Calling save automatically stops the recording process. --- ## Replay the Path Now that you have your "Golden Path" saved (either computed or recorded), you can replay it anytime. 1. **Restart:** You can restart the stack or simply clear the current plan. 2. **Load:** In the UI Inputs panel, select `planner/load_plan_from_file`: - **file_location:** `/tmp/` - **file_name:** `driven_path.csv` (or `computed_path.csv`) - Click **Send**. The planner immediately loads the file and publishes it as the **Global Plan**. The **Controller** receives this path and begins executing it immediately, retracing the recorded steps exactly. --- ## Use Cases - **Routine Patrols** -- Record a perfect lap around a facility and replay it endlessly. - **Complex Docking** -- Manually drive a complex approach to a charging station, save the plan, and use it for reliable docking. - **Multi-Robot Coordination** -- Share a single "highway" path file among multiple robots to ensure they stick to verified lanes. --- ## Next Steps - **[Automated Motion Testing](motion-testing.md)** -- Run system identification tests and record response data. - **[Point Navigation](point-navigation.md)** -- Learn the fundamentals of the navigation stack step by step. --- ```{tip} **Promote this recipe to production.** While you're shaping it, the script runs straight with `python recipe.py`. Once it's solid, drop it at `~/emos/recipes//recipe.py` and run `emos run ` -- you'll get sensor pre-flight checks, persistent logs, and a card on the dashboard so an operator can launch it from a browser. See [Running Recipes](../../getting-started/running-recipes.md) for the full development-vs-production comparison and install-mode pitfalls (especially in Container mode). ``` ``` ## File: recipes/navigation/motion-testing.md ```markdown # Automated Motion Testing **System identification and response recording made easy.** The [Kompass](https://github.com/automatika-robotics/kompass) **MotionServer** is a specialized component designed for robot calibration and system identification. It performs two critical tasks: 1. **Automated Testing** -- It sends open-loop reference commands (e.g., steps, circles) to the robot. 2. **Data Recording** -- It records the robot's actual response (Odometry) versus the sent command (`cmd_vel`) to a CSV file. This data is essential for tuning controllers, verifying kinematic constraints, or training machine learning models. --- ## The Recipe Below is a complete recipe to launch a simulation (or real robot), triggering the Motion Server via the Web UI. **Create a file named `motion_test.py`:** ```python import numpy as np import os from ament_index_python.packages import get_package_share_directory from kompass.robot import ( AngularCtrlLimits, LinearCtrlLimits, RobotGeometry, RobotType, RobotConfig, RobotFrames ) from kompass.components import ( DriveManager, DriveManagerConfig, MapServer, MapServerConfig, TopicsKeys, MotionServer, MotionServerConfig ) from kompass.ros import Topic, Launcher def run_motion_test(): kompass_sim_dir = get_package_share_directory(package_name="kompass_sim") # 1. Robot Configuration # Define physical limits (crucial for the MotionServer to generate valid test commands) my_robot = RobotConfig( model_type=RobotType.DIFFERENTIAL_DRIVE, geometry_type=RobotGeometry.Type.CYLINDER, geometry_params=np.array([0.1, 0.3]), ctrl_vx_limits=LinearCtrlLimits(max_vel=0.4, max_acc=1.5, max_decel=2.5), ctrl_omega_limits=AngularCtrlLimits(max_vel=0.4, max_acc=2.0, max_decel=2.0, max_steer=np.pi / 3), ) # 2. Configure Motion Server # Run "Circle Tests" for 10 seconds per test motion_config = MotionServerConfig( test_period=10.0, run_circle_test=True, run_step_test=False, tests_folder=os.path.expanduser("~/.kompass/tests") # Where to save CSVs ) motion_server = MotionServer(component_name="motion_server", config=motion_config) motion_server.run_type = "Event" # Wait for a trigger to start # 3. Drive Manager # Acts as the safety layer between MotionServer and the hardware driver = DriveManager( component_name="drive_manager", config=DriveManagerConfig(critical_zone_distance=0.05) ) # Handle ROS 2 distribution message types cmd_msg_type = "TwistStamped" if os.environ.get("ROS_DISTRO") in ["rolling", "jazzy", "kilted"] else "Twist" # 4. Wiring # The Driver publishes the final hardware command cmd_topic = Topic(name="/cmd_vel", msg_type=cmd_msg_type) driver.outputs(robot_command=cmd_topic) # The MotionServer listens to that SAME topic to record what was actually sent motion_server.inputs(command=cmd_topic) # 5. Context (Map Server) map_server = MapServer( component_name="global_map_server", config=MapServerConfig( map_file_path=os.path.join(kompass_sim_dir, "maps", "turtlebot3_webots.yaml"), grid_resolution=0.5 ) ) # 6. Launch launcher = Launcher() launcher.kompass(components=[map_server, driver, motion_server], multiprocessing=True) # Link Odometry (the response we want to record) odom_topic = Topic(name="/odometry/filtered", msg_type="Odometry") launcher.inputs(location=odom_topic) launcher.robot = my_robot launcher.frames = RobotFrames(world="map", odom="map", scan="LDS-01") # 7. Enable UI # Expose the RUN_TESTS input so we can trigger it from the browser launcher.enable_ui( inputs=[motion_server.get_in_topic(TopicsKeys.RUN_TESTS)], outputs=[map_server.get_out_topic(TopicsKeys.GLOBAL_MAP), odom_topic] ) launcher.bringup() if __name__ == "__main__": run_motion_test() ``` --- ## How to Run the Test ### 1. Launch the Stack Run the script you just created. Ensure your simulator (e.g., Webots or Gazebo) is running first. ```bash python3 motion_test.py ``` ### 2. Open the UI Open your browser to the local UI URL (e.g., `http://0.0.0.0:5001`). You will see the map and the robot. ### 3. Trigger the Test In the **Inputs** panel on the UI, you will see a switch or button for `run_tests`. Toggle it to **True** and click **Send**. ### 4. Watch the Robot The robot will immediately execute the configured test pattern (e.g., driving in circles). 1. **Forward Circle** -- Max Velocity / 2 2. **Inverse Circle** -- Negative Velocity 3. **Backward Circle** The robot will automatically stop after the sequence is complete. --- ## Analyzing the Data Once the tests are finished, check the folder configured in `tests_folder` (in the recipe above: `~/.kompass/tests`). You will find CSV files named by the test type (e.g., `circle_forward.csv`). **CSV Structure:** | timestamp | x | y | yaw | cmd_vx | cmd_vy | cmd_omega | | :--- | :--- | :--- | :--- | :--- | :--- | :--- | | 16234.12 | 0.0 | 0.0 | 0.0 | 0.2 | 0.0 | 0.2 | | ... | ... | ... | ... | ... | ... | ... | You can plot these columns to compare the `cmd_vx` (Reference) vs the derivative of `x` (Response) to calculate your system's step response and latency. --- ## Configuration Options You can customize the testing behavior via `MotionServerConfig`: - {material-regular}`timer;1.2em;sd-text-primary` **`test_period`** *(float, default=10.0)* -- Duration of each individual test step in seconds. - {material-regular}`straighten;1.2em;sd-text-primary` **`run_step_test`** *(bool, default=False)* -- Runs linear step inputs (forward/backward straight lines). - {material-regular}`loop;1.2em;sd-text-primary` **`run_circle_test`** *(bool, default=True)* -- Runs combined linear and angular velocity commands. - {material-regular}`folder;1.2em;sd-text-primary` **`tests_folder`** *(str)* -- Absolute path where CSV files will be saved. ```{tip} The **MotionServer** generates commands, but usually sends them to the **DriveManager** first for safety checks. For accurate recording, the MotionServer should listen to the **output** of the DriveManager (`cmd_vel`) as its input. This ensures you record exactly what was sent to the motors, including any safety overrides. ``` --- ## Next Steps - **[Path Recording & Replay](path-recording.md)** -- Save successful paths and replay them on demand. - **[Point Navigation](point-navigation.md)** -- Learn the navigation stack fundamentals step by step. --- ```{tip} **Promote this recipe to production.** While you're shaping it, the script runs straight with `python recipe.py`. Once it's solid, drop it at `~/emos/recipes//recipe.py` and run `emos run ` -- you'll get sensor pre-flight checks, persistent logs, and a card on the dashboard so an operator can launch it from a browser. See [Running Recipes](../../getting-started/running-recipes.md) for the full development-vs-production comparison and install-mode pitfalls (especially in Container mode). ``` ``` ## File: recipes/navigation/vision-tracking-rgb.md ```markdown # Vision Tracking with RGB In this tutorial we will create a vision-based target following navigation system to follow a moving target using an RGB camera input. This recipe demonstrates a core EMOS pattern: combining an [EmbodiedAgents](https://github.com/automatika-robotics/embodied-agents) perception component with a [Kompass](https://github.com/automatika-robotics/kompass) navigation controller in a single script. --- ## Before You Start ### Get and start your camera ROS 2 node Based on the type of the camera used on your robot, you need to install and launch its respective ROS 2 node provided by the manufacturer. To run and test this example on your development machine, you can use your webcam along with the `usb_cam` package: ```shell sudo apt install ros--usb-cam ros2 run usb_cam usb_cam_node_exe ``` ### Start vision detection/tracking using an ML model To implement and run this example we will need a detection model processing the RGB camera images to provide the Detection or Tracking information. The most convenient way to obtain this is to use [EmbodiedAgents](https://github.com/automatika-robotics/embodied-agents) and [RoboML](https://github.com/automatika-robotics/roboml) to deploy and serve the model locally. EmbodiedAgents provides a Vision Component, which will allow us to easily deploy a ROS node in our system that interacts with vision models. Before starting with this tutorial you need to install both packages: - Install **EMOS**: check the instructions in the [installation guide](../../getting-started/installation.md) - Install RoboML: `pip install roboml` After installing both packages, you can start `roboml` to serve the model later either on the robot (or your development machine), or on another machine in the local network or any server in the cloud. To start a RoboML RESP server, simply run: ```shell roboml-resp ``` ```{tip} Save the IP of the machine running `roboml` as we will use it later in our model client. ``` --- ## Step 1: Vision Model Client First, we need to import the `VisionModel` class that defines the model used later in the component, and a model client to communicate with the model which can be running on the same hardware or in the cloud. Here we will use a `RESPModelClient` from RoboML as we activated the RESP based model server in RoboML. ```python from agents.models import VisionModel from agents.clients import RoboMLRESPClient ``` Now let's configure the model we want to use for detections/tracking and the model client: ```python object_detection = VisionModel( name="object_detection", checkpoint="PekingU/rtdetr_r50vd_coco_o365", ) roboml_detection = RoboMLRESPClient(object_detection, host='127.0.0.1', logging_level="warn") # 127.0.0.1 should be replaced by the IP of the machine running roboml. ``` The model is configured with a name and a checkpoint. RoboML's `VisionModel` works with any [HuggingFace Transformers object detection model](https://huggingface.co/models?pipeline_tag=object-detection) (RT-DETR, DETR, Grounding DINO, YOLOS, etc.). In this example we use the RT-DETR checkpoint pretrained on COCO + Objects365, which covers over 80 [classes](https://github.com/amikelive/coco-labels/blob/master/coco-labels-2014_2017.txt) of commonly found objects. --- ## Step 2: Vision Component We start by importing the required component along with its configuration class: ```python from agents.components import Vision from agents.config import VisionConfig ``` After setting up the model client, we need to select the input/output topics to configure the vision component: ```python from agents.ros import Topic # RGB camera input topic is set to the compressed image topic image0 = Topic(name="/image_raw/compressed", msg_type="CompressedImage") # Select the output topics: detections and trackings detections_topic = Topic(name="detections", msg_type="Detections") trackings_topic = Topic(name="trackings", msg_type="Trackings") # Select the vision component configuration detection_config = VisionConfig( threshold=0.5, enable_visualization=True ) # Create the component vision = Vision( inputs=[image0], outputs=[detections_topic, trackings_topic], trigger=image0, config=detection_config, model_client=roboml_detection, component_name="detection_component", ) ``` The component inputs/outputs are defined to get the images from the camera topic and provide both detections and trackings. The `trigger` of the component is set to the image input topic so the component works in an Event-Based runtype and provides a new detection/tracking on each new image. In the component configuration, the parameter `enable_visualization` is set to `True` to get a visualization of the output on an additional pop-up window for debugging purposes. The `threshold` parameter (confidence threshold for object detection) is set to `0.5`. --- ## Step 3: Robot Configuration We can select the robot motion model, control limits and other geometry parameters using the `RobotConfig` class: ```python from kompass.robot import ( AngularCtrlLimits, LinearCtrlLimits, RobotGeometry, RobotType, RobotConfig, ) import numpy as np # Setup your robot configuration my_robot = RobotConfig( model_type=RobotType.DIFFERENTIAL_DRIVE, geometry_type=RobotGeometry.Type.CYLINDER, geometry_params=np.array([0.1, 0.3]), ctrl_vx_limits=LinearCtrlLimits(max_vel=0.4, max_acc=1.5, max_decel=2.5), ctrl_omega_limits=AngularCtrlLimits( max_vel=0.2, max_acc=2.0, max_decel=2.0, max_steer=np.pi / 3 ), ) ``` ```{seealso} See more details about the robot configuration in the [Point Navigation](point-navigation.md#step-1-robot-configuration) recipe and in the [Robot Configuration](../../navigation/robot-config.md) reference. ``` --- ## Step 4: Navigation Controller To implement the target following system we will use the `Controller` component to generate the tracking commands and the `DriveManager` to handle the safe communication with the robot driver. The controller selects between path-following (e.g. `DWA`) and vision-following modes via its **algorithm**. We pick `ControllersID.VISION_IMG` for an RGB-only follower (the RGB and RGBD followers are sibling controllers with their own configs -- see [Vision Tracking with Depth](vision-tracking-depth.md) for the RGBD path). ```python from kompass.components import Controller, ControllerConfig, DriveManager from kompass.control import ControllersID, VisionRGBFollowerConfig # Set the controller component configuration config = ControllerConfig(loop_rate=10.0, ctrl_publish_type="Sequence", control_time_step=0.3) # Init the controller and pick the RGB-only vision follower controller = Controller(component_name="my_controller", config=config) controller.algorithm = ControllersID.VISION_IMG # Set the vision tracking input to either the detections or trackings topic controller.inputs(vision_tracking=detections_topic) # Set the vision follower configuration vision_follower_config = VisionRGBFollowerConfig( control_horizon=3, enable_search=False, target_search_pause=6, tolerance=0.2 ) controller.algorithms_config = vision_follower_config # Init the drive manager with the default parameters driver = DriveManager(component_name="my_driver") ``` Here we selected a loop rate for the controller of `10Hz` and a control step for generating the commands of `0.3s`, and we selected to send the commands sequentially as they get computed. The vision follower is configured with a `control_horizon` equal to three future control time steps and a `target_search_pause` equal to 6 control time steps. We also chose to disable the search, meaning that the tracking action would end when the robot loses the target. ```{tip} `target_search_pause` is implemented so the robot would pause and wait while tracking to avoid losing the target due to quick movement and slow model response. It should be adjusted based on the inference time of the model. ``` --- ## Step 5: Launch All that is left is to add all three components to the launcher and bring up the system. ```python from kompass.ros import Launcher launcher = Launcher() # setup agents as a package in the launcher and add the vision component launcher.add_pkg( components=[vision], package_name="automatika_embodied_agents", multiprocessing=True, ros_log_level="warn", ) # setup the navigation components in the launcher launcher.add_pkg( components=[controller, driver], package_name="kompass", multiprocessing=True, ) # Set the robot config for all components launcher.robot = my_robot # Start all the components launcher.bringup() ``` --- ## Full Recipe Code ```{code-block} python :caption: vision_rgb_follower.py :linenos: import numpy as np from agents.components import Vision from agents.models import VisionModel from agents.clients import RoboMLRESPClient from agents.config import VisionConfig from agents.ros import Topic from kompass.components import Controller, ControllerConfig, DriveManager from kompass.robot import ( AngularCtrlLimits, LinearCtrlLimits, RobotGeometry, RobotType, RobotConfig, ) from kompass.control import ControllersID, VisionRGBFollowerConfig from kompass.ros import Launcher # RGB camera input topic is set to the compressed image topic image0 = Topic(name="/image_raw/compressed", msg_type="CompressedImage") # Select the output topics: detections and trackings detections_topic = Topic(name="detections", msg_type="Detections") trackings_topic = Topic(name="trackings", msg_type="Trackings") object_detection = VisionModel( name="object_detection", checkpoint="PekingU/rtdetr_r50vd_coco_o365", ) roboml_detection = RoboMLRESPClient(object_detection, host='127.0.0.1', logging_level="warn") # Select the vision component configuration detection_config = VisionConfig(threshold=0.5, enable_visualization=True) # Create the component vision = Vision( inputs=[image0], outputs=[detections_topic, trackings_topic], trigger=image0, config=detection_config, model_client=roboml_detection, component_name="detection_component", ) # Setup your robot configuration my_robot = RobotConfig( model_type=RobotType.DIFFERENTIAL_DRIVE, geometry_type=RobotGeometry.Type.CYLINDER, geometry_params=np.array([0.1, 0.3]), ctrl_vx_limits=LinearCtrlLimits(max_vel=0.4, max_acc=1.5, max_decel=2.5), ctrl_omega_limits=AngularCtrlLimits( max_vel=0.2, max_acc=2.0, max_decel=2.0, max_steer=np.pi / 3 ), ) # Set the controller component configuration config = ControllerConfig( loop_rate=10.0, ctrl_publish_type="Sequence", control_time_step=0.3 ) # Init the controller and pick the RGB-only vision follower controller = Controller(component_name="my_controller", config=config) controller.algorithm = ControllersID.VISION_IMG controller.inputs(vision_tracking=detections_topic) # Set the vision follower configuration vision_follower_config = VisionRGBFollowerConfig( control_horizon=3, enable_search=False, target_search_pause=6, tolerance=0.2 ) controller.algorithms_config = vision_follower_config # Init the drive manager with the default parameters driver = DriveManager(component_name="my_driver") launcher = Launcher() launcher.add_pkg( components=[vision], package_name="automatika_embodied_agents", multiprocessing=True, ros_log_level="warn", ) launcher.add_pkg( components=[controller, driver], package_name="kompass", multiprocessing=True, ) launcher.robot = my_robot launcher.bringup() ``` --- ## Trigger the Following Action After running your complete system you can send a goal to the controller's action server `/my_controller/track_vision_target` of type `kompass_interfaces.action.TrackVisionTarget` to start tracking a selected label (`person` for example): ```shell ros2 action send_goal /my_controller/track_vision_target kompass_interfaces/action/TrackVisionTarget "{label: 'person'}" ``` You can also re-run the previous script and activate the target search by adding the following config or sending the config along with the action send_goal: ```python vision_follower_config = VisionRGBFollowerConfig( control_horizon=3, enable_search=True, target_search_pause=6, tolerance=0.2 ) ``` ```shell ros2 action send_goal /my_controller/track_vision_target kompass_interfaces/action/TrackVisionTarget "{label: 'person', search_radius: 1.0, search_timeout: 30}" ``` --- ## Next Steps - **[Vision Tracking with Depth](vision-tracking-depth.md)** -- Extend this approach with RGBD for more robust and accurate following. - **[Runtime Model Fallback](../events-and-resilience/fallback-recipes.md)** -- Make your recipe robust by switching models on failure. --- ```{tip} **Promote this recipe to production.** While you're shaping it, the script runs straight with `python recipe.py`. Once it's solid, drop it at `~/emos/recipes//recipe.py` and run `emos run ` -- you'll get sensor pre-flight checks, persistent logs, and a card on the dashboard so an operator can launch it from a browser. See [Running Recipes](../../getting-started/running-recipes.md) for the full development-vs-production comparison and install-mode pitfalls (especially in Container mode). ``` ``` ## File: recipes/navigation/vision-tracking-depth.md ```markdown # Vision Tracking with Depth This tutorial guides you through creating a vision tracking system using a depth camera. We leverage RGBD with the `VisionRGBDFollower` in [Kompass](https://github.com/automatika-robotics/kompass) to detect and follow objects more robustly. With depth information available, this creates a more precise understanding of the environment and leads to more accurate and robust object following compared to [using RGB images alone](vision-tracking-rgb.md). --- ## Before You Start ### Setup Your Depth Camera ROS 2 Node Your robot needs a depth camera to see in 3D and get the `RGBD` input. For this tutorial, we are using an **Intel RealSense** that is available on many mobile robots and well supported in ROS 2 and in simulation. To get your RealSense camera running: ```bash sudo apt install ros--realsense2-camera # Launch the camera node to start streaming both color and depth images ros2 launch realsense2_camera rs_camera.launch.py ``` ### Start vision detection using an ML model To implement and run this example we will need a detection model processing the RGBD camera images to provide the Detection information. Similarly to the [RGB tutorial](vision-tracking-rgb.md), we will use [EmbodiedAgents](https://github.com/automatika-robotics/embodied-agents). It provides a Vision Component which will allow us to easily deploy a ROS node in our system that interacts with vision models. --- ## Step 1: Vision Component and Model Client In this example, we will set `enable_local_classifier` to `True` in the vision component so the model would be deployed directly on the robot. Additionally, we will set the input topic to be the `RGBD` camera topic. This setting will allow the `Vision` component to **publish both the depth and the RGB image data along with the detections**. ```python from agents.components import Vision from agents.config import VisionConfig from agents.ros import Topic image0 = Topic(name="/camera/rgbd", msg_type="RGBD") detections_topic = Topic(name="detections", msg_type="Detections") detection_config = VisionConfig(threshold=0.5, enable_local_classifier=True) vision = Vision( inputs=[image0], outputs=[detections_topic], trigger=image0, config=detection_config, component_name="detection_component", ) ``` ```{seealso} See all available VisionModel options in the [Models](../../intelligence/models.md) reference, and all available model clients in the [Clients](../../intelligence/clients.md) reference. ``` --- ## Step 2: Robot Configuration You can set up your robot in the same way we did in the [RGB tutorial](vision-tracking-rgb.md). Here we use an Ackermann model as an example: ```python from kompass.robot import ( AngularCtrlLimits, LinearCtrlLimits, RobotGeometry, RobotType, RobotConfig, ) import numpy as np # Setup your robot configuration my_robot = RobotConfig( model_type=RobotType.ACKERMANN, geometry_type=RobotGeometry.Type.CYLINDER, geometry_params=np.array([0.1, 0.3]), ctrl_vx_limits=LinearCtrlLimits(max_vel=1.0, max_acc=3.0, max_decel=2.5), ctrl_omega_limits=AngularCtrlLimits( max_vel=4.0, max_acc=6.0, max_decel=10.0, max_steer=np.pi / 3 ), ) ``` --- ## Step 3: Controller with VisionRGBDFollower Now we set up the `Controller` component to use the `VisionRGBDFollower`. In Kompass the controller selects between path-following and vision-following modes via its **algorithm**, and `VisionRGBDFollower` and `VisionRGBFollower` are sibling controllers that live behind `ControllersID.VISION_DEPTH` and `ControllersID.VISION_IMG` respectively. Compared to the [RGB version](vision-tracking-rgb.md), the RGBD path needs two additional inputs: - The **detections topic** from the vision component - The **depth camera info topic** for depth-to-3D projection ```python from kompass.components import Controller, ControllerConfig from kompass.control import ControllersID depth_cam_info_topic = Topic(name="/camera/aligned_depth_to_color/camera_info", msg_type="CameraInfo") config = ControllerConfig(ctrl_publish_type="Parallel") controller = Controller(component_name="controller", config=config) controller.algorithm = ControllersID.VISION_DEPTH controller.inputs(vision_detections=detections_topic, depth_camera_info=depth_cam_info_topic) ``` --- ## Step 4: Helper Components To make the system more complete and robust, we add: - `DriveManager` -- to handle sending direct commands to the robot and ensure safety with its emergency stop - `LocalMapper` -- to provide the controller with more robust local perception; to do so we also set the controller's `direct_sensor` property to `False` ```python from kompass.components import DriveManager, LocalMapper controller.direct_sensor = False driver = DriveManager(component_name="driver") mapper = LocalMapper(component_name="local_mapper") ``` --- ## Full Recipe Code ```{code-block} python :caption: vision_depth_follower.py :linenos: from agents.components import Vision from agents.config import VisionConfig from agents.ros import Topic from kompass.components import Controller, ControllerConfig, DriveManager, LocalMapper from kompass.control import ControllersID from kompass.robot import ( AngularCtrlLimits, LinearCtrlLimits, RobotGeometry, RobotType, RobotConfig, ) from kompass.ros import Launcher import numpy as np image0 = Topic(name="/camera/rgbd", msg_type="RGBD") detections_topic = Topic(name="detections", msg_type="Detections") detection_config = VisionConfig(threshold=0.5, enable_local_classifier=True) vision = Vision( inputs=[image0], outputs=[detections_topic], trigger=image0, config=detection_config, component_name="detection_component", ) # Setup your robot configuration my_robot = RobotConfig( model_type=RobotType.ACKERMANN, geometry_type=RobotGeometry.Type.CYLINDER, geometry_params=np.array([0.1, 0.3]), ctrl_vx_limits=LinearCtrlLimits(max_vel=1.0, max_acc=3.0, max_decel=2.5), ctrl_omega_limits=AngularCtrlLimits( max_vel=4.0, max_acc=6.0, max_decel=10.0, max_steer=np.pi / 3 ), ) depth_cam_info_topic = Topic(name="/camera/aligned_depth_to_color/camera_info", msg_type="CameraInfo") # Setup the controller config = ControllerConfig(ctrl_publish_type="Parallel") controller = Controller(component_name="controller", config=config) controller.algorithm = ControllersID.VISION_DEPTH controller.inputs(vision_detections=detections_topic, depth_camera_info=depth_cam_info_topic) controller.direct_sensor = False # Add additional helper components driver = DriveManager(component_name="driver") mapper = LocalMapper(component_name="local_mapper") # Bring it up with the launcher launcher = Launcher() launcher.add_pkg( components=[vision], package_name="automatika_embodied_agents", multiprocessing=True, ros_log_level="warn", ) launcher.add_pkg( components=[controller, mapper, driver], package_name="kompass", multiprocessing=True, ) # Set the robot config for all components launcher.robot = my_robot launcher.bringup() ``` ```{tip} You can take your design to the next step and make your system more robust by adding some [events](../events-and-resilience/event-driven-cognition.md) or defining some [fallbacks](../events-and-resilience/fallback-recipes.md). ``` --- ```{tip} **Promote this recipe to production.** While you're shaping it, the script runs straight with `python recipe.py`. Once it's solid, drop it at `~/emos/recipes//recipe.py` and run `emos run ` -- you'll get sensor pre-flight checks, persistent logs, and a card on the dashboard so an operator can launch it from a browser. See [Running Recipes](../../getting-started/running-recipes.md) for the full development-vs-production comparison and install-mode pitfalls (especially in Container mode). ``` ``` ## File: recipes/events-and-resilience/multiprocessing.md ```markdown # Multiprocessing & Fault Tolerance In the previous recipes we saw how to compose a complex graph of components into an intelligent embodied agent. In this recipe we look at the features EMOS provides to make the same graph **robust and production-ready** -- running every component in its own process for crash isolation, and adding component-level and process-level recovery so a transient failure doesn't bring the whole system down. ```{admonition} Prerequisites :class: important This recipe builds on [Complete Agent](../foundation/complete-agent.md). Memory needs the [eMEM](https://github.com/automatika-robotics/emem) package, which `emos install` does not add — the install command depends on your mode (see [Memory installation](../../intelligence/memory.md)). ``` ## Run Components in Separate Processes By default the launcher runs each component in its own thread. ROS, however, was designed so that each functional unit -- a component in EMOS, mapped to a node in ROS -- runs in a **separate process**, such that failure of one process does not crash the rest of the system. We enable multiprocessing by passing `multiprocessing=True` and the ROS package name to `add_pkg`: ```python launcher.add_pkg( components=all_components, package_name="automatika_embodied_agents", multiprocessing=True, ) ``` ## Component-Level Fallbacks When a component fails -- a model server drops, a sensor goes dead, an algorithm can't find a solution -- EMOS lets us register **fallback strategies** that the component executes automatically. The simplest one is *restart*: re-run the lifecycle so the component checks its inputs and connections again before returning to the active state. We attach this to every component in the recipe with a small loop: ```python from agents.ros import Action for component in all_components: component.on_fail( action=Action(component.restart), max_retries=2, ) component.fallback_rate = 1 / 10 # 0.1 Hz -- check for failures every 10s ``` `on_fail` registers the action to take when the component reports an unhealthy state; `fallback_rate` controls how often EMOS retries while the component stays unhealthy. ```{seealso} EMOS supports much richer fallback behaviour -- escalation ladders, custom handlers, the four-level health-status hierarchy. See [Status & Fallbacks](../../concepts/status-and-fallbacks.md) for the full picture. ``` ## Process-Level Crash Recovery Component-level fallbacks handle the case where a component is *running* but unhealthy. They cannot help if the entire process **crashes** -- a segfault, an OOM kill, an unhandled native exception. For that, EMOS provides `Launcher.on_process_fail`, which respawns any component process that exits with a non-zero status outside of a clean shutdown: ```python launcher.on_process_fail(max_retries=3) ``` The two layers compose: the in-process fallback tries to restart the component up to 2 times; if those fail and the process actually exits, the launcher respawns the process up to 3 times. See [Process-Level Recovery](../../concepts/status-and-fallbacks.md#process-level-recovery) for the full discussion. ## The Complete Recipe Putting it all together: ```python import re from typing import Optional import numpy as np from agents.clients import ( ChromaClient, OllamaClient, RoboMLHTTPClient, RoboMLRESPClient, ) from agents.components import ( LLM, VLM, Memory, SemanticRouter, SpeechToText, TextToSpeech, Vision, ) from agents.config import ( LLMConfig, MemoryConfig, SemanticRouterConfig, TextToSpeechConfig, VisionConfig, ) from agents.models import OllamaModel, TransformersTTS, VisionModel, Whisper from agents.ros import Action, FixedInput, Launcher, MemLayer, Route, Topic from agents.vectordbs import ChromaDB ### Models and shared clients ### whisper_client = RoboMLHTTPClient(Whisper(name="whisper")) tts_client = RoboMLHTTPClient(TransformersTTS(name="tts")) detection_client = RoboMLRESPClient( VisionModel(name="rtdetr", checkpoint="PekingU/rtdetr_r50vd_coco_o365") ) qwen_vl_client = OllamaClient( OllamaModel(name="qwen_vl", checkpoint="qwen2.5vl:latest") ) qwen_client = OllamaClient(OllamaModel(name="qwen", checkpoint="qwen3:0.6b")) embedding_client = OllamaClient( OllamaModel(name="embeddings", checkpoint="nomic-embed-text-v2-moe:latest") ) # ChromaDB is still used by SemanticRouter for route embeddings. chroma_client = ChromaClient(db=ChromaDB(), port=8080) ### Speech I/O ### audio_in = Topic(name="audio0", msg_type="Audio") query_topic = Topic(name="question", msg_type="String") query_answer = Topic(name="answer", msg_type="String") speech_to_text = SpeechToText( inputs=[audio_in], outputs=[query_topic], model_client=whisper_client, trigger=audio_in, component_name="speech_to_text", ) text_to_speech = TextToSpeech( inputs=[query_answer], trigger=query_answer, model_client=tts_client, config=TextToSpeechConfig(play_on_device=True), component_name="text_to_speech", ) ### Vision (object detection) ### image0 = Topic(name="image_raw", msg_type="Image") detections_topic = Topic(name="detections", msg_type="Detections") vision = Vision( inputs=[image0], outputs=[detections_topic], trigger=image0, config=VisionConfig(threshold=0.5), model_client=detection_client, component_name="object_detection", ) ### VQA VLM ### mllm_query = Topic(name="mllm_query", msg_type="String") mllm = VLM( inputs=[mllm_query, image0, detections_topic], outputs=[query_answer], model_client=qwen_vl_client, trigger=mllm_query, component_name="visual_q_and_a", ) mllm.set_component_prompt( template=( "Imagine you are a robot. This image has the following items: " "{{ detections }}. Answer the following about this image: {{ text0 }}" ) ) ### Introspection VLM (room classification feeding the memory) ### introspection_query = FixedInput( name="introspection_query", msg_type="String", fixed=( "What kind of a room is this? Is it an office, a bedroom or a " "kitchen? Give a one word answer, out of the given choices" ), ) introspection_answer = Topic(name="introspection_answer", msg_type="String") introspector = VLM( inputs=[introspection_query, image0], outputs=[introspection_answer], model_client=qwen_vl_client, trigger=10.0, component_name="introspector", ) def introspection_validation(output: str) -> Optional[str]: for option in ["office", "bedroom", "kitchen"]: if option in output.lower(): return option introspector.add_publisher_preprocessor(introspection_answer, introspection_validation) ### Memory (graph-backed spatio-temporal memory) ### position = Topic(name="odom", msg_type="Odometry") memory = Memory( layers=[ MemLayer(subscribes_to=detections_topic), MemLayer(subscribes_to=introspection_answer), ], position=position, model_client=qwen_client, embedding_client=embedding_client, config=MemoryConfig(db_path="/tmp/complete_agent_multiprocessing.db"), trigger=15.0, component_name="memory", ) ### Generic LLM (general Q&A) ### llm_query = Topic(name="llm_query", msg_type="String") llm = LLM( inputs=[llm_query], outputs=[query_answer], model_client=qwen_client, trigger=[llm_query], component_name="general_q_and_a", ) ### Go-to-X using LLM tool calling on Memory.locate ### goto_query = Topic(name="goto_query", msg_type="String") goal_point = Topic(name="goal_point", msg_type="PoseStamped") goto = LLM( inputs=[goto_query], outputs=[goal_point], model_client=qwen_client, trigger=goto_query, config=LLMConfig(), component_name="go_to_x", ) goto.set_component_prompt( template=( "The user asks you to go to a place. Use the available tools to " "look up the place's location in memory. Pass the place name to " "the locate tool as the ``concept`` argument. User asked: {{goto_query}}" ) ) memory.register_tools_on(goto, tools=["locate"], send_tool_response_to_model=False) _LOCATION_RE = re.compile(r"Location:\s*\(([^)]+)\)") def locate_text_to_goal_point(output: str) -> Optional[np.ndarray]: """Pull the centroid coordinates out of Memory.locate's text output.""" match = _LOCATION_RE.search(output) if not match: return try: coords = np.fromstring(match.group(1), sep=",", dtype=np.float64) except ValueError: return if coords.shape[0] == 2: coords = np.append(coords, 0.0) if coords.shape[0] != 3: return return coords goto.add_publisher_preprocessor(goal_point, locate_text_to_goal_point) ### Semantic router (uses ChromaDB for the route embeddings) ### goto_route = Route( routes_to=goto_query, samples=[ "Go to the door", "Go to the kitchen", "Get me a glass", "Fetch a ball", "Go to hallway", ], ) llm_route = Route( routes_to=llm_query, samples=[ "What is the capital of France?", "Is there life on Mars?", "How many tablespoons in a cup?", "How are you today?", "Whats up?", ], ) mllm_route = Route( routes_to=mllm_query, samples=[ "Are we indoors or outdoors", "What do you see?", "Whats in front of you?", "Where are we", "Do you see any people?", "How many things are in front of you?", "Is this room occupied?", ], ) router = SemanticRouter( inputs=[query_topic], routes=[llm_route, goto_route, mllm_route], default_route=llm_route, config=SemanticRouterConfig(router_name="go-to-router", distance_func="l2"), db_client=chroma_client, component_name="router", ) ### Per-component fallback strategies ### all_components = [ mllm, llm, goto, introspector, memory, router, speech_to_text, text_to_speech, vision, ] for component in all_components: component.on_fail( action=Action(component.restart), max_retries=2, ) component.fallback_rate = 1 / 10 # 0.1 Hz -- check for failures every 10s ### Launch (multi-process) ### launcher = Launcher() launcher.enable_ui( inputs=[query_topic, audio_in], outputs=[detections_topic, query_answer, goal_point] ) launcher.add_pkg( components=all_components, package_name="automatika_embodied_agents", multiprocessing=True, ) # Process-level crash recovery: respawn any multi-process component whose # process exits unexpectedly, up to ``max_retries`` times. launcher.on_process_fail(max_retries=3) launcher.bringup() ``` With these modifications, the same complex agent graph from [Complete Agent](../foundation/complete-agent.md) runs **as nine isolated processes**, each with its own restart policy and its own process-level safety net. A model-server outage triggers a component restart; an unrecoverable process exit triggers a process respawn. The graph as a whole keeps running. --- ```{tip} **Promote this recipe to production.** While you're shaping it, the script runs straight with `python recipe.py`. Once it's solid, drop it at `~/emos/recipes//recipe.py` and run `emos run ` -- you'll get sensor pre-flight checks, persistent logs, and a card on the dashboard so an operator can launch it from a browser. See [Running Recipes](../../getting-started/running-recipes.md) for the full development-vs-production comparison and install-mode pitfalls (especially in Container mode). ``` ``` ## File: recipes/events-and-resilience/fallback-recipes.md ```markdown # Self-Healing with Fallbacks In the real world, connections drop, APIs time out, solvers fail to converge, and serial cables vibrate loose. A "Production Ready" agent cannot simply freeze when something goes wrong. EMOS provides a unified fallback API that works identically across the **intelligence layer** (model clients) and the **navigation layer** (algorithms and hardware). In this recipe, we demonstrate both. --- ## Intelligence Layer: Model Fallback We build an agent that uses a high-intelligence model (hosted remotely) as its primary _brain_, but automatically switches to a smaller, local model if the primary one fails. ### The Strategy: Plan A and Plan B 1. **Plan A (Primary):** Use a powerful model hosted via RoboML (or a cloud provider) for high-quality reasoning. 2. **Plan B (Backup):** Keep a smaller model available locally as a safety net. 3. **The Trigger:** If the Primary model fails to respond (latency, disconnection, or server error), automatically swap to the Backup. EMOS offers two approaches for implementing this strategy — a zero-config local fallback and a manual model client swap. ### Approach 1: Built-in Local Fallback (Zero-Config) The simplest way to add resilience is `fallback_to_local()`. This one-liner tells the component to switch to a built-in local model on failure — no Ollama server, no additional clients, no extra configuration. Built-in local models are available for **LLM**, **VLM**, **SpeechToText**, and **TextToSpeech** components. Default models are lightweight and designed for on-device inference: - **LLM** — Qwen3 0.6B (via `llama-cpp-python`) - **VLM** — Moondream2 (via `llama-cpp-python`) - **STT** — Whisper tiny (via `sherpa-onnx`) - **TTS** — Kokoro (via `sherpa-onnx`) ```{note} Install the required dependency for your component: `pip install llama-cpp-python` for LLM/VLM, or `pip install sherpa-onnx` for STT/TTS. These are pre-installed in EMOS Docker containers. ``` ```python from agents.components import LLM from agents.models import TransformersLLM from agents.clients import RoboMLHTTPClient from agents.ros import Launcher, Topic, Action # Primary: A powerful model hosted remotely (e.g., via RoboML) # NOTE: This is illustrative for the sake of executing on the local machine. # For a more realistic scenario, replace this with a GenericHTTPClient # pointing to a cloud model. primary_model = TransformersLLM( name="qwen_heavy", checkpoint="Qwen/Qwen2.5-1.5B-Instruct" ) primary_client = RoboMLHTTPClient(model=primary_model) # Define Topics user_query = Topic(name="user_query", msg_type="String") llm_response = Topic(name="llm_response", msg_type="String") # Configure the LLM Component with the primary client llm_component = LLM( inputs=[user_query], outputs=[llm_response], model_client=primary_client, trigger=user_query, component_name="brain", ) # One-liner fallback: switch to built-in local model on failure. # No additional model clients, no Ollama server — just a single Action. switch_to_local = Action(method=llm_component.fallback_to_local) llm_component.on_component_fail(action=switch_to_local, max_retries=3) llm_component.on_algorithm_fail(action=switch_to_local, max_retries=3) # Launch launcher = Launcher() launcher.add_pkg( components=[llm_component], multiprocessing=True, package_name="automatika_embodied_agents", ) launcher.bringup() ``` To test this, shut down your RoboML server (or disconnect the internet) while the agent is running, and watch it seamlessly switch to the local model. ```{seealso} For the full list of built-in local models and configuration options, see [Built-in Local Models](../../intelligence/models.md#built-in-local-models). ``` ### Approach 2: Model Client Hot-Swap When you want explicit control over which backup model and client to use, the `change_model_client` + `additional_model_clients` pattern lets you define exactly what runs as the fallback. #### 1. Defining the Models First, we need to define our two distinct model clients. ```python from agents.components import LLM from agents.models import OllamaModel, TransformersLLM from agents.clients import OllamaClient, RoboMLHTTPClient from agents.config import LLMConfig from agents.ros import Launcher, Topic, Action # --- Plan A: The Powerhouse --- # A powerful model hosted remotely (e.g., via RoboML). # NOTE: This is illustrative for executing on a local machine. # For a production scenario, you might use a GenericHTTPClient pointing to # GPT-5, Gemini, HuggingFace Inference etc. primary_model = TransformersLLM( name="qwen_heavy", checkpoint="Qwen/Qwen2.5-1.5B-Instruct" ) primary_client = RoboMLHTTPClient(model=primary_model) # --- Plan B: The Safety Net --- # A smaller model running locally (via Ollama) that works offline. backup_model = OllamaModel(name="llama_local", checkpoint="llama3.2:3b") backup_client = OllamaClient(model=backup_model) ``` #### 2. Configuring the Component Next, we set up the standard `LLM` component. We initialize it using the `primary_client`. However, the magic happens in the `additional_model_clients` attribute. This dictionary allows the component to hold references to other valid clients that are waiting in the wings. ```python # Define Topics user_query = Topic(name="user_query", msg_type="String") llm_response = Topic(name="llm_response", msg_type="String") # Configure the LLM Component with the PRIMARY client initially llm_component = LLM( inputs=[user_query], outputs=[llm_response], model_client=primary_client, component_name="brain", config=LLMConfig(stream=True), ) # Register the Backup Client # We store the backup client in the component's internal registry. # We will use the key 'local_backup_client' to refer to this later. llm_component.additional_model_clients = {"local_backup_client": backup_client} ``` #### 3. Creating the Fallback Action Now we need an **Action**. In EMOS, components have built-in methods to reconfigure themselves. The `LLM` component (like all other components that take a model client) has a method called `change_model_client`. We wrap this method in an `Action` so it can be triggered by an event. ```{note} All components implement some default actions as well as component specific actions. In this case we are implementing a component specific action. ``` ```{seealso} To see a list of default actions available to all components, checkout the [Actions](../../concepts/events-and-actions.md) documentation. ``` ```python # Define the Fallback Action # This action calls the component's internal method `change_model_client`. # We pass the key ('local_backup_client') defined in the previous step. switch_to_backup = Action( method=llm_component.change_model_client, args=("local_backup_client",) ) ``` #### 4. Wiring Failure to Action Finally, we tell the component _when_ to execute this action. We don't need to write complex `try/except` blocks in our business logic. Instead, we attach the action to the component's lifecycle hooks: - **`on_component_fail`**: Triggered if the component crashes or fails to initialize (e.g., the remote server is down when the robot starts). - **`on_algorithm_fail`**: Triggered if the component is running, but the inference fails (e.g., the WiFi drops mid-conversation). ```python # Bind Failures to the Action # If the component fails (startup) or the algorithm crashes (runtime), # it will attempt to switch clients. llm_component.on_component_fail(action=switch_to_backup, max_retries=3) llm_component.on_algorithm_fail(action=switch_to_backup, max_retries=3) ``` ```{note} **Why `max_retries`?** Sometimes a fallback can temporarily fail as well. The system will attempt to restart the component or algorithm up to 3 times while applying the action (switching the client) to resolve the error. This is an _optional_ parameter. ``` #### The Complete Client Hot-Swap Recipe Here is the full code. To test this, try shutting down your RoboML server (or disconnecting the internet) while the agent is running, and watch it seamlessly switch to the local Ollama model. ```python from agents.components import LLM from agents.models import OllamaModel, TransformersLLM from agents.clients import OllamaClient, RoboMLHTTPClient from agents.config import LLMConfig from agents.ros import Launcher, Topic, Action # 1. Define the Models and Clients # Primary: A powerful model hosted remotely primary_model = TransformersLLM( name="qwen_heavy", checkpoint="Qwen/Qwen2.5-1.5B-Instruct" ) primary_client = RoboMLHTTPClient(model=primary_model) # Backup: A smaller model running locally backup_model = OllamaModel(name="llama_local", checkpoint="llama3.2:3b") backup_client = OllamaClient(model=backup_model) # 2. Define Topics user_query = Topic(name="user_query", msg_type="String") llm_response = Topic(name="llm_response", msg_type="String") # 3. Configure the LLM Component llm_component = LLM( inputs=[user_query], outputs=[llm_response], model_client=primary_client, component_name="brain", config=LLMConfig(stream=True), ) # 4. Register the Backup Client llm_component.additional_model_clients = {"local_backup_client": backup_client} # 5. Define the Fallback Action switch_to_backup = Action( method=llm_component.change_model_client, args=("local_backup_client",) ) # 6. Bind Failures to the Action llm_component.on_component_fail(action=switch_to_backup, max_retries=3) llm_component.on_algorithm_fail(action=switch_to_backup, max_retries=3) # 7. Launch launcher = Launcher() launcher.add_pkg( components=[llm_component], multiprocessing=True, package_name="automatika_embodied_agents", ) launcher.bringup() ``` --- ## Navigation Layer: Algorithm & System Fallback Navigation components face a different class of failures: optimization solvers that fail to converge, serial cables that vibrate loose, and robots that get boxed into corners. The same `on_*_fail` API handles all of these. ### Algorithm Failure: Switch Controllers If the primary high-performance algorithm (e.g., `DWA`) fails, we can switch to a simpler "safety" algorithm (like `PurePursuit`). ```python from kompass.components import Controller, DriveManager from kompass.control import ControllersID from kompass.ros import Action # Select the primary control algorithm controller = Controller(component_name="controller") controller.algorithm = ControllersID.DWA # Define the fallback: switch to PurePursuit switch_algorithm_action = Action( method=controller.set_algorithm, args=(ControllersID.PURE_PURSUIT,) ) # Fallback sequence: restart first, then switch algorithm if it fails again controller.on_algorithm_fail( action=[Action(controller.restart), switch_algorithm_action], max_retries=1 ) ``` ### System Failure: Restart Hardware Connection The `DriveManager` talks directly to low-level hardware (micro-controller/motor drivers). Transient failures -- loose USB cables, electromagnetic interference, watchdog trips -- are common and often resolved by a simple restart. ```python driver = DriveManager(component_name="drive_manager") # Restart on system failure (unlimited retries for transient hardware glitches) driver.on_system_fail(Action(driver.restart)) ``` ### Global Catch-All For any component that doesn't have specific fallback logic, the Launcher provides a blanket policy. ```python launcher.on_fail(action_name="restart") launcher.fallback_rate = 1 / 10 # 0.1 Hz (one retry every 10 seconds) ``` ```{seealso} The [Multiprocessing & Fault Tolerance](multiprocessing.md) recipe shows how to combine `launcher.on_fail()` with process isolation for a complete production setup. ``` ## The Same API, Both Layers The key insight is that **the same three hooks** work everywhere in EMOS: | Hook | Triggers When | Intelligence Example | Navigation Example | |---|---|---|---| | `on_component_fail` | Component crashes or fails to initialize | Remote model server is down | Serial port unavailable | | `on_algorithm_fail` | Inference or computation fails at runtime | WiFi drops mid-conversation | DWA solver can't converge | | `on_system_fail` | External dependency is lost | API key revoked | Motor controller resets | Each hook accepts an `Action` (or list of actions) and an optional `max_retries` parameter. This consistency means you can apply the same resilience patterns regardless of which layer you're working in. --- ```{tip} **Promote this recipe to production.** While you're shaping it, the script runs straight with `python recipe.py`. Once it's solid, drop it at `~/emos/recipes//recipe.py` and run `emos run ` -- you'll get sensor pre-flight checks, persistent logs, and a card on the dashboard so an operator can launch it from a browser. See [Running Recipes](../../getting-started/running-recipes.md) for the full development-vs-production comparison and install-mode pitfalls (especially in Container mode). ``` ``` ## File: recipes/events-and-resilience/event-driven-cognition.md ```markdown # Event-Driven Cognition Robots process a massive amount of sensory data. Running a large Vision Language Model (VLM) on every single video frame to ask "What is happening?", while possible with smaller models, is in fact computationally expensive and redundant. In this tutorial, we will use the **Event-Driven** nature of EMOS to create a smart "Reflex-Cognition" loop. We will use a lightweight detector to monitor the scene efficiently (the Reflex), and only when a specific object (a person) is found, we will trigger a larger VLM to describe them (the Cognition). One can imagine that this description can be used for logging robot's observations or parsed for triggering further actions downstream. ## The Strategy: Reflex and Cognition 1. **Reflex (Vision Component):** A fast, lightweight object detector runs on every frame. It acts as a gatekeeper. 2. **Event (The Trigger):** We define a smart event that fires only when the detector finds a "person" (and hasn't seen one recently). 3. **Cognition (VLM Component):** A more powerful VLM wakes up only when triggered by the event to describe the scene. ### 1. The Reflex: Vision Component First, we set up the `Vision` component. This component is designed to be lightweight. By enabling the local classifier, we can run a small optimized model contained within the component, directly on the edge. ```python from agents.components import Vision from agents.config import VisionConfig from agents.ros import Topic # Define Topics camera_image = Topic(name="/image_raw", msg_type="Image") detections = Topic(name="/detections", msg_type="Detections") # Output of Vision # Setup the Vision Component (The Trigger) # We use a lower threshold to ensure we catch people easily and we use a small embedded model vision_config = VisionConfig(threshold=0.6, enable_local_classifier=True) vision_detector = Vision( inputs=[camera_image], outputs=[detections], trigger=camera_image, # Runs on every frame config=vision_config, component_name="eye_detector", ) ``` The `trigger=camera_image` argument tells this component to process every single message that arrives on the `/image_raw` topic. ### 2. The Trigger: Smart Events Now, we need to bridge the gap between detection and description. We don't want the VLM to fire 30 times a second just because a person is standing in the frame. We use `Event` with `on_change` mode. This event type is perfect for state changes. It monitors a list inside a message (in this case, the `labels` list of the detections). ```python from agents.ros import Event # Define the Event # This event listens to the 'detections' topic. # It triggers ONLY if the "labels" list inside the message contains "person" # after not containing a person (within a 5 second interval). event_person_detected = Event( detections.msg.labels.contains_any(["person"]), on_change=True, # Trigger only when a change has occurred to stop repeat triggering keep_event_delay=5, # A delay in seconds ) ``` ```{note} **`keep_event_delay=5`**: This is a debouncing mechanism. It ensures that once the event triggers, it won't trigger again for at least 5 seconds, even if the person remains in the frame. This prevents our VLM from being flooded with requests and can be quite useful to prevent jittery detections, which are common especially for mobile robots. ``` ```{seealso} Events can be used to create arbitrarily complex agent graphs. Check out all the events available in the [Events](../../concepts/events-and-actions.md) documentation. ``` ### 3. The Cognition: VLM Component Finally, we set up the heavy lifter. We will use a `VLM` component powered by **Qwen-VL** running on Ollama. Crucially, this component does **not** have a topic trigger like the vision detector. Instead, it is triggered by `event_person_detected`. We also need to tell the VLM _what_ to do when it wakes up. Since there is no user typing a question, we inject a `FixedInput`, a static prompt that acts as a standing order. ```python from agents.components import VLM from agents.clients import OllamaClient from agents.models import OllamaModel from agents.ros import FixedInput description_output = Topic(name="/description", msg_type="String") # Output of VLM # Setup a model client for the component qwen_vl = OllamaModel(name="qwen_vl", checkpoint="qwen2.5vl:7b") ollama_client = OllamaClient(model=qwen_vl) # We define a fixed prompt that is injected whenever the component runs. fixed_prompt = FixedInput( name="prompt", msg_type="String", fixed="A person has been detected. Describe their appearance briefly.", ) visual_describer = VLM( inputs=[fixed_prompt, camera_image], # Takes the fixed prompt + current image outputs=[description_output], model_client=ollama_client, trigger=event_person_detected, # CRITICAL: Only runs when the event fires component_name="visual_describer", ) ``` ## Launching the Application We combine everything into a launcher. ```python from agents.ros import Launcher # Launch launcher = Launcher() launcher.add_pkg( components=[vision_detector, visual_describer], multiprocessing=True, package_name="automatika_embodied_agents", ) launcher.bringup() ``` ## See the results in the UI We can see this recipe in action if we enable the UI. We can do so by simply adding the following line in the launcher. ```python launcher.enable_ui(outputs=[camera_image, detections, description_output]) ``` ````{note} In order to run the client you will need to install [FastHTML](https://www.fastht.ml/) and [MonsterUI](https://github.com/AnswerDotAI/MonsterUI) with ```shell pip install python-fasthtml monsterui ```` The client displays a web UI on **http://localhost:5001** if you have run it on your machine. Or you can access it at **http://:5001** if you have run it on the robot. ### Complete Code Here is the complete recipe for the Event-Driven Cognition agent: ```{code-block} python :caption: Event-Driven Cognition :linenos: from agents.components import Vision, VLM from agents.config import VisionConfig from agents.clients import OllamaClient from agents.models import OllamaModel from agents.ros import Launcher, Topic, FixedInput, Event # Define Topics camera_image = Topic(name="/image_raw", msg_type="Image") detections = Topic(name="/detections", msg_type="Detections") # Output of Vision description_output = Topic(name="/description", msg_type="String") # Output of VLM # Setup the Vision Component (The Trigger) # We use a lower threshold to ensure we catch people easily and we use a small local model vision_config = VisionConfig(threshold=0.6, enable_local_classifier=True) vision_detector = Vision( inputs=[camera_image], outputs=[detections], trigger=camera_image, # Runs on every frame config=vision_config, component_name="eye_detector", ) # Define the Event # This event listens to the 'detections' topic. # It triggers ONLY if the "labels" list inside the message contains "person" # after not containing a person (within a 5 second interval). event_person_detected = Event( detections.msg.labels.contains_any(["person"]), on_change=True, # Trigger only when a change has occurred to stop repeat triggering keep_event_delay=5, # A delay in seconds ) # Setup the VLM Component (The Responder) # This component does NOT run continuously. It waits for the event. # Setup a model client for the component qwen_vl = OllamaModel(name="qwen_vl", checkpoint="qwen2.5vl:7b") ollama_client = OllamaClient(model=qwen_vl) # We define a fixed prompt that is injected whenever the component runs. fixed_prompt = FixedInput( name="prompt", msg_type="String", fixed="A person has been detected. Describe their appearance briefly.", ) visual_describer = VLM( inputs=[fixed_prompt, camera_image], # Takes the fixed prompt + current image outputs=[description_output], model_client=ollama_client, trigger=event_person_detected, # CRITICAL: Only runs when the event fires component_name="visual_describer", ) # Launch launcher = Launcher() launcher.enable_ui(outputs=[camera_image, detections, description_output]) launcher.add_pkg( components=[vision_detector, visual_describer], multiprocessing=True, package_name="automatika_embodied_agents", ) launcher.bringup() ``` --- ```{tip} **Promote this recipe to production.** While you're shaping it, the script runs straight with `python recipe.py`. Once it's solid, drop it at `~/emos/recipes//recipe.py` and run `emos run ` -- you'll get sensor pre-flight checks, persistent logs, and a card on the dashboard so an operator can launch it from a browser. See [Running Recipes](../../getting-started/running-recipes.md) for the full development-vs-production comparison and install-mode pitfalls (especially in Container mode). ``` ``` ## File: recipes/events-and-resilience/visualizing-system-graph.md ```markdown # Visualizing the System Graph EMOS recipes get rich quickly: a dozen components, topics flowing between them, events firing into actions, fallbacks layered on top. Reading a Python script doesn't always tell you whether the wiring matches what you intended -- a perception edge you thought was feeding the planner might be going to nothing, an event you thought was triggering an action might be sitting idle, and a process you thought was reacting to a topic might never have subscribed. The launcher's [Dynamic Web UI](../../concepts/web-ui.md) ships a **System Graph** view that renders the _whole running recipe_ as a draggable, resizable node graph -- components, topics, events, and actions all in one place. This recipe walks through using it on a recipe you already have. ```{seealso} For the broader feature list of the Web UI, see [Dynamic Web UI](../../concepts/web-ui.md). For the events/actions wiring conceptually, see [Events & Actions](../../concepts/events-and-actions.md). ``` --- ## Step 1: Pick (or build) any recipe Any recipe with more than one component will give you something to look at, though the graph is most useful for ones with non-trivial wiring -- multiple components feeding each other, events triggering actions, fallbacks attached. The [Complete Agent](../foundation/complete-agent.md) is a good starter (nine components, a router, memory, navigation hooks). If you're working on a navigation system, the [Cortex with Navigation](../planning-and-manipulation/cortex-navigation.md) recipe gives you an even meatier graph with both perception and motion components plus a Cortex hub. Make sure the launcher comes up with the Web UI enabled: ```python launcher.enable_ui( inputs=[...], outputs=[...], ) launcher.bringup() ``` --- ## Step 2: Open the System Graph Run the recipe and open the launcher Web UI in a browser (default: `http://localhost:5001`). You'll see the standard Dynamic Web UI with the components' settings panels and any inputs/outputs you've registered. The new **System Graph** tab sits alongside those panels.

System Graph view in the Web UI

What you'll see: - {material-regular}`hub;1.2em;sd-text-primary` **Component nodes** -- one per component, labelled with the component name, colour-coded by lifecycle state. - {material-regular}`compare_arrows;1.2em;sd-text-primary` **Topic edges** -- arrows in the direction of data flow. Input edges enter from the left of a component, output edges leave from the right. - {material-regular}`flash_on;1.2em;sd-text-primary` **Event nodes** -- yellow event markers placed between the component(s) whose topics the event watches and the action(s) it triggers. - {material-regular}`bolt;1.2em;sd-text-primary` **Action nodes** -- blue markers attached to the events that fire them. The whole graph is **draggable and resizable**, so you can pull a busy region out for a closer look without losing the rest. --- ## Step 3: Inspect anything Click any element to open its **detail card** -- nodes, edges, events, and actions all have one. - **Components** -- inputs/outputs, current lifecycle state, the model client (if any), declared `@component_action`s, registered fallbacks. - **Topics** -- name, message type, the publishing component, the subscribing component(s), the latest message preview if applicable. - **Events** -- the condition expression in serialised form (the same combinator logic `&` / `|` / `~` you wrote, rendered as a tree), the topic(s) it watches, the action(s) registered against it, and the current trigger count. - **Actions** -- the target method, the arguments (including any bound topic data), and the count of times it has fired. The detail card is the fastest way to confirm that an event you authored as `~event_a & event_b` is registered with that exact shape, that a topic you thought was subscribed actually has a subscriber, or that a component is in the lifecycle state you expect. ```{tip} If an event you expect to see never fires, the detail card's *trigger count = 0* is the easiest diagnostic. Combine it with the action-server logs surfaced in the main logging card to follow the full trace. ``` --- ## What to use it for - **Sanity-checking a new recipe** -- confirm every component you added is wired the way you intended before you start chasing bugs. - **Operator handover** -- the graph is a self-documenting handoff artefact. Screenshot it for a runbook. - **Failure investigation** -- when an event isn't firing, the detail card and trigger count usually point at the wiring problem before logs do. ``` ## File: recipes/events-and-resilience/internal-state-events.md ```markdown # Internal-State Events Most events fire on **topic data** — *"battery dropped below 15%"*, *"a person was detected"*, *"the controller reported failure"*. These are clean and declarative, but they share an assumption: the trigger condition can be expressed as a predicate over messages flowing through the system as a published topic. Sometimes that's not enough. The thing you want to react to is **internal state**: - A hardware monitor that is not being published as a topic. - A compound predicate over multiple disparate signals. - A hysteresis flag that's *sticky* across cycles. For these cases, EMOS supports **callable-based events**: an `Event` constructed from any Python callable that returns `bool`, polled at a configurable rate. The callable holds whatever state it needs — a closure variable, a class attribute, a global counter — and the Monitor polls it at `check_rate` Hz. When it returns `True`, registered Actions fire just like for any other event. ```{seealso} For the conceptual reference, see [Events & Actions](../../concepts/events-and-actions.md). For inspecting live events in the Web UI, see [Visualizing the System Graph](visualizing-system-graph.md). ``` --- ## The Mechanic An `Event` accepts three kinds of `event_condition`: 1. A `Topic` — fires on any new message. 2. A `Condition` expression — fires when a predicate over topic attributes becomes true. 3. **A `Callable` returning `bool`** — fires when polling the callable returns true. The third form is what we use here: ```python from ros_sugar.event import Event def is_low_power() -> bool: # must be type-annotated as bool ... event_low_power = Event(is_low_power, check_rate=1.0) # polled every 1 second ``` Two rules from the `Event` constructor: - The callable's return type annotation **must be `bool`**. The constructor inspects it and raises `TypeError` otherwise. - The callable **cannot be a `@component_action`** method bound to a managed component's lifecycle. Use a plain function or a regular instance method instead. `check_rate` is in **Hz**. If omitted, the event polls at the central Monitor's loop rate. For most real-world predicates, anywhere from 0.5–5 Hz is sensible — fast enough to catch transitions, slow enough to be cheap. --- ## Recipe: Idle Detection A robot that listens for voice commands should react when there's been no input for a while — dim its display, switch to a lighter local model, or just go quiet. *"Idleness"* isn't a value on any topic; it's a derived predicate over time and the most recent input. Perfect fit for a callable-based event. ### Step 1: A stateful predicate Hold the *last interaction* timestamp in a tiny helper class and expose a typed `is_idle` method: ```python import time class IdleTracker: """Tracks how long it has been since the last user interaction.""" def __init__(self, threshold_seconds: float = 60.0) -> None: self.threshold_seconds = threshold_seconds self._last_seen = time.time() def touch(self) -> None: """Mark 'just had user interaction' -- call this from your input pipeline.""" self._last_seen = time.time() def is_idle(self) -> bool: """True when we haven't seen interaction for ``threshold_seconds``.""" return (time.time() - self._last_seen) > self.threshold_seconds idle = IdleTracker(threshold_seconds=60.0) ``` Two methods — `touch()` to update state, `is_idle()` to read it. The `-> bool` annotation on `is_idle` is mandatory; the `Event` constructor inspects it. ### Step 2: Wire the state update Whenever the Speech-to-Text component publishes a transcribed query, we want to call `idle.touch()`. EMOS lets us hook a small pre-processor on the publishing side of any topic: ```python from typing import Optional def touch_on_speech(text: str) -> Optional[str]: idle.touch() return text # pass through unchanged speech_to_text.add_publisher_preprocessor(query_topic, touch_on_speech) ``` The pre-processor runs in-process before publication. It updates `idle._last_seen` and passes the message through untouched. Any topic update mechanism would work — a callback, a periodic component, a subscriber elsewhere. The point is that the *state* lives in `idle`, not in any topic. ### Step 3: Build the Event and an Action ```python from ros_sugar.actions import log from ros_sugar.event import Event event_idle = Event(idle.is_idle, check_rate=0.5) # poll every 2 seconds events_actions = { event_idle: [ log(msg="No interaction for 60s -- switching to low-power mode"), # ...switch to a smaller model, dim a display, etc. ], } ``` `check_rate=0.5` means the Monitor checks `idle.is_idle()` every 2 seconds. As long as `touch_on_speech` keeps firing, `is_idle()` returns False; the moment 60 seconds elapse without a query, the next poll returns True and the Action fires. ### Step 4: Hand the events_actions dict to the Launcher ```python from agents.ros import Launcher launcher = Launcher() launcher.add_pkg( components=[speech_to_text, vlm, text_to_speech], events_actions=events_actions, package_name="automatika_embodied_agents", multiprocessing=True, ) launcher.bringup() ``` The Monitor (or `Cortex`, if it's the recipe's monitor) takes ownership of the polling loop. From this point on, every 2 seconds the Monitor calls `idle.is_idle()` and fires the registered Actions on rising-edge transitions. --- ## When to reach for this pattern Use a callable-based event when: - The condition is **internal state** that has no business being a topic (counters, timers, sticky flags, last-seen timestamps). - The condition is a **compound predicate** over multiple sources that's easier to express as an arbitrary calculation in Python rather than as a chain of topic conditions. - You want **encapsulation** — keep the state inside the recipe rather than smearing it across publisher topics just to satisfy the trigger. Use a topic-based event when the condition genuinely is *"some message arrived"* or *"this attribute crossed a threshold"*. Topic events are cheaper (push, no polling) and more declarative. --- ## Visualisation Callable-based events show up on the [System Graph](../../concepts/web-ui.md) too: the source component (or the recipe-level dispatcher) is the upstream node, and the registered Actions are the downstream nodes. The detail card lists `check_rate` and the current trigger count, so you can confirm at runtime that the predicate is actually flipping when you expect it to. See [Visualizing the System Graph](visualizing-system-graph.md) for the tour. ``` ## File: recipes/events-and-resilience/external-reflexes.md ```markdown # External Reflexes In [Event-Driven Cognition](event-driven-cognition.md), we used a lightweight detector to wake a heavy VLM -- intelligence reacting to the world. In this recipe, we apply the same pattern to the **navigation layer**: the robot transitions from idle patrol to active person-following the moment a human appears in the camera feed. This is an **External Reflex** -- an event triggered by the environment (not an internal failure) that reconfigures the robot's behavior at runtime. --- ## The Strategy 1. **Reflex (Vision Component):** A lightweight detector runs on every frame, scanning for "person". 2. **Event (The Trigger):** Fires when "person" first appears in the detection labels. 3. **Response (Controller Reconfiguration):** Two actions execute in sequence: - Switch the Controller's algorithm to `VisionRGBDFollower` - Send a goal to the Controller's ActionServer to begin tracking --- ## Step 1: The Vision Detector We use the `Vision` component from the intelligence layer with a small embedded classifier -- fast enough to process every frame. ```python from agents.components import Vision from agents.config import VisionConfig from agents.ros import Topic image0 = Topic(name="/camera/rgbd", msg_type="RGBD") detections_topic = Topic(name="detections", msg_type="Detections") detection_config = VisionConfig(threshold=0.5, enable_local_classifier=True) vision = Vision( inputs=[image0], outputs=[detections_topic], trigger=image0, config=detection_config, component_name="detection_component", ) ``` ## Step 2: Define the Event We use `on_change=True` so the event fires only when "person" *first* appears in the detection labels -- not continuously while a person remains in frame. ```python from kompass.ros import Event event_person_detected = Event( event_condition=detections_topic.msg.labels.contains("person"), on_change=True ) ``` ## Step 3: Define the Actions When the event fires, two actions execute **in sequence**: 1. **Switch algorithm** -- reconfigure the Controller from its current mode to `VisionRGBDFollower` 2. **Trigger the ActionServer** -- send a goal specifying "person" as the tracking target ```python from kompass.actions import update_parameter, send_component_action_server_goal from kompass_interfaces.action import TrackVisionTarget # Action 1: Switch the controller algorithm switch_algorithm_action = update_parameter( component=controller, param_name="algorithm", new_value="VisionRGBDFollower" ) # Action 2: Send a tracking goal to the controller's action server action_request_msg = TrackVisionTarget.Goal() action_request_msg.label = "person" action_start_person_following = send_component_action_server_goal( component=controller, request_msg=action_request_msg, ) ``` ```{tip} Linking an Event to a **list** of Actions executes them in sequence. This lets you chain reconfiguration steps -- switch algorithm first, then send the goal. ``` ## Step 4: Wire and Launch ```python events_action = { event_person_detected: [switch_algorithm_action, action_start_person_following] } ``` --- ## Complete Recipe ```{code-block} python :caption: external_reflexes.py :linenos: import numpy as np from agents.components import Vision from agents.config import VisionConfig from agents.ros import Topic from kompass.components import Controller, ControllerConfig, DriveManager, LocalMapper from kompass.robot import ( AngularCtrlLimits, LinearCtrlLimits, RobotGeometry, RobotType, RobotConfig, ) from kompass.ros import Launcher, Event from kompass.actions import update_parameter, send_component_action_server_goal from kompass_interfaces.action import TrackVisionTarget # --- Vision Detector --- image0 = Topic(name="/camera/rgbd", msg_type="RGBD") detections_topic = Topic(name="detections", msg_type="Detections") detection_config = VisionConfig(threshold=0.5, enable_local_classifier=True) vision = Vision( inputs=[image0], outputs=[detections_topic], trigger=image0, config=detection_config, component_name="detection_component", ) # --- Robot Configuration --- my_robot = RobotConfig( model_type=RobotType.ACKERMANN, geometry_type=RobotGeometry.Type.CYLINDER, geometry_params=np.array([0.1, 0.3]), ctrl_vx_limits=LinearCtrlLimits(max_vel=1.0, max_acc=3.0, max_decel=2.5), ctrl_omega_limits=AngularCtrlLimits( max_vel=4.0, max_acc=6.0, max_decel=10.0, max_steer=np.pi / 3 ), ) # --- Navigation Components --- depth_cam_info_topic = Topic( name="/camera/aligned_depth_to_color/camera_info", msg_type="CameraInfo" ) config = ControllerConfig(ctrl_publish_type="Parallel") controller = Controller(component_name="controller", config=config) controller.inputs( vision_detections=detections_topic, depth_camera_info=depth_cam_info_topic, ) controller.algorithm = "VisionRGBDFollower" controller.direct_sensor = False driver = DriveManager(component_name="driver") mapper = LocalMapper(component_name="local_mapper") # --- Event: Person Detected --- event_person_detected = Event( event_condition=detections_topic.msg.labels.contains("person"), on_change=True, ) # --- Actions: Switch Algorithm + Start Following --- switch_algorithm_action = update_parameter( component=controller, param_name="algorithm", new_value="VisionRGBDFollower", ) action_request_msg = TrackVisionTarget.Goal() action_request_msg.label = "person" action_start_person_following = send_component_action_server_goal( component=controller, request_msg=action_request_msg, ) events_action = { event_person_detected: [switch_algorithm_action, action_start_person_following], } # --- Launch --- launcher = Launcher() launcher.add_pkg( components=[vision], ros_log_level="warn", package_name="automatika_embodied_agents", executable_entry_point="executable", multiprocessing=True, ) launcher.kompass( components=[controller, mapper, driver], events_actions=events_action, ) launcher.robot = my_robot launcher.bringup() ``` --- ```{tip} **Promote this recipe to production.** While you're shaping it, the script runs straight with `python recipe.py`. Once it's solid, drop it at `~/emos/recipes//recipe.py` and run `emos run ` -- you'll get sensor pre-flight checks, persistent logs, and a card on the dashboard so an operator can launch it from a browser. See [Running Recipes](../../getting-started/running-recipes.md) for the full development-vs-production comparison and install-mode pitfalls (especially in Container mode). ``` ``` ## File: recipes/events-and-resilience/cross-component-events.md ```markdown # Cross-Component Healing In the [Self-Healing with Fallbacks](fallback-recipes.md) recipe, we learned how a component can heal *itself* (e.g., restarting or switching algorithms). But sophisticated autonomy requires more than self-repair -- it requires **system-level awareness**, where components monitor *each other* and take corrective action. In this recipe, we use **Events** to implement cross-component healing: one component detects a failure, and a *different* component executes the recovery. --- ## Scenario A: The "Unstuck" Reflex The `Controller` gets stuck in a local minimum (e.g., the robot is facing a corner). It reports an `ALGORITHM_FAILURE` because it cannot find a valid velocity command. We detect this status and ask the `DriveManager` to execute a blind "Unblock" maneuver -- rotate in place or back up. ```{tip} All component health status topics are accessible via `component.status_topic`. ``` ### Define the Event and Action ```python from kompass.ros import Event, Action, Topic from sugar.msg import ComponentStatus # Event: Controller reports algorithm failure # keep_event_delay prevents re-triggering while recovery is in progress event_controller_fail = Event( controller.status_topic.msg.status == ComponentStatus.STATUS_FAILURE_ALGORITHM_LEVEL, keep_event_delay=60.0 ) # Action: DriveManager executes a recovery maneuver unblock_action = Action(method=driver.move_to_unblock) ``` The `keep_event_delay=60.0` ensures the unblock action fires at most once per minute, giving the controller time to recover before trying again. --- ## Scenario B: The "Blind Mode" Reflex The `LocalMapper` crashes, failing to provide the high-fidelity local map that the `Controller` depends on. Instead of halting, the `Controller` reconfigures itself to use raw sensor data directly (reactive mode). ```python from kompass.actions import update_parameter # Event: Mapper is NOT healthy # handle_once=True means this fires only ONCE during the system's lifetime event_mapper_fault = Event( mapper.status_topic.msg.status != ComponentStatus.STATUS_HEALTHY, handle_once=True ) # Action: Reconfigure Controller to bypass the mapper activate_direct_sensor_mode = update_parameter( component=controller, param_name="use_direct_sensor", new_value=True ) ``` --- ## Scenario C: Goal Handling via Events In a production system, goals often arrive from external interfaces like RViz rather than being hardcoded. Events bridge the gap: we listen for clicked points and forward them to the Planner's ActionServer. ### Define the Goal Event ```python from kompass import event from kompass.actions import ComponentActions # Fire whenever a new PointStamped arrives on /clicked_point event_clicked_point = event.OnGreater( "rviz_goal", Topic(name="/clicked_point", msg_type="PointStamped"), 0, ["header", "stamp", "sec"], ) ``` ### Define the Goal Action with a Parser The clicked point message needs to be converted into a `PlanPath.Goal`. We write a parser function and attach it to the action: ```python from kompass_interfaces.action import PlanPath from kompass_interfaces.msg import PathTrackingError from geometry_msgs.msg import Pose, PointStamped from kompass.actions import LogInfo # Create the action server goal action send_goal = ComponentActions.send_action_goal( action_name="/planner/plan_path", action_type=PlanPath, action_request_msg=PlanPath.Goal(), ) # Parse PointStamped into PlanPath.Goal def goal_point_parser(*, msg: PointStamped, **_): action_request = PlanPath.Goal() goal = Pose() goal.position.x = msg.point.x goal.position.y = msg.point.y action_request.goal = goal end_tolerance = PathTrackingError() end_tolerance.orientation_error = 0.2 end_tolerance.lateral_distance_error = 0.05 action_request.end_tolerance = end_tolerance return action_request send_goal.event_parser(goal_point_parser, output_mapping="action_request_msg") ``` ```{tip} `ComponentActions.send_srv_request` and `ComponentActions.send_action_goal` let you call **any** ROS 2 service or action server from an event -- not just EMOS services. ``` --- ## Wiring Events to Actions With all events and actions defined, we assemble the event-action dictionary. Each event maps to one or more actions: ```python events_actions = { # RViz click -> log + send goal to planner event_clicked_point: [LogInfo(msg="Got new goal point"), send_goal], # Controller stuck -> unblock maneuver event_controller_fail: unblock_action, # Mapper down -> switch controller to direct sensor mode event_mapper_fault: activate_direct_sensor_mode, } ``` --- ## Complete Recipe ```{code-block} python :caption: cross_component_healing.py :linenos: import numpy as np import os from sugar.msg import ComponentStatus from kompass_interfaces.action import PlanPath from kompass_interfaces.msg import PathTrackingError from geometry_msgs.msg import Pose, PointStamped from kompass import event from kompass.actions import Action, ComponentActions, LogInfo, update_parameter from kompass.components import ( Controller, DriveManager, Planner, PlannerConfig, LocalMapper, ) from kompass.config import RobotConfig from kompass.robot import ( AngularCtrlLimits, LinearCtrlLimits, RobotGeometry, RobotType, ) from kompass.ros import Topic, Launcher, Event # --- Robot Configuration --- my_robot = RobotConfig( model_type=RobotType.DIFFERENTIAL_DRIVE, geometry_type=RobotGeometry.Type.CYLINDER, geometry_params=np.array([0.1, 0.3]), ctrl_vx_limits=LinearCtrlLimits(max_vel=0.2, max_acc=1.5, max_decel=2.5), ctrl_omega_limits=AngularCtrlLimits( max_vel=0.4, max_acc=2.0, max_decel=2.0, max_steer=np.pi / 3 ), ) # --- Components --- planner = Planner(component_name="planner", config=PlannerConfig(loop_rate=1.0)) planner.run_type = "ActionServer" controller = Controller(component_name="controller") controller.direct_sensor = False mapper = LocalMapper(component_name="mapper") driver = DriveManager(component_name="drive_manager") if os.environ.get("ROS_DISTRO") in ["rolling", "jazzy", "kilted"]: cmd_msg_type = "TwistStamped" else: cmd_msg_type = "Twist" driver.outputs(robot_command=Topic(name="/cmd_vel", msg_type=cmd_msg_type)) # --- Cross-Component Events --- # 1. Controller stuck -> DriveManager unblocks event_controller_fail = Event( controller.status_topic.msg.status == ComponentStatus.STATUS_FAILURE_ALGORITHM_LEVEL, keep_event_delay=60.0 ) unblock_action = Action(method=driver.move_to_unblock) # 2. Mapper down -> Controller switches to direct sensor mode event_mapper_fault = Event( mapper.status_topic.msg.status != ComponentStatus.STATUS_HEALTHY, handle_once=True ) activate_direct_sensor_mode = update_parameter( component=controller, param_name="use_direct_sensor", new_value=True ) # 3. RViz click -> Planner goal event_clicked_point = event.OnGreater( "rviz_goal", Topic(name="/clicked_point", msg_type="PointStamped"), 0, ["header", "stamp", "sec"], ) send_goal = ComponentActions.send_action_goal( action_name="/planner/plan_path", action_type=PlanPath, action_request_msg=PlanPath.Goal(), ) def goal_point_parser(*, msg: PointStamped, **_): action_request = PlanPath.Goal() goal = Pose() goal.position.x = msg.point.x goal.position.y = msg.point.y action_request.goal = goal end_tolerance = PathTrackingError() end_tolerance.orientation_error = 0.2 end_tolerance.lateral_distance_error = 0.05 action_request.end_tolerance = end_tolerance return action_request send_goal.event_parser(goal_point_parser, output_mapping="action_request_msg") # --- Wire Events -> Actions --- events_actions = { event_clicked_point: [LogInfo(msg="Got new goal point"), send_goal], event_controller_fail: unblock_action, event_mapper_fault: activate_direct_sensor_mode, } # --- Launch --- odom_topic = Topic(name="/odometry/filtered", msg_type="Odometry") launcher = Launcher() launcher.kompass( components=[planner, controller, mapper, driver], events_actions=events_actions, activate_all_components_on_start=True, multi_processing=True, ) launcher.inputs(location=odom_topic) launcher.robot = my_robot launcher.bringup() ``` --- ```{tip} **Promote this recipe to production.** While you're shaping it, the script runs straight with `python recipe.py`. Once it's solid, drop it at `~/emos/recipes//recipe.py` and run `emos run ` -- you'll get sensor pre-flight checks, persistent logs, and a card on the dashboard so an operator can launch it from a browser. See [Running Recipes](../../getting-started/running-recipes.md) for the full development-vs-production comparison and install-mode pitfalls (especially in Container mode). ``` ``` ## File: recipes/events-and-resilience/composed-events.md ```markdown # Logic Gates & Composed Events In the previous recipes, we triggered actions based on single, isolated conditions -- "If Mapper Fails" or "If Person Detected". But real-world autonomy is rarely that simple. A robot shouldn't stop *every* time it sees an obstacle -- maybe only if it's moving fast. It shouldn't return home *just* because the battery is low -- maybe only after finishing its current task. In this recipe, we use **logic operators** to compose multiple conditions into smarter, more robust event triggers. --- ## Logic Operators EMOS lets you compose complex triggers using standard Python bitwise operators. This turns your Event definitions into a high-level logic circuit. | Logic | Operator | Description | Use Case | |:---|:---|:---|:---| | **AND** | `&` | All conditions must be True | Speed > 0 **AND** Obstacle Close | | **OR** | `\|` | At least one condition is True | Lidar Blocked **OR** Bumper Hit | | **NOT** | `~` | Inverts the condition | Target Seen **AND NOT** Low Battery | --- ## Navigation Example: Smart Emergency Stop **The problem:** A naive emergency stop triggers whenever an object is within 0.5m. But if the robot is docking or maneuvering in a tight elevator, this stops it unnecessarily. **The solution:** Trigger ONLY if an obstacle is close **AND** the robot is moving fast. ### 1. Define the Data Sources ```python from kompass.ros import Topic # Radar distance reading (0.2s timeout for safety-critical data) radar = Topic(name="/radar_front", msg_type="Float32", data_timeout=0.2) # Odometry (0.5s timeout) odom = Topic(name="/odom", msg_type="Odometry", data_timeout=0.5) ``` ### 2. Compose the Event ```python from kompass.ros import Event, Action # Condition A: Obstacle within 0.3m is_obstacle_close = radar.msg.data < 0.3 # Condition B: Robot moving faster than 0.1 m/s is_robot_moving_fast = odom.msg.twist.twist.linear.x > 0.1 # Composed Event: BOTH must be True event_smart_stop = Event( event_condition=(is_obstacle_close & is_robot_moving_fast), on_change=True ) ``` ### 3. Wire to Action ```python from kompass.components import DriveManager driver = DriveManager(component_name="drive_manager") # Emergency stop action stop_action = Action(method=driver.stop) events_actions = { event_smart_stop: stop_action, } ``` Now the robot stops only when it *should* -- fast approach toward an obstacle -- and ignores close objects during slow precision maneuvers. --- ## Intelligence Example: Conditional Cognition We can apply the same logic to the intelligence layer. Consider the [Event-Driven Cognition](event-driven-cognition.md) recipe where a Vision detector triggers a VLM. What if we only want the VLM to run when the robot has sufficient battery? ```python from agents.ros import Topic, Event # Detection output from the Vision component detections = Topic(name="/detections", msg_type="Detections") # Battery level topic battery = Topic(name="/battery_state", msg_type="Float32") # Condition A: Person detected (with debounce) person_detected = detections.msg.labels.contains_any(["person"]) # Condition B: Battery above 20% battery_ok = battery.msg.data > 20.0 # Composed Event: person detected AND battery sufficient event_describe_person = Event( event_condition=(person_detected & battery_ok), on_change=True, keep_event_delay=5 ) ``` The VLM only wakes up when both conditions are met -- saving compute when the battery is low. --- ## OR Logic: Redundant Sensors The `|` operator is useful for sensor redundancy. If *either* the front lidar detects a close obstacle or the bumper is pressed, trigger an emergency stop: ```python from kompass.ros import Topic, Event lidar = Topic(name="/scan_front", msg_type="Float32") bumper = Topic(name="/bumper", msg_type="Bool") is_lidar_blocked = lidar.msg.data < 0.2 is_bumper_pressed = bumper.msg.data == True event_any_collision = Event( event_condition=(is_lidar_blocked | is_bumper_pressed), on_change=True ) ``` --- ## NOT Logic: Exclusion The `~` operator inverts a condition. Use it to exclude scenarios: ```python from kompass.ros import Topic, Event # Only track the target if the robot is NOT in manual override mode manual_mode = Topic(name="/manual_override", msg_type="Bool") target_seen = detections.msg.labels.contains_any(["person"]) event_auto_track = Event( event_condition=(target_seen & ~manual_mode.msg.data), on_change=True ) ``` --- ## Event Configuration Reference All composed events support these parameters: | Parameter | Description | Default | |---|---|---| | `on_change` | Trigger only when the condition *transitions* to True (edge-triggered) | `False` | | `handle_once` | Fire only once during the system's lifetime | `False` | | `keep_event_delay` | Minimum seconds between consecutive triggers (debounce) | `0` | ```{seealso} For the full list of event types and configuration options, see the [Events & Actions](../../concepts/events-and-actions.md) reference. ``` ``` ## File: recipes/events-and-resilience/context-aware-actions.md ```markdown # Context-Aware Actions In previous recipes, our actions used **static** arguments -- pre-defined at configuration time. For example, in [Self-Healing with Fallbacks](fallback-recipes.md), we defined `Action(method=controller.set_algorithm, args=(ControllersID.PURE_PURSUIT,))` where the target algorithm is hardcoded. But what if the action depends on **what** the robot is seeing, or **where** it was told to go? Real-world autonomy requires **dynamic data injection** -- action arguments fetched from the system at the time of execution. --- ## The Concept: Static vs Dynamic | Type | Argument Set At | Example | |---|---|---| | **Static** | Configuration time | `args=(ControllersID.PURE_PURSUIT,)` | | **Dynamic** | Event firing time | `args=(command_topic.msg.data,)` | With dynamic injection, you pass a **topic message field** as an argument. EMOS resolves the actual value when the event fires, not when the recipe is written. --- ## Navigation Example: Semantic Navigation We build a system where you publish a location name (like "kitchen") to a topic, and the robot automatically looks up the coordinates and navigates there. ### 1. Define the Command Source ```python from kompass.ros import Topic # Simulates a voice command or fleet management instruction # Examples: "kitchen", "reception", "station_a" command_topic = Topic(name="/user_command", msg_type="String") ``` ### 2. Write the Lookup Function ```python import subprocess # A simple map of the environment # In a real app, this could come from a database or semantic memory WAYPOINTS = { "kitchen": {"x": 2.0, "y": 0.5}, "reception": {"x": 0.0, "y": 0.0}, "station_a": {"x": -1.5, "y": 2.0}, } def navigate_to_location(location_name: str): """Looks up coordinates and publishes a goal to the planner.""" key = location_name.strip().lower() if key not in WAYPOINTS: print(f"Unknown location: {key}") return coords = WAYPOINTS[key] topic_cmd = ( f"ros2 topic pub --once /clicked_point geometry_msgs/msg/PointStamped " f"'{{header: {{frame_id: \"map\"}}, point: {{x: {coords['x']}, y: {coords['y']}, z: 0.0}}}}'" ) subprocess.run(topic_cmd, shell=True) ``` ### 3. Define the Event and Action ```python from kompass.ros import Event, Action from sugar.msg import ComponentStatus # Trigger on any new command, but only if the mapper is healthy event_command_received = Event( event_condition=( command_topic & (mapper.status_topic.msg.status == ComponentStatus.STATUS_HEALTHY) ), ) # DYNAMIC INJECTION: command_topic.msg.data is resolved at event-fire time action_process_command = Action( method=navigate_to_location, args=(command_topic.msg.data,) ) ``` When someone publishes `"kitchen"` to `/user_command`, the Event fires and the Action calls `navigate_to_location("kitchen")` -- the string is fetched live from the topic. --- ## Intelligence Example: Dynamic Prompt Injection The same pattern works for the intelligence layer. Consider a Vision component that detects objects, and a VLM that should describe *whatever* was detected -- not just "person": ```python from agents.ros import Topic, Event, Action, FixedInput from agents.components import Vision, VLM # Vision outputs detections = Topic(name="/detections", msg_type="Detections") camera_image = Topic(name="/image_raw", msg_type="Image") # Event: any object detected event_object_detected = Event( detections.msg.labels.length() > 0, on_change=True, keep_event_delay=5 ) # Dynamic prompt: inject the detected label into the VLM query def describe_detected_object(label: str): """Called with the actual detected label at event time.""" return f"A {label} has been detected. Describe what you see." action_describe = Action( method=describe_detected_object, args=(detections.msg.labels[0],) # First detected label, resolved dynamically ) ``` --- ## Complete Navigation Recipe Launch this script, then publish a string to `/user_command` (e.g., `ros2 topic pub /user_command std_msgs/String "data: kitchen" --once`) to see the robot navigate. ```{code-block} python :caption: semantic_navigation.py :linenos: import os import subprocess import numpy as np from sugar.msg import ComponentStatus from kompass.components import ( Controller, DriveManager, Planner, PlannerConfig, LocalMapper, ) from kompass.config import RobotConfig from kompass.control import ControllersID from kompass.robot import ( AngularCtrlLimits, LinearCtrlLimits, RobotGeometry, RobotType, ) from kompass.ros import Topic, Launcher, Action, Event # --- Waypoint Database --- WAYPOINTS = { "kitchen": {"x": 2.0, "y": 0.5}, "reception": {"x": 0.0, "y": 0.0}, "station_a": {"x": -1.5, "y": 2.0}, } def navigate_to_location(location_name: str): key = location_name.strip().lower() if key not in WAYPOINTS: print(f"Unknown location: {key}") return coords = WAYPOINTS[key] topic_cmd = ( f"ros2 topic pub --once /clicked_point geometry_msgs/msg/PointStamped " f"'{{header: {{frame_id: \"map\"}}, point: {{x: {coords['x']}, y: {coords['y']}, z: 0.0}}}}'" ) subprocess.run(topic_cmd, shell=True) # --- Command Topic --- command_topic = Topic(name="/user_command", msg_type="String") # --- Robot Configuration --- my_robot = RobotConfig( model_type=RobotType.DIFFERENTIAL_DRIVE, geometry_type=RobotGeometry.Type.CYLINDER, geometry_params=np.array([0.1, 0.3]), ctrl_vx_limits=LinearCtrlLimits(max_vel=0.2, max_acc=1.5, max_decel=2.5), ctrl_omega_limits=AngularCtrlLimits( max_vel=0.4, max_acc=2.0, max_decel=2.0, max_steer=np.pi / 3 ), ) # --- Components --- planner = Planner(component_name="planner", config=PlannerConfig(loop_rate=1.0)) goal_topic = Topic(name="/clicked_point", msg_type="PointStamped") planner.inputs(goal_point=goal_topic) controller = Controller(component_name="controller") controller.direct_sensor = False controller.algorithm = ControllersID.DWA mapper = LocalMapper(component_name="mapper") driver = DriveManager(component_name="drive_manager") if os.environ.get("ROS_DISTRO") in ["rolling", "jazzy", "kilted"]: cmd_msg_type = "TwistStamped" else: cmd_msg_type = "Twist" driver.outputs(robot_command=Topic(name="/cmd_vel", msg_type=cmd_msg_type)) # --- Context-Aware Event & Action --- event_command_received = Event( event_condition=( command_topic & (mapper.status_topic.msg.status == ComponentStatus.STATUS_HEALTHY) ), ) action_process_command = Action( method=navigate_to_location, args=(command_topic.msg.data,) # Dynamic injection ) events_actions = { event_command_received: action_process_command, } # --- Launch --- launcher = Launcher() launcher.kompass( components=[planner, controller, driver, mapper], activate_all_components_on_start=True, multi_processing=True, events_actions=events_actions, ) odom_topic = Topic(name="/odometry/filtered", msg_type="Odometry") launcher.inputs(location=odom_topic) launcher.robot = my_robot launcher.bringup() ``` --- ```{tip} **Promote this recipe to production.** While you're shaping it, the script runs straight with `python recipe.py`. Once it's solid, drop it at `~/emos/recipes//recipe.py` and run `emos run ` -- you'll get sensor pre-flight checks, persistent logs, and a card on the dashboard so an operator can launch it from a browser. See [Running Recipes](../../getting-started/running-recipes.md) for the full development-vs-production comparison and install-mode pitfalls (especially in Container mode). ``` ``` ## File: advanced/configuration.md ```markdown # Configuration EMOS is built for flexibility -- and that starts with how you configure your components. Whether you are scripting in Python, editing clean and readable YAML, crafting elegant TOML files, or piping in JSON from a toolchain, EMOS lets you do it your way. No rigid formats or boilerplate structures. Just straightforward, expressive configuration -- however you like to write it. ## Configuration Formats EMOS supports four configuration methods: - [Python API](#python-api) - [YAML](#yaml) - [TOML](#toml) - [JSON](#json) Pick your format. Plug it in. Go. ## Python API Use the full power of the Pythonic API to configure your components when you need dynamic logic, computation, or tighter control. ```python from kompass.components import Planner, PlannerConfig from kompass.ros import Topic from kompass.robot import ( AngularCtrlLimits, LinearCtrlLimits, RobotGeometry, RobotType, RobotConfig, RobotFrames ) import numpy as np import math # Define your robot's physical and control characteristics my_robot = RobotConfig( model_type=RobotType.DIFFERENTIAL_DRIVE, # Type of robot motion model geometry_type=RobotGeometry.Type.CYLINDER, # Shape of the robot geometry_params=np.array([0.1, 0.3]), # Diameter and height of the cylinder ctrl_vx_limits=LinearCtrlLimits( # Linear velocity constraints max_vel=0.4, max_acc=1.5, max_decel=2.5 ), ctrl_omega_limits=AngularCtrlLimits( # Angular velocity constraints max_vel=0.4, max_acc=2.0, max_decel=2.0, max_steer=math.pi / 3 # Steering angle limit (radians) ), ) # Define the robot's coordinates frames my_frames = RobotFrames( world="map", odom="odom", robot_base="body", scan="lidar_link" ) # Create the planner config using your robot setup config = PlannerConfig( robot=my_robot, loop_rate=1.0 ) # Instantiate the Planner component planner = Planner( component_name="planner", config=config ) # Additionally configure the component's inputs or outputs planner.inputs( map_layer=Topic(name="/map", msg_type="OccupancyGrid"), goal_point=Topic(name="/clicked_point", msg_type="PointStamped") ) ``` ## YAML Similar to traditional ROS 2 launch, you can maintain all your configuration parameters in a YAML file. EMOS simplifies the standard ROS 2 YAML format -- just drop the `ros__parameters` noise: ```yaml /**: # Common parameters for all components frames: robot_base: "body" odom: "odom" world: "map" scan: "lidar_link" robot: model_type: "DIFFERENTIAL_DRIVE" geometry_type: "CYLINDER" geometry_params: [0.1, 0.3] ctrl_vx_limits: max_vel: 0.4 max_acc: 1.5 max_decel: 2.5 ctrl_omega_limits: max_vel: 0.4 max_acc: 2.0 max_decel: 2.0 max_steer: 1.0472 # ~ pi / 3 planner: inputs: map_layer: name: "/map" msg_type: "OccupancyGrid" goal_point: name: "/clicked_point" msg_type: "PointStamped" loop_rate: 1.0 ``` Common parameters placed under the `/**` key are shared across all components. Component-specific parameters are placed under the component name. ## TOML Not a fan of YAML? EMOS lets you configure your components using TOML too. TOML offers clear structure and excellent tooling support, making it ideal for clean, maintainable configs. ```toml ["/**".frames] robot_base = "body" odom = "odom" world = "map" scan = "lidar_link" ["/**".robot] model_type = "DIFFERENTIAL_DRIVE" geometry_type = "CYLINDER" geometry_params = [0.1, 0.3] ["/**".robot.ctrl_vx_limits] max_vel = 0.4 max_acc = 1.5 max_decel = 2.5 ["/**".robot.ctrl_omega_limits] max_vel = 0.4 max_acc = 2.0 max_decel = 2.0 max_steer = 1.0472 # ~ pi / 3 [planner] loop_rate = 1.0 [planner.inputs.map_layer] name = "/map" msg_type = "OccupancyGrid" [planner.inputs.goal_point] name = "/clicked_point" msg_type = "PointStamped" ``` ## JSON Prefer curly braces? Or looking to pipe configs from an ML model or external toolchain? JSON is machine-friendly and widely supported -- perfect for automating your EMOS configuration with generated files. ```json { "/**": { "frames": { "robot_base": "body", "odom": "odom", "world": "map", "scan": "lidar_link" }, "robot": { "model_type": "DIFFERENTIAL_DRIVE", "geometry_type": "CYLINDER", "geometry_params": [0.1, 0.3], "ctrl_vx_limits": { "max_vel": 0.4, "max_acc": 1.5, "max_decel": 2.5 }, "ctrl_omega_limits": { "max_vel": 0.4, "max_acc": 2.0, "max_decel": 2.0, "max_steer": 1.0472 } } }, "planner": { "loop_rate": 1.0, "inputs": { "map_layer": { "name": "/map", "msg_type": "OccupancyGrid" }, "goal_point": { "name": "/clicked_point", "msg_type": "PointStamped" } } } } ``` ## Minimal Configuration Examples For simple components that do not require full robot configuration, the config files are even more concise: ::::{tab-set} :::{tab-item} YAML ```yaml /**: common_int_param: 0 my_component_name: float_param: 1.5 boolean_param: true ``` ::: :::{tab-item} TOML ```toml ["/**"] common_int_param = 0 [my_component_name] float_param = 1.5 boolean_param = true ``` ::: :::{tab-item} JSON ```json { "/**": { "common_int_param": 0 }, "my_component_name": { "float_param": 1.5, "boolean_param": true } } ``` ::: :::: ```{note} Make sure to pass your config file to the component on initialization or to the Launcher. ``` :::{seealso} You can check complete examples of detailed configuration files in the [EMOS navigation params](https://github.com/automatika-robotics/kompass/tree/main/kompass/params). ::: ``` ## File: advanced/extending.md ```markdown # Extending EMOS EMOS is designed to be extended. This guide covers how to create custom components, deploy them as system services, use built-in services for live reconfiguration, and write robot plugins for hardware portability. ## Creating Custom Components :::{tip} To see detailed examples of packages built with EMOS, check out [Kompass](https://automatika-robotics.github.io/kompass/) (navigation) and [EmbodiedAgents](https://automatika-robotics.github.io/embodied-agents/) (intelligence). ::: :::{note} Before building your own package, review the core [design concepts](../concepts/components.md). ::: ### Step 1 -- Create a ROS 2 Package Start by creating a standard ROS 2 Python package: ```bash ros2 pkg create --build-type ament_python --license Apache-2.0 my-awesome-pkg ``` ### Step 2 -- Define Your Component Create your first functional unit (component) in a new file: ```bash cd my-awesome-pkg/my_awesome_pkg touch awesome_component.py ``` ### Step 3 -- Setup Component Configuration Extend `BaseComponentConfig` based on the [attrs](https://www.attrs.org/en/stable/) package: ```python from attrs import field, define from ros_sugar.config import BaseComponentConfig, base_validators @define(kw_only=True) class AwesomeConfig(BaseComponentConfig): """ Component configuration parameters """ extra_float: float = field( default=10.0, validator=base_validators.in_range(min_value=1e-9, max_value=1e9) ) extra_flag: bool = field(default=True) ``` ### Step 4 -- Implement the Component Initialize your component by inheriting from `BaseComponent`. Code the desired functionality in your component: ```python from ros_sugar.core import ComponentFallbacks, BaseComponent from ros_sugar.io import Topic class AwesomeComponent(BaseComponent): def __init__( self, *, component_name: str, inputs: Optional[Sequence[Topic]] = None, outputs: Optional[Sequence[Topic]] = None, config_file: Optional[str] = None, config: Optional[AwesomeConfig] = None, **kwargs, ) -> None: # Set default config if config is not provided self.config: AwesomeConfig = config or AwesomeConfig() super().__init__( component_name=component_name, inputs=inputs, outputs=outputs, config=self.config, config_file=config_file, **kwargs, ) def _execution_step(self): """ The execution step is the main (timed) functional unit in the component. Gets called automatically at every loop step (with a frequency of 'self.config.loop_rate'). """ super()._execution_step() # Add your main execution step here ``` Follow this pattern to create any number of functional units in your package. ### Step 5 -- Create an Entry Point (Multi-Process) To use your components with the EMOS Launcher in multi-process execution, create an entry point: ```python #!/usr/bin/env python3 from ros_sugar import executable_main from my_awesome_pkg.awesome_component import AwesomeComponent, AwesomeConfig # Create lists of available components/config classes _components_list = [AwesomeComponent] _configs_list = [AwesomeConfig] # Create entry point main def main(args=None): executable_main(list_of_components=_components_list, list_of_configs=_configs_list) ``` Add the entry point to the ROS 2 package `setup.py`: ```python from setuptools import find_packages, setup package_name = "my_awesome_pkg" console_scripts = [ "executable = my_awesome_pkg.executable:main", ] setup( name=package_name, version="1", packages=find_packages(), install_requires=["setuptools"], zip_safe=True, entry_points={ "console_scripts": console_scripts, }, ) ``` Build your ROS 2 package with colcon, then use the Launcher to bring up your system. ### Step 6 -- Launch with EMOS Use the EMOS Launcher to bring up your package: ```{code-block} python :caption: Using the EMOS Launcher with your package :linenos: from my_awesome_pkg.awesome_component import AwesomeComponent, AwesomeConfig from ros_sugar.actions import LogInfo from ros_sugar.events import OnLess from ros_sugar import Launcher from ros_sugar.io import Topic # Define a set of topics map_topic = Topic(name="map", msg_type="OccupancyGrid") audio_topic = Topic(name="voice", msg_type="Audio") image_topic = Topic(name="camera/rgb", msg_type="Image") # Init your components my_component = AwesomeComponent( component_name='awesome_component', inputs=[map_topic, image_topic], outputs=[audio_topic] ) # Create your events low_battery = Event(battery_level_topic.msg.data < 15.0) # Events/Actions my_events_actions: Dict[event.Event, Action] = { low_battery: LogInfo(msg="Battery is Low!") } # Create your launcher launcher = Launcher() # Add your package components launcher.add_pkg( components=[my_component], package_name='my_awesome_pkg', executable_entry_point='executable', events_actions=my_events_actions, activate_all_components_on_start=True, multiprocessing=True, ) # If any component fails -> restart it with unlimited retries launcher.on_component_fail(action_name="restart") # Bring up the system launcher.bringup() ``` --- ## Deploying as systemd Services EMOS recipes can be easily deployed as `systemd` services for production environments or embedded systems where automatic startup and restart behavior is critical. Once you have a Python script for your EMOS-based package (e.g., `my_awesome_system.py`), install it as a systemd service: ```bash ros2 run automatika_ros_sugar create_service ``` ### Arguments - ``: The full path to your EMOS Python script (e.g., `/path/to/my_awesome_system.py`). - ``: The name of the systemd service (do **not** include the `.service` extension). ### Example ```bash ros2 run automatika_ros_sugar create_service ~/ros2_ws/my_awesome_system.py my_awesome_service ``` This installs and optionally enables a `systemd` service named `my_awesome_service.service`. ### Full Command Usage ```text usage: create_service [-h] [--service-description SERVICE_DESCRIPTION] [--install-path INSTALL_PATH] [--source-workspace-path SOURCE_WORKSPACE_PATH] [--no-enable] [--restart-time RESTART_TIME] service_file_path service_name ``` **Positional Arguments:** - **`service_file_path`**: Path to the Python script to install as a service. - **`service_name`**: Name of the systemd service (without `.service` extension). **Optional Arguments:** - `-h, --help`: Show the help message and exit. - `--service-description SERVICE_DESCRIPTION`: Human-readable description of the service. Defaults to `"EMOS Service"`. - `--install-path INSTALL_PATH`: Directory to install the systemd service file. Defaults to `/etc/systemd/system`. - `--source-workspace-path SOURCE_WORKSPACE_PATH`: Path to the ROS workspace `setup` script. If omitted, it auto-detects the active ROS distribution. - `--no-enable`: Skip enabling the service after installation. - `--restart-time RESTART_TIME`: Time to wait before restarting the service if it fails (e.g., `3s`). Default is `3s`. ### What This Does This command: 1. Creates a `.service` file for `systemd`. 2. Installs it in the specified or default location. 3. Sources the appropriate ROS environment. 4. Optionally enables and starts the service immediately. Once installed, manage the service with standard `systemd` commands: ```bash sudo systemctl start my_awesome_service sudo systemctl status my_awesome_service sudo systemctl stop my_awesome_service sudo systemctl enable my_awesome_service ``` --- ## Built-in Services for Live Reconfiguration In addition to the standard [ROS 2 Lifecycle Node](https://github.com/ros2/demos/blob/rolling/lifecycle/README.rst) services, EMOS components provide a powerful set of built-in services for live reconfiguration. These services allow you to dynamically adjust inputs, outputs, and parameters on-the-fly, making it easier to respond to changing runtime conditions or trigger intelligent behavior in response to events. Like any ROS 2 services, they can be called from other Nodes or with the ROS 2 CLI, and can also be called programmatically as part of an action sequence or event-driven workflow in the launch script. ### Replacing an Input or Output with a Different Topic You can swap an existing topic connection (input or output) with a different topic online without restarting your script. The service will stop the running lifecycle node, replace the connection, and restart it. - **Service Name:** `/{component_name}/change_topic` - **Service Type:** `automatika_ros_sugar/srv/ReplaceTopic` **Example:** ```shell ros2 service call /awesome_component/change_topic automatika_ros_sugar/srv/ReplaceTopic \ "{direction: 1, old_name: '/voice', new_name: '/audio_device_0', new_msg_type: 'Audio'}" ``` ### Updating a Configuration Parameter Value The `ChangeParameter` service allows updating a single configuration parameter at runtime. You can choose whether the component remains active during the change, or temporarily deactivates for a safe update. - **Service Name:** `/{component_name}/update_config_parameter` - **Service Type:** `automatika_ros_sugar/srv/ChangeParameter` **Example:** ```shell ros2 service call /awesome_component/update_config_parameter automatika_ros_sugar/srv/ChangeParameter \ "{name: 'loop_rate', value: '1', keep_alive: false}" ``` ### Updating Multiple Configuration Parameters The `ChangeParameters` service allows updating multiple parameters at once, ideal for switching modes or reconfiguring components in batches. - **Service Name:** `/{component_name}/update_config_parameters` - **Service Type:** `automatika_ros_sugar/srv/ChangeParameters` **Example:** ```shell ros2 service call /awesome_component/update_config_parameters automatika_ros_sugar/srv/ChangeParameters \ "{names: ['loop_rate', 'fallback_rate'], values: ['1', '10'], keep_alive: false}" ``` ### Reconfiguring from a File The `ConfigureFromFile` service lets you reconfigure an entire component from a YAML, JSON, or TOML configuration file while the node is online. This is useful for applying scenario-specific settings or restoring saved configurations in a single operation. - **Service Name:** `/{component_name}/configure_from_file` - **Service Type:** `automatika_ros_sugar/srv/ConfigureFromFile` **Example YAML configuration file:** ```yaml /**: fallback_rate: 10.0 awesome_component: loop_rate: 100.0 ``` ### Executing a Component Method The `ExecuteMethod` service enables runtime invocation of any class method in the component. This is useful for triggering specific behaviors, tools, or diagnostics during runtime without writing additional interfaces. - **Service Name:** `/{component_name}/execute_method` - **Service Type:** `automatika_ros_sugar/srv/ExecuteMethod` ```{seealso} To make your recipes portable across different robot hardware, see [Robot Plugins](../concepts/robot-plugins.md). ``` ``` ## File: advanced/types.md ```markdown # Supported Types EMOS components automatically create subscribers and publishers for all inputs and outputs. This page provides a comprehensive reference of all natively supported ROS 2 message types across the full EMOS stack -- the orchestration layer (Sugarcoat), intelligence layer (EmbodiedAgents), and navigation layer (Kompass). When defining a [Topic](../concepts/topics.md), you pass the message type as a string (e.g., `Topic(name="/image", msg_type="Image")`). The framework handles all serialization, callback creation, and type conversion automatically. ## Standard Messages | Message | ROS 2 Package | Description | |:---|:---|:---| | **String** | std_msgs | Standard text message | | **Bool** | std_msgs | Boolean value | | **Float32** | std_msgs | Single-precision float | | **Float32MultiArray** | std_msgs | Array of single-precision floats | | **Float64** | std_msgs | Double-precision float | | **Float64MultiArray** | std_msgs | Array of double-precision floats | ## Geometry Messages | Message | ROS 2 Package | Description | |:---|:---|:---| | **Point** | geometry_msgs | 3D point (x, y, z) | | **PointStamped** | geometry_msgs | Timestamped 3D point | | **Pose** | geometry_msgs | Position + orientation | | **PoseStamped** | geometry_msgs | Timestamped pose | | **Twist** | geometry_msgs | Linear + angular velocity | | **TwistStamped** | geometry_msgs | Timestamped velocity | ## Sensor Messages | Message | ROS 2 Package | Description | |:---|:---|:---| | **Image** | sensor_msgs | Raw image data | | **CompressedImage** | sensor_msgs | Compressed image (JPEG, PNG) | | **Audio** | sensor_msgs | Audio stream data | | **LaserScan** | sensor_msgs | 2D lidar scan | | **PointCloud2** | sensor_msgs | 3D point cloud | | **CameraInfo** | sensor_msgs | Camera calibration and metadata | | **JointState** | sensor_msgs | Instantaneous joint position, velocity, and effort | ## Navigation Messages | Message | ROS 2 Package | Description | |:---|:---|:---| | **Odometry** | nav_msgs | Robot position and velocity | | **Path** | nav_msgs | Array of poses for navigation | | **MapMetaData** | nav_msgs | Map resolution, size, origin | | **OccupancyGrid** | nav_msgs | 2D grid map with occupancy probabilities | ## Intelligence Messages These types are defined by EmbodiedAgents for AI component communication. | Message | ROS 2 Package | Description | |:---|:---|:---| | **StreamingString** | automatika_embodied_agents | String chunk for streaming applications (e.g., LLM tokens) | | **Video** | automatika_embodied_agents | A sequence of image frames | | **Detections** | automatika_embodied_agents | 2D bounding boxes with labels and confidence scores | | **DetectionsMultiSource** | automatika_embodied_agents | Detections from multiple input sources | | **PointsOfInterest** | automatika_embodied_agents | Specific 2D coordinates of interest within an image | | **Trackings** | automatika_embodied_agents | Object tracking data including IDs, labels, and trajectories | | **TrackingsMultiSource** | automatika_embodied_agents | Object tracking data from multiple sources | ## Navigation-Specific Messages These types are defined by Kompass for navigation component communication. | Message | ROS 2 Package | Description | |:---|:---|:---| | **TwistArray** | kompass_interfaces | Array of velocity commands for trajectory candidates | ## Hardware Interface Messages | Message | ROS 2 Package | Description | |:---|:---|:---| | **RGBD** | realsense2_camera_msgs | Synchronized RGB and Depth image pair | | **JointTrajectoryPoint** | trajectory_msgs | Position, velocity, and acceleration for joints at a specific time | | **JointTrajectory** | trajectory_msgs | A sequence of waypoints for joint control | | **JointJog** | control_msgs | Immediate displacement or velocity commands for joints | ``` ## File: advanced/algorithms.md ```markdown # Navigation Algorithms Kompass, the EMOS navigation engine, provides a comprehensive suite of algorithms for both **global path planning** and **local motion control**. - {material-regular}`route;1.2em;sd-text-primary` **[Planning Algorithms](#planning-algorithms-ompl)** -- Over 25 sampling-based planners from OMPL (RRT*, PRM, KPIECE, etc.) for global path planning with collision checking. - {material-regular}`gamepad;1.2em;sd-text-primary` **[Control Algorithms](#control-algorithms)** -- Battle-tested controllers ranging from classic geometric path-followers to GPU-accelerated local planners and visual servoing. Every algorithm is natively compatible with the three primary motion models. The internal logic automatically adapts to the specific constraints of your platform: - {material-regular}`directions_car;1.2em;sd-text-primary` **ACKERMANN**: Car-like platforms with steering constraints. - {material-regular}`swap_horiz;1.2em;sd-text-primary` **DIFFERENTIAL_DRIVE**: Two-wheeled or skid-steer robots. - {material-regular}`open_with;1.2em;sd-text-primary` **OMNI**: Holonomic systems capable of lateral movement. Each algorithm is fully parameterized. Developers can tune behaviors such as lookahead gains, safety margins, and obstacle sensitivity directly through the Python API or YAML configuration. --- (control-algorithms)= ## Control Algorithms | Algorithm | Type | Key Feature | Sensors Required | | :--- | :--- | :--- | :--- | | [DWA](#dynamic-window-approach-dwa) | Sampling-based planner | GPU-accelerated velocity space planning | LaserScan, PointCloud, OccupancyGrid | | [Pure Pursuit](#pure-pursuit) | Geometric tracker | Lookahead-based path tracking with collision avoidance | LaserScan, PointCloud, OccupancyGrid (optional) | | [Stanley Steering](#stanley-steering) | Geometric tracker | Front-axle feedback for Ackermann platforms | None (pure path follower) | | [DVZ](#deformable-virtual-zone-dvz) | Reactive controller | Deformable safety bubble for fast avoidance | LaserScan | | [Vision Follower (RGB)](#vision-follower-rgb) | Visual servoing | 2D target centering with monocular camera | Detections / Trackings, RGB Image | | [Vision Follower (RGB-D)](#vision-follower-rgb-d) | Visual servoing + planner | Depth-aware following | Detections, Depth Image | | [Trajectory Cost Evaluation](#trajectory-cost-evaluation) | Cost functions | Weighted scoring for sampling-based controllers | -- | --- ## Dynamic Window Approach (DWA) **GPU-accelerated Dynamic Window Approach.** DWA is a classic local planning method developed in the 90s.[^dwa] It is a sampling-based controller that generates a set of constant-velocity trajectories within a "Dynamic Window" of reachable velocities. EMOS supercharges this algorithm using **SYCL-based hardware acceleration**, allowing it to sample and evaluate thousands of candidate trajectories in parallel on **Nvidia, AMD, or Intel** GPUs. This enables high-frequency control loops even in complex, dynamic environments with dense obstacle fields. It is highly effective for differential drive and omnidirectional robots. ### How It Works The algorithm operates in a three-step pipeline at every control cycle: 1. **Compute Dynamic Window.** Calculate the range of reachable linear and angular velocities ($v, \omega$) for the next time step, limited by the robot's maximum acceleration and current speed. 2. **Sample Trajectories.** Generate a set of candidate trajectories by sampling velocity pairs within the dynamic window and simulating the robot's motion forward in time. 3. **Score and Select.** Discard trajectories that collide with obstacles (using **FCL**). Score the remaining valid paths based on distance to goal, path alignment, and smoothness. ### Supported Sensory Inputs DWA requires spatial data to perform collision checking during the rollout phase. - LaserScan - PointCloud - OccupancyGrid ### Parameters and Default Values ```{list-table} :widths: 10 10 10 70 :header-rows: 1 * - Name - Type - Default - Description * - control_time_step - `float` - `0.1` - Time interval between control actions (sec). Must be between `1e-4` and `1e6`. * - prediction_horizon - `float` - `1.0` - Duration over which predictions are made (sec). Must be between `1e-4` and `1e6`. * - control_horizon - `float` - `0.2` - Duration over which control actions are planned (sec). Must be between `1e-4` and `1e6`. * - max_linear_samples - `int` - `20` - Maximum number of linear control samples. Must be between `1` and `1e3`. * - max_angular_samples - `int` - `20` - Maximum number of angular control samples. Must be between `1` and `1e3`. * - sensor_position_to_robot - `List[float]` - `[0.0, 0.0, 0.0]` - Position of the sensor relative to the robot in 3D space (x, y, z) coordinates. * - sensor_rotation_to_robot - `List[float]` - `[0.0, 0.0, 0.0, 1.0]` - Orientation of the sensor relative to the robot as a quaternion (x, y, z, w). * - octree_resolution - `float` - `0.1` - Resolution of the Octree used for collision checking. Must be between `1e-9` and `1e3`. * - costs_weights - `TrajectoryCostsWeights` - see [defaults](#configuration-weights) - Weights for trajectory cost evaluation. * - max_num_threads - `int` - `1` - Maximum number of threads used when running the controller. Must be between `1` and `1e2`. ``` ```{note} All previous parameters can be configured when using the DWA algorithm directly in your Python recipe or using a config file (as shown in the usage example). ``` ### Usage Example DWA can be activated by setting the `algorithm` property in the Controller configuration. ```{code-block} python :caption: dwa.py from kompass.components import Controller, ControllerConfig from kompass.robot import ( AngularCtrlLimits, LinearCtrlLimits, RobotCtrlLimits, RobotGeometry, RobotType, RobotConfig ) from kompass.control import ControllersID # Setup your robot configuration my_robot = RobotConfig( model_type=RobotType.ACKERMANN, geometry_type=RobotGeometry.Type.BOX, geometry_params=np.array([0.3, 0.3, 0.3]), ctrl_vx_limits=LinearCtrlLimits(max_vel=0.2, max_acc=1.5, max_decel=2.5), ctrl_omega_limits=AngularCtrlLimits( max_vel=0.4, max_acc=2.0, max_decel=2.0, max_steer=np.pi / 3) ) # Set DWA algorithm using the config class controller_config = ControllerConfig(algorithm="DWA") # Set YAML config file config_file = "my_config.yaml" controller = Controller(component_name="my_controller", config=controller_config, config_file=config_file) # algorithm can also be set using a property controller.algorithm = ControllersID.DWA # or "DWA" ``` ```{code-block} yaml :caption: my_config.yaml my_controller: # Component config parameters loop_rate: 10.0 control_time_step: 0.1 prediction_horizon: 4.0 ctrl_publish_type: 'Array' # Algorithm parameters under the algorithm name DWA: control_horizon: 0.6 octree_resolution: 0.1 max_linear_samples: 20 max_angular_samples: 20 max_num_threads: 3 costs_weights: goal_distance_weight: 1.0 reference_path_distance_weight: 1.5 obstacles_distance_weight: 2.0 smoothness_weight: 1.0 jerk_weight: 0.0 ``` ### Trajectory Samples Generation Trajectory samples are generated using a constant velocity generator for each velocity value within the reachable range to generate the configured maximum number of samples (see `max_linear_samples` and `max_angular_samples` in the config parameters). The shape of the sampled trajectories depends heavily on the robot's kinematic model: ::::{tab-set} :::{tab-item} Ackermann :sync: ackermann **Car-Like Motion** Note the limited curvature constraints typical of car-like steering. ::: :::{tab-item} Differential :sync: diff **Tank/Diff Drive** Includes rotation-in-place (if configured) and smooth arcs. ::: :::{tab-item} Omni :sync: omni **Holonomic Motion** Includes lateral (sideways) movement samples. ::: :::: :::{admonition} Rotate-Then-Move :class: note To ensure natural movement for Differential and Omni robots, EMOS implements a **Rotate-Then-Move** policy. Simultaneous rotation and high-speed linear translation is restricted to prevent erratic behavior. ::: ### Best Trajectory Selection A collision-free admissibility criteria is implemented within the trajectory samples generator using FCL to check the collision between the simulated robot state and the reference sensor input. Once admissible trajectories are sampled, the **Best Trajectory** is selected by minimizing a weighted cost function. You can tune these weights (`costs_weights`) to change the robot's behavior (e.g., sticking closer to the path vs. prioritizing obstacle clearance). See [Trajectory Cost Evaluation](#trajectory-cost-evaluation) for details. [^dwa]: [Dieter Fox, Wolf Burgard and Sebastian Thrun. The Dynamic Window Approach to Collision Avoidance. IEEE Robotics & Automation Magazine (Volume: 4, Issue: 1, March 1997)](https://www.ri.cmu.edu/pub_files/pub1/fox_dieter_1997_1/fox_dieter_1997_1.pdf) --- ## Pure Pursuit **Geometric path tracking with reactive collision avoidance.** Pure Pursuit is a fundamental path-tracking algorithm. It calculates the curvature required to move the robot from its current position to a specific "lookahead" point on the path, simulating how a human driver looks forward to steer a vehicle. EMOS enhances the standard implementation (based on [Purdue SIGBOTS](https://wiki.purduesigbots.com/software/control-algorithms/basic-pure-pursuit)) by adding an integrated **Simple Search Collision Avoidance** layer. This allows the robot to deviate locally from the path to avoid unexpected obstacles without needing a full replan. ### How It Works The controller executes a four-step cycle: 1. **Find Target -- Locate Lookahead.** Find the point on the path that is distance $L$ away from the robot. $L$ scales with speed ($L = k \cdot v$). 2. **Steering -- Compute Curvature.** Calculate the arc required to reach that target point based on the robot's kinematic constraints. 3. **Safety -- Collision Check.** Project the robot's motion forward using the `prediction_horizon` to check for immediate collisions. 4. **Avoidance -- Local Search.** If the nominal arc is blocked, the controller searches through `max_search_candidates` to find a safe velocity offset that clears the obstacle while maintaining progress. ### Supported Sensory Inputs To enable the collision avoidance layer, spatial data is required. - LaserScan - PointCloud - OccupancyGrid *(Note: The controller can run in "blind" tracking mode without these inputs, but collision avoidance will be disabled.)* ### Configuration Parameters ```{list-table} :widths: 15 10 10 65 :header-rows: 1 * - Name - Type - Default - Description * - lookahead_gain_forward - `float` - `0.8` - Factor to scale lookahead distance by current velocity ($L = k \cdot v$). * - prediction_horizon - `int` - `10` - Number of future steps used to check for potential collisions along the path. * - path_search_step - `float` - `0.2` - Offset step used to search for alternative velocity commands when the nominal path is blocked. * - max_search_candidates - `int` - `10` - Maximum number of search iterations to find a collision-free command. ``` ### Usage Example ```{code-block} python :caption: pure_pursuit.py from kompass.components import Controller, ControllerConfig from kompass.robot import ( AngularCtrlLimits, LinearCtrlLimits, RobotCtrlLimits, RobotGeometry, RobotType, RobotConfig ) from kompass.control import ControllersID, PurePursuitConfig # Setup your robot configuration my_robot = RobotConfig( model_type=RobotType.OMNI, geometry_type=RobotGeometry.Type.BOX, geometry_params=np.array([0.3, 0.3, 0.3]), ctrl_vx_limits=LinearCtrlLimits(max_vel=0.2, max_acc=1.5, max_decel=2.5), ctrl_omega_limits=AngularCtrlLimits( max_vel=0.4, max_acc=2.0, max_decel=2.0, max_steer=np.pi / 3) ) # Initialize the controller controller = Controller(component_name="my_controller") # Set the algorithm configuration pure_pursuit_config = PurePursuitConfig( lookahead_gain_forward=0.5, prediction_horizon=8, max_search_candidates=20 ) controller.algorithms_config = pure_pursuit_config # NOTE: You can configure more than one algorithm to switch during runtime # other_algorithm_config = .... # controller.algorithms_config = [pure_pursuit_config, other_algorithm_config] # Set the algorithm to Pure Pursuit controller.algorithm = ControllersID.PURE_PURSUIT ``` ### Performance and Results The following tests demonstrate the controller's ability to track reference paths (**thin dark blue**) and avoid obstacles (**red x**). **Nominal Tracking** -- Performance on standard geometric paths (U-Turns and Circles) without interference: ::::{grid} 1 3 3 3 :gutter: 2 :::{grid-item-card} Ackermann **U-Turn** ::: :::{grid-item-card} Differential **Circle** ::: :::{grid-item-card} Omni **Circle** ::: :::: **Collision Avoidance** -- Scenarios where static obstacles are placed directly on the global path. The controller successfully identifies the blockage and finds a safe path around it: ::::{grid} 1 3 3 3 :gutter: 2 :::{grid-item-card} Ackermann **Straight + Obstacles** ::: :::{grid-item-card} Differential **U-Turn + Obstacles** ::: :::{grid-item-card} Omni **Straight + Obstacles** ::: :::: :::{admonition} Observations :class: note * **Convergence:** Smooth convergence to the reference path across all kinematic models. * **Clearance:** The simple search algorithm successfully clears obstacle boundaries before returning to the path. * **Stability:** No significant oscillation observed during avoidance maneuvers. ::: --- ## Stanley Steering **Front-wheel feedback control for path tracking.** Stanley is a geometric path tracking method originally developed for the DARPA Grand Challenge.[^stanley] Unlike Pure Pursuit (which looks ahead), Stanley uses the **Front Axle** as its reference point to calculate steering commands. It computes a steering angle $\delta(t)$ based on two error terms: 1. **Heading Error** ($\psi_e$): Difference between the robot's heading and the path direction. 2. **Cross-Track Error** ($e$): Lateral distance from the front axle to the nearest path segment. The control law combines these to minimize error exponentially: $$ \delta(t) = \psi_e(t) + \arctan \left( \frac{k \cdot e(t)}{v(t)} \right) $$ ### Key Features - **Ackermann Native** -- Designed specifically for car-like steering geometry. Naturally stable at high speeds for these vehicles. - **Multi-Model Support** -- EMOS extends Stanley to Differential and Omni robots by applying a **Rotate-Then-Move** strategy when orientation errors are large. - **Sensor-Less** -- Does not require LiDAR or depth data. It is a pure path follower. ### Configuration Parameters ```{list-table} :widths: 10 10 10 70 :header-rows: 1 * - Name - Type - Default - Description * - heading_gain - `float` - `0.7` - Heading gain in the control law. Must be between `0.0` and `1e2`. * - cross_track_min_linear_vel - `float` - `0.05` - Minimum linear velocity for cross-track control (m/s). Must be between `1e-4` and `1e2`. * - min_angular_vel - `float` - `0.01` - Minimum allowable angular velocity (rad/s). Must be between `0.0` and `1e9`. * - cross_track_gain - `float` - `1.5` - Gain for cross-track in the control law. Must be between `0.0` and `1e2`. * - max_angle_error - `float` - `np.pi / 16` - Maximum allowable angular error (rad). Must be between `1e-9` and `pi`. * - max_distance_error - `float` - `0.1` - Maximum allowable distance error (m). Must be between `1e-9` and `1e9`. ``` ### Usage Example ```{code-block} python :caption: stanley.py from kompass.components import Controller, ControllerConfig from kompass.robot import ( AngularCtrlLimits, LinearCtrlLimits, RobotCtrlLimits, RobotGeometry, RobotType, RobotConfig ) from kompass.control import ControllersID # Setup your robot configuration my_robot = RobotConfig( model_type=RobotType.ACKERMANN, geometry_type=RobotGeometry.Type.BOX, geometry_params=np.array([0.3, 0.3, 0.3]), ctrl_vx_limits=LinearCtrlLimits(max_vel=0.2, max_acc=1.5, max_decel=2.5), ctrl_omega_limits=AngularCtrlLimits( max_vel=0.4, max_acc=2.0, max_decel=2.0, max_steer=np.pi / 3) ) # Set Stanley algorithm using the config class controller_config = ControllerConfig(algorithm="Stanley") # or ControllersID.STANLEY # Set YAML config file config_file = "my_config.yaml" controller = Controller(component_name="my_controller", config=controller_config, config_file=config_file) # algorithm can also be set using a property controller.algorithm = ControllersID.STANLEY # or "Stanley" ``` ```{code-block} yaml :caption: my_config.yaml my_controller: # Component config parameters loop_rate: 10.0 control_time_step: 0.1 ctrl_publish_type: 'Sequence' # Algorithm parameters under the algorithm name Stanley: cross_track_gain: 1.0 heading_gain: 2.0 ``` :::{admonition} Safety Note :class: warning Stanley does **not** have built-in obstacle avoidance. It is strongly recommended to use this controller in conjunction with the **Drive Manager** component to provide Emergency Stop and Slowdown safety layers. ::: [^stanley]: [Hoffmann, Gabriel M., Claire J. Tomlin, Michael Montemerlo, and Sebastian Thrun. "Autonomous Automobile Trajectory Tracking for Off-Road Driving: Controller Design, Experimental Validation and Racing." American Control Conference. 2007, pp. 2296-2301](https://ieeexplore.ieee.org/document/4282788) --- ## Deformable Virtual Zone (DVZ) **Fast, reactive collision avoidance for dynamic environments.** The DVZ (Deformable Virtual Zone) is a reactive control method introduced by R. Zapata in 1994.[^dvz] It models the robot's safety perimeter as a "virtual bubble" (zone) that deforms when obstacles intrude. Unlike sampling methods (like DWA) that simulate future trajectories, DVZ calculates a reaction vector based directly on how the bubble is being "squished" by the environment. This makes it extremely computationally efficient and ideal for crowded, fast-changing environments where rapid reactivity is more important than global optimality. ### How It Works The algorithm continuously computes a deformation vector to steer the robot away from intrusion. 1. **Define Zone -- Create Bubble.** Define a circular (or elliptical) protection zone around the robot with a nominal undeformed radius $R$. 2. **Sense -- Measure Intrusion.** Using LaserScan data, compute the *deformed radius* $d_h(\alpha)$ for every angle $\alpha \in [0, 2\pi]$ around the robot. 3. **Compute Deformation -- Calculate Metrics.** * **Intrusion Intensity ($I_D$):** How much total "stuff" is inside the zone. $I_D = \frac{1}{2\pi} \int_{0}^{2\pi}\frac{R - d_h(\alpha)}{R} d\alpha$ * **Deformation Angle ($\Theta_D$):** The primary direction of the intrusion. $\Theta_D = \frac{\int_{0}^{2\pi} (R - d_h(\alpha))\alpha d\alpha}{I_D}$ 4. **React -- Control Law.** The final control command minimizes $I_D$ (pushing away from the deformation) while trying to maintain the robot's original heading towards the goal. ### Supported Sensory Inputs DVZ relies on dense 2D range data to compute the deformation integral. - LaserScan ### Configuration Parameters DVZ balances two competing forces: **Path Following** (Geometric) vs. **Obstacle Repulsion** (Reactive). ```{list-table} :widths: 10 10 10 70 :header-rows: 1 * - Name - Type - Default - Description * - min_front_margin - `float` - `1.0` - Minimum front margin distance. Must be between `0.0` and `1e2`. * - K_linear - `float` - `1.0` - Proportional gain for linear control. Must be between `0.1` and `10.0`. * - K_angular - `float` - `1.0` - Proportional gain for angular control. Must be between `0.1` and `10.0`. * - K_I - `float` - `5.0` - Proportional deformation gain. Must be between `0.1` and `10.0`. * - side_margin_width_ratio - `float` - `1.0` - Width ratio between the deformation zone front and side (circle if 1.0). Must be between `1e-2` and `1e2`. * - heading_gain - `float` - `0.7` - Heading gain of the internal pure follower control law. Must be between `0.0` and `1e2`. * - cross_track_gain - `float` - `1.5` - Gain for cross-track error of the internal pure follower control law. Must be between `0.0` and `1e2`. ``` ### Usage Example ```{code-block} python :caption: dvz.py from kompass.components import Controller, ControllerConfig from kompass.robot import ( AngularCtrlLimits, LinearCtrlLimits, RobotCtrlLimits, RobotGeometry, RobotType, RobotConfig ) from kompass.control import LocalPlannersID # Setup your robot configuration my_robot = RobotConfig( model_type=RobotType.ACKERMANN, geometry_type=RobotGeometry.Type.BOX, geometry_params=np.array([0.3, 0.3, 0.3]), ctrl_vx_limits=LinearCtrlLimits(max_vel=0.2, max_acc=1.5, max_decel=2.5), ctrl_omega_limits=AngularCtrlLimits( max_vel=0.4, max_acc=2.0, max_decel=2.0, max_steer=np.pi / 3) ) # Set DVZ algorithm using the config class controller_config = ControllerConfig(algorithm="DVZ") # or LocalPlannersID.DVZ # Set YAML config file config_file = "my_config.yaml" controller = Controller(component_name="my_controller", config=controller_config, config_file=config_file) # algorithm can also be set using a property controller.algorithm = ControllersID.DVZ # or "DVZ" ``` ```{code-block} yaml :caption: my_config.yaml my_controller: # Component config parameters loop_rate: 10.0 control_time_step: 0.1 ctrl_publish_type: 'Sequence' # Algorithm parameters under the algorithm name DVZ: cross_track_gain: 1.0 heading_gain: 2.0 K_angular: 1.0 K_linear: 1.0 min_front_margin: 1.0 side_margin_width_ratio: 1.0 ``` [^dvz]: [Zapata, R., Lepinay, P., and Thompson, P. "Reactive behaviors of fast mobile robots". In: Journal of Robotic Systems 11.1 (1994)](https://www.researchgate.net/publication/221787033_Reactive_Motion_Planning_for_Mobile_Robots) --- ## Vision Follower (RGB) **2D Visual Servoing for target centering.** The VisionFollowerRGB is a reactive controller designed to keep a visual target (like a person or another robot) centered within the camera frame. Unlike the RGB-D variant, this controller operates purely on 2D image coordinates, making it compatible with any standard monocular camera. It calculates velocity commands based on the **relative shift** and **apparent size** of a 2D bounding box. ### How It Works The controller uses a proportional control law to minimize the error between the target's current position in the image and the desired center point. - **Horizontal Centering -- Rotation.** The robot rotates to minimize the horizontal offset of the target bounding box relative to the image center. - **Scale Maintenance -- Linear Velocity.** The robot moves forward or backward to maintain a consistent bounding box size, effectively keeping a fixed relative distance without explicit depth data. - **Target Recovery -- Search Behavior.** If the target is lost, the controller can initiate a search pattern (rotation in place) to re-acquire the target bounding box. ### Supported Inputs This controller requires 2D detection or tracking data. - Detections / Trackings (must provide Detections2D or Trackings2D) :::{admonition} Data Synchronization :class: note The Controller does not subscribe directly to raw images. It expects the detection metadata (bounding boxes) to be provided by an external vision pipeline. ::: ### Configuration Parameters ```{list-table} :widths: 20 15 15 50 :header-rows: 1 * - Name - Type - Default - Description * - **rotation_gain** - `float` - `1.0` - Proportional gain for angular control (centering the target). * - **speed_gain** - `float` - `0.7` - Proportional gain for linear speed (maintaining distance). * - **tolerance** - `float` - `0.1` - Error margin for tracking before commands are issued. * - **target_search_timeout** - `float` - `30.0` - Maximum duration (seconds) to perform search before timing out. * - **enable_search** - `bool` - `True` - Whether to rotate the robot to find a target if it exits the FOV. * - **min_vel** - `float` - `0.1` - Minimum linear velocity allowed during following. ``` ### Usage Example ```python import numpy as np from kompass.control import VisionRGBFollowerConfig # Configure the algorithm config = VisionRGBFollowerConfig( rotation_gain=0.9, speed_gain=0.8, enable_search=True ) ``` --- ## Vision Follower (RGB-D) **Depth-aware target tracking.** The VisionFollowerRGBD is a sophisticated 3D visual servoing controller. It combines 2D object detections with depth information to estimate the precise 3D position and velocity of a target. ### Key Features - **3D Projection -- Depth Fusion.** Projects 2D bounding boxes into 3D space using the depth image and camera intrinsics. - **Relative Positioning** -- Maintain a specific distance and bearing relative to the target. - **Velocity Tracking** -- Capable of estimating target velocity to provide smoother, more predictive following. - **Recovery Behaviors** -- Includes configurable **Wait** and **Search** (rotating in place) logic for when the target is temporarily occluded or leaves the field of view. ### Supported Inputs This controller requires synchronized vision and spatial data. - Detections -- 2D bounding boxes (Detections2D, Trackings2D). - Depth Image Information -- Aligned depth image info for 3D coordinate estimation. ### Configuration Parameters The RGB-D follower inherits all parameters from DWA and adds vision-specific settings. ```{list-table} :widths: 20 15 15 50 :header-rows: 1 * - Name - Type - Default - Description * - **target_distance** - `float` - `None` - The desired distance (m) to maintain from the target. * - **target_orientation** - `float` - `0.0` - The desired bearing angle (rad) relative to the target. * - **prediction_horizon** - `int` - `10` - Number of future steps to project for collision checking. * - **target_search_timeout** - `float` - `30.0` - Max time to search for a lost target before giving up. * - **depth_conversion_factor** - `float` - `1e-3` - Factor to convert raw depth values to meters (e.g., $0.001$ for mm). * - **camera_position_to_robot** - `np.array` - `[0,0,0]` - 3D translation vector $(x, y, z)$ from camera to robot base. ``` ### Usage Example ```python from kompass.control import VisionRGBDFollowerConfig config = VisionRGBDFollowerConfig( target_distance=1.5, target_orientation=0.0, enable_search=True, ) ``` --- ## Trajectory Cost Evaluation **Scoring candidate paths for optimal selection.** In sampling-based controllers like DWA, dozens of candidate trajectories are generated at every time step. To choose the best one, EMOS uses a weighted sum of several cost functions. The total cost $J$ for a given trajectory is calculated as: $$ J = \sum (w_i \cdot C_i) $$ Where $w_i$ is the configured weight and $C_i$ is the normalized cost value. ### Hardware Acceleration To handle high-frequency control loops with large sample sets, EMOS leverages **SYCL** for massive parallelism. Each cost function is implemented as a specialized **SYCL kernel**, allowing the controller to evaluate thousands of trajectory points in parallel on **Nvidia, AMD, or Intel** GPUs, significantly reducing latency compared to CPU-only implementations. See the performance gains in the [Benchmarks](./benchmarks.md) page. ### Built-in Cost Functions | Cost Component | Description | Goal | | :--- | :--- | :--- | | **Reference Path** | Average distance between the candidate trajectory and the global reference path. | **Stay on track.** Keep the robot from drifting away from the global plan. | | **Goal Destination** | Euclidean distance from the end of the trajectory to the final goal point. | **Make progress.** Favor trajectories that actually move the robot closer to the destination. | | **Obstacle Distance** | Inverse of the minimum distance to the nearest obstacle (from LaserScan/PointCloud). | **Stay safe.** Heavily penalize trajectories that come too close to walls or objects. | | **Smoothness** | Average change in velocity (acceleration) along the trajectory. | **Drive smoothly.** Prevent jerky velocity changes. | | **Jerk** | Average change in acceleration along the trajectory. | **Protect hardware.** Minimize mechanical stress and wheel slip. | ### Configuration Weights You can tune the behavior of the robot by adjusting the weights ($w_i$) in your configuration. ```{list-table} :widths: 10 10 10 70 :header-rows: 1 * - Name - Type - Default - Description * - reference_path_distance_weight - `float` - `3.0` - Weight of the reference path cost. Must be between `0.0` and `1e3`. * - goal_distance_weight - `float` - `3.0` - Weight of the goal position cost. Must be between `0.0` and `1e3`. * - obstacles_distance_weight - `float` - `1.0` - Weight of the obstacles distance cost. Must be between `0.0` and `1e3`. * - smoothness_weight - `float` - `0.0` - Weight of the trajectory smoothness cost. Must be between `0.0` and `1e3`. * - jerk_weight - `float` - `0.0` - Weight of the trajectory jerk cost. Must be between `0.0` and `1e3`. ``` :::{tip} Setting a weight to `0.0` completely disables that specific cost calculation kernel, saving computational resources. ::: --- (planning-algorithms-ompl)= ## Planning Algorithms (OMPL) EMOS integrates the **[Open Motion Planning Library (OMPL)](https://ompl.kavrakilab.org/)** for global path planning. OMPL is a generic C++ library for state-of-the-art sampling-based motion planning algorithms. EMOS provides Python bindings (via Pybind11) for OMPL through its navigation core package. The bindings enable setting and solving a planning problem using: - **SE2StateSpace** -- Convenient for 2D motion planning, providing an SE2 state consisting of position and rotation in the plane: `SE(2): (x, y, yaw)` - **Geometric planners** -- All planners listed below - **Built-in StateValidityChecker** -- Implements collision checking using [FCL](https://github.com/flexible-collision-library/fcl) to ensure collision-free paths ### Configuring OMPL ```yaml ompl: log_level: 'WARN' planning_timeout: 10.0 # (secs) Fail if solving takes longer simplification_timeout: 0.01 # (secs) Abort path simplification if too slow goal_tolerance: 0.01 # (meters) Distance to consider goal reached optimization_objective: 'PathLengthOptimizationObjective' planner_id: 'ompl.geometric.KPIECE1' ``` ### Available OMPL Planners The following 29 geometric planners are supported: - [ABITstar](#abitstar) - [AITstar](#aitstar) - [BFMT](#bfmt) - [BITstar](#bitstar) - [BKPIECE1](#bkpiece1) - [BiEST](#biest) - [EST](#est) - [FMT](#fmt) - [InformedRRTstar](#informedrrtstar) - [KPIECE1](#kpiece1) - [LBKPIECE1](#lbkpiece1) - [LBTRRT](#lbtrrt) - [LazyLBTRRT](#lazylbtrrt) - [LazyPRM](#lazyprm) - [LazyPRMstar](#lazyprmstar) - [LazyRRT](#lazyrrt) - [PDST](#pdst) - [PRM](#prm) - [PRMstar](#prmstar) - [ProjEST](#projest) - [RRT](#rrt) - [RRTConnect](#rrtconnect) - [RRTXstatic](#rrtxstatic) - [RRTsharp](#rrtsharp) - [RRTstar](#rrtstar) - [SBL](#sbl) - [SST](#sst) - [STRIDE](#stride) - [TRRT](#trrt) ### Planner Benchmark Results A planning problem was simulated using the Turtlebot3 Gazebo Waffle map. Each planner was tested over 20 repetitions with a 2-second solution search timeout. The table shows average results. | Method | Solved | Solution Time (s) | Solution Length (m) | Simplification Time (s) | |:---|:---|:---|:---|:---| | ABITstar | True | 1.071 | 2.948 | 0.0075 | | BFMT | True | 0.113 | 3.487 | 0.0066 | | BITstar | True | 1.073 | 2.962 | 0.0061 | | BKPIECE1 | True | 0.070 | 4.469 | 0.0178 | | BiEST | True | 0.062 | 4.418 | 0.0108 | | EST | True | 0.064 | 4.059 | 0.0107 | | FMT | True | 0.133 | 3.628 | 0.0063 | | InformedRRTstar | True | 1.068 | 2.962 | 0.0046 | | KPIECE1 | True | 0.068 | 5.439 | 0.0148 | | LBKPIECE1 | True | 0.075 | 5.174 | 0.0200 | | LBTRRT | True | 1.070 | 3.221 | 0.0050 | | LazyLBTRRT | True | 1.067 | 3.305 | 0.0053 | | LazyPRM | False | 1.081 | -- | -- | | LazyPRMstar | True | 1.070 | 3.030 | 0.0063 | | LazyRRT | True | 0.098 | 4.520 | 0.0160 | | PDST | True | 0.068 | 3.836 | 0.0090 | | PRM | True | 1.067 | 3.306 | 0.0068 | | PRMstar | True | 1.074 | 3.720 | 0.0085 | | ProjEST | True | 0.068 | 4.190 | 0.0082 | | RRT | True | 0.091 | 4.860 | 0.0190 | | RRTConnect | True | 0.075 | 4.780 | 0.0140 | | RRTXstatic | True | 1.071 | 3.030 | 0.0041 | | RRTsharp | True | 1.068 | 3.010 | 0.0052 | | RRTstar | True | 1.067 | 2.960 | 0.0042 | | SBL | True | 0.080 | 4.039 | 0.0121 | | SST | True | 1.068 | 2.630 | 0.0012 | | STRIDE | True | 0.068 | 4.120 | 0.0098 | | TRRT | True | 0.080 | 4.110 | 0.0109 | ### Planner Default Parameters #### ABITstar - delay_rewiring_to_first_solution: False - drop_unconnected_samples_on_prune: False - find_approximate_solutions: False - inflation_scaling_parameter: 10.0 - initial_inflation_factor: 1000000.0 - prune_threshold_as_fractional_cost_change: 0.05 - rewire_factor: 1.1 - samples_per_batch: 100 - stop_on_each_solution_improvement: False - truncation_scaling_parameter: 5.0 - use_graph_pruning: True - use_just_in_time_sampling: False - use_k_nearest: True - use_strict_queue_ordering: True #### AITstar - find_approximate_solutions: True - rewire_factor: 1.0 - samples_per_batch: 100 - use_graph_pruning: True - use_k_nearest: True #### BFMT - balanced: False - cache_cc: True - extended_fmt: True - heuristics: True - nearest_k: True - num_samples: 1000 - optimality: True - radius_multiplier: 1.0 #### BITstar - delay_rewiring_to_first_solution: False - drop_unconnected_samples_on_prune: False - find_approximate_solutions: False - prune_threshold_as_fractional_cost_change: 0.05 - rewire_factor: 1.1 - samples_per_batch: 100 - stop_on_each_solution_improvement: False - use_graph_pruning: True - use_just_in_time_sampling: False - use_k_nearest: True - use_strict_queue_ordering: True #### BKPIECE1 - border_fraction: 0.9 - range: 0.0 #### BiEST - range: 0.0 #### EST - goal_bias: 0.5 - range: 0.0 #### FMT - cache_cc: True - extended_fmt: True - heuristics: False - num_samples: 1000 - radius_multiplier: 1.1 - use_k_nearest: True #### InformedRRTstar - delay_collision_checking: True - goal_bias: 0.05 - number_sampling_attempts: 100 - ordered_sampling: False - ordering_batch_size: 1 - prune_threshold: 0.05 - range: 0.0 - rewire_factor: 1.1 - use_k_nearest: True #### KPIECE1 - border_fraction: 0.9 - goal_bias: 0.05 - range: 0.0 #### LBKPIECE1 - border_fraction: 0.9 - range: 0.0 #### LBTRRT - epsilon: 0.4 - goal_bias: 0.05 - range: 0.0 #### LazyLBTRRT - epsilon: 0.4 - goal_bias: 0.05 - range: 0.0 #### LazyPRM - max_nearest_neighbors: 8 - range: 0.0 #### LazyPRMstar No configurable parameters. #### LazyRRT - goal_bias: 0.05 - range: 0.0 #### PDST - goal_bias: 0.05 #### PRM - max_nearest_neighbors: 8 #### PRMstar No configurable parameters. #### ProjEST - goal_bias: 0.05 - range: 0.0 #### RRT - goal_bias: 0.05 - intermediate_states: False - range: 0.0 #### RRTConnect - intermediate_states: False - range: 0.0 #### RRTXstatic - epsilon: 0.0 - goal_bias: 0.05 - informed_sampling: False - number_sampling_attempts: 100 - range: 0.0 - rejection_variant: 0 - rejection_variant_alpha: 1.0 - rewire_factor: 1.1 - sample_rejection: False - update_children: True - use_k_nearest: True #### RRTsharp - goal_bias: 0.05 - informed_sampling: False - number_sampling_attempts: 100 - range: 0.0 - rejection_variant: 0 - rejection_variant_alpha: 1.0 - rewire_factor: 1.1 - sample_rejection: False - update_children: True - use_k_nearest: True #### RRTstar - delay_collision_checking: True - focus_search: False - goal_bias: 0.05 - informed_sampling: True - new_state_rejection: False - number_sampling_attempts: 100 - ordered_sampling: False - ordering_batch_size: 1 - prune_threshold: 0.05 - pruned_measure: False - range: 0.0 - rewire_factor: 1.1 - sample_rejection: False - tree_pruning: False - use_admissible_heuristic: True - use_k_nearest: True #### SBL - range: 0.0 #### SST - goal_bias: 0.05 - pruning_radius: 3.0 - range: 5.0 - selection_radius: 5.0 #### STRIDE - degree: 16 - estimated_dimension: 3.0 - goal_bias: 0.05 - max_degree: 18 - max_pts_per_leaf: 6 - min_degree: 12 - min_valid_path_fraction: 0.2 - range: 0.0 - use_projected_distance: False #### TRRT - goal_bias: 0.05 - range: 0.0 - temp_change_factor: 0.1 ```