Self-Healing with Fallbacks¶

In the real world, connections drop, APIs time out, solvers fail to converge, and serial cables vibrate loose. A “Production Ready” agent cannot simply freeze when something goes wrong.

EMOS provides a unified fallback API that works identically across the intelligence layer (model clients) and the navigation layer (algorithms and hardware). In this recipe, we demonstrate both.

Intelligence Layer: Model Fallback¶

We build an agent that uses a high-intelligence model (hosted remotely) as its primary brain, but automatically switches to a smaller, local model if the primary one fails.

The Strategy: Plan A and Plan B¶

Plan A (Primary): Use a powerful model hosted via RoboML (or a cloud provider) for high-quality reasoning.
Plan B (Backup): Keep a smaller, quantized model (like Llama 3.2 3B) loaded locally via Ollama.
The Trigger: If the Primary model fails to respond (latency, disconnection, or server error), automatically swap the component’s internal client to the Backup.

1. Defining the Models¶

First, we need to define our two distinct model clients.

from agents.components import LLM
from agents.models import OllamaModel, TransformersLLM
from agents.clients import OllamaClient, RoboMLHTTPClient
from agents.config import LLMConfig
from agents.ros import Launcher, Topic, Action

# --- Plan A: The Powerhouse ---
# A powerful model hosted remotely (e.g., via RoboML).
# NOTE: This is illustrative for executing on a local machine.
# For a production scenario, you might use a GenericHTTPClient pointing to
# GPT-5, Gemini, HuggingFace Inference etc.
primary_model = TransformersLLM(
    name="qwen_heavy",
    checkpoint="Qwen/Qwen2.5-1.5B-Instruct"
)
primary_client = RoboMLHTTPClient(model=primary_model)

# --- Plan B: The Safety Net ---
# A smaller model running locally (via Ollama) that works offline.
backup_model = OllamaModel(name="llama_local", checkpoint="llama3.2:3b")
backup_client = OllamaClient(model=backup_model)

2. Configuring the Component¶

Next, we set up the standard LLM component. We initialize it using the primary_client.

However, the magic happens in the additional_model_clients attribute. This dictionary allows the component to hold references to other valid clients that are waiting in the wings.

# Define Topics
user_query = Topic(name="user_query", msg_type="String")
llm_response = Topic(name="llm_response", msg_type="String")

# Configure the LLM Component with the PRIMARY client initially
llm_component = LLM(
    inputs=[user_query],
    outputs=[llm_response],
    model_client=primary_client,
    component_name="brain",
    config=LLMConfig(stream=True),
)

# Register the Backup Client
# We store the backup client in the component's internal registry.
# We will use the key 'local_backup_client' to refer to this later.
llm_component.additional_model_clients = {"local_backup_client": backup_client}

3. Creating the Fallback Action¶

Now we need an Action. In EMOS, components have built-in methods to reconfigure themselves. The LLM component (like all other components that take a model client) has a method called change_model_client.

We wrap this method in an Action so it can be triggered by an event.

Note

All components implement some default actions as well as component specific actions. In this case we are implementing a component specific action.

4. Wiring Failure to Action¶

Finally, we tell the component when to execute this action. We don’t need to write complex try/except blocks in our business logic. Instead, we attach the action to the component’s lifecycle hooks:

on_component_fail: Triggered if the component crashes or fails to initialize (e.g., the remote server is down when the robot starts).
on_algorithm_fail: Triggered if the component is running, but the inference fails (e.g., the WiFi drops mid-conversation).

# Bind Failures to the Action
# If the component fails (startup) or the algorithm crashes (runtime),
# it will attempt to switch clients.
llm_component.on_component_fail(action=switch_to_backup, max_retries=3)
llm_component.on_algorithm_fail(action=switch_to_backup, max_retries=3)

Note

Why max_retries? Sometimes a fallback can temporarily fail as well. The system will attempt to restart the component or algorithm up to 3 times while applying the action (switching the client) to resolve the error. This is an optional parameter.

The Complete Intelligence Fallback Recipe¶

Here is the full code. To test this, you can try shutting down your RoboML server (or disconnecting the internet) while the agent is running, and watch it seamlessly switch to the local Llama model.

from agents.components import LLM
from agents.models import OllamaModel, TransformersLLM
from agents.clients import OllamaClient, RoboMLHTTPClient
from agents.config import LLMConfig
from agents.ros import Launcher, Topic, Action

# 1. Define the Models and Clients
# Primary: A powerful model hosted remotely
primary_model = TransformersLLM(
    name="qwen_heavy", checkpoint="Qwen/Qwen2.5-1.5B-Instruct"
)
primary_client = RoboMLHTTPClient(model=primary_model)

# Backup: A smaller model running locally
backup_model = OllamaModel(name="llama_local", checkpoint="llama3.2:3b")
backup_client = OllamaClient(model=backup_model)

# 2. Define Topics
user_query = Topic(name="user_query", msg_type="String")
llm_response = Topic(name="llm_response", msg_type="String")

# 3. Configure the LLM Component
llm_component = LLM(
    inputs=[user_query],
    outputs=[llm_response],
    model_client=primary_client,
    component_name="brain",
    config=LLMConfig(stream=True),
)

# 4. Register the Backup Client
llm_component.additional_model_clients = {"local_backup_client": backup_client}

# 5. Define the Fallback Action
switch_to_backup = Action(
    method=llm_component.change_model_client,
    args=("local_backup_client",)
)

# 6. Bind Failures to the Action
llm_component.on_component_fail(action=switch_to_backup, max_retries=3)
llm_component.on_algorithm_fail(action=switch_to_backup, max_retries=3)

# 7. Launch
launcher = Launcher()
launcher.add_pkg(
    components=[llm_component],
    multiprocessing=True,
    package_name="automatika_embodied_agents",
)
launcher.bringup()

Navigation Layer: Algorithm & System Fallback¶

Navigation components face a different class of failures: optimization solvers that fail to converge, serial cables that vibrate loose, and robots that get boxed into corners. The same on_*_fail API handles all of these.

Algorithm Failure: Switch Controllers¶

If the primary high-performance algorithm (e.g., DWA) fails, we can switch to a simpler “safety” algorithm (like PurePursuit).

from kompass.components import Controller, DriveManager
from kompass.control import ControllersID
from kompass.ros import Action

# Select the primary control algorithm
controller = Controller(component_name="controller")
controller.algorithm = ControllersID.DWA

# Define the fallback: switch to PurePursuit
switch_algorithm_action = Action(
    method=controller.set_algorithm,
    args=(ControllersID.PURE_PURSUIT,)
)

# Fallback sequence: restart first, then switch algorithm if it fails again
controller.on_algorithm_fail(
    action=[Action(controller.restart), switch_algorithm_action],
    max_retries=1
)

System Failure: Restart Hardware Connection¶

The DriveManager talks directly to low-level hardware (micro-controller/motor drivers). Transient failures – loose USB cables, electromagnetic interference, watchdog trips – are common and often resolved by a simple restart.

driver = DriveManager(component_name="drive_manager")

# Restart on system failure (unlimited retries for transient hardware glitches)
driver.on_system_fail(Action(driver.restart))

Global Catch-All¶

For any component that doesn’t have specific fallback logic, the Launcher provides a blanket policy.

launcher.on_fail(action_name="restart")
launcher.fallback_rate = 1 / 10  # 0.1 Hz (one retry every 10 seconds)

The Same API, Both Layers¶

The key insight is that the same three hooks work everywhere in EMOS:

Hook	Triggers When	Intelligence Example	Navigation Example
`on_component_fail`	Component crashes or fails to initialize	Remote model server is down	Serial port unavailable
`on_algorithm_fail`	Inference or computation fails at runtime	WiFi drops mid-conversation	DWA solver can’t converge
`on_system_fail`	External dependency is lost	API key revoked	Motor controller resets

Each hook accepts an Action (or list of actions) and an optional max_retries parameter. This consistency means you can apply the same resilience patterns regardless of which layer you’re working in.