Conversational Agent

Often times robots are equipped with a speaker system and a microphone. Once these peripherals have been exposed through ROS, we can use EMOS to trivially create a conversational interface on the robot. Our conversational agent will use a multimodal LLM for contextual question/answering utilizing the camera onboard the robot. Furthermore, it will use speech-to-text and text-to-speech models for converting audio to text and vice versa. We will start by importing the relevant components that we want to string together.

from agents.components import VLM, SpeechToText, TextToSpeech

Components are basic functional units in EMOS. Their inputs and outputs are defined using ROS Topics. And their function can be any input transformation, for example the inference of an ML model. Lets setup these components one by one. Since our input to the robot would be speech, we will setup the speech-to-text component first.

SpeechToText Component

This component listens to an audio input topic, that takes in a multibyte array of audio (captured in a ROS std_msgs message, which maps to Audio msg_type in EMOS) and can publish output to a text topic. It can also be configured to get the audio stream from microphones on board our robot. By default the component is configured to use a small Voice Activity Detection (VAD) model, Silero-VAD to filter out any audio that is not speech.

However, merely utilizing speech can be problematic in robots, due to the hands free nature of the audio system. Therefore its useful to add wakeword detection, so that speech-to-text is only activated when the robot is called with a specific phrase (such as ‘ok robot’).

We will be using this configuration in our example. First we will setup our input and output topics and then create a config object which we can later pass to our component.

Note

With enable_vad set to True, the component automatically downloads and deploys Silero-VAD by default in ONNX format. This model has a small footprint and can be easily deployed on the edge. However we need to install a couple of dependencies for this to work. These can be installed with: pip install pyaudio onnxruntime

Note

With enable_wakeword set to True, the component downloads a small keyword-spotting model from sherpa-onnx and listens for the phrase given in wakeword_phrase, which is ‘ok robot’ unless you set another one, or a list of them. Nothing is trained: any phrase works with the same model, and models for other languages are published with sherpa-onnx’s keyword-spotting releases. The spotter needs pip install sherpa-onnx sentencepiece.

from agents.ros import Topic
from agents.config import SpeechToTextConfig

# Define input and output topics (pay attention to msg_type)
audio_in = Topic(name="audio0", msg_type="Audio")
text_query = Topic(name="text0", msg_type="String")

s2t_config = SpeechToTextConfig(enable_vad=True,     # option to listen for speech through the microphone, set to False if using web UI
                                enable_wakeword=True) # option to invoke the component with a wakeword, 'ok robot' by default, set to False if using web UI

Warning

The enable_wakeword option cannot be enabled without the enable_vad option.

See also

Check the available defaults and options for the SpeechToTextConfig in the API reference.

To initialize the component we also need a model client for a speech to text model. We will be using the WebSocket client for RoboML for this purpose.

Note

RoboML is an aggregator library that provides a model serving apparatus for locally serving opensource ML models useful in robotics. Learn about setting up RoboML here.

Additionally, we will use the client with a model called Whisper, a popular opensource speech to text model from OpenAI. Lets see what that looks like in code.

from agents.clients import RoboMLWSClient
from agents.models import Whisper

# Setup the model client
whisper = Whisper(name="whisper")  # Custom model init params can be provided here
roboml_whisper = RoboMLWSClient(whisper)

# Initialize the component
speech_to_text = SpeechToText(
    inputs=[audio_in],  # the input topic we setup
    outputs=[text_query], # the output topic we setup
    model_client=roboml_whisper,
    trigger=audio_in,
    config=s2t_config,  # pass in the config object
    component_name="speech_to_text"
)

The trigger parameter lets the component know that it has to perform its function (in this case model inference) when an input is received on this particular topic. In our configuration, the component will be triggered using voice activity detection on the continuous stream of audio being received on the microphone. Next we will setup our VLM component.

VLM Component

The VLM component takes as input a text topic (the output of the SpeechToText component) and an image topic, assuming we have a camera device onboard the robot publishing this topic. And just like before we need to provide a model client, this time with a VLM model. This time we will use the OllamaClient along with qwen2.5vl:latest model, an opensource multimodal LLM from the Qwen family, available on Ollama. Furthermore, we will configure our VLM component using VLMConfig. We will set stream=True to make the VLM output text be published as a stream for downstream components that consume this output. In EMOS, streaming output can be chunked using a break_character in the config (Default: ‘.’). This way the downstream TextToSpeech component can start generating audio as soon as the first sentence is produced by the LLM.

We also set chat_history=True. By default a component does not remember the conversation. Every request it sends to the model contains only the system prompt and the current input. With chat_history on, the component also sends the conversation so far, so the user can ask a follow-up question. After “what is on the table?” they can ask “what colour is it?” and the model knows what “it” refers to.

The conversation is not kept forever. The component keeps the last ten turns, which can be changed with history_size. It also makes sure that the whole request fits in the model’s context window. When the model runs on Ollama, the client knows the size of the window and the component works out its budget from it. Pictures are sent only with the turn they belong to. An earlier turn keeps a note saying that it had a picture, but the picture itself is not sent again. If the robot should be able to answer questions about something it saw a few turns ago, set history_images to the number of recent turns whose pictures should be sent along. Turns that are dropped from the conversation are forgotten. If you would rather keep a summary of them, set history_compaction="summarize". The component then asks the model to summarize the dropped turns and keeps that summary at the start of the conversation.

To start a new conversation, say “chat reset”. The phrase can be changed with history_reset_phrase. It is matched without case and without trailing punctuation, so the “Chat reset.” that comes out of speech recognition works too. The system prompt stays in place. A recipe can also do this from an event or a routine by calling the component’s reset_history action.

Note

Ollama is one of the most popular local LLM serving projects. Learn about setting up Ollama here.

Here is the code for our VLM setup.

from agents.clients.ollama import OllamaClient
from agents.models import OllamaModel
from agents.config import VLMConfig

# Define the image input topic and a new text output topic
image0 = Topic(name="image_raw", msg_type="Image")
text_answer = Topic(name="text1", msg_type="String")

# Define a model client (working with Ollama in this case)
# OllamaModel is a generic wrapper for all ollama models
qwen_vl = OllamaModel(name="qwen_vl", checkpoint="qwen2.5vl:latest")
qwen_client = OllamaClient(qwen_vl)

mllm_config = VLMConfig(stream=True, chat_history=True)  # Other inference specific parameters can be provided here

# Define an VLM component
mllm = VLM(
    inputs=[text_query, image0],  # Notice the text input is the same as the output of the previous component
    outputs=[text_answer],
    model_client=qwen_client,
    config=mllm_config,
    trigger=text_query,
    component_name="vqa" # We have also given our component an optional name
)

We can further customize our VLM component by attaching a context prompt template. This can be done at the component level or at the level of a particular input topic. In this case we will attach a prompt template to the input topic text_query.

# Attach a prompt template
mllm.set_topic_prompt(text_query, template="""You are an amazing and funny robot.
Answer the following about this image: {{ text0 }}"""
)

Notice that the template is a jinja2 template string, where the actual name of the topic is set as a variable. For longer templates you can also write them to a file and provide its path when calling this function. After this we move on to setting up our last component.

TextToSpeech Component

The TextToSpeech component setup will be very similar to the SpeechToText component. We will once again use a RoboML client, this time with the unified TransformersTTS wrapper. RoboML serves any HuggingFace Transformers TTS model – VITS, Bark, SpeechT5, SeamlessM4T, etc. – through a single class; here we keep the default checkpoint, Facebook’s MMS-TTS-eng (VITS), which is fast and lightweight enough to run comfortably on-device. The component can be configured to play audio on a playback device available onboard the robot, which we will utilize through our config. An output topic is optional for this component as we will be playing the audio directly on device.

Note

In order to utilize play_on_device you need to install a couple of dependencies as follows: pip install soundfile pyaudio

from agents.config import TextToSpeechConfig
from agents.models import TransformersTTS

# config for asynchronously playing audio on device
t2s_config = TextToSpeechConfig(play_on_device=True, stream=True)  # Set play_on_device to false if using the web UI

# Uncomment the following line for receiving output on the web UI
# audio_out = Topic(name="audio_out", msg_type="Audio")

vits = TransformersTTS(name="vits")  # default checkpoint: facebook/mms-tts-eng
roboml_vits = RoboMLWSClient(vits)
text_to_speech = TextToSpeech(
    inputs=[text_answer],
    outputs=[],  # use outputs=[audio_out] for receiving answers on web UI
    trigger=text_answer,
    model_client=roboml_vits,
    config=t2s_config,
    component_name="text_to_speech"
)

Launching the Components

The final step in this example is to launch the components. This is done by passing the defined components to the launcher and calling the bringup method. EMOS also allows us to create a web-based UI for interacting with our conversational agent recipe.

from agents.ros import Launcher

# Launch the components
launcher = Launcher()
launcher.enable_ui(inputs=[audio_in, text_query], outputs=[image0])  # specify topics
launcher.add_pkg(
    components=[speech_to_text, mllm, text_to_speech]
    )
launcher.bringup()

Et voila! We have setup a graph of three components in less than 50 lines of well formatted code. The complete example is as follows:

Multimodal Audio Conversational Agent

from agents.components import VLM, SpeechToText, TextToSpeech
from agents.config import SpeechToTextConfig, TextToSpeechConfig, VLMConfig
from agents.clients import OllamaClient, RoboMLWSClient
from agents.models import Whisper, TransformersTTS, OllamaModel
from agents.ros import Topic, Launcher

audio_in = Topic(name="audio0", msg_type="Audio")
text_query = Topic(name="text0", msg_type="String")

whisper = Whisper(name="whisper")  # Custom model init params can be provided here
roboml_whisper = RoboMLWSClient(whisper)

s2t_config = SpeechToTextConfig(enable_vad=True,     # option to listen for speech through the microphone, set to False if using web UI
                                enable_wakeword=True) # option to invoke the component with a wakeword, 'ok robot' by default, set to False if using web UI

speech_to_text = SpeechToText(
    inputs=[audio_in],
    outputs=[text_query],
    model_client=roboml_whisper,
    trigger=audio_in,
    config=s2t_config,
    component_name="speech_to_text",
)

image0 = Topic(name="image_raw", msg_type="Image")
text_answer = Topic(name="text1", msg_type="String")

qwen_vl = OllamaModel(name="qwen_vl", checkpoint="qwen2.5vl:latest")
qwen_client = OllamaClient(qwen_vl)
mllm_config = VLMConfig(stream=True, chat_history=True)  # Other inference specific parameters can be provided here

mllm = VLM(
    inputs=[text_query, image0],
    outputs=[text_answer],
    model_client=qwen_client,
    trigger=text_query,
    config=mllm_config,
    component_name="vqa",
)

t2s_config = TextToSpeechConfig(play_on_device=True, stream=True)  # Set play_on_device to false if using the web UI

# Uncomment the following line for receiving output on the web UI
# audio_out = Topic(name="audio_out", msg_type="Audio")

vits = TransformersTTS(name="vits")  # default checkpoint: facebook/mms-tts-eng
roboml_vits = RoboMLWSClient(vits)
text_to_speech = TextToSpeech(
    inputs=[text_answer],
    outputs=[],  # use outputs=[audio_out] for receiving answers on web UI
    trigger=text_answer,
    model_client=roboml_vits,
    config=t2s_config,
    component_name="text_to_speech"
)

launcher = Launcher()
launcher.enable_ui(inputs=[audio_in, text_query], outputs=[image0])  # specify topics
launcher.add_pkg(components=[speech_to_text, mllm, text_to_speech])
launcher.bringup()

Web Based UI for Interacting with the Robot

To interact with topics on the robot, EMOS can create dynamically specified UIs. This is useful if the robot does not have a microphone/speaker interface or if one wants to communicate with it remotely. We will also like to see the images coming in from the robots camera to have more context of its answers.

In the code above, we already specified the input and output topics for the UI by calling the function launcher.enable_ui. Furthermore, we can set enable_vad and enable_wakeword options in s2t_config to False and set play_on_device option in t2s_config to False. Now we are ready to use our browser based UI.

Note

An EMOS install already has what the browser front end needs. Elsewhere, install FastHTML and MonsterUI with

pip install python-fasthtml monsterui

The client serves the web UI on https://localhost:5001 if you have run it on your machine, or on https://<IP_ADDRESS_OF_THE_ROBOT>:5001 if you have run it on the robot. The certificate is self-signed, so the browser asks you to accept it once.

---

Tip

Promote this recipe to production. While you are shaping it, run the script directly with python recipe.py. Once it is solid, drop it at ~/emos/recipes/<name>/recipe.py and start it with emos run <name>, or from the dashboard. Either way every run is logged under ~/emos/logs, and an operator gets a card to launch it from a browser. Running Recipes covers the two ways of running a recipe and what differs per install mode.