Using the Testing Framework with Realtime Models

The LiveKit testing framework allows you to test your agents with both standard LLM models and realtime models. This guide covers how to properly set up and use the testing framework with realtime models.

Setting Up Realtime Model Tests

To test an agent with a realtime model, use the following pattern:

async with (
    openai.realtime.RealtimeModel(modalities=["text"]) as rt_llm,
    AgentSession(llm=rt_llm, userdata=userdata) as sess,
):
    await sess.start(YourAgent(userdata=userdata))
    result = await sess.run(user_input="...")

This pattern uses Python's async context manager to properly initialize and clean up both the realtime model and the agent session.

Important Notes

When testing with realtime models, keep these key points in mind:

• Use text modalities — When using realtime models, make sure to specify modalities=["text"] as the testing helpers are designed to work with text input and output.
• Context manager required — The realtime model must be used within an agent session context manager to ensure proper resource cleanup.
• Suppressing warnings — If you're seeing deprecation or resource warnings in Python 3.12+, you can suppress them with:

import warnings

warnings.filterwarnings("ignore", category=DeprecationWarning)
warnings.filterwarnings("ignore", category=ResourceWarning)

Basic Test Example

Here's a complete example of testing an agent with a realtime model:

import pytest
from livekit.agents import AgentSession
from livekit.plugins import openai

from your_agent import YourAgent, UserData

@pytest.mark.asyncio
async def test_agent_greeting():
    userdata = UserData()

    async with (
        openai.realtime.RealtimeModel(modalities=["text"]) as rt_llm,
        AgentSession(llm=rt_llm, userdata=userdata) as sess,
    ):
        await sess.start(YourAgent(userdata=userdata))
        result = await sess.run(user_input="Hello!")

        await result.expect.next_event().is_message(role="assistant").judge(
            rt_llm, intent="Provides a friendly greeting"
        )

Validating Function Calls

You can test that your agent correctly calls function tools:

@pytest.mark.asyncio
async def test_function_call():
    userdata = UserData()

    async with (
        openai.realtime.RealtimeModel(modalities=["text"]) as rt_llm,
        AgentSession(llm=rt_llm, userdata=userdata) as sess,
    ):
        await sess.start(YourAgent(userdata=userdata))
        result = await sess.run(user_input="Book an appointment for tomorrow")

        # Verify the function is called with expected arguments
        result.expect.next_event().is_function_call(
            name="book_appointment",
            arguments={"date": "tomorrow"}
        )

        # Verify function output is processed
        result.expect.next_event().is_function_call_output()

Testing Conversation Context

To test that your agent maintains conversation context across multiple turns:

@pytest.mark.asyncio
async def test_conversation_context():
    userdata = UserData()

    async with (
        openai.realtime.RealtimeModel(modalities=["text"]) as rt_llm,
        AgentSession(llm=rt_llm, userdata=userdata) as sess,
    ):
        await sess.start(YourAgent(userdata=userdata))

        # First turn - establish context
        result1 = await sess.run(user_input="My name is Alice")

        # Second turn - verify context is maintained
        result2 = await sess.run(user_input="What's my name?")

        await result2.expect.next_event().is_message(role="assistant").judge(
            rt_llm, intent="Correctly recalls that the user's name is Alice"
        )

Mocking Function Tools

For isolated unit tests, you can mock function tools to control their behavior:

from livekit.agents.voice.run_result import mock_tools

@pytest.mark.asyncio
async def test_with_mocked_tools():
    userdata = UserData()

    async with (
        openai.realtime.RealtimeModel(modalities=["text"]) as rt_llm,
        AgentSession(llm=rt_llm, userdata=userdata) as sess,
    ):
        await sess.start(YourAgent(userdata=userdata))

        # Mock the function to return a specific value
        with mock_tools(YourAgent, {
            "check_availability": lambda date: "Available at 2pm and 4pm"
        }):
            result = await sess.run(user_input="Check availability for Monday")

            result.expect.next_event().is_function_call(name="check_availability")
            result.expect.next_event().is_function_call_output(
                output="Available at 2pm and 4pm"
            )

Example Test Implementations

For complete reference implementations demonstrating these patterns in real-world agents, check out:

• Comprehensive Agent Testing — Complete test suite with fixtures, mocks, and conversation flows
• Doheny Surf Desk Agent — Multi-agent booking system with extensive testing patterns

Summary

Additional Resources

For more examples and advanced testing patterns, refer to our voice agents examples repository.