All products

Evaluate voice agents with AI

Build task-based tests that distinguish what the agent understood, what the caller heard and whether an external action really succeeded.

To evaluate voice agents with AI, define what would count as success for the caller. A convincing confirmation does not establish that an appointment moved. The voice-agent-evaluation skill separates conversational quality from playback timing and verified tool effects, assigning the right evidence to each claim.

A compact scenario set can expose consequential mistakes hidden by an average score. A lost “not”, a corrected date or an interruption during an operation may matter more than several harmless transcription errors. The workflow also distinguishes a mocked service response from a verified end-to-end outcome.

What it does

  • Create scenarios for normal calls, corrections and interruptions.
  • Test task-critical words, names, numbers and dates explicitly.
  • Compare generated replies with audio actually delivered.
  • Verify a tool outcome independently of the agent’s statement.
  • Record the test conditions, judging criteria and unresolved failures.

Install and use

  1. Install the skill and provide the intended call flow and permitted test evidence. Full-system checks need an authorised test instance and connected tools. Use synthetic or redacted examples where practical.
  2. Ask, “Evaluate our appointment change flow. Show separate evidence for the recognised date, the spoken reply and the actual booking update.”
  3. Inspect disputed results manually and identify which claims still need a live integration test.

FAQ

Source

Made by Rob Strayer. Original skill: voice-agent-evaluation, published on GitHub under the MIT licence.

View on GitHub