VoiceGremlin Automates Testing for AI Phone Agents

Summary

VoiceGremlin is a SaaS tool for automated tests against AI systems that answer phone calls. Its API accepts a phone number and a test case, such as verifying that an agent performs a specified action, then places a call and has bots converse with the target voice agent. An LLM-as-judge component produces a pass/fail result that can be used in a development pipeline. The product is fully working and offers a free trial without requiring a card. Its creator describes the goal as becoming “Mailosaur for phone agents,” because arranging automated calls is technically and organizationally difficult. The implementation had to address telephony-provider onboarding, bandwidth when using separate speech-to-text and text-to-speech services, prompt tuning to reduce flaky LLM-to-LLM tests, and fraud or abuse prevention. Development took about 45 days, split between business setup and engineering experiments involving telephony providers, features, models, prompts, and safeguards. VoiceGremlin is deliberately limited in scope: it does not manage test cases, host customers’ voice agents, or add upsell features. The stated stack includes Supabase, Netlify, DigitalOcean, Telnyx, Stripe, OpenRouter, Cloudflare, and Keybase.