How Latitude Monitors and Evaluates Fleets of AI Agents

Summary

Latitude presents a practical monitoring workflow for consultancies operating AI agents for multiple clients. The guide uses a Hermes example with five client deployments, two development projects, and 845 conversations collected over six weeks. Each agent is assigned a Latitude project so its conversations, costs, telemetry, and signals remain together, while Latitude's MCP lets a coding agent query evidence across projects and generate fleet or client reports. Automated flaggers make an initial pass: deterministic checks identify issues such as empty responses, tool errors, malformed JSON, and low cache usage, while sampled LLM checks look for problems such as frustration, lost context, incomplete work, unnecessary refusals, or claims that a failed tool call succeeded. Client feedback from external channels such as WhatsApp, Telegram, and Slack must be transferred manually as scores or custom signals; Latitude does not automatically read every messaging channel. Signals group related findings with their conversations, but the guide stresses that people must inspect traces before accepting a signal's diagnosis and decide whether the remedy is a tool change, prompt change, or client-process change. Slack notifications can route high-severity findings to the team, while MCP-generated reports can separate client deployments from development traffic and distinguish recorded model costs from the consultancy's broader service costs. In the example, a fleet report covered 845 conversations, a 4.66% tool-call error rate, and 15 open signals. A release comparison for a support agent showed tool-call errors falling from 11.23% in v1.6.0 to 2.63% in v1.7.0 after a tool-description change, but the author notes that the live-traffic comparison was not a controlled A/B test and that other signal-affected conversations increased. Deployment-specific failures can become ongoing evaluations after a human annotates at least one trace, with sampling and representative pass cases used to assess detector quality. As the fleet grows from five agents to fifty or more, the process still requires human ownership, review, and incident handling, but it can reduce routine transcript reading, dashboard checking, complaint copying, and report assembly.