How to Evaluate Application Responses with GPT-6 Sol in AI SDK

Summary

Vercel documents how GPT-6 Sol can evaluate application outputs through AI SDK's experimental evaluation API. An application supplies shared evidence and named questions, and the OpenAI evaluation adapter uses the Responses API with structured output to return choice, score, or Boolean answers. Choice questions select one configured option, score questions return a possibly fractional position on an ordered rubric beginning at zero, and Boolean questions return an estimated probability that a statement is true. The Boolean estimate is not a measured accuracy or a universal confidence score, while choice and score answers do not include probability distributions. The example evaluates a customer-support reply using a policy, customer message, and draft response. It separately asks whether the reply is ready, how fully it addresses the request, and whether it makes an unsupported factual claim, illustrating why coverage alone cannot determine approval. The article recommends keeping evidence sources distinct, showing supporting fields to reviewers, and routing revise or needs_review results differently. It also says the model evaluates only supplied state and does not retrieve missing evidence. The TypeScript setup installs `ai` and `@ai-sdk/openai`, configures `OPENAI_API_KEY`, and calls `openai.evaluationModel('gpt-6-sol')` with a reasoning effort such as medium. The evaluation API is experimental, so question changes and SDK updates should be checked against reviewed examples. GPT-6 Astra and GPT-6 Luna can use the same adapter when supported, but Astra requires an explicit supported reasoning level and Luna supports none. Failed, refused, truncated, or invalid evaluations should be recorded separately from a completed needs_review judgment. Vercel recommends testing models and rubrics against reviewer-labeled examples, tracking false approvals, unnecessary revisions, duration, and token use, and testing application behavior for both results and failures. The article also notes that the examples demonstrate evaluation rather than benchmark GPT-6 Sol itself.