Vibecheck Uses AI Questions to Verify Developers Understand Their Pull Requests

Summary

Vibecheck is a self-hosted merge gate for GitHub pull requests, designed for teams concerned that developers may submit AI-generated code they do not understand. It runs as a Dockerized FastAPI service with a Postgres database and uses a bring-your-own-key Anthropic Claude integration. A workflow sends each new pull request to the service, which reads the diff, creates a fixed high-level question plus scoped questions tied to parts of the change, and posts a link for the author. The author answers one question at a time in plain English, while an AI grader compares each response with its own interpretation of the diff. Teams can configure how much detail is needed to pass; correct answers can pass, incorrect answers can be retried, and the session fails when the required number of marks is no longer reachable. The tool reports a verdict commit status that can be required by GitHub branch protection, keeping the pull request blocked until the author passes. Question volume is sized to the change, at roughly one question per 15 changed lines up to a configured limit, and oversized pull requests or diffs are rejected before questioning. Vibecheck does not review code for correctness; it checks whether the author understands it. Its local-first design requires only a workflow file and a bearer token, with no GitHub App, OAuth, or webhook installation, and is intended for maintainers, engineering teams, open-source projects, and developers using AI coding tools.