38 Lessons PostHog Learned from Building AI Products
Summary
PostHog says it has built 10 AI products over four years, including PostHog AI, an installation wizard, Replay Vision, and Scouts, with nearly 100,000 monthly users across them. The company argues that AI should be justified by a real job to be done rather than by novelty: onboarding, repeated work, unfamiliar interfaces, and tasks lasting more than 30 seconds are stronger targets than generic document chat. Its installation wizard reduced roughly two hours of work to about eight minutes; users who adopted it converted to paid plans five times more often and activated twice as fast, while the MCP server is used daily by five times as many people as PostHog AI. PostHog recommends designing the agent-first version of a product, watching customers who rebuild workflows with AI, and treating context, product data, tools, and skills as the moat rather than any particular model harness. It favors shared infrastructure for prompts, streaming, consent, evals, tasks, and analytics, and says agents should be able to execute code in sandboxes and complete the same operations users can perform. The company replaced narrowly predefined query tools with a general SQL execution tool, while its agents can create context through initialization commands and semantic definitions instead of relying on oversized prompts. Other lessons include tracing inputs, outputs, latency, and cost; measuring MCP calls and page requests; starting evals from manually reviewed traces; and prioritizing reliability, clear limits, and uncertainty over raw capability. PostHog reports that more context improved its wizard more than several reasoning and planning changes, that subagents increased cost and sometimes latency, and that changing models forced major product restructuring. Finally, it says AI shifts interfaces toward approving, reviewing, undoing, and triaging agent work, while human confirmation, protection against prompt injection, visible execution steps, and purpose-built verification UIs remain important; many of these conclusions may change within a year.