SemiAnalysis Estimates Google’s Ironwood TPU Leads Nvidia B200 and B300 in Inference Cost Efficiency

Summary

A third-party SemiAnalysis estimate compares Google’s seventh-generation Ironwood TPU with Nvidia’s B200 and B300 under matched model workloads. At a benchmark of 100 output tokens per second for one user, the TPU is reported to cost 19% less than B200 and 34% less than B300; the broader estimate puts its per-million-token cost at $0.181, versus $0.222 for B200 and $0.276 for B300. The comparison is not an across-the-board performance win: B200 retains a narrow advantage in a limited 30-second response-time range, while TPU’s lower rental cost drives its stronger cost efficiency across most of the tested range. The article attributes the result to Google’s system-level design, including Ironwood’s larger HBM capacity, native FP8 support, TPU interconnects, and software optimizations for KV-cache layout, decoding, and communication workloads. It also describes TorchTPU, which uses PyTorch’s PrivateUse1 backend to make TPU a more native PyTorch device and allows frameworks such as vLLM and SGLang to reuse more of their existing scheduling and API code. The article argues that this reduces the software barrier historically associated with CUDA, while noting that the measurements come from a third-party analysis and represent specific workloads and cost assumptions. It also reports that Anthropic has committed to more than one million TPUs and that Google has begun offering direct chip sales, but does not establish that TPU has displaced Nvidia across the wider market.