OpenAI Jalapeño: Better than Nvidia Blackwell

7.4 relevance

OpenAI's custom inference chip directly impacts AI agent cost and performance, a core interest.

AI/ML newsletter.semianalysis.com

Answer the board on AI readiness

Arrive with evidence, not opinion. See how your foundation benchmarks against your peers, get a clear read, and leave with a ranked place to start.

Why It Matters

For a solutions architect focused on AI infrastructure and platform engineering, this signals a paradigm shift: custom ASICs can now outperform general-purpose GPUs on inference, which will reshape cloud deployment strategies, cost modeling, and hardware procurement decisions for LLM workloads.

Key Takeaways

Evaluate custom inference ASICs like Jalapeño for latency-sensitive and high-throughput LLM serving, as they can deliver 2-3x better perf/W than Blackwell without speculative decoding or disaggregation.

OpenAI Jalapeño: Better than Nvidia Blackwell

Summary

OpenAI's custom inference ASIC 'Jalapeño,' built with Broadcom in just 16 months, beats Nvidia Blackwell and AMD chips on throughput per watt across all tested open-source models, including DeepSeek R1 and Kimi-K2. The chip uses HBM4, achieves over 700 tokens/sec/user at concurrency 1 with single-token prediction, and is a general-purpose inference accelerator—not specialized for OpenAI's models. Performance was verified in-lab using the InferenceX benchmark suite, though AgentX results are pending.

Author

Bryan Shan — HPC