Porting Gemma-4 (2B / 4B / 12B) to AWS Inferentia2
7.7 relevance
Score Breakdown
technical depth 9
novelty 7
actionability 8
community 5
strategic 6
personal 9
Scored daily by a customisable AI persona to surface the most relevant engineering leadership news.
Detailed field report on porting Gemma-4 to AWS Inferentia2 highly actionable
Summary
Porting Gemma-4 (E2B, E4B, 12B) to AWS Inferentia2 (inf2.xlarge/8xlarge) required overcoming mixed attention heads, cross-layer KV-sharing, and vendor stack dead-ends (optimum-neuron, neuronx-distributed, Neuron vLLM). By tracing the Hugging Face forward pass directly, the author achieved correct inference at 44 tok/s (E2B), 33-39 tok/s (E4B), and 15 tok/s (12B) — token-for-token identical to CPU reference — using Neuron SDK 2.23 and torch-neuronx 2.8.0. The fix bypasses the vendor's graph builder, which cannot express KV-sharing or mixed attention types.