Porting a 128-expert MoE (Gemma-4 26B-A4B) to AWS Inferentia2 — where every rank weighted the wrong experts
Scored daily by a customisable AI persona to surface the most relevant engineering leadership news.
Deep technical dive into MoE porting on AWS Inferentia with a tricky bug.
Porting Gemma-4 26B-A4B's 128-expert MoE to AWS Inferentia2 required a workaround for its sparse top-8 routing: computing all 128 experts per token and masking non-selected ones with zero weights, collapsing the expert loop into two standard parallel linear layers (ColumnParallel and RowParallel) that Neuron's ModelBuilder can shard. The architecture also includes a shared dense MLP in parallel with the MoE per layer, and despite passing unit tests with MAXDIFF ≈ 2e-6, the first device run yielded empty output due to a bug in an unexpected location. After fixing, the model achieved SEQ_MATCH on greedy decode with 77 ms prefill.