Intel, Hugging Face report 1.7x TCO gain for GPT OSS on Google Cloud C4
Hugging Face
- Benchmarks
- Infrastructure
- #gpt-oss
- #xeon
- #google cloud
Browse Benchmarks topic page →
Intel and Hugging Face published a benchmark claiming Google Cloud C4 VMs on Intel Xeon 6 deliver 1.4x–1.7x per-vCPU throughput over C3 for the GPT OSS 120B Mixture-of-Experts model, plus a fused transformers optimization routing tokens only to selected experts. The 1.7x TCO figure is a ratio derived from throughput and per-vCPU price parity, not an independent measurement.
What the collaboration demonstrated
Intel and Hugging Face describe a joint benchmark of the GPT OSS 120B Mixture-of-Experts model (unsloth/gpt-oss-120b-BF16) on Google Cloud C4 VMs with Intel Xeon 6 processors versus prior-generation C3 VMs on 4th Gen Xeon. The workload fixes input and output at 1024 tokens and sweeps batch sizes from 1 to 64, using static KV cache and SDPA attention. The authors report 1.4x to 1.7x normalized throughput per vCPU across those batch sizes and, at batch 64, a 1.7x TCO ratio they derive from the per-vCPU throughput ratio and roughly equal price per vCPU. They also merged a transformers change routing each MoE expert only to the tokens it receives, so experts no longer process every token.
Application scenario for an inference platform engineer
A team already serving a large MoE model on CPU VMs and weighing a region or fleet refresh could use this post as a prompt to re-run its own decoding benchmark on candidate VM types rather than extrapolating the vendor ratio. Because the reported gain is per-vCPU and headline TCO depends on batch size, an engineer would need to check where their live traffic batches actually land before assuming the 1.7x applies. This remains a hypothesis to test, not a guarantee of savings.
Trial steps and measurement
A team could replicate the described setup: build the transformers commit referenced in the post, run the model in bfloat16 with left-padded fixed-length batches, a warm-up round, and generation_config set for deterministic decoding, then compute throughput as output tokens times batch size over total latency. Record per-vCPU throughput on C3 and C4 at the batch sizes matching production. Compare against an internal baseline, watch whether the expert-routing change alone shifts results, and treat any deviation from the published range as a reason to pause rather than adjust conclusions to match the post.
Trade-offs and unknowns
The headline number is vendor-reported and based on one model, one fixed workload and a short excerpt, so it should not be treated as a general property of C4 or MoE serving. Price parity is assumed rather than shown, and real bills include storage, networking and idle time omitted here. Batch 64 may be impractical for latency-sensitive traffic. Teams should not use this to size a migration without their own measurements, and should treat it as a starting signal, not a procurement justification.
Context and limitations
- Findings come from a short excerpt of a vendor-authored post; the full article and any peer review were not analyzed.
- The benchmark is vendor-published by Intel and Hugging Face, not an independent test, and uses one fixed workload and one model.
- C3 and C4 instances differ in vCPU count in the setup, so throughput normalization choices affect the headline ratio.
- Based on bounded source text, not independent testing. Practical suggestions are editorial proposals.