Mistral Releases Mixtral 8x7B, a Sparse Mixture-of-Experts Open-Weight Model
Mistral AI
- Models
- Open source
- #mixtral
- #mixture-of-experts
Mistral AI announced Mixtral 8x7B on December 11, 2023, an open-weight sparse mixture-of-experts model under Apache 2.0. A router sends each token to two of eight expert groups, so the model holds 46.7B parameters but uses 12.9B per token. Mistral reports it outperforms Llama 2 70B on most benchmarks with 6x faster inference and matches or beats GPT-3.5 on most standard benchmarks, plus a 32k-token context and five supported languages.
What Mistral announced
On December 11, 2023, Mistral AI released Mixtral 8x7B, which it describes as a high-quality sparse mixture-of-experts model with open weights, licensed under Apache 2.0. Architecturally it is a decoder-only model whose feedforward block chooses from eight distinct parameter groups; at every layer a router picks two experts per token and combines their outputs additively. The practical consequence Mistral emphasizes is that the model carries 46.7B total parameters but computes with only 12.9B per token, so it processes input and generates output at the reported speed and cost of a 12.9B model. Mistral reports that Mixtral outperforms Llama 2 70B on most benchmarks with 6x faster inference, matches or beats GPT-3.5 on most standard benchmarks, handles a 32k-token context, supports English, French, Italian, German and Spanish, and performs strongly in code generation. It also released an Instruct variant tuned with supervised fine-tuning and DPO that reaches 8.30 on MT-Bench, and noted deployment work integrating Megablocks CUDA kernels into vLLM plus Skypilot for endpoint deployment. These are publisher claims from the announcement excerpt, not results verified by this review.
A scenario for an ML platform engineer
A platform engineer maintaining a self-hosted inference service could consider Mixtral as a candidate for workloads mixing multilingual requests, code assistance and longer documents. The sparse routing is the reason to look: if Mistral's per-token figure holds in that engineer's environment, serving costs would resemble a ~12.9B-parameter dense model rather than a 46.7B one, even though quality on several tasks is claimed to sit near Llama 2 70B. Because the announcement names a 32k context window and five languages, a team could first test on prompts from those categories. This is a hypothetical evaluation path, not a claim that any integration or saving has been demonstrated outside Mistral's own reporting.
How to trial it against a baseline
A team could run a bounded comparison using its current production model as the baseline. Steps: assemble a fixed evaluation set drawn from real traffic, tagged by language, code, and long-context requests; record baseline quality judgments (human or rubric-based) and per-token latency and throughput; then run the same set on Mixtral 8x7B and its Instruct variant. Define success in advance, for example no worse quality on most tagged categories with equal or lower cost per token. Failure checks should include long-context prompts near the 32k limit and code outputs reviewed by a developer. Because Mistral's note about moderation says that without a banning prompt the model follows whatever instructions it is given, any user-facing deployment needs a human review step or a tested output filter before launch.
Trade-offs and when to hold back
Sparse routing lowers compute per token but keeps the full weight set in memory, so memory footprint and hosting prerequisites may differ from a dense 12.9B model; this excerpt does not state hardware requirements. The quality and speed comparisons are Mistral's, measured against Llama 2 and GPT-3.5 on standard benchmarks, and may not transfer to a specific domain. Moderation is not built in by default. A team should avoid adopting Mixtral where strict data-governance review is pending, where its five listed languages do not cover users, or where independent benchmark verification is a requirement. The beta mistral-small endpoint is described as early-access, so it should not be assumed production-ready.
Context and limitations
- Based on a short publisher excerpt; the full article and detailed benchmark table were not analyzed.
- Benchmark, speed and accuracy figures are Mistral's own claims from its announcement, not independent test results.
- Only the base and Instruct variants, Apache 2.0 licensing, and the beta mistral-small endpoint are described; production behavior is untested here.
- Based on bounded source text, not independent testing. Practical suggestions are editorial proposals.