FlowtandBriefAI · Software · ResearchMeet Flowtand

Mistral's Leanstral 1.5 targets Lean 4 proof engineering with an open-weight model

Announcement

Mistral AI

Browse Models topic page →

Mistral's Leanstral Team announced Leanstral 1.5, an Apache-2.0 licensed model with 119B total and 6B active parameters for formal verification in Lean 4. The team reports saturating miniF2F, solving 587/672 PutnamBench problems, and reaching 87% on FATE-H and 34% on FATE-X, plus five previously unreported bugs found across 57 tested repositories. These figures come from the publisher, not independent testing.

What Mistral announced

Mistral's Leanstral Team released Leanstral 1.5, described as a free Apache-2.0 licensed model with 119B total and 6B active parameters aimed at proof engineering in Lean 4. Training reportedly ran through mid-training, supervised fine-tuning, and reinforcement learning with CISPO across two environments: a multiturn setting where the model submits proofs and refines them using Lean compiler feedback, and a code-agent setting where it edits files, runs bash commands, and inspects goals via the Lean language server. The team reports saturating miniF2F, solving 587 of 672 PutnamBench problems, and reaching 87% on FATE-H and 34% on FATE-X, with gains on its FLTEval benchmark. It also describes an automated pipeline pairing the Aeneas Rust-to-Lean translator with property inference; across 57 repositories it flagged 47 violated properties, 11 judged genuine bugs, five previously unreported. All of these are publisher-reported results.

A scenario for a verification engineer

A verification engineer maintaining a Lean 4 codebase with partially proved theorems could consider using the model, via the described API endpoint or the suggested Mistral Vibe agent setup, to draft auxiliary lemmas or close routine goals in a scratch branch. Conditional on the team's Lean toolchain already building cleanly, the engineer might route a small set of stuck lemmas through the agent and inspect each proposed proof by compiling it and reading the term. The bug-discovery case suggests a separate, more cautious trial: a team with a Rust library and an existing Aeneas translation path could ask the model to infer correctness properties and attempt proofs, treating every flagged property as a lead for human triage rather than a confirmed defect.

How to trial it against a baseline

A team could select ten to twenty Lean 4 goals its engineers already solved, record the human time and token cost for each, then run the same goals through Leanstral 1.5 under a fixed token budget. Success would mean matching the baseline proof acceptance rate at acceptable cost, measured by compiled proofs per goal and reviewer minutes per accepted proof. Failure checks should include proofs that compile but state weaker properties than intended, and goals where the agent exhausts its budget without progress. For the code-verification idea, a team could re-run its own previously known bugs through the pipeline and measure how many the flags recover versus how many false positives require triage. A human reviewer should confirm every property and proof before merge.

Trade-offs and when to hold back

The central trade-off the release implies is token budget against solved problems: Mistral reports monotonic gains up to a 4M token budget on PutnamBench, with one AVL-tree proof spanning 2.7 million tokens and 22 compactions. Long runs may therefore be slow and costly even if per-problem cost is described as low. Prerequisites include a working Lean 4 setup and, for code verification, a Rust-to-Lean translation path such as Aeneas. Unknowns include behaviour outside the reported benchmarks, reproducibility, and how often flagged properties are false positives. A team should not rely on the model for final sign-off on safety-critical proofs or treat reported bug discoveries as confirmed without manual reproduction.

Context and limitations

  • All benchmark results and bug counts are reported by Mistral; no independent evaluation is described in the excerpt.
  • Cost comparisons cited (about $4 per PutnamBench problem versus far higher estimates for other provers) are publisher claims and not verified.
  • The excerpt is short and does not cover failure modes, hardware requirements, or reproducibility of the bug-discovery pipeline.
  • Based on bounded source text, not independent testing. Practical suggestions are editorial proposals.