Intel's DeepMath trains a small model to solve math by writing sandboxed Python
Hugging Face
Intel's AI Software Group published DeepMath, a math reasoning agent built on Qwen3-4B Thinking, fine-tuned with GRPO and run through smolagents. Rather than long-text arithmetic, the model emits short Python snippets that execute in a sandbox and feed results back into the trace. The authors report output length reductions of up to 66% on four math datasets, with accuracy often improving.
What Intel published
Intel's AI Software Group described DeepMath, a math reasoning agent built on the Qwen3-4B Thinking model and fine-tuned with GRPO (Group Relative Policy Optimization). Instead of writing out multi-step arithmetic in prose, the agent emits short Python snippets for intermediate calculations; these run in a sandbox and their outputs are inserted back into the reasoning trace. The agent is implemented with the smolagents library, with vLLM as the inference engine and an adapted TRL GRPO trainer. Sandbox constraints include an allow-list of importable modules, per-snippet timeouts, and no file or network access. Training rewards both correct answers and code use, caps completion candidates at 5k tokens, and schedules temperature from 1.2 down to 0.7. The authors benchmark on MATH500, AIME, HMMT and HLE, reporting up to 66% shorter outputs while often improving accuracy. Their ablations separate agentic inference from GRPO training, with the combined configuration performing best.
A scenario for an ML evaluation engineer
An engineer at a company that already runs a small reasoning model for internal math-heavy tasks could prototype the DeepMath pattern rather than adopting the released model wholesale: keep the current model, add a constrained executor for arithmetic steps, and compare traces against the existing plain chain-of-thought pipeline. The specific experiment is whether routing intermediate computation through sandboxed snippets lowers arithmetic mistakes on the team's own problem set without sacrificing final-answer correctness. Because the source only claims results on contest-style math, the engineer should treat domain fit as the open question, not a given.
How to trial it and measure
A team could start with a held-out set of roughly 50-100 problems where the baseline model's answers are known, recording baseline accuracy and mean trace length first. Then run the same prompts with a snippet-executing agent layer and compare both metrics, plus a manual review of whether cited snippet outputs were actually used correctly. Failure checks should include: does the sandbox reject disallowed imports, do timeouts trigger and get handled, and does the model fall back to prose when execution fails? A human reviewer should audit a sample of final answers, since shorter traces could hide unsupported steps. Success means fewer arithmetic errors and shorter outputs at comparable accuracy, measured against that recorded baseline, not against the paper's numbers.
Trade-offs and prerequisites
The approach adds infrastructure: a sandbox, an execution loop, and an agent framework such as smolagents, plus a model trained or prompted to emit code reliably. Generated-code execution remains risky despite sandboxing, and the authors themselves recommend managing attack surfaces and rate limits in any deployment. The reported gains are scoped to a 4B model and contest-style math; the authors note results may not transfer to open-ended math or formal proofs. A team should pause if its tasks lack deterministic, checkable computation, if it cannot audit what the model executes, or if it needs complete trace readability with no external runtime. Latency and cost of sandbox execution versus shorter generation are not quantified in the excerpt.
Context and limitations
- Based on a short source excerpt; the full article and complete benchmark tables were not analyzed.
- Reported accuracy and length gains come from the authors' own evaluation on contest-style math datasets (MATH500, AIME, HMMT, HLE); independent reproduction was not reviewed.
- Executing model-generated code carries security risk even with sandboxing; the authors advise managing attack surfaces and rate limits.
- Based on bounded source text, not independent testing. Practical suggestions are editorial proposals.