Reproducing a Third of ICML 2026: What 6,816 Agent Logbooks Showed
Hugging Face
- Research
- Agents
- Benchmarks
- #icml
- #reproduction
- #coding agents
Hugging Face reports that a July 2026 community challenge used coding agents to attempt reproduction of 2,226 ICML 2026 papers, publishing 6,816 logbooks and 35,908 judged claim verdicts. The organizers say about half of examined papers had at least one claim verified and 23% had a claim falsified or contested, including cases where teams disagreed.
The challenge and how it was run
Hugging Face describes an open reproductions challenge for ICML 2026 held July 15 to August 2, 2026. Rather than a 40-page PDF, organizers indexed accepted papers and extracted their core scientific claims so agents had a concrete, checkable target. Participants brought their own agents, from Claude Code and Codex to Cursor and OpenResearch's orx, and could pull a paper, its claims, and instructions with one command. Each run produced a Trackio logbook containing the write-up, the code, artifacts, and optionally the full agent trace. An automated judge running an open-weights model, GLM-5.2, re-read each logbook and issued per-claim verdicts: verified, falsified, toy, or inconclusive, with self-assessments treated as untrusted. Organizers say the challenge produced 6,816 logbooks covering 2,226 papers, with 35,908 claims judged. Where datasets or checkpoints were unavailable, participants ran toy reproductions on synthetic data. Organizers also re-verified every claimed falsification themselves before writing to authors, and report corrections in progress on some papers.
A scenario for a research engineer on a reading group
A research engineer preparing a journal-club or reading-group session could try a narrow version of this workflow instead of presenting a paper only through its abstract. The supported premise is that agents can read a paper, write code, launch experiments, and produce an auditable logbook, and that organizers found paper-level claims more checkable than whole PDFs. A team could try extracting two or three central claims from one paper, assigning one agent per claim in a separate environment, and requiring each run to publish its code and artifacts alongside the write-up. A reviewer role, analogous to the humans organizers describe steering agents, would inspect the logbooks before the session and flag runs that stopped early or used synthetic substitutes. This is a proposed use of a described workflow, not a demonstrated integration; whether it fits a given group depends on compute access and how precisely the paper states its claims.
Trial steps and how to judge success
A team could try a small pilot on five papers with clearly stated claims, recording a baseline first: how many claims a human reviewer would normally check in the same time. Each claim could be run by two independent agents to surface disagreement, since organizers report cases where teams reached opposite verdicts. Success would be measured against that baseline using the same verdict categories the organizers used, with disagreement rate and inconclusive rate tracked separately from verified or falsified counts. Failure checks should include rerunning any falsification from the paper's own text, as organizers did, and looking specifically for checks that end too early: organizers note several verified verdicts on one paper came from finite-horizon checks that stopped before scale-dependent behavior appeared. A named human reviewer should sign off on any falsification before it is shared with authors, and any self-assessment in a logbook should be treated as untrusted.
Trade-offs, prerequisites and when to hold back
The workflow assumes compute for runs, enough public artifacts to attempt a real reproduction, and time to review logbooks critically. Where datasets or checkpoints are closed, evidence drops to toy scale, and organizers report that missing artifacts were the most common cause of inconclusive outcomes. Agents themselves have documented limits: organizers describe them looping, misreading scale-dependent behavior, and occasionally building falsifications on unit mismatches, and some questions, such as whether generated images are usable, stayed with humans. The headline statistics are the organizers' own tallies from one hackathon and are not an independent audit, so they should not be cited as a measure of conference-wide reproducibility. A group without a reviewer able to re-derive the math or rerun the code should probably not present agent verdicts as findings. A team could try the pilot above and expand only if human review keeps pace with the logbooks.
Context and limitations
- Based on a short source excerpt; the full article was not analyzed.
- Figures are the organizers' own hackathon tallies, not an independent audit of ICML 2026.
- The described judge and logbook workflow reflect this one challenge, not a standard practice.
- Based on bounded source text, not independent testing. Practical suggestions are editorial proposals.