FlowtandBriefAI · Software · ResearchMeet Flowtand

Every Eval Ever Adds a Converter to Publish Structured Eval Records to Hugging Face Community Evals

Announcement

Hugging Face

Browse Benchmarks topic page →

A Hugging Face blog post announced that Every Eval Ever (EEE) and Hugging Face Community Evals are intercompatible, with a converter that turns EEE records into the YAML files Community Evals expects and links scores back to full EEE JSON records. The post reports a datastore of about 229,000 results across 22,000-plus models and 2,200 benchmarks.

What the announcement actually adds

A Hugging Face blog post dated June 30, 2026 says Every Eval Ever (EEE), launched in February 2026 by the EvalEval Coalition, is now intercompatible with Hugging Face Community Evals. EEE is described as one JSON schema for an evaluation result that records who ran it, which model, how it was accessed, generation settings, and what the metric means, optionally with a companion JSONL file of per-sample outputs. Community Evals stores scores as YAML under .eval_results/ in a model repo, with benchmark, task, and value required, and an optional source block that links back to the EEE record. The new converter maps fields such as the evaluation name to task_id and the score to value, then inserts the datastore URL as the source link. Before writing, it checks existing YAML on the main branch and in open pull requests, labelling entries already_present, score_conflict, missing_hf_model, or ready, and it opens pull requests only after the user types OPEN PRS. The post states that it handles MMLU-Pro, GPQA, HLE, and GSM8K, and reports roughly 229,000 results covering over 22,000 models and 2,200 benchmarks, gathered from 31 reporting formats. It illustrates the reporting gap with LLaMA 65B scoring both 63.7 and 48.8 on MMLU.

A model publisher preparing a card

A model publisher who already has EEE records for a model could use the converter to avoid maintaining two formats by hand. The workflow in the post is to point the tool at an EEE datastore collection, review the generated local YAML previews and the audit report, then type OPEN PRS with a commit message. This is suited to teams that already report under their organization's official Hub account, since the post says that route yields a verified checkmark on EvalEval. It is not a substitute for producing the evaluation in the first place; the converter only reshapes records that exist.

Trying it with a baseline

A team could trial the converter on one supported benchmark, for example MMLU-Pro, using a datastore collection whose results they already publish manually. First, record the baseline: the current YAML files on the model repo and any open pull requests. Run the tool, compare its audit labels against that baseline, and inspect the previews and review file rather than letting it push. A success criterion could be that every new YAML points to the correct EEE record, that no already_present entry is duplicated, and that any score_conflict is explained. Treat unexpected missing_hf_model entries and unseen score conflicts as failure checks that pause the run. A human should sign off on each pull request, since the tool is reported to require explicit confirmation.

Trade-offs and when to hold back

The trade-off is limited coverage. With only four benchmarks supported, a team evaluating on other benchmarks would still need manual YAML or wait for broader support; the post does not describe plans beyond those four. Prerequisites include an EEE datastore collection with reviewable source records and a model repo that resolves on the Hub; otherwise models are flagged missing_hf_model. Reruns reuse cached results unless --force is passed, so treat cached previews with care when the datastore changes. Because the evidence is a single blog post, including datastore size and cost claims that were not independently checked, a team should validate the converter against its own records before relying on it. Hold off if a model author disagrees with public submitted scores, since the post says model authors can close pull requests or hide results on their own repo.

Context and limitations

  • Based on a short source excerpt; the full article was not analyzed.
  • Benchmark coverage is limited to the four named benchmarks; the post does not describe behavior for others.
  • The post reports datastore figures and cost estimates without independent verification.
  • Based on bounded source text, not independent testing. Practical suggestions are editorial proposals.