Google DeepMind Proposes 10-Ability Cognitive Taxonomy for AGI and Opens Kaggle Evaluations Hackathon
Google DeepMind
- Research
- Benchmarks
- Safety & policy
- #agi
- #cognitive taxonomy
- #kaggle
Google DeepMind published a paper proposing a cognitive-science taxonomy of ten abilities for assessing AI systems and a three-stage protocol comparing systems to a representative adult sample. It also launched a Kaggle hackathon on five under-evaluated abilities, with submissions open March 17–April 16 and results June 1.
What the framework proposes
Google DeepMind announced a paper, "Measuring Progress Toward AGI: A Cognitive Taxonomy," arguing that progress toward AGI is hard to gauge because empirical evaluation tools are limited, and that cognitive science is one useful component. The taxonomy lists ten hypothesized abilities: perception, generation, attention, learning, memory, reasoning, metacognition, executive functions, problem solving, and social cognition. To assess systems across them, the authors propose a three-stage protocol: test AI systems on a broad suite of cognitive tasks using held-out sets to limit data contamination; collect human baselines from a demographically representative adult sample on the same tasks; then map each system's performance against the human distribution for each ability. The announcement also describes a Kaggle hackathon on five abilities described as having the largest evaluation gap — learning, metacognition, attention, executive functions, and social cognition — with submissions open March 17 to April 16 and results announced June 1. These are the authors' proposals; the excerpt reports no validation results.
A scenario for benchmark designers
Consider a benchmark team at a research lab that already tracks model scores on task-specific suites but cannot say much about, for example, metacognition or executive functions. A team could try mapping its existing evaluations to the ten proposed categories to expose which abilities are covered only thinly or not at all, then prioritize new task design where gaps are widest. Because the protocol calls for held-out test sets, the team could also review whether current suites risk contamination and whether human-comparison data exist for the tasks it keeps. This is a structuring exercise, not a claim that the framework has been shown to predict general intelligence.
Running a small trial
A lab could select one under-covered ability, such as executive functions, and draft a few tasks with held-out items, then collect a small human baseline from a demographically described sample following the protocol's second stage. Before comparing, it should fix a baseline: scores from an existing benchmark for the same models and a documented null result (e.g., chance-level performance on pilot items). Success would mean the new tasks produce measurable spread across models and remain stable under repeated runs; failure checks include contamination screening of test items, inter-rater agreement on human scoring, and review by at least one independent cognitive scientist. If human baselines are impractical, a team could try documenting the limitation rather than substituting model judgments for them.
Trade-offs and unknowns
The framework's value depends on whether the ten abilities can be operationalized into tasks that are valid, uncontaminated, and fair to compare against human distributions; the excerpt does not show that they can. The protocol is resource-intensive, requiring held-out sets and representative human sampling, and human baselines bring their own design choices about demographics and task framing. The hackathon targets only five abilities, so coverage remains partial by design. Teams should treat the taxonomy as a scaffal for organizing evaluation work, not as a validated measure of general intelligence, and should not delegate scoring or interpretation of results entirely to automated systems. Whether this framework improves on existing benchmark suites is an open question the paper is meant to start addressing.
Context and limitations
- Based on a short source excerpt; the full paper was not analyzed and its detailed methodology is unverified.
- The taxonomy and protocol are presented as a proposed framework; the source does not report validation results.
- Prize amounts and dates are as stated in the announcement and may change; no independent verification.
- Based on bounded source text, not independent testing. Practical suggestions are editorial proposals.