The CMU 30.3%: the enterprise agent capability gap

Carnegie Mellon University’s TheAgentCompany 2026 benchmark reports that Gemini 2.5 Pro completes 30.3% of enterprise agent tasks, the top score across the frontier LLMs the benchmark tests. GPT-5 Pro completes 28%. Claude 4 Opus 27%. The 30.3% figure gets cited frequently in vendor marketing as evidence that agents are “approaching production-readiness.” The benchmark’s own brief implies a production-readiness threshold closer to 95%.

The gap between 30.3% and 95% is the subject of this piece. Not as a reason to wait, but as the specific operating constraint the enterprise deployments that work are running within. Durable deployments exist inside that constraint: McKinsey’s 6% AI-high-performer segment attributes measurable EBIT to AI, IDC research with Lenovo (CIO.com, 25 Mar 2025) reports roughly 12% of AI POCs graduate to production while 88% never do, and Gartner’s 40% cancellation projection for agentic AI projects by end-2027 frames the downside. The deployments that survive use the same base models scoring 30.3% on the CMU benchmark. They produce durable outcomes not by working around the capability gap but by scoping narrowly enough that the 30.3% translates into 85-95% on a specific named scope.

Two propositions structure the piece:

• Capability is rising but will not reach 95% within the 3-year TCO horizon. The benchmark trajectory (24% in 2024, 30.3% in 2026) projects to 40% by late 2027. Enterprises planning on “waiting for better models” are planning on an event outside the horizon of their current business case.
• The durable cohort operates within the 30.3%, not around it. Narrow scope. Human-in-the-loop for edge cases. Governance discipline on the six GAUGE framework dimensions. The capability gap is real; it is not the variable that separates the deployments that hold up from the ones that stall.

What TheAgentCompany actually measures

The CMU benchmark places an LLM-powered agent inside a synthetic enterprise environment with five business units: HR, finance, engineering, legal, administration. The agent is given natural-language tasks a new hire at that company might receive: draft a performance review, reconcile a quarterly budget, commit a code change, prepare a compliance memo, schedule a cross-team meeting.

Tasks span difficulty from trivial (look up a specific record) to complex (draft a cross-functional change-management plan with stakeholder sign-offs). A task counts as completed if the agent’s output meets the benchmark’s acceptance criteria without human intervention during the execution. Partial completion, requiring intervention, or producing an output that fails the acceptance check all count as failures.

Gemini 2.5 Pro completes 30.3% of the tasks. The distribution across difficulty tiers is significant: roughly 55% completion on easy tasks, 25% on medium, 8% on hard. The 30.3% headline is a weighted average that under-represents how badly the best model performs on the difficult tasks the enterprise economy is actually built on.

Two observations from the methodology:

First, the benchmark is synthetic. Real enterprise deployments operate on real data with real organisational politics; the benchmark’s environment is cleaner than any production deployment. Real-world completion rates for equivalent agent scope should be read as lower than the benchmark number, not higher.

Second, the 95% production-readiness implied threshold is a benchmark artefact, not an industry standard. Different enterprise functions have different thresholds. A customer-support agent at 85% autonomous-completion with clean handoff on the remaining 15% is production-viable; a compliance-memo-drafting agent at 85% is not, because the 15% that fails is exactly the 15% that creates regulatory exposure. The threshold varies with the cost of a failure, not with benchmark convention.

The trajectory of capability

CMU’s 2024 original benchmark used Claude 3.5 Sonnet; that model completed 24% of the tasks. The 2026 update puts Gemini 2.5 Pro at 30.3%, a gain of 6.3 percentage points over roughly 18-24 months. The rate is broadly consistent with other benchmark trajectories: SWE-bench verified has similarly moved from ~15% to ~55% over a comparable window for the highest-scoring models, though SWE-bench is a narrower benchmark (code-repository bug fixes) where narrow specialisation drives faster improvement.

Projecting the CMU rate forward, the top score hits ~40% by late 2027 if the current pace holds. The capability line does not cross the 95% implicit threshold within the 3-year TCO horizon most enterprise AI business cases operate against. Planning a scale-up to assume capability catches up during the deployment’s life is planning on a favourable trajectory that is not in the data.

What the trajectory does support: scope that looks borderline today may be meaningfully more viable in 18 months. An enterprise targeting an agent at a specific scope where current capability is 60% and the required-for-production threshold is 85% might reasonably plan to re-score the deployment in 18 months and expand scope then. But that is different from planning for a capability phase-change.

What the durable cohort does with 30.3% capability

Correction (10 Jun 2026): This section previously anchored on a Stanford Digital Economy Lab “12% durable cohort clearing 300%+ ROI against an 88% failing body”. Full-text verification on 10 Jun 2026 found that figure absent from the cited playbook, which studies 51 successful deployments by design; the claim for this piece is Partial and the correction is logged at AM-031. The verified adjacent split is IDC/Lenovo’s pilot-graduation data (CIO.com, 25 Mar 2025): roughly 12% of AI POCs graduate to production, 88% never do. The operational patterns below stand on their own evidence and no longer lean on the withdrawn cohort.

The deployments that survive into durable production use the same frontier models as the pilots that stall. The distinguishing operational patterns:

Narrow scope. The agent is responsible for a specific named workflow inside a specific business process, not a general-purpose assistant. “Triage incoming customer-support tickets into one of eight categories and route accordingly” is a narrow scope. “Handle customer support” is not. The 30.3% benchmark capability translates to 85-95% on the narrow scope because the narrow scope is what the 30.3% actually covers.

Human-in-the-loop on edge cases. The 15-25% of the scope the agent cannot handle autonomously is routed to a human reviewer with context. The agent’s value is in the 75-85% of volume it handles without human time, not in the 100% of volume with no human oversight. The “autonomy” framing is vendor-convenient; the operational pattern is augmentation, not replacement.

Governance discipline across all six GAUGE dimensions. Per the deployment-failure analysis: the distinguishing variable is governance-maturity + threat-model + ROI-evidence + change-management + vendor-lock-in + compliance-posture scored together. The durable cohort runs on a 90-day review cadence. The stalled majority does not.

MTTD-for-Agents instrumented from day one. The narrow scope does not eliminate the attack surface; it contains it. The detection layer (tripwires on tool-use frequency, output-length Z-score, cross-agent delegation rate, refusal rate) is in place whether or not a specific incident class has manifested in this deployment yet.

Four patterns, implemented together. None requires model capability beyond 30.3%. All require operational discipline the stalled majority treats as nice-to-have.

When capability does limit scope

There are enterprise scopes where the 30.3% capability genuinely blocks durable deployment, and no amount of governance discipline compensates. Three examples from the benchmark’s hard-tier tasks:

Complex cross-functional coordination. Drafting a change-management plan that requires synthesising HR constraints, legal exposure, finance budget envelopes, and engineering feasibility simultaneously. The 8% completion rate on hard tasks means seven out of eight attempts fail. No governance wrapper converts a 12.5% success rate into a production-viable workflow.

Regulatory compliance memo drafting. The accuracy floor for regulatory output is 100%; a 90% agent output with 10% hallucination rate is unusable because the cost of a single wrong statement propagates. Until the capability on this specific scope crosses the accuracy floor, the scope is unfit for autonomous agent coverage. Human-drafted with AI-assisted review is the viable pattern, not the inverse.

Multi-agent coordination at scale. Benchmarks that test agents coordinating with other agents (passing work, resolving conflicts, renegotiating assignments) score substantially below single-agent task completion. The enterprises attempting multi-agent orchestration at scale in Q1 2026 are concentrated in the failure-mode data, not in the durable cohort.

For each of the three, the honest answer is: not now. Not because the general capability is insufficient, but because this specific scope sits beyond the current benchmark envelope. Re-evaluate at 18 months.

What this means for Q2-Q4 2026 deployment planning

The practical implication of the CMU data for an enterprise considering a new agentic AI deployment:

1. Score the candidate deployment on the six GAUGE dimensions before proposing it. A deployment that scores below 40 is unready regardless of capability; a deployment that scores 60+ is worth the next step.
2. Define the scope narrowly enough that benchmark capability supports the scope. Use the CMU data as the calibration: if the scope requires hard-tier capability (8% completion), re-scope or reject. If the scope is easy-tier (55% completion), the durable pattern is achievable.
3. Instrument human-in-the-loop review for edge cases from day one. The review is not a transitional step that goes away when the model improves; it is the permanent operational pattern the durable cohort runs.
4. Instrument MTTD-for-Agents detection at the agent surface. Tripwires on tool-use, output-length, delegation, refusal. This is the precondition for scaling the deployment later, per the McKinsey 23% scaling-gap analysis.
5. Plan the business case against current capability, not projected. If the deployment does not produce ROI at 30.3% capability, it does not produce ROI. The 40% projection by 2027 is a possible upside, not a dependency.

The narrow-capability / broad-incapability divergence the CMU data documents is also visible in the opposite direction. Anthropic’s Claude Mythos disclosure (April 2026) reports a 73% success rate on expert-level hacking tasks against a baseline where no prior public model could complete such tasks at all twelve months earlier. Mythos and TheAgentCompany are measuring different things, but together they describe the same calibration problem: agents can be benchmark-strong on a narrow specification and operationally inadequate against an unbounded one in the same release cycle.

The governance playbook operationalises all five into a 90-day setup. The playbook’s first 30 days concentrate on scope definition and baseline measurement, which is where the capability-scope match gets tested before any scale-up decision.

What the data does not support

Three framings that are common in 2026 vendor discourse and are not supported by the CMU data:

“Agents are almost ready; wait six months.” The benchmark’s 2024-to-2026 trajectory is 6 percentage points over 18-24 months. Six months is roughly 1-2 percentage points. Not a phase change; not a threshold crossing.

“Better models will close the gap between the deployments that hold up and the ones that stall.” Both cohorts use the same models. The gap is governance. NIST AI RMF, ISO/IEC 42001, and the EU AI Act all treat capability and governance as separate axes; the CMU data specifically isolates capability and the IDC/Lenovo pilot-graduation data specifically isolates how few deployments survive to production. The two axes do not correlate in the way “better models solve this” requires.

“30.3% is close enough to production for general autonomy.” Two out of three tasks failing is not close enough for any scope where failure propagates cost. Narrow-scope deployments with human-in-the-loop are not “almost autonomous”; they are a different operational model. Marketing the narrow-scope pattern as autonomy is the framing error that produces the stalled majority.

Holding-up note

The primary claim of this piece (that the CMU 30.3% capability figure is the current operating constraint for enterprise agentic AI deployment, that capability trajectory will not cross the 95% production-readiness threshold within the 3-year TCO horizon, and that the durable deployments operate within the 30.3% by scoping narrowly plus governance discipline, not by waiting for capability improvement) is on a 60-day review cadence. The cohort leg of the original claim, which anchored on a Stanford DEL 12%-vs-88% split, failed full-text verification on 10 Jun 2026; the claim stands Partial with the correction above. Three kinds of evidence would move the verdict:

• A frontier-model generation that crosses 50% on TheAgentCompany benchmark without a corresponding governance-discipline change in deployment patterns. Would partially weaken the “capability is not the variable” framing.
• Cross-enterprise analyses showing that deployments waiting for capability improvement produce outcomes statistically indistinguishable from deployments running the governance pattern today. Would materially weaken the scope-narrow-and-instrument recommendation.
• A benchmark refresh from CMU or equivalent showing the enterprise-task difficulty distribution has shifted such that easy-tier tasks (the 55% bucket) now include what were previously hard-tier tasks. Would expand the scope-that-works envelope.

If any land, the Holding-up record for AM-031 captures what changed, dated. Original claim stays visible. Nothing is quietly removed.