AI Productivity: The Architecture Gap

Why task-level gains don't translate to organisational improvement

Executive Summary

AI productivity gains are real at task level but don't automatically translate to organisational improvement
The bottleneck shifts, it doesn't disappear
Same pattern at the individual level — delegation erodes capability; scaffolding preserves it
Success requires architectural transformation and org redesign
This is an enterprise architecture problem, not a tooling problem

Agenda

1. The speed of change
why your mental model from 2023 is already wrong

6. Enterprise architecture as the constraint
is your organisation legible?

2. The productivity paradox
task gains real, org gains aren't

7. AI as middle manager
what the research says about substitution

3. The bottleneck cascade
what happens when you speed up coding alone

8. A worked example
copilot to dark factory

4. Why copilots hit a ceiling
and why the gap is widening

9. Adoption barriers are psychological
lessons from Trail of Bits

5. DORA: AI amplifies what you are
team health sets the floor

10. Practical recommendations
what to do this quarter

Exercise 1 — AI in your last month

EXERCISE 1 · GROUPS · ~6 MIN

In your group, go round each person:

Have you used AI in the last month?
If yes — which tool, what for, and how?
If no — why not?
Across the group, where's the spread?

The Speed of Change

Timeline

2019: First demo code completion
2022 Nov: ChatGPT public release
2025 Feb: Claude Code research preview
2025 Nov: Opus 4.5 — step change with Claude Code
2026: Agentic systems mainstream

"Anyone who thinks AI is slowing down is fatally miscalibrated"

— Jack Clark, co-founder of Anthropic (X, March 2026)

METR: Task Duration Doubling

METR time horizon of software engineering tasks

Task duration AI can complete (80% reliability) doubles every 4-7 months

From Task Success to Agentic Workflows

Rapidly Expanding Capability

Mythos Preview: Firefox JS shell exploitation

One domain, one generation: near-100x jump

Capabilities don't arrive gradually — they step-change

Firefox bugs fixed per month

$Firefox bugs fixed per month$

The Jagged Frontier

Jagged frontier: where AI is superhuman, where it's not

AI is superhuman at some things humans find easy, and terrible at others that look similar
You cannot generalise from one use case to another
Each domain needs its own capability assessment

Source: Helen Toner, Taking Jaggedness Seriously (helentoner.substack.com), adapting Ethan Mollick. The frontier isn't a smooth line where AI capability is high on one side and low on the other — it's a jagged, unpredictable shape where "easy for humans" and "easy for AI" don't track each other. Coding can be superhuman; basic arithmetic can be terrible; legal reasoning can be great in some sub-domains and hopeless in adjacent ones. For IT leaders, this has two implications: (1) you cannot extrapolate from a successful pilot in one domain to a different domain without testing, and (2) the shape of the frontier changes with each model release, so your capability map has a short shelf life. Previous slide (Mythos/Firefox) shows the speed at which any given domain can jump; this slide shows the unevenness of where the jumps happen.

The Productivity Paradox

DORA Metrics — The Gold Standard

DORA (DevOps Research and Assessment) measures software delivery:

Deployment frequency — how often you ship to production
Lead time for changes — commit to production
Change failure rate — % of deployments causing incidents
Time to restore service — recovery time after failure

These are the metrics that matter — not lines of code or PRs merged

Three Levels, Three Stories

Level	Source	Finding
Task	Anthropic	80% time savings
Developer	METR RCT	-4% to +9% (flat)
Organisation	Faros	+21% tasks, unchanged DORA

Amdahl's Law Explains the Gap

Amdahl's Law: The maximum speedup of a system is limited by the fraction that can be improved

Coding = 25-35% of the software development lifecycle
Even infinite speedup in coding → at most ~54% system improvement
The other 65-75%: requirements, design, testing, review, deployment, operations

Try It Yourself

Brooks Told Us 50 Years Ago

No Silver Bullet (1986): "The hard part of building software is the specification, design, and testing of this conceptual construct, not the labor of representing it"

Coding is accidental complexity — AI reduces this
Specification, design, testing are essential complexity — AI doesn't eliminate these

Fred Brooks drew the distinction between essential and accidental complexity in software. Essential complexity is inherent in the problem — understanding what to build, designing how it fits together, verifying it works correctly. Accidental complexity is the labour of expressing that understanding in code. AI dramatically reduces accidental complexity. But Brooks' insight is that accidental complexity was never the main cost. His estimate that five-sixths (83%) of time is spent on things other than coding aligns remarkably with the Amdahl's Law calculation — coding has always been the minority of software work. We've known this for 50 years. The industry keeps rediscovering it. Brooks also noted a 10:1 productivity variance between the best and worst developers — a ratio that maps onto organisations as well, and which AI amplifies rather than eliminates.

Faros 2025 Data — 10K Developers, 1,255 Teams

+21% tasks completed
+98% PR volume
+91% review time
+154% PR size
+9% bugs
Flat DORA metrics

The Cascade Effect

AI accelerates code

→ PR volume doubles

→ review time increases

→ quality gates overwhelmed

→ deployment unchanged

Exercise 2 — Map your bottleneck

EXERCISE 2 · GROUPS · ~8 MIN

Pick a process — e.g. feature delivery, an incident response, an onboarding, a report.

List the steps end-to-end
Estimate % of total time per step
Circle where AI is actually accelerating you today
Is that the biggest step? If not — what is, and why isn't AI touching it?

Why Copilots Hit a Ceiling

Copilots Are Task Tools

Copilots accelerate individual tasks — exactly the level where gains are real

But we've just seen: task-level gains don't translate to org-level improvement

Copilots optimise the 25-35% that Amdahl's Law says can't move the needle
They augment existing workflows — they don't transform them
The bottleneck cascade happens because we speed up tasks without changing the system

Your Data Maturity Is the Ceiling

Before AI, you could run on tribal knowledge, stale spreadsheets, and "ask Dave". Slow, but workable.

AI can only reason over data that's structured, accessible, and reliable
Gaps that used to be inconvenient become binding constraints
"Ask Dave" doesn't work when Dave isn't in the loop

AI Adoption Requires Systemic Change

Task tools → task gains (real but bounded)

Systemic improvement requires systemic change:

How code is specified (intent, not implementation)
How quality is verified (machine-checkable, not human review)
Who drives the work (domain experts, not just developers)

This is the pivot point of the presentation. Everything before this says "here's the problem." Everything after says "here's what actually works." The shift is from thinking about AI as a tool that helps individuals write code faster, to thinking about it as a force that requires re-architecting how software is built and delivered. The third bullet reflects an emerging structural change: as coding becomes commodified, domain expertise becomes the bottleneck and the differentiator. As Hamel Husain put it: "the coding is commodified but the domain expertise is the differentiator." OpenAI's Pioneers Program is already recruiting lawyers, doctors, and accountants as benchmarking partners — not developers. Teams are starting to invert: domain experts driving, with developers providing infrastructure. The implication for IT leaders: your team structures will change as the bottleneck shifts from code to specification, evaluation, and judgment.

The Competitive Dynamic

Copilots-only users vs. agent-native organisations — the gap is widening:

Microsoft rolls out Claude Code internally while selling Copilot externally
Two kinds of AI users: CLI agents vs chat-only
Productivity gap compounds over time
84% haven't redesigned roles for agents (Deloitte)

Inaction is a position, but maybe not a good one

DORA 2025: AI Amplifies What You Are

Everyone Uses AI Now

DORA 2025: 90% of respondents use AI at work

90% of developers now use AI at work — adoption is no longer the question

But Impact Varies Dramatically by Team Type

AI Magnifies Strengths AND Weaknesses

High-performing teams get more productive
Struggling teams may get worse
Aggregating across archetypes masks the signal
Invest in team health before investing in AI tools

Organisational Readiness Sets the Floor

DORA: "AI's primary role is that of an amplifier. It magnifies the strengths of high-performing organisations and the dysfunctions of struggling ones"

Ad-hoc business processes → AI can't systematise what isn't systematic
Poorly controlled data → agents inherit your data quality problems
Unclear ownership → more code, same confusion, faster

Fix the foundations first — AI won't do it for you

This is the flip side of the amplification effect. Where processes are well-defined and data is clean, AI can accelerate them. Where processes are ad-hoc, built on tribal knowledge, or contradictory, AI has nothing coherent to accelerate — it amplifies the chaos. Brooks observed a 10:1 productivity variance between the best and worst developers in 1968; that ratio maps onto organisations too. The best-run organisations will see dramatically more benefit from AI than the worst-run ones. This isn't a tooling problem — it's a management problem. If your requirements process is "ask Dave, he knows how it works" and your data is spread across three inconsistent spreadsheets and a legacy system, no amount of AI capability will deliver the gains you're expecting. Organisations that haven't invested in process clarity, data governance, and clear ownership will find AI investment delivers diminishing returns regardless of which model they use.

Enterprise Architecture as the Constraint

The Core Insight

Task-level gains != Org-level gains

The gap is architecture — and architecture here means the whole organisation, not just the code.

Your data, processes, and decision rights — how legible they are to an agent
Not your model choice, your licence count, or your tooling stack

Is Your Organisation Legible?

Can an agent read your organisation well enough to work in it?

Data — structured and queryable, or tribal knowledge and stale spreadsheets?
Process — documented and consistent, or "ask Dave, he knows how it works"?
Project state — in Jira and systems of record, or in slide decks and email threads?
Decisions — recorded with their rationale, or in someone's head?

Legibility isn't a coding problem. It's an information architecture problem. Every gap is a ceiling.

This slide ties together several earlier threads — the Data Maturity point (your ceiling is the quality of your data), the DORA amplifier finding (AI magnifies what you are), the PMO example (project state locked in slide decks), the Adoption Barriers (opacity). The common factor is legibility: can the information an agent needs to do its job actually be read by an agent? For most organisations, the honest answer across these four rows is "partially, mostly no". That's the work. Not "roll out more copilots". The "every gap is a ceiling" line is the memorable takeaway — each row where your organisation isn't legible caps the benefit AI can deliver, regardless of model capability. Callbacks are deliberate: this is where the diagnostic sections from earlier pay off as one coherent prescription.

Benefit Requires Systems Thinking

Getting benefit from AI requires value stream mapping

Map how an idea gets to production end-to-end
e.g. in development, the area from commit to production is often the best place to start

Value stream map: backlog to production

Exercise 3 — Legibility audit

EXERCISE 3 · GROUPS · ~8 MIN

Score your organisation 1–5 on each row — could an agent actually read this?

Requirements — written, current, in one place?
Data — discoverable, consistent, machine-readable?
Process — documented, or "ask Dave"?
Decision rights — clear who owns what?

Your lowest score is your ceiling. What would it take to move it by one point?

Augmentation vs Delegation

The organisation can be legible and still erode the people inside it

Delegation vs Scaffolding

Two ways to use AI. Same tool. Opposite cognitive consequences.

Delegation — just get the answer: 61% do this and show impaired persistence on later tasks
Scaffolding — get hints, clarification, critique: 27% do this with no significant impairment vs control

It isn't about how much AI you use. It's about whether you use it to avoid the cognitive work, or to structure it.

This slide establishes the central distinction that runs through the rest of the talk. Liu et al. (arXiv:2604.04721) ran a randomised trial with over 1,200 participants. The aggregate finding — AI use impairs later independent performance — is what the media will report. The moderation finding is what matters. Users who obtained direct answers from the AI were more than twice as likely to show impaired persistence on subsequent tasks than users who used AI only for hints and clarifying questions. The hint-only group showed no significant impairment versus control. Same model, same prompts available, different interaction pattern, opposite cognitive consequence. This reframes the intervention. "Just use less AI" is the wrong prescription — it trades productivity for capability. "Use AI to scaffold rather than delegate" is the right one, because it preserves both. But the distinction is almost never designed for. Enterprise AI architecture — MCP, agent frameworks, "autonomous" deployments — is optimised for delegation. The systems that would preserve human capability are the ones the market isn't building.

The Skill Cliff

Skill loss from AI delegation isn't a slope. It's a cliff.

Up to a point, you can use AI heavily and stay sharp
Past that point, capability collapses — it doesn't gradually decline
Today's frontier AI is past that point for most knowledge work

The fix isn't less AI. It's scaffolding over delegation — which has to be architected, not willed.

This is Park, Kim & Han's Enrichment Paradox in plain language. Their model — calibrated against 20 years of PISA educational outcomes data — identifies a critical threshold (they call it K*, around 0.85) below which humans maintain capability through use, and above which capability collapses through a phase transition. It's not a smooth slope, it's a cliff edge. Current frontier AI capability already exceeds the threshold (GPT-4o and Claude both scoring around 0.94). The optimistic finding: 20% mandatory practice preserves 92% of capability, and periodic AI failures actually strengthen capability 2.7-fold. The pessimistic implication: there's no slow-feedback warning before collapse. Adoption rate is not the same as delegation rate, so we may not be at the threshold in practice yet — but if the model is right, by the time you measure the decline, it's too late. The headline for IT leaders: don't wait for evidence of skill erosion before mandating practice. The evidence comes after the cliff.

The Augmentation Trap

Caosun & Aral (2026): the economics actively select against prevention.

Fully informed, rational decision-makers adopt AI even when it lowers long-run output
Why: front-loaded productivity gains outweigh back-loaded skill costs at any ordinary discount rate
The faster you discount the future, the more attractive adoption becomes — even knowing the cost

Individual rationality produces collective irrationality. The market won't fix this because the market is the cause.

Caosun & Aral's Augmentation Trap model formalises why this pattern keeps repeating across domains. The argument is not that organisations are uninformed — they explicitly model fully informed rational actors who know AI will reduce long-run output through skill erosion. The actors still adopt AI because the productivity gains arrive immediately while the skill costs accumulate over years. At any standard discount rate (5-10% annually), the present value of front-loaded gains dominates the present value of back-loaded costs. This isn't a knowledge failure or a coordination failure — it's a pure incentive failure built into how rational actors evaluate temporal trade-offs. The implication: external governance (regulators, professional bodies, unions) face the same incentive structure when evaluating organisations. They see productivity gains in the data first; capability erosion shows up much later. By the time the evidence is undeniable, the next rational action is still adoption. This is the trap.

Organisational Delegation Drift

Scaffolding at the individual level requires de-scaffolding at the organisational level.

Fewer approval layers, smaller review batches, different accountability structures
You cannot preserve human judgement at the keyboard while keeping a process architecture that assumes humans produce code at human speed

Most organisations will drift into de facto delegation without process changes — the worst of both worlds.

This is the governance point that the Augmentation Trap slide sets up but does not resolve. If individuals need to scaffold rather than delegate, the organisation has to change to let them. The review architecture most enterprises built assumes humans write code at human speed. If AI speeds generation 2–3× and review still runs at human speed, you either get the Faros bottleneck cascade we already saw — review time growing faster than generation — or you collapse review gates and accept the delegation failure mode. Forced scaffolding — plan approval before implementation, mandatory review gates, rotating "no-AI" sprints, kill switches at 3+ stuck iterations — is the defensible middle. It only works if the organisation's bottlenecks move in sync. The honest prediction: most organisations will do neither the forced-scaffolding restructure nor any equivalent redesign. They will drift into de facto delegation with unchanged process architecture. The worst of both worlds — the productivity cost of Faros-style bottlenecks, and the capability cost of skill erosion, arriving together. This is the strategic concern that survives even if Dorsey's flat-org vision does not: the substitution happens by drift, not by design.