How Much Can an AI Agent Build Without Human Intervention?

An AI coding agent can already write code, run tests, inspect errors, modify files, and iterate. But that doesn't automatically make it autonomous.

The more interesting question is: how much work can an AI agent complete when it has its own persistent environment to operate inside—and nobody is constantly guiding its next move?

That is the question behind this experiment.

Not can an AI write code?

But how far can AI agent automation go before human intervention becomes necessary?

The autonomy gap

A typical AI coding workflow looks deceptively autonomous:

Human
  │
  ├── "Build X"
  ↓
AI Coding Agent
  │
  ├── writes code
  ├── runs tests
  ├── sees error
  └── fixes code
  ↓
Human
  └── reviews / redirects / approves

The agent is doing substantial work, but the human is still acting as the control plane.

Give the agent a proper environment, however, and the loop can become:

Goal
 │
 ▼
Agent
 │
 ├── inspect environment
 ├── plan
 ├── create files
 ├── install dependencies
 ├── execute commands
 ├── observe failures
 ├── modify implementation
 ├── run tests
 └── verify result
       │
       ▼
    Artifact

That difference is the foundation of AI agent autonomy.

The environment doesn't make the model smarter.

It gives the model somewhere to act, observe, recover, and continue.

What existing evidence tells us

There is already strong evidence that AI coding agents can perform meaningful software-engineering work without a human manually writing each step.

SWE-bench evaluates agents against real GitHub issues. SWE-bench Verified is a human-validated subset of 500 tasks, specifically designed to provide a more reliable evaluation set for coding agents.

The benchmark is particularly interesting because the agent isn't simply asked to generate a code snippet.

It receives a repository and an issue and must work toward a patch that resolves the problem.

The current SWE-bench leaderboard reports % Resolved rather than simply measuring generated-code quality. That makes the benchmark much closer to an actual agent workflow: understand the task, operate on a repository, change code, and produce a solution that passes evaluation.

OpenHands' benchmark infrastructure goes even further by running SWE-bench agents inside isolated Docker workspaces. Each benchmark instance gets its own environment, and the agent can iterate inside that workspace before the resulting patch is evaluated.

That is an important architectural signal.

The environment is part of the experiment.

But SWE-bench doesn't answer our actual question

A benchmark can tell us whether an agent solved a predefined task.

It doesn't completely answer:

"What can an agent actually build when you give it an environment and a goal?"

Building something from scratch introduces a different set of problems.

For example:

Can it decide the implementation structure?
Can it install and configure dependencies?
Can it recover from build failures?
Can it test what it created?
Can it notice that its own output is broken?
Can it continue after an unexpected error?
Can it operate for a long enough horizon without a human redirecting it?

These are environment-level autonomy questions, not just code-generation questions.

This is also why newer evaluations are expanding beyond issue fixing. OpenHands' Index, for example, evaluates multiple software-engineering dimensions, including issue resolution and greenfield development.

Our experiment

Instead of asking an agent ten unrelated coding questions, we want to measure a continuous build loop.

The agent gets:

A clean environment
A repository
A natural-language objective
Access to the filesystem
A shell
The ability to install dependencies
Tests
No human intervention during execution

The human only defines the initial goal and evaluates the final result.

Experimental setup

The important part is the feedback loop.

The agent doesn't just generate code once.

It gets to act → observe → modify → test → repeat.

What should we ask the agent to build?

The experiment should use tasks that are small enough to evaluate objectively but large enough to require multiple decisions.

For example:

Task A — Build a web application

Build a small task-management web application with authentication, task creation, editing, deletion, filtering, and persistent storage.

Task B — Build an API

Build a REST API for managing projects and tasks. Include validation, persistence, error handling, tests, and API documentation.

Task C — Debug an existing application

Give the agent an intentionally broken repository and ask:

Identify the failures, fix them, run the test suite, and leave the repository in a working state.

Task D — Extend an existing project

Give it a working application and ask:

Add CSV import, validation, duplicate detection, and an export endpoint without breaking existing functionality.

These tasks are much more useful than:

"Write a Python function."

The objective is to measure end-to-end execution, not autocomplete ability.

The intervention protocol

This is where the experiment gets interesting.

We define four levels of human involvement.

Level	Human involvement
0	Human provides goal only
1	Human can approve dangerous actions
2	Human can answer agent questions
3	Human actively guides implementation

The first experiment should focus on Level 0.

The agent receives the objective and is left alone.

If it gets stuck, that counts as a failure or incomplete run.

No:

"Try using PostgreSQL."

No:

"Your API is returning 500."

No:

"You forgot the frontend."

That would contaminate the autonomy measurement.

What we measure

We shouldn't reduce autonomy to one percentage.

A better evaluation records several dimensions.

1. Task completion

Did the final artifact satisfy the original requirements?

2. Functional correctness

Do the tests pass?

3. Intervention count

How many times did a human have to intervene?

4. Recovery capability

When something failed, did the agent diagnose and recover?

5. Unnecessary changes

Did it modify unrelated files or introduce unnecessary complexity?

6. Execution continuity

Could it continue operating without losing context or requiring a restart?

7. Reproducibility

Can the resulting project actually be installed and run again from the environment?

A simple evaluation record could look like:

{
  "task": "build_task_manager",
  "completed": true,
  "tests_passed": 42,
  "tests_failed": 0,
  "human_interventions": 0,
  "recovery_events": 3,
  "unnecessary_file_changes": 2
}

The important point is that these values should come from an actual run.

They should never be filled with assumptions.

The agent loop

A minimal autonomous coding loop looks surprisingly simple.

while not task_complete:

    state = inspect_environment()

    plan = agent.plan(
        goal=goal,
        environment=state
    )

    action = agent.choose_action(plan)

    result = execute(action)

    observation = inspect(result)

    agent.update_context(observation)

In a real implementation, the loop also needs:

command timeouts
process isolation
filesystem boundaries
network controls
resource limits
logs
checkpoints
failure detection
termination conditions

The loop itself isn't the difficult part.

Making the loop safe and reliable is.

Where autonomy actually breaks

This is where the experiment matters more than the demo.

An agent can be excellent at generating code and still fail at autonomous execution.

Imagine this sequence:

Agent
 │
 ├── creates application
 ├── installs dependency
 ├── dependency fails
 ├── searches documentation
 ├── changes configuration
 ├── application starts
 ├── test fails
 ├── changes implementation
 ├── test passes
 └── declares success

That looks autonomous.

But now introduce:

Database unavailable
        ↓
Environment variable missing
        ↓
Package version conflict
        ↓
Port already occupied
        ↓
Build cache corrupted
        ↓
External API unavailable

The question changes from:

Can the model write the code?

to:

Can the agent operate a system?

That distinction is enormous.

Environment is part of intelligence

This is probably the most important observation from the experiment design.

A model has reasoning capability.

An agent needs state and action.

The environment provides:

persistent files
processes
dependencies
runtime state
tools
logs
network access
test feedback
previous actions

Without those things, an agent repeatedly reconstructs context from text.

With them, the agent can interact with an actual system.

The environment therefore isn't merely infrastructure surrounding the agent.

It becomes part of the agent's operating loop.

What "without human intervention" really means

There is a trap here.

Zero human intervention does not mean the agent is universally autonomous.

It only means:

Given a defined environment, tools, constraints, and objective, the agent can complete a particular class of tasks without requiring a human to guide intermediate decisions.

That's a much more useful definition.

An agent might autonomously build a small application but fail to:

choose a product strategy
validate whether the product is useful
understand ambiguous business requirements
judge legal consequences
decide whether production deployment is safe
recover from unknown infrastructure failures

Autonomy is therefore task- and environment-dependent.

The autonomy ladder

A useful way to think about progress is:

Level 1 — Code generation

The model produces code.

Level 2 — Tool use

The model can edit files and execute commands.

Level 3 — Task completion

The agent can complete a bounded objective.

Level 4 — Environment operation

It can install, configure, debug, test, and manage its workspace.

Level 5 — Long-horizon execution

It can continue working through multiple failures and decisions.

Level 6 — Autonomous systems

It can operate toward higher-level goals with limited human supervision.

Most current AI coding agents are somewhere across Levels 2–5 depending heavily on the task, model, tools, and environment.

Public coding benchmarks demonstrate substantial progress at bounded software-engineering tasks, but they should not be interpreted as proof of unrestricted autonomy. SWE-bench itself evaluates specific repository issues, while newer benchmark efforts explicitly broaden the task space.

The biggest limitation of our experiment

A controlled environment is also an artificial environment.

If we give an agent:

unlimited compute
perfect documentation
deterministic tests
unrestricted internet
clean repositories
no security constraints

we are not measuring production autonomy.

We're measuring autonomy under favorable conditions.

Real systems introduce:

permissions
secrets
flaky services
changing dependencies
production data
cost constraints
security boundaries
ambiguous requirements
irreversible actions

That's why the next stage of research should progressively make the environment more realistic.

What we actually learned

The important conclusion isn't:

"AI agents can build software without humans."

We already have evidence that coding agents can solve meaningful software-engineering tasks.

The more useful conclusion is:

The amount of autonomy an AI agent can achieve is strongly coupled to the environment in which it operates.

A model that can only generate text has limited agency.

A model connected to a shell can execute.

A model connected to a persistent workspace can iterate.

A model connected to tests can verify.

A model connected to logs can diagnose.

A model connected to a controlled runtime can keep operating.

And a model connected to all of these can begin behaving less like a chatbot and more like a software worker.

That's the real infrastructure problem.

So, how much can an AI agent actually build?

More than a code-generation demo suggests.

Less than the phrase "fully autonomous developer" suggests.

The boundary isn't fixed.

It moves with three variables:

          Model capability
                 │
                 ▼
        ┌─────────────────┐
        │   AI Agent      │
        └─────────────────┘
          ▲             ▲
          │             │
   Environment      Evaluation
          │             │
          └──────┬──────┘
                 ▼
          Actual autonomy

Better models increase what the agent can reason about.

Better environments increase what the agent can actually do.

Better evaluation tells us whether it really worked.

And that leads to a broader idea behind AI agent infrastructure:

If agents are going to do real work, giving them a model isn't enough. They need somewhere to work.

That environment may eventually become as important to AI agents as the development environment is to human engineers.

And that is exactly why the next question isn't simply how capable is the model?

It's:

What should an environment look like when the primary worker inside it is an AI agent?

Methodology note

This article deliberately separates published evidence from the proposed Causly experiment.

SWE-bench Verified and OpenHands provide externally reproducible evidence that agents can perform substantial software-engineering tasks inside executable environments.

The autonomy protocol described above is a research methodology, not a claim that Causly has already achieved the listed results. Any Causly-specific completion rate, intervention rate, cost, or reliability number should only be published after actually running the experiment and recording the raw runs.

That distinction matters.

If we're researching autonomous agents, the methodology itself has to be trustworthy.

Get insights in your inbox

How Much Can an AI Agent Build Without Human Intervention?

How Much Can an AI Agent Build Without Human Intervention?

The autonomy gap

What existing evidence tells us

But SWE-bench doesn't answer our actual question

Our experiment

Experimental setup

What should we ask the agent to build?

Task A — Build a web application

Task B — Build an API

Task C — Debug an existing application

Task D — Extend an existing project

The intervention protocol

What we measure

1. Task completion

2. Functional correctness

3. Intervention count

4. Recovery capability

5. Unnecessary changes

6. Execution continuity

7. Reproducibility

The agent loop

Where autonomy actually breaks

Environment is part of intelligence

What "without human intervention" really means

The autonomy ladder

Level 1 — Code generation

Level 2 — Tool use

Level 3 — Task completion

Level 4 — Environment operation

Level 5 — Long-horizon execution

Level 6 — Autonomous systems

The biggest limitation of our experiment

What we actually learned

So, how much can an AI agent actually build?

Methodology note