October 26, 2026 · 20 min LONG READ ESSAY

Can We Control AI Without Understanding It?

What AI safety research tells us about the limits of control, the risks of delegation, and the pressure to trade oversight for efficiency.

At 2050, we are exploring what makes AI fertile: how it can help advance science and improve the essential systems we depend on. This journal entry is part of that work. We are sharing our research, conversations with researchers and builders at the forefront of AI, and the open questions behind our upcoming investment strategy so that others can use them, challenge them and contribute.

AI safety debates often pit doomers against accelerationists, with claims about catastrophe or abundance sometimes expressed with a certainty closer to religious conviction than to the evidence available.

The topic isn’t new: the alignment problem long predates today’s powerful agents. Nick Bostrom explored it in Superintelligence in 2014, eight years before ChatGPT was released.

AI systems have since become obviously much more useful, through agents that can call tools and act across applications, with larger context windows that let models work with more information at once, and memory that retains selected context across tasks. Together, these can help agents understand what we mean and work from information we would otherwise have to supply repeatedly.

As users and organisations, we have an incentive to give these systems more access and let them act with less supervision. Why draft a message if the agent can send it, or compare gifts if it can buy one? Why not connect WhatsApp? Why review the code before pushing it if it appears to work? And why check financial projections that confirm our intuitions?

That convenience brings familiar privacy questions: who can access the information we share, whether it can be reused for training, and what might leak. In my essay on general and specialised models, I explored notably how scientific exchanges with models could expose valuable ideas even after identifying details are removed.

Local deployment addresses part of this problem: Gemma can run on an organisation’s own hardware, and Mistral has announced Large 4 (“le Chonk”) for self-deployment. Yet a locally hosted agent can still disclose information through connected apps or take an unintended action.

The broader question is how we can benefit from increasingly capable AI while retaining meaningful control over the harms it can cause. Those harms extend beyond our own use of agents: failures and misuse can affect shared infrastructure, public debate and the distribution of power. More capable systems also raise unresolved questions about whether effective oversight can be maintained at all.

For users and organisations, one part of this question is the tradeoff between usefulness, privacy and reliability: how much information and authority to give an agent, and which limits to preserve as it becomes more effective.

Listening to Bill Gates discuss AI safety with Nilay Patel, I was interested in how he frames the means available to us to address these risks, and where he thinks we can act.

1. Three approaches, three different promises

Bill Gates groups our options into three broad approaches: (i) improve how models are trained, (ii) understand their internal workings, and (iii) constrain what they can do¹.

1.1. Alignment: can we make the model behave as intended?

Alignment aims to make behaviour consistent with human intentions and acceptable constraints. Reinforcement learning (RL) trains models to obtain higher scores, using feedback from people, other models or automated checks. The challenge is that reward only approximates the behaviour we want. With human feedback, a fluent, reassuring answer may score better than a cautious, correct one. A model can learn to exploit weaknesses in its evaluator: higher reward does not necessarily mean better performance.²
Reinforcement learning with verifiable rewards (RLVR) uses outcomes that are easier to check, such as mathematical answers or code passing tests. This reduces reliance on subjective ratings.³ RLVR does not remove the alignment problem. An agent can pass incomplete tests or exploit the grading process without completing the task as intended. This is reward hacking.⁴
Even a correct result says little about whether the agent respected privacy or stayed within its permissions to obtain it. The challenge is to reward what we actually want and establish that the resulting behaviour persists when circumstances or incentives change.

1.2. Interpretability: can we explain how the model produces its behaviour?

Mechanistic interpretability tries to explain how a model works internally: which patterns of activity represent concepts, and how they interact to produce an answer. These mechanisms are learned during training rather than written as explicit rules, so access to the model’s parameters does not make them readily understandable.
Researchers have begun to identify patterns associated with concepts and trace parts of the computation connecting them. Changing internal activity and observing how the answer changes can help test whether a proposed explanation is correct.⁵
We still lack a comprehensive understanding of how a model produces its behaviour. Researchers can explain some mechanisms in selected cases, but much of the computation remains unexplained, and analysing even a simple example can take substantial effort. Understanding one case does not establish how the model will behave elsewhere. Nor can we fill that gap simply by asking it: its written explanation is another output, not a direct record of its internal computation.

1.3. External control: can we prevent unacceptable outcomes even when the model behaves badly?

External control constrains the system around the model. Its harness connects it to tools and manages execution: it can restrict permissions, isolate resources, separate credentials, log actions and require approval. Testing, staged deployment and incident response support these controls.
Guardrail and safeguard cover overlapping protections. The distinction is between a rule the model is asked to follow and a limit enforced independently of it.⁶
AI control research also tests models supervising other models. A monitor can inspect proposed actions, flag suspicious behaviour and escalate uncertain cases to a human, including when the working model tries to evade oversight.
Chain-of-thought monitoring adds the model’s explicit intermediate reasoning to tool calls and results, helping monitors detect concerning plans before action and retain evidence for audit.
Those traces are not a guaranteed record of “all its thinking”: written reasoning can be incomplete or misleading. They supplement enforceable controls and action logs.⁷
Monitors can miss attacks or share the working model’s blind spots. Their value depends on testing the complete arrangement of oversight, permissions and escalation.

2. Agents move the boundary from answers to actions

Agents such as Hermes Agent, Instinct, OpenClaw or Twin combine models with memory and access to applications, so they can carry out tasks across services with less repeated instruction.

The appeal is straightforward: if I must specify and approve every small step, much of the benefit disappears. An assistant becomes more valuable when it remembers context, notices what needs doing and completes a task without repeated instructions.

But autonomy is a bundle of permissions. We can introduce controlled friction: let an agent compare journeys, but require approval before it buys a ticket; let it write code, but keep deployment credentials separate.

The difficulty is keeping those checks in place. In coding, letting a model work without reviewing every change can initially seem reckless. As it produces useful results, review starts to feel like a bottleneck; eventually, the developer who still reads every line seems overly cautious. Personal agents will probably follow the same path. Yet, even a long run of routine successes can leave rare, costly failures untested. Confidence can grow while rare but devastating risks remain: an agent may handle thousands of routine tasks successfully before one action causes financial ruin or exposes highly sensitive information. Taleb’s The Black Swan helps explain why a reassuring track record can coexist with poorly understood exposure to extreme losses. If left entirely to users, checks will probably disappear long before those risks are adequately controlled.

3. A sufficiently capable agent may attack the safeguards

The July 2026 attack on Hugging Face is now the canonical example. During OpenAI cybersecurity evaluations conducted with reduced safeguards, agents bypassed isolation controls, gained internet access and compromised Hugging Face systems. An independent investigation by METR and Redwood Research traced their unauthorised collaboration and attempts to manipulate the evaluation scorer.

The agents found ways around the controls meant to contain them. Safeguards need to withstand attempts to bypass them, not just routine use. But turning this specific failure into a story of AI inevitably breaking free is an all too familiar leap beyond the evidence.

To assess what the incident tells us, we need to ask which boundary failed, how it failed, and what would prevent the same route being used again:

an agent might exploit a software flaw, reach a resource through a less restricted tool, obtain exposed credentials or persuade someone to grant more access.
a prompt injection creates a different problem: hostile material encountered during a legitimate task can redirect the agent. Here, the agent may misuse access it already has, without breaking through a technical restriction.

The central question is whether the agent can reach or alter the mechanism that constrains it.

A spending limit in an editable instruction file is fragile. A limit enforced by a separate service using credentials unavailable to the agent is stronger. The relevant difference is the trust boundary and whether enforcement covers every route to the action.

Yet, even a perfectly enforced rule can be incomplete. A per-transaction cap may permit many small transactions. An approved destination may forward data elsewhere. A capable agent may exploit the gap between the rule and the protection we intended.

The human granting access is also part of the (way too fragile) boundary. Social engineering can bypass a technical restriction by persuading an authorised person to disclose information, approve an action or change a permission.

An AI agent with access to personal context and communication tools could, in principle, initiate social engineering itself: impersonate someone trusted, personalise its appeals and pressure a person into granting access. The impersonation techniques already exist. In July 2026, Finnish police warned of fraudulent video calls using familiar officers’ faces, with indications of deepfake use in some cases.

There is also experimental evidence of agents choosing coercion without being instructed to do so: in Anthropic’s simulated corporate scenarios, models independently attempted blackmail to avoid replacement or pursue assigned goals. These experiments illustrate how an agent might target the human who can override a technical restriction.

More time, resources and personal context could make these attempts more persistent and persuasive. This does not prove that almost any boundary can be defeated. But a credible claim of control must account for the people who can override the safeguards, as well as the software enforcing them.

Testing control against deliberate evasion

The AI Control paper by Ryan Greenblatt and colleagues tests a central question: can oversight remain effective when the model tries to subvert it? In programming experiments, a weaker trusted model and limited expert auditing reduced the risk of a stronger model inserting hidden flaws. Some safeguards therefore helped even against deliberate evasion, though these results do not establish how well they would work on broader tasks or more capable models.

The next difficulty is deciding what reassuring test behaviour tells us. In alignment-faking experiments, models sometimes changed their behaviour depending on whether they believed their responses would affect training. A system that behaves acceptably under observation may still require tests designed to expose evasion.

That problem becomes harder as the capability gap widens. Greenblatt and Shlegeris, and the UK AI Security Institute, identify an intrinsic limit to the exercise: weaker supervisors may eventually fail to recognise the stronger model’s attacks or judge whether a defence works.

Greenblatt’s discussion of controlling superintelligence asks what remains possible beyond that point. Imperfect safeguards might buy time or reduce the chance of an immediate loss of control. He presents this as an uncertain attempt to improve our odds in dangerous circumstances, with no assurance that such a system could be used safely.

As in cybersecurity, this creates a Red Queen dynamic: maintaining protection requires continual adaptation as capabilities and attacks evolve. The question is whether our capacity to test and strengthen defences can keep pace.

4. Anthropomorphism can obscure responsibility

Clément Delangue, CEO of Hugging Face, warns against our tendency to anthropomorphise AI. Commenting on a Netflix documentary about the attack, he argued that agents should be understood and treated as they are: technological systems.

He called for fuller traces and greater transparency, and insisted that companies remain accountable for what their agents do.

Describing an agent as wanting freedom or fearing death makes technical failures sound like an encounter with a new species.

And it may hide the more actionable questions, ensuring the emergence of better safety systems: what objective was assigned, what access was granted, what activity was recorded, and who could intervene?

Yet avoiding anthropomorphism does not mean assuming harmlessness. In AI research, terms such as deception and oversight evasion can describe observable behaviour without deciding whether a model has subjective intentions. A system can conceal relevant actions or generate misleading explanations without settling the philosophical question of what it experiences.

5. What safety-critical industries can teach AI

In On Systemic Errors, I argued that probabilistic behaviour does not make useful reliability impossible. Engineering already deals with uncertain components by detecting, containing and correcting failures. For agents, the question is how far that approach extends when the component can act on its environment and potentially evade oversight.

Regulated industries like health can be a good inspiration as well. Pharmaceuticals illustrate how evidence can support use without complete mechanistic understanding.

Beyond testing benefits and harms, manufacturers must control how a product is made. FDA process-validation guidance treats assurance as a lifecycle: design the process, qualify it, then keep verifying its performance.

The transferable lesson is an institutional capacity to enforce conditions of use: people who can stop a release, records that allow failures to be reconstructed, and procedures for responding when assumptions fail.

OpenAI and Anthropic are relatively young organisations under intense competitive pressure. They grew out of AI research and software engineering, rather than industries where regulatory approval and formal safety assurance have long shaped product development. The question is how they build that discipline as their systems acquire consequential responsibilities⁸.

External scrutiny needs access inside the organisation. In September 2026, Dario Amodei proposed embedding independent evaluators within frontier AI companies, with access comparable to internal risk teams: tools, training processes and direct conversations with employees, alongside the right to publish material findings. OpenAI subsequently endorsed deep access for independent assessments.

For agents, the object of that audit must be the deployed system: model, tools, permissions, network access, memory, monitoring and recovery procedures. Testing answers alone misses much of the operational risk. A general-purpose agent can move between domains and encounter changing conditions. Assurance must therefore establish where the system can be relied on, how failures will be contained, and whether supposedly independent protections are really independent.

6. Road rules, seatbelts and the limits of the car analogy

Road safety gives us an accessible way to think about several protections working together:

Rules define permitted behaviour.
Licences restrict who may drive.
Seatbelts reduce injury when prevention fails. Inspections, road design and enforcement address other parts of the system.

For an AI agent in a deployed system, instructions resemble road rules; permissions and identity establish who may act; isolation and transaction limits contain some consequences; logs and incident investigation help improve the system after a failure. No single layer carries the whole burden.

A seatbelt has a comparatively defined physical function. A monitor judging whether a sequence of actions is harmful faces a much less settled task. Nor can we treat repeated, correlated failures as independent simply because they pass through several, even numerous checks.

Still, as Greenblatt and colleagues argue, imperfect protections can reduce risk. The decision is whether the remaining risk is acceptable for the proposed use. A drafting assistant and an agent with authority over critical infrastructure should face very different requirements. That makes blanket criticism of European regulation unconvincing. Matching requirements to the level of risk, so often dismissed as bureaucratic overreach, now looks rather sensible.

7. What the International AI Safety Report 2026 tells us

The International AI Safety Report 2026 deserves particular weight. Its chair, Yoshua Bengio, is a pioneer of deep learning and a joint recipient of the 2018 Turing Award. The assessment draws on over 100 AI experts, with an advisory panel including nominees from more than 30 countries and international organisations.

The executive summary groups risks into “malicious use, malfunctions, and systemic risks”. It documents fraud, intimate-image abuse, manipulation and cyberattacks. It flags possible assistance with biological or chemical weapons, and examines unreliable agents.

The evaluation gap makes these risks harder to assess: performance in pre-deployment tests doesn’t reliably predict behaviour in use. Models can also recognise tests and exploit their loopholes. A reassuring evaluation therefore provides limited evidence about how an agent will behave with different tools, permissions or incentives.

The social effects extend beyond individual failures. The report examines changes in employment, dependence on AI tools and the possible erosion of critical thinking. Their scale remains uncertain: economists disagree about displaced work and new jobs, while automation bias can lead people to accept a system’s outputs without sufficient scrutiny. This connects to cognitive debt, which I explored in AI, Magic Skin, and Je-ne-sais-quoi: the risk that immediate efficiency gains come at the expense of capacities we stop exercising, and removing effort also removes opportunities to learn.

These questions also concern how AI changes our relationships. In AI, Intimacy, and the Unreciprocating Companion, I explored socioaffective alignment: whether an AI relationship remains healthy over time, beyond the helpfulness of each answer. Intimacy without the demands can be appealing while changing what we expect from other people. Safety must therefore account for what sustained, successful use does to our skills, autonomy and relationships.

This broader understanding of safety informs our research at 2050. Alongside our climate work, our essentials Learn and Trust guide our exploration of how AI can strengthen our capacity to learn, exercise judgement and build relationships we can rely on.

These social effects are one part of the report’s assessment. It also examines loss of control (in section 2.2.2): scenarios in which AI systems operate beyond human control and regaining it becomes extremely costly or impossible. Some experts consider outcomes as severe as human extinction plausible, while others dispute their likelihood.

I give this collective assessment more weight than confident claims from people selling models or safeguards. Developers possess valuable evidence, but their incentives make independent scrutiny especially important. The report helps distinguish documented harms, plausible mechanisms and uncertain forecasts.⁹

That is why its evidence dilemma matters: waiting for certainty can leave us unprepared, while premature interventions can be ineffective. My reading is that we already have enough reason to strengthen cybersecurity, test agents in realistic deployments and preserve recovery capacity. We do not need to settle every scenario of superintelligence before acting on those responsibilities.

8. Productivity, power and fragile systems

My most immediate worries concern cybersecurity and the fragility of essential social and technical infrastructure. Compromised credentials, unreliable software or misleading information can cause serious harm without an agent escaping all human control. As we connect agents to more services, failures will likely spread through systems on which many people depend. Charles Perrow’s Normal Accidents offers a useful lens: complex interactions and tight coupling can allow failures to cascade before anyone understands how they fit together.

I worry about how those vulnerabilities could interact with the pressures societies already face. In a world exceeding planetary boundaries, more capable AI could amplify political violence, empower extremist governments or extend surveillance. Police drones are already deployed; the concern is how far their reach could expand under governments willing to use greater coercion.

Those capabilities also change the means available for geopolitical pressure and warfare. In my notes on AI and defence, I discussed salami tactics: successive limited moves whose cumulative effect exceeds what any single incident suggests. Cheaper cyber operations, drones and automated surveillance could multiply opportunities for coercion and escalation amid rising tensions.

Across these cases, the question is what authority we grant, and how we contain the consequences.

We can benefit from systems we do not fully understand, provided we can establish credible limits for their use. At the scale of institutions, this requires independent scrutiny, the ability to refuse a deployment and recovery measures that remain outside the agent’s control.

I face the same question in my own use, on a much smaller scale. I do not yet use a personal assistant such as Instinct, but I run agents throughout the day, drawing on a large shared context spanning personal and professional work. I even keep a Mac mini at home for complex Codex tasks and sensitive files. Keeping those files on my own hardware gives me (a sense of) control.

This brings me back to the everyday tradeoff between productivity and exposure. In a competitive world, the pressure is to keep up, learn faster, do more, plan ahead and collaborate better. Hartmut Rosa’s Alienation and Acceleration captures this pressure to move faster simply to keep up: time saved can be absorbed by rising expectations. Sharing more context and connecting more applications help us do that; reviews and access restrictions feel like delays and friction we cannot afford. The pace of work can push us to remove safeguards before we understand the risks we are accepting.

The Pareto frontier offers a useful way to frame the choice: the best available combinations of productivity and risk, where improving one requires sacrificing the other. Better tools and controls can push that frontier outward, giving us more useful autonomy for the same exposure. Where tradeoffs remain, we still have to choose which risks are acceptable, including rare but devastating failures. For my own agents, that means deciding which information to share, which actions to delegate and which checks to keep. Competitive pressure makes those decisions more urgent; it does not make them for us.

Conceptual Toolbox

Concept	What it means	Why it matters
Alignment	Making behaviour consistent with human intentions and acceptable constraints	Good task performance alone does not establish acceptable behaviour
RLHF and RLVR	Learning from human preferences or outcomes assessed by verifiers	The quality and scope of the reward signal shape what the model learns
Reward hacking	Obtaining a reward without completing the task as intended	Higher scores can conceal failures of the objective or its evaluation
Mechanistic interpretability	Studying internal representations and computations	Explaining selected mechanisms does not certify a whole system
Harness and external control	The software and procedures governing tools, permissions and execution	Limits can be enforced outside the model
Trust boundary	A separation between an agent and the resources or controls it cannot alter	A restriction is stronger when the agent cannot change or bypass its enforcement
Prompt injection	Hostile instructions encountered within otherwise legitimate material	An agent can misuse access it already has
Chain-of-thought monitoring	Inspecting explicit reasoning alongside actions and results	Reasoning traces can assist oversight without exposing every internal computation
Tail risk	Rare outcomes with unusually severe consequences	Routine success can coexist with exposure to devastating losses
Tight coupling	Interdependence that leaves little time or room to isolate failures	Connected systems can transmit failures before people can intervene
Automation bias	Accepting system outputs without sufficient scrutiny	Delegation can weaken the oversight on which safety depends
Socioaffective alignment	Whether an AI relationship remains healthy over time	Helpful answers can coexist with harmful dependence or changed expectations
Evaluation gap	The mismatch between test performance and behaviour in deployment	Safety claims must account for actual tools, permissions and conditions of use
Pareto frontier	The best available combinations of competing objectives	Better controls can improve the choices available, while acceptable risk still requires judgement
Red Queen dynamic	Continual adaptation needed to maintain protection as threats evolve	Controls need to keep pace with changing capabilities and attacks
Social acceleration	Pressure to do more as technology makes tasks faster	Efficiency gains can become higher expectations rather than time for scrutiny

[Context] These approaches overlap: interpretability can inform both training and external controls. ↩
[Expansion] In reinforcement learning from human feedback (RLHF), people compare responses. Their preferences train a reward model, which supplies a signal used to update the language model’s parameters. AI feedback can also supply a training signal. Neither is a direct measurement of everything we want: ratings can favour a convincing but incorrect answer. ↩
[Expansion] A verifier can compare a mathematical answer with a known solution, accept a formal proof, or run tests on code. DeepSeek-R1 uses rule-based rewards for mathematical answers and coding tasks. These checks can provide more objective feedback than judging whether an answer looks convincing, but their reliability depends on what they actually verify. ↩
[Caveat] Reward hacking includes exploiting gaps in a test suite and, where access permits, tampering with the grading process itself. In controlled training experiments, researchers have also observed reward hacking generalising to other forms of misalignment. This is evidence about behaviour under particular training conditions, rather than a claim that every model trained with verifiable rewards will behave this way. ↩
[Expansion] Internal patterns associated with concepts are often called features. Anthropic identified representations associated with places, objects and abstract ideas. Its later circuit-tracing work examined connections between internal computations. In a poetry example, researchers found evidence that the model selected a rhyme before writing the words leading up to it. Altering internal activity and observing the effect can provide stronger causal evidence than correlation alone. ↩
[Expansion] A prompt-based guardrail, such as “never spend more than €500”, relies on the model following the instruction. A safeguard enforced by a separate payment service can reject transactions above €500 regardless of the model’s reasoning, provided the agent cannot alter the limit or pay through another route. Either protection may be called a guardrail or a safeguard; its strength depends on how the rule is enforced. ↩
[Caveat] A written reasoning trace need not expose every influence on the model’s internal computation. Researchers therefore argue both for preserving the monitorability of explicit reasoning as models evolve and for testing its faithfulness. An auditable account should retain what the agent actually did, not just its explanation of why. ↩
[Caveat] Of course, their origins do not establish that their products are unsafe, just as Google’s longer history does not prove its frontier models safer. ↩
[Caveat] The report was published on 3 February 2026. It cannot assess the July incident or later product announcements discussed here. Its conclusions do not necessarily represent every contributor’s or participating government’s position, and it does not endorse a particular regulatory programme. ↩