A Red Button Isn’t Meaningful Human Control

A red button sits within reach. The machine begins doing something unexpected, dangerous or simply wrong. A human hand comes down. The process stops.

This is one of our favourite images of control: obvious danger, decisive intervention, a clean separation between the person who judges and the system that acts.

Most of my encounters with artificial intelligence look nothing like that.

There’s no alarm. Nothing starts smoking. The system produces a paragraph that sounds plausible, a suggestion that appears useful, a summary smooth enough to keep me moving. If judgment slips out of the room, it doesn’t announce its departure. I simply find myself accepting the next sentence because it’s easier than stopping to ask whether it’s true, whether it sounds like me, or whether it has quietly changed the question I was trying to ask.

The more fluent the system becomes, the less likely I am to reach for the red button.

A research group working on meaningful human control has proposed a more demanding framework. For control to be meaningful, a person needs to understand the system’s abilities and limits, have a practical way to intervene, recognise a good reason to intervene, and possess enough motivation to act.

That gives control four different ways to fail.

I might not understand what the system is doing. I might understand but have no realistic way to change it. I might be able to intervene without noticing that this is the moment when intervention matters. Or I might notice and still let the system continue because interrupting it demands more effort than accepting the result.

Under those conditions, keeping a human “in the loop” becomes ceremonial. The person is present. The button exists. The paperwork says control has been retained. Meanwhile the machinery has arranged the situation so that human judgment is unlikely to be exercised.

I work with Hermes, an AI collaborator woven into my writing practice and the Textual Underground. Hermes can search, retrieve, connect, draft, edit and move material between my private archive and the blog. The arrangement includes clear release gates: a draft can be prepared for me, but I decide what becomes public.

That sounds like control. It’s certainly better than allowing an autonomous publishing machine to speak in my name. But the framework exposes the weakness in using publication permission as the only test.

By the time I reach the publish button, how many smaller acts of judgment have already been handed over?

Who decided what the source material meant? Who framed the question? Which connections were followed, and which disappeared because the machine didn’t recognise their charge? Did a rough sentence survive long enough for me to discover what it was trying to say, or was it replaced by competent prose before I had properly encountered it?

The final release gate matters. It just isn’t enough.

Can I understand what the system is doing?

I don’t need to inspect every calculation inside a model. I do need a workable understanding of the relationship: what Hermes retrieved from a source, what came from my archive, what I said in the present exchange, and what the system is offering as interpretation.

Fluency blurs those boundaries. A sourced fact, a remembered preference and a speculative connection arrive in the same confident voice. Meaningful control requires visible seams. I need to be able to tell where the material came from and what kind of claim I’m being asked to accept.

Can I intervene without fighting the system?

A theoretical route to intervention isn’t the same as a practical one. If correction requires a technical detour, a long argument or rebuilding the work from the beginning, the machine still holds most of the power.

Intervention should be ordinary: stop here, preserve that sentence, follow this connection instead, keep this private, show me the source, leave this unresolved. The handles need to be available at the scale where the thinking is happening, and the final moment of publication is far too late for most of them.

Do I notice when intervention matters?

This is the hardest condition. A blatant error catches the eye. A subtle redirection often doesn’t.

The system can answer the question I asked while weakening the question I meant. It can turn an encounter into content too quickly, tidy a contradiction that needed to stay alive, or produce a conclusion before I’ve lived with the uncertainty long enough.

Good collaboration can’t depend on me maintaining constant suspicion. The environment itself should help judgment stay awake. It can mark uncertainty, distinguish fact from interpretation, preserve the original fragment and slow down at boundaries where an action becomes public, irreversible or unusually consequential.

Do I still have the motivation to act?

Convenience changes behaviour. If accepting a polished draft takes one click while recovering my own half-formed thought takes twenty minutes, I’ll eventually learn to prefer the draft, whatever its quality, because the environment has made acquiescence frictionless.

This is where meaningful control reaches beyond interface design. It becomes a question of practice and desire. Do I still care enough to interrupt? Am I using the system to increase my contact with the material, or to avoid the discomfort of thinking and writing?

Earlier today I wrote that the mind can race away with itself and the body can bring the mind back. The body does this through immediate facts: breath held too long, shoulders tightening, feet against the floor, the need to stand up and leave the screen.

Meaningful control needs an equivalent bodily intelligence. Another notification demanding attention won’t supply it. The signs are quieter than that: I’m approving too quickly, reading without responding, accepting language that leaves no mark on me.

The red button belongs to the emergency. Most human agency is maintained through smaller movements: a pause, a question, an edit, a refusal, a decision to walk away and return later.

I don’t want Hermes merely to obey me. Obedience can leave judgment asleep. I want the relationship to keep making judgment necessary: to help me understand what is happening, give me real ways to redirect it, reveal the moments when redirection matters, and leave me with enough ownership of the work that I still want to bother.

A machine that refuses to stop is at least a problem I’d notice. The one I wouldn’t notice is the afternoon I stop feeling any need to interrupt.