The Next Autonomous-Weapon Failure May Be A Drone That Obeys Every Rule – OpEd

Key Takeaways:

• The author argues the danger is not a “rogue” drone but an obedient one that keeps full authority after comms, nav, or sensors fail—as in the reported July Molniya strike where humans set a general aim and onboard AI picked the final object.
• Design rule: an “authority envelope” that contracts when evidence worsens and recovers only slowly with independent checks (DoD 3000.09 already wants judgment and validated limits); a human “in the loop” is not in control if they cannot see uncertainty, sources, or abort without penalty.
• Geneva CCW GGE (Aug. 31–Sept. 4) and Guterres/ICRC calls need testable permission rules and audit trails—not “ban killer robots” vs. “keep a human in the loop.”

As governments return to Geneva to debate autonomous weapons, the hardest control problem is not a machine that goes rogue. It is a machine that faithfully follows permissions that should have narrowed when battlefield evidence deteriorated.

The most frightening image in the debate over autonomous weapons is also one of the least useful: the machine that goes rogue. A drone rejects its human operator, escapes its constraints, chooses its own target and turns science fiction into battlefield reality.

Recent reporting from Ukraine has made that image more powerful. A Russian Molniya drone involved in a July strike in Zaporizhzhia that killed three civilians was reportedly equipped with onboard computing that allowed it to perform part of its final targeting autonomously. According to forensic reporting summarized by Tom’s Hardware, human operators launched the drone toward a general objective, while onboard AI apparently selected the final object it attempted to strike.

That is a serious development. But focusing on whether a weapon “goes rogue” can obscure a more ordinary and potentially more important failure.

A weapon does not need to rebel against its instructions to escape meaningful human control. It may only need to follow them perfectly after the conditions that justified those instructions have disappeared.

Imagine a drone launched on a commander-approved mission. Its communications link drops. Navigation confidence deteriorates. Smoke, camouflage, electronic warfare or a decoy makes the sensor picture less reliable. The software does exactly what engineers designed it to do: it switches to onboard guidance and continues.

Nothing has escaped. Nothing has developed a motive. Nothing has become conscious. The system may be functioning precisely as specified. The governance question is whether it should still possess the same authority.

The Problem Is Permission, Not Personality

Rebecca Gordon recently argued that lethal AI systems are outrunning the institutions expected to hold humans accountable for their use. In her CounterPunch essay, she cited my July Articles of War analysis of drone swarms and the accountability gap in Ukraine, particularly the argument that the law of armed conflict still applies but responsibility must be examined earlier when the architecture distributing decision-making authority is designed rather than reconstructed only after a weapon strikes.

That point matters because autonomy is not one thing.

An AI system that recommends a target has one kind of authority. A system that ranks targets for an operator has another. A system permitted to select an object autonomously has more. A system that can complete an engagement after losing communication with the responsible human authority exercises something more consequential still. A swarm creates another layer of complexity because a lethal outcome may emerge from interaction among several machines rather than from one identifiable decision point.

These distinctions are not technical trivia. They determine where human authority actually resides.

Every autonomous weapon already operates inside what can be understood as an authority envelope: a bounded set of actions the system is permitted to perform under stated conditions.

This is not a weapons-only principle. In an August 18 Techstrong.ai op-ed on the AI Kill Switch Act, I used authority envelope for the enforced boundary around an agentic runtime: the tools, credentials, networks and execution domains it may reach. The principle: autonomy should operate inside permissions it cannot silently rewrite.

This authority-envelope problem is also the subject of my forthcoming peer-reviewed article in Digital War, “The Authority Gap: Why Meaningful Human Control Fails at the Tier Where It Is Signed.” There I call it the terminal-systemic authority gap: the distance between the tier where a human formally authorizes an action and the upstream tiers where decisive reasoning has already been produced. If those tiers cannot be reviewed in time, a human signature can remain formally present while substantive control has already thinned.

The central governance question should therefore not be whether the machine is obedient.

It should be: Are the permissions it is obeying still justified by the evidence available to it?

Authority Should Contract When Evidence Gets Worse

This leads to a principle that should be much more explicit in autonomous-weapons design: when evidence deteriorates, autonomous authority should contract.

Suppose a weapon has an authenticated communications link, independently corroborated sensors, reliable navigation and a clearly classified object. That combination of evidence may justify one level of delegated authority.

Now remove the communications link. The system should not necessarily retain exactly the same permission state. Allow navigation integrity to fall below its validated threshold. Authority should contract again. Let sensors disagree about the target. Contract again. Introduce an environmental condition outside the system’s validated operating domain. Contract again.

This is fundamentally different from designing a weapon with one binary switch marked autonomous or not autonomous. Authority should be dynamic because the evidence supporting authority is dynamic.

The U.S. Department of Defense already recognizes part of this logic in Directive 3000.09. The directive requires relevant autonomous weapon systems to be designed so commanders and operators can exercise appropriate levels of human judgment over the use of force, and requires appropriate performance under realistic conditions. It also requires systems covered by its senior-review process to operate within stated time, geographic, environmental and operational parameters.

That is important. But the engineering principle should be made unmistakable: permission should follow evidence.

The same peer-reviewed analysis yields a design consequence: when meaningful review becomes impossible, authority should contract rather than persist by default. That makes human control a property of the architecture, not merely a promise at the final click.

The loss of evidence should never silently produce the inheritance of authority.

A Human Can Be in the Loop and Still Not Be in Control

This also exposes the weakness of one of the most familiar phrases in the debate: human in the loop. Putting a human somewhere in a process does not prove that the human controls it.

A human may receive a machine-generated recommendation after classification, target matching, proportionality inputs and other calculations have already occurred. The operator may technically possess the power to reject it.

But does the operator have enough time?

Can the operator examine the evidence underneath the recommendation?

Can the operator see uncertainty rather than merely a confidence score?

Can the operator determine whether apparently corroborating sensors actually depend on the same underlying source?

And can the operator realistically stop the process without creating an operational penalty so large that approval becomes the expected default?

If those answers are no, the human may be present while substantive control has already migrated elsewhere.

This is why the Articles of War argument was not that international humanitarian law has somehow become obsolete in autonomous warfare. The opposite is true. Responsibility remains human. The difficulty is architectural: autonomy can distribute the decisions that produce an engagement among designers, commanders, data, software, operators and pre-authorized mission rules long before the final moment of force.

Searching for responsibility only at the last click therefore risks looking at the wrong layer.

Contract Quickly. Recover Deliberately.

There is another problem that receives even less attention. If a weapon loses authority because evidence has deteriorated, what allows it to recover that authority? Suppose communications disappear and the system shifts into a constrained mode. Thirty seconds later the datalink returns.

Should full autonomous authority immediately return too?

Probably not.

One restored signal does not establish that the conditions supporting the original permission state have been restored. Navigation may still be unreliable. The target picture may still be stale. The operating environment may have changed. The communications link itself may be intermittent.

This suggests a second design principle: contract quickly; recover deliberately.

Permission should be easy to lose when the evidence supporting it fails. Higher authority should be harder to regain.

Recovery should require pre-defined evidence: perhaps multiple independent sensor confirmations, re-established navigation integrity, authenticated communications or renewed human authorization, depending on the consequence involved.

That creates an asymmetric safety rule. Systems move downward in authority rapidly under uncertainty. They move upward only when evidence earns the permission back.

This is not a prohibition on autonomy. It is a discipline for using autonomy responsibly in adversarial environments.

Geneva Needs an Engineering Answer

The timing matters.

The Convention on Certain Conventional Weapons’ Group of Governmental Experts on lethal autonomous weapon systems meets again in Geneva from August 31 to September 4. Its mandate is to continue formulating, by consensus, elements of an instrument and other possible measures governing emerging technologies in lethal autonomous weapon systems.

The diplomatic discussion is already converging on an important principle. The Chair’s summary from the first 2026 session recorded broad support for the position that responsibility and accountability cannot be transferred to machines, and that autonomous weapon systems should operate within a responsible chain of human command and control.

On August 25, UN Secretary-General António Guterres and ICRC President Mirjana Spoljaric renewed their call for new international rules, warning that the gap between technological capability and regulatory constraint has widened and that the autonomous targeting of humans risks crossing a moral red line.

Those are significant normative statements.

The difficult next step is turning them into behavior that can be specified, tested and verified.

States are unlikely to allow foreign inspectors to examine classified targeting algorithms, model weights, sensor-performance envelopes or operational doctrine. They do not necessarily need to. A verification regime can begin with narrower questions about authority behavior.

Which classes of action require fresh human authorization?

Under what evidence conditions must autonomous permissions narrow?

What happens when communications, navigation or sensing degrade?

Can a system continue selecting or engaging targets after entering an uncertain environment?

What must be true before higher permissions return?

Can an audit record reconstruct what the system was allowed to do, what evidence supported that permission and which human authority approved the relevant boundary?

These are testable questions.

They do not require an inspector to read the model’s mind. They require the system to demonstrate that certain transitions are allowed and others are impossible under declared conditions.

That makes authority architecture relevant not only to weapons design, but potentially to arms-control verification.

Beyond “Ban Killer Robots” and “Keep a Human in the Loop”

The international argument is often presented as a choice between two positions.

One side says: ban killer robots.

The other says: keep a human in the loop.

Both slogans capture legitimate concerns. Neither is sufficiently precise for the systems now moving onto battlefields.

The real questions are harder.

Where is authority located? How much has been delegated? Which evidence justified that delegation? What happens to permission when the evidence deteriorates? Which decisions remain human-reserved? Who approved the boundary? Can investigators later reconstruct whether the system remained inside it?

Those questions move autonomous-weapons governance away from speculation about machine psychology and toward something governments can actually regulate.

We do not need to establish whether an AI intended to deceive. We do not need to determine whether a drone “wanted” to continue a mission. And we do not need to solve machine consciousness before writing meaningful constraints.

States already regulate what machines are permitted to do. The challenge is making those permissions responsive to battlefield reality.

The Most Dangerous Machine May Be the Obedient One

Rebecca Gordon is right that accountability begins earlier than the instant a weapon strikes. The next step is to make that insight executable.

A future autonomous-weapon failure may indeed involve software behaving unexpectedly. But it may also be much more mundane. The machine may follow every instruction. Its sensors may deteriorate exactly as engineers anticipated. Its communications may fail in a known way. Its fallback mode may activate correctly. Its software may remain completely inside the constraints humans gave it.

And civilians may still die because those constraints allowed too much authority to survive after the evidence supporting them disappeared.

That is why the governing question is not simply whether artificial intelligence can be trusted to obey. It is whether humans designed what it is allowed to obey carefully enough in the first place.