It will soon be too late to regulate use of autonomous weapons, warns Red Cross

The International Committee of the Red Cross and rights groups have warned that it will soon be too late to put the "genie back in the bottle" as the development of autonomous weapons appears to outpace efforts to regulate them.

The stark warning came as more than 120 member states of the UN Convention on Certain Conventional Weapons concluded a week-long conference in Geneva.

Despite mounting pressure, an agreement to discuss an instrument to regulate autonomous weapons over the next two years fell short of the urgent action rights groups said was needed to address the looming threat.

"We're starting to see artificial intelligence … being fielded in the battlefield and perhaps also already in weapon systems. If we leave this unregulated, it will be much more difficult to put the genie back in the bottle," said Dr Cordula Droege, the director of international law, movement and partnerships at the ICRC.

Autonomous weapons are military systems that act without human supervision, often through the use of artificial intelligence or other forms of robotics.

As artificial intelligence has grown exponentially in recent years, militaries and manufacturers have created increasingly advanced weapons systems for both offensive and defensive purposes.

These systems include the use of AI in generating target banks, as was done by the Israeli military in Gaza, as well as semi-autonomous drones in the Ukraine-Russia war, which have "last-mile capability", where AI takes over in the drone's final approach to its target.

The president of the ICRC, Mirjana Spoljaric Egger, and the UN secretary general, António Guterres, have called for a legally binding instrument with clear prohibitions and restrictions on use of weapons. In particular, the ICRC has called for a ban on the use of autonomous weapons against personnel, as well as ensuring that weapons systems behave predictably and placing limits on weapons including restricting their use to military objectives.

Experts have said that given the rapid development of autonomous weapons and AI, the failure to create adequate regulations could have dangerous repercussions.

"It's not about banning all autonomous weapon systems, but to set limits in an international treaty of red lines that should never be crossed," said Droege. "We're very close to crossing those red lines, particularly targeting human beings through machines should be a moral red line."

Other rights groups warned that any delay towards regulation of these weapons risked allowing private industry and militaries to set the norms of how autonomous weapons are used.

Already, autonomous weapons have led to civilian harm. Three civilians were killed by a Russian drone guided by an AI system in July in Zaporizhzhia in south-eastern Ukraine. The drone tried to hit a petrol station, probably its intended target, but instead crashed into a wall, killing three people in a fountain of shrapnel.

In Gaza, Israel has used AI-assisted targeting methods to fill its target banks much faster than humanly possible since the start of the war. As a result, rights groups have said that civilians and people with tenuous links to militant groups were erroneously killed.

In Iran, a US strike that hit an elementary school, killing 156 civilians including 120 schoolchildren, reportedly was a result of faulty intelligence gathered by AI systems.

Anna Hehir, the head of military AI governance at the Future of Life Institute, said further delay to a strong treaty could see "more cases of mass civilian harm that we've seen over the past 18 months to two years", warning that some of these mass casualty events could dwarf what has already occurred. She said manufacturers and militaries were trying to develop "swarm" capabilities, where hundreds or thousands of autonomous drones act at the same time.

"This has the potential for high damage, just because of the sheer mass of systems that would be autonomous. All it takes is for one of them to face algorithmic error and then the entire swarm is utilised and led down a path based on that error … particularly if they're programmed to target humans."

There is also a risk of proliferation, as non-state groups increasingly use AI to program their weapons systems, as was the case when the Houthi movement in Yemen used Claude to "vibe code" ballistic missiles.

Armed Islamist groups in Africa's Sahel region have reportedly used AI tools to improve drone ballistics and for operational planning.

"This is the only thing that is important this week. It's not really the substance of laws. What's on the agenda here, after 13 years, is the political will finally present to start to get serious about regulation of autonomous weapons systems," said a senior diplomat with knowledge of the UN discussions.

Original source It will soon be too late to regulate use of autonomous weapons, warns Red Cross

Back to home