AWS has released UAEF, a free open-source tool that tests AI agents on 36 measures, from tool use to safety. It works across several agent frameworks and flags when a new prompt or model makes an agent worse.

Amazon Web Services has released a free tool that tests whether changes to an AI agent made it better or worse. The Universal Agent Evaluation Framework (UAEF) is open source, AWS said in a blog post on Wednesday.

AI agents are built with different software kits, such as LangGraph, LangChain, Strands Agents and Amazon Bedrock AgentCore. Each one records an agent’s steps in its own format. That means teams often have to rebuild their testing whenever they switch tools.

UAEF converts those records into one shared format, then scores the agent on 36 measures in seven areas. These include how well it picks and uses tools, the quality of its answers, safety, speed, and how it handles long conversations or teamwork with other agents. Some scores are calculated directly, while others are judged by a second AI model on Amazon Bedrock.

Teams can save each test run and compare it with an earlier one. Any score that drops by more than 5% is flagged as a regression and rated minor, moderate or severe. AWS says 12 organisations in telecoms, healthcare and financial services have used UAEF to compare agent versions before release.

The tool is free to use, but scoring with AI judges and saving results both rely on AWS services. AWS also warns that UAEF should not be used to make security decisions.

Getting agents to work reliably remains a struggle. AWS has said almost 90% of prototypes Amazon built two years ago never shipped. Startups such as Raindrop have raised money to catch agent failures in production.

Researchers have also shown that one prompt could take over AgentCore agents, which AWS disputed.