Anthropic Releases New Open-Source Tool That Evaluates How AI Models Behave

Anthropic released a new artificial intelligence (AI) tool last week that can test and gauge how an AI model behaves under normal and stressful circumstances. Dubbed Bloom, it is designed to automate the process of testing behavioural traits of models by generating a detailed set of scenarios as prompts and evaluating the responses. The San Francisco-based AI startup's AI tool is also open-source, meaning any interested developer or an AI lab can download it to test models across various traits.

Anthropic Introduces Bloom to Test Model Behaviour

In a post, the Claude maker introduced and detailed the new AI tool. Anthropic says that testing AI model's behaviour is important as it helps researchers learn if it is prone to becoming biased, prioritising self-preservation, or indulging in sycophancy. However, the process to test model behaviour so far has been manual, where researchers create a detailed set of prompts to stress-test models and then evaluate the responses. The company says it is a lengthy and complex process.

This is where Bloom comes in. Based on specific behaviours requested by a researcher, the tool creates sample evaluations locally until the trait has been captured. Then, it runs these scenarios on the target model. Anthropic claimed that Bloom integrates with a model's weights and biases for experiments at scale. It also exports “inspect-compatible” transcripts, which can be viewed within the tool.

The functioning of the AI tool can be broken down into four broad stages. First, the AI tool analyses the requested behaviour and any example transcripts shared with it to gain understanding about it. Then, it ideates evaluation scenarios that can effectively capture and measure the trait. “Each scenario specifies the situation, simulated user, system prompt, and interaction environment,” the post mentioned. Interestingly, Bloom generates new scenarios every time, instead of relying on fixed sets.

Then, all scenarios are rolled out in parallel as an AI agent simulates both the user's and the tool responses to trigger the desired behaviour in the model. Finally, a judge model is used to score each transcript for the presence of the behaviour, and a meta-judge produces analysis of the scores and data. Anthropic added that researchers can configure Bloom's behaviour by adjusting the interactions' length and modality.

Besides the tool, Anthropic has also released benchmark results of Bloom across four behaviours — delusional sycophancy, instructed long-horizon sabotage, self-preservation, and self-preferential bias. The company tested 16 different AI models, with a mix of in-house and third-party models.

Since Bloom is open-source, interested individuals can download the AI tool from the AI startup's GitHub listing. The tool is available with a permissive MIT licence for both academic and commercial use cases.