Models perform poorly on new benchmark, highlighting gaps for AI in enterprise IT.

ITBench-AA, a new benchmark from Artificial Analysis and IBM Research, reveals that leading AI models struggle with site reliability engineering (SRE) tasks. Models such as Claude Opus 4.7 and GPT-5.5 score below 50%, indicating significant room for improvement in AI’s ability to handle complex enterprise IT operations.

Artificial Analysis and IBM developed the benchmark over six months, focusing on SRE tasks that require diagnosing live systems using logs and tracing dependencies. The initial results raise concerns about current model limitations, particularly regarding investigation depth and accuracy.

For builders and operators of AI-driven solutions in enterprise environments, these findings underscore the need for more robust training data and improved model architectures to ensure reliable performance under complex conditions. As ITBench-AA expands to include FinOps and CISO tasks, developers will face new challenges and opportunities to enhance their models’ capabilities.

Future developments may involve closer collaboration between AI researchers and enterprise IT teams to refine benchmarks and develop more effective training methodologies.

What matters

• Frontier models score below 50% on first agentic enterprise IT tasks benchmark.
• Performance issues could impact model reliability in real-world scenarios.
• Next steps include expanding the benchmark to FinOps and CISO tasks.

Why it matters

Next steps include expanding the benchmark to FinOps and CISO tasks.

This GenAI News article was prepared in original wording using reporting and materials published by Hugging Face Blog. Source reference: https://huggingface.co/blog/ibm-research/itbench-aa.

Drafted by the GenAI News review pipeline.