The open-source harness measures accuracy, token use, and agent turn counts across five models instead of relying on price per token.
AWS published an open-source benchmarking harness that prices OpenAI models on Amazon Bedrock by outcomes rather than tokens. The harness runs one identical code path through the OpenAI Responses API against five models: gpt-5.6-luna, gpt-5.6-terra, and gpt-5.6-sol on Bedrock, plus gpt-5.4-mini and gpt-5.4-nano on the OpenAI API.
Organizations typically compare models by dollars per million tokens because that figure sits on every pricing page. Production work instead pays for resolved tickets, finished research briefs, and correct summaries, where accuracy and token efficiency multiply the sticker price. AWS chose mini and nano as cost-optimized baselines because teams ask whether a newer Bedrock model justifies a switch.
Agent workloads re-send a growing conversation on every turn, so turn count can dominate the bill more than per-token pricing does. The harness grades single-call accuracy on AIME, GPQA Diamond, and MMLU-Pro, plus multi-turn web research and rubric-graded professional deliverables. Bedrock models ran with reasoning disabled while API baselines ran at defaults.
Sample sizes range from 48 to 198 items, so AWS advises treating small gaps as directional. Every run writes a timestamped result JSON, and every number and chart in the post comes from those files at build time. The harness is public, so teams can run it on their own tasks before choosing a model.
What matters
- AWS published an open-source harness that benchmarks five OpenAI models on Bedrock and the API.
- Token price hides accuracy, retry volume, and agent turn count, which drive real production spend.
- Sample sizes of 48 to 198 items make small gaps directional, so teams should rerun the harness themselves.
Why it matters
Sample sizes of 48 to 198 items make small gaps directional, so teams should rerun the harness themselves.
This GenAI News article was prepared in original wording using reporting and materials published by AWS Machine Learning Blog. Source reference: https://aws.amazon.com/blogs/machine-learning/beyond-the-price-per-token-choosing-the-right-openai-model-on-amazon-bedrock-for-your-workload/.
Drafted by the GenAI News review pipeline.