The benchmark shows patches pass 45.9% of tasks with live-serving checks versus 69.4% without those checks.
SWE-Serve evaluates AI coding agents on 53 inference-engineering tasks drawn from 83 merged SGLang pull requests. The benchmark covers six families: model enablement, decoding, caching, scheduling, serving APIs, and distributed execution. The SGLang team provided input for the benchmark.
SWE-Serve evaluates changes to inference-serving software by checking the full serving path, including whether a system returns correct results through its public interface. Nineteen tasks start a live server and test patches through that complete path. Across those tasks, the same patches passed 69.4% of the time when the verifier excluded live-serving checks, but only 45.9% with the complete verifier.
The results matter for builders and operators because a patch can pass local tests yet fail when a server loads a real model and handles requests. Tasks that span multiple runtime domains, such as request handling, scheduling, model execution, and KV-cache management, show a 21.3 percentage-point lower pass rate than single-domain tasks. Every tested model exhibits that same gap.
Eleven models running with mini-swe-agent under closed-book conditions show mean pass@1 ranging from 34.6% to 75.5%. Top performers include Claude Opus 5 and GPT-5.6 Sol at 75%. The leaderboard compares model performance across tasks, and the GitHub repository lets teams run SWE-Serve on SGLang tasks. The paper documents the qualification and evaluation-integrity process.
What matters
- SWE-Serve evaluates 53 inference-engineering tasks drawn from 83 merged SGLang pull requests.
- Builders must test full serving paths because local checks miss failures that live servers catch.
- Watch whether new models close the 21.3 percentage-point gap on multi-domain tasks.
Why it matters
Watch whether new models close the 21.3 percentage-point gap on multi-domain tasks.
This GenAI News article was prepared in original wording using reporting and materials published by NVIDIA Developer Blog. Source reference: https://developer.nvidia.com/blog/how-swe-serve-exposes-the-gap-between-local-tests-and-live-serving/.
Drafted by the GenAI News review pipeline.