Combining CUDA Multi-Process Service and Triton Inference Server lets one L40S GPU handle 92.1 requests per second.
NVIDIA CUDA Multi-Process Service and NVIDIA Triton Inference Server on Amazon EC2 reduced automatic speech recognition GPU requirements by 75 percent. A fine-tuned Parakeet TDT 0.6B V2 model now serves 92.1 requests per second per L40S GPU while holding sub-second latency. The deployment cuts the footprint from 16 instances to 4.
AWS, NVIDIA, and Heidi Health published the guidance. Heidi Health, an AI Care Partner, processes more than 2.4 million clinical consultations per week across 190 countries. A single ASR request uses only 15 to 20 percent of an L40S GPU’s 142 streaming multiprocessors. CUDA’s default time-slicing leaves the remaining capacity idle and forces the company to run 16 GPU instances for peak traffic.
For teams serving latency-sensitive models, the key gain is concurrent execution on a single GPU. MPS partitions the GPU into compute instances, while Triton schedules dynamic batches and uses ONNX or TensorRT optimizations to keep latency stable. Operators can evaluate their own utilization and apply the same pattern instead of adding instances.
The reference implementation includes a container that starts the MPS daemon, runs an auto-configuration script, and launches a Triton Python backend with local attention optimization. Builders can pass a fine-tuned NeMo checkpoint as a Docker build argument and set the MPS instance count when running the image. Watch for broader adoption of this MPS serving pattern across EC2 GPU families.
What matters
- AWS and NVIDIA showed how MPS and Triton cut ASR GPU instances from 16 to 4 on EC2.
- For operators, the approach converts idle L40S compute into 92.1 requests per second per GPU.
- Watch for more production recipes combining MPS with ONNX and TensorRT optimizations.
Why it matters
Watch for more production recipes combining MPS with ONNX and TensorRT optimizations.
This GenAI News article was prepared in original wording using reporting and materials published by AWS Machine Learning Blog. Source reference: https://aws.amazon.com/blogs/machine-learning/reduce-asr-inference-costs-by-75-with-nvidia-mps-on-amazon-ec2/.
Drafted by the GenAI News review pipeline.