TorchServe is no longer actively maintained, so AWS ships a pre-built Ray Serve container for model inference.

AWS released Ray Serve Deep Learning Containers, pre-built Docker images that bundle the full inference stack. The images include PyTorch, Ray Serve, FastAPI, Uvicorn, and the CUDA runtime on the GPU variant. The launch follows the official end of maintenance for TorchServe, leaving users without security patches or framework updates.

AWS built the Ray Serve DLC on Amazon Linux 2023 for CPU and on the NVIDIA Amazon Linux 2023 image for GPU. The GPU variant includes CUDA runtime libraries and FFmpeg compiled with NVIDIA hardware acceleration. AWS validates all components together before each release to avoid version drift between the CUDA runtime, the framework, and the serving layer.

Operators running TorchServe today carry the entire dependency chain, including patching vulnerabilities and matching PyTorch and CUDA versions. The Ray Serve DLC shifts that burden to AWS, which applies security patches at build time and tests the whole inference stack. Teams can pull the container and serve models behind an HTTP endpoint immediately, cutting maintenance overhead.

The accompanying repository contains code for deploying a vision-language model on one GPU node with Ray Serve on Amazon EKS. Users need an AWS account, sufficient g5.xlarge quotas, and configured AWS CLI, eksctl, and kubectl. Watch for migration patterns as teams move from TorchServe to the maintained Ray Serve images across EKS, EC2, and SageMaker.

What matters

• AWS released Ray Serve Deep Learning Containers that bundle PyTorch, Ray, and FastAPI into one tested image.
• TorchServe lacks security patches, so teams must own dependency management and vulnerability fixes themselves.
• Watch for adoption of Ray Serve DLC on Amazon EKS, EC2, and SageMaker with GPU-optimized images.

Why it matters

Watch for adoption of Ray Serve DLC on Amazon EKS, EC2, and SageMaker with GPU-optimized images.

This GenAI News article was prepared in original wording using reporting and materials published by AWS Machine Learning Blog. Source reference: https://aws.amazon.com/blogs/machine-learning/simplify-and-support-your-torchserve-workloads-using-ray-serve-deep-learning-containers/.

Drafted by the GenAI News review pipeline.