The container bundles Ray Serve LLM, NVIDIA NIXL, and EFA to serve one model across two GPU nodes on Amazon EKS.

AWS introduced Ray LLM Deep Learning Containers for multi-node inference on Amazon EKS. The containers run Ray Serve LLM, NVIDIA Inference Transfer Library, and Elastic Fabric Adapter, and expose an OpenAI-compatible endpoint. Teams can deploy one large language model across two GPU nodes, with one node handling prefill and the other handling decode.

AWS has long offered Deep Learning Containers as pre-built Docker images that bundle frameworks, dependencies, and GPU stacks into tested combinations. A prior post introduced the Ray Serve DLC for single-node inference. The new Ray LLM DLC extends that approach to multi-node deployments for tensor, pipeline, and data parallelism, as well as prefill and decode disaggregation.

Builders gain a maintained container that removes dependency chain assembly for large models that exceed one GPU or require higher throughput. Separating prefill and decode phases prevents long prompts from stalling token generation for requests already in progress. The setup suits retrieval augmented generation, document summarization, and agentic applications.

The complete code sits in an accompanying repository, so operators can test the two-node topology and tune phase splits. Watch for AWS updates on node sizing, EFA networking, and multi-node Ray LLM support. The main tradeoff involves coordinating two GPU nodes behind one endpoint instead of scaling a single-node replica.

What matters

• AWS released multi-node Ray LLM Deep Learning Containers for Amazon EKS inference workloads.
• Builders can run large models across two GPU nodes without assembling compatible GPU stack dependencies.
• Watch for AWS guidance on tuning prefill and decode splits for RAG, summarization, and agent workloads.

Why it matters

Watch for AWS guidance on tuning prefill and decode splits for RAG, summarization, and agent workloads.

This GenAI News article was prepared in original wording using reporting and materials published by AWS Machine Learning Blog. Source reference: https://aws.amazon.com/blogs/machine-learning/run-multi-node-inference-workloads-using-ray-llm-deep-learning-containers/.

Drafted by the GenAI News review pipeline.