Deployment & Scaling · Advanced
Triton Inference Server and Dynamo: Serving Many Models Well
Triton is the general-purpose inference server: many frameworks, many models, one endpoint, with dynamic batching, model ensembles and concurrent execution. Dynamo extends the idea to disaggregated, multi-node LLM serving.
- 1.7 hours lesson
- advanced level
- NVIDIA AI Track
- Narrated presentation
What you’ll learn
- One server, many frameworks: PyTorch, ONNX, TensorRT, Python and ensemble backends
- Dynamic batching with a configurable queue delay as the core throughput/latency trade
- Instance groups and concurrent model execution on one GPU
- Model repository layout, config.pbtxt and version management
- perf_analyzer and Model Analyzer for sweeping configurations empirically
- NIM vs Triton vs Dynamo: packaged single model, general server, distributed multi-node
- Name collision with the Triton kernel language
Inside this lesson
A narrated, slide-by-slide presentation with 1 prediction exercise, 1 knowledge check, 3 videos. You answer as you go, and the platform checks your work.