Deployment & Scaling · Advanced
TensorRT and TensorRT-LLM: Compiling Models for Inference
TensorRT compiles a trained network into a hardware-specific engine — fusing kernels, selecting algorithms and lowering precision. TensorRT-LLM extends that to transformers with in-flight batching, paged KV cache and speculative decoding.
- 1.8 hours lesson
- advanced level
- NVIDIA AI Track
- Narrated presentation
What you’ll learn
- Compilation to a hardware-specific engine: kernel fusion, algorithm selection, precision lowering
- Engines are architecture-specific and must be rebuilt per GPU generation
- In-flight (continuous) batching as the dominant throughput lever
- Paged KV cache and its effect on achievable concurrency
- FP8 and 4-bit quantisation via TensorRT Model Optimizer with calibration
- Speculative decoding and tensor/pipeline parallelism
- TensorRT-LLM versus vLLM: control and peak performance versus speed of adoption
Inside this lesson
A narrated, slide-by-slide presentation with 1 prediction exercise, 1 knowledge check, 4 videos. You answer as you go, and the platform checks your work.