TensorRT 11.1 tuner details model-specific inference optimization
NVIDIA Technical Blog details the architecture and best practices of the Global Performance Tuner, an automatic, accuracy-aware performance-tuning feature released in TensorRT 11.1. It represents alternative compiler scheduling strategies as searchable build routes made from combinations of compiler knob values, then selects the fastest measured engine that meets the user-defined accuracy boundary. NVIDIA recommends re-tuning when the model, input shapes, target GPU, software stack or relevant TensorRT version changes.
According to NVIDIA Technical Blog, the feature moves TensorRT optimization from a universal scheduling strategy toward model- and environment-specific route selection while accounting for accuracy requirements.
The Global Performance Tuner is a feature for automatic performance tuning released in NVIDIA TensorRT 11.1. It provides automatic, accuracy-aware and end-to-end performance tuning within TensorRT by customizing different optimizations for different models. For more details, see Part 1 of this series, which introduces the challenges associated with scheduling (a critical stage in deep learning compiler optimization), the motivation for developing the Global Performance Tuner feature in TensorRT, and the mathematical theories established to formalize the problem.
This post, Part 2, uses the notation from Part 1 to describe the algorithm and architectural design of the Global Performance Tuner feature. It also provides best practices for using it.
Designing the Global Performance Tuner feature in TensorRT
The central design idea of the Global Performance Tuner is to represent alternative scheduling strategies as explicit and searchable build routes. A compiler knob (also called a compiler flag or compiler option) is a configuration setting that changes some aspect of the compilation pipeline, such as a heuristic, fusion decision, or code generation path.
Each tunable knob exposes a finite set of allowed values and has a default value. A build route is a concrete combination of knob values. When no build route is specified, TensorRT uses the default build route, in which every knob retains its default value.