The Global Performance Tuner is a feature for automatic performance tuning released in NVIDIA TensorRT 11.1. It provides automatic, accuracy-aware and end-to-end performance tuning within TensorRT by customizing different optimizations for different models. For more details, see Part 1 of this series, which introduces the challenges associated with scheduling (a critical stage in deep learning compiler optimization), the motivation for developing the Global Performance Tuner feature in TensorRT, and the mathematical theories established to formalize the problem.

This post, Part 2, uses the notation from Part 1 to describe the algorithm and architectural design of the Global Performance Tuner feature. It also provides best practices for using it.

Designing the Global Performance Tuner feature in TensorRT

The central design idea of the Global Performance Tuner is to represent alternative scheduling strategies as explicit and searchable build routes. A compiler knob (also called a compiler flag or compiler option) is a configuration setting that changes some aspect of the compilation pipeline, such as a heuristic, fusion decision, or code generation path.

Each tunable knob exposes a finite set of allowed values and has a default value. A build route is a concrete combination of knob values. When no build route is specified, TensorRT uses the default build route, in which every knob retains its default value.