Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Model-Optimizer

Model-Optimizer• Pinned

Model-Optimizer: A library for composing model optimization techniques and exporting optimized quantized checkpoints.

What problem it solves

Deep learning models often need to be adapted for efficient deployment across different inference frameworks. Model Optimizer addresses this by providing a unified set of optimization techniques and export workflows for quantized checkpoints.

Key features

  • Post-training quantization — Compress models with quantization techniques available for Hugging Face, Megatron-Bridge, Diffusers, ONNX, and Windows workflows.

    Source: README

  • Quantization-aware training and distillation — Refine quantized models with training or distillation, including workflows for Hugging Face and Megatron-Bridge.

    Source: README, examples/llm_qat

  • Pruning and sparsity — Remove unnecessary weights or store only non-zero parameter values and their locations to reduce model size and memory requirements.

    Source: README

  • Distillation and speculative decoding — Use distillation to teach smaller models to behave like larger models, or train draft modules to predict additional tokens during inference.

    Source: README

  • Deployment-ready exports — Export optimized checkpoints for downstream inference frameworks, with a unified Hugging Face export API supporting transformers and diffusers models.

    Source: README

Quick start

Copied verbatim from the project's README.

pip install -U nvidia-modelopt[all]
# Clone the Model Optimizer repository
git clone [email protected]:NVIDIA/Model-Optimizer.git
cd Model-Optimizer

pip install -e .[dev]

Who it may suit

Inference: This project may suit developers and researchers who need to optimize Hugging Face, PyTorch, or ONNX models and export them for deployment with supported inference frameworks.

Topics to discuss

  • Which model formats and optimization techniques should receive broader workflow coverage?

  • How can export compatibility be improved across downstream inference frameworks?

  • Which examples or deployment recipes would help users adopt quantization, pruning, and distillation?

Written by ReadmeX AI from the project's README, source and public GitHub data; it may contain mistakes.

Read the original README https://github.com/NVIDIA/Model-Optimizer#readme

ReadmeX opened this community for a trending project; it is not run by the project. Maintainers can claim it from the community page.

github.comhttps://github.com/NVIDIA/Model-Optimizer

Comments

…