Skip to content
Sign in
  • Browse all toolsFilter by category and pricing
  • Browse categoriesWriting, code, image, audio and more
  • Compare toolsPut up to three side by side
  • Read the blogGuides and comparisons

NVIDIA Model Optimizer

Open-source model optimization library for quantization, pruning, distillation and faster inference.

NVIDIA Model Optimizer is a model optimization library that compresses and accelerates deep learning models through quantization, pruning, distillation and other techniques. NVIDIA Model Optimizer serves developers and researchers preparing models for inference and is available as an open-source project under the Apache-2.0 license.

Updated 9 Nov 20266 sections

NVIDIA Model Optimizer website screenshot

What is NVIDIA Model Optimizer?

NVIDIA Model Optimizer is an open-source library for optimizing deep learning models before inference deployment. NVIDIA Model Optimizer brings quantization, pruning, neural architecture search, distillation, speculative decoding and sparsity into a shared Python workflow. The library is intended for technical users working with model compression and inference acceleration, with support for Hugging Face, PyTorch and ONNX inputs.

NVIDIA Model Optimizer separates model optimization from downstream inference execution. NVIDIA Model Optimizer produces optimized quantized checkpoints that deployment frameworks can consume, rather than serving as the inference framework itself. The repository includes documentation, examples and support matrices covering language models, vision-language models, diffusion models and individual optimization techniques.

What NVIDIA Model Optimizer can do

NVIDIA Model Optimizer provides techniques that address model size, memory footprint, inference speed and quantized model accuracy. NVIDIA Model Optimizer exposes Python APIs for composing these techniques and exporting the resulting checkpoints. The documented options include compression without training, training-based accuracy refinement and methods that change model structure or token prediction.

  • NVIDIA Model Optimizer compresses models through post-training quantization.
  • NVIDIA Model Optimizer refines quantized model accuracy through quantization-aware training and distillation.
  • NVIDIA Model Optimizer removes unnecessary weights through pruning.
  • NVIDIA Model Optimizer supports neural architecture search.
  • NVIDIA Model Optimizer trains draft modules for speculative decoding.
  • NVIDIA Model Optimizer supports sparse storage of non-zero parameters and their locations.
  • NVIDIA Model Optimizer exports transformers and diffusers models through a unified Hugging Face export API.

How NVIDIA Model Optimizer works

NVIDIA Model Optimizer follows an input, optimization and export workflow. Users supply a supported Hugging Face, PyTorch or ONNX model, apply optimization techniques through Python APIs, and export an optimized quantized checkpoint. NVIDIA Model Optimizer integrates with NVIDIA Megatron-Bridge, Megatron-LM and Hugging Face Accelerate for optimization methods that require training.

NVIDIA Model Optimizer checkpoints are prepared for downstream deployment in frameworks including SGLang, TensorRT-LLM, TensorRT and vLLM. NVIDIA Model Optimizer also provides a Hugging Face collection of ready-to-deploy pre-quantized checkpoints. The repository identifies TensorRT-LLM, vLLM and SGLang as deployment options for that collection and links to technique-specific examples.

Who NVIDIA Model Optimizer is for

NVIDIA Model Optimizer serves machine learning developers and researchers who need to prepare models for more efficient inference. NVIDIA Model Optimizer documents workflows for language models, vision-language models, diffusion models and ONNX models. The Python APIs, training integrations and deployment exports make the library relevant to teams managing the technical path from an existing model to an optimized checkpoint.

  • NVIDIA Model Optimizer supports compressing language and vision-language models before deployment.
  • NVIDIA Model Optimizer supports quantizing diffusion models.
  • NVIDIA Model Optimizer supports teaching smaller models to behave like larger models through distillation.
  • NVIDIA Model Optimizer supports preparing checkpoints for downstream inference frameworks.

NVIDIA Model Optimizer pricing and access

NVIDIA Model Optimizer is available as an open-source project under the Apache-2.0 license. NVIDIA Model Optimizer can be installed as a stable package from PyPI, installed from source in editable mode with development dependencies, or accessed through NVIDIA container images with the library pre-installed. The repository does not describe subscription plans or paid product tiers.

NVIDIA Model Optimizer strengths and limitations

NVIDIA Model Optimizer combines several optimization methods with multiple input formats, training integrations and deployment targets. NVIDIA Model Optimizer also supplies examples and support matrices to guide technique selection. These resources help users identify applicable workflows, but compatibility still needs to be checked for the chosen model and optimization method rather than assumed across all combinations.

  • NVIDIA Model Optimizer requires technical setup through packages, source installation or containers.
  • NVIDIA Model Optimizer installs third-party open-source dependencies whose license terms require review.
  • NVIDIA Model Optimizer container images have separate license terms to review before use.

NVIDIA Model Optimizer FAQ

What is NVIDIA Model Optimizer?

NVIDIA Model Optimizer is an open-source library for compressing and accelerating deep learning models. NVIDIA Model Optimizer provides quantization, pruning, neural architecture search, distillation, speculative decoding and sparsity, with Python APIs for optimization and checkpoint export.

Is NVIDIA Model Optimizer free?

NVIDIA Model Optimizer is available as an open-source project under the Apache-2.0 license, with no subscription pricing stated in the repository. NVIDIA Model Optimizer dependencies and NVIDIA container images have separate license terms that users should review.

Which model formats does NVIDIA Model Optimizer support?

NVIDIA Model Optimizer supports Hugging Face, PyTorch and ONNX model inputs. NVIDIA Model Optimizer also provides a unified Hugging Face export API for transformers and diffusers models, alongside model and technique support matrices.

Which inference frameworks work with NVIDIA Model Optimizer?

NVIDIA Model Optimizer exports optimized quantized checkpoints for downstream inference frameworks including SGLang, TensorRT-LLM, TensorRT and vLLM. NVIDIA Model Optimizer also offers a pre-quantized checkpoint collection identified as deployable on TensorRT-LLM, vLLM and SGLang.

How do you install NVIDIA Model Optimizer?

NVIDIA Model Optimizer can be installed from PyPI with pip, installed from source in editable mode, or used through NVIDIA container images with the library pre-installed. NVIDIA Model Optimizer documentation provides additional dependency and environment setup guidance.

Does NVIDIA Model Optimizer provide Python APIs and training integrations?

NVIDIA Model Optimizer provides Python APIs for composing optimization techniques and exporting optimized quantized checkpoints. NVIDIA Model Optimizer integrates with NVIDIA Megatron-Bridge, Megatron-LM and Hugging Face Accelerate for inference optimization techniques that require training.

From the community

No reviews yet

Be the first to review this tool. What you write helps the next person choose.

More tools like NVIDIA Model Optimizer

Automodel

www.github.com

Open-source PyTorch distributed training and fine-tuning for language and vision-language models

Free

Magnitude

www.github.com

Open-source inference engine that tunes local models for your hardware and connects them to coding agents.

Free

lithos-metal

www.github.com

Open-source LLM inference and local serving for Apple silicon Macs using Metal megakernels and DSpark decoding.

Free

AudioCraft

www.github.com

A PyTorch library for audio generation, music generation, audio compression and deep learning research.

Free