What is Megatron-LM?
Megatron-LM is an open-source NVIDIA project for training transformer models at scale. Megatron-LM is the reference example in a repository that also contains Megatron Core, a composable library of GPU-optimized training components. The reference implementation combines that library with pre-configured training scripts for research teams, people learning distributed training, and quick experimentation.
Megatron-LM separates example-driven training from the building blocks used to create custom frameworks. Megatron Core provides transformer components, parallelism strategies, mixed-precision support, and model architectures for framework developers and ML engineers. This distinction helps explain whether a user should begin with the repository's training examples or work directly with the underlying library.
What Megatron-LM can do
Megatron-LM brings training scripts together with Megatron Core capabilities for distributed model development. The repository includes transformer, hybrid, and multimodal model components, as well as datasets, optimizers, inference components, and post-training tools. Megatron Bridge complements the repository with bidirectional Hugging Face checkpoint conversion and production-ready recipes.
- Megatron-LM includes pre-configured scripts for transformer training.
- Megatron-LM includes tensor, pipeline, data, expert, and context parallelism through Megatron Core.
- Megatron-LM includes FP16, BF16, FP8, and FP4 mixed-precision support through Megatron Core.
- Megatron-LM includes distributed checkpoint saving, loading, and resharding.
- Megatron-LM contains post-training tools for quantization, distillation, and pruning.
- Megatron-LM includes model export components, with TensorRT-LLM given as an example.
How Megatron-LM works
Megatron-LM can be obtained by cloning the GitHub repository and installing the source with uv pip install -e . from the project directory. Megatron Core is also available from PyPI through uv pip install megatron-core. The documentation recommends Python 3.12+ and PyTorch 2.6.0+ and links to an installation guide covering NVIDIA GPU Cloud container setup.
Megatron-LM links to an end-to-end first training run that includes data preparation, followed by guidance on parallelism strategies for scaling across GPUs. For source development, tested NGC PyTorch base images are pinned in repository files. Benchmark reproduction instructions specify matching container and code versions rather than treating published throughput as a general guarantee.
Who Megatron-LM is for
Megatron-LM is intended for research teams, learners studying distributed training, and users who need a reference setup for experimentation. Megatron Core addresses a related audience of framework developers and ML engineers building custom training pipelines. The repository's examples and component structure support both exploration of training workflows and development around reusable training primitives.
- Research teams can use Megatron-LM for transformer training experiments.
- Learners can use Megatron-LM to study distributed training workflows.
- Framework developers can use Megatron Core to build custom training frameworks.
- ML engineers can use Megatron Bridge to convert Hugging Face and Megatron checkpoints.
Megatron-LM access and pricing information
Megatron-LM is distributed as an open-source project through a public GitHub repository, with instructions for cloning and installing the source. Megatron Core also has a PyPI installation path. The supplied repository text does not specify a product subscription or paid training plan; the accompanying GitHub pricing information concerns GitHub services rather than Megatron-LM.
Megatron-LM strengths and limitations
Megatron-LM combines ready-to-use training examples with composable GPU-optimized components and documentation for distributed training. The repository also points to benchmark recipes, system configurations, and reproduction instructions. Practical constraints include memory use during source builds and configuration-dependent throughput, so installation and performance evaluation require attention to the documented environment.
- Megatron-LM source builds can run out of memory; the documentation suggests limiting parallel compilation jobs with MAX_JOBS.
- Megatron-LM setup involves package installation or source installation and training configuration.
- Megatron-LM benchmark throughput depends on the model, precision, hardware, and configuration.
Megatron-LM FAQ
What is Megatron-LM?
Megatron-LM is an open-source NVIDIA reference project for training transformer models at scale. Megatron-LM combines Megatron Core with pre-configured training scripts for research teams, distributed-training learners, and users running experiments.
What is the difference between Megatron-LM and Megatron Core?
Megatron-LM is a reference example that includes Megatron Core and pre-configured training scripts. Megatron Core is a composable library with GPU-optimized building blocks, parallelism strategies, mixed-precision support, and model architectures for custom training frameworks.
How do you install Megatron-LM?
Megatron-LM can be installed by cloning the GitHub repository, entering the project directory, and running uv pip install -e . Megatron Core can also be installed from PyPI with uv pip install megatron-core. The documentation recommends Python 3.12+ and PyTorch 2.6.0+.
Which parallelism strategies does Megatron-LM support?
Megatron-LM includes tensor, pipeline, data, expert, and context parallelism through Megatron Core. Megatron-LM links to documentation explaining how these strategies scale training across GPUs and provides training examples with data preparation.
Can Megatron-LM work with Hugging Face checkpoints?
Megatron-LM is complemented by Megatron Bridge, which provides bidirectional checkpoint conversion between Hugging Face and Megatron. Megatron Bridge also provides production-ready recipes, allowing checkpoint conversion to form part of a documented training workflow.
What limitations should users know about Megatron-LM?
Megatron-LM source builds can use substantial memory, and the documentation recommends limiting parallel compilation jobs if a build runs out of memory. Published benchmark throughput also depends on the model, precision, hardware, and configuration.
From the community
No reviews yet
Be the first to review this tool. What you write helps the next person choose.




