Skip to content
Sign in
  • Browse all toolsFilter by category and pricing
  • Browse categoriesWriting, code, image, audio and more
  • Compare toolsPut up to three side by side
  • Read the blogGuides and comparisons

tokenizers

Open-source tokenization library for encoding and decoding text in research and production.

tokenizers is a text tokenization library that loads tokenizer configurations, encodes text and decodes token IDs for language model research and production. Designed for developers and researchers, tokenizers provides Rust, Python, Node.js and WebAssembly support and is available as an open-source project under the Apache-2.0 license.

Updated 9 Nov 20266 sections

tokenizers website screenshot

What is tokenizers?

tokenizers is an open-source library from Hugging Face for text tokenization in language model research and production. tokenizers provides a Rust reference implementation alongside Python, Node.js and WebAssembly bindings. The library handles the processing pipeline around tokenization, including normalization, pre-tokenization, model execution, post-processing and decoding.

tokenizers separates its Rust implementation into components for inference, configuration reading, legacy configuration conversion and training. This structure lets applications choose dependencies according to their workload. The repository also documents an ongoing release-candidate transition, so the available functionality must be distinguished from features listed on the roadmap.

What tokenizers can do

tokenizers supports several tokenization models and exposes operations for moving between text, token strings and token IDs. The documented examples cover individual text inputs and paired inputs, with encoded output containing IDs, type IDs and an attention mask. Configuration editing is separate from the compiled object used for encoding and decoding.

  • tokenizers supports BPE, Unigram, WordPiece and WordLevel models.
  • tokenizers loads pretrained configurations from the Hub or local files.
  • tokenizers encodes text and text pairs into structured outputs.
  • tokenizers returns token strings and decodes token IDs into text.
  • tokenizers allows configuration changes before compilation.
  • tokenizers offers Rust training through an opt-in feature.

How tokenizers works

tokenizers documents Python installation through pip using the prerelease option and Rust installation by adding the crate to Cargo.toml. The Python workflow begins by importing Tokenizer and loading a pretrained tokenizer or a local file. A Tokenizer holds the tokenizer.json configuration, which can be edited before calling compile() to create a CompiledTokenizer.

tokenizers uses the compiled object for encode, tokenize, decode and decode_tokens operations. Rust users can select optional capabilities: training and Hub downloads are not enabled by default. The umbrella crate preserves existing import paths, while separate crates allow applications to exclude training or legacy conversion components when those capabilities are unnecessary.

Who tokenizers is for

tokenizers targets research and production workloads involving language models and natural language processing. Researchers can use supported tokenization models and training components, while application developers can integrate encoding and decoding through the available bindings. Production engineers can select inference-focused Rust components without linking trainers, corpus readers or legacy converters into serving binaries.

  • tokenizers supports text preprocessing for language model applications.
  • tokenizers supports inspection of token strings and encoded outputs.
  • tokenizers supports customized configurations loaded from local files.
  • tokenizers supports dependency selection for Rust inference deployments.

tokenizers pricing and access

tokenizers is an open-source project distributed under the Apache-2.0 license. Access is through the public repository and documented language packages rather than a hosted subscription described on the page. The repository links to a guide, a quick tour and Rust documentation, and identifies tokenizers-wasm as the WebAssembly package on npm.

tokenizers strengths and current limitations

tokenizers combines multiple language bindings with modular Rust components and portable execution paths. SIMD is not required: every kernel has a portable path, while hardware-adapted implementations affect throughput rather than correctness. The principal constraint is release readiness, because the documented release candidate does not yet provide every feature intended for the upcoming release.

  • tokenizers lists Python training and batch encoding APIs as work to bring back.
  • tokenizers lists offset output and windowing among roadmap tasks.
  • tokenizers identifies C, C++, Java and Go bindings as planned.
  • tokenizers requires explicit Rust features for training and Hub downloads.

tokenizers FAQ

What is tokenizers?

tokenizers is an open-source text tokenization library from Hugging Face for research and production. tokenizers supports BPE, Unigram, WordPiece and WordLevel models, with pipeline components for normalization, pre-tokenization, post-processing and decoding.

Is tokenizers free?

tokenizers is available as an open-source project under the Apache-2.0 license. The repository provides installation instructions for Python and Rust and identifies additional bindings, rather than describing a paid tokenizers subscription.

Which programming languages does tokenizers support?

tokenizers lists Rust as its reference implementation and marks Python, Node.js and WebAssembly bindings as available. Ruby support is maintained in an external community repository, while C, C++, Java and Go bindings are listed as planned.

How do you load and use a tokenizer with tokenizers?

tokenizers loads a pretrained tokenizer with Tokenizer.from_pretrained or a local configuration with Tokenizer.from_file. The configuration can be edited before calling compile(), which creates the CompiledTokenizer used for encoding, token output and decoding.

Can tokenizers train tokenization models?

tokenizers provides Rust training through the optional train feature, which is disabled by default and trains only models whose features are enabled. The current release-candidate roadmap separately lists bringing training back to the Python bindings.

Does tokenizers require SIMD hardware?

tokenizers does not require SIMD because every kernel includes a portable path. Hardware-adapted implementations are available for selected operations and architectures, but the repository states that correctness does not depend on those kernels being present.

From the community

No reviews yet

Be the first to review this tool. What you write helps the next person choose.

More tools like tokenizers

Candle

www.github.com

Open-source Rust machine learning framework for model training, GPU inference, and browser-based models.

Free

Mistral Inference

www.github.com

An archived, open-source inference library for running Mistral models through Python and command-line tools.

Free

Automodel

www.github.com

Open-source PyTorch distributed training and fine-tuning for language and vision-language models

Free

datasets

www.github.com

Open-source Python library for loading, streaming, and preprocessing machine learning datasets.

Free