Skip to content
Sign in
  • Browse all toolsFilter by category and pricing
  • Browse categoriesWriting, code, image, audio and more
  • Compare toolsPut up to three side by side
  • Read the blogGuides and comparisons

datasets

Open-source Python library for loading, streaming, and preprocessing machine learning datasets.

datasets is a lightweight, open-source Python library for loading, streaming, and preprocessing public and local datasets. datasets serves machine learning developers and researchers preparing multimodal data for training and evaluation, and is freely available under the Apache-2.0 license through pip, conda, or its GitHub repository.

Updated 9 Nov 20266 sections

datasets website screenshot

What is datasets?

datasets is an open-source Python library from Hugging Face for loading and preprocessing machine learning data. datasets serves developers and researchers who need public datasets or their own local files for model training, evaluation, and inspection. The library provides a common interface across text, audio, image, video, PDF, and medical imaging data.

datasets connects to datasets maintained on the Hugging Face Hub and also accepts local data in formats including CSV, JSON, JSONL, Parquet, HDF5, and XML. The repository describes a community-driven approach to adding and sharing datasets, with documentation covering loading, processing, streaming, and contributions.

What datasets can do

datasets combines dataset access with tools for preparing data before model training or evaluation. The library uses Apache Arrow for memory-mapped storage and automatically reuses cached processing results. Its interoperability features let users move between dataset objects and established numerical, tabular, and machine learning frameworks.

  • datasets loads public Hub datasets and local files with load_dataset().
  • datasets streams examples on demand with streaming=True.
  • datasets applies transformations in parallel with map(num_proc=N).
  • datasets supports text, audio, images, video, PDF, and NIfTI data.
  • datasets converts data to and from NumPy, Pandas, Polars, Arrow, PyTorch, TensorFlow, JAX, and Spark.
  • datasets supports FAISS and Elasticsearch indexes for similarity search.
  • datasets reads and writes Hugging Face Storage Buckets.
  • datasets loads AI agent traces containing prompts, tool calls, and responses.

How datasets works

datasets centers its API on load_dataset(), which creates a dataset from a Hub identifier or local source. Users can inspect examples, select dataset splits, and apply processing functions with map(). The quick-start examples demonstrate adding a derived column, tokenizing context text with Transformers, and applying chat templates to conversation data.

datasets provides Dataset objects for indexed, memory-mapped access and IterableDataset objects for lazy streaming. DatasetDict and IterableDatasetDict group dataset splits. Users can also create datasets from Python dictionaries, lists, generators, and Pandas DataFrames. Streaming mode is intended for situations where downloading the dataset would take too long or exceed available disk space.

Who datasets is for

datasets is intended for machine learning workflows that need repeatable data loading and preparation across different modalities. Researchers can use public datasets for evaluation and training, while developers can process their own files through the same library. The documented examples also cover chat-model preparation and handling structured AI agent traces.

  • datasets prepares public datasets for model training and evaluation.
  • datasets processes local tabular and structured files for inspection.
  • datasets tokenizes chat conversations for instruction and chat-model fine-tuning.
  • datasets loads audio, image, video, PDF, and medical imaging datasets.
  • datasets streams large datasets for out-of-core processing.

datasets pricing and installation

datasets is available as an open-source project under the Apache-2.0 license. The repository provides installation through PyPI using pip install datasets and through conda using the huggingface and conda-forge channels. Users can also install the latest development version directly from the GitHub repository.

datasets installation guidance recommends a virtual environment such as venv or conda. Optional extras supply dependencies for audio, image and video processing, PDF and NIfTI handling, and PyTorch, TensorFlow, or JAX integration. The repository links to installation documentation and a quick-start guide for working with the library.

datasets strengths and limitations

datasets brings loading, preprocessing, caching, and framework conversion into a shared Python workflow. Streaming and memory-mapped storage address large-data workflows without requiring every dataset to fit in RAM. Practical constraints include installing the relevant optional dependencies and managing dataset revisions: the repository asks users to pin revisions of dataset repositories for reproducibility.

  • datasets requires a Python environment and code-based interaction.
  • datasets uses optional dependencies for some media capabilities and framework integrations.
  • datasets users should pin dataset repository revisions for reproducible results.

datasets FAQ

What is datasets?

datasets is an open-source Python library from Hugging Face for loading and preprocessing machine learning datasets. datasets supports public Hub datasets and local files, with tools for streaming, caching, parallel transformations, and preparation for model training and evaluation.

Is datasets free?

datasets is freely available as an open-source library under the Apache-2.0 license. datasets can be installed from PyPI with pip, through conda, or directly from the GitHub repository for the latest development version.

Can datasets stream data without downloading the full dataset?

datasets supports streaming through the streaming=True option in load_dataset(). datasets lets users iterate over examples on demand, which is useful when a dataset exceeds available disk space or users want to avoid waiting for a complete download.

What file formats and data types does datasets support?

datasets supports formats including CSV, JSON, JSONL, Parquet, Arrow, XML, Text, and Webdataset. datasets also handles text, audio, images, video, PDF documents, NIfTI medical imaging, and structured AI agent traces, with optional dependencies documented for specific media capabilities.

Which machine learning frameworks does datasets support?

datasets supports conversion to and from NumPy, Pandas, Polars, Arrow, PyTorch, TensorFlow, JAX, and Spark. datasets documentation also demonstrates using Transformers tokenizers to process context text and apply chat templates to conversation datasets.

From the community

No reviews yet

Be the first to review this tool. What you write helps the next person choose.

More tools like datasets

Hugging Face

www.huggingface.co

Discover, host and collaborate on machine learning models, datasets and applications.

Freemium

Gradio

www.gradio.app

Build and share machine learning web interfaces in Python without writing frontend code.

Freemium

tokenizers

www.github.com

Open-source tokenization library for encoding and decoding text in research and production.

Free

Scikit Learn

www.scikit-learn.org

Machine learning for classification, regression, clustering, preprocessing, and model selection

Freemium