Skip to content
Sign in
  • Browse all toolsFilter by category and pricing
  • Browse categoriesWriting, code, image, audio and more
  • Compare toolsPut up to three side by side
  • Read the blogGuides and comparisons

Crawl4AI

Open-source web crawling and extraction for AI, with Python, Docker, CLI and hosted API access.

Crawl4AI is a web crawling and extraction project that converts pages into Markdown and structured JSON for developers building AI agents, RAG applications and data pipelines. The Apache-2.0 crawler is free to self-host, while Crawl4AI Cloud offers free monthly credits and paid hosted API access.

Updated 9 Nov 20266 sections

At a glance

Pricing
FreemiumFree Apache-2.0 library and self-hosted server; users cover infrastructure. Crawl4AI Cloud includes 1,000 free monthly credits, plans from $10 and pay-as-you-go access.
Added
9 Nov 2026
Crawl4AI website screenshot

What is Crawl4AI?

Crawl4AI is an open-source web crawler for turning website content into Markdown and structured data for AI workflows. The repository contains the Apache-2.0 crawler that powers Crawl4AI Cloud, a hosted web scraping, extraction and search API. Developers can use the project in Python, through a command-line interface or as a Docker server.

Crawl4AI serves developers working on AI agents, RAG applications and data pipelines. The project provides control over browser behavior and extraction strategies, while the hosted service manages browsers and proxies. These deployment choices let developers choose between running the crawling infrastructure themselves and accessing crawling capabilities through an API key.

What Crawl4AI can do

Crawl4AI combines browser-based crawling with content conversion and structured extraction. Markdown generation preserves headings, lists, tables and code blocks, while filters remove menus, footers and other boilerplate. Extraction strategies cover schema-based methods without an LLM as well as LLM extraction into typed JSON schemas.

  • Crawl4AI converts page links into a numbered reference list.
  • Crawl4AI extracts structured data with CSS, XPath, regex and LLM strategies.
  • Crawl4AI supports breadth-first, depth-first and best-first deep crawling.
  • Crawl4AI runs JavaScript, waits for elements and scans pages for lazy-loaded content.
  • Crawl4AI captures screenshots, PDFs, media links and page metadata.
  • Crawl4AI processes multiple URLs with a memory-adaptive dispatcher.
  • Crawl4AI discovers URLs through sitemaps, Common Crawl and domain mapping.

How Crawl4AI works

Crawl4AI can be installed with pip, followed by crawl4ai-setup to install and configure the browser. Python users create an AsyncWebCrawler and call arun with a URL to obtain results such as Markdown. The crwl command-line interface also supports Markdown output, deep crawling and page questions that require an LLM key.

Crawl4AI can alternatively run as a Docker server with authenticated REST endpoints and MCP access. The hosted route uses an API key stored as CRAWL4AI_KEY, with requests sent to the hosted API. The documentation includes curl, Python and Node.js examples, plus hosted MCP setup for Claude Code, Codex, Cursor and OpenCode.

Who Crawl4AI is for

Crawl4AI is intended for developers who need web content in formats that AI applications can consume. RAG workflows can use the Markdown output, while data pipelines can use structured extraction. Agent developers can connect through MCP, and developers managing their own crawling environment can configure browser profiles, sessions, proxies, cookies and hooks.

  • Crawl4AI prepares website content as Markdown for RAG.
  • Crawl4AI extracts page fields into reusable structured schemas.
  • Crawl4AI preserves browser state for multi-step crawls.
  • Crawl4AI connects agents to hosted scraping and search.
  • Crawl4AI processes batches of URLs for data pipelines.

Crawl4AI pricing and plans

Crawl4AI is free under the Apache-2.0 license when used as a Python library or self-hosted server. Library users cover infrastructure costs, and Docker users cover hosting. Crawl4AI Cloud includes 1,000 free credits every month without a credit card, lists plans from $10 and offers pay-as-you-go access with live prices linked from the repository.

Crawl4AI strengths and limitations

Crawl4AI provides deployment flexibility, multiple extraction methods and browser controls within an open-source project. Developers can extract data without an LLM using CSS, XPath or regex, or use an LLM provider for typed JSON extraction. Hosted access removes browser management from the user's workflow, while self-hosting retains control over deployment and configuration.

  • Crawl4AI self-hosting requires users to cover infrastructure and manage browser settings and proxies.
  • Crawl4AI web search, SERP and answer endpoints are available through the cloud rather than the library or self-hosted server.
  • Crawl4AI marks the hosted SERP API as beta.
  • Crawl4AI marks the hosted answer endpoint as experimental.

Crawl4AI FAQ

What is Crawl4AI?

Crawl4AI is an Apache-2.0 open-source web crawler that converts pages into Markdown and structured data. Developers can run Crawl4AI with Python, the command line or Docker, or access the hosted scraping, extraction and search API.

Is Crawl4AI free?

Crawl4AI is free to use as an Apache-2.0 library or self-hosted server, with users covering infrastructure or hosting costs. Crawl4AI Cloud provides 1,000 free credits each month without a credit card, alongside paid plans from $10 and pay-as-you-go access.

Can Crawl4AI extract data without an LLM?

Crawl4AI supports structured extraction without an LLM through CSS, XPath and regex extraction strategies. Crawl4AI also supports generating reusable schemas and using LLM-based extraction to produce typed JSON, with open-source or hosted LLM providers.

Does Crawl4AI support MCP and AI coding tools?

Crawl4AI supports MCP through both the hosted service and a self-hosted Docker server. The repository describes hosted setup for Claude Code, Codex, Cursor and OpenCode, while the self-hosted server can connect Claude Code and other agents.

How do you install Crawl4AI locally?

Crawl4AI can be installed with pip install -U crawl4ai, followed by crawl4ai-setup to install and configure the browser. Crawl4AI also provides crawl4ai-doctor to check the installation, a command-line interface and a Docker server deployment option.

Does self-hosted Crawl4AI include web search?

Crawl4AI reserves the /search, /serp and /answer endpoints for the hosted cloud service. The repository's deployment comparison does not include those capabilities in the Python library or self-hosted server. The hosted SERP API is beta, and the answer endpoint is experimental.

From the community

No reviews yet

Be the first to review this tool. What you write helps the next person choose.

More tools like Crawl4AI

Olostep Web Data API

www.olostep.com

Web scraping, crawling, search and monitoring API for engineering teams building AI applications.

Freemium

Firecrawl

www.github.com

An open-source API for web search, scraping, browser interaction, and structured data extraction for AI agents.

Free

PaddleOCR

www.github.com

Open-source OCR and document parsing that converts PDFs and images into structured Markdown and JSON for AI workflows.

Free

AgentScope

www.github.com

Open-source Python framework for building AI agents, multi-agent teams, and agent applications.

Free