crawlee.dev

What It Is

Crawlee is a web scraping library built by Apify, positioned around the tagline “Build reliable web scrapers. Fast.” The main copy clearly states that it supports JavaScript and Python, and is designed to handle common crawler-engineering complexities such as blocking, crawling, proxies, and browsers. It is not a no-code scraping tool, but a developer-focused library and CLI tool suited for embedding scraping capabilities into your own projects or data pipelines.

Core Capabilities

Based on the scraped content, Crawlee’s JavaScript ecosystem is very complete. It provides multiple crawler types, including PlaywrightCrawler, PuppeteerCrawler, CheerioCrawler, JSDOMCrawler, HttpCrawler, and BasicCrawler, covering static pages, dynamically rendered pages, browser automation, and low-level HTTP crawling. It also includes built-in abstractions such as RequestQueue, RequestList, Dataset, KeyValueStore, SessionPool, ProxyConfiguration, and AutoscaledPool, supporting link enqueuing, data persistence, CSV export, proxy management, session management, parallel crawling, and scaling. The CLI can create project templates via npx crawlee create my-crawler, or on the Python side with uvx 'crawlee[cli]' create my-crawler.

Languages, Ecosystem, and Documentation

The content shows that Crawlee supports both JavaScript and Python. The JavaScript documentation is especially detailed, covering modules such as @crawlee/core, @crawlee/playwright, @crawlee/puppeteer, @crawlee/cheerio, @crawlee/http, and @crawlee/utils. It is deeply integrated with ecosystems such as Playwright, Puppeteer, Cheerio, JSDOM, LinkeDOM, Got Scraping, Impit, Docker, puppeteer-extra, and playwright-extra. The documentation quality is high, with Quick Start, Guides, Examples, API, and Changelog sections, plus many examples such as crawling sitemaps, multiple URLs, downloading files, submitting forms, taking screenshots, and exporting data.

Pricing and Self-Hosting

The page explicitly states “Forever free and open-source,” so its value for money is excellent. The content does not show any commercial hosted plans or paid support information. As an open-source library, it is naturally suitable for running locally or on your own servers. The documentation also includes topics such as Running in Docker, Running in web server, Deployment, and Running in the Cloud.

Pros, Cons, and Who It’s For

Its strengths are that it is free and open-source, modular, well documented, and provides systematic abstractions for proxies, sessions, storage, scaling, and anti-blocking-related capabilities. The limitation is that it still requires developers to write selectors and handle changes in page structure; the page also notes that it “won’t fix broken selectors for you.” It is suitable for data engineers, backend developers, and automation teams familiar with JS/TS or Python who need to maintain crawler systems over the long term.

Access from China

The content does not provide information about access from mainland China, mirrors, payment methods, or network availability, so this remains unknown. If access to npm, Playwright browser downloads, or the official documentation is affected by network conditions, you may consider configuring an npm mirror, using a proxy, or choosing alternatives such as Scrapy, Playwright, Puppeteer, or Selenium.