htcrawl.org

What It Is

Htcrawl is a Node.js module designed for recursively crawling single-page applications (SPAs). It uses Headless Chrome to load and analyze web applications and is built on top of Puppeteer, so it can reuse Puppeteer’s browser automation capabilities. Rather than being a traditional static crawler, it is more focused on dynamic page analysis, request interception, and automated interaction.

Core Capabilities

In terms of functionality, Htcrawl focuses on event-driven crawling: it can listen for XHR, fetch, JSONP, WebSocket, form submissions, DOM additions, input field filling, event triggering, navigation, redirects, and related processes. Its API provides methods such as start, navigate, reload, clickToNavigate, and waitForRequestsCompletion, and also allows access to Puppeteer’s browser and page instances. It offers a fairly rich set of configuration options as well, including proxy settings, cookies, HTTP authentication, custom headers, POST loading, maximum recursion depth, Ajax chain length, CSP bypassing, and navigation timeouts.

Pricing and Open Source

The text clearly states that Htcrawl is free software, licensed under GNU GPL v2 or later, and can be redistributed and modified. Installation options include npm installation and cloning the source code from GitHub. There is no mention of a commercial edition, cloud service, paid support, or enterprise licensing, so it is better suited to developer teams that can maintain it themselves.

Pros and Cons

Its strengths are that it closely matches the real runtime environment of SPAs and can capture dynamic DOM changes and asynchronous requests. Its event system is detailed, making it suitable for building DOM-XSS scanners, security testing tools, or advanced content crawling scripts. Being based on Puppeteer also reduces the cost of low-level browser control. The downside is that it is more of a low-level library than a complete application; capabilities such as DOM-XSS scanning require users to implement their own logic. While the documentation includes APIs and examples, it contains spelling errors, inconsistent use of the names htcrawl/htcap, and lacks information on maintenance status, performance tuning, troubleshooting, and community support.

Who It’s For and Access from China

Htcrawl is suitable for Node.js developers, security researchers, automation testing engineers, and tool authors who need to analyze SPA request flows. It is less suitable for users looking for an out-of-the-box product with a visual interface or reporting system. The text does not provide information about access from China. Accessing npm and GitHub to obtain the source code may be affected by the local network environment, but this alone is not enough to draw a firm conclusion. Alternatives include Puppeteer, Playwright, Crawlee, and Selenium; for security scanning, htcap may also be worth watching.