URL and endpoint discovery
Which pages, parameters and API paths exist?
Archive-based tools pull historic URLs from the Wayback Machine and Common Crawl without touching the site. Crawlers request the live site and follow links and JavaScript. Both surface forgotten endpoints and parameters.
Listed alphabetically. The order is not a ranking.
- cariddiActive
Crawls a list of domains and scans the pages it fetches for endpoints, secrets, API keys, error messages and notable file types.
3.8k stars · last push November 5, 2026 · v1.4.6 · GPL-3.0
- getallurls (gau)Passive
Pulls known historical URLs for a domain from the Wayback Machine, Common Crawl, AlienVault OTX and URLScan.
5.2k stars · last push October 12, 2026 · v2.2.4 · MIT
- getJSActive
Extracts JavaScript source URLs from web pages or piped HTTP responses, as a CLI and as a Go package.
902 stars · last push August 29, 2026 · v2.0.0 · MIT
- HakrawlerActive
Lightweight Go crawler that reads URLs from stdin and prints the links and JavaScript file locations it finds.
5.2k stars · last push August 5, 2026 · 2.1 · GPL-3.0
- KatanaActive
Web crawler that maps URLs and endpoints on a target site, with an optional headless browser mode for JavaScript-heavy apps.
18k stars · last push November 9, 2026 · v1.8.0 · MIT
- URLFinderPassive
Passive URL discovery tool from ProjectDiscovery that gathers a domain's known URLs from public archives and intel sources.
921 stars · last push November 9, 2026 · v0.0.3 · MIT
Minimal stdin/stdout tool that lists URLs the Wayback Machine has recorded for a domain and its subdomains.
4.6k stars · last push May 1, 2024 · v0.1.0 · No licence file