Skip to content
Deepak Gupta markRecon Atlas

URL and endpoint discovery

Hakrawler

Lightweight Go crawler that reads URLs from stdin and prints the links and JavaScript file locations it finds.

ActiveScope a pentestAudit your own exposure

5.2k stars · last push August 5, 2026 · 2.1 · GPL-3.0

GitHub repository

What it finds

Hakrawler crawls each URL piped to it and outputs discovered URLs and JavaScript file locations. The README describes it as a simple implementation of the Gocolly library. Flags control depth, subdomain inclusion, headers, proxying and JSON output.

  • URLs
  • JavaScript file locations
  • forms (with -s source tagging)

Does it touch the target?

Sends traffic to the target's systems. Use only with the owner's permission.

README: "Fast golang web crawler for gathering URLs and JavaScript file locations." It requests the target's pages directly; there are flags to disable redirects and TLS verification and to route requests through a proxy.

API keys: The README documents no API keys or external services.

Install

Install with go
go install github.com/hakluke/hakrawler@latest

From the project's documentation

Install with package
sudo apt install hakrawler

From the project's documentation

How to run it

Examples use reserved documentation hosts such as example.com. Replace them with a domain you own or are authorised to test.

Crawl a single URL
echo https://example.com | hakrawler

From the project's documentation

Crawl a list of URLs with a 5-second limit per URL
cat urls.txt | hakrawler -timeout 5

From the project's documentation

Include subdomains in scope
echo https://example.com | hakrawler -subs

From the project's documentation

Send all requests through a proxy
cat urls.txt | hakrawler -proxy http://localhost:8080

From the project's documentation

Output formats: plain text (stdout), JSON (-json)

Limits the project documents

  • Often returns no URLs when the given domain redirects to a subdomain outside scope; specify the final URL or use -subs.
  • The Kali apt package installs an older version that the author says may be buggy and lacks features.
  • Default crawl depth is 2.

Deepak's take

Analysis, not documentation

Use it when
Use it for a quick, low-setup crawl inside a shell pipeline, for example after probing which hosts respond.
Skip it when
Skip it for JavaScript-heavy apps that render links client-side; Katana's headless mode covers those. Skip it too when no traffic to the target is allowed.
What it will miss
Its README documents no headless browser, so routes built by client-side JavaScript at runtime may not appear.

Facts read from the project's README and docs on November 10, 2026. Repository data as of November 11, 2026. Listed under GitHub's reconnaissance topic. Licence: GPL-3.0. Hakrawler is a trademark or project name of its owners; this library is not affiliated with it.