PAGES
← all tools
◆ Reconnaissance & OSINT

hakrawler

2.1 open-source

Fast web crawler that gathers URLs and JavaScript file locations from target websites.

source ↗ rss ↗ What is a cyber toolchain?

Summary

hakrawler is an open-source command-line crawler, written in Go, that reads URLs from stdin and gathers the links and JavaScript file locations reachable from them. It costs nothing to use and installs via go install, a Docker image, or apt on Kali, then runs as a pipeable step accepting single or bulk URLs with options for timeouts, proxying, and subdomain scope. It suits penetration testers and bug bounty researchers doing reconnaissance ahead of vulnerability testing, and its README shows it chained with tools like haktrails and httpx for subdomain enumeration and live-host checks before crawling. First committed in 2019 with 27 contributors and three tracked releases, it has had no commits in the past year.

Fast web crawler that gathers URLs and JavaScript file locations from target websites.

How hakrawler helps

Which pages and endpoints on this target can I reach by following links from a single starting URL?

crawls outward from each seeded address and prints every link it discovers, giving a flat list of reachable paths before any active testing begins

Where are the JavaScript files served by this site, so I can review them for secrets or endpoints?

walks the same crawl and reports the location of every script file it finds alongside the page links

Does this target's attack surface extend beyond the domain I was given?

when scope is widened to include subdomains, links pointing at any subdomain of the seed are followed and reported rather than dropped at the domain boundary

Can I fold this into a pipeline that goes from bare domain to crawled URLs without touching a browser?

reads URLs from standard input and writes discovered links to standard output, so it chains after subdomain and live-host tools and before whatever consumes the URL list

Why did crawling this domain come back empty?

if the seed URL redirects to a subdomain not included in scope, nothing is followed from it, so the empty result traces back to a scope mismatch rather than a crawl failure

Do my requests to this target need to go through a monitoring proxy while I recon it?

routes every request the crawl makes through a specified proxy address so traffic can be inspected or logged as it happens

all 7 features, with the evidence for each →

Languages

Not documented

Features

7 capabilities · 5 backed by code, an API document or a real run

Built from hakrawler's releases, docs, README and our own runs. “Seen” marks the month we first spotted a release the vendor didn't date. Group names and tags are our labels.

Compare with peers →

  • hakrawler covers 1 of the 13 network discovery and scanning capabilities we track.

What hakrawler offers you

  • Command-line interface2
  • JSON data export1
All capabilities 7 capabilities
Crawl timeout control verified Limits the maximum time spent crawling each URL. 1 release · 3 other sources · first seen May 2022

release

command line

  • --timeout — Maximum time to crawl each URL from stdin, in seconds. 2.1 · May 2022 · command-line history
  • -timeout — Maximum time to crawl each URL from stdin, in seconds. (default -1) 2.1 · May 2022 · command-line history

example

  • Crawl with 5-second timeout per URL cat urls.txt | hakrawler -timeout 5
JSON output verified Outputs crawl results in JSON format. 1 release · 3 other sources · first seen May 2022

Also counts toward: JSON data export

release

docs

  • Fast web crawler that gathers URLs and JavaScript file locations from target websites.

command line

example

  • Export scan results as JSON for automated processing or integration with other tools hakrawler -url https://example.com -json 2.1 · May 2022 · source
Proxy support verified Routes crawl traffic through a specified proxy URL. 1 release · 3 other sources · first seen May 2022

Also counts toward: Command-line interface

release

command line

example

  • Route traffic through a local proxy server (e.g., Burp Suite) for inspection or interception hakrawler -url https://example.com -proxy http://127.0.0.1:8080 2.1 · May 2022 · source
  • Route all requests through a proxy cat urls.txt | hakrawler -proxy http://localhost:8080
Subdomain crawling verified Optionally includes subdomains when crawling a target. 3 other sources · first seen May 2022

Shared capability: Attack surface inventory

command line

example

  • Chain: find subdomains then crawl all echo google.com | haktrails subdomains | httpx | hakrawler
  • Crawl including subdomains echo https://google.com | hakrawler -subs
Page size limit verified Restricts the maximum page size (in KB) fetched during crawling. 1 other source · first seen May 2022

Also counts toward: Command-line interface

command line

Batch URL crawling shipped suggested Crawls multiple URLs provided from a file. 1 other source

Suggested capability: Attack surface inventory. This is our guess, not confirmed.

example

  • Crawl multiple URLs from file cat urls.txt | hakrawler
Single URL crawling shipped Crawls a single, specified URL. 1 other source

example

  • Crawl a single URL echo https://google.com | hakrawler

Lines in monospace are the tool's own words: help text parsed from its source, or an endpoint from its API document. Under release, a line is our summary of a dated release. Under docs, it is quoted from the tool's own documentation or website. Each links back to the source it came from.

Score card

  1. Shipping velocity releases a month, last 12 months
  2. Documentation quality release-notes rubric, 0–100
  3. Breaking-change discipline how rarely a release breaks users, and notice given
  4. Maintenance risk quiet time against its own cadence, maintainers
  5. Innovation & surface new capabilities, API and MCP surface
  6. Community health
  7. Contributor trend distinct committers, this year against last
  8. Maintainer concentration bus factor now against a year ago
  9. Issue backlog trend issues closed per issue opened, 12 months
  10. Responsiveness median time to a maintainer’s first reply

Nine scores for hakrawler, each built from the releases, notes and API history this site tracks — and, on Researcher and Business, where it stands among every measured tool in its category and how that has moved.

Score cards are part of every paid plan.

See plans →

Each score is 0–100 on a published rubric, derived from data this site already shows. A dimension we cannot measure says so — it is never scored 0 and never enters a rank.

How this is scored
  • Shipping velocity — releases a month over the last 12 months, on a log scale where 16 a month scores 100. Needs 3 releases in the window.
  • Documentation quality — the release-notes rubric published on the analytics page, unchanged.
  • Breaking-change discipline — the share of releases carrying a breaking change (none scores 100, half or more scores 0). Only projects that label breaking changes are measured: a clean record cannot be told from an unlabelled one.
  • Maintenance risk — quiet time measured against the tool's own release cadence (quiet longer than 90 days and three typical gaps scores 0), plus the number of active maintainers and whether contributors are growing, where known. Higher is lower risk.
  • Innovation & surface — new capabilities in the feature ledger this year, API and MCP operations added, and the share of commits that are features: the mean of whichever we hold.
  • Community health (open-source tools only) — each is a direction, not a size: contributors this year against last, the bus factor now against a year ago, and issues closed per issue opened, where steady scores 50, doubled 100 and halved 0; plus the median time to a maintainer's first reply (an hour scores about 100, a month or more 0). A commercial card has no repository and carries none of the four.
  • Documented delivery (commercial tools) — when a vendor's documented new capabilities have slowed against its own baseline, that counts against the overall standing as heavily as two low dimensions. Steady delivery, or too little history to judge, leaves the standing alone; it is never a sixth score.
  • Open-source tools are scored on repository signals; commercial tools on their published changelog. Ranks compare tools within one category and one profile and need at least five measured peers; the overall standing needs three ranked dimensions. Best-in-breed or consolidator is a shape, not a quality, and is never scored.

Release-note detail over time

See how much detail hakrawler puts in its release notes, quarter by quarter, and compare it with tracked tools in the same category. This history uses up to ten retained, reviewed notes per tool; sparse or blank older quarters may reflect that limit.

Checking release-note history…

Explore the full chart →

Releases

1 entry

  1. May 20221

    1 entry

    1. 2.1 May 23, 2022 · release notes

      hakrawler v2.1 adds proxy support, timeouts, JSON output, and page size limits

      └──▷ GET THIS VERSION 
      $ git clone --branch 2.1 https://github.com/hakluke/hakrawler.git
# already have the repo? check out this version:
$ git checkout 2.1
      └──▷ TRY IT
      Export scan results as JSON for automated processing or integration with other tools
      $ hakrawler -url https://example.com -json
      Route traffic through a local proxy server (e.g., Burp Suite) for inspection or interception
      $ hakrawler -url https://example.com -proxy http://127.0.0.1:8080
      • -json flag to output findings in JSON format
      • -proxy option to route requests through a specified proxy
      • -timeout setting to configure request timeout duration

      3 details in Everything

    Project health

    Issue, contributor and maintainer activity in hakluke/hakrawler, 2023-10 – 2026-09, as of 2026-09-25. Twelve-month figures first; the trend lines run the whole window.

    1. Contributors 1 ±0 vs prior 12 months
    2. Maintainers 0 +0 joined · −0 left, last 12 months
    3. Bus factor 0 committers behind half the commits
    4. Community-filed not measured of issues opened, last 12 months
    5. Median time to close not measured issues closed, last 12 months
    6. Open backlog 5 of 97 issues ever filed
    1. Issues opened / month 0 · 0 communitymaintainers
    2. Open backlog 5
    3. Contributors / month 0
    4. Commits / month 0
    5. Stars 5,133
    Issue themes, last 12 months not measured

    Counts from the project's public repository and git history; no individuals are identified. Maintainers are the committers covering 80% of the last year's commits. how we measure this

    Libraries its release ships

    What its latest release locks, checked against known advisories.

    None of its release's images build from a lockfile we can read.

    Add the badge to your README

    Maintain this project? The badge links readers straight to this page, so they can follow every release here.

    Tracked by Cyber Toolchain Choose a badge style and copy the code →
    compare.md

      Free the public facts, no sign-in Any paid plan all other sections Researcher & Business pricing history and category rank

      my-toolchain — 0 tools

        Betas, release candidates and previews. Turn it off to see final releases only. It also applies to changes you ask for over MCP when you're signed in.
        browse all tools → score card for your toolchain →
        cyber-toolchain / feedback

        Send us feedback