gospider
open-sourceThe section your link pointed to isn't on gospider's page any more, so we've brought you to the top.
GoSpider is a fast web spider written in Go that crawls websites, parses robots.txt and sitemap.xml, and extracts URLs from JavaScript files.
Summary
gospider is an open-source command-line web crawler written in Go, free to use, built for fast reconnaissance against one or many target sites at once. It parses robots.txt and sitemap.xml, extracts and verifies links from JavaScript files, spots AWS S3 buckets and subdomains in response bodies, and pulls historical URLs from the Wayback Machine, Common Crawl, VirusTotal, and AlienVault, with output formatted for grepping or JSON parsing. It is aimed at penetration testers and bug bounty hunters doing recon rather than continuous monitoring, and its README describes it as part of the Osmedeus recon engine. First committed in 2020 and carrying ten releases since, the project has had no commits in the past year, so treat it as stable but no longer actively maintained.
GoSpider is a fast web spider written in Go that crawls websites, parses robots.txt and sitemap.xml, and extracts URLs from JavaScript files.
How gospider helps
What every reachable page and endpoint under this domain actually is?
crawls the site following links, sitemap, and robots.txt entries and lists every URL it discovers, so you get a map of the site's real surface instead of just the pages you already knew about
Which old or forgotten endpoints still exist outside my current site map?
pulls historical URLs for the target from the Wayback Machine, Common Crawl, VirusTotal, and AlienVault archives, surfacing paths that no longer show up in navigation but may still respond
Have any secrets or extra endpoints leaked into the client-side code?
parses JavaScript files pulled from the site, extracts embedded links and paths, and verifies which of them are live
Is this application or its assets exposing cloud storage or forgotten hosts?
scans response bodies for AWS S3 bucket references and subdomain names, flagging them alongside the page they were found on
Can I recon every site in a scope at once without running separate scans?
takes a list of sites and crawls them in parallel with configurable thread and concurrency counts, producing one combined output set
Which results actually matter once I filter out the noise?
supports narrowing output by response length and restricting scope with whitelist or blacklist rules, and can format results as JSON or plain grep-able text for pipelining into other tools
Languages
Command line
| option | found in | since | description |
|---|
Values are placeholders taken from each option’s declared default. Nothing is executed here — the output shown is a recording of a run that already happened.
Score card
-
Shipping velocity releases a month, last 12 months
-
Documentation quality release-notes rubric, 0–100
-
Breaking-change discipline how rarely a release breaks users, and notice given
-
Maintenance risk quiet time against its own cadence, maintainers
-
Innovation & surface new capabilities, API and MCP surface
- Community health
-
Contributor trend distinct committers, this year against last
-
Maintainer concentration bus factor now against a year ago
-
Issue backlog trend issues closed per issue opened, 12 months
-
Responsiveness median time to a maintainer’s first reply
Nine scores for gospider, each built from the releases, notes and API history this site tracks — and, on Researcher and Business, where it stands among every measured tool in its category and how that has moved.
Score cards are part of every paid plan.
See plans → Already subscribed? Sign in →Trajectory
Blank means never measured; nothing here is a filled-in zero.
Peer standings are included with Researcher and Business. See how this tool ranks against its competitors, using data we don't publish anywhere else.
See plans →Each score is 0–100 on a published rubric, derived from data this site already shows. A dimension we cannot measure says so — it is never scored 0 and never enters a rank.
How this is scored
- Shipping velocity — releases a month over the last 12 months, on a log scale where 16 a month scores 100. Needs 3 releases in the window.
- Documentation quality — the release-notes rubric published on the analytics page, unchanged.
- Breaking-change discipline — the share of releases carrying a breaking change (none scores 100, half or more scores 0). Only projects that label breaking changes are measured: a clean record cannot be told from an unlabelled one.
- Maintenance risk — quiet time measured against the tool's own release cadence (quiet longer than 90 days and three typical gaps scores 0), plus the number of active maintainers and whether contributors are growing, where known. Higher is lower risk.
- Innovation & surface — new capabilities in the feature ledger this year, API and MCP operations added, and the share of commits that are features: the mean of whichever we hold.
- Community health (open-source tools only) — each is a direction, not a size: contributors this year against last, the bus factor now against a year ago, and issues closed per issue opened, where steady scores 50, doubled 100 and halved 0; plus the median time to a maintainer's first reply (an hour scores about 100, a month or more 0). A commercial card has no repository and carries none of the four.
- Documented delivery (commercial tools) — when a vendor's documented new capabilities have slowed against its own baseline, that counts against the overall standing as heavily as two low dimensions. Steady delivery, or too little history to judge, leaves the standing alone; it is never a sixth score.
- Open-source tools are scored on repository signals; commercial tools on their published changelog. Ranks compare tools within one category and one profile and need at least five measured peers; the overall standing needs three ranked dimensions. Best-in-breed or consolidator is a shape, not a quality, and is never scored.
Release-note detail over time
See how much detail gospider puts in its release notes, quarter by quarter, and compare it with tracked tools in the same category. This history uses up to ten retained, reviewed notes per tool; sparse or blank older quarters may reflect that limit.
Checking release-note history…
Releases
Nothing shipped since we started watching. Releases and docs updates land here as they happen.
Project health
Issue, contributor and maintainer activity in jaeles-project/gospider, 2023-10 – 2026-09, as of 2026-09-25. Twelve-month figures first; the trend lines run the whole window.
- Contributors 0 ±0 vs prior 12 months
- Maintainers 0 +0 joined · −0 left, last 12 months
- Bus factor 0 committers behind half the commits
- Community-filed not measured of issues opened, last 12 months
- Median time to close not measured issues closed, last 12 months
- Open backlog 48 of 61 issues ever filed
- Issues opened / month 0 · 0 communitymaintainers
- Open backlog 48
- Contributors / month 0
- Commits / month 0
- Stars 2,998
Counts from the project's public repository and git history; no individuals are identified. Maintainers are the committers covering 80% of the last year's commits. how we measure this
Libraries its release ships
What its latest release locks, checked against known advisories.None of its release's images build from a lockfile we can read.
Add the badge to your README
Maintain this project? The badge links readers straight to this page, so they can follow every release here.
Choose a badge style and copy the code →