colly

No entries match these filters. Clear filters

1. v2.2.0 Mar 27, 2025 · release notes

   Colly v2.2.0 adds context.Context support, Stop() in queues, MaxRequests cap, Host/custom header support, and content sniffing for HTML parsing.

   └──▷ GET THIS VERSION

   copy

   $ git clone --branch v2.2.0 https://github.com/gocolly/colly.git
   # already have the repo? check out this version:
   $ git checkout v2.2.0

   └──▷ USE IT
   Cancel all in-flight scraping requests when a deadline or signal fires, using a standard context.
   go copy

   ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
   defer cancel()
   
   c := colly.NewCollector()
   c.Request("GET", "https://example.com", nil, ctx, nil)

   Stop a running queue mid-crawl, e.g. after a match condition is met during OSINTOSINTOpen-source intelligence, collecting and analyzing information from publicly available sources. Investigators use it to learn about people, infrastructure or threats. collection.
   go copy

   q, _ := queue.New(4, &queue.InMemoryQueueStorage{MaxSize: 10000})
   // ... add URLs ...
   go func() {
       time.Sleep(5 * time.Second)
       q.Stop()
   }()
   q.Run(c)

   • Adds context.Context support to collectors, enabling cancellation and deadline propagation across scraping requests.
   • Supports Stop() in the queue, allowing graceful halt of queued crawl jobs.
   • Adds MaxRequests parameter to cap the total number of requests a collector will make.
   • Adds Host and custom header support, giving finer control over outbound request headers.
   • Implements content sniffing for HTML parsing, improving handling of responses with missing or incorrect Content-Type headers.

   +1 more show less

   • Applies URL filters (checkFilters) consistently on redirects, not just initial requests.

   6 details in Everything

2. v2.0.0 Nov 28, 2019 · release notes

   Colly v2.0.0 adds Collector.SetRedirectHandler, SetClient, HasVisited, and ChildTexts APIs

   └──▷ GET THIS VERSION

   copy

   $ git clone --branch v2.0.0 https://github.com/gocolly/colly.git
   # already have the repo? check out this version:
   $ git checkout v2.0.0

   • Adds Collector.HasVisited method to check if an URL has already been visited during a crawl
   • Replaces the RedirectHandler member with Collector.SetRedirectHandler function for redirect logic

   2 details in Everything

   └──▷ BREAKING ON UPGRADE

   • !The Collector.RedirectHandler struct field is removed; use Collector.SetRedirectHandler to configure redirects instead
3. v1.2.0 Feb 13, 2019 · release notes

   colly v1.2.0 adds HEAD request shortcut and URL availability check

   └──▷ GET THIS VERSION

   copy

   $ git clone --branch v1.2.0 https://github.com/gocolly/colly.git
   # already have the repo? check out this version:
   $ git checkout v1.2.0

   • Adds new request shortcut for HEAD requests
   • Adds ability to check URL availability before visiting

   2 details in Everything

4. v1.1.0 Aug 28, 2018 · release notes

   Colly v1.1.0 introduces ForEachWithBreak, gzip support, and Appengine context integration.

   └──▷ GET THIS VERSION

   copy

   $ git clone --branch v1.1.0 https://github.com/gocolly/colly.git
   # already have the repo? check out this version:
   $ git checkout v1.1.0

   └──▷ USE IT
   Stop scraping a list of URLs as soon as a specific item is found, avoiding unnecessary network requests.
   text copy

   html.ForEachWithBreak(".item", func(e *colly.HTMLNode) bool {
     if e.Attr("data-skip") == "true" {
       return false // Stop iteration
     }
     fmt.Println(e.Text)
     return true
   })

   Parse a JSON array containing structs where one field is a pointer, now supported by the unmarshaler.
   text copy

   var items []MyStruct
   err := collector.UnmarshalJSON(resp.Body, &items)
   if err != nil {
     log.Fatal(err)
   }

   Update Appengine integration to use the new context-aware API signature.
   text copy

   func Handle(w http.ResponseWriter, r *http.Request) {
     c := colly.NewAppengine(colly.AppengineCtx(r.Context()))
     // ...
   }

   • context.Context is now passed to the Appengine integration instead of http.Request. colly/appengine
   • New Accept HTTP header is added by default to every request in all collectors
   • Unmarshaling now supports slices of pointers and structs, enabling complex data parsing from JSON responses
   • Added ForEachWithBreak method to HTMLElement to allow early termination during element iteration
   • Response body decompression now supports gzip encoded content automatically

   5 details in Everything

   └──▷ BREAKING ON UPGRADE

   • !The Appengine integration API now requires context.Context instead of the previous http.Request parameter
5. v1.0.0 May 14, 2018 · release notes

   Colly v1.0.0 first major release with rate limiting, request caching, proxy switchers, storage backends, debuggers and configurable extensions.

   └──▷ GET THIS VERSION

   copy

   $ git clone --branch v1.0.0 https://github.com/gocolly/colly.git
   # already have the repo? check out this version:
   $ git checkout v1.0.0

   • extensions package provides helpers like RandomUserAgent to generate random User Agents
   • Configurable via environment variables for fine-tuning without rebuilding
   • Supports Redis and SQLite3 storage backends alongside the default memory backend
   • extensions package includes RequestCaching support to cache responses and reduce external traffic
   • Proxies/proxy switchers allow distributing requests among multiple hosts for scaling

   +2 more show less

   • Supports distributed scraping via request queue management across worker threads
   • Debuggers interface with LogDebugger for console output and WebDebugger for web interface inspection

   7 details in Everything