https://cybertoolchain.io/tools/colly New releases and features in colly, tracked by The Cyber Toolchain. en Thu, 27 Mar 2025 10:47:28 GMT https://github.com/gocolly/colly/releases/tag/v2.2.0 https://github.com/gocolly/colly/releases/tag/v2.2.0 Thu, 27 Mar 2025 10:47:28 GMT Colly v2.2.0 adds context.Context support, Stop() in queues, MaxRequests cap, Host/custom header support, and content sniffing for HTML parsing. • Adds `context.Context` support to collectors, enabling cancellation and deadline propagation across scraping requests. • Supports Stop() in the queue, allowing graceful halt of queued crawl jobs. • Adds `MaxRequests` parameter to cap the total number of requests a collector will make. • Adds Host and custom header support, giving finer control over outbound request headers. • Implements content sniffing for HTML parsing, improving handling of responses with missing or incorrect Content-Type headers. • Applies URL filters (checkFilters) consistently on redirects, not just initial requests. Full history on Cyber Toolchain: https://cybertoolchain.io/tools/colly