For site owners and server admins
The BlueFrog crawler.
If you found this page in your logs: this is who fetched your site, what it took, and how to make it stop.
Who it is.
Two user agents, one company. Both link here or to the tools page so you can always tell which one you saw.
BlueFrogTools/1.0 (+https://bluefroganalytics.com/tools)- The free tools. A person typed a URL into a check and this fetched it for them, on demand, once. If a host refuses that user agent outright, one retry is made with an ordinary browser user agent so the person still gets an answer.
BlueFrogBot/1.0 (+https://bluefroganalytics.com/crawler)- The batch scanner behind Infrasight and the OpsBlu data: the index of 18M+ websites fingerprinted, re-checked on a schedule, recognising 3,127 technologies from one fetch.
Requests come from BlueFrogAnalytics LLC in Portland, Oregon. The reverse DNS of our addresses resolves to bluefroganalytics.com.
What it fetches.
- The homepage. One GET of the URL as given (or https://<domain>/), following up to ten redirects. Bodies are capped at 5 MB; nothing larger is read.
- /robots.txt. Read before any batch fetch and honoured (see below). On a visitor-run check it is only read when the check is about robots.txt.
- A sitemap. The sitemap(s) declared in robots.txt, or /sitemap.xml, fetched once to count URLs and check freshness. Pages listed in it are not crawled.
- One JavaScript render. Only when the plain HTML is not enough to identify the stack (a blank app shell). A single headless Chrome load of the same homepage, no clicks, no scrolling, no forms, nothing submitted.
What it never does.
- Never logs in, never fills a form, never follows links beyond the homepage, robots.txt and the sitemap.
- Never fetches images, video, fonts or downloads; only HTML, text and XML.
- Never runs from a private address or against one: localhost, LAN and cloud-metadata ranges are refused before any request leaves.
- Never ships your HTML to anyone: a fetched page is cached for ten minutes so ten tools asked about the same site cost the site one request, then it is dropped. What is kept is the result (status, headers, scores, the technologies detected).
How often.
Monthly for the full index. Each site in the index is re-fetched about once a month; a site that changes its stack is noticed on the next pass, not sooner. That is one homepage, one robots.txt and one sitemap per site per month, plus the single JavaScript render when the HTML alone could not identify the stack.
On demand when a visitor runs a check. Someone pasting your URL into a free tool triggers one fetch of that URL, right then, with the tools user agent. The result is cached for ten minutes, so a visitor clicking through several tools costs your site one request, not ten.
robots.txt and rate limits.
The batch scanner reads your robots.txt first and honours Disallow rules for BlueFrogBot and for *. A disallowed site is skipped for that pass and the ones after it; it is not retried under another name.
Visitor-run checks are a person asking about one URL, the same as opening it in a browser, so they fetch the URL typed and nothing else; they do not walk your site.
Both agents hold one connection per host and pace to at most one request per second per host, with a 30-second timeout. A 429 or 503 answer backs the scanner off that host for the rest of the pass. Crawl-delay, where set, is honoured up to 60 seconds.
How to block it.
Add this to your robots.txt and the batch scanner stops on its next pass (within a month, usually within a day):
User-agent: BlueFrogBot
Disallow: /To stop the on-demand tools from fetching your site as well, refuse the user agent at your edge:
User-agent: BlueFrogTools
Disallow: /Or email us: hello@bluefroganalytics.com from an address at the domain, or use the contact page, with the domain and "do not crawl". It is added to the exclusion list by hand within one business day and stays excluded. To have data about your site removed as well, use data rights.
See what the crawler sees.
The same fetch, pointed at your own site. Free, no login.