THE CRAWLER

CrawldBot, and how to stop it.

If you found this page from your access logs, this is everything about what our crawler does and how to turn it off.

How to identify it

Every request we make carries this user agent:

CrawldBot/1.0 (+https://crawld.co/bot)

The URL in the string points here, so the bot always identifies itself and always links to instructions for blocking it. We do not crawl under a browser user agent.

What it fetches

Public HTML, and the resources a page references that are needed to judge it: stylesheets, the robots.txt, the sitemap, structured data. It does not submit forms, does not follow links behind authentication, and does not attempt to access anything a signed-out visitor could not.

Why it might be visiting you

There are two reasons, and they behave differently:

  • Someone scanned their own site. A customer, or a visitor running the free scorecard on a domain they control. This is a one-off crawl of a capped number of pages.
  • Someone is monitoring you as a competitor. Crawld's SEO Watchdog lets a customer track up to three competitor sites for changes to titles, meta descriptions, headings, body copy, pricing and links. If that is why we are here, the limits below are what constrain it.

The limits it works within

  • 12 pages per site on a competitor crawl. It does not walk your whole site.
  • One request at a time. No parallel fetching, ever. It is not capable of loading your server.
  • Once a week. The scheduler runs on a 168-hour interval.
  • robots.txt is obeyed, including crawl-delay.

How to block it

Add this to your robots.txt and we will stop on the next scheduled run:

User-agent: CrawldBot
Disallow: /

To allow most of the site but keep it out of one area:

User-agent: CrawldBot
Disallow: /internal/
Disallow: /staging/

If you would rather not wait for the next run, or you believe the crawler is misbehaving, contact us and we will add the domain to a permanent exclusion list.

What we do with what we collect

Crawl results are stored against the account that requested them and are used to produce that account's scorecard or change report. We do not resell crawl data and we do not train models on the content we fetch.