PLATFORM
What a scan needs from your site.
There is nothing to install. The requirements are about what your site exposes and what your infrastructure permits.
For the site being scanned
- A publicly reachable URL over HTTPS. Pages behind authentication are never fetched.
- A resolvable public hostname. Addresses in private space are rejected by the SSRF guard.
- robots.txt that permits CrawldBot, if you want it crawled. A disallow is obeyed and the scan will report reduced coverage.
- Server-rendered content, ideally. Not a requirement, but a page that paints entirely client-side is what most answer-engine crawlers see as empty, which is itself one of the most common findings.
Allowlisting the crawler
If a WAF or bot filter sits in front of your site, identify us by user agent:
CrawldBot/1.0 (+https://crawld.co/bot)We do not crawl under a browser user agent, and we do not rotate identities to get around filtering. If we are blocked, the scan reports lower coverage rather than working around it.
For using the dashboard
- A current version of any major browser. Nothing exotic is required.
- JavaScript enabled for the application. This marketing site works without it.
- Cookies enabled for authenticated sessions.
For the API
- Anything that can make an HTTPS request and parse JSON.
- The ability to poll: scan submission returns immediately with an id, not a result.
- In production, a CAPTCHA token on scan submission. The check fails closed, so a scan without one is rejected.