The next generation of web crawlers
CrawlerX reads each page the way a browser shows it, after its scripts run. It checks what search engines and AI crawlers are served, and reads the llms.txt files written for AI tools.
How to recognize and block it
CrawlerX sends this User-Agent header, and its robots.txt token is CrawlerX. Until the new name reaches every crawl, requests can carry its earlier name, FathomSpider, so match both.
CrawlerX/1.0 (+https://crawlerx.ai)To block it, answer either name with a 403 at your server or firewall. In Cloudflare, create a custom rule with this expression and set the action to Block.
(http.user_agent contains "CrawlerX") or
(http.user_agent contains "FathomSpider")A robots.txt rule keeps competitor snapshots away, but a default audit reads it and crawls the pages it blocks anyway, so only the 403 stops both. There is no published IP range to match. Technical details has the nginx rule, the robots.txt lines and how to slow CrawlerX down.
What makes it new
On a full audit, CrawlerX opens every page that returned HTML with a 2xx status in headless Chromium and reads it after its scripts run, at desktop and phone sizes. For your home page and 3 others, it sends the request again under the user agents of Googlebot, Bingbot and five AI crawlers, and records where your server answers them differently. Those requests come from CrawlerX, not from the companies that run those crawlers. It also requests /llms.txt, /llms-full.txt and your home page as Markdown, the versions some sites publish for AI tools.
Contact
Email bot@crawlerx.ai with your domain and, if you can, the time and request line from your logs.