CitedBot
CitedBot is the crawler behind CitedBy site audits: it crawls your pages, analyses them and reports what to fix. Its User-Agent header is:
Mozilla/5.0 (compatible; CitedBot/1.0; +https://citedby.eu/bot)
When it visits
Only when a site owner or a visitor asks CitedBy to audit that site. It does not crawl on its own, and it does not build an index of the web.
How it behaves
- It reads your robots.txt first and obeys it. It follows the group
User-agent: CitedBotand, if there is none, the groupUser-agent: *. - It waits at least 1 second between requests to your site.
- It stops at a page limit set by the plan of the audit: 500, 1,000, 5,000 or 10,000 pages.
- It only reads public pages. It does not log in, submit forms or run scripts.
Block it
Add this to your robots.txt and it will not fetch any page:
User-agent: CitedBot
Disallow: /
To keep it out of one folder only, use Disallow: /folder/ instead.
Contact
Questions or a request to stop crawling: hello@citedby.eu. We answer the same business day.