SearchEngineOptimizationBot
Our crawler. If it visited your site, someone who verified they own that site asked us to map how its pages link to each other.
SearchEngineOptimizationBot/1.0 (+https://searchengineoptimization.io/bot)It builds a link map, nothing more.
This is a lightweight structural crawl. It requests pages, reads the links on them, and follows the internal ones to build a picture of how a site connects to itself. It does not copy your content, submit forms, run scripts against your systems, or collect personal data. It is separate from any ranking, backlink, or full page-audit data in the product, which come from other sources.
How it behaves
- Only verified sitesIt only ever crawls a domain after the person who added it proved they control it, by DNS record, an uploaded file, or Google Search Console. It does not crawl the open web, and it will not visit a site nobody has verified.
- It identifies itselfEvery request carries the user-agent above and links back to this page, so you can always tell it apart from anonymous traffic in your logs.
- It does not use robots.txtBecause it runs only on a property whose owner has already authorised it, it maps the whole site rather than obeying robots.txt. Finding a page you meant to keep out of search is part of what the map is for. The owner controls it by removing the site.
- It's gentleOne page at a time, with a pause between requests and a hard cap on how many pages a single crawl fetches. It is not a load test.
Talk to the site owner, or us.
If this crawler reached a site you run and you did not add it to searchengineoptimization.io yourself, someone with verified access to the domain did. Start with whoever controls the site. If you think that's wrong, get in touch and we will look into it.