Robots.txt and sitemaps, explained without the jargon
Two files sitting quietly on your domain control whether search engines can read your site at all. Most people never look at them. When one is wrong the effect is total, and it is invisible from the front of the website.
· 5 min read
What robots.txt actually does
Robots.txt is a plain text file at yourdomain.com/robots.txt. It tells crawlers which parts of your site they may fetch. That is the whole job.
It is a set of instructions, not a lock. Well-behaved crawlers such as Google and Bing follow it. It is not a security feature and it will not hide anything from a determined visitor, so private material needs a login, not a robots rule.
Type your domain followed by /robots.txt into a browser right now. If you get a page of text, read it. If you get a 404, that is fine and means everything is allowed.
The one line that can hide your whole site
If robots.txt contains Disallow: / with nothing else, you have told every search engine to ignore the entire site.
This happens more often than you would think. It is the default in many staging environments, and it survives the move to production when nobody thinks to check. A site can sit invisible for months while the owner assumes SEO is slow.
If you see that line and your site is meant to be public, that is your problem and nothing else matters until it is gone. Fix it, then use Google Search Console to request a recrawl rather than waiting.
What a sitemap is for
A sitemap is an XML file listing the pages you want indexed. It usually lives at yourdomain.com/sitemap.xml, though it can be anywhere as long as you say where.
It does not force anything to be indexed. It is closer to handing someone a contents page. Google still decides what to include, but a sitemap means discovery does not depend on your internal links being perfect.
Small sites with good navigation can survive without one. Sites with pages that are not linked from anywhere prominent, which is most sites over about thirty pages, benefit noticeably.
Make the two files talk to each other
Add a line to robots.txt pointing at your sitemap. It looks like this, on its own line: Sitemap: https://yourdomain.com/sitemap.xml
This is the single most skipped step. Without it, a crawler has to guess where your sitemap lives, and if yours sits at a non-standard path such as sitemap-index.xml, it may never find it at all.
One line, thirty seconds, and it removes the guesswork permanently.
Keep the sitemap honest
A sitemap full of dead URLs teaches crawlers to trust it less. Every entry should return a 200 and be a page you genuinely want in search results.
Common rubbish to remove: pages that redirect, pages carrying a noindex tag, tag and category archives that duplicate other pages, and old URLs from a site migration three years ago.
If your sitemap is generated by a plugin or your platform, check what it is actually producing rather than assuming. Auto-generated does not mean correct.
Blocking and hiding are different jobs
This trips up almost everyone at some point. Robots.txt controls whether a crawler may fetch a page. A noindex tag, which sits in the page's own HTML, controls whether it appears in results. They sound similar and they do opposite halves of the job.
The trap is using both at once on the same page. If you block a page in robots.txt, the crawler never fetches it, so it never reads the noindex tag inside it. The page can then still show up in results, usually with no description, because Google knows the address exists from links elsewhere and was never allowed to look inside and see that you did not want it listed.
The rule to remember: to keep a page out of search results, let crawlers fetch it and give it a noindex tag. Use robots.txt for things you want left alone entirely, such as an admin area or a search results page that would waste crawl budget.
Check it once, then leave it
This is not ongoing work. Get both files right, submit the sitemap once in Google Search Console and Bing Webmaster Tools, and you are done until the site changes structurally.
Worth rechecking after a redesign, a platform migration, or any time a developer touches the server config. Those are the moments a stray Disallow reappears.
The tool runs these two checks automatically against your live domain and shows a pass or fail, which is the first thing on the action plan precisely because everything else depends on it. Once that is green, the ranking guide covers what to look at next.