Two small files decide what a crawler may read and what it should be pointed at. Reading both for a minute catches the mistakes that quietly hide pages.
A sitemap is an invitation and robots.txt is a rulebook. Neither one is complicated, and both are plain text you can read in a browser tab. The trouble is that they are usually written once and never looked at again, which is how a site ends up asking to be indexed while robots.txt quietly blocks the way.
Read robots.txt the way a crawler does
A crawler fetches robots.txt first and treats every rule as applying to a path prefix. The two rules that matter most are Disallow, which removes a path from consideration, and Allow, which puts it back. A single broad Disallow line at the top of the file is the most common way a whole section of a site disappears from search.
The Sitemap line at the bottom is separate from the rules above it. It is a pointer, not a permission, so a page can be allowed by robots.txt and still never be discovered if nothing links to it.
- Confirm no Disallow line covers a path you want read.
- Confirm Allow lines still match the URLs they were written for.
- Confirm the Sitemap line points at the same domain the crawler is on, including or excluding the trailing slash consistently.
Read the sitemap like a list of promises
Every URL in a sitemap is a claim: this page exists, it loads, and it is the address you want shown. Claims that are false cost more than a missing entry, because a crawler has to spend a request finding out.
The five checks worth doing before a submit
- Fetch robots.txt in the browser and read every Disallow line out loud against your real paths.
- Open the sitemap and pick three URLs at random; load each one and confirm it returns the page, not a redirect to somewhere else.
- Confirm every URL in the sitemap is self-canonical, meaning it points at its own address rather than a variant.
- Check that no URL in the sitemap carries a noindex instruction in the page head.
- Submit the sitemap only after the first four checks pass, because a submit speeds up a discovery that would otherwise happen on a crawl.
Why a submit is not a fix
Pointing a search console at a sitemap does not repair a blocked path or a redirect chain. It only shortens the wait for a crawler to notice pages it was already allowed to reach. The fixes happen in the two files and in the markup, not in the submit form.