◎ SEOlust
Technical

Sitemap vs Robots Coverage Checker

Deep-check how your XML sitemap and robots.txt work together: sitemap declaration, reachability, index expansion, URL counts, and how many URLs are blocked.

All tools
Enter a domain and we auto-discover robots.txt and the sitemap, or paste a specific sitemap URL.

Verifies your sitemap is declared in robots.txt and reachable, expands sitemap indexes, counts URLs, and reports how many are blocked by robots.txt. For a focused list of just the conflicts, see the Sitemap vs Robots Conflict Checker.

Sitemap vs Robots.txt Coverage Checker: Audit Crawlability End to End

Enter a domain and this tool audits how your XML sitemap and robots.txt work together — confirming the sitemap is declared and reachable, expanding sitemap indexes, counting URLs, and flagging every URL you submit for indexing but block from crawling.

What the coverage checker does

A sitemap tells search engines what to crawl; robots.txt tells them what they may not. When the two disagree, crawl budget is wasted and pages silently fail to index. This deep-check inspects the whole pipeline:

  • Finds and parses robots.txt, and lists every Sitemap: directive it declares.
  • Confirms the sitemap is reachable and detects whether it is a sitemap index or a URL set.
  • Expands sitemap indexes and counts the URLs across child sitemaps.
  • Evaluates each URL against robots.txt rules to see how many are blocked from crawling.
  • Flags off-host and non-HTTPS URLs that do not belong in the sitemap.

Why blocked sitemap URLs hurt SEO

Listing a URL in your sitemap is a request to index it. Disallowing that same URL in robots.txt tells crawlers to stay away. Google resolves this contradiction by refusing to crawl the page, which usually means it will not be indexed — and Search Console reports it as 'Indexed, though blocked by robots.txt' or excludes it entirely. Fixing these conflicts is one of the highest-leverage technical SEO cleanups you can do.

How to fix what it finds

Work through the report top to bottom:

  • If robots.txt does not declare your sitemap, add a Sitemap: line so crawlers discover it automatically.
  • If the sitemap is unreachable, fix the URL or server response before anything else.
  • For every blocked URL, decide: it either belongs in the index (then allow it in robots.txt) or it does not (then remove it from the sitemap).
  • Remove off-host and non-HTTPS URLs so the sitemap contains only canonical, same-site https URLs.

How this differs from a conflict checker

A conflict checker gives you the raw list of blocked URLs. This coverage checker adds the surrounding audit — sitemap declaration, reachability, index expansion, URL totals, host and protocol hygiene, and the overall blocked percentage — so you understand not just which URLs conflict but whether your crawl setup is healthy as a whole.

FAQ

Do I need to enter the sitemap URL?
No. Enter just the domain and the tool checks robots.txt for a declared sitemap, then falls back to /sitemap.xml. You can also paste an exact sitemap URL.
Does it follow sitemap index files?
Yes. If the sitemap is an index, it fetches the child sitemaps (up to 20) and aggregates their URLs, up to 2,000 total for speed.
How is 'blocked' determined?
Each sitemap URL's path is tested against your robots.txt rules for Googlebot using longest-match precedence, where Allow beats Disallow on ties — the same way Google evaluates them.
Will blocked URLs always fail to index?
Usually. Google will not crawl a disallowed URL, so it typically will not index its content. Occasionally a blocked URL is indexed URL-only from external links, but without a useful snippet.
Is my site's data stored?
No. The tool fetches your public robots.txt and sitemap at request time and does not store the results.

Related tools

Pro tip: pair this tool with Trailing Slash Consistency Checker and Duplicate Parameter URL Finder for a faster SEO workflow.