What it is
An XML sitemap lists your canonical URLs and when they last changed. A sitemap index points at several sitemaps, which is how large sites stay under the per-file limits.
Referencing it from robots.txt with a Sitemap: line is what makes it discoverable without anyone guessing the path.
Why an agent cares
Anything reachable only through a JavaScript-driven menu, a search box, or pagination is effectively invisible without a sitemap.
It is also the cheapest freshness signal you have — lastmod tells a crawler what to re-fetch rather than making it re-crawl everything.
How to fix it
Generate it at build time so it can never drift from the routes that actually exist.
- Next.js: export a
sitemap.tsfrom the app directory. - Include only canonical, indexable URLs — no redirects, no
noindexpages. - Add
Sitemap: https://example.com/sitemap.xmlto robots.txt.
How we test it
We follow the Sitemap: lines in robots.txt first, then fall back to /sitemap.xml and /sitemap_index.xml, and confirm a real urlset or sitemapindex document comes back.