Your post is live. Nothing knows it exists.
A page can render perfectly, sit linked on your index, and still never get crawled, because the one file that lists your URLs was never updated. Here is where that seam usually is, and how to check it in ten minutes.
You publish a post. You open it in the browser and it looks right. It shows up on the blog index with the correct title and date. Three weeks later somebody searches for the exact headline and finds nothing. The page works. It is just not on any list that a crawler reads.
This is the most boring category of bug in web work and the most expensive one for a small business, because nothing alerts you. A broken deploy screams. A page that renders but was never registered anywhere makes no sound at all. Publishing is a chain of four or five independent steps, and every one of them can fail quietly while the others keep succeeding.
Where the seam is in our own build
Our site is static HTML with no framework. The blog pages are generated from a list of posts in a build script: add an entry, run the script, and you get the post page plus an updated index. That part is one source of truth and it is hard to get wrong. The sitemap is a different story. The block of demo URLs gets rewritten automatically by its own script, and everything else in that file is maintained by hand.
So the build can produce a page that the sitemap has never heard of, and nothing anywhere will complain. The file is valid XML. The page returns 200. The index links to it. Two lists exist that both claim to describe the site, and only one of them is generated.
A sitemap does not make anything get indexed. It is a hint. But when it is stale, it is usually a symptom that your record of what you published no longer matches what you deployed.
How to check, in about ten minutes
Do this after every publish until it is automatic, or until a script does it for you. None of it requires tooling beyond a browser and curl.
- Open sitemap.xml directly in the browser and count the URLs. Compare that count against the pages you actually deployed.
- Fetch the page with curl and read the raw HTML. Confirm the link from the index appears in the source, not only after JavaScript runs.
- Read robots.txt line by line. One stray Disallow blocks an entire directory and looks like nothing on the site itself.
- Check that the page returns 200 directly, not through a redirect chain that ends somewhere else.
- Compare the canonical tag on the page against the URL in the sitemap, character for character. They must be the same string.
- A week later, search for the exact title with the site: prefix in Google. If nothing comes back, the chain is broken somewhere above.
That canonical check catches a trap specific to sites with pretty URLs. Our Apache config serves the same page at two addresses, with and without the .html. If the sitemap lists one form, the internal links use the other, and the canonical tag names a third, you have taught the crawler that you have three pages of identical content and no opinion about which one matters.
What correct looks like
- Sitemap edited by hand
- Two lists, one site
- Mismatch nobody sees
- Advertised 404s
- One source of truth
- Sitemap generated in the same run
- Deploy folder diffed against sitemap
- Build fails loudly
One list. The thing that generates your pages should also generate the lines in your sitemap, from the same data, in the same run. If adding a post requires editing two files, then sooner or later somebody edits one. That is not a discipline problem, it is a design problem, and the fix is to remove the second file from human hands.
Underneath that, add a check that fails loudly. Walk the deploy folder, collect every HTML file meant to be public, parse the URLs out of the sitemap, and diff the two sets. Anything on disk but not in the sitemap is an error. Anything in the sitemap but not on disk is a worse one, because it means you are advertising a 404.
If you are the owner rather than the engineer, you only need one question: what generates the sitemap. If the answer is a person, you already know what will happen the week that person is busy. If the answer is the same script that generates the pages, ask to see the check that compares them. Both answers take under a minute to give, and the difference between them is whether your last six months of publishing is findable.