Reading a page the way a crawler reads it
A search result is built from three fields: the `<title>`, the `<meta name="description">` and the address. A shared link is built from a fourth set: Open Graph (`og:title`, `og:description`, `og:image`, `og:url`, `og:site_name`) for most platforms, and `twitter:*` for X, which also falls back to the `og:` tags by name. This page reads all of them, then shows the two previews side by side with the fallbacks applied, so a page with only `twitter:*` is shown as a page with only `twitter:*` rather than as a page with a card.
The reading is a scan of the text rather than a parse of a live document, which is deliberate: it is what a crawler without JavaScript receives, and it is why a `<meta>` written inside a comment or inside a `<script>` is correctly ignored. The tag reader is the one the minifier uses, so a `>` inside a quoted attribute value does not end a tag here either.
A page with nothing to say about a field is not given a default: the checks report it as missing, and the preview shows what would actually be shown - which for a page with no `og:image` is a grey box, and for a page with no description is a sentence a search engine picked out of the body.
Twenty checks, and why they are a checklist rather than a score
Three of them are failures rather than warnings, because a page cannot really ship without them: a title, a description, and a canonical - plus Open Graph tags, without which a shared link is a bare address. The rest are warnings: a title that is longer than the widely quoted 60 characters, a description shorter than 70, a viewport that does not set `width=device-width`, a missing `<html lang>`, no favicon, an `og:image` that is relative (so a crawler cannot fetch it), an `og:image` whose declared shape is not the card’s 1.91 to 1, an `og:url` that disagrees with the canonical, no `twitter:card`, a tag written twice, JSON-LD that does not parse, no `<h1>` or more than one, and images without `alt`.
The one number this page refuses to invent is the pixel width of a title. Google truncates a result by pixel width, and how many characters fit depends on the letters: sixty `i`s and sixty `W`s are not the same width. So the counts here are exact character counts, the cut marks sit where the guidance is usually quoted, and the page says which of the two it is doing.
The ordering is the ordering of repairs: a title, a description and a canonical first, the cards after them, and the body checks last if a whole page was pasted rather than only a head. A page that passes all twenty is not thereby ranked: not one of these tags is a ranking factor on its own. They decide how a result looks once it is in a list, not whether it gets there.
sitemap.xml: what belongs in one, and what a crawler does with it
A `<loc>` must be an absolute address, and the builder refuses anything else by name and line number rather than writing a file with a relative path in it and hoping. What may follow an address is `<lastmod>`, `<changefreq>` and `<priority>`, in that order, and the page writes them in the order the schema lists them. A URL that is written twice is dropped the second time and reported; characters a sitemap cannot carry literally - non-ASCII, and spaces - are percent-encoded, and the report says which addresses were changed, because `%C3%A9` is a change a reader should know about.
Two limits are real and are checked: 50,000 URLs and 50 MB uncompressed in one file. Beyond that a crawler reads part of the file and stops, so the report says so and suggests a sitemap index. The extras are worth exactly what they are worth, and the page says which: `lastmod` is believed when it is accurate, and a date of “today” on every URL teaches a crawler to ignore the field; `changefreq` and `priority` are ignored by Google entirely. A sitemap does not get a page crawled - a link from a page that is already crawled does that. It says the address exists.
robots.txt: the two lines that look the same and mean the opposite
`Disallow:` with nothing after it allows everything. `Disallow: /` blocks everything. Both look like rules, both are one line, and the second one has taken real sites out of search. The builder and the explanation on the page come from the same list of lines, so the table cannot describe something the file does not say, and the meaning of each line is printed beside it: which paths are blocked, what a `/` would do if it were in that list, and why the `Sitemap:` line is there at all.
The file is a request rather than access control, and it is public: a path listed in it is a path a scraper now knows about. It also works per host, so a subdomain needs its own file, and so does the same site served over both `http` and `https`. `Crawl-Delay` is offered because the directive exists and some crawlers read it, with the honest note that Google ignores it and that a real rate limit belongs in the crawler’s own console.
Danfio