How Search Engines Crawl and Index a Website

"Google doesn't have my site in its index" and "Google isn't crawling my site" sound like the same problem and usually get diagnosed the same way — they aren't the same problem, and treating them as one leads to fixing the wrong thing. Crawling and indexing are two distinct stages, run by different processes, with different failure modes. Ranking is a third stage on top of both. This guide walks through the real mechanics of each.

Stage one: discovery

Before anything can be crawled, Google has to know a URL exists. Discovery happens through a handful of specific paths:

• Following links from pages Google already knows about — this is the original, foundational mechanism, and it's still how most of the web gets found.
• XML sitemaps, which explicitly list URLs you want considered, submitted through Search Console or referenced in robots.txt.
• Manually requesting indexing for a specific URL through Search Console.
• Redirects from previously known URLs, and canonical tags pointing to URLs not yet crawled directly.

A URL with no links pointing to it, absent from any sitemap, and never manually submitted is effectively invisible — an orphan page, discoverable only by accident. Discovery is the precondition for everything that follows.

Stage two: crawling

Once a URL is known, Googlebot decides whether and when to actually fetch it — that decision itself is shaped by crawl budget: your server's capacity and Google's own interest in the page. See the crawl budget guide for the mechanics of that decision, but the short version is: crawling is Googlebot sending an HTTP request to your server and downloading whatever comes back, checked first against your robots.txt rules to confirm it's allowed to fetch that path at all.

If the page relies on JavaScript to display its real content, crawling captures the raw HTML first, and a separate rendering pass may happen afterward to execute the JavaScript and see the fully built page — a process covered in detail in the JavaScript SEO guide. Crawling a page successfully means Google now has its content. It does not yet mean anything about whether that content will be indexed.

Stage three: indexing

Indexing is where Google actually processes the crawled content and decides whether to store it in a form that's eligible to appear in search results. This is a separate, later evaluation, and it's where several distinct checks happen:

Directive checks

Google checks for explicit instructions: a noindex meta tag or HTTP header tells Google outright not to index the page, regardless of content quality. This is a deliberate site-owner instruction, different from robots.txt (which controls crawling access) and different from a canonical tag (which addresses duplication) — see robots.txt vs. noindex vs. canonical for how the three interact.

Canonicalization

If Google finds multiple URLs with substantially similar or duplicate content, it picks one as the canonical version to index and treats the others as variants, using signals like your canonical tag, redirects, internal linking patterns, and URL structure. Google's choice doesn't always match the URL you intended — this is one of the more common, least-understood reasons a specific page doesn't show up as indexed even though a very similar version of it does.

Quality and content evaluation

Google evaluates the actual content, structure, and signals of the page to judge whether it's worth indexing at all. Thin, low-value, or auto-generated content is a common reason a page gets crawled but sits indefinitely as "Crawled - currently not indexed" — Google saw it, and decided against storing it.

Ranking is a separate, fourth thing

Being indexed only makes a page eligible to appear in results for relevant queries — it says nothing about where. Ranking happens at query time, when Google evaluates every indexed page it considers relevant to a specific search and orders them using a much larger set of signals (relevance, authority, page experience, and more). A page can be fully, cleanly indexed and still rank poorly, or not appear for a given query at all, simply because other indexed pages are judged more relevant or authoritative for that particular search.

Checking where a specific page actually stands

The URL Inspection tool in Google Search Console reports exactly where a URL sits in this pipeline: whether it's been discovered, whether it's been crawled and when, whether it's indexed, and — if not indexed — the specific reason Google gives. That specificity matters: "not indexed because of a noindex tag" and "not indexed because Google chose a different canonical" require completely different fixes, even though both show up as "my page isn't in Google."