How Search Engines Work

The instructor is ready

How Search Engines Work

Interactive digital-human course

How Search Engines Work

This training explains how search engines discover, index, and rank web content for users seeking information.

My workspace32 minFree to watch

What you’ll learn

  1. 01How Search Engines Work: Crawling, Indexing, and ServingWelcome to How Search Engines Work. This course is designed for technical creators who need a grounded, systems-level understanding of the mechanisms behind search. We'll move past marketing metaphors and focus on the actual infrastructure. Our goal is to trace a single URL through its entire lifecycle, from initial discovery to its appearance as a search result. The process is built on three core pillars: crawling, indexing, and ranking, which is also called serving. We'll treat these as distinct, logical stages in a pipeline. In the next slide, we'll start by examining the first major constraint of that pipeline: the scale of the web itself.How Search Engines Work: Crawling, Indexing, and Serving1 min
  2. 02The Scale Problem: Web Size, Freshness, and Crawl BudgetNow let's confront the scale problem directly. The web contains trillions of pages, and exhaustive crawling is simply impossible. Search engines must make resource allocation decisions, and that is where crawl budget enters the picture. Google defines crawl budget as the set of URLs it can and wants to crawl on your site. It is not a fixed allowance, but rather a dynamic calculation governed by two independent forces. The first is crawl rate limit, which is a ceiling based on your server's capacity. If your server's Time to First Byte climbs past 500 milliseconds or you start returning frequent 5xx errors, Googlebot automatically backs off to protect your infrastructure. The second force is crawl demand, which reflects how much Google actually wants to crawl your pages, driven by signals like popularity, freshness, and what Google calls your perceived inventory. The binding constraint is whichever factor is lower. If demand is high but your server is slow, capacity caps you. If your server is fast but demand is low, Google simply will not crawl more. The critical takeaway for today is that wasted crawl budget is no longer just a technical SEO issue. Every crawl spent on a duplicate, a redirect chain, or a low-value parameterized URL is a crawl not spent on a page you want indexed. And a page that does not get crawled cannot be indexed, which means it cannot appear in any AI-powered search surface. Let's move on to how Google discovers URLs and manages its crawl queue.The Scale Problem: Web Size, Freshness, and Crawl Budgetdevelopers.google.comindexcraft.inincremys.com+22 min
  3. 03Discovery: URL Sources and the Crawl QueueNow, let's examine how Googlebot actually discovers URLs and decides which ones to crawl. The crawl queue is fed by a few primary sources, with XML sitemaps, external backlinks, and the URL Inspection tool being the most significant. Once a URL enters the queue, it isn't processed on a first-come, first-served basis. Priority is determined by a combination of authority signals, freshness indicators, and your server's historical performance. The overall pace of crawling is governed by two competing forces: crawl demand, driven by a URL's popularity and how stale it might be, and the crawl rate limit, which is strictly controlled by your server's health and response times. We also need to be precise about the control mechanisms. A robots.txt file or a `noindex` tag can block access, but a `nofollow` attribute is just a hint. It does not prevent crawling, especially if the URL is discovered through another path. Understanding this queue logic is fundamental to managing a large site's crawl budget. Next, we'll move inside the crawler to see how fetching actually works at scale in 'Inside the Crawler: Fetching at Scale'.Discovery: URL Sources and the Crawl Queuedevelopers.google.comdevelopers.google.comdevelopers.google.cn+22 min
  4. 04Inside the Crawler: Fetching at ScaleNow let's look inside the crawler itself and how it fetches at scale. When Googlebot requests a URL, it fetches only the first two megabytes of the resource, including HTTP headers. Any bytes beyond that cutoff are entirely ignored. They are not fetched, not rendered, and not indexed. The fetch limit is separate for each embedded resource, like CSS and JavaScript files, so large inline scripts in the HTML can push critical content past the two-megabyte mark. Server performance also directly governs crawl behavior. Time to First Byte acts as a throttle: fast servers get more concurrent connections, while slow responses cause Googlebot to back off. Similarly, if your origin returns frequent five-hundred-series errors or timeouts, the crawler will reduce activity to protect your infrastructure. Finally, the Web Rendering Service operates statelessly. It clears cookies, local storage, and session data between requests. This means paywalls and login walls will block rendering entirely, because WRS arrives with no prior session and no authentication state. Next, we will explore the two-wave indexing process: crawl, then render.Inside the Crawler: Fetching at Scaledevelopers.google.cndevelopers.google.comvegaseotalks.com+22 min
  5. 05The Two-Wave Indexing Process: Crawl, Then RenderNow let's step through the two-wave indexing process. The first wave is an immediate HTML crawl, where Googlebot fetches the raw document. It extracts the initial text, meta tags, and links, processing only the first two megabytes of the file. Anything beyond that cutoff is simply ignored. The second wave is where things get more complex. The URL enters a deferred rendering queue, feeding into the Web Rendering Service, or WRS. This is a headless Chromium environment that executes your JavaScript. It's critical to understand that the WRS operates statelessly. No cookies, local storage, or service workers persist between rendering passes. This means the rendering delay can range from hours to weeks, which is why server-side rendering and static site generation are absolutely critical for reliable indexing today. Furthermore, keep in mind that major AI crawlers, like GPTBot and ClaudeBot, do not execute JavaScript at all. For them, your initial HTML must be completely self-sufficient. This two-wave reality has a direct impact on how your content is parsed and tokenized next.The Two-Wave Indexing Process: Crawl, Then Renderdevelopers.google.cndevelopers.google.comvegaseotalks.com+22 min
  6. 06Parsing and Extraction: From HTML to Meaningful TokensNow, let's look at what happens once the raw HTML is fetched: parsing and extraction. The goal here is to turn a messy HTML document into meaningful tokens the indexer can actually use. Googlebot parses the HTML to build the Document Object Model, or DOM tree. But it doesn't stop at static source code. The Web Rendering Service executes JavaScript, so content injected by scripts, iframes, and even shadow DOM is included in the final rendered tree. From this complete structure, Google extracts the visible text and discovers all outgoing links. It also pulls out critical metadata, such as the canonical URL, hreflang tags for internationalization, and structured data in JSON-LD format. However, JSON-LD must be syntactically perfect. A single trailing comma or a missing quote makes the entire block unparsable, which blocks eligibility for rich results entirely. Next, we'll explore how all this extracted text gets organized into a searchable structure, in 'The Inverted Index: How Documents Become Searchable.'Parsing and Extraction: From HTML to Meaningful Tokenssupport.google.comsupport.google.comsupport.google.com+21 min
  7. 07The Inverted Index: How Documents Become SearchableLet's walk through the inverted index, the core data structure that makes a search engine fast. Instead of mapping documents to the words they contain, an inverted index maps each unique token to the list of documents where it appears. When a query comes in, the engine doesn't scan every page; it goes straight to the relevant tokens and retrieves the matching document IDs. Before that lookup happens, the text goes through essential preprocessing. Tokenization breaks the raw text into individual terms. Stemming reduces words to their root forms, so "running" and "ran" both land on the same token. Normalization handles case, accents, and other variations to collapse equivalent terms. For multilingual text, this pipeline becomes more complex, but the goal remains the same: turn messy human language into clean, searchable tokens. Modern indexes take this a step further by integrating vector embeddings. Alongside the traditional token-to-document mapping, we now store dense numerical vectors that capture the semantic meaning of the text. This is what powers hybrid search: a single query can now combine precise lexical matching using an algorithm like BM25 with semantic retrieval based on vector similarity. The search engine runs both retrieval paths in parallel, fuses the results using a technique like Reciprocal Rank Fusion, and returns a single ranked list. This hybrid approach is also the retrieval backbone for AI features like Retrieval-Augmented Generation, AI Overviews, and intent-driven retrieval. Next, we'll look at how the search engine interprets the query itself through Query Processing, Intent, Entities, and the Knowledge Graph.The Inverted Index: How Documents Become Searchabledocs.opensearch.orglinkedin.com2 min
  8. 08Query Processing: Intent, Entities, and the Knowledge GraphNow, let's step inside the query processor. When a user types a search, the first job is to refine what they actually meant. This means query rewriting, automatic spelling correction, and synonym expansion all happen before the real work begins. Next, named entity recognition, or NER, classifies tokens into types like Person, Organization, or Location. After that, the Knowledge Graph disambiguates the term. For example, it resolves the company "Apple" from the edible fruit. Finally, state-of-the-art models like BERT parse the context bidirectionally, while MUM extends this capability to multimodal tasks, such as understanding a query that combines an image with text across multiple languages. These layers together translate a raw string into a precise, machine-readable intent. Next, we'll explore how that intent moves into the ranking stage with the multi-stage architecture.Query Processing: Intent, Entities, and the Knowledge Graphdocs.opensearch.orglinkedin.com1 min
  9. 09Ranking: The Multi-Stage ArchitectureNow let's look at the multi-stage architecture that actually ranks search results. Think of it as a funnel, not a single formula. It starts with candidate retrieval, a broad sweep that pulls potentially relevant pages from the index. This is followed by a first-pass ranking, which applies lighter, faster signals to narrow the pool. Then, a second-pass ranking layer applies the most computationally expensive and nuanced signals to the final candidates. The old PageRank and link quality signals have evolved into sophisticated topical authority assessments. Google is not just counting links; it is measuring whether your site is a genuine, focused authority on a subject. User interactions, like clicks and dwell time, are used as training data to improve the models, but they are not direct, real-time ranking factors. A key modern signal is the Information Gain score. This rewards content that teaches the search engine something new, something not already present in the top-ranking pages. Finally, site-level classifiers look at your entire domain. A pattern of thin, unhelpful pages can drag down the ranking potential of your genuinely strong articles. This is why a holistic site quality strategy is non-negotiable. Next, let's connect this directly to the E-E-A-T core ranking signals.Ranking: The Multi-Stage Architecture2 min
  10. 10E-E-A-T: The Core Ranking Signals in 2026Now let's look at the core ranking signals that define quality in 2026. Google's evaluation framework is commonly summarized as E-E-A-T: Experience, Expertise, Authoritativeness, and Trustworthiness. These are not simple numeric scores, but qualitative judgments made by human raters, whose feedback then trains machine-learning classifiers. The most critical thing to understand here is the Helpful Content System. It is no longer a separate update; it was integrated into the core algorithm. It now functions as a site-wide classifier. This means a pattern of thin, unoriginal content on one part of your site can suppress the rankings of your genuinely good pages. To counter this, Google's systems heavily weight Information Gain. This metric measures whether your content teaches the reader something new, compared to the pages already ranking. In this environment, demonstrating deep topical authority and providing original insights completely outweighs a high publishing volume. The goal is to build a site where nothing qualifies for demotion. Next, we'll move into how these quality signals are assembled and presented to the user in the slide titled Serving: Assembling the Modern SERP.E-E-A-T: The Core Ranking Signals in 20262 min
  11. 11Serving: Assembling the Modern SERPNow let's look at how the modern search results page is actually assembled. When a user hits enter, what they see is far from just ten blue links. In fact, over 65 percent of queries now trigger some form of SERP feature, from AI Overviews and knowledge panels to featured snippets. These rich elements dominate the space above the fold, pushing traditional organic results further down the page. The assembly itself is highly personalized. Real-time signals like the user's location, search history, and device type are factored in to tailor the entire page. This shift has also led to the rise of zero-click searches, where a question is answered directly on the results page. While that can reduce overall clicks, when your content is the one cited inside these features, the traffic you do get is often highly qualified. Next, we'll focus on the most prominent of these new features: AI Overviews, and how they are becoming the new front door to search.Serving: Assembling the Modern SERPdocs.opensearch.orglinkedin.com1 min
  12. 12AI Overviews: The New Front Door to SearchNow, let's turn to the feature that's reshaping the top of the search results page: AI Overviews. These are the AI-generated summaries that synthesize answers from multiple sources, placing a citation-rich block above the organic listings. For complex queries, this is becoming the new front door to search. But being cited here isn't random. The data shows seventy-one percent of citations come from pages already ranking in the top ten organic positions. Simply put, if you are not already on page one, your content is rarely eligible to be referenced. Beyond traditional ranking, there's a critical architectural requirement. Major AI crawlers like GPTBot and ClaudeBot do not render JavaScript. They only read the initial HTML. This means server-side rendering or static site generation is essential for visibility in these AI systems. Fortunately, the work you do for featured snippets directly strengthens your AI Overview citation potential. The same clear, structured answers that win position zero also make your content a reliable source for AI models. Next, we'll take these principles into a practical audit of crawlability and indexability.AI Overviews: The New Front Door to Searchdocs.opensearch.orglinkedin.comdevelopers.google.cn+22 min
  13. 13Practical Audit: Crawlability and IndexabilityLet's move into a practical audit of crawlability and indexability. Start in Google Search Console's Crawl Stats report. Review the 90-day trend for total crawl requests, and confirm host status is green. Watch for server errors coded 5xx, and check for DNS or connectivity failures. Next, cross-reference with the Page Indexing report. Look for pages that were crawled but not indexed, and note the reasons. For server health, monitor Time to First Byte, DNS resolution time, and keep the error rate below zero point one percent per any four-hour window. Use the URL Inspection Tool to examine the rendered HTML, the canonical URL Google selected, and which structured data was detected. Finally, validate your JSON hyphen LD with the Rich Results Test. Fix syntax errors like trailing commas, unclosed braces, and missing colons immediately. These diagnostics connect crawling activity to real indexing outcomes. Next, we'll look at structuring content for modern indexability.Practical Audit: Crawlability and Indexabilitysupport.google.comsupport.google.comsupport.google.com+22 min
  14. 14Structuring Content for Modern IndexabilityLet's operationalize indexability. Start with semantic HTML and a clean information architecture. This gives crawlers an unambiguous content hierarchy, so they know which sections carry the most weight. Next, direct your crawl budget deliberately. Strong internal linking pushes Googlebot toward your most valuable pages and away from thin or duplicate content. Now, a critical point on structured data. JSON-LD must be syntactically perfect. A single trailing comma, a smart quote pasted from a word processor, or a missing brace will make the entire block unparseable. Google's parser simply stops. Validate every block with the Rich Results Test before publishing. Finally, choose a rendering strategy that works for all crawlers. Server-side rendering or static site generation should be your foundation. This ensures Googlebot, GPTBot, and ClaudeBot all see complete content immediately. Reserve client-side JavaScript for interactivity only. Never use it to deliver critical content or structured data. With these fundamentals in place, we can turn to the next slide: Diagnosing and Recovering from Ranking Drops.Structuring Content for Modern Indexabilitysupport.google.comsupport.google.comsupport.google.com+22 min
  15. 15Diagnosing and Recovering from Ranking DropsNow let's walk through a systematic approach to diagnosing and recovering from ranking drops. When you see a traffic dip, the first step is to cross-reference the timeline with known algorithm updates and any recent changes you made to the site. This helps you separate an external shift from a self-inflicted technical regression. Next, determine whether the problem is site-wide or page-level. A sharp drop on a specific template often points to a technical issue, while a gradual decline across the domain suggests a broader quality signal. Recovery requires action at both levels. Prune thin content that adds no original value. Improve engagement by demonstrating real, first-hand expertise. Remember that the Helpful Content System evaluates sites holistically. A large volume of weak pages can suppress the rankings of your genuinely strong articles. This is not a quick fix. Google's documentation is clear that recovery requires sustained, site-wide quality improvement over months. Patience and a structured audit are your most effective tools. Let's move on to the final slide, where we'll consolidate the key takeaways and build an action plan for technical creators.Diagnosing and Recovering from Ranking Drops2 min
  16. 16Key Takeaways and Action Plan for Technical CreatorsLet's pull these lessons together for your daily work. First, crawl budget is the gatekeeper. Server health and clean URL inventory determine whether your pages even get discovered. Second, indexability and quality signals, especially E E A T, determine eligibility. But the differentiator today is Information Gain. Your content must teach something the current top results do not. Third, verify that your critical content is server-rendered or statically generated. Client-side JavaScript should only add interactivity, not hold your entire content payload hostage. Finally, optimize for entity understanding, not just keywords. You want Google's models to clearly identify your topics, resolve them in the Knowledge Graph, and see your content as the authoritative source. This is the modern technical creator's mandate. Thank you for joining this walkthrough. Now go build what the search engines are learning to reward.Key Takeaways and Action Plan for Technical Creatorsdevelopers.google.comindexcraft.inincremys.com+22 min

Sources consulted

Web sources consulted while building this course.

How Search Engines Work