How Search Engines Actually Work

Quick Answer

When you type three words into a search engine and hit Enter, the search engine does not search the live internet.

Searching billions of live websites on demand would require issuing billions of HTTP requests across undersea fiber cables, waiting for distant web servers to respond, and parsing gigabytes of HTML—taking hours or days per search.

Instead, a search engine operates in two completely separate time loops:

1. The Asynchronous Ingestion Loop (Months, Days, Minutes):

   • Crawling: Automated programs called crawlers (such as Googlebot) continuously traverse the web 24 hours a day, following billions of hyperlinks and downloading HTML pages into massive storage clusters.
   • Indexing: The engine shreds every downloaded webpage into individual words and compiles a gigantic reverse lookup table called an Inverted Index. Instead of storing "Document A contains words X, Y, Z", the index stores "Word X appears in Documents A, M, and Z".
   • Link Graphing (PageRank): The engine treats every hyperlink on the internet as a vote of confidence, computing mathematical authority scores for every known webpage.

2. The Real-Time Serving Loop (Under 150 Milliseconds):

   • Query Parsing: The engine expands synonyms, corrects typos, and extracts semantic intent from your search words.
   • Candidate Retrieval (Recall): The query is broadcast across thousands of memory-sharded server racks. Each shard consults its slice of the Inverted Index, using algorithms like BM25 to pull the top ~1,000 candidate documents in under 20 milliseconds.
   • Neural Re-Ranking (Precision): Machine learning rankers (such as RankBrain and transformer cross-encoders) evaluate these 1,000 candidates against hundreds of real-time signals—content relevance, authoritativeness, freshness, user location, and click probabilities.
   • Snippet Generation & Delivery: The top 10 results are packaged with highlighted summary snippets and returned to your screen before you can blink.

OFFLINE / CONTINUOUS (Petabyte Scale)
[ Web Crawlers ] ──► [ HTML Tokenizer ] ──► [ Inverted Index ] + [ PageRank Graph ]
                                                    │
════════════════════════════════════════════════════╪══════════════════════════════
ONLINE / REAL-TIME (< 150 Milliseconds)             │
[ User Query ] ──► [ Intent & Vector ] ──► [ Fast Recall (~1,000) ]
                                                    │
                                                    ▼
                                           [ Neural Re-Ranker ]
                                         (BERT / GBDT / 200+ Signals)
                                                    │
                                                    ▼
                                           [ Top 10 Ranked Results ]

---

Phase 1: Crawling the Planetary Web

Before an engine can search anything, it must discover that a webpage exists.

The web has no master registry of URLs. It is an open, decentralized graph where anyone can buy a domain and host pages. Search engines discover new and updated pages using autonomous distributed bots known as web crawlers or spiders.

   ┌────────────────────────────────────────────────────────┐
   │                    CRAWL FRONTIER                      │
   │   Priority Queue: Prioritized by PageRank, recency,    │
   │   and update frequency. Politeness rate-limited.       │
   └───────────────────────────┬────────────────────────────┘
                               │ Dispatches Target URL
                               ▼
   ┌────────────────────────────────────────────────────────┐
   │                  CRAWLER FETCH WORKER                  │
   │   1. Checks robots.txt (RFC 9309) cache                │
   │   2. Resolves IP via DNS (Anycast)                     │
   │   3. Establishes TCP/TLS & requests HTTP/2 GET         │
   │   4. Renders dynamic JavaScript via headless browser   │
   └───────────────────────────┬────────────────────────────┘
                               │
               ┌───────────────┴───────────────┐
               ▼                               ▼
     [ Raw HTML Document ]           [ Extracted Hyperlinks ]
               │                               │
               ▼                               ▼
    Piped to Indexing Pipeline      Pushed back to Crawl Frontier

1. The Crawl Frontier & Politeness

At any given moment, a crawler maintains a queue of billions of candidate URLs called the Crawl Frontier:

• Priority: Pages with high authority (such as major news sites or university domains) are recrawled every few minutes. Static personal blogs may only be visited once every few weeks.
• Politeness (Rate Limiting): If a crawler blasted a small blog with 5,000 requests per second, it would knock the web server offline (a denial-of-service). Crawlers enforce strict per-host rate limits, pausing between requests to the same domain.
• The Robots Exclusion Protocol (RFC 9309): Before crawling any website, the bot requests https://domain.com/robots.txt. If the webmaster specified Disallow: /private/, the crawler honors the directive and skips those URLs.

2. Rendering Dynamic JavaScript

In the early days of the web, crawlers only parsed static raw HTML strings. Today, millions of modern websites are Single Page Applications built on React, Next.js, or Vue that render blank HTML until JavaScript executes.

Modern crawlers run a two-pass architecture:

1. Immediate Pass: The crawler reads static raw HTML and indexes visible text immediately.
2. Render Queue: The page is placed into a headless browser rendering farm (using headless Chromium). The engine executes JavaScript, builds the full DOM, waits for network requests to resolve, and captures the dynamically rendered content.

3. Canonicalization & Duplicate Detection

The crawler extracts all <a href="..."> links from the page and feeds them back into the Crawl Frontier.

To prevent infinite loops (such as calendars linking to /next-month infinitely) and duplicate content (such as http://site.com, https://site.com, and https://site.com/index.html?ref=twitter), the engine computes cryptographic hashes (such as SimHash) of the text. If two URLs produce near-identical SimHash fingerprints, the engine clusters them and designates a single Canonical URL.

---

Phase 2: The Inverted Index — The Secret to Instant Search

Once HTML is downloaded, how can an engine scan 50 billion pages for three words in 0.02 seconds?

If you opened a 1,000-page textbook and wanted to find every mention of the word "photosynthesis", you would never read the book from page 1 to 1000. You would flip to the Index at the back of the book, locate "Photosynthesis", and read the list of page numbers: pages 42, 108, 215.

Search engines do the exact same thing on a planetary scale. This data structure is called an Inverted Index.

Constructing the Index

Consider three web documents:

• Doc 1: "UPI moves money between banks instantly."
• Doc 2: "Credit cards move money across payment networks."
• Doc 3: "Banks settle money using central bank reserves."

The indexing pipeline processes the text through four transformations:

[ Raw Sentence: "UPI moves money between banks instantly." ]
                      │
                      ▼
[ Tokenization ]: [ "upi", "moves", "money", "between", "banks", "instantly" ]
                      │
                      ▼
[ Stopword Filter & Normalization ]: [ "upi", "move", "money", "bank", "instant" ]
                      │
                      ▼
[ Inverted Postings Table ]:
  "bank"    ──► [ Doc 1 (pos 5), Doc 3 (pos 1, pos 5) ]
  "card"    ──► [ Doc 2 (pos 2) ]
  "money"   ──► [ Doc 1 (pos 3), Doc 2 (pos 4), Doc 3 (pos 3) ]
  "move"    ──► [ Doc 1 (pos 2), Doc 2 (pos 3) ]
  "upi"     ──► [ Doc 1 (pos 1) ]

The Postings List & Delta Encoding

In production, a search term like "money" appears on over 500 million web pages. Storing 500 million 64-bit document IDs would require 4 gigabytes of RAM for a single word.

Search engines compress these Postings Lists using Delta Encoding:

• Instead of storing raw document IDs: [100450, 100455, 100462, 100500]
• The engine sorts them and stores only the difference (delta) from the previous number: [100450, +5, +7, +38]
• Because these deltas are small integers, algorithms like Variable Byte Encoding (Varint) or SIMD-PForDelta pack them into tight bit arrays, compressing multi-terabyte indexes into server RAM.

When you search for "bank money", the engine does not inspect web pages. It simply fetches the postings list for "bank" and the postings list for "money", and performs a blazing-fast bitwise intersection of their memory pointers.

---

Phase 3: The Search Engine Architecture

The diagram below maps the complete end-to-end pipeline: from asynchronous web discovery to real-time query parsing and neural re-ranking:

---

Phase 4: Determining Relevance — BM25 Scoring

When 5 million web pages contain your search terms, how does the engine decide which ones are most relevant to the words?

Modern search engines rely on Okapi BM25 (Best Matching 25), a probabilistic relevance ranking algorithm formulated by Stephen Robertson and Karen Spärck Jones.

BM25 calculates a score based on three core components:

$$BM25(D, Q) = \sum_{i=1}^{N} IDF(q_i) \cdot \frac{TF(q_i, D) \cdot (k_1 + 1)}{TF(q_i, D) + k_1 \cdot \left(1 - b + b \cdot \frac{|D|}{\text{avgdl}}\right)}$$

1. Inverse Document Frequency (IDF)

How rare is the word?

• Common words like "the", "how", or "internet" appear on millions of pages and receive an IDF score close to zero.
• Rare, highly specific words like "photolithography" or "RTGS" appear on few pages and receive a massive IDF multiplier.

2. Term Frequency Saturation ($TF$)

If a webpage mentions the word "apple" once, it might be relevant. If it mentions it 5 times, it is much more likely to be relevant.

However, in crude early search algorithms, spammers discovered they could repeat the word "apple" 10,000 times in hidden white text on a white background to rank #1. BM25 prevents this with diminishing returns (saturation):

• Moving from 1 mention to 5 mentions increases the score substantially.
• Moving from 50 mentions to 100 mentions provides almost zero additional score benefit (controlled by parameter $k_1 \approx 1.2$).

3. Document Length Normalization ($|D| / \text{avgdl}$)

A 50,000-word book is naturally more likely to mention your search term than a concise 500-word article, simply by virtue of its length.

• BM25 penalizes excessively long documents that only mention the term in passing.
• It rewards concise documents where the search term represents a higher density of the overall content (controlled by parameter $b \approx 0.75$).

---

Phase 5: Determining Authority — The PageRank Algorithm

Relevance alone is not enough. If someone creates a spam website repeating accurate financial keywords, how does the search engine know whether to trust that page over the official Reserve Bank of India website?

In 1996, Stanford graduate students Larry Page and Sergey Brin developed PageRank.

The Web as a Directed Graph

PageRank made a profound observation: A hyperlink from Page A to Page B is an endorsement.

    ┌───────────────┐
    │  Wikipedia    │ ──────────────┐
    │  (Very High)  │               │
    └───────────────┘               ▼
                               ┌───────────────────┐
                               │ Target Authority  │
                               │ Page (e.g. RBI)   │
                               └───────────────────┘
    ┌───────────────┐               ▲
    │ The New York  │ ──────────────┘
    │ Times (High)  │
    └───────────────┘

However, PageRank is not a simple link-counting contest. A link from a low-quality personal blog is worth very little. A link from Wikipedia, the BBC, or Harvard University carries monumental authority.

The Random Surfer Model

Mathematically, PageRank models the behavior of an imaginary user browsing the web:

1. The user lands on a random webpage.
2. They click random hyperlinks on that page.
3. Occasionally (with probability $1 - d \approx 0.15$), they get bored, abandon the link chain, and jump to a completely random URL elsewhere on the web.

The PageRank equation defines the probability $PR(u)$ that the random surfer lands on page $u$:

$$PR(u) = \frac{1 - d}{N} + d \sum_{v \in B_u} \frac{PR(v)}{L(v)}$$

Where:

• $B_u$ is the set of all pages linking to page $u$.
• $PR(v)$ is the PageRank authority of the linking page.
• $L(v)$ is the total number of outbound links on page $v$ (if a page links to 100 different sites, its endorsement power is divided by 100).
• $d$ is the damping factor (usually set to $0.85$).

Because the PageRank of every page depends on the PageRank of the pages linking to it, you cannot solve this with simple arithmetic. The engine uses power iteration: it initializes every page with an equal score, multiplies the entire billion-node matrix repeatedly, and watches the numbers converge until the mathematical vector stabilizes into global authority scores.

---

Phase 6: Modern Neural Search — Beyond Keywords

For the first twenty years of the web, search engines matched text: if your query was "curb weight of swift", the engine looked for pages containing the exact string "curb weight" and "swift".

If you searched "what is the name of that movie where dreams take place inside other dreams?", early keyword matching failed because the movie page for Inception might never contain the phrase "dreams take place inside other dreams".

Modern search engines solve this with Deep Learning and Dense Vector Embeddings (as explored in our architectural guide on How Large Language Models Generate Text).

Query: "medication for severe headache"
                    │
                    ▼ (Bi-Encoder Neural Model)
    Vector: [ 0.231, -0.842, 0.512, ... 768 dimensions ]
                    │
                    ▼ (Cosine Similarity in Vector Space)
Document: "Clinical efficacy of sumatriptan in acute migraine treatment"
    Vector: [ 0.228, -0.839, 0.518, ... 768 dimensions ]
                    │
                    ▼
MATCH: High Semantic Proximity (Even with Zero Shared Words!)

1. Bi-Encoders (Dense Retrieval)

Using models based on transformer architectures (such as Google's BERT or ColBERT), text is projected into a high-dimensional mathematical space. Documents discussing "acute migraine" and queries asking about "severe headache" land in the exact same vector cluster, allowing the search engine to retrieve semantically identical pages even if they share zero vocabulary.

2. The Two-Stage Ranking Funnel

Why don't search engines run heavy neural models across all 50 billion pages for every query?

Running a deep transformer across 50 billion pages would require thousands of GPU-seconds per search—costing millions of dollars per minute and taking seconds to respond.

Search engines solve this through a two-stage funnel:

1. Stage 1: High-Recall Candidate Retrieval (Fast & Coarse):
   • Scans the memory-sharded Inverted Index using BM25 and lightweight approximate nearest-neighbor vector search (HNSW).
   • Filters 50,000,000,000 pages down to the top 1,000 candidate documents in under 20 milliseconds.
2. Stage 2: High-Precision Neural Re-Ranking (Slow & Deep):
   • The 1,000 candidates are passed to heavy machine learning rankers (Cross-Encoders and Gradient Boosted Decision Trees).
   • These models examine rich cross-attention between every query word and document sentence, evaluating over 200 signals: content depth, author expertise (E-E-A-T), click-through rates, page load speed, mobile usability, and geographic intent.
   • The model selects and orders the definitive Top 10 search results.

---

What Happens in Those 150 Milliseconds

Here is the exact timeline of a single search query, from keystroke to rendered SERP (Search Engine Results Page):

Time Window	Component	Mechanical Action
0 – 15 ms	Edge Anycast Gateway	TLS termination, IP geolocation lookup, and request routing to regional datacenter.
15 – 30 ms	Query Engine	Spellcheck, entity recognition, synonym expansion, and embedding generation.
30 – 55 ms	Index Leaf Clusters	Thousands of index shards scan postings lists in parallel RAM; BM25 scores top 1,000 candidate docs.
55 – 90 ms	Neural Re-ranker	Transformer cross-encoders score 1,000 candidates across 200+ signals, outputting top 10 ranked IDs.
90 – 115 ms	Document Store	Forward index fetches page titles, canonical URLs, and dynamic snippet extracts.
115 – 140 ms	Result Aggregator	Injects knowledge graph panels, local maps, image carousels, and ads; serializes HTML payload.
140 – 150 ms	Browser Delivery	Response packets stream back over HTTP/2 across fiber backbones to render on your screen.

---

Why This Architecture Matters

The architecture of a modern search engine represents one of the most sophisticated engineering achievements in human history.

It balances an impossible trade-off: planetary scale vs. sub-second latency.

By severing ingestion from serving—shredding the web asynchronously into an inverted index and evaluating candidates through a multi-stage funnel—search engines turn the chaotic, unorganized explosion of human knowledge across 50 billion pages into an organized, instant answer box.

---

To understand why two different people typing the exact same query can see entirely different search rankings, explore our companion guide on Why Search Results Differ Between People. To discover how network packets physically navigate transoceanic fiber cables to reach search engine datacenters, read How the Internet Actually Works or trace the browser rendering loop in What Happens When You Type a Website Address.