Search Engines & Information Retrieval
How distributed web crawlers traverse hyperlinks, compile multi-terabyte inverted indexes, and retrieve relevant results across 50 billion pages in 150 milliseconds.
Pieces in this series
How Search Engines Actually Work
Web crawling, inverted indexing, PageRank link graphs, and neural vector retrieval across 50 billion pages
Search engines do not search the live web when you click search. They search a pre-built, compressed Inverted Index—a reverse dictionary mapping words to document postings—scored across 200+ signals including link graph authority (PageRank), term specificity (BM25), and transformer-based semantic embeddings.
Why Search Results Differ Between People
Geographic IP Anycast routing, session history vectors, freshness ranking, and A/B experiment flight allocation
Search engines do not maintain a single static leaderboard of the web. Search results diverge based on geographic IP resolution, implicit session context, device capabilities, language preferences, real-time freshness signals, and multi-armed bandit A/B experimentation buckets running across server clusters.