AltaVista Search Engine Architecture

AltaVista Indexing and Search Engine - By Mike Burrows - Recreated by Changshu Liu [ http:// csliu .com ]

Goals
• General purpose
• Good query performance
• Scale to hundreds of gigabytes
• Compact index/query representation
• Queries possible during updates
• Reasonable update performance

Non-Goals
• Scale beyond a terabyte
• Document parsing
• Query parsing
• Ranking for query results

Structure of Inverted Files
Chose flat inverted files that map words to lists of locations where those words occur
Words are null-terminated byte strings
Locations are 64-bit unsigned integers
Client picks what locations mean. No predefined notion of document, page or word number

Documents
A document is contiguous in location space
Documents do not overlap
Location space is allocated densely. The first document is at location 1
Word endDoc at last location of document
All document structure encoded with words
For example: begintitle , endtitle

Inverted File Format
Words ordered lexicographically
Each word followed by list of locations
Common word prefixes are compressed
Locations are encoded as deltas
Deltas stored in as few bytes as possible
2 bytes is common
Full-text index occupies about 30% text size. Word-in-document (non-positional) index is about 10%

Obvious format for deltas:
Continuation Bits Indicate Delta Boundaries
Key operation: Find first location at least X
Better format for efficient scanning:
Deltas packed into aligned 64-bit word
First byte contains continuation bits

Parsing a Delta
Observation:
Choose instructions to dual-issue well.
Fixed word structure allows prefetch.
Avoid branch mispredictions.
6 instr. to extract+sum+compare a delta
extql b, tp, x ; get next delta
addq tp, l, tp ; point to next delta
mskql x, l, x ; cut delta to length
srl l, 3, l ; get next delta length
addq cur, x, cur ; add delta to location
bge cur, done ; bail if done
With loop overhead, 35 instr/64-bit word.
10 cycles/64-bit word.

Index Stream Reader (ISRs)
An interface for reading the result of a query as an ascending sequence of locations
Lazily evaluated
ISRs are objects with methods:
Loc() – Return Current Location
Next() – Advance to next location
Seek(X) – Advance to first location at least X.
Subtype ISRP adds:
Prev() – Return previous location
Used for fielded queries (e.g. in title)
No methods move backwards

ISR Implementations
file — reads inverted files;
seek () method is the delta parsing loop
or — merges two or more ISRs
not — returns locations not in argument ISR
and — constraint solver ( AND , NEAR etc)
and other, specialized ISRs
and & not cannot support prev ()

ISR And—constraint solver
Arguments: list of ISR s , list of Constraint s
Constraint types: (A and B are ISRs)
1. loc ( A ) ≤ loc ( B ) + K
2. prev ( A ) ≤ loc ( B ) + K
3. loc ( A ) ≤ prev ( B ) + K
4. prev ( A ) ≤ prev ( B ) + K
If each word takes a location, constraints for two-word phrase “ a b ” are:
loc ( A ) < loc ( B )
loc ( B ) ≤ loc ( A ) + 1

Let E , BT , ET be ISRPs of words:
enddoc , begintitle , endtitle
Constraints for conjunction: a and b
prev ( E ) < loc ( A ) loc ( A ) ≤ loc ( E )
prev ( E ) < loc ( B ) loc ( B ) ≤ loc ( E )
Constraints for field query: title: A
prev ( BT ) < loc ( A ) loc ( A ) ≤ loc ( ET )
prev ( BT ) < loc ( ET ) loc ( ET ) ≤ loc ( BT )

Solver Algorithm
While (Unsatisfied Constraints)
Pick Unsatisfied Constraint()
Satisfy Constraint()
To Satisfy
loc ( A ) ≤ loc ( B ) + K:
seek ( B, loc ( A ) − K )
prev ( A ) ≤ loc ( B ) + K:
seek ( B, prev ( A ) − K )
loc ( A ) ≤ prev ( B ) + K:
seek ( B, loc ( A ) − K )
next ( B )
prev ( A ) ≤ prev ( B ) + K:
seek ( B, prev ( A ) − K )
next ( B )

Some Metrics
(performance based on AltaVista Web index)
20K lines of code
Indexes around 1.5GByte/Hr/600MHz CPU
Queries take about 100 cycles/query/MByte
Queries are CPU bound:
95% in user space, 5% in kernel
Memory bus is currently under-utilized

Breakdown of user CPU time
30% inner loop
15% constraint solver
15% higher level seek code
7% ranking code
0.2% merging results
Miss Ratios:
2% I-cache
8% D-cache
8% level-2 cache
40% level-3 cache

Postmortem
Successes
ISRs are a good abstraction
Flat location space
Representing structure as words
Regrets
No ability to run ISRs backwards
Wish ISR constraint solver were less complex

AltaVista Site Architecture - By Mike Burrows - Recreated by Changshu Liu [ http://xcybercloud.blogspot.com ]

Structure of the Site
Front-Ends: Alpha Workstations
Back-Ends:
4-10 CPU Alpha Servers
8GBytes RAM / 150 GBytes Disc.
Organized in Groups of 4-10 Machines
Each machine has 1/Nth of the whole index
Broad Routers 0 Broad Routers 1 FDDI Router Front End 0 Front End 1 Front End N-1 FDDI Router Front End N

Handling Failures
Disc: RAID controllers with spare discs
Back-ends: front-ends use other groups
Frond-ends: hot-spare grabs IP address
FDDI: manual replacement of cold spare
Site: failover via manual DNS change

RAID
Reconstructing a disc takes 30 minutes.
Disc performance is crippled
Except a few discs to fail a month
Need daily schedule for checking discs.
GUI annoying when checking 60 controllers
Once a disc failed with no error reported
Corrupted index file
On first day, the only non-RAID device (root disc) failed during demo for press

File System
Need a Journaling File System
Write Ahead Log
FSCK(consistency checker) takes ours
Software/Memory errors destroy file systems
Restoring 300GB from tape doesn’t work
Tape may be in error
Too slow
Important to replicate data in spinning disk

Back-Ends
Back-Ends were Digital 8400’s (Turoblaser)
Huge cards with large connectors
Pins are on backplane, not card
RAID setup took hours on separate machine
Console interrupt is a boon

Front-Ends
Biggest Problems:
Poorly-Tested software
Operator Error
Automatic restart dealt with former
A trivial IP failover scheme dealt with latter

HTTP Server
Original NCSA httpd was abysmal
Forked too often
Synchronous name resolution
Logs writes to full disc
Prone to denial of service attacks
Fixed with new first http server
Never Forks: aggravates software test issues
Submit limits: sockets/threads/requests rate

Load Balance
Front-End
DNS round robin
Backend
Front-Ends will group similar queries to the same specific backend for cache

Overload Handling
Back-ends take short-cuts when verloaded
Ultimately, they can refuse service
Front-ends have spare capacity to avoid site appearing completely dead

Reference
The AltaVista Indexing and Search Engine
Mike Burrows, Compaq SRC
Production Date: 01/18/2000
Link: http://uwtv.org/programs/displayevent.aspx?rid=2123

http://blog.csliu.com	24
http://csliu.com	10
http://feeds2.feedburner.com	2
url_unknown	1
http://feeds.feedburner.com	1

AltaVista Search Engine Architecture

by changshuliu on Jul 07, 2011

Accessibility

Categories

Tags

Upload Details

Usage Rights

5 Embeds 38

Statistics

AltaVista Search Engine Architecture — Presentation Transcript