The document describes building a searchable index of URLs from Common Crawl datasets hosted cheaply on Amazon S3. It discusses the goals of building an index that can return results in seconds, be hosted cheaply at scale, and be accessible from any programming language. It then outlines the process used to extract the URLs and locations from archives, construct the index as a B-prefix tree stored in blocks in S3, and make it available for querying.

1. Indexing the URLs in
Providing Random Access by Domain
and Country
by Scott Robertson
Thursday, January 31, 13

2. Using Amazon S3 to Host
Cheap Searchable Datasets
^
ssive
Ma
Thursday, January 31, 13

3. We will Cover
‣ Goals
‣ Structure
‣ Construction
‣ Questions
Thursday, January 31, 13

4. About Me: Scott Robertson
‣ Product Guy
‣ Entrepreneur
‣ Bit Twiddler
Thursday, January 31, 13

5. Moving to a cluster is Hard
May
ftp
http
???
ens ive
Exp
&
mp lex
Co
Thursday, January 31, 13

6. Git push your script
ftp
http
May
scheduler dependency
manager
auto scaled cluster
adaptors
we handle the rest
Thursday, January 31, 13

7. Shameless Plug
We’re looking for
interesting big data
work!
Thursday, January 31, 13

8. 2 Million Archives
5 Billion Web Pages ~100 MB
Thursday, January 31, 13

9. Question
“How do I get all the pages for a
domain or country out of Common
Crawl?”
Thursday, January 31, 13

10. Answer
“Download all the archives and
unzip them”
That takes hours!
Thursday, January 31, 13

11. Goals
We Need an Index Which:
• Return results in seconds
• We can host cheaply
• Highly Available
• Accessible in any program language
Thursday, January 31, 13

12. Stats
‣ 5 Billion Pages
‣ Average URL Length: 66 bytes
‣ Location Pointer: 28 Bytes
‣ 5B x 66 x 28 = One Huge Index!
~ 437 GB
Thursday, January 31, 13

13. AmazonS3
1 prefix 1 2 ...
+ 3 prefix 1 4 ... 5 prefix N 6 ...
er s
item 1 item 2 item 3 item 4 item 5 item 6
lun te
Vo Suckers
+ X
construction ~$400
=monthly hosting
~$50
Thursday, January 31, 13

14. Price Comparison
Approximate Monthly Cost
Amazon Dynamo $437
Amazon RDS * $663
Roll your own** $622
* 4 SQL servers 1 master with 3 read replicas
** 4 $2,000 servers amortized over 36 months plus hosting costs
Thursday, January 31, 13

15. S3 Features
1. Byte range request
2. Multipart uploads
Thursday, January 31, 13

16. Byte Range Requests = Random Access
437GB
...
GET /url_index HTTP/1.1
...
Thursday, January 31, 13

17. Byte Range Requests = Random Access
437GB
...
GET /url_index HTTP/1.1
...
Range: 0-65536
Thursday, January 31, 13

18. Byte Range Requests = Random Access
437GB
...
GET /url_index HTTP/1.1
...
Range: 1310720-1376256
Thursday, January 31, 13

19. Byte Range Requests = Random Access
437GB
...
GET /url_index HTTP/1.1
...
Range: 8519680 - 8585216
Thursday, January 31, 13

20. Byte Range Requests = Random Access
437GB
...
GET /url_index HTTP/1.1
...
Thursday, January 31, 13

21. Multi Part Uploads = Distributed Builds
...
Thursday, January 31, 13

22. Index Structure
index blocks
header block 1 block 2 block 3 ... block n
data blocks
Series of 65,536 bytes blocks
except for the header
Thursday, January 31, 13

23. It’s a B-prefix Tree!
1 prefix 1 2 ...
3 prefix 1 4 ... 5 prefix N 6 ...
item 1 item 2 item 3 item 4 item 5 item 6
Thursday, January 31, 13

24. It’s a B-prefix Tree!
1 prefix 1 2 ...
3 prefix 1 4 ... 5 prefix N 6 ...
item 1 item 2 item 3 item 4 item 5 item 6
Thursday, January 31, 13

25. It’s a B-prefix Tree!
1 prefix 1 2 ...
3 prefix 1 4 ... 5 prefix N 6 ...
item 1 item 2 item 3 item 4 item 5 item 6
Thursday, January 31, 13

26. It’s a B-prefix Tree!
1 prefix 1 2 ...
3 prefix 1 4 ... 5 prefix N 6 ...
item 1 item 2 item 3 item 4 item 5 item 6
Thursday, January 31, 13

27. It’s a B-prefix Tree!
1 prefix 1 2 ...
3 prefix 1 4 ... 5 prefix N 6 ...
item 1 item 2 item 3 item 4 item 5 item 6
Thursday, January 31, 13

28. It’s a B-prefix Tree!
1 prefix 1 2 ...
3 prefix 1 4 ... 5 prefix N 6 ...
item 1 item 2 item 3 item 4 item 5 item 6
Thursday, January 31, 13

29. Reversed Host URLs
http://www.lolcats.com/hairless/300.html
url stored as
com.lolcats.blog/i-shaved-my-cat-today.html:http
com.lolcats.www/hairless/300.html:http
Thursday, January 31, 13

30. The Header
block size block count
header
Thursday, January 31, 13

31. Calculate offsets
(Block Size x Block Number)
+ Header Size
Thursday, January 31, 13

32. Index Block
4 byte Block Number
1 prefix 1 0 2 prefix 2 0 3 prefix 3 0 4
...
index block
prefix null terminator
Thursday, January 31, 13

33. What’s a Prefix?
the smallest part of a url that
distinguishes one sorted url
from another
Thursday, January 31, 13

34. Example Prefix
sorted urls
com.example/baz/page1
com.example/foo/page3
com.example/foo/page4 “com.l”
com.lolcats/fuzzy/
com.lolcats/hairless/300.html
Thursday, January 31, 13

35. Searching an index block
1 prefix 1 0 2 prefix 2 0 3 prefix 3 0 4
...
index block
while target > prefix
...
target_block_number = last_block_number
Thursday, January 31, 13

36. Data Block
url 1 0 segment date partition file offset url 2
data block
item
(url and arc file location)
Thursday, January 31, 13

37. Data Block
variable number of characters
url 1 0 segment date partition file offset url 2
data block
null terminator 28 byte location pointer
Thursday, January 31, 13

38. Location Pointer
s3://aws-publicdatasets/common-crawl/parse-output/segment/
1341690147253/1341711067356_4321.arc.gz
➥
segment date part
Thursday, January 31, 13

39. Construction
Thursday, January 31, 13

40. Step 1: Extract URL Locations
arcfile url and location output
D MAP
a-f
RE
A MAP
DU g-m
T MAP
CE n-z
A MAP
Sort and stored in 1,000 files
Thursday, January 31, 13

41. Step 2: Construct Index
e url and location prefixes index
a-f MAP
RE
MAP
DU g-m RE
MAP
CE n-z
Thursday, January 31, 13

42. Step 3: Upload to S3
AmazonS3
a-f
MAP
g-m MAP
MAP
n-z
MAP
index
Thursday, January 31, 13

43. What’s Next?
‣ Release “Real Soon Now” TM
‣ Post Code to Git Hub
‣ Build a Query Site/API
Thursday, January 31, 13

44. Questions?
Email me -> scott@triv.io
Thursday, January 31, 13