1. Warcbase: Scaling ‘Out’, ‘In’, and
‘Down’ HBase for Web Archiving
Jimmy Lin
University of Maryland
@lintool
Ian Milligan
University of Waterloo
@ianmilligan1
HBaseCon2015
Thursday, May 7, 2015

2. Source: Wikipedia (All Souls College, Oxford)
From the Ivory Tower…

3. Source: Wikipedia (Factory)
… to building sh*t that works

4. Gupta et al.WTF:The Who to Follow Service at Twitter.WWW 2013
Lin and Kolcz. Large-Scale Machine Learning at Twitter. SIGMOD 2012
Busch et al. Earlybird: Real-Time
Search at Twitter. ICDE 2012
Mishne et al. Fast Data in the Era of Big Data:Twitter's Real-
Time Related Query Suggestion Architecture. SIGMOD 2013.
Leibert et al. Automatic Management of Partitioned,
Replicated Search Services. SoCC 2011
I worked on…
– data products to surface relevant content to users
– analytics infrastructure to support data science

5. Source: https://www.flickr.com/photos/bongtongol/3491316758/

6. circa ~2010
~150 people total
~60 Hadoop nodes
~6 people use analytics stack daily
circa ~2012
~1400 people total
10s of Ks of Hadoop nodes, multiple DCs
10s of PBs total Hadoop DW capacity
~100 TB ingest daily
dozens of teams use Hadoop daily
10s of Ks of Hadoop jobs daily

7. Source: Wikipedia (All Souls College, Oxford)
And back!

8. “That sucks.”
Big Data for Good
Source: http://www.flickr.com/photos/yoononn/3300080563/
“The best minds of my generation are thinking about how
to make people click ads.”
– Jeff Hammerbacher

9. Source: http://images3.nick.com/nick-assets/shows/images/house-of-anubis/flipbooks/hidden-room/hidden-room-04.jpg
Big Data for Good
Cultural Heritage Preservation

11. Source: Wikipedia (Opposition to United States involvement in the Vietnam War)
When does an event become “history”?

12. When does an event become “history”?
~20-30 years later
The history of the 1960s were written in the 1980s!

16. This scandal was brought to you by the digital revolution.
That meant we could access all the information we wanted,
when we wanted it, anytime, anywhere, and when the story
broke in January 1998, it broke online. It was the first time
the traditional news was usurped by the Internet for a major
news story, a click that reverberated around the world…
… it was before social media, but people could still comment
online, email stories, and, of course, email cruel jokes. News
sources plastered photos of me all over to sell newspapers,
banner ads online, and to keep people tuned to the TV.
Monica Lewinsky – The Price of Shame
TED Talk, March 2015
“
”
So, where are those web pages now?

17. When does an event become “history”?
~20-30 years later
Historians are getting ready to write about the 90s…
Right about now!

18. Can you write a history of the
1990s without the web?
So, where are those web pages now?

20. Oct 25, 2001
… to look back over any
of the 10 billion web pages
archived… since 1996 …
Internet Archive’s Archive-It Service:WAaaS

22. And beyond…
70+ web archiving efforts worldwide!

23. (fortunately or unfortunately)
… don’t really care

24. What would a web archiving platform built on
modern big data infrastructure look like?

25. Wayback machine: monolithicTomcat application
Scalable storage of archived data
Efficient random access
Scalable processing and analytics
Scalable storage and access of derived data
Desiderata
Some work by the Internet Archive,
Common Crawl, and others…
Petabox by Internet Archive;
NAS, SAN, etc. by others
Ad hoc storage in
flat text WAT files

26. Scalable storage of archived data
Efficient random access
Scalable processing and analytics
Scalable storage and access of derived data
Desiderata
HDFS
HBase
Hadoop
HBase

27. Warcbase
An open-source platform for managing web
archives built on and H and
http://warcbase.org/
Source: Wikipedia (Archive)
WARC = Web ARChive (file format)

28. Bigtable use case: storing web crawls!
Row key: domain-reversed URL
Column family: contents
Raw data
Value: raw HTML
Column qualifier:
Column family: anchor
Derived data
Value: anchor text
Column qualifier: source URL

29. Warcbase: HBase data model
Row key: domain-reversed URL
Column family: c
Value: raw source
Column qualifier: MIME type
Why this?
“Fat Row”
Timestamp: crawl date
Saves a query…
But you need to do a scan…
In most cases, you need to do
a short scan anyway….

30. WARC data
Ingestion
Applications
and Services
Processing Analytics
text analysis, link analysis, …

31. Scalability?
We got 99 problems but scalability ain’t one…
– Jay-Z
Scalability of Warcbase limited by Hadoop/HBase
Applications are lightweight clients

32. WARC data
Ingestion
Applications
and Services
Processing Analytics
text analysis, link analysis, …

33. Sample dataset: crawl of the 108th U.S. Congress
Monthly snapshots, January 2003 to January 2005
1.15 TB gzipped ARC files
29 million captures of 7.8 million unique URLs
23.8 million captures are HTML files
Hadoop/HBase cluster:
16 nodes, dual quad-core processors, 3 × 2TB disks each

34. Temporal Browsing
View a particular version of a captured web page
Follow links to contemporaneous pages
Internet Archive’s Wayback machine
OpenWayback

35. OpenWayBack + Warcbase Integration
Implementation:
OpenWayback frontend for rendering
Offloads storage management to HBase
Transparently scales out with HBase

38. Topic Model Explorer
Implementation
LDA on each temporal slice
Adaptation of Termite visualization

39. Topic 0
Topic 1Topic 1
Topic 2
Topic 3
Topic 4
Topic 5Topic 5
Topic 6
Topic 7
Topic 8
Topic 9
Topic 10
Topic 11
Topic 12
Topic 13
Topic 14
Topic 15
Topic 16
Topic 17
Topic 18
Topic 19
forces
development
community
local
state
school
children
international
foreign
nations
grants
training
schools
students
college
services
president
funding
states
year
united
federal
government
grant
program
support
programs
education
child
security
national
forces
development
community
local
state
school
children
international
foreign
nations
grants
training
schools
students
college
services
president
funding
states
year
united
federal
government
grant
program
support
programs
education
child
security
national

40. Webgraph Explorer
Implementation
Link extraction with Hadoop, site-level aggregation
Computation of standard graph statistics
d3 interactive visualization

44. Scale “out”: use lots of commodity machines
Scaling Options
Scale “up”: use a beefier machine
Scale “in”: use a single commodity machine

45. Warcbase: here.
Mac Pro (Late 2013)
3 GHz 8-Core Intel Xeon E5
64 GB RAM, 1TB SSD
OS X 10.10.2

46. Scale “In”: Experiments
Wide Web scrape (2011) – Internet Archive wide0002 crawl
Sample of 100 WARC files (~100 GB)
Warcbase running on the Mac Pro:
Ingestion in ~1 hour
Extraction of webgraph using Pig in ~55 minutes
Result: 17m links, .ca subset of 1.7m links
Visualization in Gephi…

47. Internet Archive
Wide Web Scrap
from 2011

49. But they can probably afford a Mac Pro
What’s the big deal?
Historians probably can’t afford Hadoop clusters…
How will this change historical scholarship?
Visual graph analysis on longitudinal data, select subsets
for further textual analysis
… all on your desktop!
Drill down to examine individual pages

50. Scale “out”: use lots of commodity machines
Scaling Options
Scale “up”: use a beefier machine
Scale “down”: use a wimpy machine
Scale “in”: use a single commodity machine

51. Warcbase: here.
Raspberry Pi B+ (2012)
Broadcom BCM2835 SoC
700 MHz ARM11 processor
512 MB RAM
up to 128 GB storage (microSD)
Raspbian Linux

52. Scale “Down”: Experiments
Columbia University’s Human Rights Web Archive
43 GB of crawls from 2008 (1.68 million records)
Warcbase running on the Raspberry Pi (standalone mode)
Ingestion in ~27 hours (17.3 records/second)
Random browsing (avg over 100 pages): 2.1s page load
Same pages in Internet Archive: 2.9s page load
Draws 2.4 Watts
Jimmy Lin. Scaling Down Distributed Infrastructure on Wimpy Machines
for Personal Web Archiving.Temporal Web Analytics Workshop 2015.

53. Throw in search, lightweight analytics, …
What’s the big deal?
Store every page you’ve ever visited in your pocket!
What will you do with the web in your pocket?
How will this change how you interact with the web?
When was the last time you searched to refind?

54. Source: http://images3.nick.com/nick-assets/shows/images/house-of-anubis/flipbooks/hidden-room/hidden-room-04.jpg
Questions?
In conclusion…
Interesting big data problem
Underexplored
“Warm and fuzzy”