This document provides an overview of existing web archives and their use for research. It discusses several major web archives including the Internet Archive, Common Crawl, Pandora Archive, and national archives. For each, it describes their size and collection strategies, as well as positives and negatives for research use. The talk concludes with examples of how existing web archives in Australia are being used for research.

1. Internet Content as
Research Data
Australian National University
August 2012, Canberra
Monica Omodei

2. Research Examples
• Social networking • Political Science
• Lexicography • Media Studies
• Linguistics • Contemporary history
• Network Science
Data-driven science is migrating from the
natural sciences to humanities and social
science

3. Talk
Structure
• Exis0ng
web
archives
• Web
archive
use
cases
• Bringing
archives
together
• Crea0ng
your
own
archive
• It’s
ge>ng
harder
–
challenges
• Web
data
mining
&
analysis

4. Exis0ng
web
archives
• Internet
Archive
• Common
Crawl
• Pandora
Archive
• Internet
Memory
Founda0on
Archive
• Other
na0onal
archives
• Research,
University
Library
archives

5. Common
Collec0on
Strategies
• Crawl
Scope
&
Focus
1) Thema0c/Topical
(elec0ons,
events,
global
warming…)
2) Resource-­‐specific
(video,
pdf,
etc.)
3) Broad
survey
(domain
wide
for
.com/.net/.org/.edu/.gov)
4) Exhaus0ve
(end
of
life, closure crawls, natl domains)
5) Frequency-­‐Based
• Key
Inputs:
nomina0ons
from
subject
ma^er
experts,
prior
crawl
data,
registry
data,
trusted
directories,
wikipedia,
twi^er

6. Internet Archive’s Web Archive
Positives
– Very broad – 175+ billion web instances
– Historic – started 1996
– Publicly accessible
– Time-based URL search
– API access
– Not constrained by legislation – covered by
fair use and fast take-down response

7. Internet
Archive’s
Web
Archive
Negatives
– Because of size can’t search by keyword
– Because of size crawling is fully automated –
ergo QA is not possible

11. Common
Crawl
• Non-­‐profit
founda0on
building
an
open
crawl
of
the
web
to
seed
research
and
innova0on
• Currently
5
billion
pages
• Stored
on
Amazon’s
S3
• Accessible
via
MapReduce
processing
in
Amazon’s
EC2
compute
cloud
• Wholesale
extrac0on,
transforma0on,
and
analysis
of
web
data
cheap
and
easy

12. Common
Crawl
Nega0ves
• Not
designed
for
human
browsing
but
for
machine
access
• Objec0ve
is
to
support
large-­‐scale
analysis
and
text
mining/indexing
–
not
long-­‐term
preserva0on
• Some
costs
are
involved
for
direct
extrac0on
of
data
from
S3
storage
using
Requester-­‐Pays
API

13. Pandora
Archive
• Posi0ves
– Quality
checked
– Targeted
Australian
content
with
selec0on
policy
– Historical
–
started
1996
– Bibliocentric
approach
–web
sites/publica0ons
selected
for
archiving
are
catalogued
(see
Trove)
– Keyword
search
– Publicly
accessible
– You
can
nominate
Australian
web
sites
for
inclusion
-­‐
pandora.nla.gov.au/
registra0on_form.html

15. Pandora
Archive
• Nega0ves
– labour
intensive
thus
quite
small
– significant
content
missed
because
permission
to
copy
refused
• Situa0on
will
improve
markedly
if
Legal
Deposit
provisions
extended
to
digital
publica0ons
• Broader
coverage
will
be
achieved
when
infrastructure
is
upgraded
hence
reducing
labour
costs
for
checking/fixing
crawls

16. Pandora
Archive
Stats
• Size
–
6.32
TB
• Number
of
Files
>
140
million
• Number
of
‘0tles’
>
30.5K
• Number
of
0tle
instances
>
73.5K

21. Which archived sites are popular ?
• Measure: filtered, aggregated web access
log data which counts access to titles "
• Examined top 30 archived titles (# of
accesses) for each year 2009 to 2012"
• Selected some to examine and speculate
as to why they might be popular"
• Selected those with consistently high
ranking, and ones that were very variable
between years

22. Reasons for popularity of archived
version
• Were once popular and are now
decommissioned, particularly if domain
name continues to exist and redirects to
the archive"
• May not be that popular as live sites but
their live site links prominently to Pandora
as an archive for their content"
• Popular referencing sources cite the
archive as well as the live site (if it still
exists)

26. Improving visibility and usage of
Pandora archive
• Articles about interesting content on the
Australia Web Archives blog –http://
blogs.nla.gov.au/australias-web-archives/"
• More effort to identify archived sites that are
no longer ʻliveʼ"
• Market automatic redirect services to web
site owners/managers"
• Allow Google to index archive content for
ʻnon-liveʼ sites (problematic)"
• Install Twittervane - draws
site
nomina0ons
for
archiving
based
on
trending
Twi^er
topics.
"

27. .au
Domain
Annual
Snapshots
• Annual
crawls
since
2005
commissioned
from
Internet
Archive
• Includes
sites
on
servers
located
in
Australia
as
well
as
.au
domain
• Robots.txt
respected
except
for
inline
images
and
stylesheets
• No
public
access
–
researcher
access
protocols
are
being
developed
• Full
text
search
–
suited
to
searching
archives
• Separate
.gov
crawl
publicly
accessible
soon

28. Australian
web
domain
crawls
Year
2005
2006
2007
2008
2009
2011
Files
185
596
516
1
billion
765
660
million
million
million
million
million
Hosts
811,523
1,046,038
1,247,614
3,038,658
1,074,645
1,346,549
crawled
Size
(TBs)
6.69
19.04
18.47
34.55
24.29
30.71

29. Internet
Memory
Founda0on
• Number
of
European
partners
• LiWA
–
Living
Web
Archives:
next
genera0on
Web
archiving
methods
and
tools
• LAWA
–
Longitudinal
Analy0cs
of
Web
Archive
Data:
experimental
testbed
for
large-­‐scale
data
analy0cs
• ARCOMEM
(Collect-­‐All
ARchives
to
COmmunity
MEMories)
leveraging
social
media
for
Intelligent
Preserva0on
• SCAPE
–
Scalable
Preserva0on
Environments

31. Other
Na0onal
Archives
• List
of
Interna0onal
Internet
Preserva0on
Consor0um
member
archives
–
netpreserve.org/about/archiveList.php
• Some
are
whole
domain
archives,
some
are
selec0ve
archives,
many
are
both
• Some
have
public
access,
others
you
will
need
to
nego0ate
access
for
research
• Most
archives
have
been
collected
using
the
heritrix
open-­‐source
crawler
and
thus
use
the
standard
format
(warc
ISO
format)

32. Research
Archives
• California
Digital
Library
• Harvard
University
Libraries
• Columbia
University
Libraries
• University
of
North
Texas
….
and
many
more
• WebCITE
-­‐
webcita0on.org
(cita0on
service
archive)

33. Example:
Columbia
University
• Member
of
the
IIPC
• They
use
the
ArchiveIt
service
• A
Research
library
that
sees
web
archiving
as
fundamental
to
their
collec0ng
• They
complement
and
coordinate
with
other
web
archives
• Their
collec0ng
focus
is
thema0c
–
eg
human
rights,
historic
preserva0on,
NY
religious
ins0tu0ons
• They
also
archive
web
content
as
part
of
personal
and
organisa0onal
archives
(c.f.
manuscripts
coll)
• Archive
their
own
web
site
regularly

35. Bringing
Archives
Together
• Common
standards
and
APIs
• Memento
project
–
adding
0me
to
the
web
– Aggregates
CDX
files
(URL
index)
from
mul0ple
archives
– Has
a
Firefox
plug-­‐in
which
allows
0me-­‐based
browsing
– Ini0a0ve
of
Los
Alamos
Laboratories
– See
h^p://www.mementoweb.org/demo/

37. Common
Use
Cases
for
a
web
archive
• Content
discovery
• Nostalgia
queries
• Web
site
restora0on
and
file
recovery
• Domain
name
valua0on
• Fall-­‐back
for
link-­‐rot
• Prior
art
analysis
and
patent/copyright
infringement
research
• Legal
cases
• Topic
analysis,
web
trends
analysis,
popularity
analysis,
network
analysis,
linguis0c
analysis

38. Create
your
own
Archive
• Use
a
subscrip0on
service
• Build
your
own
web
archiving
infrastructure
with
open
source
sonware
(
ie
Heritrix
and
Wayback)
• Use
web
cita0on
services
that
create
archive
copies
as
you
bookmark
pages

39. Subscrip0on
Services
• archive-­‐it.org
(service
operated
by
non-­‐profit
Internet
Archive
since
2006)
• archivethe.net
(service
operated
by
non-­‐profit
Internet
Memory
Founda0on)
• California
Digital
Library
Web
Archiving
Service
-­‐
cdlib.org/services/uc3/was.html
• OCLC
Harvester
Service
-­‐
oclc.org/
webharvester/overview/default.htm

41. Install
web
archiving
system
locally
• Easy-­‐to-­‐deploy
web
archiving
toolkit
not
yet
available
• Ins0tu0onal
web
archiving
infrastructure
is
feasible
and
has
been
established
at
a
number
of
universi0es
for
use
by
researchers
–
needs
IT
systems
engineers
to
set
up
though
• Archives
can
be
deposited
with
the
NLA
for
long-­‐term
preserva0on

42. Personal
Web
Archiving
• WARCreate
–
recently
released
free
tool
which
creates
wayback-­‐consumable
warc
files
from
any
web
page
• Google
Chrome
extension
• Enables
preserva0on
by
users
from
their
desktop
• Can
target
content
unreachable
by
crawlers
• Brings
WARC
to
personal
digital
archiving
• What
you
do
with
the
WARC
files
is
up
to
you
• Install
suite
provided
to
set
up
local
Wayback
instance
and
Memento
0megate

43. Current
challenges
• Database-­‐driven
features
and
func0ons
• Complex
and
varying
URI
formats
and
non-­‐
standard
link
implementa0ons
eg
Twi^er
• Dynamically
generated
ever-­‐changing
URIs
– For
serving
the
same
resources
• Rich
Media
–
eg
streamed
media
with
custom
apps
and
ant-­‐collec0on
measures
• Scripted
incremental
display
and
page-­‐loading

44. …
more…
• Scripted
HTML
forms
• Mul0-­‐sourced
embedded
material
• Dynamic
authen0ca0on
e.g.
captchas,
cross-­‐
site
authen0ca0on,
user-­‐sensi0ve
embeds
• Alternate
display
based
on
browser
or
device,
or
other
parameter
• Site
architecture
designed
to
inhibit
crawling
and
indexing
–
but
if
poorly
done
even
‘polite’
harvesters
like
Heritrix
may
crash
their
server

45. ..
but
wait,
there’s
more
…
• Server-­‐side
scripts
and
remote
procedure
calls
–
the
full
variety
of
paths
through
a
site
are
now
onen
hidden
in
remote/opaque
server-­‐
side
code
–
not
a
new
problem
but
now
effects
80+%
of
online
resources
• HTML
5
web
sockets
–
effec0vely
codifies
incremental
updates
without
page
reloads
• Mobile
publishing

46. Transac0onal
Web
Archiving
• Useful
for
ins0tu0onal
archiving
– Best
for
record-­‐keeping
purposes
-­‐
when
challenged
in
court
about
content
on
web
site
– Can
be
used
to
ensure
URL
persistence
eg
when
site
has
a
make-­‐over
–
can
intercept
404s
– No
‘gaps’
c.f.
crawl
approach
–
every
change
in
accessed
content
is
archived
– However
requires
code
snippet
to
be
installed
on
web
server
– Open
source
sonware
being
developed
by
Los
Alamos
Labs

47. Web Data Mining & Analysis –
What is it? Why Do It?
Innovation is increasingly driven from Large scale
Data Analysis
Need fast iteration to understand the right
questions to ask
More minds able to contribute = more value
(perceived and real) placed on the importance
of the data
Increased demand for/value of the data = more
funding to support it
Need to surface the Information amongst all
that data…

48. Platform & Toolkit: Overview
• Software
– Apache Hadoop
– Apache Pig
• Data/File format
– WARC
– CDX
– WAT (new!)

49. Apache Hadoop
• HDFS
– Distributed storage
– Durable, default 3x replication
– Scalable: Yahoo! 60+PB HDFS
• MapReduce
– Distributed computation
– You write Java functions
– Hadoop distributes work across cluster
– Tolerates failures

50. File formats and data: WARC

51. File formats and data: CDX
• Index used to browse WARC-based archive
• Space-delimited text file
• Only essential the essential metadata needed
by Wayback
– URL
– Content Digest
– Capture Timestamp
– Content-Type
– HTTP response code
– etc.

52. File formats and data: WAT
• Yet Another Metadata Format! ☺ ☹
• Not preservation format
• Data exchange and analysis
• Less than full WARC, more than CDX
• Essential metadata for many types of analysis
• Avoids barriers to data exchange: copyright,
privacy
• Work-in-progress: we want your feedback

53. File formats and data: WAT
• WAT is WARC ☺
– WAT records are WARC
metadata records
File formats & data:
– WARC-Refers-To header • CDX: 53 MB
identifies original WARC
record
• WAT: 443 MB
• WAT payload is JSON
• WARC: 8,651 MB
– Compact
– Hierarchical
– Supported by every
programming environ

54. Some
References
• h^p://en.wikipedia.org/wiki/Web_archiving
• h^p://netpreserve.org/about/archiveList.php
• Web
Archives:
The
Future(s)
-­‐
h^p://www.netpreserve.org/publica0ons/
2011_06_IIPC_WebArchives-­‐TheFutures.pdf
• h^p://matkelly.com/warcreate/
• Common
Crawl:
h^p://commoncrawl.org/
data/accessing-­‐the-­‐data/

55. Contacts
• Webarchive
@
nla.gov.au
• Secretariat
@
internetmemory.org
• Queries
about
the
internet
archive
web
archive
h^p://iawebarchiving.wordpress.com/
• Queries
about
Archive-­‐It
service
h^p://www.archive-­‐it.org/contact-­‐us
momodei
@
nla.gov.au
(un0l
31
Aug
2012
)
or
monica.omodei
@
gmail.com