Ian Milligan's work focuses on using computational methods to analyze web archives, particularly a case study of the Canadian web. The study involves over 622,000 distinct URLs and utilizes various tools and formats, including WARC files, for indexing and text analysis. The document emphasizes the importance of historians adapting to digital resources to enhance their research capabilities.

1. Ian Milligan (@ianmilligan1)
Assistant Professor of History
i2millig@uwaterloo.ca
Clustering Search
to Navigate A Case
Study of the
Canadian World
Wide Web as a
Historical Resource

2. Why?
!
Historians need to think
about Computational
Methods in an era of
web archives.

3. INTERNET ARCHIVE
~ 10,240 TBs
LIBRARY of CONGRESS
~ 200 TBs
est. HOLDINGS:

4. The 80TB Wide Web
Scrape
[March - December 2011]

5. Wayback
Machine
or
WARC files?

6. Building a .ca sample:
!
622,365 distinct URLs /
8,512,275 overall URLs =
7.31% in case study

7. WARC
Web ARChive file format
ISO 28500:2009

8. filesdump.py available at https://github.com/ianmilligan1/
Historian-WARC-1/tree/master/WARC/warc-tools-mandel
WARC File
WARC-Tools/Lynx!
(warcfilter.py,
warchtmlindex.py
and filesdump.py)
Indexing
CDX Files !
(finding aids)

9. Full Text Index
Clustering
Workbench
Other sorts
of text
analysis

14. https://github.com/ianmilligan1/Historian-WARC-1/tree/
master/WARC/warc-tools-mandel
WARC File WARC-Tools/Lynx!
(warchtmlindex.py
and filesdump.py)
Indexing

15. Downside is you still
have to know what
you’re looking for.

17. Playing with
images?

18. Ian Milligan
Assistant Professor of History
i2millig@uwaterloo.ca
Thanks (to you all
and to funders).
!
http://
ianmilligan.ca/