Intro to Web Archiving

Intro to Web Archiving
Dr. Michele C. Weigle, @weiglemc
Web Sciences and Digital Libraries (WS-DL) Group, @WebSciDL
Department of Computer Science
Old Dominion University
June 26, 2018
ODU Machine Learning and Data Sciences Camp

@weiglemc, @WebSciDL
ODU WS-DL Group
• Web Sciences and Digital Libraries
– digital preservation
– web archiving
– web science (social media analysis, web usage analysis)
• Our recent work has been featured in the popular
press
June 26, 2018 2
@WebSciDL
http://ws-dl.cs.odu.edu/
http://ws-dl.blogspot.com/

ODU WS-DL Group
• Scott Ainsworth
• Sawood Alam
• Lulwah Alkwai
• Mohamed Aturban
• Brian Griffin
• Hussam Hallak
• Shawn Jones
• Mat Kelly
• Corren McCoy
• Louis Nguyen
• Alexander Nwala
June 26, 2018 3
PhD Students
• Nauman Siddique
• Miranda Smith
MS Students
Coming in Fall 2018!
• Dr. Sampath Jayarathna
• Dr. Jian Wu
• Dr. Michael L. Nelson
• Dr. Michele C. Weigle
Faculty
@WebSciDL
http://ws-dl.cs.odu.edu/
http://ws-dl.blogspot.com/

What is the past web?
June 26, 2018 4

The Web holds our stories
June 26, 2018 5

But webpages can disappear
• Average lifespan of a webpage: 50-100 days
• A year after publication, about 11% of
content shared on social media will be gone.
June 26, 2018
SalahEldeen and Nelson, "Losing My Revolution: How Many Resources Shared on Social Media Have Been Lost?", TPDL 2012
http://ws-dl.blogspot.com/2012/02/2012-02-11-losing-my-revolution-year.html
6

Maybe it's archived?
June 26, 2018 7
https://archive.org/web

Why archives matter
• Malaysia Airlines Flight
17 (MH17)
• Ukrainian separatists
originally took credit for
downing a transport
plane in that location
• Later deleted the post
• Internet Archive had
archived the post before
deletion
June 26, 2018 8
http://www.csmonitor.com/World/Europe/2014/0717/Web-
evidence-points-to-pro-Russia-rebels-in-downing-of-MH17-video

We can use archives to tell stories
June 26, 2018 9
similar to our Hurricane Katrina example: https://www.slideshare.net/phonedude/why-careaboutthepast
https://www.nytimes.com/2016/11/17/insider/in-13-
headlines-the-drama-of-election-night.html

If something's gone from the live
web, check a web archive
June 26, 2018 10

Web archives to the rescue!
June 26, 2018 11
https://twitter.com/brian3354/status/966081774194511874

Internet Archive's Wayback Machine
has gone mainstream
June 26, 2018 12
"God bless you Internet Archive"
- Rachel Maddow, Dec 12, 2016
Last Week Tonight, Mar 18, 2018
Jill Lepore, "The Cobweb", The New Yorker, Jan 26, 2015

But Wayback is not Google
• Wayback Machine has no full-text search
– too big to be indexed
– 654 billion web pages, 9 petabytes of data
– growing at 20 TB/week
• Enter URL and pick a date
June 26, 2018 13
"It’s more like a phone book than like an archive."
-Jill Lepore, The New Yorker

What do people think the Wayback
Machine is?
June 26, 2018 14
https://www.politico.com/story/2018/04/25/joy-reid-anti-gay-posts-550213

What do people think the Wayback
Machine is?
June 26, 2018 15
https://www.cnn.com/2018/02/16/politics/richard-pinedo-guilty-plea/index.html
https://www.politico.com/story/2018/04/25/joy-reid-anti-gay-posts-550213
https://web.archive.org/web/20180115103952/https:/auctionessistance.com/

Caches are not archives
June 26, 2018 16
http://ws-dl.blogspot.com/2018/01/2018-01-02-link-to-web-archives-not.html
http://www.wired.co.uk/article/russia-propaganda-online-blog-longform-medium-posts
https://webcache.googleusercontent.com/search?q=cache:qwqnGPqC2vsJ:https://medium.com/
%40TheFoundingSon/huffington-post-vs-whiteness-and-white-women-
1e67193085d4+&cd=15&hl=en&ct=clnk&gl=uk

Is it really that important to archive
instead of just taking a screenshot?
June 26, 2018 17
https://twitter.com/AngryBlackLady/status/990032514080108544
https://twitter.com/phonedude_mln/status/990070331737100288

We should be doing both
June 26, 2018 18
https://twitter.com/conspirator0/status/1000475042017366017

“If you see something, save
something”
June 26, 2018 19
https://blog.archive.org/2017/01/25/see-something-save-something/

There's more than just the Internet
Archive
June 26, 2018 20
http://timetravel.mementoweb.org/list/20020908180610/http://blog.reidreport.com/

TimeTravel
June 26, 2018 21
http://timetravel.mementoweb.org

Pro tip: submit pages to multiple
archives
June 26, 2018 22
https://twitter.com/phonedude_mln/status/998948823845261312

We've built tools to help people
submit webpages to multiple archives
• Mink – Google Chrome extension
• #icanhazmemento – Twitter bot
• ArchiveNow – Python module, Docker
container, local web service
June 26, 2018 23

Mink
June 26, 2018 24
“Archive What I See Now: Bringing Institutional Web Archiving Tools to the Individual Researcher”,
2014-2017, HK-50181-14
Mat Kelly, Michael L. Nelson and Michele C. Weigle, "Mink: Integrating the Live and Archived Web Viewing
Experience Using Web Browsers and Memento," JCDL 2014, poster.
http://ws-dl.blogspot.com/2014/10/2014-10-03-integrating-live-and.html
Google Chrome extension
Submit currently viewed
webpage to public
archives
https://github.com/machawk1/
Mink

#icanhazmemento
June 26, 2018 25
http://ws-dl.blogspot.com/2015/07/2015-07-22-i-can-haz-memento.html
Twitter bot
Include #icanhazmemento in a
tweet with a URL
Bot replies with a link to the
memento of the page closest to
the time of the tweet
If page not archived, bot submits
URL to multiple public archives,
replies with a link to the
memento in Time Travel
Alexander Nwala, "2015-07-22: I Can Haz Memento,"
https://github.com/anwala/icanhazmemento

ArchiveNow
June 26, 2018 26
Mohamed Aturban, Mat Kelly, Sawood Alam, John Berlin, Michael L. Nelson and Michele C. Weigle,
"ArchiveNow: Simplified, Extensible, Multi-Archive Preservation," JCDL 2018, poster.
http://ws-dl.blogspot.com/2017/02/2017-02-22-archive-now-archivenow.html
Python module, Docker
container
Submit URI to multiple
archives
“Towards a Web-Centric Approach for Capturing the Scholarly Record”, 2016-2019
https://github.com/oduwsdl/archivenow

Memento: Time Travel for the Web
Access mementos in
multiple web archives
Memento’s core
components:
• A bridge between
present and past: link
and content
negotiation
• A bridge between past
and present: link
June 26, 2018 27

Memento Aggregator
June 26, 2018 28

Memento Aggregator
June 26, 2018 29

How can I use Memento?
June 26, 2018
Memento for Chrome
http://ws-dl.blogspot.com/2013/10/2013-10-14-right-click-to-past-memento.html
http://ws-dl.blogspot.com/2014/10/2014-10-03-integrating-live-and.html
http://timetravel.mementoweb.org
30
Mink

Use Mink to view the odu.edu of the
past
June 26, 2018 31

Click the Mink icon
June 26, 2018 32

Then choose your datetime
June 26, 2018 33

Archived odu.edu
June 26, 2018 34

Fixing 404 Pages: Google Results Page
June 26, 2018 35

Fixing 404 Pages: Result Page
June 26, 2018 36
http://www.clashmusic.com/news/johnny-marr-leaves-the-cribs

Fixing 404 Pages: Scrolling Down
June 26, 2018 37

Fixing 404 Pages: Server Up, Page 404
June 26, 2018 38

Fixing 404 Pages: Using Mink
June 26, 2018 39

Fixing 404 Pages: Archived Page 2011-
04-16
June 26, 2018 40

#whatdiditlooklike
June 26, 2018 41
http://ws-dl.blogspot.com/2015/01/2015-02-05-what-did-it-look-like.html
Twitter bot
Include #whatdiditlooklike in a
tweet with a URL
Bot generates animated GIF of first
memento of each year
Bot replies with a link to entry in
Tumblr
Tumblr:
http://whatdiditlooklike.mementoweb.org/
Source:
https://github.com/anwala/wdill
Alexander Nwala, "2015-02-05: What Did It Look Like?,"

Use web archives to save the current
web and view the past web
• Web Science and Digital Libraries (WS-DL) group at
ODU
– ws-dl.blogspot.com, @WebSciDL (Twitter)
• Websites/Tools for web archiving
– Internet Archive's Wayback Machine - archive.org/web
– On-demand archiving - archive.is
– Memento Time Travel - timetravel.mementoweb.org
– Mink - matkelly.com/mink/
– #icanhazmemento
– #whatdiditlooklike
June 26, 2018 42

Intro to Web Archiving

More Related Content

Similar to Intro to Web Archiving

More from Michele Weigle

Recently uploaded

Intro to Web Archiving