Memento 101

Memento & Access to Resource Versions
Herbert Van de Sompel
http://mementoweb.org/
Memento
Uniform and Robust Access to Resource Versions
Memento has received funding
from
The Library of Congress
Andrew W. Mellon Foundation
IIPC
1

Memento Makes Navigating the Web’s Past Easy
2
RFC 7089 (2013) Van de Sompel, H., Nelson, M.L., Sanderson, R.
HTTP Framework for Time-Based Access to Resource States - Memento
http://tools.ietf.org/html/rfc7089

Today
Select Date
June 20 1997
June 5 1997
From archive.today
Memento: Access Versions via the Original URI and a Datetime
3

Today
Select Date
June 27 2011
May 29 2011
From
Internet Archive
Memento: Access Versions via the Original URI and a Datetime
4

The Memento protocol achieves this by introducing
a uniform, datetime-based, version access capability
that integrates the Present and Past Web.
5

Problem Statement …
6

Resources
7

Resources have Representations
8

Resources have Representations that Change over Time
9

Only the Current Representation is Available from a Resource
10

Old Representations are Lost Forever
11

But … Archived/Version Resources Exist
12

There are resource versions on
the Web, in:
• Web Archives;
• Content Management
Systems;
• Search engine caches;
• Transactional archives.
13

Web Archive
Archived Resource
URI-M - http://web.archive.org/web/20010911203610/http://www.cnn.com/
URI-R - http://www.cnn.com/

Web Archive
Archived Resource
URI-M - https://archive.today/UD0d6
URI-R - http://www.w3.org/

Version Resource
URI-M - http://en.wikipedia.org/w/index.php?title=September_11_attacks&oldid=282333
CMS
URI-R - http://en.wikipedia.org/wiki/September_11_attacks

Search Engine Cache
Cached Resource
URI-R – http://ghr.nlm.nih.gov/handbook/basics/dna
URI-M - http://webcache.googleusercontent.com/search?q=cache:kDmDc1PIA38J:
ghr.nlm.nih.gov/handbook/basics/dna+&cd=2&hl=en&ct=clnk&gl=us

Archived Resource
Transactional Archive
URI-R - http://dans.knaw.nl/en
URI-M -
http://www.theresourcedepot.com/000010/memento/20130418204153/http://dans.knaw.
nl/en

But, without Memento, the Web
handles these version
resources poorly:
• Cannot talk, in URI terms,
about a resource as it used to
exist
• Cannot access a prior version
knowing the current one
• Cannot access the current
version knowing a prior one
Solutions are ad hoc and
localized
19

Without Memento, the Current and Past Web Lack Integration
20
• Going from Current to Past
Web is a matter of (manual)
discovery
• Navigating the Past Web is
only possible within the
boundary of a single web
archive, versioning system
• Memento integrates the
Current And Past Web by
means of an extension of
HTTP
• Memento turns archives,
versioning systems into
infrastructure rather than
destinations

Systems with Resource Versions
system type stores URI-R and URI-M
web archive observations over time different baseURL
CMS history same baseURL
search engine cache one recent observation different baseURL
transactional archive history different baseURL
These systems have different characteristics
but the Memento protocol allows uniform versions access to their resources
21

The Memento Framework:
Protocol to Integrate Present and Past Web
Overview
22

The Memento protocol:
• Regards the Web as a big
Content Management System
• Introduces an interoperable
approach to access resource
versions across the Web
• Does not build new archives
but leverages all systems that
host versions
23

Memento’s approach to access
resource versions:
• Is distributed: versions may
exist on several servers
• Uses time as a global version
indicator
• Is based on the primitives of
the Web: resource, state,
representation, content
negotiation, link
24

Memento’s approach to access
resource versions has two
components:
• Access to a single
archived/version resource –
via datetime negotiation with
a TimeGate
• Access to an overview of
existing versions – by
requesting a TimeMap
25

26
Memento Protocol Resource Types
Original Resource: Resource that exists or used to exist;
we are interested in accessing a past state of it
Memento: Resource that is a prior version of the Original
Resource; it encapsulates a past state of the Original Resource
TimeGate: Resource that “decides”, based on a given datetime,
which is the temporally best Memento for an Original Resource
TimeMap: Resource that provides a list of known Mementos for
an Original Resource as well as their datetime

Datetime Negotiation
27

28
Original Resource and Mementos

29
Bridge from Present to Past

30
Bridge from Present to Past

31
Bridge from Past to Present

32
Bridge from Past to Present

33
Memento Datetime Negotiation Component

34
Memento Protocol Datetime Negotiation Patterns
The different Patterns are discussed in RFC 7089
Here, we deal with URI-R <> URI-G <> URI-M and 302 style negotiation
can coincide with
can coincide with
302 or 200 style negotiation can be used

35
Memento Datetime Negotiation - Client Server Interaction
Yes, G
It’s at M

Memento Datetime Negotiation - HTTP Flow
HEAD R, [Accept-Datetime]
[Link  G]
302  M, Vary, Link  R,[M,T]
200, Memento-Datetime, Link  R,[G,M,T]
HEAD G, Accept-Datetime
GET M, [Accept-Datetime]
[…]== optional

37
Original Resource Provides No Link – Client Intelligence

38
Original Resource Gone – Client Intelligence

39
Original Resource Gone – Server Due Dilligence

40
Original Resource’s Server Gone – Client Intelligence

41
Memento Aggregator

42
TimeGates
A list of TimeGates provided by major web archives as well as
by-proxy TimeGates provided for other systems is maintained at
http://mementoweb.org/depot/
http://timetravel.mementoweb.org/guide/api/#registry

TimeMaps
43

44
TimeMap
• multiple TimeMap serializations possible
• application-link/format mandatory
• When TimeMaps become too large, they can
be broken up and paged

45
TimeMaps
A list of TimeMaps provided by major web archives as well as
by-proxy TimeMaps provided for other systems is maintained at
http://timetravel.mementoweb.org/guide/api/#registry

HTTP Headers
46

The HTTP Headers used in the Memento Protocol
• Define two new headers:
– request: Accept-Datetime:
– response: Memento-Datetime:
• Introduce new content for two existing headers:
– response: Vary: ; Link:
• Use one existing header without modification:
– response: Location:, TCN:
47

HTTP Request Headers for Datetime Negotiation
• Accept-Datetime:
o Issued against TimeGate, [Original Resource, Memento]
o Header value: desired datetime of a Memento
Accept-Datetime: Mon, 12 Oct 2009 14:20:33 GMT
48

HTTP Response Headers for Datetime Negotiation
• Memento-Datetime:
o Returned by Mementos only
- Even when not as a result of datetetime negotiation
o Header value: Archival datetime of the Memento
- Resource has not and will not change beyond that date
o This header is sticky:
- Once returned, a server must always return it with same
value
- Must also be preserved when Mementos are mirrored at
different URIs
o This header is crucial to allow a client to understand it has
arrived at a Memento
Memento-Datetime: Mon, 12 Oct 2009 14:20:33 GMT
49

HTTP Response Headers Datetime Negotiation
• Vary:
o Returned by TimeGate
o Similar to regular content negotiation
o Header value: accept-datetime
• Regular content negotiation (e.g. media type) can be used too but
a TimeGate must first meet the datetime preference, and then – if
possible – the other content negotiation preferences
• Note: accept-datetime in Vary header is crucial to allow a
client to understand it has arrived at a TimeGate
Vary: accept-datetime
50

• Location:
o Returned by TimeGate
o Similar to regular content negotiation
o Header value: URI of the Memento selected by the TimeGate
Location:
http://web.archive.org/web/20010911223004/http://cnn.co
m
51

• Link:
o Returned by Original Resource, TimeGate and Mementos
o Various new Relation Types are introduced:
- “original” – points to Original Resource
- “timegate” – points to TimeGate
- “memento” – points to Memento
- “timemap” – points to TimeMap
o A TimeGate must provide the “original” link
o A Memento must provide the “original” link
o All other links are encouraged but optional
52
HTTP Link Header: RFC 5988

• Link:
o The following ”memento” links that point at special Mementos,
known to the responding server, are optional but very useful:
- First and last Memento known to the server, e.g. ”memento
first”
- Memento prior and after the selected Memento, e.g.
“”memento predecessor-version”
- Selected Memento
- Temporal order of Mementos is expressed using existing
relation types from RFC 5829 and RFC 5988: first, last,
next, prev, successor-version, predecessor-
version
53

• Link:
o Attributes for a ”memento” Link:
- datetime (mandatory): datetime of the Memento pointed at
by the link
- license (optional): license associated with the Memento
o Attributes for a ”timemap” Link:
- type (recommended): MIME type of TimeMap serialization
- from, until (optional): to convey the temporal interval of
Memento datetimes covered by the TimeMap
54

Memento Datetime Negotiation - HTTP Flow
HEAD R, [Accept-Datetime]
[Link  G]
302  M, Vary, Link  R [M T]
200, Memento-Datetime, Link  R [G M T]
HEAD G, Accept-Datetime
GET M, [Accept-Datetime]
[timegate]
original [memento timemap]
original [timegate memento timemap]

HTTP Interactions
56

Datetime Negotiation Flow: Step 1
57

58

59

60

61

62

TimeMap Access Flow: Step 1
63

64

65

66

67

68

TimeMap Access Flow: Step 6 with Index TimeMap
69

TimeMap Access Flow: Step 6 with Paging TimeMap
70

Resource Versioning and Memento
71

Common Resource Versioning Approach

Version Resources
(*) Tim Berners-Lee (1996) http://www.w3.org/DesignIssues/Generic.html
(*)

Version Resources and Associated Generic Resource
(*)
(*)
(*) Tim Berners-Lee (1996) http://www.w3.org/DesignIssues/Generic.html

Memento Bridges Between Generic & Specific Resources

Stepwise Support for the Memento Protocol – Step 1
• Provide Memento protocol HTTP response headers to convey version
date and links
o Provide Memento-Datetime header to express version date
o Provide Link header with “original” link to point from version
resource to generic resource
o Provide Link header with appropriate “memento” links to allow
navigating between versions
- In combination with links with other relation types, e.g.
“first”, “last”, “prev”, “next”, “predecessor-version”,
“successor-version”
78

• Response to HTTP HEAD/GET against
http://www.w3.org/TR/2004/PR-webarch-20041105/
79

80
• Publish a TimeMap, at, say,
http://www.w3.org/TR/timemap/webarch/
• For the generic resource and for each version resource, provide a
Link header with “timemap” link that points at the TimeMap

http://www.w3.org/TR/2004/PR-webarch-20041105/
81

• Response to HTTP GET against
http://www.w3.org/TR/timemap/webarch/
82

83
• Expose a TimeGate, at, say,
http://www.w3.org/TR/timegate/webarch/
• Reponses for generic resource, version resources, TimeGate,
TimeMap as shown in slides 56-70
• Note that Patterns for datetime negotiation other than the one
shown in those slides are described in RFC 7089

Memento and Linked Data
84

User & Developer Tools
87

Memento for Chrome
88http://bit.ly/memento-for-chrome

Time Travel Find – Search Page
http://timetravel.mementoweb.org/

Time Travel Find – Result Page
http://timetravel.mementoweb.org/list/20100428103432/http://stanford.edu

Time Travel Find – Result Page
http://timetravel.mementoweb.org/list/20140428052227/http://coptr.digipres.org/Main_Page

Time Travel Reconstruct – Search Page
http://timetravel.mementoweb.org/

Time Travel Reconstruct – Result Page
http://timetravel.mementoweb.org/reconstruct/20100428103432/http://stanford.edu

Memento for MediaWiki Extensions
96http://bit.ly/memento-for-mediawiki

Generic TimeGate Server (1/2)
https://github.com/mementoweb/timegate

Generic TimeGate Server (2/2)
https://github.com/mementoweb/timegate

SiteStory Transactional Archive for Apache Servers
https://mementoweb.github.io/SiteStory/

100
Memento Aggregator
Coverage: See http://mementoweb.org/depot/ and
http://labs.mementoweb.org/aggregator_config/archivelist.xml

Various Memento Tools for Users & Developers
101http://mementoweb.org/tools/

Time Travel APIs
102

Time Travel APIs
http://timetravel.mementoweb.org/guide/api/

URI that Redirects to a Memento
http://timetravel.mementoweb.org/memento/20100428103432/http://stanford.edu

URI that Redirects to a JSON Description of a Memento
http://timetravel.mementoweb.org/api/json/20100428103432/http://stanford.edu

JSON Format for TimeMaps
http://mementoweb.org/guide/timemap-json/

DIY TimeMap - Index TimeMap Lists Potential TimeMap URIs
http://timetravel.mementoweb.org/timemap/json/http://stanford.edu
SPEED

WDI TimeMap - Regular (Index) TimeMap
http://labs.mementoweb.org/timemap/link/http://stanford.edu
COVERAGE

Time Travel Archive Registry
http://labs.mementoweb.org/aggregator_config/archivelist.xml

Robust Links
110

How to Reference Resources
• Create a Capture in Internet Archive, archive.today, perma.cc,
webcitation
• Existing practice for linking to such captures:
o Link to URI of Capture
o Lose Original URI
o Lose Capture Datetime
• Problems with existing practice:
o Impossible to visit the original URI, if desired
o Requires the permanent existence/uptime of the archive that
holds the capture
- One link rot problem replaced by another
Van de Sompel, H. et al. (2013) Thoughts on referencing, linking, reference rot
http://mementoweb.org/missing-link/

Permanent Existence/Uptime of Archives?
Capture of http://webcitation.org dated July 17 2013
https://archive.today/eAETp

http://webcitation.org/ on August 6 2014

Remnant of discontinued web archive http://mummify.it captured on February 14 2014
https://web.archive.org/web/20140214233752/https://www.mummify.it/

http://www.themoscowtimes.com/news/article/russia-bans-wayback-machine-internet-archive-over-
islamic-state-video/510074.html

Hacking Original URI, Capture Datetime from Capture URI?
URI of Capture Original URI Datetime T
https://web.archive.org/web/20140214233752/https://
www.mummify.it
yes yes
https://archive.today/eAETp no no
http://perma.cc/4RH7-999Q?type=source no no
http://en.wikipedia.org/w/index.php?title=Coil_(band)
&oldid=388321480
no no

Using Capture URI to find Captures in Other Web Archives?

Reference Resources Robustly
• When referencing resources include:
o Original URI – Allows revisiting the URI as it is at the time of
reading, if the URI is still operational
o Snapshot URI – Allows revisiting the snapshot, if one was
created, and if the web archive in which it was created is still
operational
o Original URI & Date/Time allows revisiting a snapshot
created around the Date/Time in any web archive around the
world (using Memento infrastructure)
Van de Sompel, H. et al. (2013) Thoughts on referencing, linking, reference rot
http://mementoweb.org/missing-link/

Reference Resources Actionably
• When referencing resources, use Link Decorations to convey
Original URI, Snapshot URI, Date/Time
<a href=“http://www.stanford.edu”
data-versionurl=“http://archive.is/FAy6o”
data-versiondate=“2014-08-15” >
<a href=“http://www.stanford.edu”
Herbert Van de Sompel et al. (2015) Robust Links - Link Decorations
http://robustlinks.mementoweb.org/spec/
<a href=“http://archive.is/FAy6o”
data-originalurl=“http://www.stanford.edu”

No Link Decorations? Insert Page Date!
• Include page date to allow retrieving Mementos of linked resources
from around page publication date
<html>
<head lang=“en”
itemtype=“http://schema.org/WebPage”
itemid=“http://robustlinks.mementoweb.org/spec/”>
<meta itemprop=“datePublished” content=“2015-01-23”>
Herbert Van de Sompel et al. (2015) Robust Links - Link Decorations
http://robustlinks.mementoweb.org/spec/

Robust Links via Link Decoration, JavaScript, Time Travel API
• JavaScript makes link decorations actionable
http://robustlinks.mementoweb.org/demo/uri_references_js.html
JavaScript makes
the info actionable

Pointers
123

Pointers
• Memento site - http://mementoweb.org/about/
• Time Travel site – http://timetravel.mementoweb.org
• RFC 7089 - http://tools.ietf.org/html/rfc7089 (text version),
http://www.mementoweb.org/guide/rfc/ (HTML version)
• Memento Development List -
http://groups.google.com/group/memento-dev/
• Memento GitHub projects - https://github.com/mementoweb/
• Client and Server software and tools -
http://mementoweb.org/tools/
• Information on TimeGates and TimeMaps for major systems -
• IIPC list of software and tools related to web archiving -
http://netpreserve.org/web-archiving/tools-and-software
124

Additional Details
125

Fixed Resource
• The resource is its own Memento, i.e. it is a stable resource
o Resource that was born stable or became stable; it will not change
anymore, e.g. PermaLink resources on news sites
o Resource provides:
- Link header with ”original” link pointing to itself
- Memento-Datetime header
o Note the difference with Last-Modified header: no promise
resource will not change anymore
- Details at http://ws-dl.blogspot.com/2010/11/2010-11-05-
memento-datetime-is-not-last.html
126

Fixed Resource
http://a.example.org
127

Memento Without TimeGate
• The resource is a Memento but there is no TimeGate available for it
o e.g. snapshot of resource when server is being retired
- Link header with ”original” link revealing the URI of
Original Resource
- Memento-Datetime header
128

Memento Without TimeGate
http://arxiv.example.net/web/20010321203610/http://
a.example.org
129

Intermediate Resource
• The resource issues a redirect to a TimeGate, a Memento, another
intermediate resource
o Plays an active role in the Memento framework
- Link header with ”original” link revealing the URI of
Original Resource
130

Intermediate Resource
• Response to HTTP HEAD/GET against a resource that redirects to a
TimeGate
131

Resource Excluded from Datetime Negotiation
• e.g. JavaScript, logos, banners added by web archives
o Resource always needs to be used in its current state
o In order to flag it is excluded from datetime negotiation, this
resource provides:
- Link header with ”type” link that has as value
http://mementoweb.org/terms/donotnegotiat
e
132

Resource Excluded from Datetime Negotiation
• Response to HTTP HEAD/GET against a resource that is excluded
from datetime negotiation
133

Memento of a Redirect
• HTTP responses with 3XX codes are also archived
o e.g. web archives hold on to “301 Moved Permanently” and “302
Found” whereas Linked data archives preserve “303 See Other”
• The Memento’s response must have the same HTTP status code as
the original
• Memento headers are as usual
• Memento clients need to understand that the redirect (URI in Location
header) can be to an Original Resource or to a Memento
o If an Original Resource, the client must proceed to find an
appropriate Memento for it
134

• Response in April 2008 to HTTP HEAD/GET against
http://a.example.org
135

• Response to a HTTP HEAD/GET of a Memento of that 2008 redirect,
whereby the redirect is unchanged, i.e. it is to the resource to which
the redirect originally led
136

• Response to a HTTP HEAD/GET of a Memento of that 2008 redirect,
whereby the redirect is rewritten, i.e. it leads to a Memento of the
resource to which the redirect originally led
137

http://mementoweb.org/
Memento
Uniform and Robust Access to Resource Versions
Memento has received funding
from
The Library of Congress
Andrew W. Mellon Foundation
IIPC
138

Memento 101

More Related Content

Viewers also liked

Similar to Memento 101

More from Herbert Van de Sompel

Recently uploaded

Memento 101