Summarizing archival collections using storytelling techniques

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Summarizing archival collections
using storytelling techniques
Yasmin AlNoamany
Michele C. Weigle
Michael L. Nelson
Old Do...

Archive-It, a subscription-based service,
allows creation of collections
2
> 3,000
collections
> 340
institutions
> 10B ar...

3
Collection
title
Collection
categorization
based on the
curator
Seed URI
Metadata
about the
collection
Text search
box
T...

Collection understanding and collection
summarization are not currently supported
Not easy to answer “what’s in that colle...

There is more than one collection about
“Egyptian Revolution”
5
• “2010-2011 Arab Spring” https://archive-it.org/collectio...

6
One of at least seven Human Rights collections…

Our early attempts at collection understanding
tried to include everything…
9
“Visualizing digital collections at Archive-...

1000s of seeds X 1000s of archived pages ==
Conventional Vis Methods Not Applicable
10

Idea:
Storytelling
11

Stories in literature
Story elements: setting, characters, sequence, exposition, conflict,
climax, resolution
Once upon a ...

Stories in social media
“It's hard to define a story, but I know it when I see it” (Alexander, 2008)
basically, just arran...

“Storytelling” is becoming a popular
technique in social media
14

What are the limitations of
storytelling services?
15

The Egyptian Revolution on Storify
16

Bookmarking, not preserving!
17

Despite these limitations, how do we
combine storytelling & archives?
18

Use interface people already know how to use
to summarize collections
Archived collectionsStorytelling services
Archived e...

We sample k mementos from N pages of the
collection (k << N) to create a summary story
S
1
S
2
S
3
S
4
S
2
S
1
S
3
Collect...

Yasmin hand-crafted stories to summarize the
Egyptian Revolution collection for her son, Yousof
https://storify.com/yasmin...

How do we generate this automatically?
22

Collections have two dimensions:
{Fixed, Sliding} X {Page, Time}
t1 t3t2 t5t4 tk
…
URI
Time
t6
23
…
…

Fixed Page, Fixed Time
A desktop Chrome user-agent
http://www.cnn.com/2014/02/24/world/africa/egypt-
politics/index.html?h...

Feb 1 Feb 1 Feb 2
Feb 4 Feb 5 Feb 7
Feb 9 Feb 11 Feb 11
25
Fixed Page, Sliding Time

Feb. 11, 2011
Mubarak resigns
26
Sliding Page, Fixed Time

Jan 27 Jan 31
Feb 7Feb 4
Feb 11 Feb 11
Feb 2
Jan 25
Feb 10
27
Sliding Page, Sliding Time

The Dark and Stormy Archives (DSA) framework
Establish a
baseline
Reduce the candidate
pool of archived pages
Select good
...

Establish a baseline of
social media stories
"Characteristics of Social Media Stories”, TPDL 2015, IJDL 2016.
29

What is the length of a story
(the number of resources per story)?
This story has
31 resources
1
3
2
30

What are the types of resources that
compose a story?
Quotes
Video
31
This story has
• 19 quotes
• 8 images
• 4 videos

What are the most frequently used domains?
Twitter.com
Twitter.com
Twitter.com
32
This story has
• 90% twitter.com
• 7% in...

Top 25 domains represents 92%
of all domains
33

What differentiates a popular story?
(popular = stories with the top 25% of views)
19,795 views 64 views
34

The distributions for the features of the stories
• Based on Kruskal-Wallis test, at the p ≤ 0.05 significance level, the ...

Do popular stories have a lower decay rate?
The 75th percentile of decay rate per popular story is 10% of the resources,
w...

We found that 28 mementos is a good
number for the resources in the stories.
37

Establish a baseline of current
Archive-It collections
"Characteristics of Social Media Stories. What makes a good story?"...

The mean and median number of
URIs in a collection
This collection has 435 seed URIs 39

The mean and median number of
mementos per URI
This seed URI has 16 mementos 40

The most frequent used domains
abcnews.go.com
blogspot.com
This collection has 30% abcnews.com, 10% blogspot.com, 3% faceb...

Archive-It top 25 is fundamentally
different than Storify top 25
42

Archive-It top 25 is fundamentally
different than Storify top 25
43
Twitter
is #10
not #1

What we archive and
what we share on social media
are different subsets of the web
(seeds != shares)
44
see also: Brunelle...

Detecting off-topic pages
"Detecting Off-Topic Pages in Web Archives”, TPDL 2015, IJDL 2016.
45

Archive-It provides their partners with tools
that allow them to build themed collections
46

Archive-It tools are about HTTP events /
mechanics, not “content”
47

These tools won’t detect that > 60% of
mementos of hamdeensabahy.com are off-topic
May 13, 2012: The page started as
on-to...

How do we automatically detect
off-topic pages?
49

Textual content
cosine similarity, intersection of the most frequent terms,
Jaccard similarity
Method Similarity
cosine 0....

Textual content
cosine similarity, intersection of the most frequent terms,
Jaccard similarity
Method Similarity
cosine 0....

Semantics of the text
Web based kernel function using the search engine (SE)
52
Sahami and Heilman, A Web-based Kernel Fun...

Semantics of the text
Web based kernel function using the search engine (SE)
Method Similarity
SE-Kernel 0.7
53
Sahami and...

Structural methods
no. of words, content-length
100 109
Method % change
WordCount 0.09
54

Structural methods
no. of words, content-length
100 109
100 5
Method % change
WordCount 0.09
Method % change
WordCount -0....

We built a gold standard data set to
evaluate the methods
56

We manually labeled 15,760 mementos
Egypt Revolution and Politics
URI-Rs: 136
URI-Ms: 6,886
Off-topic URI-Ms: 384
Occupy M...

Evaluated 6 methods + combos at 21 thresholds
Averaged the results at each threshold over the three gold standard
collecti...

Average precision of 0.89 on 18 different
Archive-It collections
59
(Cosine,WordCount) with (0.10,-0.85) thresholds

How do we dynamically divide the
collections into appropriate slices?
(in other words, how do we pick just 28?)
60

We expected most collections to look like this…
The Global Food Crisis collection at Archive-It
61

This is what we found
Egypt Revolution and Politics
Human RightsApril 16 Archive Virginia Tech Shooting
Jasmine Revolution...

Selecting representative pages for
generating stories
(skipping clustering details, but goal is k=28)
63

Quality metrics for selecting mementos
• In the DSA, memento quality Mq is calculated as
following:
Mq = (1 − wm*Dm) + wql...

We prefer a higher quality memento (Dm)
http://wayback.archive-it.org/2358/20110201231457/
http://news.blogs.cnn.com/categ...

We prefer pages with attractive snippets
https://wayback.archive-it.org/2358/20110207193404/http://news.blogs.cnn.com/2011...

We prefer deep links over
high level domains (Sql)
Feb. 11, 2011: the homepage of BBC on Storify
Feb. 11, 2011: the homepa...

Social media pages may not produce
good snippets (Sqc)
http://wayback.archive-it.org/1784/20100131023240/http:/twitter.com...

Visualizing stories in Storify
69

Remember Yasmin’s hand-crafted stories?
70

Remember Yasmin’s hand-crafted stories?
71

We extract the metadata of the pages
and order them chronologically
{ "elements":[
{
"permalink":"http://wayback.archive-i...

Example of an automatically generated story
73
Notice the good
metadata: images,
titles with dates,
favicons

Evaluating the Dark and Stormy
Archive framework
(how good are the automatically generated stories?)
74

Evaluation is tricky!
(two perfectly good stories could have non-overlapping k=28
elements!)
• We use human evaluators (vi...

Our guidelines for expert archivists at Archive-It
for generating stories from the collections
76

We received 23 stories for 10
Archive-It collections
SPST is “Sliding Page, Sliding Time”
SPFT is “Sliding Page, Fixed Tim...

https://storify.com/mturk_exp/3649b1s-57218803f5db94d11030f90b 78
• Generated by domain experts
• Sliding Page, Sliding Ti...

Automatically generated stories from
archived collections
1. Obtain the seed list and the TimeMap of URIs from the front-e...

https://storify.com/mturk_exp/3649b0s
80
• Automatically generated story
• Sliding Page, Sliding Time
• The Boston Maratho...

Random stories
28 mementos were randomly selected from each
collection before excluding off-topic and duplicate
pages
81

https://storify.com/mturk_exp/3649b2s-57227227bb79 048c2d0388dc 82
• Randomly generated story
• Sliding Page, Sliding Time...

https://storify.com/mturk_exp/3649bads 83
if someone prefers this story,
we exclude their results
• Poorly generated story...

MT experiment setup
• Three HITs for each story (69 HITs to evaluate 23 stories); two
comparisons per HIT:
• HIT1: human v...

A sample HIT
85

DSA == Human
(Human,DSA) > Random
86

Automatic versus Human
87
Sliding Page, Sliding Time Sliding Page, Fixed Time Fixed Page, Sliding Time

Human versus Random
Sliding Page, Sliding Time Sliding Page, Fixed Time Fixed Page, Sliding Time
88

Automatic versus Random
89
Sliding Page, Sliding Time Sliding Page, Fixed Time Fixed Page, Sliding Time

Success!
DSA-generated stories are just as good as stories
generated by human experts
90

Use interface people already know how to use
to summarize collections
Archived collectionsStorytelling services
Archived e...

Share Slideshare

LinkedIn
Facebook
Twitter

Embed

Size (px)

Show related Slideshows at end

WordPress Shortcode

Link

Share
Email

Where Can We Post Stories Summarizi... by Shawn Jones 1946 views
Robust Linking to Web Resources by Martin Klein 1775 views
Combining Heritrix and PhantomJS fo... by Michael Nelson 3584 views
Profiling Web Archive Coverage for... by Michael Nelson 4351 views
More Archives, More Better by Michael Nelson 1642 views
We Need Multiple, Independent Web A... by Michael Nelson 2304 views

View on Slideshare

1 of 91 Ad

View on Slideshare

1 of 91 Ad