Archiving Web News (captioned)

Archiving Web News
A Case Study

©2012 Sven Aas, Mount Holyoke College

Archiving Web News
Sven Aas
Web Team Lead
Mount Holyoke College

saas@mtholyoke.edu
sven.aas@gmail.com
@svenaas

#archwn

©2012 Sven Aas, Mount Holyoke College

The Early Years (’98–01)
Many years ago Mount Holyoke College published news stories as static web pages.

Text

(Wayback Machine)

#archwn ©2012 Sven Aas, Mount Holyoke College

The Portal (’01–08)
Then a custom portal system was built with the capability to publish news stories.

(Wayback Machine)


The CMS (’06–12)
Later a CMS was purchased, but not for News, so the portal was still employed.

(Wayback Machine)


The News System (’08–)
Then I built a custom system to house the portal news content and we've been running happily ever since.


Enter the Archivist
Archives received an NHPRC grant to pilot electronic records archiving projects including news.

LITS (Library, Information, and Technology Services)

DAPS (Digital Assets and Preservation Services)

ASC (Archives & Special Collections)

NHPRC (National Historical Publications and Records Commission)


What We Had


Numbers
There are two kinds of news items: stories and links. I’ll get to those in a minute

1998–2010:

• 34 News Channels
• 6,111 News Items
• 916 Links
• 5,195 Stories
• 5,260 News Images

News Item
Every news item has

• Title
• Thumbnail
• Summary
Some have

• Long Title

Stories and Links

Stories have

• Body
• Sidebar Image
Links have
• URL


Metadata
• Creation time
• Modiﬁcation time
• Creator
• Channel assignments
• Display start/end dates
• URLs where stories were available

Goals

• Preserve each news item.
• Maintain characters and markup in
text as they appeared.

• Preserve thumbnail and sidebar
images as they were presented.


Out of Scope

• Original (or current) presentation
• Higher resolution source images


Challenges


Consolidation

Challenge: Not all items were in the
Newswire system.
Approach: Staff and student
programmers worked to import them.


September 12, 2001
These strays were probably from an bulk operation or import into the portal system which took place on that date.

Challenge: A large number of news
items shared a single “creation” date.
Approach: Students determined real
dates where possible. An annotation
was provided in the export warning
about approximated dates.


Organization
Should we export each channel and let duplication of items take place?

Challenge: News items may be
crosslisted in various channels.
Approach: Consider channels as
secondary to items. Export each item
once, and group by year.


Preservation and Access
Where do we put this stuff? We had multiple digital asset management systems in place.

Challenge: Preserve news items and
make them available in an appropriate
digital asset management system.
Approach: Use our existing dSPACE
server.


Data

Challenge: Save news items in an
archivally appropriate and useful
format.
Approach: Render each item as a PDF.


Metadata

Challenge: Making key metadata
available to digital asset management
system.
Approach: Export Dublin Core
metadata XML ﬁle for each news item.


Import
Since I was exporting from an app I controlled I was able to favor choices which made things easier for dSPACE.

Challenge: Facilitate bulk rather than
individual import into dSPACE.
Approach: Cater to dSPACE’s
expectations in export folder and ﬁle
names and structure.


Output structure


Items

For each item three ﬁles are generated:

• contents
• dublin_core.xml
• [item_id].pdf


Sample contents

5682676.pdf


Sample XML


Sample PDF


Folders
Exported items are ordered chronologically and grouped by year. dSPACE looks for these numbered folders.

Each set of item ﬁles is in its own folder:

• item_000/
• item_001/
• item_002/


Details
PDF Export:

• pdfkit and wkhtmltopdf
Date/Time format:

• 1998-03-01T00:00:00-05:00
Channel references:

• By name rather than ID.

Problems


<i>citations<i>
HTML tags caused problems when they showed up in the XML metadata. So we had to strip tags.

Yeah, that’s right.
When titles in our Dublin Core XML
contained tags they were XML-escaped
on import.
Solution: Strip tags from titles.


 
But tag stripping turned non-breaking spaces into problems, so we had to modify the effects of tag stripping.

Uh-huh.
Some summary ﬁelds contained only a
 . Tag stripping turned this into a
 , which showed up as ? in
dSPACE.
Solution: Patch Builder library to avoid
exporting XML entities.


&
But we overdid it. We NEEDED to encode ampersands and apostrophes, and now we had to do it manually.

We overdid it. Ampersands and
apostrophes weren’t getting encoded in
the XML, and they needed to.
We had to manually encode these.


'
Of course, that wasn’t simple either. You have to ﬁgure out what your tools are assuming.

Complication:

• XML has ﬁve core entities: & < > ' "
• HTML has 252
• XHTML has 253
Guess which one’s in XML but not in
HTML ...


Export and Transfer


On December 19, 2011

• All content was downloaded to a
workstation and the export was run
there.

• The ﬁles were MD5 hashed and
electronically transferred to DAPS
and Archives.


And all was well.


But then I received an email from Erin Fahy in DAPS.

But then ...


##

• OK. So it turns out some old
summaries contained ##some unusual
markup##.
• At one point this markup was used to
locate a hyperlink, but for years it had
been stripped on display.

• So we decided to strip it too.


But you know it’s
never that simple ...


'

Remember '?
Well, something changed and PDF
headers which contained an ' got
screwed up.
This wasn’t a problem back in
December! What happened?


Approaches were tried.

.gsub(/'/, "'")


Approaches were tried.
"'"
"'"
"'"
'''
''
etc.

They didn’t work. None of them worked.


What worked?

"'''"
Are you kidding me?

It’s a long story. You can ask me later.


But it worked.
And there was much rejoicing.

Monty Python and the Holy Grail
#archwn ©Python (Monty) Pictures Ltd. ©2012 Sven Aas, Mount Holyoke College

Archiving Web News (captioned)

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (15)

Recently uploaded

Recently uploaded (20)

Archiving Web News (captioned)

Editor's Notes