This document discusses opportunities for the web archiving community to collaborate on building tools and standards together. It notes that most web archiving is currently done by a small number of large centralized organizations, but there is potential for more distributed and collaborative efforts. The document proposes several areas for collaboration, such as developing common APIs, modularizing archiving workflows into smaller components, creating community forums, and establishing shared standards and definitions. The goal is to strengthen web archiving by encouraging more participation and cooperation across different archives and organizations.

1. Building Web Archiving
Technology, Together
Nicholas Taylor
Web Archiving Service Manager
Stanford University Libraries
Web Archives 2015: Capture, Curate, Analyze
November 13, 2015

2. overview
• why build together?
• community for
collaborative work
• APIs for collaborative
work
“LAX on take off” by Doug under CC BY-NC-ND 2.0

3. not a programmer
“Bug” by Randall Munroe under CC BY-NC 2.5

4. aspiring OSS contributor
GitHub: “nullhandle (Nicholas Taylor)”

5. studying the landscape
“2010 Grand Canyon Celebration of Art 172” by Grand Canyon National Park under CC BY 2.0

6. a centralized enterprise
60%
25%
14%
63%
20%
16%
0%
10%
20%
30%
40%
50%
60%
70%
External Local Both
2011 2013
NDSA: “Web Archiving in the U.S.: A 2013 Survey”

7. a centralized enterprise
0 0
1
0
2
0
1
0
1
0
3 3
1
2
4
2
6
4
1
0
2
0
0
1
1
0
1 3
5
3
4 2
2
5
6
15
0
2
4
6
8
10
12
14
16
18
20
1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013
Number of organizations Archive-It Partner as of 2013
NDSA: “Web Archiving in the U.S.: A 2013 Survey”

8. minimal local preservation
19%
81%
20%
80%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
Transferred Haven't transferred
2011 2013
NDSA: “Web Archiving in the U.S.: A 2013 Survey”

9. evolving web
“Light Writing - Spider Web” by oz dean under CC BY-ND 2.0

10. opportunities for preservation
“standing out” by kenda bustami under CC BY 2.0

11. opportunities for research
“Exploring the Canadian Political Interest Group and Political Parties Web Sphere” by Ian Milligan under Standard YouTube License

12. not the only one
HUL: “Web Archiving Environmental Scan Home”
CDL: “Announcing a New Partnership”

13. a response

14. COMMUNITY
“Why we love Peckham, P1020468crop” by Eye magazine under CC BY-NC-SA 2.0

15. community analysis
SAA Web
Archiving
Roundtabl
e
Archive-It
Partners
IIPC
NDSA: “Web Archiving in the U.S.: A 2013 Survey”

16. Archive-It
Archive-It: “Archive-It 5.0 Feature Requests”

17. IIPC
Open HUB: “Open Wayback”

18. models of software production
(irrespective of license)
• sole source
– single developer
• closed source
– team/corporate dev; no outside contributions
• club source
– pool resources for solo/team/corporate dev
• community source
– direct and distributed community participation
• open source
– grassroots, democratic, meritocratic participation
Tom Cramer: “Collaborative Open Source Software Production & APIs”

19. club source examples
• Archivematica, AtoM (Artefactual)
• ArchivesSpace (Lyrasis)
• Bitcurator (Educopia)
• Fedora (DuraSpace)
• JHOVE (OPF)
• LOCKSS (Stanford University)
• Omeka (George Mason University)

20. community source examples

21. community architecture
• privileges community over code
• recognizes distribution of investment
• embraces community diversity
• models open processes and governance
• encourages varied contributions
• serves community needs

22. STANDARDS
“P1050827” by Rebecca Siegel under CC BY 2.0

23. success of a standard
• capture: DeDuplicator, Heritrix, python-
heritrix, SiteStory, WAIL, WARCreate,
WarcMITMProxy, WarcProxy, Webrecorder,
wget, Wpull
• access: OpenWayback, pywb, warc-proxy,
WarcManager, Wayback Machine, Web
Archive Discovery, WebArchivePlayer
• utilities: JHOVE2, JWAT, Megawarc, pylibwarc,
WARCAT, Warcbase, warctools, Web Archive
Commons

24. web archiving lifecycle
Internet Archive: “The Web Archiving Life Cycle Model”

25. missed opportunities?
Appraisal
and
Selection
Scoping
Data
Capture
Storage and
Organization
QA and
Analysis
Metadata /
Description
Access
/ Use /
Reuse
Preservation
Risk
Management
ACT
Archive-It
AtN
BCWeb
CDL WAS
DigiBoard
Islandora
WARC
Solution Pack
Netarchive
Suite
PageFreezer
UNT
Nomination
Tool
WCT

26. smaller, modular components
“Giant Rubik's Cube” by Francois Lamotte under CC BY 2.0

27. smaller projects do better
small projects (<$1 million) large projects (>$10 million)
on time/budget challenged failedon time/budget challenged failed
Standish Group: “Chaos Manifesto 2013: Thing Big, Act Small”

28. IIPC community interest in APIs
contribution type
% of
respondents
# of
respondents
help define functional
requirements
94% 15
contribute use cases 81% 13
help define technical details 69% 11
help schedule and run
meetings
19% 3
implement and test 6% 1
Andrea Goethals: “Results of the Web Archiving API Survey of IIPC Members”

29. API candidates
• capture tool/proxy
interconnect
• capture tool
management
• data import/export
• query + extraction
• integrity audit + repair
• descriptive metadata
• logs + analytics
• renderings/derivative
formats
• federated data
delivery
• federated replay
• federated full-text
search

30. let’s combine forces
“Stages of flow” by Peter Thoeny under CC BY-NC-SA 2.0