Courtney Mumma presenteerde op de studiedag Een web van webarchieven (17-11-2016, NCDD/Beeld en Geluid/Netwerk Digitaal Erfgoed) over de web crawling tools en services van the Internet Archive.

1. Web Crawling Tools and Services from
the Internet Archive: Archive-It and
Contract Crawling
Courtney C. Mumma, Internet Archive
November 17, 2016 - Dutch Institute for Sound and Vision

2. Talk overview
● Archiving the web at IA
● Partnerships and services
○ Contract crawls
○ Archive-It
■ Research Services
○ Interoperability & Distributed Preservation
● New technology for new challenges

3. The Internet Archive
Non-Profit Library
Founded in 1996 by Brewster Kahle
Universal Access to All Knowledge

4. 30,000,000,000,000,000 Bytes Archived
(30 PetaBytes)
20 Years of Archiving the Web
500,000,000,000+ URLs

5. 1996
US Presidential Campaigns with Smithsonian
218,342,520
Web Captures

6. 1997
First Full Crawl
525,362,846
Web Captures

7. 1998
Donation of Crawl to the Library Of Congress
1,166,891,826
Web Captures

8. 2000
US Presidential Campaigns with the Library of Congress
6,153,042,235
Web Captures

9. 2001
Launch of the WayBack Machine
12,082,859,018
Web Captures

10. 2003
International Internet Preservation Consortium
Founded
38,868,116,181
Web Captures

11. 2006
Archive-It Started
103,943,903,726
Web Captures

12. 2007
Ireland
184,277,909,308
Web Captures

13. 2008
National Archive Government Crawls
209,160,715,829
Web Captures

14. 2009
Archive-It Adds its 100th Partner
7 National Library Partners
225,658,093,516
Web Captures

15. 2010
Broad and Survey Web-Scale Crawls
246,744,306,660
Web Captures

16. 2015
Archive-It Adds its 400th Partner
467,195,419,06
9
Web Captures

18. Global Wayback
● Broad snapshot
● Deep crawl on popular
sites
● Broad crawl on known
domains
● No more 404s
● On-demand
● Donated and targeted crawls
● https://web-beta.archive.org/
with KEYWORD SEARCH
and more!

19. Support Open Source Software

20. Web Archiving Partnerships
and Services

21. Domain Scale Web Preservation

22. Contract Crawling
Domain-scale
• Run by Internet Archive
• Average 300 million URLs per collection
Partial List of Partners
• National Libraries of Australia and New Zealand
• U.S. National Archives and Library of Congress
• Luxembourg National Library
• Israel National Library
Partial List of Collections
• Iraq War (2003-2011)
• 2005 US Supreme Court Nominations

23. Archive-It
Curated, Selective Web Archiving

24. Archive-It
Web based - nothing to install
Fully hosted service with
unlimited support
Simple to select, manage, scope
and catalog with metadata
10 different crawl frequencies
Includes quick access and
storage
html, videos, audio, social
media, PDFs, images, news
Full text search
Restricted access options

28. How our partners use Archive-It
● Enhance and supplement traditional offline collections
○ archives, topical collections
● Support records retention and archival policies
● Capture event-based content
○ Spontaneous
○ Planned
● Individual organizations and Consortial collaboration

29. Research Services

30. Goals of Archive-It Research Services
● Expand access models for web archives
● Enable new insights into collections
● Leverage Internet Archive infrastructure for large-scale
processing to produce datasets for research
● Facilitate computational analysis and new use cases
● Increase use, visibility, and value of Archive-It partner
collections

31. Web Archives Datasets
Archive-It Research Services
http://bit.ly/ait_ars

32. Exploring the Canadian Political Interest Group and
Political Parties Web Sphere via WAT files

33. Named Entities in the Human Rights Collection

34. Systems Interoperability and Distributed
Preservation

35. Lost in the maze in Labyrinth (1986, LucasFilm, screen capture)
WARCs, CDXs
and derivatives
Access
Storage
Preservation
Content Mgmt
Web Archiving Tools

37. APIs
(*application programming interfaces)
● Interoperability
● Flexibility and modularity
● Loose coupling of services (so we can improve pieces as
needed)
● Scalability - Bulk data upload and download

38. New technology to face new challenges

40. Ongoing efforts
• Open Wayback
• Social media / Dynamic content
– Brozzler and Umbra (Archive-It)
– Social Feed Manager (GWU)
• URL nomination tools (UNT)
• Capture tools (GWU, IA, Rhizome)
• WASAPI - Community building and API
• Memento
BROZZLER!

41. ????s
THANK YOU
courtney@archive.org