The document discusses various tools and technologies for web archiving, including Heritrix for crawling, the Web Curator Tool (WCT) and NetarchiveSuite for the full workflow from selection to access, WARC tools for working with archive files, NutchWax for indexing, and Wayback for public access to archives. It provides brief descriptions and comparisons of each tool's functionality and suitability for different archiving tasks and levels of technical expertise.

1. Web Archiving
Tools and Technology
Dan Chudnov - GWU Libraries
dchud at gwu edu
@dchud
IS&T Workshop, April 2, 2013
Washington DC USA
Tuesday, April 2, 13

2. select scope crawl process access
unt nom X X
tool
heritrix X X
wct X X X X
netarchive
X X X X X
suite
warc tools X
nutchwax X X
wayback X
Tuesday, April 2, 13

3. select
• what to collect
• who authorizes
• when
• what order
Tuesday, April 2, 13

4. scope
•how much
• robots.txt
• what to leave out
• which doors not
to open
Tuesday, April 2, 13

5. crawl
• start with seeds
• find, queue, follow links
• be kind to each site
• parallelize across sites
• schedule, log,
checkpoint, resume
• bundle
Tuesday, April 2, 13

6. process
• lump, split, bundle,
rebundle
• quality control
• index, surrogate,
reorder, prep for access
• store, distribute,
preserve
Tuesday, April 2, 13

7. access
• browse
• search
• known items
• patterns
• needles
Tuesday, April 2, 13

8. select scope crawl process access
unt nom X X
tool
heritrix X X
wct X X X X
netarchive
X X X X X
suite
warc tools X
nutchwax X X
wayback X
Tuesday, April 2, 13

9. UNT URL Nomination Tool
• collaborative
selection
• collect seed lists
• attach metadata
• agree on scope
• feed crawlers
Tuesday, April 2, 13

10. heritrix
• free software from
Internet Archive
• easy to start with
• difficult to master
• powerful, configurable,
confusing
Tuesday, April 2, 13

11. heritrix cont’d
• two major versions, “1” and “3”
• WCT and NetArchive embed “1”
• “1” - minimal UI
• “3” - even less
• iterate early - long learning curve
• best available tool
Tuesday, April 2, 13

12. heritrix cont’d
Tuesday, April 2, 13

13. Web Curator Tool
• free software from
NLNZ / BL
• full crawling workflow
suite
• select, obtain
permissions, authorize
• schedule, crawl w/
heritrix 1
Tuesday, April 2, 13

14. WCT cont’d
• quality review
• statistics, hierarchy
visualization, pruning
• troubleshooting
• task notifications
• reporting
Tuesday, April 2, 13

15. WCT cont’d
Tuesday, April 2, 13

16. NetarchiveSuite
• free software from
netarkivet.dk
• used by State and University
Library, The Royal Library in
Denmark
• complete solution from
selection to access
Tuesday, April 2, 13

17. NetarchiveSuite cont’d
Tuesday, April 2, 13

18. NetarchiveSuite cont’d
• selection, scoping,
scheduling
• crawling, troubleshooting,
tweaking
• system dashboard, quality
assurance
• heritrix and wayback
Tuesday, April 2, 13

19. warc-tools
• command-line tools for
arc/warc
• validate, summarize,
filter
• bundle / rebundle,
convert, index
Tuesday, April 2, 13

20. NutchWax
• free software
• index / search of ARC
data
• development slowed /
stopped but still used
Tuesday, April 2, 13

21. searching
web archives
is hard
Tuesday, April 2, 13

22. wayback
• free software from
Internet Archive
• public web access to
web archives
• what you’ve seen at
archive.org
Tuesday, April 2, 13

23. wayback cont’d
Tuesday, April 2, 13

24. wayback cont’d
Tuesday, April 2, 13

25. Tuesday, April 2, 13