Much U. S. federal data are perceived as being at risk of becoming inaccessible through lack of maintenance and funding shifts. Organizations such as the End of Term Project and Data Rescue have emerged to coordinate the backup and rescue of at-risk federal data. Tufts University conducted a curated harvest to back up potentially at-risk federal, environmental and social justice geospatial data and associated tabular data. Tufts ongoing harvest has recovered over 40 TB of data. Tufts developed the Crawler to crawl all data: unzip files, identify file types, sizes, local directories, and harvest and process all related metadata using data mining and natural language processing (NLP) techniques. The Crawler results in detailed collection level and layer level metadata analytics for assessment, search and discovery.

1. Tufts Spatial Data Rescue:
Crawling at-risk Government
Data
Kyle Monahan
Statistics and Research Technology Specialist
Tufts University
FOSS4G | Boston, MA | 8/17/2017

2. Background
•What is a data rescue?
• Methods and techniques to identify, store and preserve
datasets
• Predominantly data associated with government entities
• *.gov, *.mil, *.edu, *.org, etc.
• Especially critical during election transitions
12/23/2017 FOSS4G Conference | Boston, MA 2

3. Background - History
12/23/2017 KMM 3

4. Example: 2008 End of Term Harvest
12/23/2017 FOSS4G Conference | Boston, MA 4
• National Archives and Records Administration announced they
would be unable to rescue data as they did in 2004.
• International Internet Preservation Consortium (IIPC) responded
by organizing a crawl:
• California Digital Library
• Internet Archive
• Government Printing Office
• Library of Congress
• University of North Texas
• Goal: “comprehensive harvest” (EOTerm Archive, 2016)

5. Example: 2008 End of Term Harvest
12/23/2017 FOSS4G Conference | Boston, MA 5
• Consisted of three main crawls:
• Pre-election
• Post-election
• Post-inauguration
• Produced over 16 TB of data
• And 160,211,356 URIs (Phillips, 2016)
End of Term, 2008

6. Methods of Tufts Crawl
12/23/2017 FOSS4G Conference | Boston, MA 6
• Access to up-to-date Federal data is critical for our Data Lab
• GIS and statistics classes rely on federal data (e.g. US Census,
TRI, HUD)

7. Methods of Tufts Crawl
12/23/2017 FOSS4G Conference | Boston, MA 7
• Inquired about key data for faculty
and staff at Tufts
• Also reached out to the Open
Geoportal community
• Created a list of critical data
sources that enable research and
learning at Tufts and beyond
Google Docs
OGP Outreach

8. Methods of Tufts Crawl
12/23/2017 FOSS4G Conference | Boston, MA 8
• Used an FTP program called Filezilla
• Can re-initiate connections after
failure
• Ran on multiple computers
overnight, set to mirror different
FTP sites.

9. Methods of Tufts Crawl
12/23/2017 FOSS4G Conference | Boston, MA 9
•Also completed collection of data
from speed-limited locations by
traditional mail
•Placed 128 GB flash drive in an
envelope
• Caught in a storm, but still much
faster than dial-up speed

10. Summary of Results
12/23/2017 FOSS4G Conference | Boston, MA 10
16
31
40
0
10
20
30
40
50
2008 2012 2017
DataRecused,TB
Year of Harvest
Data Rescue, Estimated Harvest
From Tufts alone – likely much
higher for all data rescues!Source: Phillips, 2016

11. Results of Tufts Crawl
12/23/2017 FOSS4G Conference | Boston, MA 11
Word Cloud
(highest frequency terms)

12. Development of Tufts Crawler
12/23/2017 FOSS4G Conference | Boston, MA 12
•High volume of data – much of it
zipped
• Some further compressed inside zip
files
•Needed a lightweight tool to
assess what data was captured
• Solution  Python script

13. Development of Tufts Crawler
12/23/2017 FOSS4G Conference | Boston, MA 13
•Packaged the Python script in a
GUI using Tkinter
• Object-oriented layer on Tcl/Tk
•Allows for users unfamiliar with
Python to use the tool
•Provides a simple interface and
clear results

14. Results of Tufts Crawler
12/23/2017 FOSS4G Conference | Boston, MA 14
Unzips
files
Records type
of file
Organizes XML
data

15. Using the Tufts Crawler – Take a Look
12/23/2017 FOSS4G Conference | Boston, MA 15

16. Summary & Future Work
12/23/2017 FOSS4G Conference | Boston, MA 16
• Tufts identified federal data
perceived “at-risk”
• Harvested over 40 TB of data, mostly
compressed
• Developed Tufts Crawler to unpack
and categorize types, sizes and other
metadata.
• Future work: pack into .exe, estimate
progress bar

17. Acknowledgements
12/23/2017 FOSS4G Conference | Boston, MA 17
• Tufts Geospatial Team: Carolyn
Talmadge, Chris Barnett, Szuhui Wu,
Annie Swafford, Kristen Lee, Adrian
Sharpe, Patrick Florance.
• Graduate students: Sam Boiler.
• Others: Faculty and members of OGP
who assisted in data selection,
DataRescue Boston, the #DataRefuge
slack channel, all in Bromfield House.

18. Thank you!
Kyle M. Monahan
Statistics & Research Technology Specialist
Tufts University
kyle.monahan@tufts.edu
12/23/2017 FOSS4G Conference | Boston, MA 18
kylemonahan.info datalab.tufts.edu
For more information:

19. Questions?
5 minutes
12/23/2017 FOSS4G Conference | Boston, MA 19

20. Extra Slides – Python Code
12/23/2017 FOSS4G Conference | Boston, MA 20

21. Extra Slides – Python Code
12/23/2017 FOSS4G Conference | Boston, MA 21

22. Extra Slides – Details about tkinter GUI
12/23/2017 FOSS4G Conference | Boston, MA 22