Download to read offline
![We’re here to help http://archaeologydataservice.ac.uk/ http://guides.archaeologydataservice.ac.uk/ [email_address]](https://image.slidesharecdn.com/cshmanagarchdata-110321115651-phpapp01/85/Managing-large-and-complex-data-sets-34-320.jpg)
More Related Content
Similar to Managing large and complex data sets
Managing large and complex data sets
- 1. Managing large andcomplex data sets: … THE CHALLENGES OF ARCHIVING AND ONLINE DELIVERY CATHERINE HARDMAN
- 2. The problem….in 1996My lithics report here, on floppy disc
- 3. The ADS: someancient history The Archaeology Data Service: set up in 1996 one of five AHDS subject centres based within the University of York Funding: initially received funding from Arts and Humanities Research Council (AHRC) Joint Information Systems Committee (JISC) Presently receives core funding from AHRC alongside cross-sectoral, project-based funding.
- 4. What do wedo? Our remit: “ To support research, learning and teaching with high quality and dependable digital resources.” In practice this means three key things: That ADS collect and preserve datasets That we allow full, easy and free access to these And that we additionally provide guidance and support to data creators
- 5. No need fordigital preservation Domesday Book: Publisher: William of Normandy (1086) – still readable
- 6. Where’s preservation whenyou need it? Domesday Disc: Publisher: BBC (1986) –nearly lost
- 7. Why is itimportant?
- 8. Michener, W.K., Brunt,J.W., Helly, J.J., Kirchner, T.B. and Stafford, S.G. 1997. Nongeospatial Metadata for the Ecological Sciences. Ecological Applications. 7: 330-342. What’s the problem? Information Entropy
- 9. The scale ofthe problem in the 1990s Strategies for protecting physical media Findings and Recommendations from ‘Digital Data in Archaeology: A Survey of User Needs’ Condron et al 1999
- 10. Protecting Physical media… never the twain
- 11. The scale ofthe problem in the 1990s The popularity of storage options Findings and Recommendations from ‘Digital Data in Archaeology: A Survey of User Needs’ Condron et al 1999
- 12. 8" Floppy 3.5"Floppy 5.25" Floppy 12" Optical Disk 5.25" Optical Disk CD-ROM Sparq Disk Cartridge Zip Disk Click! DVD-ROM Jaz Disk Floptical Disk Punch Tape Rectangular Hole Punch Card IBM 3480 DLT Tape DG90M Tape DC4_120 8mmD-eight QIC DC600 G2000 Tape 4mm Tape Ditto Max 9-Track Ree l Cassette tape Memory Stick MultiMedia Card SD Memory Card xD Picture Card Smart Media CompactFlash Travan
- 13. Why is itall so difficult? Deterioration of the storage medium Obsolescence of the storage medium Failure to document the format adequately Obsolescence of the software Obsolescence of the hardware Long-term management
- 14. How do wedo it? Open Archival Information System (OAIS)
- 15.
- 16. Migration based approach& controlled ingest Aim to connect with data producers early on in their project lifecycles to ensure that preservation planning is a key consideration during the project rather than an afterthought.
- 17. Guides to helpyou do all that.
- 18. It hasn’t reallygot much easier The goal posts keep moving!
- 19. The size ofdigital archives held by different types of archaeological bodies http://ads.ahds.ac.uk/ A rchaeology D ata S ervice
- 20. Big Data ProjectRoughly how much data would be generated by a single project?
- 21. Which of thesedata collection techniques do you carry out? Technologies used 12% 4% 4% 3% 8% 1% 3% 11% 9% 9% 7% 14% 3% 12% 3D Laser Scanning Sidescan Sonar Multibeam Scanning Single Beam Scanning Geophysics Acoustic Tracking Sub bottom profiling Geographic (eg GIS) Lidar Digital Video Video Movie Clips Still Images CAD (2D or 3D) Other
- 22. What are themain software packages you use ?
- 23. Do you havean archiving policy for the data sets / types in question?
- 24.
- 25. When you starta new project …would you consider using existing datasets?
- 26. This is theopportunity!
- 27.
- 28. Making the inaccessibleaccessible to make available unpublished fieldwork reports in an easily retrievable fashion. There are currently 8018 reports available and this number is increasing steadily through the OASIS project in England and Scotland.
- 29. Blurring the distinction… … between publication and archives …
- 30.
- 31.
- 32. What does thatmean for you? Plan for reuse Plan for reuse Plan for reuse Plan for reuse
- 33. How do youdo that? Include a data management plan (use the DCCs) Order your data File naming strategy Version control Back-up (in the field) Consider your file formats Dissemination plan (and it’s longevity) What does the long term look like? Discuss requirements with an archive
- 34. We’re here tohelp http://archaeologydataservice.ac.uk/ http://guides.archaeologydataservice.ac.uk/ [email_address]
Editor's Notes
- #21 How big is your data? – asked in order to get a idea of scale of the problem So you’ll see there is some quite big data being produced out there – some people producing over 200GB for a project
- #22 We ran an online questionnaire to find out about users and uses of big data – I’ll just skim through some of the things that came out of it: We got 48 responses. this is one of the first questions we asked. Wanted to get an idea of the data collection techniques that people are using to create big data. You’ll see there’s a wide range of technologies including the ones I mentioned on an earlier slide.
- #23 Of the 101 software packages entered into the online form a staggering 52 are unique (that is after editing for things like lower and upper case character differences). It seems the world of ‘big data’ is very fragmented.
- #24 This is an interesting one. We asked if people had an archival policy for the data sets in question. Only 48% of respondents note that they have a policy in place Of these many noted that these policies were localised and incomplete - not formal written policy. A proper system of digital archiving should involve continuous active management of the data, putting data on a dvd and putting it in a drawer is not really a stable archival policy. A formal archival policy as we see it should ideally be based on the OAIS system – continuous active management of data to ensure its survival into the future.
- #26 Overwhelming “yes” to this question.... Some of the reasons that were cited: monitoring over time avoiding duplication Saving time/money Of course – re-use just isn’t possible unless someone is archiving and providing access to this data