Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Managing provenance in the Social Sciences: the Data Documentation Initiative (DDI)


Published on

Slides from webinar: Provenance and social science data. Presented on 15 March 2017. Presenter was Dr Steve McEachern, Director Australian Data Archive

FULL webinar recording:

1. Dr Steve McEachern (Director, Aust Data Archive) Data Documentation Initiative (DDI: A free, international standard for describing data produced by surveys and other observational methods in the social, behavioral, economic, and health sciences. It can document and manage different stages in the research data lifecycle, eg conceptualization, collection, processing, distribution, discovery, and archiving. Documenting data with DDI facilitates understanding, interpretation, and use -- by people, software systems, and computer networks.

Published in: Data & Analytics
  • Did you try ⇒ ⇐?. They know how to do an amazing essay, research papers or dissertations.
    Are you sure you want to  Yes  No
    Your message goes here
  • My brother found Custom Writing Service ⇒ ⇐ and ordered a couple of works. Their customer service is outstanding, never left a query unanswered.
    Are you sure you want to  Yes  No
    Your message goes here

Managing provenance in the Social Sciences: the Data Documentation Initiative (DDI)

  1. 1. Managing provenance in the Social Sciences: The Data Documentation Initiative (DDI) Dr. Steve McEachern Director, Australian Data Archive
  2. 2. ADA in Brief • The Social Science Data Archive (now ADA) was set up in 1981, housed in the Research School of Social Sciences, with a mission to collect and preserve Australian social science data on behalf of the social science research community • The Archive holds over 5000 datasets from around 1500 studies, including national election studies; public opinion polls; social attitudes surveys, censuses, aggregate statistics, administrative data and many other sources. • Data holdings are sourced from academic, government and private sectors.
  3. 3. So what is a data archive? • ‘A “trusted system” that provides... an accessible and comprehensive service empowering researchers to locate, request, retrieve and use data resources in a simple, seamless and cost effective way, while at the same time protecting the privacy, confidentiality and intellectual property rights of those involved.’ Social Sciences and Humanities Research Council of Canada. “National Data Archive Consultation Final Report: Building Infrastructure for Access to and Preservation of Research Data in Canada” URL: [20 November 2003].
  4. 4. The Data Documentation Initiative standard
  5. 5. About DDI • A structured metadata specification of and for the community • Two major development lines – XML Schemas – DDI Codebook – DDI Lifecycle • Additional specifications: – Controlled vocabularies – RDF vocabularies for use with Linked Data • Model based version is in development – with serialisations in XML and RDF – Includes support for provenance and process models • Managed by the DDI Alliance –
  6. 6. DDI-Codebook • XML based, first published in 2000 • Four sections: 1. Document description: characteristics of the DDI XML document itself 2. Study description: characteristics of the Study (project) that the DDI is describing (including Related Materials: documents associated with the project, such as questionnaires, codebooks, etc.) 3. File description: characteristics of the physical data files 4. Variable description: characteristics of the variables in the data file
  7. 7. DDI Lifecycle Model S03 7 Metadata Reuse
  8. 8. Why can DDI Lifecycle do more? • It is machine-actionable – not just documentary • It’s more complex with a tighter structure • It manages metadata objects through a structured identification and reference system that allows sharing between organizations • It has greater support for related standards • Reuse of metadata within the lifecycle of a study and between studies S05 8
  9. 9. DDI Lifecycle Features • Support for CAI instruments • Support for longitudinal surveys • Focus on comparison, both by design and after-the-fact (harmonization) • Robust record and file linkages for complex data files • Support for geographic content (shape and boundary files) • Capability for registries and question banks
  10. 10. Provenance in DDI
  11. 11. DDI Codebook • Human readable provenance • Studies: – Attribution – Methodology – Data processing, collection, etc. – Related materials: questionnaires, technical reports, … • Variables: – Variable name, values, labels, type – Question text – Notes
  12. 12. DDI Lifecycle • Machine actionable provenance • Studies: – Attribution – Methodology – Data processing, collection, etc. – Related materials: questionnaires, technical reports, … • Variables: – Questions – Variables – Code lists – Universes (i.e. population) – All are maintainable and re-usable – Allows provenance of concepts across studies
  13. 13. DDI 4 (a.k.a. Views) • Machine actionable provenance • Variable hierarchy: – Conceptual Variable – Represented Variable – Instance Variable – Each inherits from the level above • Management of codes and categories across the lifecycle – E.g. management of a set of missing values • Management and transformation of individual datum(s) – Process model for data transformation and validation
  14. 14. Managing and Depositing Data: ADA and DDI
  15. 15. Approach • Core archive website: – • Sub-archives focussed on specialised thematic or methodological areas - eg. • “Add-on” systems for complex analysis or visualisation tasks: – Nesstar – GIS: – Longitudinal visualisation: Panemalia – Historical census data:
  16. 16. OAIS architecture
  17. 17. Data deposit: ADAPT
  18. 18. Archival processing Manual system with some automation tools 1. Deposit: – Review of ADAPT submission – Storage via ADAPT to file store 2. Data processing: – File format conversion (usually to SPSS for processing) – Privacy/confidentiality review – Data cleaning (in consultation with depositor) 3. Metadata processing: – DDI-C metadata creation in Nesstar Publisher 4. Publishing: – Archival storage and access format creation – Data publication to Nesstar server – Metadata publication to Nesstar and ADA CMS
  19. 19. The ADA study page Study information is available through the tabs at the top of the study: • Study: information including the investigators, abstract, sample, data collection methods, and access requirements. • Variables: a list of variables available in a quantitative dataset • Related Materials: additional documentation, links and other related studies (eg. others in the series) that may interest you The study page is also the access point for the ADA Nesstar system, for: • Analysis of quantitative data online, • Download of data to your own computer.
  20. 20. The ADA Study Page
  21. 21. Questions? Steven McEachern