Tutorial data management presentation from the 2016 AGU/ASLO/TOS Ocean Sciences Meeting; focused on making long-tail data discover-able.

1. Moving Beyond Planning to
Implementation: Open-Source
Tools…
Josh Young
Ocean Sciences Meeting
February 24, 2016

2. Who is Unidata?

3. Why at Ocean Sciences?

4. Scope
Imagine a project:
• that includes a well-thought out and
documented data management plan,
• and robust implementation of that
plan through out the project and
beyond.
• This talk is not for that project; it is for
the rest of us.

5. So why do we care about data
management?
• Internal reasons: do good research,
write papers, get tenure, win more
grants.
• External reasons: public access &
reproducibility
 Risk of becoming dark data (Heidorn,
2008)

6. Why care about external access?
• Intangibles for an Investigator
• Maybe someday I’ll benefit from someone else’s
data
• Maybe I’ll learn something through informal dialogue
• Most science funding is from public resources and
should/could be considered a public trust resource
• Peer pressure
• Tangibles for an Investigator
• Increased efficiency
• My funders require it.

7. So why do we care about data
management?
• Internal reasons: do good research,
write papers, get tenure, win more
grants.
• External reasons: greater impact

8. WorkflowsInternal

9. Public-Access Workflows

10. What is the DMRC & do we really
need another Data Plan Project?
• Probably not
• The DMRC is not a Data Plan tool
• Unidata community requested help
with implementation
• Therefore, the DMRC is primarily a
curated list of tools for implementation

11. The DMRC

12. What the DMRC Offers
• Highlights requirements from funding
agencies;
• Points to Best Practices developed by
others in the Data Management
space;
• Sorts available tools by best practice;
• Details available tools.

13. Requirements
• Highlight data management funding
requirements from NASA, NOAA,
NSF
• These are the agencies that fund our
community so we try to stay up to
date, but remember the agency
posted information is always the
authority

14. Activity Best Practices
& Possible Tools
Activity column based on DataOne Best Practices

15. The DMRC Points to Tools

16. The DMRC Points to Tools

17. The DMRC Points to Tools

18. The DMRC Points to Tools

19. The DMRC Points to Tools

20. The DMRC Explains the LDM

21. The DMRC Explains the TDS

22. The DMRC Explains RAMADDA

23. What We Are Exploring
• Dataverse by Harvard
• Designed for sharing, archiving, and
citing data
• Allows you to create a DOI
• Allows you to store and make data
accessible in perpetuity

24. What We Are Exploring
Known Dataverse Characteristics:
• Largest single file limited to 10GB
• No limit to number of files
• Users create their own Dataverse
• Designate private or public
• Open to data from all science disciplines
• Does not corrupt at least some software
files (e.g. IDV bundles)
• FREE

25. What We Are Exploring
Possible Dataverse Contributions:
• Description (providing DOIs)
• Sharing (access for perpetuity)
• Preservation (static copy for perpetuity)
• Cost (free) very suitable for projects that
might otherwise become long-tail data

26. Activity Best Practices
& Possible Tools
Activity column based on DataOne Best Practices

27. Open Source Access to Code

28. We Welcome Your Resource
Suggestions!
• Please visit:
http://goo.gl/forms/Ngp4Xu9nGr

29. Example Workflow Implementation
• Radar and Lidar data from the
University of Wyoming King Air
• Millersville University Plains Elevated
Convection at Night (PECAN) data
• North Carolina State University WRF
North Atlantic Model Outputs

30. Part of a larger effort: Agile Data
Curation
• Means taking implementable steps to
improve data management for
external access.
• Philosophically, it attempts to apply
lessons from agile software
development to data management.

31. Agile Curation Principles,
2nd Generation
(J.Young, K.Benedict, & C. Lenhardt, AGU 2015 Fall Meeting)
1) Delivery, access, use and citation of
research data are the primary measures of
success.
2) Maximize the impact of research data
through the continuous integration of
curation activities
3) Support unanticipated needs for and uses
of research data (and documentation) and
develop flexible systems to capture new
uses.

32. Agile Curation Principles,
2nd Generation
4) Make data open and accessible as early in the
process as possible.
5) Encourage crowd-sourced / community
feedback to improve and enhance the data.
Provide basic metadata for data available early
in the process even if the data are not finalized.
6) Identify key individuals in a research project
that have the requisite motivation, knowledge,
or ability to learn and get out of their way.

33. Agile Curation Principles,
2nd Generation continued
7) Data creators and data curators should work
closely throughout the data life story to ensure
the most efficient and streamlined process.
8) Identify the most effective method(s) for
maintaining close communication between the
data creators and curators involved and use
them.
9) Target the steady delivery of incremental
improvements to research data discovery,
access and use that is consistent with a
sustainable level of effort and available funding.

34. Agile Curation Principles,
2nd Generation continued
10) Start with the basics and only make systems
more complex as needed, while maintaining a
low bar to entry.
11)Continuous attention to technical excellence
and good design enhances agility.
12)Continuously develop a community of data
providers, curators and users that participate in
the evolution of the research data systems.

35. We Welcome Your Stories
• Please email: jwyoung@ucar.edu

36. Balancing infrastructure development & scientific
advancement to create sustainable, multidisciplinary
solutions
M. Chan
• Advance science
• Meet grand challenges
• Leverage shared
cyberinfrastructure
technology
NSF’s EarthCube
Cyber
Infrastructure
Science
RCNs
Building
Blocks
Interactive
Activities
End User
Workshops
EC
Committees
GOALS

37. Get Involved!
Science
Committee
Technology &
Architecture
Committee
Liaison
Team
LEADERSHIP
COUNCIL
Office
Council of
Data
Facilities
Engagement
Team
• Talk to EarthCube
Participants!
• Attend EarthCube
Workshops!
• Join the mailing list at
earthcube.org
• Apply for funding (EC Travel
Grants, Distinguished
Lecturers)
• Follow on twitter @earthcube

38. Unidata is one of the University Corporation
for Atmospheric Research (UCAR)'s
Community Programs (UCP), and is
funded primarily by the National Science
Foundation (Grant NSF-1344155).