DMVitals: A Data Management Assessment Recommendations Tool - IASSIST 2012


Published on

Published in: Technology, Business
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Andrew (1 of 6)
  • Andrew (2 of 6)
  • Andrew (3 of 6)
  • Andrew (4 of 6)
  • Andrew (5 of 6)
  • Andrew (6 of 6)
  • Sherry (1 of 5) (start @ around 7 min.)The columns represent the questions from our Data Interview process, Using Data management best practice statementsFrom UVa sources (ISPRO, SciDaC Guidelines) – Information Security, Policy and Records Office (ISPRO) Information Technology Security Risk ManagementANDS long-term sustainability storage modelWe then associated DM practices with research data interview questions & responsesthese data management best practices are listed under them.Using the answers from the interview, then coded “YES”, “NO”, or “NULL” to each corresponding Best Practice.
  • Sherry (2 of 5)Each Best practice statement is then mapped to one of 8 data management categories (FileFormatsDataTypes, Organization of Files, Security StorageBackups, Copyright Priv Confidentiality, Data DocumentationMetadata) Worksheet tabs @ bottomNote that in this current version, we are only using 5 of the management categories (Funding Guidelines, Archiving & Sharing, & Citing Data)And then each practice is given a “weight” for the 5 sustainability levels (Leastsust., fair, satisf, good, more sust)The responses from the Interview sheet are used (linked) to create a ratio of total # Yes (for current best practices) to total possible score.Ranks (subjectively) practices to sustainability levels. This is done for each category. The ratio for each value is then recorded on the Report sheet….
  • Sherry (3 of 5)Top chart has @ DM category and the resultant sustainabilty index (displayed as a % - per ratio)Rank researchers’ current DM practices according to level of “sustainability” and get an Average Sustainability Index(less sustainable, Fair, Satisfactory, Good, more sustainable) based on the ratio of best practices in use vs. total possible best practicesWith 5 levels of sustainability, we divided the ratio values into 5 groupings: 0 – 20%, 21- 40%, 41 – 60%, 61% - 80 %, >81%)Mapped the Avg. of Sustainability Index on the Crowston/Qin Capability Maturity Model for Scientific Data Management.Data management maturity index of Crowston & Qin Capability Maturity Model (CMM) for Scientific Data Management (SDM)--------Along with the score that is generated with a target to improve& includes actionable recommendations – Those practices not being done, are marked with “X”. And include action statements on how to improve. DM consultants then sorts the action statements by phases. – customizable, to help researchers get things done, move some actions to later phases
  • Sherry (4 of 5)General information. Sustainability Chart and then the action statements grouped into phases.Phase 1 (short-term)Phase 2 (long-term)Phase 3 (future)Once the report is created on the DMVitals (spreadsheet)…. We thenCreate report with their grade of DM sustainability and list of tasks divided into implementation phases. We then sit down with the researcher go over the recommendations and make adjustments on what actions are done in each phase. It’s the start of our Data Management Implementation.Action statements in each phase are tweaked as needed. The default gives a relative sense of sustainability & what actin to do. But can be customizable.
  • Sherry (5 of 5) finish by 12 min. (15 @ max)Just to recap how we use the DMVitals to create DM Recommendations from our Assessments.In this step you could add your institution’s policies and other best practices local to you.Even the “ranking” of sustainability can be adjusted per discipline, or institution. – where we put the best practices in columns – from Least sustain… to more sustain…Action statements definitely will require local customizations, it’s the actions that your researchers need to do for your institution. – they can include naming specific groups to go to get help.As I said at UVa, we meet with the researcher and customize the recommendations. Finally the report that you create is slightly different for each researcher based on their time and needs.
  • Andrew (1 of 3)Helps with identifying gaps in domain knowledge and/or skill areas (in which topical areas are people weakest, is it limited to certain domains, etc.). Very useful for targeted training and promotion of services and software/tools.
  • Andrew (2 of 3)
  • Andrew (3 of 3)
  • DMVitals: A Data Management Assessment Recommendations Tool - IASSIST 2012

    1. 1. DMVitals:A Data Management Assessment Recommendations Tool Andrew Sallans, Head of Strategic Data Initiatives Sherry Lake, Senior Scientific Data Consultant IASSIST 2012 - June 6, 2012
    2. 2. Interviews/Assessment Preface• Over past two years we conducted about 25 data interviews – Focus on learning about research data practices at UVa and identifying service needs/opportunities – Intention of leading into consulting opportunities• Ended up with conundrum of how to manage “unique” conditions of each research environment against common characteristics of data management within domains and institutional framework 2
    3. 3. Consulting Workflow Distribute final report Send initial report to and begin DMConduct Data researcher for Implementation with Interview approval/review Researcher Produce "Data Code Data Extract action Interview Report" Interview statements from answers in the “DM Vitals “DM Vitals” tool Recommendations Report" 3
    4. 4. Recommendation Requirements• Must be a fast process• Must create actionable and repeatable recommendations• Must reduce subjectivity• Must weigh all assessment factors• Must address present DM condition while showing path for improvement 4
    5. 5. Components of the DMVitals• Data management best practice statements – UVa sources (ISPRO, SciDaC Guidelines) – ANDS long-term sustainability scoring model• 8 data management categories• Data interview questions and responses• Data management maturity index of Crowston & Qin Capability Maturity Model (CMM) for Scientific Data Management (SDM) 5
    6. 6. Crowston & Qin Capability Maturity Model for SDM Crowston, K. & J. Qin. (2010). A capability maturity model for scientific data management. In: Proceedings of the American Society for Information Science and Technology, October 24-26, 2010, Pittsburgh, PA. (Poster) 6
    7. 7. DMVitals Interview Sheet 7
    8. 8. DMVitals Section View 8
    9. 9. DMVitals Report View 9
    10. 10. Consulting Recommendations View 10
    11. 11. DMVitals Workflow Recap Rank Create report researchers’ Create “action” with a grade forAssociate DM current DM statements for sustainabilitypractices with practices researchers and list of tasksresearch data according to that correlate divided into interview level of with each level implementation “sustainability” phases 11
    12. 12. DMVitals for Aggregate Learning Research Data Management Sustainability (Example - not real data) 70% 60%% of Sustainability Guidelines Met 50% 40% Total 30% 20% 10% 0% STS ASTRO BIO BME CHEME CHEM CIVE CS PATH MOLPHYS CELLBIO 12
    13. 13. Major Challenges1. Assessment tool design, specifically dealing with appropriate weighting, false positives, double negatives2. Social/ethical implications of giving such focused feedback and criticism to researchers3. Broader issue of motivations and incentives 13
    14. 14. DMVitals Next Steps• Release plan for others to use – Starting to develop versions of package – Will begin to make stable releases available on our website, along with development roadmap• Collaboration opportunities for expansion – Interested in collaborations to drive further development and integration into services – Seeking collaborators now! 14
    15. 15. References• Australian National Data Service. (2011) ANDS and Data Storage. Available: Last accessed May 30, 2012.• Crowston, K., & Qin J. (2010). A capability maturity model for scientific data management. American Society for Information Science and Technology Annual Meeting. Pittsburg, PA. Working Paper available: management-0. Last accessed May 23, 2012.• Digital Curation Center. (2011). CARDIO. Available: Last accessed May 30, 2012.• Information Technology Security (2010). University of Virginia Information Technology Security Risk Management (ITS-RM) Program. Available: Last accessed May 23, 2012.• University of Virginia Library (2011). Scientific Data Consulting Data Management Home. Web Site available: Last access May 30, 2012. 15
    16. 16. Acknowledgements and Contact Information• Acknowledgements – Susan Borda • UVa SciDaC Intern Summer 2011 • Recent graduate of Syracuse GSLIS program • Starting as Digital Curation Librarian at University of California – Merced this summer• Contact Information – Andrew Sallans, – Sherry Lake, 16