Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Beautiful Research Data (Structured Data and Open Refine)


Published on

"Unlike traditional academic conferences, the Roots & Routes Summer Institute features a combination of informal presentations, seminar-style discussions of shared materials, hands-on workshops on a variety of digital tools, and small-group project development sessions. The institute welcomes participants from a range of disciplines with an interest in engaging with digital scholarship; technical experience is not a requirement. Graduate students (MA and PhD), postdoctoral fellows and faculty are all encouraged to apply."

Published in: Software, Technology
  • Be the first to comment

Beautiful Research Data (Structured Data and Open Refine)

  1. 1. Beautiful Research Data Kirsta Stapelfeldt, Coordinator UTSC Library’s Digital Scholarship Unit
  2. 2. In this presentation ● Part One: preparing to create machine- readable data at the onset of a research endeavour ● Part Two: Working with “messy,” datasets
  3. 3. Benefits of machine-readable data ● Easier to query for new insights ● Easier to mount in a computing environment ● Easier to share with others
  4. 4. Just a .csv + Fusion Tables ● Fusion tables is an experimental, web-based chrome app ● Took a spreadsheet that Natalie has been working on and loaded it into the app ● Results have not been massaged at all ● We can expect additional benefits from having structured data in the future
  5. 5. Part one In which you have no research data...yet
  6. 6. Best Case Scenario You start by utilizing some best practices 4 Pieces of low-hanging fruit...
  7. 7. 1. No word documents ● database (even a spreadsheet) not .docs ● avoid a lot of style information in your research documents (such as bolding and italicizing text, or moving things to other areas of the page using the tab key or spacebar) ● Why?
  8. 8. Look beyond the surface. & n &nsbp; &nsbp; &nsbp; &nsbp; no thank you!
  9. 9. Beauty is more than browser deep
  10. 10. 2. Use consistent formats for elements such as date & language ● i.e. dates recorded consistently where possible (05/25/2014)
  11. 11. 3. Taxonomies & Standards ● use controlled vocabularies for keywords, place names, person names of relevance o using an open format for a place name can make geocoding much easier o stay consistent in a given language
  12. 12. 4. Text Encoding ● Ensure you are using Unicode (UTF-8) ● How do you know ? o Notepad can be your friend o Test a sample between systems
  13. 13.
  14. 14. Changing the way you think about your research process Draw a picture
  15. 15. 1. Think small. Atomistic information (what is the smallest meaningful unit of information you are collecting?) For example: ● A person’s name, religion, and DOB ● Mention of a location or name ● Repeated occurrence
  16. 16. 2. Connect the dots. What are the relationships between your data elements? Useful tool: The Entity Relationship Diagram
  17. 17. Draft Dragomans Content Model
  18. 18. Crow’s Foot Notation Exercise - Building an ERD
  19. 19. Part two Your data is a mess
  20. 20. Tools for dealing with messy data ● Regular Expressions ● Open Refine
  21. 21. Regular Expressions: Find & Replace on Steroids ● Available in most productivity suites (iWork, Microsoft Word, Libre Office/Open Office) ● Often syntax is a little different across systems
  22. 22. “The regular expression (?<=.) {2,}(?=[A-Z]) matches at least two spaces occurring after period (.) and before an upper case letter as highlighted in the text above.”
  23. 23. Open Refine ● Similar to spreadsheet software ● Installed on your computer, but used through your browser ● “Power Tool” for messy data Following will draw heavily from this lesson - with-openrefine (Thanks to Seth van Hooland, Ruben Verborgh, Max De Wilde)
  24. 24. Base Assumption of Open Refine ● You have “structured data” ● some consistent and machine-readable logic has been applied to your data o Excel, .csv, XML ● you may have structured data and not know it o Check export options from any software you regularly use
  25. 25. 1. Remove duplicates 2. Remove blanks 3. Make data atomistic (smallest meaningful unit) 4. Keep terms/formats consistent
  26. 26. powerhouse-museum/phm- Choose file & select Next...
  27. 27. Set appropriate options and “Create Project”
  28. 28. Project is created with 75,814 rows.
  29. 29. 1. Look for Blank Records See if any RecordIDs are blank by using a numeric facet
  30. 30. “Non-numeric” rows are blank.
  31. 31. Hovering over the cell makes an “edit” link visible
  32. 32. The “blank” fields actually contained a single whitespace. You can delete the whitespace and then select “Apply to All Identical Cells” -
  33. 33. A confirmation message will always show up noting what you’ve done, and giving you a chance to “undo”
  34. 34. 2. Look for Duplicate Records using Record ID (since it should be unique)
  35. 35. Sorting is a visual tool only unless you “Reorder rows permanently”
  36. 36. “Blank down” will delete the second instance of a duplicated “Record ID”
  37. 37. Then, we can facet the “Record ID” column by blank records.
  38. 38. the “true” facet contains all the blank records.
  39. 39. Clicking the “true” link will narrow to the blank records, which can then be removed.
  40. 40. 3. Make data atomistic “Category” contains numerous categories separated by the “|” character
  41. 41. You can tell the system to split the cells using this character.
  42. 42. Now only single categories appear.
  43. 43. Creating a text facet on “Categories” brings up all the options in this column. We can “cluster” to detect similar terms that might have variances in spelling or capitalization 4. Make terms consistent
  44. 44. This interface allows you to select which term is authoritative. You can then merge terms together.
  45. 45. a couple of additional features... The “Undo/Redo” tab allows you to back up in steps to the creation of your project, if you make a mistake.
  46. 46. A “text filter” can allow you to search in a column (by regular expression too!)
  47. 47. Refine has its own set of regular expressions that can be used to perform functions on data.
  48. 48. Functions A full list of these is available on Github.
  49. 49. Finally, projects can be exported as Refine projects, but also in a number of additional structured formats. Do this frequently.
  50. 50. Structured data is beautiful data. Make a plan to create structured data during your research Clean legacy data or data you inherit, by becoming a regular expression (regex) expert and/or using a tool like OpenRefine.
  51. 51. Go to your library or ITS department to see if you can get support. Thanks for listening to me!