This document provides an overview of cleaning and analyzing messy data using OpenRefine. It discusses profiling data to assess its current state, cleaning data by correcting issues, exploring data by checking, filtering, and cleaning it. It also discusses sorting, faceting, detecting blanks and duplicates, removing blanks and duplicates, performing simple and advanced transformations like clustering and using regular expressions. Resources for learning more about OpenRefine and data cleaning are also provided.

1. DATA
JOURNALISM
Dr. Bahareh Heravi
@Bahareh360
Week 8
Cleaning and Analysing Data

2.
DATA
is
o(en
ugly
&
MESSY

3. Data Profiling
Assess current state of your data.
Data Cleaning
Correct the issues you found during‘data profiling’.

4. Exploring data
Checking data
Filtering data
Cleaning data
Reshaping data
Annotating data
Linking data

5. Dataset
Powerhouse Museum objects collection
Download from:
http://data.freeyourmetadata.org/powerhouse-
museum/phm-collection.tsv
Open Refine and load the dataset.

6. Sorting data

7. Faceting data
To select a subset of your data to work on.
To get useful insight into your data.
To apply a transformation to a subset of your data.

8. Types of Facets
Text facets for text
Numeric facets for number and dates
Predefined/customised facets

9. Text facets
Text facets used for faceting text
Examples:County or city names, TD names

10. Text facets

11. Numeric facets
Numeric facets used for faceting numerical values
and ranges.
Examples:Expenditure,crime rate

12. Numeric facets

13. Detecting blanks

14. Removing blanks

15. Detecting duplicates

16. Removing duplicates
Warning:
If we remove all the original records will also be
removed!

17. Removing duplicates

18. Removing duplicates
Now you can remove.
Facet by blank

19. Congratulations you have removed
all blank and duplicate values.

20. Simple cell transformations

21. Advanced data operations
Clustering
Transformations
Multi-valued cells
Derived columns
Splitting data across columns
Regular Expressions
GREL(General Refine Expression Language)

22. Multi-valued cells
To split a cell in

23. Clustering
To cluster similar (syntactically) items together.
To be used to fix inconsistencies,typos,etc.
Examples in the dataset: Agricultural equipment
Agricultural Equipment
Costume
Costumes

24. Clustering

25. Clustering

26. Transforming cell values

27. Transforming cell valuesGREL
(General
Refine
Expression
Language)

28. Resources
Using OpenRefine by
RubbenVerborgh and Max DeWilde
http://freeyourmetadata.org/cleanup/
Cleaning Data with Refine, School of Data
The Bastard Book of Regular Expressions by Dan Nguyen
GREL:https://github.com/OpenRefine/OpenRefine/wiki/General-Refine-Expression-Language

29.
Ques8ons?
Bahareh
R.
Heravi
@Bahareh360