This document discusses key questions to consider regarding data for research projects that utilize large datasets. It outlines questions about the suitability, type, source, size, format, cleaning requirements, quality, handling of missing data, and evaluation of the data analysis. Specific topics covered include determining if the data is appropriately labeled for the research questions, common data types, sources for finding datasets, reasonable dataset sizes, common data formats, methods for cleaning and assessing data quality, and techniques for evaluating the analysis outcomes.

1. Some Key Questions
about you Data
Brian Mac Namee
Brendan Tierney
Damian Gordon

2. The Data
 If the data is the key consideration in your research
(although not all projects will necessarily be
concerned with large datasets) it is important to
consider several questions for those projects that
do.

3. Overview
 How suitable is the data?
 What is the type of the data?
 Where will you get it from?
 What size is the dataset?
 What format is it in?
 How much cleaning is required?
 What is the quality of the data?
 How do you deal with missing data?
 How will you evaluate your analysis?
 etc.

4. Suitability: Dataset
 Determining the suitability of the data is a vital
consideration, it is not sufficient to simply locate a
dataset that is thematically linked to your research
question, it must be appropriate to explore the
questions that you want to ask.
 For example, just because you want to do Credit
Card Fraud detection and you have a dataset that
contains Credit Card transactions or was used in
another Credit Card Fraud project, does not mean
that it will be suitable for your project.

5. Suitability: Labelling
 Is the data already labelled?
 This is very important for supervised learning
problems.
 To take the credit card fraud example again, you
can probably get as many credit card transactions
as you like but you probably won't be able to get
them marked up as fraudulent and non-fraudulent.

6. Suitability: Labelling
 The same thing goes for a lot of text analytics
problems - can you get people to label thousands of
documents as being interesting or non-interesting to
them so that you can train a predictive model?
 The availability of labelled data is a key
consideration for any supervised learning problem.
 The areas of semi-supervised learning and active
learning try to address this problem and have some
very interesting open research questions.

7. Suitability: Labelling
 Two important considerations:
 The Curse of Dimensionality – When the dimensionality
increases, the volume of the space increases so fast that
the available data becomes sparse. In order to obtain a
statistically sound result, the amount of data you need
often grows exponentially with the dimensionality.
 The No Free Lunch Theorem - Classifier performance
depends greatly on the characteristics of the data to be
classified. There is no single classifier that works best on
all given problems.

8. Suitability: Labelling
 Also remember for labelling, you might be aiming
for one of three goals:
 Binary classifications – classifying each data item to one
of two categories.
 Multiclass classifications - classifying each data item to
more than two categories.
 Multi-label classifications - classifying each data item to
multiple target labels.

9. Types of Data
 Federated data
 High dimensional data
 Descriptive data
 Longitudinal data
 Streaming data
 Web (scraped) data
 Numeric vs. categorical vs. text data
 etc.

10. Locating Datasets
 http://researchmethodsdataanalysis.blogsp
 e.g.
 http://www.kdnuggets.com/datasets/
 http://www.google.com/publicdata/directory
 http://opendata.ie/
 http://lib.stat.cmu.edu/datasets/

11. Size of the Dataset
 What is a reasonable size of a dataset?
 Obviously it vary a lot from problem to problem, but
in general we would recommend at least 10
features (columns) in the dataset, and we’d like to
see thousands of instances.

12. Format of the Data
 TXT (Text file)
 MIME (Multipurpose Internet Mail Extensions)
 XML (Extensible Markup Language)
 CSV (Comma-Separated Values)
 ACSII (American Standard Code for Information
Interchange)
 etc.

13. Cleaning of Data
 Parsing
 Correcting
 Standardizing
 Matching
 Consolidating

14. Quality of the Data
 Frequency counts
 Descriptive statistics (mean, standard deviation,
median)
 Normality (skewness, kurtosis, frequency
histograms, normal probability plots)
 Associations (correlations, scatter plots)

15. Missing Data?
 Imputation
 Partial imputation
 Partial deletion
 Full analysis
 Also consider database nullology

16. Evaluating the Analysis
 How confident are you in the outcomes of your
analysis?
 Area under the Curve
 Misclassification Error
 Confusion Matrix
 N-fold Cross Validation
 Test predictions using the real-world

17. The Data
 Other questions?