06 Data

Stat405 Data

Hadley Wickham
Monday, 14 September 2009

1. Group work
2. Motivating problem
3. Loading & saving data
4. Factors & characters


Group project
Want to help your groups become
effective teams.
We’ll spend 15 minutes getting you into
teams, and establishing expectations.
See handouts.
Final project weighting for team
citizenship.


Firing & Quitting
You may ﬁre a non-participating team
member, but you need to meet with me
and issue a written warning.
If you feel that you are doing all the work
in your team, you may quit. You’ll also
need to meet with me and give a written
warning to the rest of your team.


State regulated payoffs: how can be
sure they’re honest? CC by-nc-nd: http://www.ﬂickr.com/photos/amoleji/2979221622/


Where are we going?
In the next few weeks we will be
focussing our attention on some slot
machine data. We want to ﬁgure out if
the slot machine is paying out at the rate
the manufacturer claims.
To do this, we’ll need to learn more about
data formats and how to write functions.


Loading data
read.table(): white space separated
read.table(sep="t"): tab separated
read.csv(): comma separated
read.fwf(): ﬁxed width
load(): R binary format
All take file argument


Why csv?

Simple.
Compatible with all statistics software.
Human readable (in 20 years time you will
still be able to extract data from it).


Your turn
Download baseball and slots csv files from
website. Practice using read.csv() to
load into R.
Guess the name of the function you might
use to write the R object back to a csv file
on disk. Practice using it.
What happens if you read in a file you
wrote with this method?


batting <- read.csv("batting.csv")
players <- read.csv("players.csv")
slots <- read.csv("slots.csv")

write.csv(slots, "slots-2.csv")
slots2 <- read.csv("slots-2.csv")
str(slots)
str(slots2)

# Better
write.table(slots, file = "slots-3.csv",
sep=",", row = F)
slots3 <- read.csv("slots-3.csv")


Working directory
Remember to set your working directory.
From the terminal (linux or mac): the
working directory is the directory you’re in
when you start R
On windows: setwd(choose.dir())
On the mac: ⌘-D


Saving data

# For long-term
write.table(slots, file = "slots-3.csv",
sep=",", row = F)

# For short-term caching
save(slots, file = "slots.rdata")


.csv .rdata

read.csv() load()
write.table(sep = ",",
row = F) save()

Only data frames Any R object
Can be read by any
program
Only by R
Short term caching of
Long term expensive computations


Cleaning
I cleaned up slots.csv for you to practice
with. The original data was slots.txt.
Your next task is to performing the
cleaning yourself.
This should always be the first step in an
analysis: ensure that your data is available
as a clean csv file. Do this in once in a
file called clean.r.


Your turn

Take two minutes to ﬁnd as many
differences as possible between
slots.txt and slots.csv.
What did I do to clean up the ﬁle?


Cleaning

• Convert from space delimited to csv
• Add variable names
• Convert uninformative numbers to
informative labels


Variable names
names(slots)
names(slots) <- c("w1", "w2", "w3",
"prize", "night")
dput(names(slots))

This is a general pattern we’ll see a lot of


Factors
• R’s way of storing categorical data
• Have ordered levels() which:
• Control order on plots and in table()
• Are preserved across subsets
• Affect contrasts in linear models


# Creating a factor
x <- sample(5, 20, rep = T)
a <- factor(x)
b <- factor(x, levels = 1:10)
c <- factor(x, labels = letters[1:5])

levels(a); levels(b); levels(c)
table(a); table(b); table(c)


# Subsets
b2 <- b[1:5]
levels(b2)
table(b2)

# Remove extra levels
b2[, drop=T]
factor(b2)

# Convert to character
b3 <- as.character(b)
table(b3)
table(b3[1:5])


as.numeric(a)
as.numeric(b)
as.numeric(c)

d <- factor(x, labels = 2^(1:5))
as.numeric(d)
as.character(d)
as.numeric(as.character(d))


Character vs. factor
Characters don’t remember all levels.
Tables of characters always ordered
alphabetically
By default, strings converted to factors
when loading data frames.
Use stringsAsFactors = F to turn off for
one data frame, or
options(stringsAsFactors = F)


Character vs. factor

Use a factor when there is a well-deﬁned
set of all possible values.
Use a character vector when there are
potentially inﬁnite possibilities.


Quiz
Take one minute to decide which data
type is most appropriate for each of the
following variables collected in a medical
experiment:
Subject id, name, treatment, sex,
address, race, eye colour, birth city, birth
state.


Your turn
Convert w1, w2 and w3 to 0 Blank (0)
factors with labels from 1 Single Bar (B)
adjacent table 2 Double Bar (BB)
Rearrange levels in terms 3 Triple Bar (BBB)
of value: DD, 7, BBB, BB, 5 Double Diamond (DD)
B, C, 0
6 Cherries (C)
Save as a csv ﬁle
7 Seven (7)
Read in and look at levels.
Compare to input with
stringsAsFactors = F


slots <- read.table("slots.txt")
names(slots) <- c("w1", "w2", "w3", "prize", "night")

levels <- c(0, 1, 2, 3, 5, 6, 7)
labels <- c("0", "B", "BB", "BBB", "DD", "C", "7")

slots$w1 <- factor(slots$w1, levels = levels, labels = labels)

write.table(slots, "slots.csv", sep=",", row=F)


06 Data

More Related Content

Viewers also liked

Similar to 06 Data

More from Hadley Wickham

Recently uploaded

06 Data