12 adv-manip

  • 1,319 views
Uploaded on

 

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
1,319
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
56
Comments
0
Likes
1

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Stat405 Advanced data manipulation 2 Hadley Wickham Thursday, 30 September 2010
  • 2. 1. Colloquium Monday 2. String basics 3. Group-wise transformations 4. Practice challenges Thursday, 30 September 2010
  • 3. Colloquium on Summer research experiences When: Monday, 4:00 - 5:00 Where: DH 1070 Coffee and Cookies will be served ahead of time Thursday, 30 September 2010
  • 4. Speakers 1. James Rigby -summer institute in bioinformatics 2. Gabi Quart - internship at Deutche Bank 3. Liz Jackson - Survey methodology summer program 4. Ollie McDonald - internship at Novartis in Switzerland 5. Christine Peterson - research in the Med center 6. Joseph Egbulefu - research at Rice Thursday, 30 September 2010
  • 5. String basics Thursday, 30 September 2010
  • 6. install.packages("stringr") library(stringr) str_length("Hadley") str_c(letters, LETTERS) str_c(letters, LETTERS, sep = " ") str_c("H", "a", "d", "l", "e", "y", collapse = "") tolower("Hadley") toupper("Hadley") str_sub("Hadley", 1, 3) str_sub("Hadley", -1) Thursday, 30 September 2010
  • 7. Your turn Add new columns that give: The length of each name The first letter of the name The last letter of the name Thursday, 30 September 2010
  • 8. library(stringr) bnames <- read.csv("baby-names2.csv.bz2", stringsAsFactors = FALSE) bnames <- transform(bnames, length = str_length(name), first = str_sub(name, 1, 1), last = str_sub(name, -1, -1)) Thursday, 30 September 2010
  • 9. Your turn Explore how the average length of names has changed over time (for each sex) Thursday, 30 September 2010
  • 10. library(plyr) sy <- ddply(bnames, c("sex", "year"), summarise, avg_length = weighted.mean(length, prop)) library(ggplot2) qplot(year, avg_length, data = sy, colour = sex, geom = "line") Thursday, 30 September 2010
  • 11. # Another approach syl <- ddply(bnames, c("sex", "length", "year"), summarise, prop = sum(prop)) qplot(year, prop, data = syl, colour = sex, geom = "line") + facet_wrap(~ length) Thursday, 30 September 2010
  • 12. Transformations Thursday, 30 September 2010
  • 13. Transformations What about group-wise transformations? e.g. what if we want to compute the rank of a name within a sex and year? This task is easy if we have a single year & sex, but hard otherwise. Thursday, 30 September 2010
  • 14. Transformations What about group-wise transformations? e.g. what if we want to compute the rank of a name within a sex and year? This task is easy if we have a single year & sex, but hard otherwise. How would you do it for a single group? Thursday, 30 September 2010
  • 15. one <- subset(bnames, sex == "boy" & year == 2008) one$rank <- rank(-one$prop, ties.method = "first") # or one <- transform(one, rank = rank(-prop, ties.method = "first")) head(one) What if we want to transform every sex and year? Thursday, 30 September 2010
  • 16. Workflow 1. Extract a single group 2. Figure out how to solve it for just that group 3. Use ddply to solve it for all groups Thursday, 30 September 2010
  • 17. Workflow 1. Extract a single group 2. Figure out how to solve it for just that group 3. Use ddply to solve it for all groups How would you use ddply to calculate all ranks? Thursday, 30 September 2010
  • 18. bnames <- ddply(bnames, c("sex", "year"), transform, rank = rank(-prop, ties.method = "first")) Thursday, 30 September 2010
  • 19. ddply + transform = group-wise transformation ddply + summarise = per-group summaries ddply + subset = per-group subsets Thursday, 30 September 2010
  • 20. Tools You know have all the tools to solve 95% of data manipulation problems in R. It’s just a matter of figuring out which tools to use, and how to combine them. The following challenges will give you some practice. Thursday, 30 September 2010
  • 21. Challenges Thursday, 30 September 2010
  • 22. Warmups Which names were most popular in 1999? Work out the average proportion for each name. List the 10 names with the highest average proportions. Thursday, 30 September 2010
  • 23. # Which names were most popular in 1999? subset(bnames, year == 1999 & rank < 10) subset(bnames, year == 1999 & prop == max(prop)) # Average usage overall <- ddply(bnames, "name", summarise, prop = mean(prop)) # Top 10 names head(arrange(overall, desc(prop)), 10) Thursday, 30 September 2010
  • 24. Challenge 1 How has the total proportion of babies with names in the top 1000 changed over time? How has the popularity of different initials changed over time? Thursday, 30 September 2010
  • 25. sy <- ddply(bnames, c("year","sex"), summarise, prop = sum(prop), npop = sum(prop > 1/1000)) qplot(year, prop, data = sy, colour = sex, geom = "line") qplot(year, npop, data = sy, colour = sex, geom = "line") Thursday, 30 September 2010
  • 26. Challenge 2 For each name, find the year in which it was most popular, and the rank in that year. (Hint: you might find which.max useful). Print all names that have been the most popular name at least once. Thursday, 30 September 2010
  • 27. most_pop <- ddply(bnames, "name", summarise, year = year[which.max(prop)], rank = min(rank)) most_pop <- ddply(bnames, "name", subset, prop == max(prop)) subset(most_pop, rank == 1) # Double challenge: Why is this last one wrong? Thursday, 30 September 2010
  • 28. Challenge 3 What name has been in the top 10 most often? (Hint: you'll have to do this in three steps. Think about what they are before starting) Thursday, 30 September 2010
  • 29. top10 <- subset(bnames, rank <= 10) counts <- count(top10, c("sex", "name")) ddply(counts, "sex", subset, freq == max(freq)) head(arrange(counts, desc(freq)), 10) Thursday, 30 September 2010
  • 30. Homework No homework this week. Use what you’ve learned to make your projects even better! Thursday, 30 September 2010