12 adv-manip

1,584 views
1,541 views

Published on

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,584
On SlideShare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
65
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

12 adv-manip

  1. 1. Stat405 Advanced data manipulation 2 Hadley Wickham Thursday, 30 September 2010
  2. 2. 1. Colloquium Monday 2. String basics 3. Group-wise transformations 4. Practice challenges Thursday, 30 September 2010
  3. 3. Colloquium on Summer research experiences When: Monday, 4:00 - 5:00 Where: DH 1070 Coffee and Cookies will be served ahead of time Thursday, 30 September 2010
  4. 4. Speakers 1. James Rigby -summer institute in bioinformatics 2. Gabi Quart - internship at Deutche Bank 3. Liz Jackson - Survey methodology summer program 4. Ollie McDonald - internship at Novartis in Switzerland 5. Christine Peterson - research in the Med center 6. Joseph Egbulefu - research at Rice Thursday, 30 September 2010
  5. 5. String basics Thursday, 30 September 2010
  6. 6. install.packages("stringr") library(stringr) str_length("Hadley") str_c(letters, LETTERS) str_c(letters, LETTERS, sep = " ") str_c("H", "a", "d", "l", "e", "y", collapse = "") tolower("Hadley") toupper("Hadley") str_sub("Hadley", 1, 3) str_sub("Hadley", -1) Thursday, 30 September 2010
  7. 7. Your turn Add new columns that give: The length of each name The first letter of the name The last letter of the name Thursday, 30 September 2010
  8. 8. library(stringr) bnames <- read.csv("baby-names2.csv.bz2", stringsAsFactors = FALSE) bnames <- transform(bnames, length = str_length(name), first = str_sub(name, 1, 1), last = str_sub(name, -1, -1)) Thursday, 30 September 2010
  9. 9. Your turn Explore how the average length of names has changed over time (for each sex) Thursday, 30 September 2010
  10. 10. library(plyr) sy <- ddply(bnames, c("sex", "year"), summarise, avg_length = weighted.mean(length, prop)) library(ggplot2) qplot(year, avg_length, data = sy, colour = sex, geom = "line") Thursday, 30 September 2010
  11. 11. # Another approach syl <- ddply(bnames, c("sex", "length", "year"), summarise, prop = sum(prop)) qplot(year, prop, data = syl, colour = sex, geom = "line") + facet_wrap(~ length) Thursday, 30 September 2010
  12. 12. Transformations Thursday, 30 September 2010
  13. 13. Transformations What about group-wise transformations? e.g. what if we want to compute the rank of a name within a sex and year? This task is easy if we have a single year & sex, but hard otherwise. Thursday, 30 September 2010
  14. 14. Transformations What about group-wise transformations? e.g. what if we want to compute the rank of a name within a sex and year? This task is easy if we have a single year & sex, but hard otherwise. How would you do it for a single group? Thursday, 30 September 2010
  15. 15. one <- subset(bnames, sex == "boy" & year == 2008) one$rank <- rank(-one$prop, ties.method = "first") # or one <- transform(one, rank = rank(-prop, ties.method = "first")) head(one) What if we want to transform every sex and year? Thursday, 30 September 2010
  16. 16. Workflow 1. Extract a single group 2. Figure out how to solve it for just that group 3. Use ddply to solve it for all groups Thursday, 30 September 2010
  17. 17. Workflow 1. Extract a single group 2. Figure out how to solve it for just that group 3. Use ddply to solve it for all groups How would you use ddply to calculate all ranks? Thursday, 30 September 2010
  18. 18. bnames <- ddply(bnames, c("sex", "year"), transform, rank = rank(-prop, ties.method = "first")) Thursday, 30 September 2010
  19. 19. ddply + transform = group-wise transformation ddply + summarise = per-group summaries ddply + subset = per-group subsets Thursday, 30 September 2010
  20. 20. Tools You know have all the tools to solve 95% of data manipulation problems in R. It’s just a matter of figuring out which tools to use, and how to combine them. The following challenges will give you some practice. Thursday, 30 September 2010
  21. 21. Challenges Thursday, 30 September 2010
  22. 22. Warmups Which names were most popular in 1999? Work out the average proportion for each name. List the 10 names with the highest average proportions. Thursday, 30 September 2010
  23. 23. # Which names were most popular in 1999? subset(bnames, year == 1999 & rank < 10) subset(bnames, year == 1999 & prop == max(prop)) # Average usage overall <- ddply(bnames, "name", summarise, prop = mean(prop)) # Top 10 names head(arrange(overall, desc(prop)), 10) Thursday, 30 September 2010
  24. 24. Challenge 1 How has the total proportion of babies with names in the top 1000 changed over time? How has the popularity of different initials changed over time? Thursday, 30 September 2010
  25. 25. sy <- ddply(bnames, c("year","sex"), summarise, prop = sum(prop), npop = sum(prop > 1/1000)) qplot(year, prop, data = sy, colour = sex, geom = "line") qplot(year, npop, data = sy, colour = sex, geom = "line") Thursday, 30 September 2010
  26. 26. Challenge 2 For each name, find the year in which it was most popular, and the rank in that year. (Hint: you might find which.max useful). Print all names that have been the most popular name at least once. Thursday, 30 September 2010
  27. 27. most_pop <- ddply(bnames, "name", summarise, year = year[which.max(prop)], rank = min(rank)) most_pop <- ddply(bnames, "name", subset, prop == max(prop)) subset(most_pop, rank == 1) # Double challenge: Why is this last one wrong? Thursday, 30 September 2010
  28. 28. Challenge 3 What name has been in the top 10 most often? (Hint: you'll have to do this in three steps. Think about what they are before starting) Thursday, 30 September 2010
  29. 29. top10 <- subset(bnames, rank <= 10) counts <- count(top10, c("sex", "name")) ddply(counts, "sex", subset, freq == max(freq)) head(arrange(counts, desc(freq)), 10) Thursday, 30 September 2010
  30. 30. Homework No homework this week. Use what you’ve learned to make your projects even better! Thursday, 30 September 2010

×