Recommended
More Related Content
Similar to intro-to-metaprogramming-in-r.pdf
Similar to intro-to-metaprogramming-in-r.pdf (20)
More from K. Matthew Dupree
Recently uploaded
Recently uploaded (20)
intro-to-metaprogramming-in-r.pdf
- 4. y tho? how it works my journey
- 6. programs that understand or modify programs
- 9. df | > filter(col1 = = 1, col2 = = 1) df.query('col1 = = 1 & col2 = = 1')
- 15. -Matt Harrison, Author of E ff ective Pandas “I've had people say, “This is the worst code that I've ever seen."…And then, on the flip side, I get people like, "This is awesome. This changed how I write code. My life is much better…”
- 16. a1 = autos[cols] cyls = autos.cylinders.fillna(0) cyls2 = cyls.astype('int8') a1['cylinders'] = cyls2 displ = a1.displ displ2 = displ.fillna(0) displ3 = displ2.astype('float16') a1.displ = displ3 a1.drive = autos.drive.fillna('Other').astype('category') a1['automatic'] = autos.trany.str.contains('Auto') speed = autos.trany.str.extract(r'(d)+') speedfill = speed.fillna('20') speedint = speedfill.astype('int8') a1['speeds'] = speedint a1.createdOn = pd.to_datetime( autos.createdOn).dt.tz_localize('America/New_York') a1.ffs = autos.eng_dscr.str.contains('FFS') a1['highway08'] = autos.highway08.astype('int8') a1['city08'] = autos.city08.astype('int8') a1['comb08'] = autos.comb08.astype('int16') a1['fuelCost08'] = autos.fuelCost08.astype('int16') a1['range'] = autos.range.astype('int16') a1['make'] = autos.make.astype('category') a3 = a1.drop(columns=['trany', 'eng_dscr'])
- 17. (autos [cols] # create var .pipe(get_var, 'df3') .assign(cylinders=autos.cylinders.fillna(0).astype('int8'), displ=autos.displ.fillna(0).astype('float16'), drive=autos.drive.fillna('Other').astype('category'), automatic=autos.trany.str.contains('Auto'), speeds=autos.trany.str.extract( r'(d)+').fillna('20').astype('int8'), createdOn=pd.to_datetime(autos.createdOn.replace({' EDT': '-04 : 00', ' EST': '-05 : 00'}, regex=True), utc=True).dt.tz_convert('America/New_York'), ffs=autos.eng_dscr.str.contains('FFS') ) # debug pipe .pipe(lambda df: display(df) or df) .astype({'highway08': 'int8', 'city08': 'int16', 'comb08': 'int16', 'fuelCost08': 'int16', 'range': 'int16', 'year': 'int16', 'make': 'category'}) .drop(columns=['trany', 'eng_dscr']) )
- 18. (autos [cols] # create var .pipe(get_var, 'df3') .assign(cylinders=autos.cylinders.fillna(0).astype('int8'), displ=autos.displ.fillna(0).astype('float16'), drive=autos.drive.fillna('Other').astype('category'), automatic=autos.trany.str.contains('Auto'), speeds=autos.trany.str.extract( r'(d)+').fillna('20').astype('int8'), createdOn=pd.to_datetime(autos.createdOn.replace({' EDT': '-04 : 00', ' EST': '-05 : 00'}, regex=True), utc=True).dt.tz_convert('America/New_York'), ffs=autos.eng_dscr.str.contains('FFS') ) # debug pipe .pipe(lambda df: display(df) or df) .astype({'highway08': 'int8', 'city08': 'int16', 'comb08': 'int16', 'fuelCost08': 'int16', 'range': 'int16', 'year': 'int16', 'make': 'category'}) .drop(columns=['trany', 'eng_dscr']) )
- 19. | > > .pipe
- 20. | > > .pipe Cmd + Shift + M
- 21. -Wes McKinney “I wish we could do pipes in python, but we don’t have nonstandard evaluation.”
- 25. postgres_con_as_df | > mutate(col1 = if(x > 100) 100 else x) | > group_by(col1) | > summarize(mean(col2))
- 26. postgres_con_as_df | > mutate(col1 = if(x > 100) 100 else x) | > group_by(col1) | > summarize(mean(col2)) #> <SQL> CASE WHEN (`x` > 100) THEN 100 WHEN NOT (`x` > 100) THEN x END
- 29. df | > group_by(col1) | > summarize(postgres_udf(col2)) SELECT postgres_udf(col2) FROM df GROUP BY col1
- 32. How it works
- 33. df | > filter(col1 = = 1, col2 = = 1) df.query('col1 = = 1 & col2 = = 1')
- 34. df < - data.frame(x = 1 : 10, y = 11 : 20, z = 21 : 30) df$x + df$y + df$z
- 35. df < - data.frame(x = 1 : 10, y = 11 : 20, z = 21 : 30) df$x + df$y + df$z = = with(df, x + y + z)
- 36. df < - data.frame(x = 1 : 10, y = 11 : 20, z = 21 : 30) df$x + df$y + df$z = = with(df, x + y + z)
- 37. how many lines of code to implement with?
- 40. -Hadley Wickam, Advanced R “The name is a portmanteau of quoting and closure, because a quosure both quotes the expression and encloses the environment.”
- 41. -Hadley Wickam, Advanced R “The name is a portmanteau of quoting and closure, because a quosure both quotes the expression and encloses the environment.”
- 42. Add 1 to x vs She said, “Add 1 to x”
- 43. Add 1 to x vs She said, “Add 1 to x” x < - 42 x + 1 [1] 43 quote(x + 1) x + 1
- 44. -Hadley Wickam, Advanced R “The name is a portmanteau of quoting and closure, because a quosure both quotes the expression and encloses the environment.”
- 45. -Hadley Wickam, Advanced R “The name is a portmanteau of quoting and closure, because a quosure both quotes the expression and encloses the environment.”
- 46. She said, “Add 1 to x” … Now, do what she said Crap. I can’t. I forgot what x is
- 47. environments map variables to values
- 48. She said, “Add 1 to x” … Now, do what she said Its 43 because x was 42
- 52. Now, do what she said
- 54. Now, do what she said… and add the vectors from this data frame to the environment
- 55. df < - data.frame(x = 1 : 10, y = 11 : 20, z = 21 : 30) df$x + df$y + df$z = = with(df, x + y + z)
- 56. with(df, x + y + z) She said, “x + y + z” Now, do what she said, and add the vectors from df to the environment
- 58. postgres_con_as_df | > mutate(col1 = if(x > 100) 100 else x) | > group_by(col1) | > summarize(mean(col2))
- 59. postgres_con_as_df | > mutate(col1 = if(x > 100) 100 else x) | > group_by(col1) | > summarize(mean(col2)) #> <SQL> CASE WHEN (`x` > 100) THEN 100 WHEN NOT (`x` > 100) THEN x END
- 62. My meta programming journey
- 65. snowflake_df | > mutate(col1 = ifelse(x > 100, 100, x)) | > collect()
- 66. snowflake_df | > mutate(col1 = ifelse(x > 100, 100, x)) | > collect() snowflake_df | > count()
- 67. snowflake_df | > mutate(col1 = ifelse(x > 100, 100, x)) | > collect()
- 68. snowflake_df | > mutate(col1 = ifelse(x > 100, 100, x)) | > collect()
- 69. snowflake_df | > mutate(col1 = ifelse(x > 100, 100, x)) | > collect() Cmd + Shift + C
- 73. bad joins comparing small groups bugs in mutation code data quality issues
- 77. how can we use meta- programming to make data analysis faster and safer?