Big and Fat: Using MongoDB with Deep and Diverse Data

1,712 views

Published on

Delivered at MongoDC

Published in: Technology
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,712
On SlideShare
0
From Embeds
0
Number of Embeds
270
Actions
Shares
0
Downloads
14
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

Big and Fat: Using MongoDB with Deep and Diverse Data

  1. 1. Big and Fat Using MongoDB with deep and diverse datasets: A case study
  2. 2. About me • My name is Jeremy McAnally • “Software architect” at Intridea • Write a lot of books, OSS, etc. • http://github.com/jm • http://twitter.com/jm • http://railsupgradehandbook.com • http://wickhamhousebrand.com
  3. 3. MongoDB analytics just like mom used to make. http://mongobase.com
  4. 4. Preface The Application™
  5. 5. Disclaimer We moved to (mostly) sql.
  6. 6. YAK SHAVE YAK SHAVE YAK SHAVE YAK SHAVE YAK SHAVE YAK SHAVE YAK SHAVE YAK SHAVE YAK SHAVE YAK SHAVE YAK SHAVE YAK SHAVE YAK SHAVE YAK SHAVE YAK SHAVE YAK SHAVE YAK SHAVE YAK SHAVE YAK SHAVE YAK SHAVE YAK SHAVE YAK SHAVE YAK SHAVE YAK SHAVE YAK SHAVE YAK SHAVE YAK SHAVE YAK SHAVE YAK SHAVE YAK SHAVE YAK SHAVE YAK SHAVE YAK SHAVE YAK SHAVE YAK SHAVE YAK SHAVE YAK SHAVE YAK SHAVE YAK SHAVE YAK SHAVE YAK SHAVE YAK SHAVE YAK SHAVE YAK SHAVE YAK SHAVE YAK SHAVE YAK SHAVE YAK SHAVE
  7. 7. Lesson 1 Abstraction is a double- edged sword.
  8. 8. Abstract away! Talking to all data (no matter the source) the same way will keep you sane.
  9. 9. !"#$"%&%'()*+,,*!#$(-#.#/!0#12)3+345%6%789'%!"#$":2; !"#$"-#</=%>?%@!@ %%A?"0"%&%>B-/?CC#/0D?E1FA?"0"F;-GDE>1,!"#$HD>%&I%!JFD>FK; %%L%J---K %%/?MM#E0"%&%>B-/?CC#/0D?E1F/?MM#E0"F;-GDE>12NO=#$#2%&I%2"!M 10=D"-<>MDEH/?!E0P%0=D"-M?>#$<0?$H/?!E0;%&&%Q2; #E>
  10. 10. !"#$"%&%R"#$-<CC !"#$"-#</=%>?%@!@ %%A?"0"%&%S?"0-GDE>1,!"#$HD>%&I%!-D>; %%L%J---K %%/?MM#E0"%&%4?MM#E0-O=#$#12"!M10=D"-<>MDEH/?!E0P% %%%%%%%%%%%%%%%%%%%%%%%%%%%%0=D"-M?>#$<0?$H/?!E0;%&&%Q2; #E>
  11. 11. !"#$"%&%R"#$-<CC !"#$"-#</=%>?%@!@ %%A?"0"%&%S?"0-GDE>1,!"#$HD>%&I%!-D>; %%L%J---K %%/?MM#E0"%&%4?MM#E0-OD0=HGDT#H0=DEU" #E>
  12. 12. ...but wait! MongoDB has a lot of features that will perform better and be less (and often better) code.
  13. 13. A=<$M</D"0"%&%VW S<0D#E0-<CC-#</=%>?%@A<0D#E0@ %%A<0D#E0-A$#"/$DA0D?E"-#</=%>?%@A$#"/$DA0D?E@ %%%%A=<$M</D"0"JA$#"/DA0D?E-E<M#K%@@&%X %%%%A=<$M</D"0"JA$#"/DA0D?E-E<M#K%Y&%Z %%#E> #E>
  14. 14. A=<$M</D"0"%&%VW S<0D#E0-<CC-#</=%>?%@A<0D#E0@ %%A<0D#E0-A$#"/$DA0D?E"-#</=%>?%@A$#"/$DA0D?E@ %%%%A=<$M</D"0"JA$#"/DA0D?E-E<M#K%@@&%X %%%%A=<$M</D"0"JA$#"/DA0D?E-E<M#K%Y&%Z %%#E> #E>SLOW AS CRAP
  15. 15. M<A%&%2G!E/0D?E1;V %%%%0=D"-A$#"/$DA0D?E"-G?$3</=1 %%%%%%G!E/0D?E1A;%V% %%%%%%%%#MD01A-E<M#P%V%/?!E0%,%Z%W;: %%%%W;W2% %% $#>!/#%&%2G!E/0D?E1[P%T;%V %%T<$%E!MB#$%&%X: %%G?$%T-G?$3</=1G!E/0D?E1;%V %%%%E!MB#$%Y&%TJDK-/?!E0: %%W;: %%$#0!$E%V%/?!E0%,%E!MB#$%W:% W2% %% A=<$M"%&%A<0D#E0"-M<AH$#>!/#1M<AP%$#>!/#;
  16. 16. M<A%&%2G!E/0D?E1;V %%%%0=D"-A$#"/$DA0D?E"-G?$3</=1 %%%%%%G!E/0D?E1A;%V% %%%%%%%%#MD01A-E<M#P%V%/?!E0%,%Z%W;: %%%%W;W2% %% $#>!/#%&%2G!E/0D?E1[P%T;%V %%T<$%E!MB#$%&%X: %%G?$%T-G?$3</=1G!E/0D?E1;%V %%%%E!MB#$%Y&%TJDK-/?!E0: %%W;: %%$#0!$E%V%/?!E0%,%E!MB#$%W:% W2% %% A=<$M"%&%A<0D#E0"-M<AH$#>!/#1M<AP%$#>!/#;
  17. 17. Lesson 2 Schema design matters.
  18. 18. Lesson 2 Schema design matters. DATAMODEL
  19. 19. Embedding works. Embedding documents is a smart decision in a lot of cases.
  20. 20. )3+345%6%789'%A<0D#E0"%]^383%D>&_Z_: )3+345%6%789'%A$#"/$DA0D?E"%]^383%A<0D#E0HD>&_Z_: )3+345%6%789'%<AA?DE0M#E0"%]^383%A<0D#E0HD>&_Z_: )3+345%6%789'%/?E0</0"%]^383%A<0D#E0HD>&_Z_: )3+345%6%789'%/C<DM"%]^383%A<0D#E0HD>&_Z_: - - -
  21. 21. ...but watch it. You can also hit a ton of performance and design issues.
  22. 22. OUR GIANT DOCUMENT Mongo’s Pre-Allocated Space
  23. 23. Patient Pharmacy “Reference” Pharmacy Search, listing, etc.
  24. 24. Lesson 3 Don’t go nuts.
  25. 25. Schemaless is fun! Having schemaless data has its own battery of advantages.
  26. 26. Schemaless Joy
  27. 27. Schemaless Joy • Transforming data models is a delight
  28. 28. Schemaless Joy • Transforming data models is a delight • Formless data isn’t awkward
  29. 29. Schemaless Joy • Transforming data models is a delight • Formless data isn’t awkward • Arbitrary embedding is awesome
  30. 30. Schemaless Joy • Transforming data models is a delight • Formless data isn’t awkward • Arbitrary embedding is awesome • Building to work with schemaless data can lead to some really powerful app concepts
  31. 31. ...but be wary. Going nuts will create headaches for you.
  32. 32. Schemaless Pain
  33. 33. Schemaless Pain • Weird app behavior
  34. 34. Schemaless Pain • Weird app behavior • Huge, long-running data transformations
  35. 35. Schemaless Pain • Weird app behavior • Huge, long-running data transformations • Annoying data transforms for development env’s
  36. 36. Schemaless Pain • Weird app behavior • Huge, long-running data transformations • Annoying data transforms for development env’s • Difficult to version data models
  37. 37. Lesson 4 Dig deep.
  38. 38. I%>B-$!E4?MM<E>1V2"#$T#$)0<0!"2%,%ZW; V % 2T#$"D?E2%,%2Z-`-a2P % 2!A0DM#2%,%bcP % 2C?/<C5DM#2%,%25=!%d?T%Ze%_XZX%XZ,`b,ae% f'5gXQXX%13)5;2P % 2UC?B<C+?/[2%,%V % % 20?0<C5DM#2%,%bcXXQ_bXP % % 2C?/[5DM#2%,%Zh`X`XP % % 2$<0D?2%,%X-XXZeZ_eZchh_bXbXhc_ % WP % 2M#M2%,%V % % 2BD0"2%,%c`P % % 2$#"D>#E02%,%_P % % 2TD$0!<C2%,%_abcP % % 2"!AA?$0#>2%,%0$!#P % % 2M<AA#>2%,%X % WP % 2/?EE#/0D?E"2%,%V % % 2/!$$#E02%,%ZP % % 2<T<DC<BC#2%,%Zbbbb % WP
  39. 39. 2?A/?!E0#$"2%,%V %%2DE"#$02%,%XP %%2i!#$(2%,%ZP %%2!A><0#2%,%XP %%2>#C#0#2%,%XP %%2U#0M?$#2%,%XP %%2/?MM<E>2%,%a W
  40. 40. 2/?EE#/0D?E"2%,%V %%2/!$$#E02%,%ZP %%2<T<DC<BC#2%,%Zbbbb W
  41. 41. j#$#M(g'/kE<CC("g'</l??[gS$?,m%n#$#M(M/<E<CC(N%M?EU?"0<0 /?EE#/0#>%0?,%Z_h-X-X-Z DE"#$0o"%i!#$(o"%!A><0#o"%>#C#0#o"%U#0M?$#o"%/?MM<E>o"%M<AA#>%%T"Dp#%%%%$#"%q%C?/[#>%q%D>.%MD""%%/?EE% %%%%%%%X%%%%%%%X%%%%%%%%X%%%%%%%%X%%%%%%%%%X%%%%%%%%%Z%%%%%%X%%%_abc%%%%%%a%%%%%%%%X%%%%%%%%%%X%%%%%Z% %%%%%%%X%%%%%%%X%%%%%%%%X%%%%%%%%X%%%%%%%%%X%%%%%%%%%Z%%%%%%X%%%_abc%%%%%%a%%%%%%%%X%%%%%%%%%%X%%%%%Z% %%%%%%%X%%%%%%%X%%%%%%%%X%%%%%%%%X%%%%%%%%%X%%%%%%%%%Z%%%%%%X%%%_abc%%%%%%a%%%%%%%%X%%%%%%%%%%X%%%%%Z% %%%%%%%X%%%%%%%X%%%%%%%%X%%%%%%%%X%%%%%%%%%X%%%%%%%%%Z%%%%%%X%%%_abc%%%%%%a%%%%%%%%X%%%%%%%%%%X%%%%%Z% %%%%%%%X%%%%%%%X%%%%%%%%X%%%%%%%%X%%%%%%%%%X%%%%%%%%%Z%%%%%%X%%%_abc%%%%%%a%%%%%%%%X%%%%%%%%%%X%%%%%Z% %%%%%%%X%%%%%%%X%%%%%%%%X%%%%%%%%X%%%%%%%%%X%%%%%%%%%Z%%%%%%X%%%_abc%%%%%%a%%%%%%%%X%%%%%%%%%%X%%%%%Z% %%%%%%%X%%%%%%%X%%%%%%%%X%%%%%%%%X%%%%%%%%%X%%%%%%%%%Z%%%%%%X%%%_abc%%%%%%a%%%%%%%%X%%%%%%%%%%X%%%%%Z% %%%%%%%X%%%%%%%X%%%%%%%%X%%%%%%%%X%%%%%%%%%X%%%%%%%%%Z%%%%%%X%%%_abc%%%%%%a%%%%%%%%X%%%%%%%%%%X%%%%%Z%
  42. 42. >B-H<>MDE4?MM<E>1V%>D<U+?UUDEU%,%Z%W;
  43. 43. >B-/!$$#E09A1; V%DEA$?U,%J%V%2?AD>2%,%aQ%P%2?A2%,%2i!#$(2%P%2E"2%,% 2G!E>B-A<$0D#"2%P %%%%%%%%%%%%%%2i!#$(2%,%2V%"/?$#%,%Z-X%W2%P%2DE+?/[2%,%Z%W %%%%%%%%%%K W
  44. 44. I%>B-?AC?U-NM<DE-GDE>1; V%20"2%,%V%202%,%Z_bXXcaQccXXXP%2D2%,%Z%WP%2?A2%,%2D2P%2E"2%,%2MDEU V%20"2%,%V%202%,%Z_bXXcaQcbXXXP%2D2%,%Z%WP%2?A2%,%2E2P%2E"2%,%22P%2 V%20"2%,%V%202%,%Z_bXXcaQhbXXXP%2D2%,%Z%WP%2?A2%,%2E2P%2E"2%,%22P%2 V%20"2%,%V%202%,%Z_bXXcaQeZXXXP%2D2%,%Z%WP%2?A2%,%2D2P%2E"2%,%2MDEU V%20"2%,%V%202%,%Z_bXXcaQeZXXXP%2D2%,%_%WP%2?A2%,%2D2P%2E"2%,%2MDEU
  45. 45. V%20"2%,% %%V%202%,%Z_bXXcaQccXXXP% %%%%2D2%,%Z% %%WP% %%2?A2%,%2D2P% %%2E"2%,%2MDEU-G??2P% %%2?2%,%V% %%%%%2HD>2%,%9Bn#/0r>12`/#`/#/#<BBZBcQZQeXXXXXZ2;P% %%%%%2GD#C>2%,%_% %%W% W
  46. 46. That’s all I got. Questions?

×