Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Patella railsconf 2012

3,067 views

Published on

This talk will feature: memcache, resque, a bit of metaprogramming, a look at caching in the wild and code that fixes some usual problems, and a fairly epic SQL query with some nice Postgres features you should know about.

Published in: Technology
  • Be the first to comment

Patella railsconf 2012

  1. PATELLAMEMOIZATION INTO MEMCACHED DONE IN RESQUEJEFF DWYERPATIENTSLIKEME@JDWYAH
  2. TODAYEngineers will never be successful if we are the brake oninnovation.Technique for innovating safelyLearn a bit about meta-programming
  3. TODAYSetup the problemSketch the solutionNitty Gritty Details
  4. TODAYSetup the problemSketch the solutionNitty Gritty Details
  5. 1) NIH PRESENTATIONIN 4 WEEKS!Integrate clinicaltrials.gov into our siteSearch by trial typeSearch by trial phaseSearch by trial conditions mapped from Mesh to MeddraSearch by trial facility locations… • Location search…
  6. WE HAVE A CHOICE
  7. WHAT IS RIGHTPostGIS spatial database extensions for PostgreSQLMongoDB built in support for two dimensional geospatial indexes
  8. AND WHAT IS EASYsqrt(pow(69.1 * (clinical_trial_locations.lat - 40.948073),2) + pow(53.0 * (clinical_trial_locations.lng - -90.36871),2)) AS distance
  9. CHOOSE THE EASY!
  10. CHOOSE THE EASY!Who knows if location is even important?Who knows if this project is even important?MongoDB requires dev setup, automated stagingsetup, production setup, monitoring.
  11. BUT, OH GOD THEHUMANITY<query plan pic>
  12. BUT, OH GOD THEHUMANITY<query plan pic>
  13. 2) PATIENT LIKE ME SEARCH
  14. 2) PATIENT LIKE ME SEARCH
  15. PATIENT SEARCH RANKINGVery basic searchPlus very complex orderingNot as many great solutions in this spaceN^2 similarity matrix @ 100k patients about 4 TBAnd did I mention it’s N^2?Postgres is an amazingly viable solution.
  16. ELEGANT CODE…
  17. LOVELY SQL…
  18. BUT IT’S JUST THISSIDE OF ‘REAL-TIME’One second queries just don’t fly.And oh, yeah 16 people hitting it at the same time wouldclobber the servers.
  19. 3) A FORWARD LOOKINGTIME MACHINEMaybe those were aberrations?Crazy right?
  20. A FORWARD LOOKINGTIME MACHINE
  21. AND HERE’S MY CEO PROMISING ITAT TED
  22. A FORWARD LOOKINGTIME MACHINE
  23. STEPPING BACKConflict • Relational data is most easily queried relationally. • Relational queries don’t necessarily scale and stay in the millisecond range • Denormalized queries & special solutions scale • But take longer to implement • (note) This isn’t just SQL, I’m talking about anything slowWe want to experiment/fail fast • But we don’t want…
  24. DON’T WANT TO LET THIS:
  25. TURN INTO:
  26. TURN INTO: :-C
  27. TURN INTO: :-C
  28. TODAYSetup the problemSketch the solutionNitty Gritty Details
  29. WHAT WE WANTTrivially easy way for developers to declare that somemethods are not to be run without adult supervision.Consistent framework so that ops doesn’t need to be afraidof new, sometimes expensive experiments.
  30. SOLUTION SPACEDoing it right all the time • Too slow and expensive • Slows innovation
  31. SOLUTION SPACEMemoization • Brilliant • Functional Programming Nirvana • No cache-key shenanagins • But also no expiry… • There’s just one thing… • It only works in a single request
  32. SOLUTION SPACEMemcached • Great • Simple to setup. • Could be simpler. Handmade cache keys feels wrong. • But it doesn’t solve our :-C problem. • The first request still slams the server. • So you do some cache warming thing… • But this is a PITA again.
  33. WHAT COULD MAKETHIS SIMPLER?Remove one constraint.A basic Rails.cache.fetch guarantees you a result • But no performance guaranteeFlip that deal around. • Guarantee performance • Don’t guarantee a result • It’s ok not to know the answer!
  34. BUT IT NEEDS A NAME!
  35. TECHNOLOGIES!Memoization into Memcached with everything calculated inResque.
  36. TODAYSetup the problemSketch the solutionNitty Gritty Details
  37. PATELLA DEVELOPERINTERFACE
  38. SEND LATERSuper easy way to just do something later while in the samecontext.Most workers are real boring.Single worker for suffices for many background jobs.Makes testing/development easier by bypassing Resque inconfiguration.AR extension. Coordinates logging / monitoring.
  39. SENDLATER User.send_later :expensive, arg1, arag2
  40. SENDLATER RESQUEWORKER
  41. MEMOIZATIONSendLater gets things calculated in Resque, but that’s step 1.We still need:Memoization.Stored in memcached.
  42. THIS IS NOT A GOOD SLIDE
  43. PATELLA RESULT
  44. THE METHOD
  45. WITH PATELLA
  46. THE REPLACEMENT
  47. THE ONE THAT DOESTHE WORK
  48. MAYBE IT’S BETTER NOW?
  49. DOG PILE
  50. THE REPLACEMENT
  51. DOG PILE
  52. LONG ARGUMENTS
  53. LONG ARGUMENTS
  54. LONG ARGUMENTS
  55. SOFT EXPIRATIONMemcached is great, but it doesn’t tell you when somethingexpires.Our strategy was to add a ‘soft_expiry’This gets stored along with the result.Then recalculate if soft_expiry < now()
  56. ABJALWAYS BE JSONBeware putting not JSON in memcachedYou really don’t want to know
  57. META IS MAGIC
  58. REAL LIFE
  59. PRETTY BORINGExcept that it works. Round 1: Major Pain Points Round 2: Magic Scaling SprinklesSuper alpha gem here: https://github.com/kbrock/patellaAlternative https://github.com/csquared/rack-worker Very REST-ish, request based.
  60. JOE@JOERODRIGUEZ
  61. AMY@AMYNEWELL
  62. KEENAN@KBROCK
  63. WINFIELD@WPETERSON

×