Patella railsconf 2012

2,995 views

Published on

This talk will feature: memcache, resque, a bit of metaprogramming, a look at caching in the wild and code that fixes some usual problems, and a fairly epic SQL query with some nice Postgres features you should know about.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,995
On SlideShare
0
From Embeds
0
Number of Embeds
811
Actions
Shares
0
Downloads
27
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide
  • I have an axe to grind. Forming working teams. Makeup. Engineering described as the brake. Technique: I think it relates well to what Aaron was saying about concurrency. This needs to be easy.Dispel a little meta-programming fear
  • Going to try to ramp up from problem statement to solution description, to general features of the solution to full blown meta programming analysis.There is a alpha gem, not actually in production. But I’m going to get into the meta programming that wrote it, because honestly it’s more there for display than for you to jump out and use.Moreover I think some people are scared of this stuff and I want to try to clear up a bit about that.
  • Pattern identification. 3 examples
  • Trials have centers every center has many recruiting locationsWe have a theory that distance to the center is a big deal. Differentiator.
  • See! They can all live together happily. I bring you a message of joy and peace
  • It’s one of those death star queries. It’s like your poor little server is Alderan.
  • Company name, but we don’t do a great job
  • Clear weighted factorsJaccard similarityDistance search without any new installsEasily extensible.
  • Beautiful. Makes my heart sing.Postrgres CTEA partridge in a pear tree
  • Third example. What kind of pattern is here?
  • That’s a crazy idea, you can’t do that.
  • The idea here is that we’re going to be able to predict the course of your disease progression based on everyone else who’s like you.It actually did work.
  • I’m sure this doesn’t remind you of any of the entrepreneurs you work with.
  • This is our ops guyThe guy with the pager
  • Who here knows what memoization is?
  • Note class method.Expiry. Soft expiration. Why soft? Memcache not super versatile. Good enough for us.No backgrounding. This lets us use the same pattern in places where we always want a result / don’t want to write an async UI.
  • This is a nice general purpose queuing thing.ANY method in your app get’s auto backgrounding with no setup.
  • NEXT: we’re going to do those two. Learn about meta-programming. Stick with me!
  • This is not a good slide. So we’re just going to look at it for a long long time.Until you get it.
  • Step 1: we have the interface, but what does it return?
  • ----- Meeting Notes (4/24/12 08:05) -----need to look up in memcahce and return patella result
  • This is what gets called from Resque
  • This is not a real
  • Who knows what this is?
  • The fetch ALWAYS returns something but doesn’t always kick off a new job.
  • Patella railsconf 2012

    1. PATELLAMEMOIZATION INTO MEMCACHED DONE IN RESQUEJEFF DWYERPATIENTSLIKEME@JDWYAH
    2. TODAYEngineers will never be successful if we are the brake oninnovation.Technique for innovating safelyLearn a bit about meta-programming
    3. TODAYSetup the problemSketch the solutionNitty Gritty Details
    4. TODAYSetup the problemSketch the solutionNitty Gritty Details
    5. 1) NIH PRESENTATIONIN 4 WEEKS!Integrate clinicaltrials.gov into our siteSearch by trial typeSearch by trial phaseSearch by trial conditions mapped from Mesh to MeddraSearch by trial facility locations… • Location search…
    6. WE HAVE A CHOICE
    7. WHAT IS RIGHTPostGIS spatial database extensions for PostgreSQLMongoDB built in support for two dimensional geospatial indexes
    8. AND WHAT IS EASYsqrt(pow(69.1 * (clinical_trial_locations.lat - 40.948073),2) + pow(53.0 * (clinical_trial_locations.lng - -90.36871),2)) AS distance
    9. CHOOSE THE EASY!
    10. CHOOSE THE EASY!Who knows if location is even important?Who knows if this project is even important?MongoDB requires dev setup, automated stagingsetup, production setup, monitoring.
    11. BUT, OH GOD THEHUMANITY<query plan pic>
    12. BUT, OH GOD THEHUMANITY<query plan pic>
    13. 2) PATIENT LIKE ME SEARCH
    14. 2) PATIENT LIKE ME SEARCH
    15. PATIENT SEARCH RANKINGVery basic searchPlus very complex orderingNot as many great solutions in this spaceN^2 similarity matrix @ 100k patients about 4 TBAnd did I mention it’s N^2?Postgres is an amazingly viable solution.
    16. ELEGANT CODE…
    17. LOVELY SQL…
    18. BUT IT’S JUST THISSIDE OF ‘REAL-TIME’One second queries just don’t fly.And oh, yeah 16 people hitting it at the same time wouldclobber the servers.
    19. 3) A FORWARD LOOKINGTIME MACHINEMaybe those were aberrations?Crazy right?
    20. A FORWARD LOOKINGTIME MACHINE
    21. AND HERE’S MY CEO PROMISING ITAT TED
    22. A FORWARD LOOKINGTIME MACHINE
    23. STEPPING BACKConflict • Relational data is most easily queried relationally. • Relational queries don’t necessarily scale and stay in the millisecond range • Denormalized queries & special solutions scale • But take longer to implement • (note) This isn’t just SQL, I’m talking about anything slowWe want to experiment/fail fast • But we don’t want…
    24. DON’T WANT TO LET THIS:
    25. TURN INTO:
    26. TURN INTO: :-C
    27. TURN INTO: :-C
    28. TODAYSetup the problemSketch the solutionNitty Gritty Details
    29. WHAT WE WANTTrivially easy way for developers to declare that somemethods are not to be run without adult supervision.Consistent framework so that ops doesn’t need to be afraidof new, sometimes expensive experiments.
    30. SOLUTION SPACEDoing it right all the time • Too slow and expensive • Slows innovation
    31. SOLUTION SPACEMemoization • Brilliant • Functional Programming Nirvana • No cache-key shenanagins • But also no expiry… • There’s just one thing… • It only works in a single request
    32. SOLUTION SPACEMemcached • Great • Simple to setup. • Could be simpler. Handmade cache keys feels wrong. • But it doesn’t solve our :-C problem. • The first request still slams the server. • So you do some cache warming thing… • But this is a PITA again.
    33. WHAT COULD MAKETHIS SIMPLER?Remove one constraint.A basic Rails.cache.fetch guarantees you a result • But no performance guaranteeFlip that deal around. • Guarantee performance • Don’t guarantee a result • It’s ok not to know the answer!
    34. BUT IT NEEDS A NAME!
    35. TECHNOLOGIES!Memoization into Memcached with everything calculated inResque.
    36. TODAYSetup the problemSketch the solutionNitty Gritty Details
    37. PATELLA DEVELOPERINTERFACE
    38. SEND LATERSuper easy way to just do something later while in the samecontext.Most workers are real boring.Single worker for suffices for many background jobs.Makes testing/development easier by bypassing Resque inconfiguration.AR extension. Coordinates logging / monitoring.
    39. SENDLATER User.send_later :expensive, arg1, arag2
    40. SENDLATER RESQUEWORKER
    41. MEMOIZATIONSendLater gets things calculated in Resque, but that’s step 1.We still need:Memoization.Stored in memcached.
    42. THIS IS NOT A GOOD SLIDE
    43. PATELLA RESULT
    44. THE METHOD
    45. WITH PATELLA
    46. THE REPLACEMENT
    47. THE ONE THAT DOESTHE WORK
    48. MAYBE IT’S BETTER NOW?
    49. DOG PILE
    50. THE REPLACEMENT
    51. DOG PILE
    52. LONG ARGUMENTS
    53. LONG ARGUMENTS
    54. LONG ARGUMENTS
    55. SOFT EXPIRATIONMemcached is great, but it doesn’t tell you when somethingexpires.Our strategy was to add a ‘soft_expiry’This gets stored along with the result.Then recalculate if soft_expiry < now()
    56. ABJALWAYS BE JSONBeware putting not JSON in memcachedYou really don’t want to know
    57. META IS MAGIC
    58. REAL LIFE
    59. PRETTY BORINGExcept that it works. Round 1: Major Pain Points Round 2: Magic Scaling SprinklesSuper alpha gem here: https://github.com/kbrock/patellaAlternative https://github.com/csquared/rack-worker Very REST-ish, request based.
    60. JOE@JOERODRIGUEZ
    61. AMY@AMYNEWELL
    62. KEENAN@KBROCK
    63. WINFIELD@WPETERSON

    ×