This talk will feature: memcache, resque, a bit of metaprogramming, a look at caching in the wild and code that fixes some usual problems, and a fairly epic SQL query with some nice Postgres features you should know about.
I have an axe to grind. Forming working teams. Makeup. Engineering described as the brake. Technique: I think it relates well to what Aaron was saying about concurrency. This needs to be easy.Dispel a little meta-programming fear
Going to try to ramp up from problem statement to solution description, to general features of the solution to full blown meta programming analysis.There is a alpha gem, not actually in production. But I’m going to get into the meta programming that wrote it, because honestly it’s more there for display than for you to jump out and use.Moreover I think some people are scared of this stuff and I want to try to clear up a bit about that.
Pattern identification. 3 examples
Trials have centers every center has many recruiting locationsWe have a theory that distance to the center is a big deal. Differentiator.
See! They can all live together happily. I bring you a message of joy and peace
It’s one of those death star queries. It’s like your poor little server is Alderan.
Company name, but we don’t do a great job
Clear weighted factorsJaccard similarityDistance search without any new installsEasily extensible.
Beautiful. Makes my heart sing.Postrgres CTEA partridge in a pear tree
Third example. What kind of pattern is here?
That’s a crazy idea, you can’t do that.
The idea here is that we’re going to be able to predict the course of your disease progression based on everyone else who’s like you.It actually did work.
I’m sure this doesn’t remind you of any of the entrepreneurs you work with.
This is our ops guyThe guy with the pager
Who here knows what memoization is?
Note class method.Expiry. Soft expiration. Why soft? Memcache not super versatile. Good enough for us.No backgrounding. This lets us use the same pattern in places where we always want a result / don’t want to write an async UI.
This is a nice general purpose queuing thing.ANY method in your app get’s auto backgrounding with no setup.
NEXT: we’re going to do those two. Learn about meta-programming. Stick with me!
This is not a good slide. So we’re just going to look at it for a long long time.Until you get it.
Step 1: we have the interface, but what does it return?
----- Meeting Notes (4/24/12 08:05) -----need to look up in memcahce and return patella result
This is what gets called from Resque
This is not a real
Who knows what this is?
The fetch ALWAYS returns something but doesn’t always kick off a new job.
1) NIH PRESENTATIONIN 4 WEEKS!Integrate
clinicaltrials.gov into our siteSearch by trial typeSearch by trial phaseSearch by trial conditions mapped from Mesh to MeddraSearch by trial facility locations… • Location search…
PATIENT SEARCH RANKINGVery basic searchPlus
very complex orderingNot as many great solutions in this spaceN^2 similarity matrix @ 100k patients about 4 TBAnd did I mention it’s N^2?Postgres is an amazingly viable solution.
STEPPING BACKConflict • Relational data
is most easily queried relationally. • Relational queries don’t necessarily scale and stay in the millisecond range • Denormalized queries & special solutions scale • But take longer to implement • (note) This isn’t just SQL, I’m talking about anything slowWe want to experiment/fail fast • But we don’t want…
WHAT WE WANTTrivially easy way
for developers to declare that somemethods are not to be run without adult supervision.Consistent framework so that ops doesn’t need to be afraidof new, sometimes expensive experiments.
SOLUTION SPACEMemcached • Great •
Simple to setup. • Could be simpler. Handmade cache keys feels wrong. • But it doesn’t solve our :-C problem. • The first request still slams the server. • So you do some cache warming thing… • But this is a PITA again.
WHAT COULD MAKETHIS SIMPLER?Remove one
constraint.A basic Rails.cache.fetch guarantees you a result • But no performance guaranteeFlip that deal around. • Guarantee performance • Don’t guarantee a result • It’s ok not to know the answer!
SEND LATERSuper easy way to
just do something later while in the samecontext.Most workers are real boring.Single worker for suffices for many background jobs.Makes testing/development easier by bypassing Resque inconfiguration.AR extension. Coordinates logging / monitoring.