•intro to news.me•our technology stack on aws•why use aws?•a few parting tips
news.me for email
news.me for iphone
this app is great for the subway
web beta coming soon. beta testers?
“Finding more of anything never staysa problem long on the web...What is hard is finding less, butordered by quality, relevance orurgency.” - Clay Shirky
our mission... deliver an amazing newsexperience, and make it fast
avoid this...sad newspaper boy == fail whale
•queuing•beanstalk(d)•caching•redis news.me stack•memcache•database•mongo (1.8 with replica sets). EBS backed•s3
news.me apis (ec2 + s3) kinda big content data store™ api-1.news.me memcache memcache mongoelb api-2.news.me mongo api-n.news.me redis redis
some api notes•have to be fast: few hundred milliseconds•returning full-text feeds for offline caching•but feeds can be 300K+ (not great for 3/4G networks)•gzip! (400K down to 80K)•balancing speed with utility (again, for offline caching)
news.me backend (ec2 + s3)content from: kinda big content data store™ aggregator-1 mongo aggregator-2 mongo redis aggregator-n redis
some aggregation notes•aggregating ~30 million tweets and fb shares (with links) per day•the streams move pretty fast. had some growing pains with mongo•kinda big content data store™ = article content and metadata stored inmemcache + multiple ec2 instances + s3•kinda big content data store™ = over 1 TB of content data•auto scale the aggregators when we need them - helps keep the costdown :-)
why use aws?easy to scalefocus on productpay for what you needcommunity (all of Betaworks companies are onAWS)
do it for the back rubs!
a few tips...decouple everything (makes dealing with failureand scaling easier)measure everything!keep it lean - one persondon’t reinvent the wheel