1.
From check-ins to recommendations
Jon Hoffman @hoffrocket
QCon NYC – June 11, 2014

2.
Watch the video with slide
synchronization on InfoQ.com!
http://www.infoq.com/presentations
/scale-foursquare
InfoQ.com: News & Community Site
• 750,000 unique visitors/month
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• News 15-20 / week
• Articles 3-4 / week
• Presentations (videos) 12-15 / week
• Interviews 2-3 / week
• Books 1 / month

3.
Presented at QCon New York
www.qconnewyork.com
Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide

4.
About Foursquare

5.
Scaling in two parts
• Part one: data storage
• Part two: application complexity

6.
Part 1: Data Storage
2009

7.
Table splits
DB.A
DB Checkins
Venues
Checkins
Users
Friends
Venues
DB.B
Users
Friends

8.
Replication
Master
RW
Slave
RO
Slave
RO

9.
Outgrowing our hardware
• Not enough RAM for indexes and
working data set
• 100 writes/second/disk

10.
Sharding
• Manage ourselves in application code on
top of postgres?
• Use something called Cassandra?
• Use something called HBase?
• Use something called Mongo?

11.
Besides Mongo
• Memcache
• Elastic search
– nearby venue search
– user search
• Custom data services
– Read only key value server
– in memory cache with business logic

12.
HFile Service: Read only KV Store
Hadoop HFile Servers
MR HDFS
hfile_0_a
hfile_0_b
hfile_1_b
hfile_0
hfile_1
Application
Servers
Zookeeper:
- data type to machine mapping
- key hash to shard mapping
hfile_1_a

13.
Caching Services
Mongo Oplog Tailer Kafka
Kafka
Consumers
Redis
Cache
Servers
Application
Servers
getUserVenueCounts(
1: list<i64> userIds
2: list<ObjectId> venues)

14.
Part 2: application complexity
2009

15.
RPC Tracing

16.
Throttles

17.
Remember the goats?

18.
Monolithic problems
• Compiling all the code, all the time
• Deploying all the code all the time
• Hard to isolate cause of performance
regressions and resource leaks

19.
SOA Infancy
• Single codebase, Multiple builds
Web
API
Offline

20.
Finagle Era
• Twitter’s scala based RPC library
service
Geocoder
{
GeocodeResponse
geocode(
1:
GeocodeRequest
r
)
}

21.
Benefits
• Independent compile targets
• Fined grained control on releases and
bug fixes
• Functional isolation

22.
Problems
• Duplication in packaging and
deployment efforts
• Hard to trace execution problems
• Hard to define/change where things live
• Networks aren’t reliable

23.
Builds and deploys
• single service definition file
• consistent build packaging
• simple deployment of canary & fleet
./service_releaser
–j
service_name

24.
Monitoring
• healthcheck endpoint over http
• consistent metric names
• dashboard for every service

25.
Distributed Tracing

26.
Exception Aggregation

27.
Application Discovery
• Finagle Server Sets + ZK

28.
Circuit Breaking
• Fast failing RPC calls after some error
rate threshold
• Loosely based on Netflix’s hystrix

29.
SOA Problem Recap
• Duplication in packaging and deployment efforts
– Build and deploy automation
• Hard to trace execution problems
– Monitoring consistency
– Distributed Tracing
– Error aggregation
• Hard to define/change where things live
– Application discovery with zookeeper
• Networks aren’t reliable
– Circuit breaking

30.
Organization
• Smaller teams owning front to back
implementation of features
• Desire to have quick deploy cycles on
new API endpoints

31.
Remote Endpoints
Wouldn’t it be cool if a developer
could expose a new API endpoint
without redeploying our still
monolithic API server?

32.
Remote Endpoint Benefits
• Very easy to experiment with new
endpoints
• Tight contract for service interaction
– JSON responses
– all http params passed along
• Clear path to breaking off more chunks
from API monolith

33.
Future work: Part 3?
• Further isolating services with
independent storage layers?
• Completely automated continuous
deployment
• Hybrid immutable/mutable data storage
– mongo & hfile & cache service

34.
Thanks!
• Want to build these things?
https://foursquare.com/jobs
• jon@foursquare.com

35.
Watch the video with slide synchronization on
InfoQ.com!
http://www.infoq.com/presentations/scale-foursquare