Scaling Wanelo.com 100x in Six Months

Scaling

100x
in six months

by Eric Saxby & Konstantin Gredeskoul
April 2013

Proprietary and
Thursday, April 18, 13 Confidential 1

What is Wanelo?
■ Wanelo (“Wah-nee-lo” from Want, Need
Love) is a global platform for shopping.

Proprietary and

■ It’s marketing-free shopping across
100s of thousands of unique stores

Proprietary and

Personal Activity Feed...
Proprietary and

iOS + Android
Proprietary and

Early Decisions

Proprietary and

Early Decisions

■ Optimize for iteration speed, not
performance

Proprietary and

Early Decisions

performance

■ Keep scalability in mind, track metrics,
and ﬁx as needed

Proprietary and

Early Decisions

performance

■ Keep scalability in mind, track metrics,
and ﬁx as needed

■ Introduce many levels of caching early

Proprietary and

Technology Timeline

Proprietary and

Technology Timeline

■ 2010 - 2011
Wanelo v1 stack is Java, JSP, MySQL, Hibernate
90K lines of code, 53+ DB tables, no tests

Proprietary and

Technology Timeline

■ 2010 - 2011

■ May 2012 - June 2012
Rewrite from scratch to RoR on PostgreSQL (v2)

Proprietary and

Technology Timeline

■ 2010 - 2011

■ May 2012 - June 2012
Rewrite from scratch to RoR on PostgreSQL (v2)
■ Ruby app is 10K LOC, full test coverage, 8
database tables, less features

Proprietary and

The “Big” Rewrite

Proprietary and


More info here....

Proprietary and


More info here....

building.wanelo.com/
http://

Proprietary and

Growth Timeline

Proprietary and

Growth Timeline
■ 06/2012 - RoR App Relaunches

Proprietary and

Growth Timeline
■ 2-3K requests per minute (RPM) peak

Proprietary and

Growth Timeline

■ 08/2012 - iOS App is launched

Proprietary and

Growth Timeline

■ 10-40K RPM peak

Proprietary and

Growth Timeline

■ 10-40K RPM peak

■ 12/2012 - Android app launched

Proprietary and

Growth Timeline

■ 10-40K RPM peak

■ 40-120K RPM peak

Proprietary and

Growth Timeline

■ 10-40K RPM peak


■ 03/2013 - #24 top free apps iTunes

Proprietary and

Growth Timeline

■ 10-40K RPM peak


■ 03/2013 - #24 top free apps iTunes
Proprietary and

Requests Per Minute (RPM)

Proprietary and

Current Numbers...

■ 4M active monthly users
■ 5M products saved 700M times
■ 8M products saved per day
■ 200k stores

Proprietary and

Backend Stack & Key Vendors
■ MRI Ruby 1.9.3 & Rails 3.2
■ PostgreSQL 9.2.4, Solr 3.6
■ Joyent Cloud, SmartOS
ZFS, ARC, raw IO performance, SmartOS, CPU bursting, dTrace

■ Circonus, Chef + Opscode
Monitoring, graphing, alerting, automation

■ Amazon S3 + Fastly CDN
■ NewRelic, statsd, Graphite, nagios

Proprietary and

Wanelo Web Architecture

nginx
6 x 2GB
haproxy

unicorn x 14 sidekiq
20 x 8GB
4 x 8GB
haproxy pgbouncer twemproxy haproxy pgbouncer twemproxy

Solr PostgreSQL Redis MemCached

Proprietary and

This talk is about:

Proprietary and

This talk is about:

1. How much trafﬁc can your database handle?

Proprietary and

This talk is about:


2. Special report on counters

Proprietary and

This talk is about:



3. Scaling database reads

Proprietary and

This talk is about:



3. Scaling database reads

4. Scaling database writes

Proprietary and

1.
How much traffic can your
database handle?

Thursday, April 18, 13 15

PostgreSQL is Awesome!

Proprietary and

■ Does a fantastic job of not corrupting
your data

Proprietary and

your data

■ Streaming replication in 9.2 is
extremely reliable

Proprietary and

your data

extremely reliable

■ Won’t write to a read-only replica

Proprietary and

your data

extremely reliable


■ But... No master/master replication
Proprietary and

your data

extremely reliable


■ But... No master/master replication
(good!)
Proprietary and

Is the database healthy?

Proprietary and

What’s healthy?

Proprietary and

What’s healthy?

■ Able to respond quickly to queries from
application (< 4ms disk seek time)

Proprietary and

What’s healthy?


■ Has enough room to grow

Proprietary and

What’s healthy?


■ Has enough room to grow

■ How do we know when we’re
approaching a dangerous threshold?

Proprietary and

Oops!

NewRelic Latency (yellow = database)

Proprietary and

pg_stat_statements

■ Maybe your app is to blame for
performance...

select

query,
calls,
total_time

from

pg_stat_statements

order
by
total_time
desc
limit
12;

Proprietary and

pg_stat_statements

■ Maybe your app is to blame for
performance...

select

query,
calls,
total_time

from

pg_stat_statements

order
by
total_time
desc
limit
12;

Similar to Percona Toolkit, but runs all the
time collecting stats.

Proprietary and

pg_stat_statements

Proprietary and

pg_stat_user_indexes

■ Using indexes as much as you think
you are?

■ Using indexes at all?

Proprietary and

pg_stat_user_tables
■ Full table scans? (seq_scan)

Proprietary and

Throw that in a graph

Reads/second for one large table, daily

Proprietary and

Non-linear changes

Suspicious spike!

Proprietary and

Correlate different data

Deployments! Aha!

Proprietary and

Utilization vs Saturation

# of Active PostgreSQL connections

Proprietary and


Red line: % of max connections established
Purple: % of connections in query

Proprietary and

Disk reads/writes

green: reads, red: writes

Proprietary and

Disk reads/writes

green: reads, red: writes

Usage increases, but are the disks saturated?

Proprietary and


Proprietary and


[
How much are you waiting on disk?

Proprietary and

File system cache (ARC)

Proprietary and

Watch the right things

Hit ratio of the ﬁle system cache (ARC)

Proprietary and

Room to grow...

Size (including indexes) of a key table

Proprietary and

Working set in RAM?

Adding index increases the size

Proprietary and

Collect all the data you can

Once we knew where to look, graphs added
later could explain behavior we could
only guess at earlier

Proprietary and

2.
Special report on
Counters and Pagination


Problem #1: DB Latency Up...

Proprietary and


■ iostat shows 100% disk busy

Proprietary and


■ iostat shows 100% disk busy

device

r/s

w/s

Mr/s

Mw/s
wait
actv

svc_t

%w

%b

sd1

384.0
1157.5

48.0

116.8

0.0

8.8

5.7

2
100

sd1

368.0
1117.9

45.7

106.3

0.0

8.0

5.4

2
100

sd1

330.3
1357.5

41.3

139.1

0.0

9.5

5.6

2
100

Proprietary and

Problem #1: Diagnostics

Proprietary and

■ Database is running very very hot.
Initial investigation shows large number of counts.

Proprietary and


■ Turns out anytime you page with Kaminari, it
always does a count(*)!

Proprietary and


■ Turns out anytime you page with Kaminari, it
always does a count(*)!

SELECT
"stores".*
FROM
"stores"

WHERE
(state
=
'approved')

LIMIT
20
OFFSET
0

SELECT
COUNT(*)
FROM
"stores"
WHERE
(state
=
'approved')

Proprietary and

Problem #1: Pagination

Proprietary and


■ Doing count(*) is pretty expensive, as DB
must scan many rows (either the actual table
or an index)

Proprietary and


Proprietary and


■ We are paginating everything! Even inﬁnite
scroll is a paged view behind the scenes.

Proprietary and


■ We are paginating everything! Even inﬁnite
scroll is a paged view behind the scenes.

■ But we really DON’T want to run count(*) for
every paged view.

Proprietary and


■ We are showing most popular stores
■ Maybe it’s OK to hard-code the total number to,
say, 1000?

Proprietary and


■ We are showing most popular stores
■ Maybe it’s OK to hard-code the total number to,
say, 1000?

■ How do we tell Kaminari NOT to issue a
count query in this case?

Proprietary and

Problem #1: Pagination (ctd)

Proprietary and

Solution #1: Monkey Patch!!

Proprietary and

Solution #1: Pass in the
counter

Proprietary and

Solution #1: Pass in the
counter

SELECT
"stores".*
FROM
"stores"
WHERE
(state
=

'approved')
LIMIT
20
OFFSET
0

Proprietary and

Problem #2: Count Draculas
■ AKA: We still are doing too many counts!

Proprietary and

Problem #2: Count Draculas
■ AKA: We still are doing too many counts!

■ Rails makes it so easy to do it the lazy way.

Proprietary and

Problem #2: Too Many Counts!
■ But it just doesn’t scale well

Proprietary and

Problem #2: Too Many Counts!
■ But it just doesn’t scale well

■ Fortunately, Rails has just a feature for this...

Proprietary and

Counter Caches
■ Unfortunately, it has one massive issue:

Proprietary and

Counter Caches

■ It causes database deadlocks at high volume

Proprietary and

Counter Caches


■ Because many ruby processes are creating child
records concurrently

Proprietary and

Counter Caches



■ Each is executing a callback, trying to update
counter_cache column on the parent, requiring
row-level lock

Proprietary and

Counter Caches



■ Each is executing a callback, trying to update
counter_cache column on the parent, requiring
row-level lock

■ Deadlocks ensue
Proprietary and

Possible Solution:
Use Background Jobs

Proprietary and

Possible Solution:
Use Background Jobs
■ It works like this:

Proprietary and

Possible Solution:
Use Background Jobs
■ As the record is created, we enqueue a request
to recalculate counter_cache on the parent

Proprietary and

Possible Solution:
Use Background Jobs
■ As the record is created, we enqueue a request
to recalculate counter_cache on the parent

■ The job performs a complete recalculation of
the counter cache and is idempotent

Proprietary and

Solution #2: Explained

Proprietary and

■ Sidekiq with UniqueJob extension

Proprietary and


■ Short wait for “buffering”

Proprietary and



■ Serialize updates via small number of workers

Proprietary and



■ Serialize updates via small number of workers

■ Can temporarily stop workers (in an
emergency) to alleviate DB load

Proprietary and

Solution #2: Code

Proprietary and

Things are better. BUT...

Proprietary and

Still too many fucking counts!

Proprietary and


■ Even doing count(*) from workers is too
much on the databases

Proprietary and



■ We need to stop doing count(*) in DB. But
keep counter_caches. How?

Proprietary and



■ We need to stop doing count(*) in DB. But
keep counter_caches. How?

■ We could use Redis for this.

Proprietary and

save product product_id

Solution #3:
Counts Deltas unicorn

counter_cache column

1. INCR product_id 2. ProductCountWorker.enqueue product_id

Redis Redis
PostgreSQL
Counters Sidekiq

4. GET
3. Dequeue
5. RESET

5. SQL Update INCR by N

sidekiq

Proprietary and

save product product_id

Solution #3:
Counts Deltas unicorn

■ Web request increments
counter value in Redis
counter_cache column

■ Enqueues request to
update counter_cache 1. INCR product_id 2. ProductCountWorker.enqueue product_id

■ Background Job picks up
a few minutes later, reads Redis Redis
PostgreSQL
Redis delta value, and Counters Sidekiq
removes it.

■ Updates counter_cache 4. GET
5. RESET
3. Dequeue

column by incrementing it
by delta. 5. SQL Update INCR by N

sidekiq

Proprietary and

Define counter_cache_on...

■ Internal GEM, will open source soon!
Proprietary and

Can now use counter caches
in pagination!

Proprietary and

3.
Scaling reads


Multiple optimization cycles

■ Caching
action caching, fragment, CDN

■ Personalization via AJAX
Cache the entire page, then add
personalized details

■ 25ms/req memcached time is cheaper than
12ms/req of database time

Proprietary and

Cache optimization

40% hit ratio! Woo!
Wait... is that even good?

Proprietary and

Cache optimization

Increasing your hit ratio means less
queries against your database

Proprietary and

Cache optimization

Caveat: even low hit ratio caches
can save your ass. You’re removing
load from the DB, remember?

Proprietary and

Cache saturation

Blue: cache writes How long before your caches
Red: automatic evictions start evicting data?

Proprietary and

Ajax personalization

Proprietary and

Nice!

■ Rails Action Caching
Runs before_ﬁlters, so A/B experiments can still run

■ Extremely fast pages
4ms application time for some of our
computationally heaviest pages

■ Could be served via CDN in the future
Proprietary and

Sad trombone...

■ Are you actually logged in?
Pages don’t know until Ajax successfully runs

■ Selenium AND Jasmine tests!

Proprietary and

Read/write splitting
■ Sometime in December 2012...

Proprietary and

■ Database reaching 100% saturation

Proprietary and


■ Latency starting to increase non-linearly

Proprietary and



■ We need to distribute database load

Proprietary and



■ We need to distribute database load

■ We need to use read replicas!

Proprietary and

DB adapters for read/write
■ Looked at several, including DbCharmer

Proprietary and

DB adapters for read/write
■ Looked at several, including DbCharmer

■ Features / Conﬁgurability / Stability
■ Thread safety? This may be Ruby, but some
people do actually use threads.

■ If I tell you it’s a read-only replica, DON’T
ISSUE WRITES

■ Failover on errors?

Proprietary and

Chose Makara, by TaskRabbit
■ Used in production
■ We extended it to work with PostgreSQL
■ Works with Sidekiqs (thread-safe!)
■ Failover code is very simple. Simple is
sometimes better.

https://github.com/taskrabbit/makara

Proprietary and

We rolled out Makara and...
■ 1 master, 3 read-only async replicas

Proprietary and

We rolled out Makara and...
■ 1 master, 3 read-only async replicas

Wait, what?
Proprietary and

A note about graphs

■ NewRelic is great!
■ Not easy to predict when your
systems are about to fall over
■ Use something else to visualize
Database and disk saturation

Proprietary and

3 days later, in production

■ 3 read replicas distributing load from master

■ app servers and sidekiqs create lots of
connections to DB backends

Proprietary and

3 days later, in production

■ 3 read replicas distributing load from master

■ app servers and sidekiqs create lots of
connections to DB backends

■ Mysterious spikes in errors at high trafﬁc

Proprietary and

Replication! Doh!

Replication lag (yellow)
correlates with application errors (red)

Proprietary and

Replication lag! Doh!
■ Track latency sending xlog to slaves
select client_addr,
pg_xlog_location_diff(sent_location, write_location)
from pg_stat_replication;

■ Track latency applying xlogs on slaves
select pg_xlog_location_diff(
pg_last_xlog_receive_location(),
pg_last_xlog_replay_location()),
extract(epoch from now()) -
extract(epoch from pg_last_xact_replay_timestamp());

Proprietary and

Eventual Consistency

Proprietary and

■ Some code paths should always go to
master for reads (ie, after signup)

Proprietary and


■ Application should be resilient to
getting RecordNotFound to tolerate
replication delays

Proprietary and


■ Application should be resilient to
getting RecordNotFound to tolerate
replication delays

■ Not enough to scale reads.
Writes become the bottleneck.
Proprietary and

Write load delays replication

Replicas are busy trying to apply XLOGs
and serve heavy read trafﬁc

Proprietary and

4.
Scaling database writes


First, No-Brainers:
■ Move stuff out of the DB. Easiest ﬁrst.

Proprietary and

First, No-Brainers:
■ Tracking user activity is very easy to do
with a database table. But slow.

Proprietary and

First, No-Brainers:

■ 2000 inserts/sec while also handling site
critical data? Not a good idea.

Proprietary and

First, No-Brainers:

■ 2000 inserts/sec while also handling site
critical data? Not a good idea.

■ Solution:
UDP packets to rsyslog, ASCII delimited ﬁles, log-
rotate, analyze them later

Proprietary and

Next: Async Commits

Proprietary and

Next: Async Commits

■ PostgreSQL supports delayed
(batched) commits

Proprietary and

Next: Async Commits

(batched) commits

■ Delays fsync for some # of
microseconds

Proprietary and

Next: Async Commits

(batched) commits

■ Delays fsync for some # of
microseconds

■ At high volume helps disk IO

Proprietary and

PostgreSQL Async Commits

Proprietary and

ZFS Block Size

Proprietary and

ZFS Block Size

■ Default ZFS block size is 128Kb

Proprietary and

ZFS Block Size

■ PostgreSQL block size is 8Kb

Proprietary and

ZFS Block Size

■ Small writes require lots of bandwidth

Proprietary and

ZFS Block Size

■ Small writes require lots of bandwidth
device

r/s

w/s

Mr/s

Mw/s
wait
actv

svc_t

%w

%b

sd1

384.0
1157.5

48.0

116.8

0.0

8.8

5.7

2
100

sd1

368.0
1117.9

45.7

106.3

0.0

8.0

5.4

2
100

sd1

330.3
1357.5

41.3

139.1

0.0

9.5

5.6

2
100

Proprietary and

ZFS Block Size (ctd.)

Proprietary and


■ Solution: change ZFS block size to 8K:

Proprietary and

device

r/s

w/s

Mr/s

Mw/s
wait
actv

svc_t

%w

%b

sd1

384.0
1157.5

48.0

116.8

0.0

8.8

5.7

2
100

sd1

368.0
1117.9

45.7

106.3

0.0

8.0

5.4

2
100

sd1

330.3
1357.5

41.3

139.1

0.0

9.5

5.6

2
100


Proprietary and

device

r/s

w/s

Mr/s

Mw/s
wait
actv

svc_t

%w

%b

sd1

384.0
1157.5

48.0

116.8

0.0

8.8

5.7

2
100

sd1

368.0
1117.9

45.7

106.3

0.0

8.0

5.4

2
100

sd1

330.3
1357.5

41.3

139.1

0.0

9.5

5.6

2
100

device

r/s

w/s

Mr/s

Mw/s
wait
actv

svc_t

%w

%b
sd1

130.3

219.9

1.0

4.4

0.0

0.7

2.1

0

37
sd1

329.3

384.1

2.6

11.3

0.0

1.9

2.6

1

78
sd1

335.3

357.0

2.6

8.7

0.0

1.8

2.6

1

80
sd1

354.0

200.3

2.8

4.9

0.0

1.6

3.0

0

84
sd1

465.3

100.7

3.6

1.7

0.0

2.1

3.7

0

91

Proprietary and

Next: Vertical Sharding

Proprietary and


■ Move out largest table into its own
master database (150 inserts/sec)

Proprietary and



■ Remove any SQL joins, do them in
application, drop foreign keys

Proprietary and



■ Remove any SQL joins, do them in
application, drop foreign keys

■ Switch model to establish_connection
to another DB. Fix many broken tests.

Proprietary and

Vertical Sharding
unicorns

haproxy pgbouncer twemproxy

PostgreSQL
saves master
PostgreSQL PostgreSQL PostgreSQL
main replica main replica main master

streaming replication

Proprietary and

Vertical Sharding: Results

Proprietary and

Vertical Sharding: Results

■ Deploy All Things!

Proprietary and

Future: Services Approach
unicorns

haproxy pgbouncer twemproxy

http / json

PostgreSQL PostgreSQL PostgreSQL sinatra services app
main replica main replica main master

streaming replication
Shard1 Shard2 Shard3

Proprietary and

In Conclusion. Tasty gems :)
https://github.com/wanelo/pause

https://github.com/wanelo/spanx

https://github.com/wanelo/redis_with_failover

https://github.com/kigster/ventable

Proprietary and

■ distributed rate limiting using redis




Proprietary and


■ rate-limit-based IP blocker for nginx



Proprietary and



■ attempt another redis server if available


Proprietary and



■ attempt another redis server if available

■ observable pattern with a twist

Proprietary and

Thanks.
Comments? Questions?
https://github.com/wanelo
https://github.com/wanelo-chef

@kig & @sax
@kig & @ecdysone
@kigster & @sax

Proprietary and

Scaling Wanelo.com 100x in Six Months

Scaling Wanelo.com 100x in Six Months

Recommended

More Related Content

Recently uploaded

Recently uploaded(20)

Featured

Featured(20)

Scaling Wanelo.com 100x in Six Months

Editor's Notes