Sharding Redis at Flite

1,287
-1

Published on

Slides from San Francisco Redis Meetup, August 6, 2013.

Sharding Redis at Flite

Published in: Technology, Business
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,287
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
28
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Sharding Redis at Flite

  1. 1. SHARDING REDIS at FLITE Eugene Feingold & Chakri Kodali
  2. 2. WHAT IS FLITE? •  Display advertising platform •  Create and publish rich, interactive ads •  Serve ad impressions •  Collect and analyze metrics, batch AND realtime
  3. 3. WHAT IS FLITE? •  Display advertising platform •  Create and publish rich, interactive ads •  Serve ad impressions •  Collect and analyze metrics, batch AND realtime with
  4. 4. REALTIME YOU SAY? •  Realtime monitoring of ad performance •  Debugging of ads, with instant feedback on triggered events
  5. 5. METRICS ARCHITECTURE JVM JVM JVM ad ad ad ad ad ad ad node.js node.js pipelined write of ~30 items
  6. 6. WRITING TO REDIS Persistence: Up to 4 hours for most data Amount: ~30 writes per event, pipelined Granularity: By second, minute, hour Write Types: •  HSET - JSON blobs (Event body data) •  LPUSH - Lists of events (Events by session) •  HINCRBY - Simple counters (Events by ad) •  ZINCRBY - Sorted set counters (To retrieve “Top 100”) •  PUBSUB - Stream event data (Debugger)
  7. 7. READING FROM REDIS Redis transactions are used extensively like multi, exec Read types: •  HGETALL - get all data for event •  HGET - get event counts by ad •  LRANGE - list of all events by session •  ZREVRANGE - top 100 ads with highest number of events •  SUBSCRIBE
  8. 8. AND ALL OF THIS AT SCALE •  Daily traffic peaks: 100k - 200k events per minute •  Peaks are really plateaus that last for hours •  Read load is negligible by comparison, but reads must be fast •  In fact, everything must be fast: <1 sec latency for debugger to work
  9. 9. WHAT’S WRONG WITH THIS PICTURE? JVM JVM JVM ad ad ad ad ad ad ad node.js node.js
  10. 10. WHAT’S WRONG WITH THIS PICTURE? Bottleneck! JVM JVM JVM ad ad ad ad ad ad ad node.js node.js
  11. 11. SCALABILITY FAIL! What did it look like? 2-3x of our usual load
  12. 12. SCALABILITY FAIL! Note how only 1 core is being used 14,000 open connections! What did it look like?
  13. 13. A QUICK SOLUTION? •  MOAR Megahurtz!!: m2.2xl is already about as fast as Amazon gets. •  Redis-As-A-Service: Expensive, not fast enough for even our usual load. •  twemproxy: Twitter’s sharding solution. Doesn’t support all commands: PING, MULTI, INFO, MGET, etc…
  14. 14. WHAT ABOUT JEDIS’S NATIVE SHARDING? •  No pipelining •  No pubsub •  Complicated consistent hashing mechanism makes reading in other environments more difficult
  15. 15. LET’S ROLL OUR OWN! Goals: Speed, speed, speed Not Goals: Fault tolerance, redundancy, resiliency
  16. 16. HOW HARD CAN IT BE? Sharding method: Java hashCode of key for every item written JVMad event items node.js Write to many Read from one
  17. 17. WHAT HAPPENED? Before Sharding After Sharding Items per Event 30 30 Items written per Event per Redis 30 10 Redis Connections per Event 1 3 Connections per second per Redis box n n Reality: When n gets to around 500, Redis maxes out CPU and starts rejecting connections. Theory: Since Redis claims to be able to handle 70k connections per second, the amount of data being sent per connection is the problem.
  18. 18. SO HOW HARD CAN IT BE? HARD.
  19. 19. TAKE TWO Sharding method: Java hashCode of EVENT key for every item written. A single key now lives on multiple Redis boxes JVMad event items node.js Write to one Read from many
  20. 20. BETTER! Before Sharding After Take 1 After Take 2 Items per Event 30 30 30 Items written per Event per Redis 30 10 30 Redis Connections per Event 1 3 1 Connections per second per Redis box n n n/3 More load can be easily accommodated by adding boxes
  21. 21. CODING CHALLENGES Java •  Managing multiple connection pools •  Managing multiple pipelines •  Automatic health checks node.js •  Finding hashing function that works in different environments •  Managing multiple pipelines •  Fanout requests and merging response once pipeline is executed
  22. 22. SINGLE-REDIS JEDIS WORKFLOW On application startup: 1.  Initialize jedisPool with connection info Every time: 1.  Jedis jedisClient = jedisPool.getClient(); 2.  Pipeline pipeline = jedisClient.pipelined(); 3.  State your business 4.  pipeline.sync(); 5.  jedisPool.returnResource(jedisClient);
  23. 23. SHARDED JEDIS WORKFLOW On application startup: 1.  Initialize n jedisPools with connection info Every time: 1.  Jedis jedisClient = jedisPool.getClient(); 2.  Pipeline pipeline = jedisClient.pipelined(); 3.  State your business 4.  pipeline.sync(); 5.  jedisPool.returnResource(jedisClient);
  24. 24. SHARDED JEDIS WORKFLOW On application startup: 1.  Initialize n jedisPools with connection info Every time: 1.  Jedis jedisClient = jedisPool.getClient(); Which pool? 2.  Pipeline pipeline = jedisClient.pipelined(); Which client? 3.  State your business To whom? 4.  pipeline.sync(); Which pipeline? 5.  jedisPool.returnResource(jedisClient); Return what where?
  25. 25. SHARDED JEDIS WORKFLOW RedisNodeManager Spring-created singleton JedisPoolManagerJedisPoolManagerJedisPoolManagerRedisPoolManager
  26. 26. SHARDED JEDIS WORKFLOW RedisNodeManager Spring-created singleton JedisPoolManagerJedisPoolManagerJedisPoolManagerRedisPoolManager RedisPipelineManager JedisPoolManagerJedisPoolManagerJedisPoolManagerRedisPoolManager getPipelineManager()
  27. 27. SHARDED JEDIS WORKFLOW RedisNodeManager Spring-created singleton JedisPoolManagerJedisPoolManagerJedisPoolManagerRedisPoolManager RedisPipelineManager JedisPoolManagerJedisPoolManagerJedisPoolManagerRedisPoolManager Pipeline getPipelineManager() getPipeline(shardKey)
  28. 28. LET’S LOOK AT SOME CODE!
  29. 29. OPERATIONAL DETAILS •  Redis is single threaded o  You can run multiple Redises on one server o  Bind each Redis instance to a specific core
  30. 30. QUESTIONS?
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×