Your SlideShare is downloading. ×

Sharding Redis at Flite

1,040

Published on

Slides from San Francisco Redis Meetup, August 6, 2013. …

Slides from San Francisco Redis Meetup, August 6, 2013.

Sharding Redis at Flite

Published in: Technology, Business
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,040
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
27
Comments
0
Likes
2
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. SHARDING REDIS at FLITE Eugene Feingold & Chakri Kodali
  • 2. WHAT IS FLITE? •  Display advertising platform •  Create and publish rich, interactive ads •  Serve ad impressions •  Collect and analyze metrics, batch AND realtime
  • 3. WHAT IS FLITE? •  Display advertising platform •  Create and publish rich, interactive ads •  Serve ad impressions •  Collect and analyze metrics, batch AND realtime with
  • 4. REALTIME YOU SAY? •  Realtime monitoring of ad performance •  Debugging of ads, with instant feedback on triggered events
  • 5. METRICS ARCHITECTURE JVM JVM JVM ad ad ad ad ad ad ad node.js node.js pipelined write of ~30 items
  • 6. WRITING TO REDIS Persistence: Up to 4 hours for most data Amount: ~30 writes per event, pipelined Granularity: By second, minute, hour Write Types: •  HSET - JSON blobs (Event body data) •  LPUSH - Lists of events (Events by session) •  HINCRBY - Simple counters (Events by ad) •  ZINCRBY - Sorted set counters (To retrieve “Top 100”) •  PUBSUB - Stream event data (Debugger)
  • 7. READING FROM REDIS Redis transactions are used extensively like multi, exec Read types: •  HGETALL - get all data for event •  HGET - get event counts by ad •  LRANGE - list of all events by session •  ZREVRANGE - top 100 ads with highest number of events •  SUBSCRIBE
  • 8. AND ALL OF THIS AT SCALE •  Daily traffic peaks: 100k - 200k events per minute •  Peaks are really plateaus that last for hours •  Read load is negligible by comparison, but reads must be fast •  In fact, everything must be fast: <1 sec latency for debugger to work
  • 9. WHAT’S WRONG WITH THIS PICTURE? JVM JVM JVM ad ad ad ad ad ad ad node.js node.js
  • 10. WHAT’S WRONG WITH THIS PICTURE? Bottleneck! JVM JVM JVM ad ad ad ad ad ad ad node.js node.js
  • 11. SCALABILITY FAIL! What did it look like? 2-3x of our usual load
  • 12. SCALABILITY FAIL! Note how only 1 core is being used 14,000 open connections! What did it look like?
  • 13. A QUICK SOLUTION? •  MOAR Megahurtz!!: m2.2xl is already about as fast as Amazon gets. •  Redis-As-A-Service: Expensive, not fast enough for even our usual load. •  twemproxy: Twitter’s sharding solution. Doesn’t support all commands: PING, MULTI, INFO, MGET, etc…
  • 14. WHAT ABOUT JEDIS’S NATIVE SHARDING? •  No pipelining •  No pubsub •  Complicated consistent hashing mechanism makes reading in other environments more difficult
  • 15. LET’S ROLL OUR OWN! Goals: Speed, speed, speed Not Goals: Fault tolerance, redundancy, resiliency
  • 16. HOW HARD CAN IT BE? Sharding method: Java hashCode of key for every item written JVMad event items node.js Write to many Read from one
  • 17. WHAT HAPPENED? Before Sharding After Sharding Items per Event 30 30 Items written per Event per Redis 30 10 Redis Connections per Event 1 3 Connections per second per Redis box n n Reality: When n gets to around 500, Redis maxes out CPU and starts rejecting connections. Theory: Since Redis claims to be able to handle 70k connections per second, the amount of data being sent per connection is the problem.
  • 18. SO HOW HARD CAN IT BE? HARD.
  • 19. TAKE TWO Sharding method: Java hashCode of EVENT key for every item written. A single key now lives on multiple Redis boxes JVMad event items node.js Write to one Read from many
  • 20. BETTER! Before Sharding After Take 1 After Take 2 Items per Event 30 30 30 Items written per Event per Redis 30 10 30 Redis Connections per Event 1 3 1 Connections per second per Redis box n n n/3 More load can be easily accommodated by adding boxes
  • 21. CODING CHALLENGES Java •  Managing multiple connection pools •  Managing multiple pipelines •  Automatic health checks node.js •  Finding hashing function that works in different environments •  Managing multiple pipelines •  Fanout requests and merging response once pipeline is executed
  • 22. SINGLE-REDIS JEDIS WORKFLOW On application startup: 1.  Initialize jedisPool with connection info Every time: 1.  Jedis jedisClient = jedisPool.getClient(); 2.  Pipeline pipeline = jedisClient.pipelined(); 3.  State your business 4.  pipeline.sync(); 5.  jedisPool.returnResource(jedisClient);
  • 23. SHARDED JEDIS WORKFLOW On application startup: 1.  Initialize n jedisPools with connection info Every time: 1.  Jedis jedisClient = jedisPool.getClient(); 2.  Pipeline pipeline = jedisClient.pipelined(); 3.  State your business 4.  pipeline.sync(); 5.  jedisPool.returnResource(jedisClient);
  • 24. SHARDED JEDIS WORKFLOW On application startup: 1.  Initialize n jedisPools with connection info Every time: 1.  Jedis jedisClient = jedisPool.getClient(); Which pool? 2.  Pipeline pipeline = jedisClient.pipelined(); Which client? 3.  State your business To whom? 4.  pipeline.sync(); Which pipeline? 5.  jedisPool.returnResource(jedisClient); Return what where?
  • 25. SHARDED JEDIS WORKFLOW RedisNodeManager Spring-created singleton JedisPoolManagerJedisPoolManagerJedisPoolManagerRedisPoolManager
  • 26. SHARDED JEDIS WORKFLOW RedisNodeManager Spring-created singleton JedisPoolManagerJedisPoolManagerJedisPoolManagerRedisPoolManager RedisPipelineManager JedisPoolManagerJedisPoolManagerJedisPoolManagerRedisPoolManager getPipelineManager()
  • 27. SHARDED JEDIS WORKFLOW RedisNodeManager Spring-created singleton JedisPoolManagerJedisPoolManagerJedisPoolManagerRedisPoolManager RedisPipelineManager JedisPoolManagerJedisPoolManagerJedisPoolManagerRedisPoolManager Pipeline getPipelineManager() getPipeline(shardKey)
  • 28. LET’S LOOK AT SOME CODE!
  • 29. OPERATIONAL DETAILS •  Redis is single threaded o  You can run multiple Redises on one server o  Bind each Redis instance to a specific core
  • 30. QUESTIONS?

×