Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Top 10 Ways to Scale with Redis
OCT 2019 | DAVE NIELSEN
In-memory Multi-model Database
2
Optionally Persistent
3
4
50,420,000
41,190,000
30,200,000
21,120,000
11,380,000
5,020,000
3 6 12 18 24 26
ops/sec
# ofnodes
ClusterThroughput(@ 1...
5
Developers
+
Redis
Redis Top Differentiators
Simplicity ExtensibilityPerformance
NoSQL Benchmark
1
Redis Data Structures
2 3
Redis Modules
6
...
Performance: The Most Powerful Database
Highest Throughput at Lowest Latency
in High Volume of Writes Scenario
Least Serve...
“REDIS IS FULL OF DATA STRUCTURES!”
2 Simplicity: Data Structures - Redis’ Building Blocks
Simplicity: Redis Data Structures – ’Lego’ Building Blocks
Lists
[ A → B → C → D → E ]
Hashes
{ A: “foo”, B: “bar”, C: “ba...
• Add-ons that use a Redis API to seamlessly support additional
use cases and data structures.
• Enjoy Redis’ simplicity, ...
• RediSearch
• RedisTimerseries
• ReJSON
• Redis Graph
• Rebloom
• Neural-Redis
• Redis-Cell
• Redis-Tdigest
• Redis-ML
Ex...
Deep Dive
Real Time
Analytics
User Session
Store
Real Time Data
Ingest
High Speed
Transactions
Job & Queue
Management
Time Series Da...
Manage Session Stores w/ Redis Hash
• An chunk of data that is connected to one “user” of a service
–”user” can be a simple visitor
–or proper user with an ac...
Session Storage Uses Cases
Traditional
• Username
• Preferences
• Name
• “Stateful” data
Intelligent
• Traditional +
• Not...
In a simple world
Internet Server Database
Good problems
Internet Server Database
Traffic Grows… Struggles
Good solution
Internet Server Database
performance restored
Session
storage on the
server
More good problems
Internet Server Database
Session
storage on the
server
Struggling
Problematic Solutions
Internet Server Database
Session
storage on the
server
Load balanced
Session
storage on the
server
Multiple Servers + On-server Sessions?
Server DatabaseRobin
Server #1 – Hello Robin!
Multiple Servers + On-server Sessions?
Server DatabaseRobin
Server #3 – Hello ????
Better solution
Internet Server Database
Load balanced
Redis
Session Storage
User Session Store
• The Problem
• Maintain session state across
multiple servers
• Multiple session variables
• High spee...
Redis Hashes Example - https://redis.io/commands#hash
userid 8754
name dave
ip 10:20:104:31
hits 1
lastpage home
hash key:...
Managing Queues w/ Redis Lists
Managing Queues of Work
• The Problem
• Tasks need to be worked on asynch
to reduce block/wait times
• Lots of items to be...
Redis Lists Example - https://redis.io/commands#list
LPUSH queue1 orange
LPUSH queue1 green
LPUSH queue 1 blue
RPUSH queue...
Redis Lists Example - https://redis.io/commands#list
blue green orange .. ..
RPOPLPUSH queue1 queue2
red
LPUSH queue1 oran...
Managing Tags w/ Redis Sets
Managing Tags Example
• The Problem
• Loads of tags
• Find items with particular tags
• High speed/low latency required
Al...
Redis Sets Example – https://redis.io/commands#set
tag1 tag22 tag24 tag28
SADD article:1 tag:1 tag:22 tag:24 tag:28
SADD t...
Managing Leaderboards w/ Redis Sorted Sets
Leaderboard with Sorted Sets Example
• The Problem
• MANY users playing a game or
collecting points
• Display real-time le...
Redis Sorted Sets - https://redis.io/commands#sorted_set
ZADD game:1 10000 id:1
ZADD game:1 21000 id:2
ZADD game:1 34000 i...
Searching within a Geospatial Index
Search within a Geographic Area Example
• The Problem
• MANY moving items within a
geographical area
• Display real-time l...
Redis Geospatial Index - https://redis.io/commands#geo
GEOADD locations 37.25 13.916667 “Campobello di Licata”
GEOADD loca...
Fire and Forget with Pub/Sub
Fire and Forget with Pub/Sub
• The Problem
• Communicate with clients in real-
time
Why Redis Rocks
• PUBLISH
• SUBSCRIBE
Pub/Sub Demo: https://redis.io/commands#pubsub
$ redis-server
$ keys *
$ flushall
1. PUBLISH channel1 "hi1"
4. PUBLISH cha...
Reliable messaging with Redis Streams
Reliable messaging with Redis Streams
• The Problem
• Communicate with clients in real-
time without missing data
Why Redi...
Redis Streams Example - https://redis.io/commands#stream
redis-server
keys *
flushall
1. XADD mystream1 * greeting "hello ...
More examples
47
Redis Data Structures
1. Strings - Cache SQL queries, pages, fast data
2. Bitmaps - Store 1s/0s (Online/Offline) like P...
In-memory, Keys & Data Structures
Redis Keys – Ground Rules - https://redis.io/commands#sorted_set
• It’s all about the keys
• No Querying
• No Indexes
• No...
Thank you!
dave@redislabs.com
50
Upcoming SlideShare
Loading in …5
×

of

10 Ways to Scale Your Website Silicon Valley Code Camp 2019 Slide 1 10 Ways to Scale Your Website Silicon Valley Code Camp 2019 Slide 2 10 Ways to Scale Your Website Silicon Valley Code Camp 2019 Slide 3 10 Ways to Scale Your Website Silicon Valley Code Camp 2019 Slide 4 10 Ways to Scale Your Website Silicon Valley Code Camp 2019 Slide 5 10 Ways to Scale Your Website Silicon Valley Code Camp 2019 Slide 6 10 Ways to Scale Your Website Silicon Valley Code Camp 2019 Slide 7 10 Ways to Scale Your Website Silicon Valley Code Camp 2019 Slide 8 10 Ways to Scale Your Website Silicon Valley Code Camp 2019 Slide 9 10 Ways to Scale Your Website Silicon Valley Code Camp 2019 Slide 10 10 Ways to Scale Your Website Silicon Valley Code Camp 2019 Slide 11 10 Ways to Scale Your Website Silicon Valley Code Camp 2019 Slide 12 10 Ways to Scale Your Website Silicon Valley Code Camp 2019 Slide 13 10 Ways to Scale Your Website Silicon Valley Code Camp 2019 Slide 14 10 Ways to Scale Your Website Silicon Valley Code Camp 2019 Slide 15 10 Ways to Scale Your Website Silicon Valley Code Camp 2019 Slide 16 10 Ways to Scale Your Website Silicon Valley Code Camp 2019 Slide 17 10 Ways to Scale Your Website Silicon Valley Code Camp 2019 Slide 18 10 Ways to Scale Your Website Silicon Valley Code Camp 2019 Slide 19 10 Ways to Scale Your Website Silicon Valley Code Camp 2019 Slide 20 10 Ways to Scale Your Website Silicon Valley Code Camp 2019 Slide 21 10 Ways to Scale Your Website Silicon Valley Code Camp 2019 Slide 22 10 Ways to Scale Your Website Silicon Valley Code Camp 2019 Slide 23 10 Ways to Scale Your Website Silicon Valley Code Camp 2019 Slide 24 10 Ways to Scale Your Website Silicon Valley Code Camp 2019 Slide 25 10 Ways to Scale Your Website Silicon Valley Code Camp 2019 Slide 26 10 Ways to Scale Your Website Silicon Valley Code Camp 2019 Slide 27 10 Ways to Scale Your Website Silicon Valley Code Camp 2019 Slide 28 10 Ways to Scale Your Website Silicon Valley Code Camp 2019 Slide 29 10 Ways to Scale Your Website Silicon Valley Code Camp 2019 Slide 30 10 Ways to Scale Your Website Silicon Valley Code Camp 2019 Slide 31 10 Ways to Scale Your Website Silicon Valley Code Camp 2019 Slide 32 10 Ways to Scale Your Website Silicon Valley Code Camp 2019 Slide 33 10 Ways to Scale Your Website Silicon Valley Code Camp 2019 Slide 34 10 Ways to Scale Your Website Silicon Valley Code Camp 2019 Slide 35 10 Ways to Scale Your Website Silicon Valley Code Camp 2019 Slide 36 10 Ways to Scale Your Website Silicon Valley Code Camp 2019 Slide 37 10 Ways to Scale Your Website Silicon Valley Code Camp 2019 Slide 38 10 Ways to Scale Your Website Silicon Valley Code Camp 2019 Slide 39 10 Ways to Scale Your Website Silicon Valley Code Camp 2019 Slide 40 10 Ways to Scale Your Website Silicon Valley Code Camp 2019 Slide 41 10 Ways to Scale Your Website Silicon Valley Code Camp 2019 Slide 42 10 Ways to Scale Your Website Silicon Valley Code Camp 2019 Slide 43 10 Ways to Scale Your Website Silicon Valley Code Camp 2019 Slide 44 10 Ways to Scale Your Website Silicon Valley Code Camp 2019 Slide 45 10 Ways to Scale Your Website Silicon Valley Code Camp 2019 Slide 46 10 Ways to Scale Your Website Silicon Valley Code Camp 2019 Slide 47 10 Ways to Scale Your Website Silicon Valley Code Camp 2019 Slide 48 10 Ways to Scale Your Website Silicon Valley Code Camp 2019 Slide 49 10 Ways to Scale Your Website Silicon Valley Code Camp 2019 Slide 50
Upcoming SlideShare
What to Upload to SlideShare
Next
Download to read offline and view in fullscreen.

0 Likes

Share

Download to read offline

10 Ways to Scale Your Website Silicon Valley Code Camp 2019

Download to read offline

Redis has 10 different data structures (String, Hash, List, Set, Sorted Set, Bit Array, Bit Field, Hyperloglog, Geospatial Index, Streams) plus Pub/Sub and many Redis Modules. In this talk, Dave will give 10 examples of how to use these data structures to scale your website. I will start with the basics, such as a cache and User session management. Then I demonstrate user generated tags, leaderboards and counting things with hyberloglog. I will with a demo of Redis Pub/Sub vs Redis Streams which can be used to scale your Microservices-based architecture.

Related Books

Free with a 30 day trial from Scribd

See all
  • Be the first to like this

10 Ways to Scale Your Website Silicon Valley Code Camp 2019

  1. 1. Top 10 Ways to Scale with Redis OCT 2019 | DAVE NIELSEN
  2. 2. In-memory Multi-model Database 2
  3. 3. Optionally Persistent 3
  4. 4. 4 50,420,000 41,190,000 30,200,000 21,120,000 11,380,000 5,020,000 3 6 12 18 24 26 ops/sec # ofnodes ClusterThroughput(@ 1 msec Latency)
  5. 5. 5 Developers + Redis
  6. 6. Redis Top Differentiators Simplicity ExtensibilityPerformance NoSQL Benchmark 1 Redis Data Structures 2 3 Redis Modules 6 Lists Hashes Bitmaps Strings Bit field Streams Hyperloglog Sorted Sets Sets Geospatial Indexes
  7. 7. Performance: The Most Powerful Database Highest Throughput at Lowest Latency in High Volume of Writes Scenario Least Servers Needed to Deliver 1 Million Writes/Sec Benchmarks performed by Avalon Consulting Group Benchmarks published in the Google blog 7 1 Serversusedtoachieve1Mwrites/sec
  8. 8. “REDIS IS FULL OF DATA STRUCTURES!” 2 Simplicity: Data Structures - Redis’ Building Blocks
  9. 9. Simplicity: Redis Data Structures – ’Lego’ Building Blocks Lists [ A → B → C → D → E ] Hashes { A: “foo”, B: “bar”, C: “baz” } Bitmaps 0011010101100111001010 Strings "I'm a Plain Text String!” Bit field {23334}{112345569}{766538} Key 9 2 ”Retrieve the e-mail address of the user with the highest bid in an auction that started on July 24th at 11:00pm PST” ZREVRANGE 07242015_2300 0 0= Streams {id1=time1.seq1(A:“xyz”, B:“cdf”), d2=time2.seq2(D:“abc”, )} Hyperloglog 00110101 11001110 Sorted Sets { A: 0.1, B: 0.3, C: 100 } Sets { A , B , C , D , E } Geospatial Indexes { A: (51.5, 0.12), B: (32.1, 34.7) }
  10. 10. • Add-ons that use a Redis API to seamlessly support additional use cases and data structures. • Enjoy Redis’ simplicity, super high performance, infinite scalability and high availability. Extensibility: Modules Extend Redis Infinitely • Any C/C++/Go program can become a Module and run on Redis. • Leverage existing data structures or introduce new ones. • Can be used by anyone; Redis Enterprise Modules are tested and certified by Redis Labs. • Turn Redis into a Multi-Model database 10 3
  11. 11. • RediSearch • RedisTimerseries • ReJSON • Redis Graph • Rebloom • Neural-Redis • Redis-Cell • Redis-Tdigest • Redis-ML Extensibility: Modules Extend Redis Infinitely 11 3 • Redis-Rating • Redis-Cuckoofilter • Cthulhu • Redis Snowflake • redis-roaring • Session Gate • ReDe • TopK • countminsketch
  12. 12. Deep Dive
  13. 13. Real Time Analytics User Session Store Real Time Data Ingest High Speed Transactions Job & Queue Management Time Series Data Complex Statistical Analysis Notifications Distributed Lock Caching Geospatial Data Streaming Data Machine Learning 13 Very Large Data Sets Search
  14. 14. Manage Session Stores w/ Redis Hash
  15. 15. • An chunk of data that is connected to one “user” of a service –”user” can be a simple visitor –or proper user with an account • Often persisted between client and server by a token in a cookie* –Cookie is given by server, stored by browser –Client sends that cookie back to the server on subsequent requests –Server associates that token with data • Often the most frequently used data by that user –Data that is specific to the user –Data that is required for rendering or common use • Often ephemeral and duplicated A user session store is…
  16. 16. Session Storage Uses Cases Traditional • Username • Preferences • Name • “Stateful” data Intelligent • Traditional + • Notifications • Past behavior –content surfacing –analytical information –personalization
  17. 17. In a simple world Internet Server Database
  18. 18. Good problems Internet Server Database Traffic Grows… Struggles
  19. 19. Good solution Internet Server Database performance restored Session storage on the server
  20. 20. More good problems Internet Server Database Session storage on the server Struggling
  21. 21. Problematic Solutions Internet Server Database Session storage on the server Load balanced Session storage on the server
  22. 22. Multiple Servers + On-server Sessions? Server DatabaseRobin Server #1 – Hello Robin!
  23. 23. Multiple Servers + On-server Sessions? Server DatabaseRobin Server #3 – Hello ????
  24. 24. Better solution Internet Server Database Load balanced Redis Session Storage
  25. 25. User Session Store • The Problem • Maintain session state across multiple servers • Multiple session variables • High speed/low latency required Why Redis Rocks • Hashes are perfect for this! • HSET lets you save session variables as key/value pairs • HGET to retrieve values • HINCRBY to increment any field within the hash structure
  26. 26. Redis Hashes Example - https://redis.io/commands#hash userid 8754 name dave ip 10:20:104:31 hits 1 lastpage home hash key: usersession:1 HMSET usersession:1 userid 8754 name dave ip 10:20:104:31 hits 1 HMGET usersession:1 userid name ip hits HINCRBY usersession:1 hits 1 HSET usersession:1 lastpage “home” HGET usersession:1 lastpage HDEL usersession:1 lastpage Hashes store a mapping of keys to values – like a dictionary or associative array – but faster EXPIRE usersession:1 10 DEL usersession:1 or
  27. 27. Managing Queues w/ Redis Lists
  28. 28. Managing Queues of Work • The Problem • Tasks need to be worked on asynch to reduce block/wait times • Lots of items to be worked on • Assign items to worker process and remove from queue at the same time • Similar to buffering high speed data- ingestion Why Redis Rocks • Lists are perfect for this! • LPUSH, RPUSH add values at beginning or end of queue • RPOPLPUSH – pops an item from one queue and pushes it to another queue
  29. 29. Redis Lists Example - https://redis.io/commands#list LPUSH queue1 orange LPUSH queue1 green LPUSH queue 1 blue RPUSH queue1 red LPUSH adds values to head of list RPUSH adds value to tail of list blue green orange .. .. red
  30. 30. Redis Lists Example - https://redis.io/commands#list blue green orange .. .. RPOPLPUSH queue1 queue2 red LPUSH queue1 orange LPUSH queue1 green LPUSH queue1 blue RPUSH queue1 red RPOPLPUSH pops a value from one list and pushes it to another list LLEN queue1 LINDEX queue1 0 LRANGE queue1 0 2
  31. 31. Managing Tags w/ Redis Sets
  32. 32. Managing Tags Example • The Problem • Loads of tags • Find items with particular tags • High speed/low latency required Also used for: • Recommending Similar Purchases • Recommending Similar Posts Why Redis Rocks • Sets are unique collections of strings • SADD to add tags to each article • SISMEMBER to check if an article has a given tag • SMEMBERS to get all the tags for an article • SINTER to find which articles are tagged with tag1, tag2 and tag77
  33. 33. Redis Sets Example – https://redis.io/commands#set tag1 tag22 tag24 tag28 SADD article:1 tag:1 tag:22 tag:24 tag:28 SADD tag:1 article:1 SADD tag:1 article:3 SADD tag:2 article:22 SADD tag:2 article:14 SADD tag:2 article:3 SISMEMBER article:1 tag:1 SMEMBERS article:1 article 1 article 1 article 3 …. tag 1 article 3 article 14 article 22 .. tag 2 SINTER tag:1 tag:2
  34. 34. Managing Leaderboards w/ Redis Sorted Sets
  35. 35. Leaderboard with Sorted Sets Example • The Problem • MANY users playing a game or collecting points • Display real-time leaderboard. • Who is your nearest competition • Disk-based DB is too slow Why Redis Rocks • Sorted Sets are perfect! • Automatically keeps list of users sorted by score • ZADD to add/update • ZRANGE, ZREVRANGE to get user • ZRANK will get any users rank instantaneously
  36. 36. Redis Sorted Sets - https://redis.io/commands#sorted_set ZADD game:1 10000 id:1 ZADD game:1 21000 id:2 ZADD game:1 34000 id:3 ZADD game:1 35000 id:4 ZADD game:1 44000 id:3 or ZINCRBY game:1 10000 id:3 34000 id:3 35000 id:4 21000 id:2 10000 id:1 ZREVRANGE game:1 0 0 ZREVRANGE game:1 0 1 WITHSCORES 44000 id:3 + 10000id:3
  37. 37. Searching within a Geospatial Index
  38. 38. Search within a Geographic Area Example • The Problem • MANY moving items within a geographical area • Display real-time locations. • What is the nearest item NOW • SQL Queries are too slow Why Redis Rocks • GEOADD to add an item • Redis Geo uses Sorted Sets so related Z-commands such as ZRANGE & ZREM are useful too • GEODIST to find distance between to points • GEORADIUS to find all points within an area
  39. 39. Redis Geospatial Index - https://redis.io/commands#geo GEOADD locations 37.25 13.916667 “Campobello di Licata” GEOADD locations 41.9 12.5 Rome GEOADD locations -122.375 37.618889 SFO GEOADD locations -122.3931400 37.7681300 Redisconf ZRANGE locations 0 -1 Rome 41.9, 12.5 Campobello di Licata 37.25, 13.916667 SFO -122.375 37.618889 RedisConf -122.3931400 37.7681300 GEODIST locations ”Campobello di Licata” Redisconf mi GEOPOS locations Redisconf SFO GEOHASH locations Redisconf ZSCORE locations Redisconf GEORADIUS locations -122.41942 37.77493 15 mi GEORADIUSBYMEMBER locations Redisconf 15 mi WITHDIST ASC ZREM locations SFO
  40. 40. Fire and Forget with Pub/Sub
  41. 41. Fire and Forget with Pub/Sub • The Problem • Communicate with clients in real- time Why Redis Rocks • PUBLISH • SUBSCRIBE
  42. 42. Pub/Sub Demo: https://redis.io/commands#pubsub $ redis-server $ keys * $ flushall 1. PUBLISH channel1 "hi1" 4. PUBLISH channel1 "hello2" 6. PUBLISH channel1 "hellooo 3" 9. PUBLISH channel1 "hello world 4" 3. SUBSCRIBE channel12. SUBSCRIBE channel1 5. CONTROL C 7. redis-cli 8. SUBSCRIBE channel1 Will receive: 4. hello2 9. hello world 4 Will receive: 4. hello2 6. hellooo 3 9. hello world 4 Setup Publisher Subscriber 1 Subscriber 2
  43. 43. Reliable messaging with Redis Streams
  44. 44. Reliable messaging with Redis Streams • The Problem • Communicate with clients in real- time without missing data Why Redis Rocks • XADD • XREAD
  45. 45. Redis Streams Example - https://redis.io/commands#stream redis-server keys * flushall 1. XADD mystream1 * greeting "hello 1" 5. XADD mystream1 * greeting "hello 2" 8. XADD mystream1 * greeting ”hello 3" 3. XREAD COUNT 10 STREAMS mystream1 0 7. XREAD COUNT 10 STREAMS mystream1 0 10. XREAD COUNT 10 STREAMS mystream1 0 2. XREAD COUNT 10 STREAMS mystream1 0 4. CONTROL C 6. redis-cli 9. XREAD COUNT 10 STREAMS mystream1 0 Will receive: 1. hello 1 5. hello 2 8. hello 3 Will receive: 1. hello 1 5. hello 2 8. hello 3 Setup Producer Consumer 1 Consumer 2
  46. 46. More examples
  47. 47. 47 Redis Data Structures 1. Strings - Cache SQL queries, pages, fast data 2. Bitmaps - Store 1s/0s (Online/Offline) like Pinterest 3. Bit Fields - Store arrays of complex numbers 4. Hashes - Store record-like info (user sessions, favorites, etc.) 5. Lists - Store ordered data (for ingesting data) 6. Sets - Store unique/unordered data (tags, unique IPs, etc.) 7. Sorted Sets - Store real-time sorted data like a MMPORG Leaderboard 8. Geospacial Indexes - Store real-time location (Uber, Lyft, etc.) 9. Hyperloglog - Store probabilistic data (Google Search count) 10. Streams - Store and share event data (similar to Kafka) Lists Hashes Bitmaps Strings Bit fields Streams Hyperloglog Sorted Sets Sets Geospatial Indexes
  48. 48. In-memory, Keys & Data Structures
  49. 49. Redis Keys – Ground Rules - https://redis.io/commands#sorted_set • It’s all about the keys • No Querying • No Indexes • No Schemas • Keys must be unique (aka primary key for the database)
  50. 50. Thank you! dave@redislabs.com 50

Redis has 10 different data structures (String, Hash, List, Set, Sorted Set, Bit Array, Bit Field, Hyperloglog, Geospatial Index, Streams) plus Pub/Sub and many Redis Modules. In this talk, Dave will give 10 examples of how to use these data structures to scale your website. I will start with the basics, such as a cache and User session management. Then I demonstrate user generated tags, leaderboards and counting things with hyberloglog. I will with a demo of Redis Pub/Sub vs Redis Streams which can be used to scale your Microservices-based architecture.

Views

Total views

181

On Slideshare

0

From embeds

0

Number of embeds

0

Actions

Downloads

7

Shares

0

Comments

0

Likes

0

×