This document summarizes how LINE messaging uses Redis. It discusses:
- How LINE has scaled Redis from 3 nodes in 2011 to over 14,000 nodes today to support over 25 billion messages per day.
- The key ways LINE uses Redis, including for storing sequences, caches, secondary indexes, and local queues.
- Challenges LINE faced in scaling their in-house Redis cluster to over 1,000 nodes and workarounds they developed.
- How LINE monitors Redis for slow commands, bursting operations, and connections to address performance issues.
- Future work LINE is doing with Redis 4 and improving latency and scalability.
2. About me
● : (Shunsuke Nakamura)
● : LINE LINE 1 7
● : Redis team lead, messaging server tech lead
●
● : LINE HBase scalability
● :
● Storage infrastructure using HBase behind LINE messages
https://www.slideshare.net/naverjapan/storage-infrastructure-using-hbase-behind-line-
messages
● HBase Redis 100 / LINE
https://www.slideshare.net/linecorp/a-5-47983106
3. Redis Talk for LINE messaging
●
● RedisConf18 @ San Francisco
● Redis at LINE, 25 Billion Messages Per Day
https://www.slideshare.net/RedisLabs/redisconf18-redis-at-
line-25-billion-messages-per-day (open today!)
● Tech Planet 2015 @ Korea
● LINE Redis Cluster
https://www.slideshare.net/lovejinstar/redis-at-line-tech-
planet-2015
●
4. Agenda
● How LINE messaging uses Redis
● Redis details for LINE messaging
● Challenge to Redis3.2 official cluster
● Redis hotspots daily handling
5. ● 2011 LINE messaging Redis
● messaging in-memory
● Redis 3 x2 (master/slave) client side sharding
Redis for LINE messaging (As of 2011.6)
7. ● sequences
● user/group/message sequence ID
● event revision
● caches
● message event TTL time series data
● immutable read heavy data
LINE is powered by Redis
● storages
● secondary index (ex. follower list)
● CAS (ex. unread badge count)
● local queue in each API servers
● API server async task ( IO )
● IO RPC context
8. 1.App. calls sendMessage thrift API
2.Acquire messageId from Redis sequence
3.Store Message, Event into Redis caches
1.Store into HBase storage
2.Enqueue task to local Redis queue if failed
4.Check receiver info with Redis storages
5.Deliver and notify event to receivers
How messaging
uses Redis
G a t e w a y
A P I s e r v e r s
…
queue
… …
… … …
1. sendMessage
sequence
caches
storages
2.acquire
messageId
3.store
Message
4. check
receiver info
5. notify
9. Redis server at LINE
● Redis version 2.8 ~ 3.2
●
● Multiple nodes per host
● 10 Gbps network
● Redis node NIC interrupt CPU affinity
● Redis node
● standalone : master slave 1 1, host role slave
● disk less : No BGSAVE backup, No AOF, No Virtual Memory
● non HA : No Sentinel, No slave read
10. In-house cluster at LINE
● 3.0 official cluster (2011~2012)
● Proxy-less client side sharding
● ZooKeeper: shard
● Cluster Manager Server:
● LINE Redis Client
● ZooKeeper
● Redis key hash (MurmurHash3)
shard
● shard master Redis command
11. Redis client at LINE
● Jedis (Sync) or Lettuce (Async) Java client
● commands ser/des template
● Redis command client side metrics
● Availability
● Back pressure: ZooKeeper
● Circuit breaker: fail fast
● read Replicated cluster
12. Replicated clusters
● replica Redis cluster cache read
● client cluster random
origin storage fallback
(read-through cache)
● : LINE official account (followers1 )
account
13. Recent works in LINE Redis team
● Redis3.2 Official cluster
● Async Redis client
● Redis
14. Challenge to Redis3.2 official cluster
● : In-house cluster 7 ...
● dynamic resizing
● cluster / 2 cluster migration
● Redis OSS
● REAL cache cluster 3.2 cluster
● in-house cluster client Jedis Cluster client
● REAL service crush test
●
=> Redis3.2 cluster
how to resize in-house cluster
15. 3 issues of Redis3.2 cluster for LINE
● Gossip traffic 1,000 nodes
● Redis official document
“High performance and linear scalability up to 1,000 nodes.”
https://redis.io/topics/cluster-spec
● => LINE in-house cluster = 1,400 master nodes
● gossip issue: https://github.com/antirez/redis/issues/3929
● PSYNC1 [#2]
● => Redis
● clustering memory overhead [#3]
● => memory bound cluster