Download as PDF, PPTX
![KEYS – Iterating all Keys
di = dictGetSafeIterator(c->db->dict);
allkeys = (pattern[0] == '*' && pattern[1] == '0');
while((de = dictNext(di)) != NULL) {
……
stringmatchlen(pattern,plen,key,sdslen(key),0)
}](https://image.slidesharecdn.com/troubleshootingredis-160515111926/85/Troubleshooting-redis-17-320.jpg)
![FlushAll – Deleting all items
for (i = 0; i < ht->size && ht->used > 0; i++) {
dictEntry *he, *nextHe;
if ((he = ht->table[i]) == NULL) continue;
while(he) {
nextHe = he->next;
dictFreeKey(d, he);
dictFreeVal(d, he);
zfree(he);
ht->used--;
he = nextHe;
}
}](https://image.slidesharecdn.com/troubleshootingredis-160515111926/85/Troubleshooting-redis-18-320.jpg)
Uploaded byDaeMyung Kang
Troubleshooting redis
This document discusses troubleshooting Redis. Some key points: - Redis is single-threaded, so commands like KEYS, FlushAll, and deleting large collections can be slow. It's better to use SCAN instead of KEYS. - Creating Redis database snapshots (RDB files) and rewriting the append-only file (AOF) can cause high disk I/O and CPU usage. It's best to disable automatic rewrites. - Monitoring memory usage and fragmentation is important to avoid performance issues. The maxmemory setting also needs monitoring to prevent out-of-memory errors. - Network and replication failures need solutions like DNS failover or using Zookeeper for coordination to maintain high availability of Redis
More Related Content
![[2018] MySQL 이중화 진화기](https://cdn.slidesharecdn.com/ss_thumbnails/cloudinfra03-190131073325-thumbnail.jpg?width=640&height=640&fit=bounds)
![[135] 오픈소스 데이터베이스, 은행 서비스에 첫발을 내밀다.](https://cdn.slidesharecdn.com/ss_thumbnails/35-171016061446-thumbnail.jpg?width=640&height=640&fit=bounds)
Troubleshooting redis
- 1.
- 2. About me •Senior SoftwareEngineer in KAKAO •Redis/Twemproxy Contributor •Redis-doc project merger. •Apache Tajo Commiter
- 3.
- 4.
- 5.
- 6. Kakaostory Service Stack •ForStorage •MariaDB(Master/Slave for HA) •Hbase •Cassandra •For Cache •Redis •Arcus • (Memcached variant, opensource, supporting collections)
- 7. Redis 5.2TB, 274 Servers (Arcus:3.3TB, 137 Servers)
- 8. Why Redis? •As lookasideCache for service data •Example) •User Profile Information •Feeds •Activities •Friends •Notifications
- 9. Agenda •Single Threaded •Memory Fragmentation •RedisTroubleshooting cases •Redis Monitoring •Redis HA
- 10.
- 11. Redis Event Loop Client#1 Client #2 …… Client #N Redis Event Loop I/O Multiplexing Process Command command #1 command #2
- 12.
- 13.
- 14.
- 15.
- 16.
- 17. KEYS – Iteratingall Keys di = dictGetSafeIterator(c->db->dict); allkeys = (pattern[0] == '*' && pattern[1] == '0'); while((de = dictNext(di)) != NULL) { …… stringmatchlen(pattern,plen,key,sdslen(key),0) }
- 18. FlushAll – Deletingall items for (i = 0; i < ht->size && ht->used > 0; i++) { dictEntry *he, *nextHe; if ((he = ht->table[i]) == NULL) continue; while(he) { nextHe = he->next; dictFreeKey(d, he); dictFreeVal(d, he); zfree(he); ht->used--; he = nextHe; } }
- 19.
- 20. Command Item CountTime flushall 1,000,000 1000ms(1 second) FlushAll
- 21. Delete collections Item CountTime list 1,000,000 1000ms(1 second) set 1,000,000 1000ms(1 second) Sorted set 1,000,000 1000ms(1 second) hash 1,000,000 1000ms(1 second) You can use Xscan commands from 2.8.x
- 22. Using Multiple Instances ina Physical Server (can use more cpus)
- 23. Fork for Creating RDB, AOFRewrite
- 24. Maximum 2x Memory DiskIO CPU Load/Usage
- 25. CPU 4 core,32G Memory Mem: 24G Mem: 8G Mem: 8G Mem: 8G more Reliable
- 26. Set CPU Affinity usingtaskset
- 27. Divide NIC InterruptCPU and Redis Process CPU
- 28.
- 29.
- 30. Memory Fragmentation #2 Used_memoryRSS Starting to use Arcus at this case
- 31.
- 32.
- 33.
- 34.
- 35.
- 36.
- 37.
- 38.
- 39.
- 40.
- 41.
- 42. Problem #2 All WriteCommands Fail
- 43. “MISCONF Redis isconfigured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.”
- 44. Reason if (((server.stop_writes_on_bgsave_err && server.saveparamslen> 0 && server.lastbgsave_status == C_ERR) || server.aof_last_write_status == C_ERR) && server.masterhost == NULL && (c->cmd->flags & CMD_WRITE || c->cmd->proc == pingCommand)) { … }
- 45.
- 46. Problem #3 Using DefaultOption
- 47.
- 48. SAVE 900 1 SAVE300 10 SAVE 60 10000
- 49. Heavy Disk IO HighCpu Load with creating RDB
- 50.
- 51. Problem #4 Using SwapMemory
- 52. Redis using 28G onsingle 32G machine
- 53.
- 54. Monitor Redis Server andkeep within bounds
- 55. Problem #5 Simultaneous AOFRewrite
- 56. A 256GB SingleMachine Redis 26GB Redis 26GB Redis 26GB Redis 26GB Redis 26GB Redis 26GB Redis 26GB Redis 26GB
- 57. Simultaneous AOF Rewrite Redis 26GB Redis 26GB Redis 26GB Redis 26GB Redis 26GB Redis 26GB Redis 26GB Redis 26GB AOFRewrite AOF Rewrite AOF Rewrite AOF Rewrite AOF Rewrite AOF Rewrite AOF Rewrite AOF Rewrite
- 58. Stop all AOFRewrites
- 59. Turn off Automatic AOFRewrite
- 60.
- 61. Manually Run AOFRewrite
- 62. Problem #6 Replication isBroken with Network Line Failure
- 63. All redis replication arebroken by Network line failure
- 64. What Happens if network isrecovered
- 65.
- 66. All slaves automatically tryto reconnect to master.
- 67. Slave of noone
- 68.
- 69.
- 70. Memory Allocation Fail sysctlvm.overcommit_memory=1
- 71.
- 72.
- 73. config set client-output-buffer-limit "slave1024mb 1024mb 60"
- 74. Problem #8 Hash TableExpansion
- 75. Redis Dict –Hash Table Expansion #1
- 76. Redis Dict –Hash Table Expansion #2
- 77. Redis Dict –Hash Table Expansion #3
- 78.
- 79.
- 80.
- 81.
- 82.
- 83. Hash Table Expansion isneeded.
- 84. 4G * 2= 8G. You need 20G(12G + 8G)
- 85.
- 86. Need a featurethat can Set Initial size of Hash Table (Not Supported) https://github.com/antirez/redis/pull/2812
- 87.
- 88. Monitoring is important asmuch as Management
- 89. Redis Monitoring Metrics FactorSystem or Redis Info CPU Usage, Load System Network Inbound/outbound System Client connections Maxclient setting Info Key size Processed commands Redis Memory Usage, RSS(very Important) Redis Disk Usage, IO System Expired Keys, Evicted Keys Redis
- 90.
- 91. Using DNS forFailover
- 92.
- 93. DNS HA Flow DetectA Redis Failure Change B can write Change DNS A with B Send A Client Kill New clients Will connect to B B Config rewrite
- 94.
- 95.
- 96.
- 97.
- 98. Zookeeper with RedisInformation
- 99. Zookeeper with Redis ApplicationServers ZooKeeper Redis Shard-1 Redis Shard-2 Redis Shard-3 Redis Cluster Monitor Get Redis Shard Information Health Check Update Shard Info Event: Node Add or Remove, Master change
- 100. Summary •Redis is SingleThreaded •Creating RDB or AOF Rewrite is expensive •Don’t use KEYS command. •Don’t use default redis configuration. •Monitoring is very importatnt.
- 101.