Performance tuning

1. JON HADDAD PRINCIPAL CONSULTANT, THE LAST PICKLE CASSANDRA PERFORMANCE TUNING

2. WHY AM I HERE?

3. ACT 1: MEASURE

4. THROUGHPUT, LATENCY, ERROR RATE

5. REQUESTS / SECOND

6. 1MS P99

7. WE NEED A METHODOLOGY

8. OPERATING SYSTEM

9. WHAT’S THE BOTTLENECK?

10. DISK IO NETWORK IO CPU USAGE

12. CASSANDRA HAS METRICS

13. JMX

14. NODETOOL

15. 15 $ nodetool tablehistograms movielens ratings_by_user movielens/ratings_by_user histograms Percentile SSTables Write Latency Read Latency Partition Size Cell Count (micros) (micros) (bytes) 50% 0.00 17.08 0.00 3973 215 75% 0.00 20.50 0.00 9887 446 95% 0.00 29.52 0.00 20501 1109 98% 0.00 42.51 0.00 24601 1331 99% 0.00 51.01 0.00 29521 1597 Min 0.00 5.72 0.00 925 51 Max 0.00 43388.63 0.00 51012 2299

16. 15 $ nodetool tablehistograms movielens ratings_by_user movielens/ratings_by_user histograms Percentile SSTables Write Latency Read Latency Partition Size Cell Count (micros) (micros) (bytes) 50% 0.00 17.08 0.00 3973 215 75% 0.00 20.50 0.00 9887 446 95% 0.00 29.52 0.00 20501 1109 98% 0.00 42.51 0.00 24601 1331 99% 0.00 51.01 0.00 29521 1597 Min 0.00 5.72 0.00 925 51 Max 0.00 43388.63 0.00 51012 2299

18. VISUALIZE METRICS

20. ANALYZE CALL TREES AND INTERNALS

21. JAVA FLIGHT RECORDER

22. YOURKIT

24. FLAME GRAPHS

26. ACT 2: COMMON OPTIMIZATIONS

27. READ HEAVY? FIX YOUR READ AMPLIFICATION

28. COMPRESSION

29. DEFAULTS ARE MEH

30. CHUNK SIZE (64K)

31. 30 $ nodetool cfhistograms movielens movies movielens/movies histograms Percentile SSTables Write Latency Read Latency Partition Size Cell Count (micros) (micros) (bytes) 50% 1.00 0.00 545.79 149 7 75% 1.00 0.00 654.95 179 7 95% 1.00 0.00 654.95 215 8 98% 1.00 0.00 654.95 215 10 99% 1.00 0.00 654.95 258 10 Min 1.00 0.00 379.02 61 6 Max 1.00 0.00 654.95 310 12

32. 30 $ nodetool cfhistograms movielens movies movielens/movies histograms Percentile SSTables Write Latency Read Latency Partition Size Cell Count (micros) (micros) (bytes) 50% 1.00 0.00 545.79 149 7 75% 1.00 0.00 654.95 179 7 95% 1.00 0.00 654.95 215 8 98% 1.00 0.00 654.95 215 10 99% 1.00 0.00 654.95 258 10 Min 1.00 0.00 379.02 61 6 Max 1.00 0.00 654.95 310 12

33. 250X

34. compression = {'chunk_length_in_kb': '4', ‘class’: ‘org.apache.cassandra.io.compress.LZ4Compressor’}

35. READ-AHEAD

36. 34 $ blockdev --report RO RA SSZ BSZ StartSec Size Device ro 256 512 1024 0 91176960 /dev/loop0 ro 256 512 1024 0 87863296 /dev/loop1 ro 256 512 1024 0 91176960 /dev/loop2 rw 256 512 4096 0 6001175126016 /dev/sda

37. 34 $ blockdev --report RO RA SSZ BSZ StartSec Size Device ro 256 512 1024 0 91176960 /dev/loop0 ro 256 512 1024 0 87863296 /dev/loop1 ro 256 512 1024 0 91176960 /dev/loop2 rw 256 512 4096 0 6001175126016 /dev/sda

38. 256K?

39. REMINDER: 258 BYTES PER PARTITION

40. 1000X

41. TURN IT OFF

42. 320K PER SSTABLE ACCESS

43. DISABLE DYNAMIC SNITCH

44. 41

45. 41

46. COMPACTION

47. TWCS (NEVER DTCS)

48. 44 Today Yesterday 2 Days Ago

50. ROW CACHE

51. 47 $ nodetool info ID : b07fa110-7deb-49aa-a020-22e1540d0f5c Gossip active : true Native Transport active: true Load : 6.03 MiB Generation No : 1516820539 Uptime (seconds) : 586 Heap Memory (MB) : 67.40 / 4016.00 Off Heap Memory (MB) : 0.01 Data Center : datacenter1 Rack : rack1 Exceptions : 0 Key Cache : entries 26, size 2.35 KiB, capacity 100 MiB, 113 hits, 140 requests, 0.807 recent hit rate, 14400 save period in seconds Row Cache : entries 1, size 346 bytes, capacity 100 MiB, 1 hits, 2 requests, 0.500 recent hit rate, 0 save period in seconds Counter Cache : entries 0, size 0 bytes, capacity 50 MiB, 0 hits, 0 requests, NaN recent hit rate, 7200 save period in seconds Chunk Cache : entries 33, size 2.06 MiB, capacity 480 MiB, 47 misses, 264 requests, 0.822 recent hit rate, NaN microseconds miss latency Percent Repaired : 100.0% Token : (invoke with -T/--tokens to see all 256 tokens)

52. 47 $ nodetool info ID : b07fa110-7deb-49aa-a020-22e1540d0f5c Gossip active : true Native Transport active: true Load : 6.03 MiB Generation No : 1516820539 Uptime (seconds) : 586 Heap Memory (MB) : 67.40 / 4016.00 Off Heap Memory (MB) : 0.01 Data Center : datacenter1 Rack : rack1 Exceptions : 0 Key Cache : entries 26, size 2.35 KiB, capacity 100 MiB, 113 hits, 140 requests, 0.807 recent hit rate, 14400 save period in seconds Row Cache : entries 1, size 346 bytes, capacity 100 MiB, 1 hits, 2 requests, 0.500 recent hit rate, 0 save period in seconds Counter Cache : entries 0, size 0 bytes, capacity 50 MiB, 0 hits, 0 requests, NaN recent hit rate, 7200 save period in seconds Chunk Cache : entries 33, size 2.06 MiB, capacity 480 MiB, 47 misses, 264 requests, 0.822 recent hit rate, NaN microseconds miss latency Percent Repaired : 100.0% Token : (invoke with -T/--tokens to see all 256 tokens)

53. COUNTERS

54. 49 $ nodetool info … Counter Cache : entries 1, size 112 bytes, capacity 50 MiB, 15 hits, 16 requests, 0.938 recent hit rate, 7200 save period in seconds …

55. ACT 3: JVM TUNING (CMS + PARNEW)

56. 51 Eden OldS0 S1

60. COPYING / PROMOTION IS SLOW

61. WRITE HEAVY WORKLOADS

62. READ HEAVY WORKLOADS

63. QUESTIONS?

Performance tuning

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Performance tuning

Similar to Performance tuning (20)

More from Jon Haddad

More from Jon Haddad (17)

Recently uploaded

Recently uploaded (20)

Performance tuning