SlideShare a Scribd company logo
1 of 59
Cassandra troubleshooting:
    out of the shadows

      Benjamin Black, b@b3k.us
Introducing: This Guy
The Allegory of the Cave
Most people start
troubleshooting problems
interpreting shadows on the
wall.
Common shadows.
Paths out of the cave.
Combination of
basic system tools
&
nodetool/JMX
I’m using RP.
My ring is very unbalanced.
I’m using RP.
My ring is very unbalanced.




                   WTF?
nodetool ring
Address   Status   Load       Range                  Ring
                      148873535527910577765226390751398592511
10.248.54.192 Up      5.59 GB                       0 |<--|
10.248.254.15 Up     10.58 GB     42535295865117307932921825928971026431 | ^
10.248.135.239Up      11.01 GB     85070591730234615865843651857942052863 v |
10.248.223.191Up       5.42 GB   106338239662793269832304564822427566079 | ^
10.248.122.240Up       5.51 GB   127605887595351923798765477786913079295 v |
10.248.34.80 Up      5.45 GB    148873535527910577765226390751398592511 |-->|
Address   Status   Load       Range                  Ring
                      148873535527910577765226390751398592511
10.248.54.192 Up      5.59 GB                       0 |<--|
10.248.254.15 Up     10.58 GB     42535295865117307932921825928971026431 | ^
10.248.135.239Up      11.01 GB     85070591730234615865843651857942052863 v |
10.248.223.191Up       5.42 GB   106338239662793269832304564822427566079 | ^
10.248.122.240Up       5.51 GB   127605887595351923798765477786913079295 v |
10.248.34.80 Up      5.45 GB    148873535527910577765226390751398592511 |-->|
Autobootstrap
+
Automatic token assignment
Automatic token algorithm:

Assign a token that will give me
half the range of
the most loaded node.
32
16 16
8 8 16
8888
44888
444488
Address   Status   Load       Range                  Ring
                      148873535527910577765226390751398592511
10.248.54.192 Up      5.59 GB                       0 |<--|
10.248.254.15 Up     10.58 GB     42535295865117307932921825928971026431 | ^
10.248.135.239Up      11.01 GB     85070591730234615865843651857942052863 v |
10.248.223.191Up       5.42 GB   106338239662793269832304564822427566079 | ^
10.248.122.240Up       5.51 GB   127605887595351923798765477786913079295 v |
10.248.34.80 Up      5.45 GB    148873535527910577765226390751398592511 |-->|
nodetool move
+
Manual token assignment
0-(2**127 - 1)
def tokens(nodes)
 0.upto(nodes - 1) do |n|
  p (n * (2**127 - 1) / nodes)
 end
end
=> tokens(6)
0
283568639100782052886145506193140
17621
567137278201564105772291012386280
35242
850705917302346158658436518579420
52863
113427455640312821154458202477256
070484
141784319550391026443072753096570
088105
YES:

This means you need to change tokens on
most of the nodes in your cluster whenever
you add a node.
Writes are fast.
Reads keep getting slower.
Writes are fast.
Reads keep getting slower.




                   WTF?
iostat -x
look at %util
nodetool tpstats
Pool Name                    Active   Pending      Completed
STREAM-STAGE                      0         0              0
RESPONSE-STAGE                    0         0         516280
ROW-READ-STAGE                    8      4096        1164326
LB-OPERATIONS                     0         0              0
MESSAGE-DESERIALIZER-POOL         1    682008        1818682
GMFD                              0         0           6467
LB-TARGET                         0         0              0
CONSISTENCY-MANAGER               0         0         661477
ROW-MUTATION-STAGE                0         0         998780
MESSAGE-STREAMING-POOL            0         0              0
LOAD-BALANCER-STAGE               0         0              0
FLUSH-SORTER-POOL                 0         0              0
MEMTABLE-POST-FLUSHER             0         0              4
FLUSH-WRITER-POOL                 0         0              4
AE-SERVICE-STAGE                  0         0              0
HINTED-HANDOFF-POOL               0         0              3
Pool Name                    Active   Pending      Completed
STREAM-STAGE                      0         0              0
RESPONSE-STAGE                    0         0         516280
ROW-READ-STAGE                    8      4096        1164326
LB-OPERATIONS                     0         0              0
MESSAGE-DESERIALIZER-POOL         1    682008        1818682
GMFD                              0         0           6467
LB-TARGET                         0         0              0
CONSISTENCY-MANAGER               0         0         661477
ROW-MUTATION-STAGE                0         0         998780
MESSAGE-STREAMING-POOL            0         0              0
LOAD-BALANCER-STAGE               0         0              0
FLUSH-SORTER-POOL                 0         0              0
MEMTABLE-POST-FLUSHER             0         0              4
FLUSH-WRITER-POOL                 0         0              4
AE-SERVICE-STAGE                  0         0              0
HINTED-HANDOFF-POOL               0         0              3
Pool Name                    Active   Pending      Completed
STREAM-STAGE                      0         0              0
RESPONSE-STAGE                    0         0         516280
ROW-READ-STAGE                    8      4096        1164326
LB-OPERATIONS                     0         0              0
MESSAGE-DESERIALIZER-POOL         1    682008        1818682
GMFD                              0         0           6467
LB-TARGET                         0         0              0
CONSISTENCY-MANAGER               0         0         661477
ROW-MUTATION-STAGE                0         0         998780
MESSAGE-STREAMING-POOL            0         0              0
LOAD-BALANCER-STAGE               0         0              0
FLUSH-SORTER-POOL                 0         0              0
MEMTABLE-POST-FLUSHER             0         0              4
FLUSH-WRITER-POOL                 0         0              4
AE-SERVICE-STAGE                  0         0              0
HINTED-HANDOFF-POOL               0         0              3
Pool Name                    Active   Pending      Completed
STREAM-STAGE                      0         0              0
RESPONSE-STAGE                    0         0         516280
ROW-READ-STAGE                    8      4096        1164326
LB-OPERATIONS                     0         0              0
MESSAGE-DESERIALIZER-POOL         1    682008        1818682
GMFD                              0         0           6467
LB-TARGET                         0         0              0
CONSISTENCY-MANAGER               0         0         661477
ROW-MUTATION-STAGE                0         0         998780
MESSAGE-STREAMING-POOL            0         0              0
LOAD-BALANCER-STAGE               0         0              0
FLUSH-SORTER-POOL                 0         0              0
MEMTABLE-POST-FLUSHER             0         0              4
FLUSH-WRITER-POOL                 0         0              4
AE-SERVICE-STAGE                  0         0              0
HINTED-HANDOFF-POOL               0         0              3
YOU ARE OUT OF
DISK BANDWIDTH
You can:
Throttle reads at clients
Adjust memtable settings
    (size/ops/time)
Less frequent memtable flush

 Less frequent compaction

Less disk bandwidth demand
Add more nodes
Add more spindles per node
Switch to SSDs
I inserted a bunch of data.
Now my nodes are flapping.
I inserted a bunch of data.
Now my nodes are flapping.




                  WTF?
iostat -x
look at %util
vmstat
look at swap
INFO 13:27:35,309
DiskAccessMode 'auto' determined to be mmap,
indexAccessMode is mmap
mmap() in Cassandra
consumes up to 2GB.
mmap() in Cassandra
consumes up to 2GB.
Per segment.
NOT tracked as JVM heap.




           *See: https://issues.apache.org/jira/browse/CASSANDRA-1214
NOT tracked as JVM heap.
JVM heap not locked in
memory.

           *See: https://issues.apache.org/jira/browse/CASSANDRA-1214
When your data set exceeds
memory,
this is likely.
Swapping can delay gossip
long enough to cause a
node to be marked down.
<DiskAccessMode>mmap_index_only</
DiskAccessMode>
or
disk_access_mode: mmap_index_only
On Linux: swappiness=0
INFO 13:27:35,309
DiskAccessMode isstandard,
indexAccessMode is mmap
Most people start
troubleshooting problems
interpreting shadows on the
wall.
You can now see the path
and the sunlight outside.
YOU CAN HELP!
What things have confused
you?
What problems have you
solved?
What tools have you used to
solve them?
GET INVOLVED!
http://wiki.apache.org/cassandra
#cassandra on freenode

More Related Content

Similar to Cassandra Summit 2010 - Operations & Troubleshooting Intro

Chicago-Java-User-Group-Meetup-Some-Garbage-Talk-2015-01-14
Chicago-Java-User-Group-Meetup-Some-Garbage-Talk-2015-01-14Chicago-Java-User-Group-Meetup-Some-Garbage-Talk-2015-01-14
Chicago-Java-User-Group-Meetup-Some-Garbage-Talk-2015-01-14
Jayesh Thakrar
 
Automated Interpretation of Wireline and LWD Formation Testing Dynamic Data
Automated Interpretation of Wireline and LWD Formation Testing Dynamic DataAutomated Interpretation of Wireline and LWD Formation Testing Dynamic Data
Automated Interpretation of Wireline and LWD Formation Testing Dynamic Data
Society of Petroleum Engineers
 
Ejercicio metodo simplificado de edificio de 4 niveles
Ejercicio metodo simplificado de edificio de 4 nivelesEjercicio metodo simplificado de edificio de 4 niveles
Ejercicio metodo simplificado de edificio de 4 niveles
hooke007
 

Similar to Cassandra Summit 2010 - Operations & Troubleshooting Intro (20)

Polardb percona-19
Polardb percona-19Polardb percona-19
Polardb percona-19
 
Tabla de resultados
Tabla de resultadosTabla de resultados
Tabla de resultados
 
Geometry Commands
Geometry CommandsGeometry Commands
Geometry Commands
 
ECE469 proj2_Lakshmi Yasaswi Kamireddy
ECE469 proj2_Lakshmi Yasaswi KamireddyECE469 proj2_Lakshmi Yasaswi Kamireddy
ECE469 proj2_Lakshmi Yasaswi Kamireddy
 
Linux networking
Linux networkingLinux networking
Linux networking
 
Bảng giá cadisun năm 2017
Bảng giá cadisun năm 2017 Bảng giá cadisun năm 2017
Bảng giá cadisun năm 2017
 
I pv6 dhcp
I pv6 dhcpI pv6 dhcp
I pv6 dhcp
 
Elecon Wrom Gear Catalogue| Sumit Engineers
Elecon  Wrom Gear Catalogue| Sumit EngineersElecon  Wrom Gear Catalogue| Sumit Engineers
Elecon Wrom Gear Catalogue| Sumit Engineers
 
Chicago-Java-User-Group-Meetup-Some-Garbage-Talk-2015-01-14
Chicago-Java-User-Group-Meetup-Some-Garbage-Talk-2015-01-14Chicago-Java-User-Group-Meetup-Some-Garbage-Talk-2015-01-14
Chicago-Java-User-Group-Meetup-Some-Garbage-Talk-2015-01-14
 
PROJECT 8th SEM - DEVELOPMENT OF SOME INTEGRATED DECISION-MAKING FRAMEWORK FO...
PROJECT 8th SEM - DEVELOPMENT OF SOME INTEGRATED DECISION-MAKING FRAMEWORK FO...PROJECT 8th SEM - DEVELOPMENT OF SOME INTEGRATED DECISION-MAKING FRAMEWORK FO...
PROJECT 8th SEM - DEVELOPMENT OF SOME INTEGRATED DECISION-MAKING FRAMEWORK FO...
 
Automated Interpretation of Wireline and LWD Formation Testing Dynamic Data
Automated Interpretation of Wireline and LWD Formation Testing Dynamic DataAutomated Interpretation of Wireline and LWD Formation Testing Dynamic Data
Automated Interpretation of Wireline and LWD Formation Testing Dynamic Data
 
Calculation template
Calculation templateCalculation template
Calculation template
 
Debugging linux issues with eBPF
Debugging linux issues with eBPFDebugging linux issues with eBPF
Debugging linux issues with eBPF
 
K lateral placas
K lateral placasK lateral placas
K lateral placas
 
Share
ShareShare
Share
 
2nd Presentation NAME 338
2nd Presentation NAME 3382nd Presentation NAME 338
2nd Presentation NAME 338
 
IP Transit : Simple Math - Simple Calculation
IP Transit : Simple Math - Simple CalculationIP Transit : Simple Math - Simple Calculation
IP Transit : Simple Math - Simple Calculation
 
NAME 338 , Presentation 1
NAME 338 , Presentation 1NAME 338 , Presentation 1
NAME 338 , Presentation 1
 
Ejercicio metodo simplificado de edificio de 4 niveles
Ejercicio metodo simplificado de edificio de 4 nivelesEjercicio metodo simplificado de edificio de 4 niveles
Ejercicio metodo simplificado de edificio de 4 niveles
 
ac motor speed reducer,AC vertical electro motor with 30:1 gearbox,ac motor
ac motor speed reducer,AC vertical electro motor with 30:1 gearbox,ac motorac motor speed reducer,AC vertical electro motor with 30:1 gearbox,ac motor
ac motor speed reducer,AC vertical electro motor with 30:1 gearbox,ac motor
 

Recently uploaded

Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...
FIDO Alliance
 
CORS (Kitworks Team Study 양다윗 발표자료 240510)
CORS (Kitworks Team Study 양다윗 발표자료 240510)CORS (Kitworks Team Study 양다윗 발표자료 240510)
CORS (Kitworks Team Study 양다윗 발표자료 240510)
Wonjun Hwang
 
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptxHarnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
FIDO Alliance
 

Recently uploaded (20)

Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
 
Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...
 
Working together SRE & Platform Engineering
Working together SRE & Platform EngineeringWorking together SRE & Platform Engineering
Working together SRE & Platform Engineering
 
AI+A11Y 11MAY2024 HYDERBAD GAAD 2024 - HelloA11Y (11 May 2024)
AI+A11Y 11MAY2024 HYDERBAD GAAD 2024 - HelloA11Y (11 May 2024)AI+A11Y 11MAY2024 HYDERBAD GAAD 2024 - HelloA11Y (11 May 2024)
AI+A11Y 11MAY2024 HYDERBAD GAAD 2024 - HelloA11Y (11 May 2024)
 
Top 10 CodeIgniter Development Companies
Top 10 CodeIgniter Development CompaniesTop 10 CodeIgniter Development Companies
Top 10 CodeIgniter Development Companies
 
AI mind or machine power point presentation
AI mind or machine power point presentationAI mind or machine power point presentation
AI mind or machine power point presentation
 
Continuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
Continuing Bonds Through AI: A Hermeneutic Reflection on ThanabotsContinuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
Continuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
 
Overview of Hyperledger Foundation
Overview of Hyperledger FoundationOverview of Hyperledger Foundation
Overview of Hyperledger Foundation
 
Event-Driven Architecture Masterclass: Challenges in Stream Processing
Event-Driven Architecture Masterclass: Challenges in Stream ProcessingEvent-Driven Architecture Masterclass: Challenges in Stream Processing
Event-Driven Architecture Masterclass: Challenges in Stream Processing
 
CORS (Kitworks Team Study 양다윗 발표자료 240510)
CORS (Kitworks Team Study 양다윗 발표자료 240510)CORS (Kitworks Team Study 양다윗 발표자료 240510)
CORS (Kitworks Team Study 양다윗 발표자료 240510)
 
Frisco Automating Purchase Orders with MuleSoft IDP- May 10th, 2024.pptx.pdf
Frisco Automating Purchase Orders with MuleSoft IDP- May 10th, 2024.pptx.pdfFrisco Automating Purchase Orders with MuleSoft IDP- May 10th, 2024.pptx.pdf
Frisco Automating Purchase Orders with MuleSoft IDP- May 10th, 2024.pptx.pdf
 
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptxHarnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
 
Design and Development of a Provenance Capture Platform for Data Science
Design and Development of a Provenance Capture Platform for Data ScienceDesign and Development of a Provenance Capture Platform for Data Science
Design and Development of a Provenance Capture Platform for Data Science
 
Portal Kombat : extension du réseau de propagande russe
Portal Kombat : extension du réseau de propagande russePortal Kombat : extension du réseau de propagande russe
Portal Kombat : extension du réseau de propagande russe
 
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)
 
State of the Smart Building Startup Landscape 2024!
State of the Smart Building Startup Landscape 2024!State of the Smart Building Startup Landscape 2024!
State of the Smart Building Startup Landscape 2024!
 
JohnPollard-hybrid-app-RailsConf2024.pptx
JohnPollard-hybrid-app-RailsConf2024.pptxJohnPollard-hybrid-app-RailsConf2024.pptx
JohnPollard-hybrid-app-RailsConf2024.pptx
 
Microsoft CSP Briefing Pre-Engagement - Questionnaire
Microsoft CSP Briefing Pre-Engagement - QuestionnaireMicrosoft CSP Briefing Pre-Engagement - Questionnaire
Microsoft CSP Briefing Pre-Engagement - Questionnaire
 
الأمن السيبراني - ما لا يسع للمستخدم جهله
الأمن السيبراني - ما لا يسع للمستخدم جهلهالأمن السيبراني - ما لا يسع للمستخدم جهله
الأمن السيبراني - ما لا يسع للمستخدم جهله
 
Cyber Insurance - RalphGilot - Embry-Riddle Aeronautical University.pptx
Cyber Insurance - RalphGilot - Embry-Riddle Aeronautical University.pptxCyber Insurance - RalphGilot - Embry-Riddle Aeronautical University.pptx
Cyber Insurance - RalphGilot - Embry-Riddle Aeronautical University.pptx
 

Cassandra Summit 2010 - Operations & Troubleshooting Intro

  • 1. Cassandra troubleshooting: out of the shadows Benjamin Black, b@b3k.us
  • 3. The Allegory of the Cave
  • 4. Most people start troubleshooting problems interpreting shadows on the wall.
  • 6. Paths out of the cave.
  • 7. Combination of basic system tools & nodetool/JMX
  • 8. I’m using RP. My ring is very unbalanced.
  • 9. I’m using RP. My ring is very unbalanced. WTF?
  • 11. Address Status Load Range Ring 148873535527910577765226390751398592511 10.248.54.192 Up 5.59 GB 0 |<--| 10.248.254.15 Up 10.58 GB 42535295865117307932921825928971026431 | ^ 10.248.135.239Up 11.01 GB 85070591730234615865843651857942052863 v | 10.248.223.191Up 5.42 GB 106338239662793269832304564822427566079 | ^ 10.248.122.240Up 5.51 GB 127605887595351923798765477786913079295 v | 10.248.34.80 Up 5.45 GB 148873535527910577765226390751398592511 |-->|
  • 12. Address Status Load Range Ring 148873535527910577765226390751398592511 10.248.54.192 Up 5.59 GB 0 |<--| 10.248.254.15 Up 10.58 GB 42535295865117307932921825928971026431 | ^ 10.248.135.239Up 11.01 GB 85070591730234615865843651857942052863 v | 10.248.223.191Up 5.42 GB 106338239662793269832304564822427566079 | ^ 10.248.122.240Up 5.51 GB 127605887595351923798765477786913079295 v | 10.248.34.80 Up 5.45 GB 148873535527910577765226390751398592511 |-->|
  • 14. Automatic token algorithm: Assign a token that will give me half the range of the most loaded node.
  • 15. 32 16 16 8 8 16 8888 44888 444488
  • 16. Address Status Load Range Ring 148873535527910577765226390751398592511 10.248.54.192 Up 5.59 GB 0 |<--| 10.248.254.15 Up 10.58 GB 42535295865117307932921825928971026431 | ^ 10.248.135.239Up 11.01 GB 85070591730234615865843651857942052863 v | 10.248.223.191Up 5.42 GB 106338239662793269832304564822427566079 | ^ 10.248.122.240Up 5.51 GB 127605887595351923798765477786913079295 v | 10.248.34.80 Up 5.45 GB 148873535527910577765226390751398592511 |-->|
  • 18. 0-(2**127 - 1) def tokens(nodes) 0.upto(nodes - 1) do |n| p (n * (2**127 - 1) / nodes) end end
  • 20. YES: This means you need to change tokens on most of the nodes in your cluster whenever you add a node.
  • 21. Writes are fast. Reads keep getting slower.
  • 22. Writes are fast. Reads keep getting slower. WTF?
  • 25. Pool Name                    Active   Pending      Completed STREAM-STAGE                      0         0              0 RESPONSE-STAGE                    0         0         516280 ROW-READ-STAGE                    8      4096        1164326 LB-OPERATIONS                     0         0              0 MESSAGE-DESERIALIZER-POOL         1    682008        1818682 GMFD                              0         0           6467 LB-TARGET                         0         0              0 CONSISTENCY-MANAGER               0         0         661477 ROW-MUTATION-STAGE                0         0         998780 MESSAGE-STREAMING-POOL            0         0              0 LOAD-BALANCER-STAGE               0         0              0 FLUSH-SORTER-POOL                 0         0              0 MEMTABLE-POST-FLUSHER             0         0              4 FLUSH-WRITER-POOL                 0         0              4 AE-SERVICE-STAGE                  0         0              0 HINTED-HANDOFF-POOL               0         0              3
  • 26. Pool Name                    Active   Pending      Completed STREAM-STAGE                      0         0              0 RESPONSE-STAGE                    0         0         516280 ROW-READ-STAGE                    8      4096        1164326 LB-OPERATIONS                     0         0              0 MESSAGE-DESERIALIZER-POOL         1    682008        1818682 GMFD                              0         0           6467 LB-TARGET                         0         0              0 CONSISTENCY-MANAGER               0         0         661477 ROW-MUTATION-STAGE                0         0         998780 MESSAGE-STREAMING-POOL            0         0              0 LOAD-BALANCER-STAGE               0         0              0 FLUSH-SORTER-POOL                 0         0              0 MEMTABLE-POST-FLUSHER             0         0              4 FLUSH-WRITER-POOL                 0         0              4 AE-SERVICE-STAGE                  0         0              0 HINTED-HANDOFF-POOL               0         0              3
  • 27. Pool Name                    Active   Pending      Completed STREAM-STAGE                      0         0              0 RESPONSE-STAGE                    0         0         516280 ROW-READ-STAGE                    8      4096        1164326 LB-OPERATIONS                     0         0              0 MESSAGE-DESERIALIZER-POOL         1    682008        1818682 GMFD                              0         0           6467 LB-TARGET                         0         0              0 CONSISTENCY-MANAGER               0         0         661477 ROW-MUTATION-STAGE                0         0         998780 MESSAGE-STREAMING-POOL            0         0              0 LOAD-BALANCER-STAGE               0         0              0 FLUSH-SORTER-POOL                 0         0              0 MEMTABLE-POST-FLUSHER             0         0              4 FLUSH-WRITER-POOL                 0         0              4 AE-SERVICE-STAGE                  0         0              0 HINTED-HANDOFF-POOL               0         0              3
  • 28. Pool Name                    Active   Pending      Completed STREAM-STAGE                      0         0              0 RESPONSE-STAGE                    0         0         516280 ROW-READ-STAGE                    8      4096        1164326 LB-OPERATIONS                     0         0              0 MESSAGE-DESERIALIZER-POOL         1    682008        1818682 GMFD                              0         0           6467 LB-TARGET                         0         0              0 CONSISTENCY-MANAGER               0         0         661477 ROW-MUTATION-STAGE                0         0         998780 MESSAGE-STREAMING-POOL            0         0              0 LOAD-BALANCER-STAGE               0         0              0 FLUSH-SORTER-POOL                 0         0              0 MEMTABLE-POST-FLUSHER             0         0              4 FLUSH-WRITER-POOL                 0         0              4 AE-SERVICE-STAGE                  0         0              0 HINTED-HANDOFF-POOL               0         0              3
  • 29. YOU ARE OUT OF DISK BANDWIDTH
  • 31. Throttle reads at clients
  • 32. Adjust memtable settings (size/ops/time)
  • 33. Less frequent memtable flush Less frequent compaction Less disk bandwidth demand
  • 35. Add more spindles per node
  • 37. I inserted a bunch of data. Now my nodes are flapping.
  • 38. I inserted a bunch of data. Now my nodes are flapping. WTF?
  • 41. INFO 13:27:35,309 DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
  • 43. mmap() in Cassandra consumes up to 2GB. Per segment.
  • 44. NOT tracked as JVM heap. *See: https://issues.apache.org/jira/browse/CASSANDRA-1214
  • 45. NOT tracked as JVM heap. JVM heap not locked in memory. *See: https://issues.apache.org/jira/browse/CASSANDRA-1214
  • 46. When your data set exceeds memory, this is likely.
  • 47. Swapping can delay gossip long enough to cause a node to be marked down.
  • 51. Most people start troubleshooting problems interpreting shadows on the wall.
  • 52. You can now see the path and the sunlight outside.
  • 54. What things have confused you?
  • 55. What problems have you solved?
  • 56. What tools have you used to solve them?

Editor's Notes