Cassandra Summit 2010 - Operations & Troubleshooting Intro

Cassandra troubleshooting: out of the shadows Benjamin Black, b@b3k.us

Introducing: This Guy

The Allegory of the Cave

Most people start troubleshooting problems interpreting shadows on the wall.

Common shadows.

Paths out of the cave.

Combination of basic system tools & nodetool/JMX

I’m using RP. My ring is very unbalanced.

I’m using RP. My ring is very unbalanced. WTF?

nodetool ring

Address Status Load Range Ring 148873535527910577765226390751398592511 10.248.54.192 Up 5.59 GB 0 |<--| 10.248.254.15 Up 10.58 GB 42535295865117307932921825928971026431 | ^ 10.248.135.239Up 11.01 GB 85070591730234615865843651857942052863 v | 10.248.223.191Up 5.42 GB 106338239662793269832304564822427566079 | ^ 10.248.122.240Up 5.51 GB 127605887595351923798765477786913079295 v | 10.248.34.80 Up 5.45 GB 148873535527910577765226390751398592511 |-->|

Autobootstrap + Automatic token assignment

Automatic token algorithm: Assign a token that will give me half the range of the most loaded node.

32 16 16 8 8 16 8888 44888 444488

Address Status Load Range Ring 148873535527910577765226390751398592511 10.248.54.192 Up 5.59 GB 0 |<--| 10.248.254.15 Up 10.58 GB 42535295865117307932921825928971026431 | ^ 10.248.135.239Up 11.01 GB 85070591730234615865843651857942052863 v | 10.248.223.191Up 5.42 GB 106338239662793269832304564822427566079 | ^ 10.248.122.240Up 5.51 GB 127605887595351923798765477786913079295 v | 10.248.34.80 Up 5.45 GB 148873535527910577765226390751398592511 |-->|

nodetool move + Manual token assignment

0-(2**127 - 1) def tokens(nodes) 0.upto(nodes - 1) do |n| p (n * (2**127 - 1) / nodes) end end

=> tokens(6) 0 283568639100782052886145506193140 17621 567137278201564105772291012386280 35242 850705917302346158658436518579420 52863 113427455640312821154458202477256 070484 141784319550391026443072753096570 088105

YES: This means you need to change tokens on most of the nodes in your cluster whenever you add a node.

Writes are fast. Reads keep getting slower.

Writes are fast. Reads keep getting slower. WTF?

iostat -x look at %util

nodetool tpstats

Pool Name Active Pending Completed STREAM-STAGE 0 0 0 RESPONSE-STAGE 0 0 516280 ROW-READ-STAGE 8 4096 1164326 LB-OPERATIONS 0 0 0 MESSAGE-DESERIALIZER-POOL 1 682008 1818682 GMFD 0 0 6467 LB-TARGET 0 0 0 CONSISTENCY-MANAGER 0 0 661477 ROW-MUTATION-STAGE 0 0 998780 MESSAGE-STREAMING-POOL 0 0 0 LOAD-BALANCER-STAGE 0 0 0 FLUSH-SORTER-POOL 0 0 0 MEMTABLE-POST-FLUSHER 0 0 4 FLUSH-WRITER-POOL 0 0 4 AE-SERVICE-STAGE 0 0 0 HINTED-HANDOFF-POOL 0 0 3

YOU ARE OUT OF DISK BANDWIDTH

You can:

Throttle reads at clients

Adjust memtable settings (size/ops/time)

Less frequent memtable ﬂush Less frequent compaction Less disk bandwidth demand

Add more nodes

Add more spindles per node

Switch to SSDs

I inserted a bunch of data. Now my nodes are ﬂapping.

I inserted a bunch of data. Now my nodes are ﬂapping. WTF?

iostat -x look at %util

vmstat look at swap

INFO 13:27:35,309 DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap

mmap() in Cassandra consumes up to 2GB.

mmap() in Cassandra consumes up to 2GB. Per segment.

NOT tracked as JVM heap. *See: https://issues.apache.org/jira/browse/CASSANDRA-1214

NOT tracked as JVM heap. JVM heap not locked in memory. *See: https://issues.apache.org/jira/browse/CASSANDRA-1214

When your data set exceeds memory, this is likely.

Swapping can delay gossip long enough to cause a node to be marked down.

<DiskAccessMode>mmap_index_only</ DiskAccessMode> or disk_access_mode: mmap_index_only

On Linux: swappiness=0

INFO 13:27:35,309 DiskAccessMode isstandard, indexAccessMode is mmap

Most people start troubleshooting problems interpreting shadows on the wall.

You can now see the path and the sunlight outside.

YOU CAN HELP!

What things have confused you?

What problems have you solved?

What tools have you used to solve them?

GET INVOLVED!

http://wiki.apache.org/cassandra

#cassandra on freenode

http://www.dbthink.com	249
http://peter78.wordpress.com	35
http://coral.corp.apple.com	2
http://xianguo.com	2
https://coral.corp.apple.com	2
http://www.slideshare.net	1
http://reader.youdao.com	1

Cassandra Summit 2010 - Operations & Troubleshooting Intro

by Benjamin Black on Aug 10, 2010

Accessibility

Categories

Upload Details

Usage Rights

7 Embeds 292

Statistics

Cassandra Summit 2010 - Operations & Troubleshooting Intro Presentation Transcript