Uploaded byzznate
4,909 views

Advanced Apache Cassandra Operations with JMX

Nodetool is a command line interface for managing a Cassandra node. It provides commands for node administration, cluster inspection, table operations and more. The nodetool info command displays node-specific information such as status, load, memory usage and cache details. The nodetool compactionstats command shows compaction status including active tasks and progress. The nodetool tablestats command displays statistics for a specific table including read/write counts, space usage, cache usage and latency.

Embed presentation

Downloaded 91 times
1 / 131
2 / 131
3 / 131
4 / 131
5 / 131
6 / 131
7 / 131
8 / 131
9 / 131
10 / 131
11 / 131
12 / 131
13 / 131
14 / 131
15 / 131
16 / 131
17 / 131
18 / 131
19 / 131
20 / 131
21 / 131
22 / 131
23 / 131
24 / 131
25 / 131
26 / 131
27 / 131
28 / 131
29 / 131
30 / 131
31 / 131
32 / 131
33 / 131
34 / 131
35 / 131
36 / 131
37 / 131
38 / 131
39 / 131
40 / 131
41 / 131
42 / 131
43 / 131
44 / 131
45 / 131
46 / 131
47 / 131
48 / 131
49 / 131
50 / 131
51 / 131
52 / 131
53 / 131
54 / 131
55 / 131
56 / 131
57 / 131
58 / 131
59 / 131
60 / 131
61 / 131
62 / 131
63 / 131
64 / 131
65 / 131
66 / 131
67 / 131
68 / 131
69 / 131
70 / 131
71 / 131
72 / 131
73 / 131
74 / 131
75 / 131
76 / 131
77 / 131
78 / 131
79 / 131
80 / 131
81 / 131
82 / 131
83 / 131
84 / 131
85 / 131
86 / 131
87 / 131
88 / 131
89 / 131
90 / 131
91 / 131
92 / 131
93 / 131
94 / 131
95 / 131
96 / 131
97 / 131
98 / 131
99 / 131
100 / 131
101 / 131
102 / 131
103 / 131
104 / 131
105 / 131
106 / 131
107 / 131
108 / 131
109 / 131
110 / 131
111 / 131
112 / 131
113 / 131
114 / 131
115 / 131
116 / 131
117 / 131
118 / 131
119 / 131
120 / 131
121 / 131
122 / 131
123 / 131
124 / 131
125 / 131
126 / 131
127 / 131
128 / 131
129 / 131
130 / 131
131 / 131
JMX and Cassandra Understanding tooling for advanced operations
Hi, I'm Nate. @zznate https://www.linkedin.com/in/zznate http://www.slideshare.net/zznate/ Co-Founder, CTO The Last Pickle Apache Cassandra PMC member User since 2009 (v0.4) Austin, Texas Wellington, NZ
TLP: not just "Aaron and Nate"
Lots of information in the next 30 minutes
1. Tools 2. Visibility and exploration 3. Operations and invocation 4. Access controls JMX and Cassandra
1. Summary of the tools
nodetool mx4j jmxsh|jmxterm jconsole Tools:
Nodetool: the cruise line of cluster administration
output of nodetool help (87 commands) usage: nodetool [(-u <username> | --username <username>)] [(-pw <password> | --password <password>)] [(-h <host> | --host <host>)] [(-p <port> | --port <port>)] [(-pwf <passwordFilePath> | --password-file <passwordFilePath>)] <command> [<args>] The most commonly used nodetool commands are: assassinate Forcefully remove a dead node without re-replicating any data. Use as a last resort if you cannot removenode bootstrap Monitor/manage node's bootstrap process cleanup Triggers the immediate cleanup of keys no longer belonging to a node. By default, clean all keyspaces clearsnapshot Remove the snapshot with the given name from the given keyspaces. If no snapshotName is specified we will remove all snapshots compact Force a (major) compaction on one or more tables compactionhistory Print history of compaction compactionstats Print statistics on compactions decommission Decommission the *node I am connecting to* describecluster Print the name, snitch, partitioner and schema version of a cluster describering Shows the token ranges info of a given keyspace disableautocompaction Disable autocompaction for the given keyspace and table disablebackup Disable incremental backup disablebinary Disable native transport (binary protocol) disablegossip Disable gossip (effectively marking the node down) disablehandoff Disable storing hinted handoffs disablehintsfordc Disable hints for a data center disablethrift Disable thrift server drain Drain the node (stop accepting writes and flush all tables) enableautocompaction Enable autocompaction for the given keyspace and table enablebackup Enable incremental backup enablebinary Reenable native transport (binary protocol) enablegossip Reenable gossip enablehandoff Reenable future hints storing on the current node enablehintsfordc Enable hints for a data center that was previsouly disabled enablethrift Reenable thrift server failuredetector Shows the failure detector information for the cluster flush Flush one or more tables gcstats Print GC Statistics getcompactionthreshold Print min and max compaction thresholds for a given table getcompactionthroughput Print the MB/s throughput cap for compaction in the system getendpoints Print the end points that owns the key getinterdcstreamthroughput Print the Mb/s throughput cap for inter-datacenter streaming in the system getlogginglevels Get the runtime logging levels getsstables Print the sstable filenames that own the key getstreamthroughput Print the Mb/s throughput cap for streaming in the system gettraceprobability Print the current trace probability value gossipinfo Shows the gossip information for the cluster help Display help information info Print node information (uptime, load, ...) invalidatecountercache Invalidate the counter cache invalidatekeycache Invalidate the key cache invalidaterowcache Invalidate the row cache join Join the ring listsnapshots Lists all the snapshots along with the size on disk and true size. move Move node on the token ring to a new token netstats Print network information on provided host (connecting node by default) pausehandoff Pause hints delivery process proxyhistograms Print statistic histograms for network operations rangekeysample Shows the sampled keys held across all keyspaces rebuild Rebuild data by streaming from other nodes (similarly to bootstrap) rebuild_index A full rebuild of native secondary indexes for a given table refresh Load newly placed SSTables to the system without restart refreshsizeestimates Refresh system.size_estimates reloadtriggers Reload trigger classes removenode Show status of current node removal, force completion of pending removal or remove provided ID repair Repair one or more tables replaybatchlog Kick off batchlog replay and wait for finish resetlocalschema Reset node's local schema and resync resumehandoff Resume hints delivery process ring Print information about the token ring scrub Scrub (rebuild sstables for) one or more tables setcachecapacity Set global key, row, and counter cache capacities (in MB units) setcachekeystosave Set number of keys saved by each cache for faster post-restart warmup. 0 to disable setcompactionthreshold Set min and max compaction thresholds for a given table setcompactionthroughput Set the MB/s throughput cap for compaction in the system, or 0 to disable throttling sethintedhandoffthrottlekb Set hinted handoff throttle in kb per second, per delivery thread. setinterdcstreamthroughput Set the Mb/s throughput cap for inter-datacenter streaming in the system, or 0 to disable throttling setlogginglevel Set the log level threshold for a given class. If both class and level are empty/null, it will reset to the initial configuration setstreamthroughput Set the Mb/s throughput cap for streaming in the system, or 0 to disable throttling settraceprobability Sets the probability for tracing any given request to value. 0 disables, 1 enables for all requests, 0 is the default snapshot Take a snapshot of specified keyspaces or a snapshot of the specified table status Print cluster information (state, load, IDs, ...) statusbackup Status of incremental backup statusbinary Status of native transport (binary protocol) statusgossip Status of gossip statushandoff Status of storing future hints on the current node statusthrift Status of thrift server stop Stop compaction stopdaemon Stop cassandra daemon tablehistograms Print statistic histograms for a given table tablestats Print statistics on tables toppartitions Sample and print the most active partitions for a given column family tpstats Print usage statistics of thread pools truncatehints Truncate all hints on the local node, or truncate hints for the endpoint(s) specified. upgradesstables Rewrite sstables (for the requested tables) that are not on the current version (thus upgrading them to said current version) verify Verify (check data checksum for) one or more tables version Print cassandra version See 'nodetool help <command>' for more information on a specific command.
nodetool commands: info ID : fed6dd09-667c-4cc5-b66a-85a1cbce1b58 Gossip active : true Thrift active : false Native Transport active: true Load : 219.97 KB Generation No : 1473125362 Uptime (seconds) : 28802 Heap Memory (MB) : 86.96 / 495.00 Off Heap Memory (MB) : 0.00 Data Center : datacenter1 Rack : rack1 Exceptions : 0 Key Cache : entries 27, size 2.15 KB, capacity 24 MB, 274 hits, 313 requests, 0.875 recent hit rate, 14400 save period in seconds Row Cache : entries 0, size 0 bytes, capacity 0 bytes, 0 hits, 0 requests, NaN recent hit rate, 0 save period in seconds Counter Cache : entries 0, size 0 bytes, capacity 12 MB, 0 hits, 0 requests, NaN recent hit rate, 7200 save period in seconds Token : -9223372036854775808
nodetool commands: info ID : fed6dd09-667c-4cc5-b66a-85a1cbce1b58 Gossip active : true Thrift active : false Native Transport active: true Load : 219.97 KB Generation No : 1473125362 Uptime (seconds) : 28802 Heap Memory (MB) : 86.96 / 495.00 Off Heap Memory (MB) : 0.00 Data Center : datacenter1 Rack : rack1 Exceptions : 0 Key Cache : entries 27, size 2.15 KB, capacity 24 MB, 274 hits, 313 requests, 0.875 recent hit rate, 14400 save period in seconds Row Cache : entries 0, size 0 bytes, capacity 0 bytes, 0 hits, 0 requests, NaN recent hit rate, 0 save period in seconds Counter Cache : entries 0, size 0 bytes, capacity 12 MB, 0 hits, 0 requests, NaN recent hit rate, 7200 save period in seconds Token : -9223372036854775808
nodetool commands: info ID : fed6dd09-667c-4cc5-b66a-85a1cbce1b58 Gossip active : true Thrift active : false Native Transport active: true Load : 219.97 KB Generation No : 1473125362 Uptime (seconds) : 28802 Heap Memory (MB) : 86.96 / 495.00 Off Heap Memory (MB) : 0.00 Data Center : datacenter1 Rack : rack1 Exceptions : 0 Key Cache : entries 27, size 2.15 KB, capacity 24 MB, 274 hits, 313 requests, 0.875 recent hit rate, 14400 save period Row Cache : entries 0, size 0 bytes, capacity 0 bytes, 0 hits, 0 requests, NaN recent hit rate, 0 save period in seconds Counter Cache : entries 0, size 0 bytes, capacity 12 MB, 0 hits, 0 requests, NaN recent hit rate, 7200 save period in seconds Token : -9223372036854775808
nodetool commands: info ID : fed6dd09-667c-4cc5-b66a-85a1cbce1b58 Gossip active : true Thrift active : false Native Transport active: true Load : 219.97 KB Generation No : 1473125362 Uptime (seconds) : 28802 Heap Memory (MB) : 86.96 / 495.00 Off Heap Memory (MB) : 0.00 Data Center : datacenter1 Rack : rack1 Exceptions : 0 Key Cache : entries 27, size 2.15 KB, capacity 24 MB, 274 hits, 313 requests, 0.875 recent hit rate, 14400 save period Row Cache : entries 0, size 0 bytes, capacity 0 bytes, 0 hits, 0 requests, NaN recent hit rate, 0 save period in seconds Counter Cache : entries 0, size 0 bytes, capacity 12 MB, 0 hits, 0 requests, NaN recent hit rate, 7200 save period in seconds Token : -9223372036854775808 tip: nodetool setcachecapacity <key> <row> <counter>
$ nodetool compactionstats -H pending tasks: 2101 id type keyspace table completed total progress d14fc600-6e3f-11e6-a01c-8bcdc0e93c44 Compaction ads ad_events 742.58 MB 4.1 TB 0.02% Active compaction remaining time : 37h18m47s nodetool commands: compactionstats
$ nodetool compactionstats -H pending tasks: 2101 id type keyspace table completed total progress d14fc600-6e3f-11e6-a01c-8bcdc0e93c44 Compaction ads ad_events 742.58 MB 4.1 TB 0.02% Active compaction remaining time : 37h18m47s nodetool commands: compactionstats
nodetool commands: compactionstats $ nodetool compactionstats -H pending tasks: 2101 id type keyspace table completed total progress d14fc600-6e3f-11e6-a01c-8bcdc0e93c44 Compaction ads ad_events 742.58 MB 4.1 TB 0.02% Active compaction remaining time : 37h18m47s
nodetool commands: tablestats Keyspace: jmx_adventure Read Count: 19 Read Latency: 0.3552105263157895 ms. Write Count: 3 Write Latency: 0.17766666666666667 ms. Pending Flushes: 0 Table: account_event SSTable count: 1 Space used (live): 5326 Space used (total): 5326 Space used by snapshots (total): 0 Off heap memory used (total): 51 SSTable Compression Ratio: 0.6517241379310345 Number of keys (estimate): 3 Memtable cell count: 0 Memtable data size: 0 Memtable off heap memory used: 0 Memtable switch count: 1 Local read count: 19 Local read latency: 0.386 ms Local write count: 3 Local write latency: 0.207 ms Pending flushes: 0 Bloom filter false positives: 0 Bloom filter false ratio: 0.00000 Bloom filter space used: 16 Bloom filter off heap memory used: 8 Index summary off heap memory used: 27 Compression metadata off heap memory used: 16 Compacted partition minimum bytes: 51 Compacted partition maximum bytes: 149 Compacted partition mean bytes: 104 Average live cells per slice (last five minutes): 2.263157894736842 Maximum live cells per slice (last five minutes): 3 Average tombstones per slice (last five minutes): 1.0 Maximum tombstones per slice (last five minutes): 1
nodetool commands: tablestats Keyspace: jmx_adventure Read Count: 19 Read Latency: 0.3552105263157895 ms. Write Count: 3 Write Latency: 0.17766666666666667 ms. Pending Flushes: 0 Table: account_event SSTable count: 1 Space used (live): 5326 Space used (total): 5326 Space used by snapshots (total): 0 Off heap memory used (total): 51 SSTable Compression Ratio: 0.6517241379310345 Number of keys (estimate): 3 Memtable cell count: 0 Memtable data size: 0 Memtable off heap memory used: 0 Memtable switch count: 1 Local read count: 19 Local read latency: 0.386 ms Local write count: 3 Local write latency: 0.207 ms Pending flushes: 0 Bloom filter false positives: 0 Bloom filter false ratio: 0.00000 Bloom filter space used: 16 Bloom filter off heap memory used: 8 Index summary off heap memory used: 27 Compression metadata off heap memory used: 16 Compacted partition minimum bytes: 51 Compacted partition maximum bytes: 149 Compacted partition mean bytes: 104 Average live cells per slice (last five minutes): 2.263157894736842 Maximum live cells per slice (last five minutes): 3 Average tombstones per slice (last five minutes): 1.0 Maximum tombstones per slice (last five minutes): 1 tombstones per read partition size read and write latency memtable usage index summary usage
nodetool commands: tablestats Keyspace: jmx_adventure Read Count: 19 Read Latency: 0.3552105263157895 ms. Write Count: 3 Write Latency: 0.17766666666666667 ms. Pending Flushes: 0 Table: account_event SSTable count: 1 Space used (live): 5326 Space used (total): 5326 Space used by snapshots (total): 0 Off heap memory used (total): 51 SSTable Compression Ratio: 0.6517241379310345 Number of keys (estimate): 3 Memtable cell count: 2 Memtable data size: 122 Memtable off heap memory used: 96 Memtable switch count: 1 Local read count: 19 Local read latency: 0.386 ms Local write count: 3 Local write latency: 0.207 ms Pending flushes: 0 Bloom filter false positives: 0 Bloom filter false ratio: 0.00000 Bloom filter space used: 16 Bloom filter off heap memory used: 8 Index summary off heap memory used: 27 Compression metadata off heap memory used: 16 Compacted partition minimum bytes: 51 Compacted partition maximum bytes: 149 Compacted partition mean bytes: 104 Average live cells per slice (last five minutes): 2.263157894736842 Maximum live cells per slice (last five minutes): 3 Average tombstones per slice (last five minutes): 1.0 Maximum tombstones per slice (last five minutes): 1 tombstones per read partition size read and write latency memtable usage index summary usage
nodetool commands: tablestats Keyspace: jmx_adventure Read Count: 19 Read Latency: 0.3552105263157895 ms. Write Count: 3 Write Latency: 0.17766666666666667 ms. Pending Flushes: 0 Table: account_event SSTable count: 1 Space used (live): 5326 Space used (total): 5326 Space used by snapshots (total): 0 Off heap memory used (total): 51 SSTable Compression Ratio: 0.6517241379310345 Number of keys (estimate): 3 Memtable cell count: 0 Memtable data size: 0 Memtable off heap memory used: 0 Memtable switch count: 1 Local read count: 19 Local read latency: 0.386 ms Local write count: 3 Local write latency: 0.207 ms Pending flushes: 0 Bloom filter false positives: 0 Bloom filter false ratio: 0.00000 Bloom filter space used: 16 Bloom filter off heap memory used: 8 Index summary off heap memory used: 27 Compression metadata off heap memory used: 16 Compacted partition minimum bytes: 51 Compacted partition maximum bytes: 149 Compacted partition mean bytes: 104 Average live cells per slice (last five minutes): 2.263157894736842 Maximum live cells per slice (last five minutes): 3 Average tombstones per slice (last five minutes): 1.0 Maximum tombstones per slice (last five minutes): 1 tombstones per read partition size read and write latency memtable usage index summary usage
nodetool commands: tablestats Keyspace: jmx_adventure Read Count: 19 Read Latency: 0.3552105263157895 ms. Write Count: 3 Write Latency: 0.17766666666666667 ms. Pending Flushes: 0 Table: account_event SSTable count: 1 Space used (live): 5326 Space used (total): 5326 Space used by snapshots (total): 0 Off heap memory used (total): 51 SSTable Compression Ratio: 0.6517241379310345 Number of keys (estimate): 3 Memtable cell count: 0 Memtable data size: 0 Memtable off heap memory used: 0 Memtable switch count: 1 Local read count: 19 Local read latency: 0.386 ms Local write count: 3 Local write latency: 0.207 ms Pending flushes: 0 Bloom filter false positives: 0 Bloom filter false ratio: 0.00000 Bloom filter space used: 16 Bloom filter off heap memory used: 8 Index summary off heap memory used: 27 Compression metadata off heap memory used: 16 Compacted partition minimum bytes: 51 Compacted partition maximum bytes: 149 Compacted partition mean bytes: 104 Average live cells per slice (last five minutes): 2.263157894736842 Maximum live cells per slice (last five minutes): 3 Average tombstones per slice (last five minutes): 1.0 Maximum tombstones per slice (last five minutes): 1 tombstones per read partition size read and write latency memtable usage index summary usage
nodetool commands: tablestats Keyspace: jmx_adventure Read Count: 19 Read Latency: 0.3552105263157895 ms. Write Count: 3 Write Latency: 0.17766666666666667 ms. Pending Flushes: 0 Table: account_event SSTable count: 1 Space used (live): 5326 Space used (total): 5326 Space used by snapshots (total): 0 Off heap memory used (total): 51 SSTable Compression Ratio: 0.6517241379310345 Number of keys (estimate): 3 Memtable cell count: 0 Memtable data size: 0 Memtable off heap memory used: 0 Memtable switch count: 1 Local read count: 19 Local read latency: 0.386 ms Local write count: 3 Local write latency: 0.207 ms Pending flushes: 0 Bloom filter false positives: 0 Bloom filter false ratio: 0.00000 Bloom filter space used: 16 Bloom filter off heap memory used: 8 Index summary off heap memory used: 27 Compression metadata off heap memory used: 16 Compacted partition minimum bytes: 51 Compacted partition maximum bytes: 149 Compacted partition mean bytes: 104 Average live cells per slice (last five minutes): 2.263 Maximum live cells per slice (last five minutes): 3 Average tombstones per slice (last five minutes): 1.0 Maximum tombstones per slice (last five minutes): 1 tombstones per read partition size read and write latency memtable usage index summary usage
nodetool commands: tablehistograms $ nodetool tablehistograms jmx_adventure account_event jmx_adventure/account_event histograms Percentile SSTables Write Latency Read Latency Partition Size Cell Count (micros) (micros) (bytes) 50% 1.00 263.21 315.85 103 1 75% 2.00 315.85 545.79 149 1 95% 2.00 315.85 1131.75 149 1 98% 2.00 315.85 1131.75 149 1 99% 2.00 315.85 1131.75 149 1 Min 1.00 35.43 152.32 51 0 Max 2.00 315.85 1131.75 149 1
nodetool commands: proxyhistograms $ nodetool proxyhistograms proxy histograms Percentile Read Latency Write Latency Range Latency (micros) (micros) (micros) 50% 454.83 1131.75 1629.72 75% 785.94 12108.97 8409.01 95% 4866.32 12108.97 20924.30 98% 129557.75 12108.97 20924.30 99% 129557.75 12108.97 20924.30 Min 219.34 315.85 943.13 Max 129557.75 12108.97 20924.30
JMX on the HTTP mx4j
JMX on the HTTP mx4jbroken in 3.x
JMX on the HTTP when it does work, it looks like this (2.2.7 and below):
JMX on the HTTP when it does work, it looks like this: ewww gross.
JMX on the HTTP (with JSON!) via Jolokia https://jolokia.org/
JMX on the HTTP (with JSON!) via Jolokia https://jolokia.org/ JSON: it's ok.
JMX on the HTTP (with JSON!) via Jolokia http://search.maven.org/remotecontent?filepath=org/jolokia/jolokia-jvm/1.3.4/jolokia-jvm-1.3.4- agent.jar 1. download: jolokia-jvm-1.3.4-agent.jar 2. place in: $CASSANDRA_HOME/lib 3. in cassandra-env.sh, add: JVM_OPTS="$JVM_OPTS - javaagent:$CASSANDRA_HOME/lib/jolokia- jvm-1.3.4-agent.jar"
JMX on the command line jmxsh jmxterm http://wiki.cyclopsgroup.org/jmxterm/download.html https://github.com/davr/jmxsh
jmxsh jmxterm JMX on the command line http://wiki.cyclopsgroup.org/jmxterm/download.html https://github.com/davr/jmxsh Almost identical!
jmxsh jmxterm JMX on the command line http://wiki.cyclopsgroup.org/jmxterm/download.html https://github.com/davr/jmxsh Almost identical! Not updated in years!
JMX on the command line $ java -jar $HOME/services/jmxterm-1.0-alpha-4-uber.jar -l 127.0.0.1:7200 Welcome to JMX terminal. Type "help" for available commands. $>bean org.apache.cassandra.net:type=MessagingService #bean is set to org.apache.cassandra.net:type=MessagingService $>get TimeoutsPerHost #mbean = org.apache.cassandra.net:type=MessagingService: TimeoutsPerHost = { 127.0.0.1 = 0; 127.0.0.3 = 0; };
JMX on the command line $ java -jar $HOME/services/jmxterm-1.0-alpha-4-uber.jar -l 127.0.0.1:7200 Welcome to JMX terminal. Type "help" for available commands. $>bean org.apache.cassandra.net:type=MessagingService #bean is set to org.apache.cassandra.net:type=MessagingService $>get TimeoutsPerHost #mbean = org.apache.cassandra.net:type=MessagingService: TimeoutsPerHost = { 127.0.0.1 = 0; 127.0.0.3 = 0; };
JMX on the command line $ java -jar $HOME/services/jmxterm-1.0-alpha-4-uber.jar -l 127.0.0.1:7200 Welcome to JMX terminal. Type "help" for available commands. $>bean org.apache.cassandra.net:type=MessagingService #bean is set to org.apache.cassandra.net:type=MessagingService $>get TimeoutsPerHost #mbean = org.apache.cassandra.net:type=MessagingService: TimeoutsPerHost = { 127.0.0.1 = 0; 127.0.0.3 = 0; };
JMX on the command line $ java -jar $HOME/services/jmxterm-1.0-alpha-4-uber.jar -l 127.0.0.1:7200 Welcome to JMX terminal. Type "help" for available commands. $>bean org.apache.cassandra.net:type=MessagingService #bean is set to org.apache.cassandra.net:type=MessagingService $>get TimeoutsPerHost #mbean = org.apache.cassandra.net:type=MessagingService: TimeoutsPerHost = { 127.0.0.1 = 0; 127.0.0.3 = 0; };
JMX on the command line $ echo "set -b org.apache.cassandra.db:type=CompactionManager MaximumCompactorThreads 6" | java -jar jmxterm-1.0-alpha-4-uber.jar -l 172.17.41.232:7199 && echo "set -b org.apache.cassandra.db:type=CompactionManager CoreCompactorThreads 6" | java -jar jmxterm-1.0-alpha-4-uber.jar -l 172.17.41.232:7199 setting compaction threads in one shot
JMX on the command line $ echo "set -b org.apache.cassandra.db:type=CompactionManager MaximumCompactorThreads 6" | java -jar jmxterm-1.0-alpha-4-uber.jar -l 172.17.41.232:7199 && echo "set -b org.apache.cassandra.db:type=CompactionManager CoreCompactorThreads 6" | java -jar jmxterm-1.0-alpha-4-uber.jar -l 172.17.41.232:7199 just echo and 'pipe' setting compaction threads in one shot
$>bean org.apache.cassandra.db:type=BatchlogManager #bean is set to org.apache.cassandra.db:type=BatchlogManager $>info #mbean = org.apache.cassandra.db:type=BatchlogManager #class name = org.apache.cassandra.batchlog.BatchlogManager # attributes %0 - TotalBatchesReplayed (long, r) # operations %0 - int countAllBatches() %1 - void forceBatchlogReplay() #there's no notifications $>get TotalBatchesReplayed #mbean = org.apache.cassandra.db:type=BatchlogManager: TotalBatchesReplayed = 0; $>run countAllBatches #calling operation countAllBatches of mbean org.apache.cassandra.db:type=BatchlogManager #operation returns: 0 JMX on the command line: general syntax
$>bean org.apache.cassandra.db:type=BatchlogManager #bean is set to org.apache.cassandra.db:type=BatchlogManager $>info #mbean = org.apache.cassandra.db:type=BatchlogManager #class name = org.apache.cassandra.batchlog.BatchlogManager # attributes %0 - TotalBatchesReplayed (long, r) # operations %0 - int countAllBatches() %1 - void forceBatchlogReplay() #there's no notifications $>get TotalBatchesReplayed #mbean = org.apache.cassandra.db:type=BatchlogManager: TotalBatchesReplayed = 0; $>run countAllBatches #calling operation countAllBatches of mbean org.apache.cassandra.db:type=BatchlogManager #operation returns: 0 JMX on the command line: general syntax Set the bean
$>bean org.apache.cassandra.db:type=BatchlogManager #bean is set to org.apache.cassandra.db:type=BatchlogManager $>info #mbean = org.apache.cassandra.db:type=BatchlogManager #class name = org.apache.cassandra.batchlog.BatchlogManager # attributes %0 - TotalBatchesReplayed (long, r) # operations %0 - int countAllBatches() %1 - void forceBatchlogReplay() #there's no notifications $>get TotalBatchesReplayed #mbean = org.apache.cassandra.db:type=BatchlogManager: TotalBatchesReplayed = 0; $>run countAllBatches #calling operation countAllBatches of mbean org.apache.cassandra.db:type=BatchlogManager #operation returns: 0 'info' for what is available JMX on the command line: general syntax
$>bean org.apache.cassandra.db:type=BatchlogManager #bean is set to org.apache.cassandra.db:type=BatchlogManager $>info #mbean = org.apache.cassandra.db:type=BatchlogManager #class name = org.apache.cassandra.batchlog.BatchlogManager # attributes %0 - TotalBatchesReplayed (long, r) # operations %0 - int countAllBatches() %1 - void forceBatchlogReplay() #there's no notifications $>get TotalBatchesReplayed #mbean = org.apache.cassandra.db:type=BatchlogManager: TotalBatchesReplayed = 0; $>run countAllBatches #calling operation countAllBatches of mbean org.apache.cassandra.db:type=BatchlogManager #operation returns: 0 'get' an attribute JMX on the command line: general syntax
$>bean org.apache.cassandra.db:type=BatchlogManager #bean is set to org.apache.cassandra.db:type=BatchlogManager $>info #mbean = org.apache.cassandra.db:type=BatchlogManager #class name = org.apache.cassandra.batchlog.BatchlogManager # attributes %0 - TotalBatchesReplayed (long, r) # operations %0 - int countAllBatches() %1 - void forceBatchlogReplay() #there's no notifications $>get TotalBatchesReplayed #mbean = org.apache.cassandra.db:type=BatchlogManager: TotalBatchesReplayed = 0; $>run countAllBatches #calling operation countAllBatches of mbean org.apache.cassandra.db:type=BatchlogManager #operation returns: 0 'run' an operation JMX on the command line: general syntax
JMX on JConsole
JMX on JConsole Note: Complex-type (like arrays) are grayed out
JMX on JConsole Note: Complex-type (like arrays) are grayed out tool-specific invocations
JMX on JConsole Getting object name and attributes for CLI invocation
A note about remote connections
cassandra-env.sh using LOCAL_JMX="no"
cassandra-env.sh false not there by default using LOCAL_JMX="no"
2. Visibility and exploration
org.apache.cassandra.metrics.* (Hopefully you saw Alain's monitoring talk)
troubleshooting cluster communication
cluster communication Looking at small vs. large tasks (>64k) provides an idea of network efficiency
cluster communication what does this node think of the others?
cluster communication Click the value to see PhiValues for other nodes
Commitlog efficiency: waiting on commit If this stays close to commitlog_size_in_mb there are bigger issues
Commitlog efficiency: waiting on commit
Commitlog efficiency: waiting on commit If this is high, you need to tune Time spent waiting for sync. This will climb if sync is lagging sync interval
Commitlog efficiency: segment allocation time spent waiting on segment allocation
Tombstone impact and compaction SSTable max local deletion time: 2147483647 Compression ratio: 0.5167785234899329 Estimated droppable tombstones: 0.3175 SSTable Level: 0 Repaired at: 0 Estimated tombstone drop times: 1471577046: 2 1471577146: 2 la-8-big-Data.db
Tombstone impact and compaction SSTable max local deletion time: 2147483647 Compression ratio: 0.5167785234899329 Estimated droppable tombstones: 0.3175 SSTable Level: 0 Repaired at: 0 Estimated tombstone drop times: 1471577046: 2 1471577146: 2 la-8-big-Data.db Go see Alain's other talk
Tombstone impact and compaction SSTable max local deletion time: 2147483647 Compression ratio: 0.5167785234899329 Estimated droppable tombstones: 0.3175 SSTable Level: 0 Repaired at: 0 Estimated tombstone drop times: 1471577046: 2 1471577146: 2 la-8-big-Data.db Go see Alain's other talk Or just read his blog: http://thelastpickle.com/blog/2016/07/27/about-deletes-and-tombstones.html
droppable tombstone ratio clicking on the value provides a handy chart!
3. Useful management incantations
Routing client requests away Set to '50.0' for routing away requests
Figuring out if DynamicSnitch is kneecapping you https://issues.apache.org/jira/browse/ CASSANDRA-6908
$>get Scores #mbean = org.apache.cassandra.db:type=DynamicEndpointSnitch: Scores = { /10.10.20.31 = 2.244811534881592; /10.10.20.30 = 0.5457598567008972; /10.10.20.37 = 2.7038445472717285; /10.10.20.36 = 3.0801687240600586; /10.10.20.39 = 6.231578826904297; /10.10.20.38 = 3.1578946113586426; /10.10.20.33 = 12.003381729125977; /10.10.20.32 = 3.348876476287842; /10.10.20.35 = 1.5086206197738647; /10.10.20.34 = 4.9235992431640625; /10.20.1.46 = 2.621400833129883; /10.20.1.44 = 1.3118916749954224; /10.20.1.45 = 1.6884760856628418; /10.30.50.62 = 2.4780631123519523; /10.30.50.63 = 2.3196894221189543; /10.30.50.61 = 1.2922532529365727; /10.20.1.50 = 7.66961669921875; /10.20.1.48 = 1.686340570449829; /10.30.50.68 = 0.9854942156774241; /10.20.8.81 = 0.7212558388710022; /10.30.50.78 = 2.4760895589502847; /10.30.50.79 = 1.2857443196017568; /10.30.50.76 = 0.7804878048780488; /10.30.50.77 = 1.8351790061811122; /10.20.1.102 = 2.4894514083862305; /10.20.1.103 = 0.5889776945114136; /10.20.1.106 = 0.8040626645088196; /10.20.1.104 = 1.4327855110168457; /10.20.1.105 = 0.75789475440979; }; DynamicSnitch: ouch! most common with real fast hardware - dramatic fluctuations - 22x difference!
Turn it off via StorageService#updateSnitch: GossipingPropertyFileSnitch, false, 0, 0, 0 DynamicSnitch: disabling
Turn it off via StorageService#updateSnitch: GossipingPropertyFileSnitch, false, 0, 0, 0 DynamicSnitch: disabling false
Turn it off via StorageService#updateSnitch: GossipingPropertyFileSnitch, false, 0, 0, 0 DynamicSnitch: disabling I have no idea what will happen if you change the snitch
Controlling Compaction
CompactionManager: compaction throttling++ - MaximumCompactorThreads (change this first!!!) - CoreCompactorThreads (tip: see jmxterm example from earlier for CLI invocation)
CompactionManager: compact a set of stables one or more "*-Data.db" files, comma-delimited
jmx_adventure/account-7029d8406d8411e6b3678b7e928ec63e/mb-2-big-Data.db Version dependent filename CompactionManager: compact a set of stables
jmx_adventure/account-7029d8406d8411e6b3678b7e928ec63e/mb-2-big-Data.db Version dependent filename keyspace CompactionManager: compact a set of stables
jmx_adventure/account-7029d8406d8411e6b3678b7e928ec63e/mb-2-big-Data.db Version dependent filename table + version UUID CompactionManager: compact a set of stables
jmx_adventure/account-7029d8406d8411e6b3678b7e928ec63e/mb-2-big-Data.db Version dependent filename data file CompactionManager: compact a set of stables
CompactionManager: stop compaction type of compaction, most commonly: COMPACTION, VALIDATION, ANTI_COMPACTION, INDEX_SUMMARY https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/db/compaction/OperationType.java
Compaction identifier of active compaction CompactionManager: stop compaction
Find the identifier of active compactions here (or nodetool compactionstats) CompactionManager: stop compaction
Update compaction strategy CompactionParametersJson: Editable JSON
CompactionParametersJson: Editable JSON test compaction settings on single node!!! Update compaction strategy
Managing hinted handoff
HintedHandoffManager
Delete all hints or by host HintedHandoffManager
Pause and resume delivery by host HintedHandoffManager
StorageProxy: Adjust hint delivery settings
StorageProxy: Adjust hint delivery settings On/off switch for hints
StorageProxy: Adjust hint delivery settings How long to keep hints in milliseconds (3 hours by default)
StorageProxy: Adjust hint delivery settings How many hints to send at a time
Adjusting operation timeouts
StorageProxy: Adjust operation timeouts Amount of time spent in LWT Contention
StorageProxy: Adjust operation timeouts Amount of time spent in LWT Contention See Chris' LWT presentation!
StorageProxy: Adjust operation timeouts timeout for counter writes
StorageProxy: Adjust operation timeouts Timeout for range queries (poorly bounded select statements)
StorageProxy: Adjust operation timeouts Timeout for single read operation
StorageProxy: Adjust operation timeouts Timeout for other miscellaneous inter-cluster operations
StorageProxy: Adjust operation timeouts Timeout for write operations
StorageProxy: Adjust operation timeouts special shoutout to TruncateRpcTimeout cqlsh> truncate table account; TimeoutException...
StorageProxy: Adjust operation timeouts If you truncate, increase this by an order of magnitude
Index sampling and summary SSTable
Index summaries Invoke occasionally on a single node
Index summaries Invoke occasionally on a single node Warning: expensive operation!
Index summaries watch the metrics for tuning
Index summaries Table: account_event ... Index summary off heap memory used: 27732 ... combine with nodetool tablestats, then adjust: min_index_interval, max_index_interval
StorageService ...miscellaneous, etc
StorageService: batch size thresholds Size of the largest batch in KB
StorageService: tombstone thresholds Adjust failure and warning to remove log messages
StorageService: tombstone thresholds Adjust failure and warning to remove log messages If you adjust these, you have bigger problems
force stop repair Does just what it says.
bulk load (sync and async) File path as arguments: /some/path/${keyspace}/${table}/
Recover commit log segments Helpful for reloading commitlogs from a backup
4. Access Control: locking down JMX
nmap -Pn -p7199 -oG logs/cass.gnmap 54.88.0.0/14 Haha! You exposed a command plane to the internet
nmap -Pn -p7199 -oG logs/cass.gnmap 54.88.0.0/14 Haha! You exposed a command plane to the internet Shameless Plug!
Why do I need to secure JMX?
Why do I need to secure JMX? StorageService.truncate("your_keyspace", "your_table")
Why do I need to secure JMX? StorageService.truncate("your_keyspace", "your_table")
Why do I need to secure JMX? StorageService.truncate("your_keyspace", "your_table") Now with internet access!!
Why do I need to secure JMX? StorageService.truncate("your_keyspace", "your_table") Now with internet access!! Works as Advertised!
Why do I need to secure JMX? StorageService.removeNode("coworker_IPADR")
Why do I need to secure JMX? StorageService.removeNode("coworker_IPADR") Fun for the whole development team!
Securing JMX SSL setup is like node to node and client to server http://docs.oracle.com/javase/8/docs/technotes/guides/management/agent.html
Securing JMX JMX Authentication is straightforward and well documented $JAVA_HOME/jre/lib/management/jmxremote.access $JAVA_HOME/jre/lib/management/ jmxremote.password.template http://docs.oracle.com/javase/8/docs/technotes/guides/management/agent.html
Securing JMX $JAVA_HOME/jre/lib/management/jmxremote.access $JAVA_HOME/jre/lib/management/ jmxremote.password.template Now you can: nodetool -u admin -pw secret compactionstats http://docs.oracle.com/javase/8/docs/technotes/guides/management/agent.html JMX Authentication is straightforward and well documented
Securing JMX $JAVA_HOME/jre/lib/management/jmxremote.access $JAVA_HOME/jre/lib/management/ jmxremote.password.template Now you can: nodetool -u admin -pw secret compactionstats Tip: -pwf option will read the password from a file http://docs.oracle.com/javase/8/docs/technotes/guides/management/agent.html JMX Authentication is straightforward and well documented
Securing JMX $JAVA_HOME/jre/lib/management/jmxremote.access $JAVA_HOME/jre/lib/management/ jmxremote.password.template Now you can: nodetool -u admin -pw secret compactionstats JMX Authentication is straightforward and well documented RBAC for JMX Authentication and Authorization is now supported https://issues.apache.org/jira/browse/CASSANDRA-10091
Thanks!@zznate

More Related Content

PDF
How Development Teams Cut Costs with ScyllaDB.pdf
byScyllaDB
 
PPTX
Exactly-Once Made Easy: Transactional Messaging in Apache Pulsar - Pulsar Sum...
byStreamNative
 
PPTX
Cassandra Troubleshooting 3.0
byJ.B. Langston
 
PDF
9 DevOps Tips for Going in Production with Galera Cluster for MySQL - Slides
bySeveralnines
 
PDF
MySQL on AWS RDS
byMydbops
 
PDF
Introduction to Apache Cassandra
byRobert Stupp
 
PDF
Storing tree structures with MongoDB
byVyacheslav
 
PPTX
トランザクションをSerializableにする4つの方法
byKumazaki Hiroki
 
How Development Teams Cut Costs with ScyllaDB.pdf
byScyllaDB
 
Exactly-Once Made Easy: Transactional Messaging in Apache Pulsar - Pulsar Sum...
byStreamNative
 
Cassandra Troubleshooting 3.0
byJ.B. Langston
 
9 DevOps Tips for Going in Production with Galera Cluster for MySQL - Slides
bySeveralnines
 
MySQL on AWS RDS
byMydbops
 
Introduction to Apache Cassandra
byRobert Stupp
 
Storing tree structures with MongoDB
byVyacheslav
 
トランザクションをSerializableにする4つの方法
byKumazaki Hiroki
 

What's hot

PDF
Deep Dive into Stateful Stream Processing in Structured Streaming with Tathag...
byDatabricks
 
PDF
Bringing Kafka Without Zookeeper Into Production with Colin McCabe | Kafka Su...
byHostedbyConfluent
 
PDF
Optimizing Kubernetes Resource Requests/Limits for Cost-Efficiency and Latenc...
byHenning Jacobs
 
PDF
Learning how AWS implement AWS VPC CNI
byHungWei Chiu
 
PPTX
High Availability and Disaster Recovery in PostgreSQL - EQUNIX
byJulyanto SUTANDANG
 
PDF
Gardener: Managed Kubernetes on Your Terms
bySAP HANA Cloud Platform
 
PPTX
OpenStack High Availability
byJakub Pavlik
 
PPTX
Ansible: Infrastructure as Code for OpenShift
byIgnacio Sánchez Ginés
 
PDF
How netflix manages petabyte scale apache cassandra in the cloud
byVinay Kumar Chella
 
PDF
Event Streaming in Retail with Apache Kafka
byKai Wähner
 
PDF
Istio service mesh introduction
byKyohei Mizumoto
 
PDF
Apache kafka 모니터링을 위한 Metrics 이해 및 최적화 방안
bySANG WON PARK
 
PDF
Multi Master PostgreSQL Cluster on Kubernetes
byOhyama Masanori
 
PDF
Go, memcached, microservices
bymosa siru
 
KEY
大規模環境でRailsと4年間付き合ってきて@ クックパッド * 食べログ合同勉強会
byTakayuki Kyowa
 
PPTX
Jvm tuning for low latency application & Cassandra
byQuentin Ambard
 
PDF
Meet Spilo, Zalando’s HIGH-AVAILABLE POSTGRESQL CLUSTER - Feike Steenbergen
bydistributed matters
 
PPTX
急速に進化を続けるCNIプラグイン Antrea
byMotonori Shindo
 
PPTX
ストリーム処理におけるApache Avroの活用について（NTTデータ テクノロジーカンファレンス 2019 講演資料、2019/09/05）
byNTT DATA Technology & Innovation
 
PPTX
Redisの特徴と活用方法について
byYuji Otani
 
Deep Dive into Stateful Stream Processing in Structured Streaming with Tathag...
byDatabricks
 
Bringing Kafka Without Zookeeper Into Production with Colin McCabe | Kafka Su...
byHostedbyConfluent
 
Optimizing Kubernetes Resource Requests/Limits for Cost-Efficiency and Latenc...
byHenning Jacobs
 
Learning how AWS implement AWS VPC CNI
byHungWei Chiu
 
High Availability and Disaster Recovery in PostgreSQL - EQUNIX
byJulyanto SUTANDANG
 
Gardener: Managed Kubernetes on Your Terms
bySAP HANA Cloud Platform
 
OpenStack High Availability
byJakub Pavlik
 
Ansible: Infrastructure as Code for OpenShift
byIgnacio Sánchez Ginés
 
How netflix manages petabyte scale apache cassandra in the cloud
byVinay Kumar Chella
 
Event Streaming in Retail with Apache Kafka
byKai Wähner
 
Istio service mesh introduction
byKyohei Mizumoto
 
Apache kafka 모니터링을 위한 Metrics 이해 및 최적화 방안
bySANG WON PARK
 
Multi Master PostgreSQL Cluster on Kubernetes
byOhyama Masanori
 
Go, memcached, microservices
bymosa siru
 
大規模環境でRailsと4年間付き合ってきて@ クックパッド * 食べログ合同勉強会
byTakayuki Kyowa
 
Jvm tuning for low latency application & Cassandra
byQuentin Ambard
 
Meet Spilo, Zalando’s HIGH-AVAILABLE POSTGRESQL CLUSTER - Feike Steenbergen
bydistributed matters
 
急速に進化を続けるCNIプラグイン Antrea
byMotonori Shindo
 
ストリーム処理におけるApache Avroの活用について（NTTデータ テクノロジーカンファレンス 2019 講演資料、2019/09/05）
byNTT DATA Technology & Innovation
 
Redisの特徴と活用方法について
byYuji Otani
 

Viewers also liked

PDF
Bulk Loading Data into Cassandra
byDataStax
 
PDF
ETL With Cassandra Streaming Bulk Loading
byalex_araujo
 
PDF
DataStax: Extreme Cassandra Optimization: The Sequel
byDataStax Academy
 
PDF
Hardening cassandra for compliance or paranoia
byzznate
 
PDF
Cassandra at teads
byRomain Hardouin
 
ODP
Introduction to Apache Cassandra
byKnoldus Inc.
 
PDF
Open Source Monitoring for Java with JMX and Graphite (GeeCON 2013)
byCyrille Le Clerc
 
PDF
The Last Pickle: Hardening Apache Cassandra for Compliance (or Paranoia).
byDataStax Academy
 
PDF
How Cassandra Deletes Data (Alain Rodriguez, The Last Pickle) | Cassandra Sum...
byDataStax
 
PPTX
Cassandra Metrics
byChris Lohfink
 
PDF
Cassandra Summit 2014: Performance Tuning Cassandra in AWS
byDataStax Academy
 
PPTX
Cassandra in Operation
byniallmilton
 
PDF
CQL performance with Apache Cassandra 3.0 (Aaron Morton, The Last Pickle) | C...
byDataStax
 
PPTX
Cassandra Troubleshooting (for 2.0 and earlier)
byJ.B. Langston
 
PDF
Hardening cassandra q2_2016
byzznate
 
PPTX
Building Data Pipelines with SMACK: Designing Storage Strategies for Scale an...
byDataStax
 
PDF
Cassandra 3.0
byRobert Stupp
 
PPTX
Performance tuning - A key to successful cassandra migration
byRamkumar Nottath
 
PDF
Jolokia - JMX on Capsaicin (Devoxx 2011)
byroland.huss
 
PDF
Wayne State University & DataStax: World's best data modeling tool for Apache...
byDataStax Academy
 
Bulk Loading Data into Cassandra
byDataStax
 
ETL With Cassandra Streaming Bulk Loading
byalex_araujo
 
DataStax: Extreme Cassandra Optimization: The Sequel
byDataStax Academy
 
Hardening cassandra for compliance or paranoia
byzznate
 
Cassandra at teads
byRomain Hardouin
 
Introduction to Apache Cassandra
byKnoldus Inc.
 
Open Source Monitoring for Java with JMX and Graphite (GeeCON 2013)
byCyrille Le Clerc
 
The Last Pickle: Hardening Apache Cassandra for Compliance (or Paranoia).
byDataStax Academy
 
How Cassandra Deletes Data (Alain Rodriguez, The Last Pickle) | Cassandra Sum...
byDataStax
 
Cassandra Metrics
byChris Lohfink
 
Cassandra Summit 2014: Performance Tuning Cassandra in AWS
byDataStax Academy
 
Cassandra in Operation
byniallmilton
 
CQL performance with Apache Cassandra 3.0 (Aaron Morton, The Last Pickle) | C...
byDataStax
 
Cassandra Troubleshooting (for 2.0 and earlier)
byJ.B. Langston
 
Hardening cassandra q2_2016
byzznate
 
Building Data Pipelines with SMACK: Designing Storage Strategies for Scale an...
byDataStax
 
Cassandra 3.0
byRobert Stupp
 
Performance tuning - A key to successful cassandra migration
byRamkumar Nottath
 
Jolokia - JMX on Capsaicin (Devoxx 2011)
byroland.huss
 
Wayne State University & DataStax: World's best data modeling tool for Apache...
byDataStax Academy
 

Similar to Advanced Apache Cassandra Operations with JMX

PPTX
Cassandra Troubleshooting for 2.1 and later
byJ.B. Langston
 
PPTX
Riak add presentation
byIlya Bogunov
 
PDF
Cassandra for Sysadmins
byNathan Milford
 
PDF
Advanced Operations
byDataStax Academy
 
PDF
Standing Up Your First Cluster
byDataStax Academy
 
PDF
Performance tuning
byJon Haddad
 
PDF
Cassandra Community Webinar | In Case of Emergency Break Glass
byDataStax
 
PDF
Cassandra Community Webinar August 29th 2013 - In Case Of Emergency, Break Glass
byaaronmorton
 
PDF
Cassandra SF 2013 - In Case Of Emergency Break Glass
byaaronmorton
 
PPTX
Ops Jumpstart: MongoDB Administration 101
byMongoDB
 
PPTX
Monitoring MongoDB’s Engines in the Wild
byTim Vaillancourt
 
KEY
Deployment Strategies (Mongo Austin)
byMongoDB
 
PDF
MongoDB and server performance
byAlon Horev
 
PDF
Cassandra Day London 2015: Diagnosing Problems in Production
byDataStax Academy
 
PDF
Cassandra Day Atlanta 2015: Diagnosing Problems in Production
byDataStax Academy
 
PDF
Cassandra Day Chicago 2015: Diagnosing Problems in Production
byDataStax Academy
 
PDF
Diagnosing Problems in Production - Cassandra
byJon Haddad
 
PDF
Diagnosing Problems in Production (Nov 2015)
byJon Haddad
 
PDF
High performance Infrastructure Oct 2013
byServer Density
 
PDF
Joel Jacobson (Datastax) - Diagnosing Cassandra Problems in Production
byOutlyer
 
Cassandra Troubleshooting for 2.1 and later
byJ.B. Langston
 
Riak add presentation
byIlya Bogunov
 
Cassandra for Sysadmins
byNathan Milford
 
Advanced Operations
byDataStax Academy
 
Standing Up Your First Cluster
byDataStax Academy
 
Performance tuning
byJon Haddad
 
Cassandra Community Webinar | In Case of Emergency Break Glass
byDataStax
 
Cassandra Community Webinar August 29th 2013 - In Case Of Emergency, Break Glass
byaaronmorton
 
Cassandra SF 2013 - In Case Of Emergency Break Glass
byaaronmorton
 
Ops Jumpstart: MongoDB Administration 101
byMongoDB
 
Monitoring MongoDB’s Engines in the Wild
byTim Vaillancourt
 
Deployment Strategies (Mongo Austin)
byMongoDB
 
MongoDB and server performance
byAlon Horev
 
Cassandra Day London 2015: Diagnosing Problems in Production
byDataStax Academy
 
Cassandra Day Atlanta 2015: Diagnosing Problems in Production
byDataStax Academy
 
Cassandra Day Chicago 2015: Diagnosing Problems in Production
byDataStax Academy
 
Diagnosing Problems in Production - Cassandra
byJon Haddad
 
Diagnosing Problems in Production (Nov 2015)
byJon Haddad
 
High performance Infrastructure Oct 2013
byServer Density
 
Joel Jacobson (Datastax) - Diagnosing Cassandra Problems in Production
byOutlyer
 

More from zznate

PDF
An Introduction to the Vert.x framework
byzznate
 
ODP
Meetup cassandra for_java_cql
byzznate
 
ODP
Introduciton to Apache Cassandra for Java Developers (JavaOne)
byzznate
 
ODP
Introduction to apache_cassandra_for_developers-lhg
byzznate
 
ODP
Nyc summit intro_to_cassandra
byzznate
 
PPT
Introduction to apache_cassandra_for_develope
byzznate
 
PDF
Software Development with Apache Cassandra
byzznate
 
PDF
Stampede con 2014 cassandra in the real world
byzznate
 
ODP
Meetup cassandra sfo_jdbc
byzznate
 
PDF
Successful Software Development with Apache Cassandra
byzznate
 
PDF
Seattle C* Meetup: Hardening cassandra for compliance or paranoia
byzznate
 
PPT
Hector v2: The Second Version of the Popular High-Level Java Client for Apach...
byzznate
 
KEY
Oscon 2012 tdd_cassandra
byzznate
 
PPTX
Strata west 2012_java_cassandra
byzznate
 
PDF
Apachecon cassandra transport
byzznate
 
PDF
Intravert atx meetup_condensed
byzznate
 
An Introduction to the Vert.x framework
byzznate
 
Meetup cassandra for_java_cql
byzznate
 
Introduciton to Apache Cassandra for Java Developers (JavaOne)
byzznate
 
Introduction to apache_cassandra_for_developers-lhg
byzznate
 
Nyc summit intro_to_cassandra
byzznate
 
Introduction to apache_cassandra_for_develope
byzznate
 
Software Development with Apache Cassandra
byzznate
 
Stampede con 2014 cassandra in the real world
byzznate
 
Meetup cassandra sfo_jdbc
byzznate
 
Successful Software Development with Apache Cassandra
byzznate
 
Seattle C* Meetup: Hardening cassandra for compliance or paranoia
byzznate
 
Hector v2: The Second Version of the Popular High-Level Java Client for Apach...
byzznate
 
Oscon 2012 tdd_cassandra
byzznate
 
Strata west 2012_java_cassandra
byzznate
 
Apachecon cassandra transport
byzznate
 
Intravert atx meetup_condensed
byzznate
 

Recently uploaded

PDF
Using Change Data Capture (CDC) to Ingest Data into Apache Kafka
byGiovanni Marigi
 
PDF
JR : Quality Random Data from the Command line
byGiovanni Marigi
 
PDF
Building Voice Agents with Google Agent Development Kit
bySuresh Peiris
 
PDF
UiPath Coded Agents - A Developer Driven Agentic Automation Era
byUiPathCommunity
 
PDF
A "Kill Switch" (Emergency Off Switch) for AI?
byScott M. Graffius
 
PDF
SEO in Charleston SC | Local SEO Strategies to Grow Your Business.pdf
bycustomdigitalsolutio1
 
PDF
Louisville User Group: Transforming Technical Debt into Technical Wealth
byLynda Kane
 
PDF
Chapter 2 Computer Threat .pdf
byGetnet Tigabie Askale -(GM)
 
PDF
Top 10 Dedicated Server Providers in Stockholm (Updated Edition).pdf
byAtal Networks
 
PDF
XonTel Plus IP-PBX Business Phone System
byXonTel Technology
 
PDF
AI Data Analyst: Smarter Insights for Modern Real Estate Decisions
byLeni Analyst
 
PDF
SAP WalkMe Overview.pdf SAP WalkMe Overview.pdf
byMurniatiSimbolon2
 
PPTX
Building Real-Time Voice AI — Udaiappa Ramachandran
byUdaiappa Ramachandran
 
PPTX
KTU-PEMET413 -MODULE 2-POLYMER MATRIX COMPOSITES - LECTURE 1.pptx
byVINAY B
 
PDF
Josh Chu Boston - An Innovative Technology Executive
byJosh Chu Boston
 
PPTX
02_Extended Warehouse Management Functions and Features.pptx
byUdayakumar218983
 
PPT
Waste water treatment plant operation and maintenance
byDuggineniRamakrishna
 
PPTX
Lecture 05. API Security for DevSecOps Eng.pptx
byvinocol222
 
PPTX
Lecture 04. about Black heads and Hackerss.pptx
byvinocol222
 
PPTX
AI_in_Cybersecurity_Insights_Case_Studies.pptx
bywrksmith9
 
Using Change Data Capture (CDC) to Ingest Data into Apache Kafka
byGiovanni Marigi
 
JR : Quality Random Data from the Command line
byGiovanni Marigi
 
Building Voice Agents with Google Agent Development Kit
bySuresh Peiris
 
UiPath Coded Agents - A Developer Driven Agentic Automation Era
byUiPathCommunity
 
A "Kill Switch" (Emergency Off Switch) for AI?
byScott M. Graffius
 
SEO in Charleston SC | Local SEO Strategies to Grow Your Business.pdf
bycustomdigitalsolutio1
 
Louisville User Group: Transforming Technical Debt into Technical Wealth
byLynda Kane
 
Chapter 2 Computer Threat .pdf
byGetnet Tigabie Askale -(GM)
 
Top 10 Dedicated Server Providers in Stockholm (Updated Edition).pdf
byAtal Networks
 
XonTel Plus IP-PBX Business Phone System
byXonTel Technology
 
AI Data Analyst: Smarter Insights for Modern Real Estate Decisions
byLeni Analyst
 
SAP WalkMe Overview.pdf SAP WalkMe Overview.pdf
byMurniatiSimbolon2
 
Building Real-Time Voice AI — Udaiappa Ramachandran
byUdaiappa Ramachandran
 
KTU-PEMET413 -MODULE 2-POLYMER MATRIX COMPOSITES - LECTURE 1.pptx
byVINAY B
 
Josh Chu Boston - An Innovative Technology Executive
byJosh Chu Boston
 
02_Extended Warehouse Management Functions and Features.pptx
byUdayakumar218983
 
Waste water treatment plant operation and maintenance
byDuggineniRamakrishna
 
Lecture 05. API Security for DevSecOps Eng.pptx
byvinocol222
 
Lecture 04. about Black heads and Hackerss.pptx
byvinocol222
 
AI_in_Cybersecurity_Insights_Case_Studies.pptx
bywrksmith9
 

Advanced Apache Cassandra Operations with JMX

  • 1.
    JMX and Cassandra Understandingtooling for advanced operations
  • 2.
    Hi, I'm Nate. @zznate https://www.linkedin.com/in/zznate http://www.slideshare.net/zznate/ Co-Founder,CTO The Last Pickle Apache Cassandra PMC member User since 2009 (v0.4) Austin, Texas Wellington, NZ
  • 3.
    TLP: not just"Aaron and Nate"
  • 4.
    Lots of information inthe next 30 minutes
  • 5.
    1. Tools 2. Visibilityand exploration 3. Operations and invocation 4. Access controls JMX and Cassandra
  • 6.
    1. Summary ofthe tools
  • 7.
  • 8.
    Nodetool: the cruiseline of cluster administration
  • 9.
    output of nodetoolhelp (87 commands) usage: nodetool [(-u <username> | --username <username>)] [(-pw <password> | --password <password>)] [(-h <host> | --host <host>)] [(-p <port> | --port <port>)] [(-pwf <passwordFilePath> | --password-file <passwordFilePath>)] <command> [<args>] The most commonly used nodetool commands are: assassinate Forcefully remove a dead node without re-replicating any data. Use as a last resort if you cannot removenode bootstrap Monitor/manage node's bootstrap process cleanup Triggers the immediate cleanup of keys no longer belonging to a node. By default, clean all keyspaces clearsnapshot Remove the snapshot with the given name from the given keyspaces. If no snapshotName is specified we will remove all snapshots compact Force a (major) compaction on one or more tables compactionhistory Print history of compaction compactionstats Print statistics on compactions decommission Decommission the *node I am connecting to* describecluster Print the name, snitch, partitioner and schema version of a cluster describering Shows the token ranges info of a given keyspace disableautocompaction Disable autocompaction for the given keyspace and table disablebackup Disable incremental backup disablebinary Disable native transport (binary protocol) disablegossip Disable gossip (effectively marking the node down) disablehandoff Disable storing hinted handoffs disablehintsfordc Disable hints for a data center disablethrift Disable thrift server drain Drain the node (stop accepting writes and flush all tables) enableautocompaction Enable autocompaction for the given keyspace and table enablebackup Enable incremental backup enablebinary Reenable native transport (binary protocol) enablegossip Reenable gossip enablehandoff Reenable future hints storing on the current node enablehintsfordc Enable hints for a data center that was previsouly disabled enablethrift Reenable thrift server failuredetector Shows the failure detector information for the cluster flush Flush one or more tables gcstats Print GC Statistics getcompactionthreshold Print min and max compaction thresholds for a given table getcompactionthroughput Print the MB/s throughput cap for compaction in the system getendpoints Print the end points that owns the key getinterdcstreamthroughput Print the Mb/s throughput cap for inter-datacenter streaming in the system getlogginglevels Get the runtime logging levels getsstables Print the sstable filenames that own the key getstreamthroughput Print the Mb/s throughput cap for streaming in the system gettraceprobability Print the current trace probability value gossipinfo Shows the gossip information for the cluster help Display help information info Print node information (uptime, load, ...) invalidatecountercache Invalidate the counter cache invalidatekeycache Invalidate the key cache invalidaterowcache Invalidate the row cache join Join the ring listsnapshots Lists all the snapshots along with the size on disk and true size. move Move node on the token ring to a new token netstats Print network information on provided host (connecting node by default) pausehandoff Pause hints delivery process proxyhistograms Print statistic histograms for network operations rangekeysample Shows the sampled keys held across all keyspaces rebuild Rebuild data by streaming from other nodes (similarly to bootstrap) rebuild_index A full rebuild of native secondary indexes for a given table refresh Load newly placed SSTables to the system without restart refreshsizeestimates Refresh system.size_estimates reloadtriggers Reload trigger classes removenode Show status of current node removal, force completion of pending removal or remove provided ID repair Repair one or more tables replaybatchlog Kick off batchlog replay and wait for finish resetlocalschema Reset node's local schema and resync resumehandoff Resume hints delivery process ring Print information about the token ring scrub Scrub (rebuild sstables for) one or more tables setcachecapacity Set global key, row, and counter cache capacities (in MB units) setcachekeystosave Set number of keys saved by each cache for faster post-restart warmup. 0 to disable setcompactionthreshold Set min and max compaction thresholds for a given table setcompactionthroughput Set the MB/s throughput cap for compaction in the system, or 0 to disable throttling sethintedhandoffthrottlekb Set hinted handoff throttle in kb per second, per delivery thread. setinterdcstreamthroughput Set the Mb/s throughput cap for inter-datacenter streaming in the system, or 0 to disable throttling setlogginglevel Set the log level threshold for a given class. If both class and level are empty/null, it will reset to the initial configuration setstreamthroughput Set the Mb/s throughput cap for streaming in the system, or 0 to disable throttling settraceprobability Sets the probability for tracing any given request to value. 0 disables, 1 enables for all requests, 0 is the default snapshot Take a snapshot of specified keyspaces or a snapshot of the specified table status Print cluster information (state, load, IDs, ...) statusbackup Status of incremental backup statusbinary Status of native transport (binary protocol) statusgossip Status of gossip statushandoff Status of storing future hints on the current node statusthrift Status of thrift server stop Stop compaction stopdaemon Stop cassandra daemon tablehistograms Print statistic histograms for a given table tablestats Print statistics on tables toppartitions Sample and print the most active partitions for a given column family tpstats Print usage statistics of thread pools truncatehints Truncate all hints on the local node, or truncate hints for the endpoint(s) specified. upgradesstables Rewrite sstables (for the requested tables) that are not on the current version (thus upgrading them to said current version) verify Verify (check data checksum for) one or more tables version Print cassandra version See 'nodetool help <command>' for more information on a specific command.
  • 10.
    nodetool commands: info ID: fed6dd09-667c-4cc5-b66a-85a1cbce1b58 Gossip active : true Thrift active : false Native Transport active: true Load : 219.97 KB Generation No : 1473125362 Uptime (seconds) : 28802 Heap Memory (MB) : 86.96 / 495.00 Off Heap Memory (MB) : 0.00 Data Center : datacenter1 Rack : rack1 Exceptions : 0 Key Cache : entries 27, size 2.15 KB, capacity 24 MB, 274 hits, 313 requests, 0.875 recent hit rate, 14400 save period in seconds Row Cache : entries 0, size 0 bytes, capacity 0 bytes, 0 hits, 0 requests, NaN recent hit rate, 0 save period in seconds Counter Cache : entries 0, size 0 bytes, capacity 12 MB, 0 hits, 0 requests, NaN recent hit rate, 7200 save period in seconds Token : -9223372036854775808
  • 11.
    nodetool commands: info ID: fed6dd09-667c-4cc5-b66a-85a1cbce1b58 Gossip active : true Thrift active : false Native Transport active: true Load : 219.97 KB Generation No : 1473125362 Uptime (seconds) : 28802 Heap Memory (MB) : 86.96 / 495.00 Off Heap Memory (MB) : 0.00 Data Center : datacenter1 Rack : rack1 Exceptions : 0 Key Cache : entries 27, size 2.15 KB, capacity 24 MB, 274 hits, 313 requests, 0.875 recent hit rate, 14400 save period in seconds Row Cache : entries 0, size 0 bytes, capacity 0 bytes, 0 hits, 0 requests, NaN recent hit rate, 0 save period in seconds Counter Cache : entries 0, size 0 bytes, capacity 12 MB, 0 hits, 0 requests, NaN recent hit rate, 7200 save period in seconds Token : -9223372036854775808
  • 12.
    nodetool commands: info ID: fed6dd09-667c-4cc5-b66a-85a1cbce1b58 Gossip active : true Thrift active : false Native Transport active: true Load : 219.97 KB Generation No : 1473125362 Uptime (seconds) : 28802 Heap Memory (MB) : 86.96 / 495.00 Off Heap Memory (MB) : 0.00 Data Center : datacenter1 Rack : rack1 Exceptions : 0 Key Cache : entries 27, size 2.15 KB, capacity 24 MB, 274 hits, 313 requests, 0.875 recent hit rate, 14400 save period Row Cache : entries 0, size 0 bytes, capacity 0 bytes, 0 hits, 0 requests, NaN recent hit rate, 0 save period in seconds Counter Cache : entries 0, size 0 bytes, capacity 12 MB, 0 hits, 0 requests, NaN recent hit rate, 7200 save period in seconds Token : -9223372036854775808
  • 13.
    nodetool commands: info ID: fed6dd09-667c-4cc5-b66a-85a1cbce1b58 Gossip active : true Thrift active : false Native Transport active: true Load : 219.97 KB Generation No : 1473125362 Uptime (seconds) : 28802 Heap Memory (MB) : 86.96 / 495.00 Off Heap Memory (MB) : 0.00 Data Center : datacenter1 Rack : rack1 Exceptions : 0 Key Cache : entries 27, size 2.15 KB, capacity 24 MB, 274 hits, 313 requests, 0.875 recent hit rate, 14400 save period Row Cache : entries 0, size 0 bytes, capacity 0 bytes, 0 hits, 0 requests, NaN recent hit rate, 0 save period in seconds Counter Cache : entries 0, size 0 bytes, capacity 12 MB, 0 hits, 0 requests, NaN recent hit rate, 7200 save period in seconds Token : -9223372036854775808 tip: nodetool setcachecapacity <key> <row> <counter>
  • 14.
    $ nodetool compactionstats-H pending tasks: 2101 id type keyspace table completed total progress d14fc600-6e3f-11e6-a01c-8bcdc0e93c44 Compaction ads ad_events 742.58 MB 4.1 TB 0.02% Active compaction remaining time : 37h18m47s nodetool commands: compactionstats
  • 15.
    $ nodetool compactionstats-H pending tasks: 2101 id type keyspace table completed total progress d14fc600-6e3f-11e6-a01c-8bcdc0e93c44 Compaction ads ad_events 742.58 MB 4.1 TB 0.02% Active compaction remaining time : 37h18m47s nodetool commands: compactionstats
  • 16.
    nodetool commands: compactionstats $nodetool compactionstats -H pending tasks: 2101 id type keyspace table completed total progress d14fc600-6e3f-11e6-a01c-8bcdc0e93c44 Compaction ads ad_events 742.58 MB 4.1 TB 0.02% Active compaction remaining time : 37h18m47s
  • 17.
    nodetool commands: tablestats Keyspace:jmx_adventure Read Count: 19 Read Latency: 0.3552105263157895 ms. Write Count: 3 Write Latency: 0.17766666666666667 ms. Pending Flushes: 0 Table: account_event SSTable count: 1 Space used (live): 5326 Space used (total): 5326 Space used by snapshots (total): 0 Off heap memory used (total): 51 SSTable Compression Ratio: 0.6517241379310345 Number of keys (estimate): 3 Memtable cell count: 0 Memtable data size: 0 Memtable off heap memory used: 0 Memtable switch count: 1 Local read count: 19 Local read latency: 0.386 ms Local write count: 3 Local write latency: 0.207 ms Pending flushes: 0 Bloom filter false positives: 0 Bloom filter false ratio: 0.00000 Bloom filter space used: 16 Bloom filter off heap memory used: 8 Index summary off heap memory used: 27 Compression metadata off heap memory used: 16 Compacted partition minimum bytes: 51 Compacted partition maximum bytes: 149 Compacted partition mean bytes: 104 Average live cells per slice (last five minutes): 2.263157894736842 Maximum live cells per slice (last five minutes): 3 Average tombstones per slice (last five minutes): 1.0 Maximum tombstones per slice (last five minutes): 1
  • 18.
    nodetool commands: tablestats Keyspace:jmx_adventure Read Count: 19 Read Latency: 0.3552105263157895 ms. Write Count: 3 Write Latency: 0.17766666666666667 ms. Pending Flushes: 0 Table: account_event SSTable count: 1 Space used (live): 5326 Space used (total): 5326 Space used by snapshots (total): 0 Off heap memory used (total): 51 SSTable Compression Ratio: 0.6517241379310345 Number of keys (estimate): 3 Memtable cell count: 0 Memtable data size: 0 Memtable off heap memory used: 0 Memtable switch count: 1 Local read count: 19 Local read latency: 0.386 ms Local write count: 3 Local write latency: 0.207 ms Pending flushes: 0 Bloom filter false positives: 0 Bloom filter false ratio: 0.00000 Bloom filter space used: 16 Bloom filter off heap memory used: 8 Index summary off heap memory used: 27 Compression metadata off heap memory used: 16 Compacted partition minimum bytes: 51 Compacted partition maximum bytes: 149 Compacted partition mean bytes: 104 Average live cells per slice (last five minutes): 2.263157894736842 Maximum live cells per slice (last five minutes): 3 Average tombstones per slice (last five minutes): 1.0 Maximum tombstones per slice (last five minutes): 1 tombstones per read partition size read and write latency memtable usage index summary usage
  • 19.
    nodetool commands: tablestats Keyspace:jmx_adventure Read Count: 19 Read Latency: 0.3552105263157895 ms. Write Count: 3 Write Latency: 0.17766666666666667 ms. Pending Flushes: 0 Table: account_event SSTable count: 1 Space used (live): 5326 Space used (total): 5326 Space used by snapshots (total): 0 Off heap memory used (total): 51 SSTable Compression Ratio: 0.6517241379310345 Number of keys (estimate): 3 Memtable cell count: 2 Memtable data size: 122 Memtable off heap memory used: 96 Memtable switch count: 1 Local read count: 19 Local read latency: 0.386 ms Local write count: 3 Local write latency: 0.207 ms Pending flushes: 0 Bloom filter false positives: 0 Bloom filter false ratio: 0.00000 Bloom filter space used: 16 Bloom filter off heap memory used: 8 Index summary off heap memory used: 27 Compression metadata off heap memory used: 16 Compacted partition minimum bytes: 51 Compacted partition maximum bytes: 149 Compacted partition mean bytes: 104 Average live cells per slice (last five minutes): 2.263157894736842 Maximum live cells per slice (last five minutes): 3 Average tombstones per slice (last five minutes): 1.0 Maximum tombstones per slice (last five minutes): 1 tombstones per read partition size read and write latency memtable usage index summary usage
  • 20.
    nodetool commands: tablestats Keyspace:jmx_adventure Read Count: 19 Read Latency: 0.3552105263157895 ms. Write Count: 3 Write Latency: 0.17766666666666667 ms. Pending Flushes: 0 Table: account_event SSTable count: 1 Space used (live): 5326 Space used (total): 5326 Space used by snapshots (total): 0 Off heap memory used (total): 51 SSTable Compression Ratio: 0.6517241379310345 Number of keys (estimate): 3 Memtable cell count: 0 Memtable data size: 0 Memtable off heap memory used: 0 Memtable switch count: 1 Local read count: 19 Local read latency: 0.386 ms Local write count: 3 Local write latency: 0.207 ms Pending flushes: 0 Bloom filter false positives: 0 Bloom filter false ratio: 0.00000 Bloom filter space used: 16 Bloom filter off heap memory used: 8 Index summary off heap memory used: 27 Compression metadata off heap memory used: 16 Compacted partition minimum bytes: 51 Compacted partition maximum bytes: 149 Compacted partition mean bytes: 104 Average live cells per slice (last five minutes): 2.263157894736842 Maximum live cells per slice (last five minutes): 3 Average tombstones per slice (last five minutes): 1.0 Maximum tombstones per slice (last five minutes): 1 tombstones per read partition size read and write latency memtable usage index summary usage
  • 21.
    nodetool commands: tablestats Keyspace:jmx_adventure Read Count: 19 Read Latency: 0.3552105263157895 ms. Write Count: 3 Write Latency: 0.17766666666666667 ms. Pending Flushes: 0 Table: account_event SSTable count: 1 Space used (live): 5326 Space used (total): 5326 Space used by snapshots (total): 0 Off heap memory used (total): 51 SSTable Compression Ratio: 0.6517241379310345 Number of keys (estimate): 3 Memtable cell count: 0 Memtable data size: 0 Memtable off heap memory used: 0 Memtable switch count: 1 Local read count: 19 Local read latency: 0.386 ms Local write count: 3 Local write latency: 0.207 ms Pending flushes: 0 Bloom filter false positives: 0 Bloom filter false ratio: 0.00000 Bloom filter space used: 16 Bloom filter off heap memory used: 8 Index summary off heap memory used: 27 Compression metadata off heap memory used: 16 Compacted partition minimum bytes: 51 Compacted partition maximum bytes: 149 Compacted partition mean bytes: 104 Average live cells per slice (last five minutes): 2.263157894736842 Maximum live cells per slice (last five minutes): 3 Average tombstones per slice (last five minutes): 1.0 Maximum tombstones per slice (last five minutes): 1 tombstones per read partition size read and write latency memtable usage index summary usage
  • 22.
    nodetool commands: tablestats Keyspace:jmx_adventure Read Count: 19 Read Latency: 0.3552105263157895 ms. Write Count: 3 Write Latency: 0.17766666666666667 ms. Pending Flushes: 0 Table: account_event SSTable count: 1 Space used (live): 5326 Space used (total): 5326 Space used by snapshots (total): 0 Off heap memory used (total): 51 SSTable Compression Ratio: 0.6517241379310345 Number of keys (estimate): 3 Memtable cell count: 0 Memtable data size: 0 Memtable off heap memory used: 0 Memtable switch count: 1 Local read count: 19 Local read latency: 0.386 ms Local write count: 3 Local write latency: 0.207 ms Pending flushes: 0 Bloom filter false positives: 0 Bloom filter false ratio: 0.00000 Bloom filter space used: 16 Bloom filter off heap memory used: 8 Index summary off heap memory used: 27 Compression metadata off heap memory used: 16 Compacted partition minimum bytes: 51 Compacted partition maximum bytes: 149 Compacted partition mean bytes: 104 Average live cells per slice (last five minutes): 2.263 Maximum live cells per slice (last five minutes): 3 Average tombstones per slice (last five minutes): 1.0 Maximum tombstones per slice (last five minutes): 1 tombstones per read partition size read and write latency memtable usage index summary usage
  • 23.
    nodetool commands: tablehistograms $nodetool tablehistograms jmx_adventure account_event jmx_adventure/account_event histograms Percentile SSTables Write Latency Read Latency Partition Size Cell Count (micros) (micros) (bytes) 50% 1.00 263.21 315.85 103 1 75% 2.00 315.85 545.79 149 1 95% 2.00 315.85 1131.75 149 1 98% 2.00 315.85 1131.75 149 1 99% 2.00 315.85 1131.75 149 1 Min 1.00 35.43 152.32 51 0 Max 2.00 315.85 1131.75 149 1
  • 24.
    nodetool commands: proxyhistograms $nodetool proxyhistograms proxy histograms Percentile Read Latency Write Latency Range Latency (micros) (micros) (micros) 50% 454.83 1131.75 1629.72 75% 785.94 12108.97 8409.01 95% 4866.32 12108.97 20924.30 98% 129557.75 12108.97 20924.30 99% 129557.75 12108.97 20924.30 Min 219.34 315.85 943.13 Max 129557.75 12108.97 20924.30
  • 25.
    JMX on theHTTP mx4j
  • 26.
    JMX on theHTTP mx4jbroken in 3.x
  • 27.
    JMX on theHTTP when it does work, it looks like this (2.2.7 and below):
  • 28.
    JMX on theHTTP when it does work, it looks like this: ewww gross.
  • 29.
    JMX on theHTTP (with JSON!) via Jolokia https://jolokia.org/
  • 30.
    JMX on theHTTP (with JSON!) via Jolokia https://jolokia.org/ JSON: it's ok.
  • 31.
    JMX on theHTTP (with JSON!) via Jolokia http://search.maven.org/remotecontent?filepath=org/jolokia/jolokia-jvm/1.3.4/jolokia-jvm-1.3.4- agent.jar 1. download: jolokia-jvm-1.3.4-agent.jar 2. place in: $CASSANDRA_HOME/lib 3. in cassandra-env.sh, add: JVM_OPTS="$JVM_OPTS - javaagent:$CASSANDRA_HOME/lib/jolokia- jvm-1.3.4-agent.jar"
  • 32.
    JMX on thecommand line jmxsh jmxterm http://wiki.cyclopsgroup.org/jmxterm/download.html https://github.com/davr/jmxsh
  • 33.
    jmxsh jmxterm JMX on thecommand line http://wiki.cyclopsgroup.org/jmxterm/download.html https://github.com/davr/jmxsh Almost identical!
  • 34.
    jmxsh jmxterm JMX on thecommand line http://wiki.cyclopsgroup.org/jmxterm/download.html https://github.com/davr/jmxsh Almost identical! Not updated in years!
  • 35.
    JMX on thecommand line $ java -jar $HOME/services/jmxterm-1.0-alpha-4-uber.jar -l 127.0.0.1:7200 Welcome to JMX terminal. Type "help" for available commands. $>bean org.apache.cassandra.net:type=MessagingService #bean is set to org.apache.cassandra.net:type=MessagingService $>get TimeoutsPerHost #mbean = org.apache.cassandra.net:type=MessagingService: TimeoutsPerHost = { 127.0.0.1 = 0; 127.0.0.3 = 0; };
  • 36.
    JMX on thecommand line $ java -jar $HOME/services/jmxterm-1.0-alpha-4-uber.jar -l 127.0.0.1:7200 Welcome to JMX terminal. Type "help" for available commands. $>bean org.apache.cassandra.net:type=MessagingService #bean is set to org.apache.cassandra.net:type=MessagingService $>get TimeoutsPerHost #mbean = org.apache.cassandra.net:type=MessagingService: TimeoutsPerHost = { 127.0.0.1 = 0; 127.0.0.3 = 0; };
  • 37.
    JMX on thecommand line $ java -jar $HOME/services/jmxterm-1.0-alpha-4-uber.jar -l 127.0.0.1:7200 Welcome to JMX terminal. Type "help" for available commands. $>bean org.apache.cassandra.net:type=MessagingService #bean is set to org.apache.cassandra.net:type=MessagingService $>get TimeoutsPerHost #mbean = org.apache.cassandra.net:type=MessagingService: TimeoutsPerHost = { 127.0.0.1 = 0; 127.0.0.3 = 0; };
  • 38.
    JMX on thecommand line $ java -jar $HOME/services/jmxterm-1.0-alpha-4-uber.jar -l 127.0.0.1:7200 Welcome to JMX terminal. Type "help" for available commands. $>bean org.apache.cassandra.net:type=MessagingService #bean is set to org.apache.cassandra.net:type=MessagingService $>get TimeoutsPerHost #mbean = org.apache.cassandra.net:type=MessagingService: TimeoutsPerHost = { 127.0.0.1 = 0; 127.0.0.3 = 0; };
  • 39.
    JMX on thecommand line $ echo "set -b org.apache.cassandra.db:type=CompactionManager MaximumCompactorThreads 6" | java -jar jmxterm-1.0-alpha-4-uber.jar -l 172.17.41.232:7199 && echo "set -b org.apache.cassandra.db:type=CompactionManager CoreCompactorThreads 6" | java -jar jmxterm-1.0-alpha-4-uber.jar -l 172.17.41.232:7199 setting compaction threads in one shot
  • 40.
    JMX on thecommand line $ echo "set -b org.apache.cassandra.db:type=CompactionManager MaximumCompactorThreads 6" | java -jar jmxterm-1.0-alpha-4-uber.jar -l 172.17.41.232:7199 && echo "set -b org.apache.cassandra.db:type=CompactionManager CoreCompactorThreads 6" | java -jar jmxterm-1.0-alpha-4-uber.jar -l 172.17.41.232:7199 just echo and 'pipe' setting compaction threads in one shot
  • 41.
    $>bean org.apache.cassandra.db:type=BatchlogManager #bean isset to org.apache.cassandra.db:type=BatchlogManager $>info #mbean = org.apache.cassandra.db:type=BatchlogManager #class name = org.apache.cassandra.batchlog.BatchlogManager # attributes %0 - TotalBatchesReplayed (long, r) # operations %0 - int countAllBatches() %1 - void forceBatchlogReplay() #there's no notifications $>get TotalBatchesReplayed #mbean = org.apache.cassandra.db:type=BatchlogManager: TotalBatchesReplayed = 0; $>run countAllBatches #calling operation countAllBatches of mbean org.apache.cassandra.db:type=BatchlogManager #operation returns: 0 JMX on the command line: general syntax
  • 42.
    $>bean org.apache.cassandra.db:type=BatchlogManager #bean isset to org.apache.cassandra.db:type=BatchlogManager $>info #mbean = org.apache.cassandra.db:type=BatchlogManager #class name = org.apache.cassandra.batchlog.BatchlogManager # attributes %0 - TotalBatchesReplayed (long, r) # operations %0 - int countAllBatches() %1 - void forceBatchlogReplay() #there's no notifications $>get TotalBatchesReplayed #mbean = org.apache.cassandra.db:type=BatchlogManager: TotalBatchesReplayed = 0; $>run countAllBatches #calling operation countAllBatches of mbean org.apache.cassandra.db:type=BatchlogManager #operation returns: 0 JMX on the command line: general syntax Set the bean
  • 43.
    $>bean org.apache.cassandra.db:type=BatchlogManager #bean isset to org.apache.cassandra.db:type=BatchlogManager $>info #mbean = org.apache.cassandra.db:type=BatchlogManager #class name = org.apache.cassandra.batchlog.BatchlogManager # attributes %0 - TotalBatchesReplayed (long, r) # operations %0 - int countAllBatches() %1 - void forceBatchlogReplay() #there's no notifications $>get TotalBatchesReplayed #mbean = org.apache.cassandra.db:type=BatchlogManager: TotalBatchesReplayed = 0; $>run countAllBatches #calling operation countAllBatches of mbean org.apache.cassandra.db:type=BatchlogManager #operation returns: 0 'info' for what is available JMX on the command line: general syntax
  • 44.
    $>bean org.apache.cassandra.db:type=BatchlogManager #bean isset to org.apache.cassandra.db:type=BatchlogManager $>info #mbean = org.apache.cassandra.db:type=BatchlogManager #class name = org.apache.cassandra.batchlog.BatchlogManager # attributes %0 - TotalBatchesReplayed (long, r) # operations %0 - int countAllBatches() %1 - void forceBatchlogReplay() #there's no notifications $>get TotalBatchesReplayed #mbean = org.apache.cassandra.db:type=BatchlogManager: TotalBatchesReplayed = 0; $>run countAllBatches #calling operation countAllBatches of mbean org.apache.cassandra.db:type=BatchlogManager #operation returns: 0 'get' an attribute JMX on the command line: general syntax
  • 45.
    $>bean org.apache.cassandra.db:type=BatchlogManager #bean isset to org.apache.cassandra.db:type=BatchlogManager $>info #mbean = org.apache.cassandra.db:type=BatchlogManager #class name = org.apache.cassandra.batchlog.BatchlogManager # attributes %0 - TotalBatchesReplayed (long, r) # operations %0 - int countAllBatches() %1 - void forceBatchlogReplay() #there's no notifications $>get TotalBatchesReplayed #mbean = org.apache.cassandra.db:type=BatchlogManager: TotalBatchesReplayed = 0; $>run countAllBatches #calling operation countAllBatches of mbean org.apache.cassandra.db:type=BatchlogManager #operation returns: 0 'run' an operation JMX on the command line: general syntax
  • 46.
  • 47.
    JMX on JConsole Note:Complex-type (like arrays) are grayed out
  • 48.
    JMX on JConsole Note:Complex-type (like arrays) are grayed out tool-specific invocations
  • 49.
    JMX on JConsole Gettingobject name and attributes for CLI invocation
  • 50.
    A note aboutremote connections
  • 51.
  • 52.
    cassandra-env.sh false not there bydefault using LOCAL_JMX="no"
  • 53.
    2. Visibility andexploration
  • 54.
  • 55.
  • 56.
    cluster communication Looking atsmall vs. large tasks (>64k) provides an idea of network efficiency
  • 57.
    cluster communication what doesthis node think of the others?
  • 58.
    cluster communication Click thevalue to see PhiValues for other nodes
  • 59.
    Commitlog efficiency: waitingon commit If this stays close to commitlog_size_in_mb there are bigger issues
  • 60.
  • 61.
    Commitlog efficiency: waitingon commit If this is high, you need to tune Time spent waiting for sync. This will climb if sync is lagging sync interval
  • 62.
    Commitlog efficiency: segmentallocation time spent waiting on segment allocation
  • 63.
    Tombstone impact andcompaction SSTable max local deletion time: 2147483647 Compression ratio: 0.5167785234899329 Estimated droppable tombstones: 0.3175 SSTable Level: 0 Repaired at: 0 Estimated tombstone drop times: 1471577046: 2 1471577146: 2 la-8-big-Data.db
  • 64.
    Tombstone impact andcompaction SSTable max local deletion time: 2147483647 Compression ratio: 0.5167785234899329 Estimated droppable tombstones: 0.3175 SSTable Level: 0 Repaired at: 0 Estimated tombstone drop times: 1471577046: 2 1471577146: 2 la-8-big-Data.db Go see Alain's other talk
  • 65.
    Tombstone impact andcompaction SSTable max local deletion time: 2147483647 Compression ratio: 0.5167785234899329 Estimated droppable tombstones: 0.3175 SSTable Level: 0 Repaired at: 0 Estimated tombstone drop times: 1471577046: 2 1471577146: 2 la-8-big-Data.db Go see Alain's other talk Or just read his blog: http://thelastpickle.com/blog/2016/07/27/about-deletes-and-tombstones.html
  • 66.
    droppable tombstone ratio clickingon the value provides a handy chart!
  • 67.
  • 68.
    Routing client requestsaway Set to '50.0' for routing away requests
  • 69.
    Figuring out ifDynamicSnitch is kneecapping you https://issues.apache.org/jira/browse/ CASSANDRA-6908
  • 70.
    $>get Scores #mbean =org.apache.cassandra.db:type=DynamicEndpointSnitch: Scores = { /10.10.20.31 = 2.244811534881592; /10.10.20.30 = 0.5457598567008972; /10.10.20.37 = 2.7038445472717285; /10.10.20.36 = 3.0801687240600586; /10.10.20.39 = 6.231578826904297; /10.10.20.38 = 3.1578946113586426; /10.10.20.33 = 12.003381729125977; /10.10.20.32 = 3.348876476287842; /10.10.20.35 = 1.5086206197738647; /10.10.20.34 = 4.9235992431640625; /10.20.1.46 = 2.621400833129883; /10.20.1.44 = 1.3118916749954224; /10.20.1.45 = 1.6884760856628418; /10.30.50.62 = 2.4780631123519523; /10.30.50.63 = 2.3196894221189543; /10.30.50.61 = 1.2922532529365727; /10.20.1.50 = 7.66961669921875; /10.20.1.48 = 1.686340570449829; /10.30.50.68 = 0.9854942156774241; /10.20.8.81 = 0.7212558388710022; /10.30.50.78 = 2.4760895589502847; /10.30.50.79 = 1.2857443196017568; /10.30.50.76 = 0.7804878048780488; /10.30.50.77 = 1.8351790061811122; /10.20.1.102 = 2.4894514083862305; /10.20.1.103 = 0.5889776945114136; /10.20.1.106 = 0.8040626645088196; /10.20.1.104 = 1.4327855110168457; /10.20.1.105 = 0.75789475440979; }; DynamicSnitch: ouch! most common with real fast hardware - dramatic fluctuations - 22x difference!
  • 71.
    Turn it offvia StorageService#updateSnitch: GossipingPropertyFileSnitch, false, 0, 0, 0 DynamicSnitch: disabling
  • 72.
    Turn it offvia StorageService#updateSnitch: GossipingPropertyFileSnitch, false, 0, 0, 0 DynamicSnitch: disabling false
  • 73.
    Turn it offvia StorageService#updateSnitch: GossipingPropertyFileSnitch, false, 0, 0, 0 DynamicSnitch: disabling I have no idea what will happen if you change the snitch
  • 74.
  • 75.
    CompactionManager: compaction throttling++ -MaximumCompactorThreads (change this first!!!) - CoreCompactorThreads (tip: see jmxterm example from earlier for CLI invocation)
  • 76.
    CompactionManager: compact aset of stables one or more "*-Data.db" files, comma-delimited
  • 77.
  • 78.
  • 79.
  • 80.
  • 81.
    CompactionManager: stop compaction typeof compaction, most commonly: COMPACTION, VALIDATION, ANTI_COMPACTION, INDEX_SUMMARY https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/db/compaction/OperationType.java
  • 82.
    Compaction identifier ofactive compaction CompactionManager: stop compaction
  • 83.
    Find the identifierof active compactions here (or nodetool compactionstats) CompactionManager: stop compaction
  • 84.
  • 85.
    CompactionParametersJson: Editable JSON testcompaction settings on single node!!! Update compaction strategy
  • 86.
  • 87.
  • 88.
    Delete all hintsor by host HintedHandoffManager
  • 89.
    Pause and resumedelivery by host HintedHandoffManager
  • 90.
    StorageProxy: Adjust hintdelivery settings
  • 91.
    StorageProxy: Adjust hintdelivery settings On/off switch for hints
  • 92.
    StorageProxy: Adjust hintdelivery settings How long to keep hints in milliseconds (3 hours by default)
  • 93.
    StorageProxy: Adjust hintdelivery settings How many hints to send at a time
  • 94.
  • 95.
    StorageProxy: Adjust operationtimeouts Amount of time spent in LWT Contention
  • 96.
    StorageProxy: Adjust operationtimeouts Amount of time spent in LWT Contention See Chris' LWT presentation!
  • 97.
    StorageProxy: Adjust operationtimeouts timeout for counter writes
  • 98.
    StorageProxy: Adjust operationtimeouts Timeout for range queries (poorly bounded select statements)
  • 99.
    StorageProxy: Adjust operationtimeouts Timeout for single read operation
  • 100.
    StorageProxy: Adjust operationtimeouts Timeout for other miscellaneous inter-cluster operations
  • 101.
    StorageProxy: Adjust operationtimeouts Timeout for write operations
  • 102.
    StorageProxy: Adjust operationtimeouts special shoutout to TruncateRpcTimeout cqlsh> truncate table account; TimeoutException...
  • 103.
    StorageProxy: Adjust operationtimeouts If you truncate, increase this by an order of magnitude
  • 104.
    Index sampling andsummary SSTable
  • 105.
  • 106.
    Index summaries Invoke occasionallyon a single node Warning: expensive operation!
  • 107.
    Index summaries watch themetrics for tuning
  • 108.
    Index summaries Table: account_event ... Indexsummary off heap memory used: 27732 ... combine with nodetool tablestats, then adjust: min_index_interval, max_index_interval
  • 109.
  • 110.
    StorageService: batch sizethresholds Size of the largest batch in KB
  • 111.
    StorageService: tombstone thresholds Adjustfailure and warning to remove log messages
  • 112.
    StorageService: tombstone thresholds Adjustfailure and warning to remove log messages If you adjust these, you have bigger problems
  • 113.
    force stop repair Doesjust what it says.
  • 114.
    bulk load (syncand async) File path as arguments: /some/path/${keyspace}/${table}/
  • 115.
    Recover commit logsegments Helpful for reloading commitlogs from a backup
  • 116.
    4. Access Control:locking down JMX
  • 117.
    nmap -Pn -p7199 -oG logs/cass.gnmap 54.88.0.0/14 Haha! You exposed a command plane to the internet
  • 118.
    nmap -Pn -p7199 -oG logs/cass.gnmap 54.88.0.0/14 Haha! You exposed a command plane to the internet Shameless Plug!
  • 119.
    Why do Ineed to secure JMX?
  • 120.
    Why do Ineed to secure JMX? StorageService.truncate("your_keyspace", "your_table")
  • 121.
    Why do Ineed to secure JMX? StorageService.truncate("your_keyspace", "your_table")
  • 122.
    Why do Ineed to secure JMX? StorageService.truncate("your_keyspace", "your_table") Now with internet access!!
  • 123.
    Why do Ineed to secure JMX? StorageService.truncate("your_keyspace", "your_table") Now with internet access!! Works as Advertised!
  • 124.
    Why do Ineed to secure JMX? StorageService.removeNode("coworker_IPADR")
  • 125.
    Why do Ineed to secure JMX? StorageService.removeNode("coworker_IPADR") Fun for the whole development team!
  • 126.
    Securing JMX SSL setupis like node to node and client to server http://docs.oracle.com/javase/8/docs/technotes/guides/management/agent.html
  • 127.
    Securing JMX JMX Authenticationis straightforward and well documented $JAVA_HOME/jre/lib/management/jmxremote.access $JAVA_HOME/jre/lib/management/ jmxremote.password.template http://docs.oracle.com/javase/8/docs/technotes/guides/management/agent.html
  • 128.
    Securing JMX $JAVA_HOME/jre/lib/management/jmxremote.access $JAVA_HOME/jre/lib/management/ jmxremote.password.template Now youcan: nodetool -u admin -pw secret compactionstats http://docs.oracle.com/javase/8/docs/technotes/guides/management/agent.html JMX Authentication is straightforward and well documented
  • 129.
    Securing JMX $JAVA_HOME/jre/lib/management/jmxremote.access $JAVA_HOME/jre/lib/management/ jmxremote.password.template Now youcan: nodetool -u admin -pw secret compactionstats Tip: -pwf option will read the password from a file http://docs.oracle.com/javase/8/docs/technotes/guides/management/agent.html JMX Authentication is straightforward and well documented
  • 130.
    Securing JMX $JAVA_HOME/jre/lib/management/jmxremote.access $JAVA_HOME/jre/lib/management/ jmxremote.password.template Now youcan: nodetool -u admin -pw secret compactionstats JMX Authentication is straightforward and well documented RBAC for JMX Authentication and Authorization is now supported https://issues.apache.org/jira/browse/CASSANDRA-10091
  • 131.