This document summarizes some lesser known features in Apache Cassandra 2.1, including: 1) Cassandra's logging was changed to use Logback, allowing for faster and more configurable logging through a logback.xml file. 2) New default paths were added in Cassandra 2.1 for data, commit logs, and configurations to keep directories cleaner. 3) A number of command line parameters and YAML configuration options were added for more control over logging levels, commit log handling, compaction settings, and more. 4) Enhancements were made to the CQL shell cqlsh and nodetool for additional debugging and management capabilities.

1. #CASSANDRASUMMIT
LESSER KNOWN FEATURES
OF CASSANDRA 2.1
Aaron Morton
@aaronmorton
Co-Founder & Principal Consultant
Licensed under a Creative Commons Attribution-NonCommercial 3.0 New Zealand License

2. About The Last Pickle.
!
Work with clients to deliver and improve Apache Cassandra
based solutions.
Apache Cassandra Committer, DataStax MVP, Hector
Maintainer, Apache Usergrid Committer.
Based in New Zealand & USA.

3. Configuration
CQL3
Tools
Internals

4. Change Logging to Logback (CASSANDRA-5883)
Faster
Auto Reloading
Auto Archiving
logback.xml

5. Change Logging to Logback (CASSANDRA-5883)
<appender name="FILE"
class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>${cassandra.logdir}/system.log</file>
<rollingPolicy
class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy">
<fileNamePattern>${cassandra.logdir}/system.log.%i.zip</
fileNamePattern>
<minIndex>1</minIndex>
<maxIndex>20</maxIndex>
</rollingPolicy>
</appender>
!
<logger name=“org.apache.cassandra.Foo” level="ERROR"/>

6. Change Logging to Logback (CASSANDRA-5883)
TIP: Change compression so
zless and zgrep work.
!
${cassandra.logdir}/system.log.%i.gz

7. Change Logging to Logback (CASSANDRA-5883)
$ ls -lah logs/
963K system.log
129K system.log.1.gz
158K system.log.2.gz
158K system.log.3.gz
158K system.log.4.gz
158K system.log.5.gz
158K system.log.6.gz
158K system.log.7.gz
158K system.log.8.gz
157K system.log.9.gz

8. Command Line Parameters

9. New Default Paths (CASSANDRA-7136)
// bin/cassandra.in.sh
CASSANDRA_HOME="`dirname "$0"`/.."
cassandra_storagedir=“$CASSANDRA_HOME/data"
!
// bin/cassandra
-Dcassandra.logdir=$CASSANDRA_HOME/logs"
-Dcassandra.storagedir=$cassandra_storagedir"

10. New Default Paths (CASSANDRA-7136)
No change for package installs.
!
Leave YAML settings blank for
new defaults.

11. New Default Paths (CASSANDRA-7136)
$ ls -lh cassandra/v2.1/
-rw-r--r--@ CHANGES.txt
-rw-r--r--@ LICENSE.txt
-rw-r--r--@ NEWS.txt
-rw-r--r--@ NOTICE.txt
drwxr-xr-x@ bin
drwxr-xr-x@ conf
drwxr-xr-x data
drwxr-xr-x@ lib
drwxr-xr-x logs
drwxr-xr-x@ pylib
drwxr-xr-x@ tools

12. New Default Paths (CASSANDRA-7136)
$ ls -lah cassandra/
drwxr-xr-x@ apache-cassandra-1.0.12
drwxr-xr-x@ apache-cassandra-1.1.12
drwxr-xr-x@ apache-cassandra-1.2.18
drwxr-xr-x@ apache-cassandra-2.0.10
drwxr-xr-x@ apache-cassandra-2.1.0-rc7
lrwxr-xr-x current -> v2.0/
lrwxr-xr-x v1.0 -> apache-cassandra-1.0.12
lrwxr-xr-x v1.1 -> apache-cassandra-1.1.12
lrwxr-xr-x v1.2 -> apache-cassandra-1.2.18/
lrwxr-xr-x v2.0 -> apache-cassandra-2.0.10
lrwxr-xr-x v2.1 -> apache-cassandra-2.1.0-rc7

13. Custom Configuration Loader (CASSANDRA-5045)
-Dcassandra.config.loader
!
Provide a custom class that
implements
o.a.c.config.ConfigurationLoader.

14. Custom Configuration Loader (CASSANDRA-5045)
Useful when embedding or storing
configuration elsewhere.

15. Add QueryHandler Interface (CASSANDRA-6659)
-Dcassandra.custom_query_handler_class
!
Provide a custom class that
implements
o.a.c.cql3.QueryHandler.

16. Add QueryHandler Interface (CASSANDRA-6659)
Custom implementation to handle
all CQL3 queries.
!
For example logging, slow query log,
stubbing Cassandra.

17. Add QueryHandler Interface (CASSANDRA-6659)
public interface QueryHandler
{
ResultMessage process(String query, QueryState state,…);
!
ResultMessage.Prepared prepare(String query, QueryState state);
!
ParsedStatement.Prepared getPrepared(MD5Digest id);
!
CQLStatement getPreparedForThrift(Integer id);
!
ResultMessage processPrepared(CQLStatement statement, …);
!
ResultMessage processBatch(BatchStatement statement, …);

18. Pluggable o.a.c.metrics Reporters (CASSANDRA-4430)
!
-Dcassandra.metricsReporterConfigFile
!
https://github.com/addthis/metrics-reporter-
config

19. Pluggable o.a.c.metrics Reporters (CASSANDRA-4430)
Reporters push metrics to
Graphite, Ganglia, Riemann
etc.

20. Pluggable o.a.c.metrics Reporters (CASSANDRA-4430)
riemann:
-
period: 1
timeunit: 'SECONDS'
hosts:
- host: 'localhost'
port: 5555
predicate:
color: "white"
useQualifiedName: true
patterns:
- "^org.apache.cassandra.metrics.ClientRequest.+"
- "^org.apache.cassandra.metrics.DroppedMessage.+"

21. Selective Commit Log Replay (CASSANDRA-4809)
-Dcassandra.replayList
!
-Dcassandra.replayList=Foo.Bar,…

22. Selective Commit Log Replay (CASSANDRA-4809)
Use for selective Point in Time
recovery using Commit Log
Archiving.

23. Add Option To Disable STCS In L0 (CASSANDRA-6621)
!
-Dcassandra.disable_stcs_in_l0
!
May be useful when
Bootstrapping.

24. Use JMX_PORT For The RMI Port (CASSANDRA-7087)
!
// conf/cassandra-env.sh
-Dcom.sun.management.jmxremote.rmi.port=$JMX_PORT

25. YAML Configuration

26. Enabled Hints Per Data Center (CASSANDRA-6157)
!
hinted_handoff_enabled: DC1,DC2
!
Disable Hints for a troubled
DC.
!
!

27. Add Paranoid Disk Failure Option (CASSANDRA-6646)
!
disk_failure_policy: stop_paranoid
!
Implement STOP behaviour
for CorruptSSTableException.

28. Commit Log Failure Handling (CASSANDRA-6364)
!
commit_failure_policy: stop
!

29. Commit Log Failure Handling (CASSANDRA-6364)
!
ERROR [PERIODIC-COMMIT-LOG-SYNCER] 2014-… CommitLog.java:350 -
Failed to persist commits to disk. Commit disk failure policy is
stop; terminating thread
!
java.io.IOException: Something broke.
at org.apache.cassandra.db.commitlog.AbstractCommitLogService
$1.run(AbstractCommitLogService.java:123)
at java.lang.Thread.run(Thread.java:744)

30. Add file_cache_size_in_mb setting (CASSANDRA-5661)
!
# Total memory to use for sstable-reading buffers.
# Defaults to
# the smaller of 1/4 of heap or 512MB
# file_cache_size_in_mb: 512

31. Add file_cache_size_in_mb setting (CASSANDRA-5661)
File Readers have a 64 KB
buffer.
Compressed File Readers have
two 64 KB buffers.

32. Add file_cache_size_in_mb setting (CASSANDRA-5661)
Discard cached file readers
512ms after last access for a
file.

33. Add file_cache_size_in_mb setting (CASSANDRA-5661)
Maintain a maximum
combined buffer size for all file
readers.

34. Improve 2.1 flush defaults (CASSANDRA-7551)
!
# memtable_flush_writers defaults to the smaller of (number of
# disks, number of cores), with a minimum of 2 and a maximum of 8.
#
# If your data directories are backed by SSD, you should increase
# this to the number of cores.
# memtable_flush_writers: 8

35. New concurrent_compactors Defaults (CASSANDRA-7139)
!
# concurrent_compactors defaults to the smaller of (number
# of disks, number of cores), with a minimum of 2 and a
# maximum of 8.
#
# If your data directories are backed by SSD, you should
# increase this to the number of cores.
# concurrent_compactors: 1

36. Throw Error When Frame Too Large (CASSANDRA-5981)
!
# native_transport_max_frame_size_in_mb: 256
!
Improves Existing Behaviour,
InvalidRequestException now
thrown.

37. Add broadcast_rpc_address Option (CASSANDRA-5899)
!
broadcast_rpc_address: 1.2.3.4
!
Broadcast to clients via node
discovery.

38. Add listen_interface and rpc_interface (CASSANDRA-7417)
!
rpc_interface: eth1
!
listen_interface: eth0

39. Add listen_interface and rpc_interface (CASSANDRA-7417)
Must have a only one IP
Address.

40. Drop queries reading many Tombstones (CASSANDRA-6117)
Replaces
tombstone_debug_threshold

41. Drop queries reading many Tombstones (CASSANDRA-6117)
!
tombstone_warn_threshold: 1000

42. Drop queries reading many Tombstones (CASSANDRA-6117)
WARN [main] …
!
SliceQueryFilter.java:236 - Read 2 live and 4
tombstoned cells in system.schema_columnfamilies (see
tombstone_warn_threshold). 2147483631 columns was
requested, slices=[-],
delInfo={deletedAt=-9223372036854775808,
localDeletion=2147483647}

43. Drop query that using Tombstones (CASSANDRA-6117)
!
tombstone_failure_threshold: 100000

44. Drop query that using Tombstones (CASSANDRA-6117)
!
ERROR [SharedPool-Worker-7] …
!
SliceQueryFilter.java:212 - Scanned
over 100000 tombstones in
system.schema_columns; query aborted
(see tombstone_fail_threshold)

45. Log a Warning for Large Batches (CASSANDRA-6487)
!
batch_size_warn_threshold_in_kb: 5

46. Log a Warning for Large Batches (CASSANDRA-6487)
!
WARN [Native-Transport-Requests:553]…
!
BatchStatement.java (line 228) Batch of prepared
statements for
[Keyspace1.Standard1] is of size 6165,
exceeding specified threshold of 5120 by 1045.

47. Preemptive opening compaction result (CASSANDRA-6916)
Replaces
compaction_preheat_key_cache: true,
preheat_kernel_page_cache: false
and
populate_io_cache_on_flush
table property.

48. Preemptive opening compaction result (CASSANDRA-6916)
With
sstable_preemptive_open_interval_in_mb: 50

49. Preemptive opening compaction result (CASSANDRA-6916)
Pre heats Key Cache and IO
Cache, and reads from partial
SSTables.

50. Throttle Inter-DC Streaming Traffic (CASSANDRA-6596)
!
inter_dc_stream_throughput_outbound_megabits_per_sec: 0
Applied after
stream_throughput_outbound_megabits_per_sec: 200

51. Configuration
CQL3
Tools
Internals

52. Alias Support For SELECT Statement (CASSANDRA-5075)
!
cqlsh:dev> select foo as total_foo from foo;
!
total_foo
-----------
bar
baz
!
(2 rows)

53. An Official Way To Disable Compaction (CASSANDRA-5074)
!
create table no_compaction_for_you (
foo text,
bar text,
primary key (foo, bar)
) with
compaction =
{
'enabled' : false,
'class' : 'SizeTieredCompactionStrategy'
};

54. Aggressive Tombstone Compactions (CASSANDRA-6563)
!
CREATE TABLE foo (
foo text primary key,
bar text
) WITH
compaction =
{
'class' : 'SizeTieredCompactionStrategy',
'unchecked_tombstone_compaction' : 'true'
};

55. Aggressive Tombstone Compactions (CASSANDRA-6563)
Disables check for other row
fragments.

56. Configuration
CQL3
Tools
Internals

57. cqlsh

58. Pass cqlsh Statements via Command Line CASSANDRA-7172
!
bin/cqlsh --execute="select distinct foo from dev.foo;"
!
foo
-------
three
one
five
two
four
!
(5 rows)

59. Include CQL in Error Messages (CASSANDRA-7111)
!
cqlsh:dev> SELECT foo bar from foo;
!
<ErrorMessage code=2000 [Syntax error in CQL query]
message="line 1:11 no viable alternative at input 'bar'">
cqlsh:dev>
!

60. nodetool

61. Set & Get Logging Levels via nodetool (CASSANDRA-7090)
!
nodetool getlogginglevels
Logger Name Log Level
ROOT INFO
com.thinkaurelius.thrift ERROR
!
nodetool setlogginglevel org.apache.cassandra.db.Row DEBUG
!
nodetool getlogginglevels
Logger Name Log Level
ROOT INFO
com.thinkaurelius.thrift ERROR
org.apache.cassandra.db.Row DEBUG

62. Add listsnapshots To nodetool (CASSANDRA-5742)
!
nodetool listsnapshots
!
Snapshot Details:
Snapshot name Keyspace name Column family True size Size on disk
pre-migration Keyspace1 Standard1 508.03 MB 555.59 MB
all-good Keyspace1 Standard1 549.01 MB 803.68 MB
!
Total TrueDiskSpaceUsed: 1.03 GB

63. Add Snapshot Space Used To cfstats (CASSANDRA-6231)
!
Table: Standard1
SSTable count: 4
Space used (live), bytes: 76875505
Space used (total), bytes: 76875505
Space used by snapshots (total), bytes: 1108391939

64. Save Compaction History (CASSANDRA-5078)
!
nodetool compactionhistory
!
id keyspace columnfamily compacted_at bytes_in bytes_out
31...9d Keyspace1 Standard1 1410287780455 488451903 174360500
51...9d Keyspace1 Standard1 1410287834272 527847260 256812363

65. Repairing Between Specific Replicas (CASSANDRA-6440)
!
nodetool repair
[(-dc <specific_dc> | --in-dc <specific_dc>)...]
[(-hosts <specific_host> | --in-hosts <specific_host>)]

66. Reset SSTable LCS Compaction Level (CASSANDRA-5271)
!
sstablelevelreset --really-reset Keyspace1 Standard1
!
!
Changing level from 1 to 0 on …/Keyspace1-Standard1-ka-35-Data.db
!
Skipped .../Keyspace1-Standard1-ka-33-Data.db since it is already
on level 0
!
Changing level from 1 to 0 on …/Keyspace1-Standard1-ka-32-Data.db
!
Skipped .../Keyspace1-Standard1-ka-34-Data.db since it is already
on level 0
!

67. Configuration
Schema
Tools
Internals

68. Promote Partition Index (CASSANDRA-2319)
Index of Cells created when
partition is larger than
column_index_size_in_kb
(64).

69. Promote Partition Index (CASSANDRA-2319)
Column Index promoted into
the Key Cache.
Improves reading from wide
Partitions.

70. Remove Keyspace.switchLock (CASSANDRA-5549)
Removed
memtable_flush_queue_size
Added
memtable_cleanup_threshold

71. Remove Keyspace.switchLock (CASSANDRA-5549)
When the Flush Queue was
full Mutations would be
blocked.
(Blocked FlushWriter tasks.)

72. Unique Column Family Directories (CASSANDRA-5202)
Prevents “re-using”data from
dropped Column Families.

73. Unique Column Family Directories (CASSANDRA-5202)
ls -lah data/data/Keyspace1/
!
drwxr-xr-x Counter1-89df6990386511e48bb41d4c6fbb569d
drwxr-xr-x Counter3-89f20730386511e48bb41d4c6fbb569d
drwxr-xr-x Standard1-89df4280386511e48bb41d4c6fbb569d

74. Specify Cells To Cache Per Row (CASSANDRA-5357)
create table row_cache (
foo text,
bar text,
primary key (foo, bar)
) with
rows_per_partition_to_cache = 10;

75. Use hibernate when join_ring is false (CASSANDRA-6961)
Use when returning a node
that as been down for longer
than the Hint window.

76. Use hibernate when join_ring is false (CASSANDRA-6961)
nodetool join

77. Add Manifest Describing Snapshot Files (CASSANDRA-6326)
!
{
"files": [
"Keyspace1-Standard1-ka-1-Data.db",
"Keyspace1-Standard1-ka-3-Data.db",
"Keyspace1-Standard1-ka-2-Data.db",
"Keyspace1-Standard1-ka-5-Data.db",
"Keyspace1-Standard1-ka-4-Data.db"
]
}

78. Track min/max Clustered Values (CASSANDRA-5514)
!
CREATE TABLE temperature
(
weatherstation_id text,
event_time timestamp,
temperature text,
PRIMARY KEY (weatherstation_id,event_time)
);

79. Track min/max Clustered Values (CASSANDRA-5514)
!
SELECT
event_time, temperature
FROM
temperature
WHERE
weatherstation_id='1234ABCD'
AND
event_time >= '2013-04-03 07:01:00'
AND
event_time <= '2013-04-03 07:04:00';

80. Track min/max Clustered Values (CASSANDRA-5514)
Skips SSTables that will not
contain relevant data.

81. The Take Away
Use Cassandra v2.0.10
!
Get testing on v2.1.0

82. Thanks.
#CASSANDRASUMMIT
!

83. Aaron Morton
@aaronmorton
!
Co-Founder & Principal Consultant
www.thelastpickle.com
!
Licensed under a Creative Commons Attribution-NonCommercial 3.0 New Zealand License