Cassandra at rest encryption, inter-node communication encryption, client-server communication encryption, authentication, authorization, and securing JMX management were discussed. The document provided guidance on implementing encryption at rest using commercial and open source options, setting up SSL for inter-node and client-server communication using self-signed certificates, implementing authentication and authorization best practices from RBMS, and securing JMX access.
1. CASSANDRA DAY ATLANTA 2015
HARDENING CASSANDRA
FOR COMPLIANCE
(OR PARANOIA)
Nate McCall
@zznate
#CassandraSummit
Co-Founder & Sr.Technical Consultant
Licensed under a Creative Commons Attribution-NonCommercial 3.0 New Zealand License
2. AboutThe Last Pickle.
Work with clients to deliver and improve
Apache Cassandra based solutions.
Based in New Zealand,Australia & USA.
9. Overview:
By necessity we abstract and encapsulate, making
it easier to have un-intended side effects
knife ssh -x ${knifeUser}
-a hostname
-E $chefEnv "role:${role}" "${executeCommand}"
Then eventually:
rm -rf /var/lib/cassandra/*
12. Overview: System Hardening
Limit ļ¬lesystem access.
Only C* needs
/var/lib/cassandra
(conļ¬guration management does not need access
after initialization)
14. Overview: System Hardening: Limit Network Access
C* only needs inbound on:
- 9042/9160 (clients)
- 7000/7001 (cluster)
- 7199 (JMX)
15. Encryption at Rest
Inter-node Communication
Client-Server Communication
Authentication and Authorization
Management andTooling
16. Encryption At Rest:
Why it's a good idea:
- Do you know how your cloud provider is actually
using storage under the hood?
- What does IT do with decommissioned disks? (Are
you sure?)
18. Encryption At Rest:
It's a distributed system.
Make sure you understand the HA Story
in context of your needs.
Otherwise you introduced an SOPF!
eg. "EBS snapshots of the key server"
Is this going to work for you?
20. Encryption At Rest: Open Source
dmCrypt (block)
eCrypt-FS (ļ¬le)
Major vendors
probably support both
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/5/html/Installation_Guide/ch29s02.html
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Storage_Administration_Guide/ch-efs.html
22. Encryption At Rest: Commercial
DataStax Enterprise*:
CREATETABLE users
...
WITH compression_parameters:sstable_compression = 'Encryptor'
and compression_parameters:cipher_algorithm = 'AES/ECB/PKCS5Padding'
and compression_parameters:secret_key_strength = 128;
*commit log not included! (eCrypt-fs to the rescue)
23. Encryption At Rest: Commercial-ish
EBS Encryption!
(a.k.a. "not my problem")
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSEncryption.html
24. Encryption at Rest
Inter-Node Communication
Client-Server Communication
Authentication and Authorization
Management andTooling
31. Inter-Node Communication: SSL: Certiļ¬cates Done Right
BYO Certiļ¬cate Authority
A CA in this case provides the root certiļ¬cate which can be used
to validate a node's identity without needing direct knowledge of
that node.
32. Inter-Node Communication: SSL: Certiļ¬cates Done Right
Don't do this:
http://docs.datastax.com/en/cassandra/2.1/cassandra/security/secureSSLCertiļ¬cates_t.html <- BAD!
LOL!
33. Inter-Node Communication: SSL: Certiļ¬cates Done Right
What we should do:
1. create the CA with a "root" certiļ¬cate
2. create certiļ¬cates for each node
3. export each node's certiļ¬cates to be signed by CA
4. sign each nodes certiļ¬cate with the CA
5. add the CA root public certiļ¬cate into each node's key store
5. add the signed result back into that node's key store
6. import CA root public certiļ¬cate into each node's trust store (or build
one and copy it around)
34. Inter-Node Communication: SSL: Certiļ¬cates Done Right
Be your own CA
openssl req
-config gen_ca_cert.conf
-new -x509
-keyout ca-key
-out ca-cert
-days 365
35. Inter-Node Communication: SSL: Certiļ¬cates Done Right
'gen_ca_cert.conf'
[ req ]
distinguished_name = req_distinguished_name
prompt = no
output_password = mypass
default_bits = 2048
[ req_distinguished_name ]
C = US
ST = TX
L = Austin
O = TLP
OU = TestCluster
CN = TestClusterMasterCA
emailAddress = info@thelastpickl.com
36. Inter-Node Communication: SSL: Certiļ¬cates Done Right
What we just did
- `req`:An OpenSSL sub-command saying that we are (in this case) creating a PKCS#10 X.509 Certiļ¬cate (see
https://www.openssl.org/docs/manmaster/apps/req.html for full details)
The following parameters are all options of "-req"
`-conļ¬g`: The path to the conļ¬g ļ¬le (avoids having to provide information on STDIN)
`-new`: This is a new signing request we are making
`-x509`: The output will be a self-signed certiļ¬cate we can use as a root CA
`-keyout`: The ļ¬lename to which we will write our key
`-out`: The ļ¬lename to which we will write our certiļ¬cate
`-days`: The number of days for which the generated certiļ¬cate will be valid
38. Inter-Node Communication: SSL: Certiļ¬cates Done Right
What we just did
`-genkeypair`: the keytool command to generate a public/private key pair combination
`-keyalg`:The algorithm to use, RSA in this case*
`-alias`:An alias to use for this public/private key pair. It needs to identify the public key when imported into a
key store. I usually use some form of $hostname-cassandra
`-keystore`:The location of our key store (created if it does not already exist)
`-storepass`:The password for the key store
`-keypass`:The password for the key
`-validity`:The number of days for which this key pair will be valid
`-keysize`:The size of the key to generate
https://docs.oracle.com/javase/8/docs/technotes/tools/unix/keytool.html
39. Inter-Node Communication: SSL: Certiļ¬cates Done Right
A note about "-dname":
CN=node1
Common Name: Best practice is to use hostname
OU=SSL-verification-cluster
Organizational Unit: I like to use the environment speciļ¬c cluster name
O=TheLastPickle
Organization
C=US"
Country
Note: Hostname veriļ¬cation can be done againstĀ subjectAlternativeName
Add "-extĀ SAN=DNS:thelastpickle.com" to keytool invocation if desired.
40. Inter-Node Communication: SSL: Certiļ¬cates Done Right
Exporting certiļ¬cates
for signing by the CA
keytool
-keystore node1-server-keystore.jks
-alias node1
-certreq
-file node1_cert_sr
-keypass awesomekeypass
-storepass awesomekeypass
41. Inter-Node Communication: SSL: Certiļ¬cates Done Right
Sign each node's
certiļ¬cate with the CA
openssl x509
-req
-CA ca-cert
-CAkey ca-key
-in node1_cert_sr
-out node1_cert_signed
-days 365
-CAcreateserial
-passin pass:mypass
42. Inter-Node Communication: SSL: Certiļ¬cates Done Right
What did we just do:
- `x509` Use the display and signing subcommand (https://www.openssl.org/docs/manmaster/apps/
x509.html)
The following parameters are options of "x509"
- `-req` We are signing a certiļ¬cate request as opposed to a certiļ¬cate
- `-CA` The CA cert ļ¬le created above
- `-CAKey` The CA key ļ¬le created above
- `-in` The certiļ¬cate request we are signing
- `-out` The newly-signed certiļ¬cate ļ¬le to create
- `-days` The number of days for which the signed certiļ¬cate will be valid
- `-CAcreateserial` Create a serialnumber for this CSR
- `-passin` The keypassword source.The arguments to 'passin' have their own formatting instructions.
See 'Pass Phrase Arguments' on https://www.openssl.org/docs/manmaster/apps/openssl.html
43. Inter-Node Communication: SSL: Certiļ¬cates Done Right
Add the CA's public key to the
node's key stores
keytool
-keystore node1-server-keystore.jks
-alias CARoot
-import
-file ca-cert
-noprompt
-keypass awesomekeypass
-storepass awesomekeypass
44. Inter-Node Communication: SSL: Certiļ¬cates Done Right
Import that node's signed certiļ¬cate
back into it's trust store
keytool
-keystore node1-server-keystore.jks
-alias node1
-import
-file node1_cert_signed
-keypass awesomekeypass
-storepass awesomekeypass
45. Inter-Node Communication: SSL: Certiļ¬cates Done Right
Build a trust store
from the CA's public certiļ¬cate
(this can be shared among all nodes)
keytool
-keystore generic-server-truststore.jks
-alias CARoot -importcert
-file ca-cert
-keypass mypass -storepass
truststorepass -noprompt
48. Inter-Node Communication: SSL Done Right
server_encryption_options:
internode_encryption: all
keystore: conf/server-keystore.jks
keystore_password: awesomekeypass
truststore: conf/server-truststore.jks
truststore_password: truststorepass
protocol: TLS
algorithm: SunX509
store_type: JKS
cipher_suites: [TLS_RSA_WITH_AES_256_CBC_SHA]
require_client_auth: true
If you don't set this to
true, you might as well
turn encryption off
49. Inter-Node Communication: SSL Done Right
Also: Key store password and key password
*must be the same*
server_encryption_options:
...
keystore_password: awesomekeypass
require_client_auth: true
...
keytool ... -keypass awesomekeypass ...
50. Inter-Node Communication: SSL Done Right
Export restrictions?
Use128 bit keys
cipher_suites: [TLS_RSA_WITH_AES_128_CBC_SHA]
http://www.oracle.com/technetwork/java/javase/downloads/jce8-download-2133166.html
http://docs.oracle.com/javase/8/docs/technotes/guides/security/SunProviders.html#importlimits
51. Inter-Node Communication: SSL: Certiļ¬cates Done Right
Trouble?
-Djavax.net.debug=ssl
http://docs.oracle.com/javase/7/docs/technotes/guides/security/jsse/ReadDebug.html
52. Inter-Node Communication: SSL: Certiļ¬cates Done Right
This Just In!
Netļ¬ix Lemur:
"x.509 Certiļ¬cate Orchestration
Framework"
http://techblog.netļ¬ix.com/2015/09/introducing-lemur.html
https://github.com/Netļ¬ix/lemur
60. Client-Server Communication
client_encryption_options:
enabled: false
keystore: conf/client-keystore.jks
keystore_password: clientkeypass
truststore: conf/client-truststore.jks
truststore_password: clienttrustpass
protocol: TLS
algorithm: SunX509
store_type: JKS
cipher_suites: [TLS_RSA_WITH_AES_256_CBC_SHA]
require_client_auth: true
If you don't set this to
true, you might as well
turn encryption off
61. Encryption at Rest
Inter-Node Communication
Client-Server Communication
Authentication and Authorization
Management andTooling
69. Authentication and Authorization:Authorization
These tables have
default read permission
for every authenticated user:
system.schema_keyspace
system.schema_columns
system.schema_columnfamilies
system.local
system.peers
73. Management andTooling: Securing JMX
Options as of 2.1.8
will look familiar:
JVM_OPTS="$JVM_OPTS -Djavax.net.ssl.keyStore=/path/to/keystore"
JVM_OPTS="$JVM_OPTS -Djavax.net.ssl.keyStorePassword=<keystore-password>"
JVM_OPTS="$JVM_OPTS -Djavax.net.ssl.trustStore=/path/to/truststore"
JVM_OPTS="$JVM_OPTS -Djavax.net.ssl.trustStorePassword=<truststore-password>"
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.ssl.need.client.auth=true"
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.registry.ssl=true"
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.ssl.enabled.protocols=<enabled-protocols>"
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.ssl.enabled.cipher.suites=<enabled-cipher-
suites>"
74. Management andTooling: Securing JMX
Thus:
as of 2.1.8, nodetool can use SSL*
* https://issues.apache.org/jira/browse/CASSANDRA-9090
75. Management andTooling: Securing JMX
Conļ¬gure authentication
for JMX*
Now you can do:
nodetool status -u monitor -pw monitorpass
* http://docs.datastax.com/en/cassandra/2.1/cassandra/security/secureJmxAuthentication.html
76. Management andTooling: Securing JMX
JMX user/password and role
conļ¬guration is ļ¬exible
For details, take a look at the defaults:
$JAVA_HOME/jre/lib/management/jmxremote.access
$JAVA_HOME/jre/lib/management/jmxremote.password.template