Seattle C* Meetup: Hardening cassandra for compliance or paranoia

HARDENING CASSANDRA
FOR COMPLIANCE
(OR PARANOIA)
Nate McCall
@zznate
Seattle Cassandra Users
Co-Founder & Sr.Technical Consultant
Licensed under a Creative Commons Attribution-NonCommercial 3.0 New Zealand License

AboutThe Last Pickle.
Work with clients to deliver and improve
Apache Cassandra based solutions.
Based in New Zealand,Australia & USA.

Encryption at Rest
Inter-node Communication
Client-Server Communication
Authentication and Authorization
Management andTooling

Overview:
Security is usually
necessitated by
Industry Guidelines
(depending on organization size)

Overview: Industry Guidelines
- Financial: PCI-DSS
- Healthcare: HIPAA
- Defense/Govt: FIPS
- TelCo: CPNI

Overview: Industry Guidelines
Their heart is in the right place...
(Do you really need node to node SSL if it never leaves a backplane?)

Overview:
However:
good security practices
can save you

Overview:
By necessity we abstract and encapsulate, making
it easier to have un-intended side effects
knife ssh -x ${knifeUser}
-a hostname
-E $chefEnv "role:${role}" "${executeCommand}"
Then eventually:
rm -rf /var/lib/cassandra/*

Overview:
Basic systems hardening
can also protect from
common operational errors

Overview: System Hardening
Limit account access
"sudo su -" = trouble

Limit ﬁlesystem access.
Only C* needs
/var/lib/cassandra
(conﬁguration management does not need access
after initialization)

Networking
- Limit network access
- network segmentation and ACLs
- iptables, etc

Overview: System Hardening: Limit Network Access
C* only needs inbound on:
- 9042/9160 (clients)
- 7000/7001 (cluster)
- 7199 (JMX)

Encryption At Rest:
Why it's a good idea:
- Do you know how your cloud provider is actually
using storage under the hood?
- What does IT do with decommissioned disks? (Are
you sure?)

Encryption At Rest:
There are good
open source and
commercial options
for at-rest encryption.
But...

Encryption At Rest:
It's a distributed system.
Make sure you understand the HA Story
in context of your needs.
Otherwise you introduced an SPOF!
eg. "EBS snapshots of the key server"
Is this going to work for you?

Encryption At Rest:
File level
vs.
Block level

Encryption At Rest: Open Source
dmCrypt (block)
eCrypt-FS (ﬁle)
Major vendors
probably support both
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/5/html/Installation_Guide/ch29s02.html
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Storage_Administration_Guide/ch-efs.html

Encryption At Rest: Commercial
- Cloudera Gazzang!
-Vormetric

Encryption At Rest: Commercial
DataStax Enterprise*:
CREATETABLE users
...
WITH compression_parameters:sstable_compression = 'Encryptor'
and compression_parameters:cipher_algorithm = 'AES/ECB/PKCS5Padding'
and compression_parameters:secret_key_strength = 128;
*commit log not included! (eCrypt-fs to the rescue)

Encryption At Rest: Commercial-ish
EBS Encryption!
(a.k.a. "Not my problem.")
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSEncryption.html

Encryption at Rest
Inter-Node Communication
Management andTooling

Inter-Node Communication: SSL
Inter-Node encryption can be rolled
out on production
with no downtime*
* Depending on client consistency levels

Why is it a good idea?

Inter-Node Communication: SSL:Why It's a Good Idea:
Every operation
is a
Message
on the wire

It takes one Message
to insert an admin account
into the system_auth table
from: http://icanhascheeseburger.com

-Dcassandra.write_survey=true

SSL Certiﬁcates:
a brief interlude

Inter-Node Communication: SSL: Certiﬁcates Done Right
Don't do this:
http://docs.datastax.com/en/cassandra/2.1/cassandra/security/secureSSLCertiﬁcates_t.html <- BAD!
LOL!

BYO Certiﬁcate Authority
A CA provides the root certiﬁcate which can be used to validate a
node's identity without needing direct knowledge of that node.

What we should do:
1. create the CA with a "root" certificate
2. create certificates for each node
3. export each node's certificates to be signed by CA
4. sign each nodes certificate with the CA
5. add the CA root public certificate into each node's key store
5. add the signed result back into that node's key store
6. import CA root public certificate into each node's trust store (or build
one and copy it around)

Creating your own CA
openssl req
-config gen_ca_cert.conf
-new -x509
-keyout ca-key
-out ca-cert
-days 365

'gen_ca_cert.conf'
[ req ]
distinguished_name = req_distinguished_name
prompt = no
output_password = mypass
default_bits = 2048
[ req_distinguished_name ]
C = US
ST = TX
L = Austin
O = TLP
OU = TestCluster
CN = TestClusterMasterCA
emailAddress = info@thelastpickl.com

What we just did
- `req`:An OpenSSL sub-command saying that we are (in this case) creating a PKCS#10 X.509 Certificate (see
https://www.openssl.org/docs/manmaster/apps/req.html for full details)
The following parameters are all options of "-req"
`-config`: The path to the config file (avoids having to provide information on STDIN)
`-new`: This is a new signing request we are making
`-x509`: The output will be a self-signed certificate we can use as a root CA
`-keyout`: The filename to which we will write our key
`-out`: The filename to which we will write our certificate
`-days`: The number of days for which the generated certificate will be valid

Per-Node Certiﬁcate Creation
keytool -genkeypair
-keyalg RSA
-alias node1
-keystore node1-server-keystore.jks
-storepass awesomekeypass
-keypass awesomekeypass
-validity 365 -keysize 2048
-dname "CN=node1, OU=SSL-verification-cluster, O=TheLastPickle, C=US"

What we just did
`-genkeypair`: the keytool command to generate a public/private key pair combination
`-keyalg`:The algorithm to use, RSA in this case*
`-alias`:An alias to use for this public/private key pair. It needs to identify the public key when imported into a
key store. I usually use some form of $hostname-cassandra
`-keystore`:The location of our key store (created if it does not already exist)
`-storepass`:The password for the key store
`-keypass`:The password for the key
`-validity`:The number of days for which this key pair will be valid
`-keysize`:The size of the key to generate
https://docs.oracle.com/javase/8/docs/technotes/tools/unix/keytool.html

A note about "-dname":
CN=node1
Common Name: Best practice is to use hostname
OU=SSL-verification-cluster
Organizational Unit: I like to use the environment speciﬁc cluster name
O=TheLastPickle
Organization
C=US"
Country
Note: Hostname veriﬁcation can be done against subjectAlternativeName
Add "-ext SAN=DNS:thelastpickle.com" to keytool invocation if desired.

Exporting certiﬁcates
for signing by the CA
keytool
-alias node1
-certreq
-file node1_cert_sr

Sign each node's
certiﬁcate with the CA
openssl x509
-req
-CA ca-cert
-CAkey ca-key
-in node1_cert_sr
-out node1_cert_signed
-days 365
-CAcreateserial
-passin pass:mypass

What did we just do:
- `x509` Use the display and signing subcommand (https://www.openssl.org/docs/manmaster/apps/
x509.html)
The following parameters are options of "x509"
- `-req` We are signing a certificate request as opposed to a certificate
- `-CA` The CA cert file created above
- `-CAKey` The CA key file created above
- `-in` The certificate request we are signing
- `-out` The newly-signed certificate file to create
- `-days` The number of days for which the signed certificate will be valid
- `-CAcreateserial` Create a serialnumber for this CSR
- `-passin` The keypassword source.The arguments to 'passin' have their own formatting instructions.
See 'Pass Phrase Arguments' on https://www.openssl.org/docs/manmaster/apps/openssl.html

Add the CA's public key to the
node's key stores
keytool
-alias CARoot
-import
-file ca-cert
-noprompt

Import that node's signed certiﬁcate
back into it's trust store
keytool
-alias node1
-import
-file node1_cert_signed

Build a trust store
from the CA's public certiﬁcate
(this can be shared among all nodes)
keytool
-keystore generic-server-truststore.jks
-alias CARoot -importcert
-file ca-cert
-keypass mypass -storepass
truststorepass -noprompt

OMG!
A Trust Chain!

Inter-Node Communication: SSL Done Right
server_encryption_options:
internode_encryption: all
keystore: conf/server-keystore.jks
keystore_password: awesomekeypass
truststore: conf/server-truststore.jks
truststore_password: truststorepass
protocol: TLS
algorithm: SunX509
store_type: JKS
cipher_suites: [TLS_RSA_WITH_AES_256_CBC_SHA]
require_client_auth: true

internode_encryption: all
keystore: conf/server-keystore.jks
truststore: conf/server-truststore.jks
truststore_password: truststorepass
protocol: TLS
algorithm: SunX509
store_type: JKS
If you don't set this to
true, you might as well
turn encryption off

Also: Key store password and key password
*must be the same*
...
...
keytool ... -keypass awesomekeypass ...

Export restrictions?
Use128 bit keys
http://www.oracle.com/technetwork/java/javase/downloads/jce8-download-2133166.html
http://docs.oracle.com/javase/8/docs/technotes/guides/security/SunProviders.html#importlimits

Trouble?
-Djavax.net.debug=ssl
http://docs.oracle.com/javase/7/docs/technotes/guides/security/jsse/ReadDebug.html

Netflix Lemur:
"x.509 Certificate Orchestration Framework"
http://techblog.netflix.com/2015/09/introducing-lemur.html
https://github.com/Netflix/lemur

HashicorpVault
"secures, stores, and tightly controls access to
tokens, passwords, certiﬁcates,API keys, and
other secrets in modern computing. "
https://www.vaultproject.io/

Service level certiﬁcate management
should have nothing to do with front-
end certiﬁcates

internode_authenticator:
Stronger node identity veriﬁcation

Inter-Node Communication:Authenticator
Available commercially
in DSE

Pretty easy
to roll your own

o.a.c.auth.IInternodeAuthenticator
boolean authenticate(InetAddress remoteAddress, int remotePort);
void validateConfiguration() throws ConfigurationException;

Same CA setup
as Server-to-Server
(same limitations, too)

client_encryption_options:
enabled: true
keystore: conf/client-keystore.jks
keystore_password: clientkeypass
truststore: conf/client-truststore.jks
truststore_password: clienttrustpass
protocol: TLS
algorithm: SunX509
store_type: JKS

client_encryption_options:
enabled: false
keystore: conf/client-keystore.jks
keystore_password: clientkeypass
truststore: conf/client-truststore.jks
truststore_password: clienttrustpass
protocol: TLS
algorithm: SunX509
store_type: JKS
If you don't set this to
true, you might as well
turn encryption off

Authentication and Authorization:
Best practices
are the same as for
RDBMS

Authentication and Authorization: Best Practices
Segment users:
- Admin
- Per-service CRUD
- CRUD (reporting)

Authentication and Authorization: Best Practices
Limit access to
tables
and keyspaces

Authentication and Authorization:
Role-based access control
(RBAC) in 2.2

Authentication and Authorization:Authentication
authenticator: PasswordAuthenticator
Tip: keep your read-only cqlsh credentials in
$HOME/.cassandra/cqlshrc
of the system's admin account

Authentication and Authorization:Authorization
authorizer: CassandraAuthorizer
TIP: turn permissions_validity_in_ms WAY UP
default is 2000 (2 seconds!!)
can use permissions_update_interval_in_ms
for async refresh if needed

Authentication and Authorization:Authorization
These tables have
default read permission
for every authenticated user:
system.schema_keyspace
system.schema_columns
system.schema_columnfamilies
system.local
system.peers

Authentication and Authorization:Authentication
IMPORTANT:
"Please increase system_auth keyspace
replication factor if you use this..."
RF=Number of Nodes

Encryption at Rest
Management and Tooling

Mangement andTooling:
Securing JMX

Management andTooling: Securing JMX
Options as of 2.1.8
will look familiar:
JVM_OPTS="$JVM_OPTS -Djavax.net.ssl.keyStore=/path/to/keystore"
JVM_OPTS="$JVM_OPTS -Djavax.net.ssl.keyStorePassword=<keystore-password>"
JVM_OPTS="$JVM_OPTS -Djavax.net.ssl.trustStore=/path/to/truststore"
JVM_OPTS="$JVM_OPTS -Djavax.net.ssl.trustStorePassword=<truststore-password>"
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.ssl.need.client.auth=true"
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.registry.ssl=true"
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.ssl.enabled.protocols=<enabled-protocols>"
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.ssl.enabled.cipher.suites=<enabled-cipher-
suites>"

Thus:
as of 2.1.8, nodetool can use SSL*
* https://issues.apache.org/jira/browse/CASSANDRA-9090

Conﬁgure authentication
for JMX*
Now you can do:
nodetool status -u monitor -pw monitorpass
* http://docs.datastax.com/en/cassandra/2.1/cassandra/security/secureJmxAuthentication.html

JMX user/password and role
conﬁguration is ﬂexible
For details, take a look at the defaults:
$JAVA_HOME/jre/lib/management/jmxremote.access
$JAVA_HOME/jre/lib/management/jmxremote.password.template

Nate McCall
@zznate
Co-Founder & Sr.Technical Consultant
www.thelastpickle.com
Seattle Cassandra Users

Seattle C* Meetup: Hardening cassandra for compliance or paranoia

Recommended

Recommended

More Related Content

What's hot

What's hot (17)

Similar to Seattle C* Meetup: Hardening cassandra for compliance or paranoia

Similar to Seattle C* Meetup: Hardening cassandra for compliance or paranoia (20)

More from zznate

More from zznate (13)

Recently uploaded

Recently uploaded (20)

Seattle C* Meetup: Hardening cassandra for compliance or paranoia