Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Seattle C* Meetup: Hardening cassandra for compliance or paranoia

1,630 views

Published on

Details how to secure Apache Cassandra clusters. Covers client to server and server to server encryption, securing management and tooling, using authentication and authorization, as well as options for encryption at rest.

Published in: Data & Analytics

Seattle C* Meetup: Hardening cassandra for compliance or paranoia

  1. 1. HARDENING CASSANDRA FOR COMPLIANCE (OR PARANOIA) Nate McCall @zznate Seattle Cassandra Users Co-Founder & Sr.Technical Consultant Licensed under a Creative Commons Attribution-NonCommercial 3.0 New Zealand License
  2. 2. AboutThe Last Pickle. Work with clients to deliver and improve Apache Cassandra based solutions. Based in New Zealand,Australia & USA.
  3. 3. OVERVIEW
  4. 4. Encryption at Rest Inter-node Communication Client-Server Communication Authentication and Authorization Management andTooling
  5. 5. Overview: Security is usually necessitated by Industry Guidelines (depending on organization size)
  6. 6. Overview: Industry Guidelines - Financial: PCI-DSS - Healthcare: HIPAA - Defense/Govt: FIPS - TelCo: CPNI
  7. 7. Overview: Industry Guidelines Their heart is in the right place... (Do you really need node to node SSL if it never leaves a backplane?)
  8. 8. Overview: However: good security practices can save you
  9. 9. Overview: By necessity we abstract and encapsulate, making it easier to have un-intended side effects knife ssh -x ${knifeUser} -a hostname -E $chefEnv "role:${role}" "${executeCommand}" Then eventually: rm -rf /var/lib/cassandra/*
  10. 10. Overview: Basic systems hardening can also protect from common operational errors
  11. 11. Overview: System Hardening Limit account access "sudo su -" = trouble
  12. 12. Overview: System Hardening Limit filesystem access. Only C* needs /var/lib/cassandra (configuration management does not need access after initialization)
  13. 13. Overview: System Hardening Networking - Limit network access - network segmentation and ACLs - iptables, etc
  14. 14. Overview: System Hardening: Limit Network Access C* only needs inbound on: - 9042/9160 (clients) - 7000/7001 (cluster) - 7199 (JMX)
  15. 15. Encryption at Rest Inter-node Communication Client-Server Communication Authentication and Authorization Management andTooling
  16. 16. Encryption At Rest: Why it's a good idea: - Do you know how your cloud provider is actually using storage under the hood? - What does IT do with decommissioned disks? (Are you sure?)
  17. 17. Encryption At Rest: There are good open source and commercial options for at-rest encryption. But...
  18. 18. Encryption At Rest: It's a distributed system. Make sure you understand the HA Story in context of your needs. Otherwise you introduced an SPOF! eg. "EBS snapshots of the key server" Is this going to work for you?
  19. 19. Encryption At Rest: File level vs. Block level
  20. 20. Encryption At Rest: Open Source dmCrypt (block) eCrypt-FS (file) Major vendors probably support both https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/5/html/Installation_Guide/ch29s02.html https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Storage_Administration_Guide/ch-efs.html
  21. 21. Encryption At Rest: Commercial - Cloudera Gazzang! -Vormetric
  22. 22. Encryption At Rest: Commercial DataStax Enterprise*: CREATETABLE users ... WITH compression_parameters:sstable_compression = 'Encryptor' and compression_parameters:cipher_algorithm = 'AES/ECB/PKCS5Padding' and compression_parameters:secret_key_strength = 128; *commit log not included! (eCrypt-fs to the rescue)
  23. 23. Encryption At Rest: Commercial-ish EBS Encryption! (a.k.a. "Not my problem.") http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSEncryption.html
  24. 24. Encryption at Rest Inter-Node Communication Client-Server Communication Authentication and Authorization Management andTooling
  25. 25. Inter-Node Communication: SSL Inter-Node encryption can be rolled out on production with no downtime* * Depending on client consistency levels
  26. 26. Inter-Node Communication: SSL Why is it a good idea?
  27. 27. Inter-Node Communication: SSL:Why It's a Good Idea: Every operation is a Message on the wire
  28. 28. Inter-Node Communication: SSL:Why It's a Good Idea: It takes one Message to insert an admin account into the system_auth table from: http://icanhascheeseburger.com
  29. 29. Inter-Node Communication: SSL:Why It's a Good Idea: -Dcassandra.write_survey=true
  30. 30. Inter-Node Communication: SSL SSL Certificates: a brief interlude
  31. 31. Inter-Node Communication: SSL: Certificates Done Right Don't do this: http://docs.datastax.com/en/cassandra/2.1/cassandra/security/secureSSLCertificates_t.html <- BAD! LOL!
  32. 32. Inter-Node Communication: SSL: Certificates Done Right BYO Certificate Authority A CA provides the root certificate which can be used to validate a node's identity without needing direct knowledge of that node.
  33. 33. Inter-Node Communication: SSL: Certificates Done Right What we should do: 1. create the CA with a "root" certificate 2. create certificates for each node 3. export each node's certificates to be signed by CA 4. sign each nodes certificate with the CA 5. add the CA root public certificate into each node's key store 5. add the signed result back into that node's key store 6. import CA root public certificate into each node's trust store (or build one and copy it around)
  34. 34. Inter-Node Communication: SSL: Certificates Done Right Creating your own CA openssl req -config gen_ca_cert.conf -new -x509 -keyout ca-key -out ca-cert -days 365
  35. 35. Inter-Node Communication: SSL: Certificates Done Right 'gen_ca_cert.conf' [ req ] distinguished_name = req_distinguished_name prompt = no output_password = mypass default_bits = 2048 [ req_distinguished_name ] C = US ST = TX L = Austin O = TLP OU = TestCluster CN = TestClusterMasterCA emailAddress = info@thelastpickl.com
  36. 36. Inter-Node Communication: SSL: Certificates Done Right What we just did - `req`:An OpenSSL sub-command saying that we are (in this case) creating a PKCS#10 X.509 Certificate (see https://www.openssl.org/docs/manmaster/apps/req.html for full details) The following parameters are all options of "-req" `-config`: The path to the config file (avoids having to provide information on STDIN) `-new`: This is a new signing request we are making `-x509`: The output will be a self-signed certificate we can use as a root CA `-keyout`: The filename to which we will write our key `-out`: The filename to which we will write our certificate `-days`: The number of days for which the generated certificate will be valid
  37. 37. Inter-Node Communication: SSL: Certificates Done Right Per-Node Certificate Creation keytool -genkeypair -keyalg RSA -alias node1 -keystore node1-server-keystore.jks -storepass awesomekeypass -keypass awesomekeypass -validity 365 -keysize 2048 -dname "CN=node1, OU=SSL-verification-cluster, O=TheLastPickle, C=US"
  38. 38. Inter-Node Communication: SSL: Certificates Done Right What we just did `-genkeypair`: the keytool command to generate a public/private key pair combination `-keyalg`:The algorithm to use, RSA in this case* `-alias`:An alias to use for this public/private key pair. It needs to identify the public key when imported into a key store. I usually use some form of $hostname-cassandra `-keystore`:The location of our key store (created if it does not already exist) `-storepass`:The password for the key store `-keypass`:The password for the key `-validity`:The number of days for which this key pair will be valid `-keysize`:The size of the key to generate https://docs.oracle.com/javase/8/docs/technotes/tools/unix/keytool.html
  39. 39. Inter-Node Communication: SSL: Certificates Done Right A note about "-dname": CN=node1 Common Name: Best practice is to use hostname OU=SSL-verification-cluster Organizational Unit: I like to use the environment specific cluster name O=TheLastPickle Organization C=US" Country Note: Hostname verification can be done against subjectAlternativeName Add "-ext SAN=DNS:thelastpickle.com" to keytool invocation if desired.
  40. 40. Inter-Node Communication: SSL: Certificates Done Right Exporting certificates for signing by the CA keytool -keystore node1-server-keystore.jks -alias node1 -certreq -file node1_cert_sr -keypass awesomekeypass -storepass awesomekeypass
  41. 41. Inter-Node Communication: SSL: Certificates Done Right Sign each node's certificate with the CA openssl x509 -req -CA ca-cert -CAkey ca-key -in node1_cert_sr -out node1_cert_signed -days 365 -CAcreateserial -passin pass:mypass
  42. 42. Inter-Node Communication: SSL: Certificates Done Right What did we just do: - `x509` Use the display and signing subcommand (https://www.openssl.org/docs/manmaster/apps/ x509.html) The following parameters are options of "x509" - `-req` We are signing a certificate request as opposed to a certificate - `-CA` The CA cert file created above - `-CAKey` The CA key file created above - `-in` The certificate request we are signing - `-out` The newly-signed certificate file to create - `-days` The number of days for which the signed certificate will be valid - `-CAcreateserial` Create a serialnumber for this CSR - `-passin` The keypassword source.The arguments to 'passin' have their own formatting instructions. See 'Pass Phrase Arguments' on https://www.openssl.org/docs/manmaster/apps/openssl.html
  43. 43. Inter-Node Communication: SSL: Certificates Done Right Add the CA's public key to the node's key stores keytool -keystore node1-server-keystore.jks -alias CARoot -import -file ca-cert -noprompt -keypass awesomekeypass -storepass awesomekeypass
  44. 44. Inter-Node Communication: SSL: Certificates Done Right Import that node's signed certificate back into it's trust store keytool -keystore node1-server-keystore.jks -alias node1 -import -file node1_cert_signed -keypass awesomekeypass -storepass awesomekeypass
  45. 45. Inter-Node Communication: SSL: Certificates Done Right Build a trust store from the CA's public certificate (this can be shared among all nodes) keytool -keystore generic-server-truststore.jks -alias CARoot -importcert -file ca-cert -keypass mypass -storepass truststorepass -noprompt
  46. 46. Inter-Node Communication: SSL: Certificates Done Right OMG! A Trust Chain!
  47. 47. Inter-Node Communication: SSL Done Right server_encryption_options: internode_encryption: all keystore: conf/server-keystore.jks keystore_password: awesomekeypass truststore: conf/server-truststore.jks truststore_password: truststorepass protocol: TLS algorithm: SunX509 store_type: JKS cipher_suites: [TLS_RSA_WITH_AES_256_CBC_SHA] require_client_auth: true
  48. 48. Inter-Node Communication: SSL Done Right server_encryption_options: internode_encryption: all keystore: conf/server-keystore.jks keystore_password: awesomekeypass truststore: conf/server-truststore.jks truststore_password: truststorepass protocol: TLS algorithm: SunX509 store_type: JKS cipher_suites: [TLS_RSA_WITH_AES_256_CBC_SHA] require_client_auth: true If you don't set this to true, you might as well turn encryption off
  49. 49. Inter-Node Communication: SSL Done Right Also: Key store password and key password *must be the same* server_encryption_options: ... keystore_password: awesomekeypass require_client_auth: true ... keytool ... -keypass awesomekeypass ...
  50. 50. Inter-Node Communication: SSL Done Right Export restrictions? Use128 bit keys cipher_suites: [TLS_RSA_WITH_AES_128_CBC_SHA] http://www.oracle.com/technetwork/java/javase/downloads/jce8-download-2133166.html http://docs.oracle.com/javase/8/docs/technotes/guides/security/SunProviders.html#importlimits
  51. 51. Inter-Node Communication: SSL: Certificates Done Right Trouble? -Djavax.net.debug=ssl http://docs.oracle.com/javase/7/docs/technotes/guides/security/jsse/ReadDebug.html
  52. 52. Inter-Node Communication: SSL: Certificates Done Right Netflix Lemur: "x.509 Certificate Orchestration Framework" http://techblog.netflix.com/2015/09/introducing-lemur.html https://github.com/Netflix/lemur
  53. 53. Inter-Node Communication: SSL: Certificates Done Right HashicorpVault "secures, stores, and tightly controls access to tokens, passwords, certificates,API keys, and other secrets in modern computing. " https://www.vaultproject.io/
  54. 54. Inter-Node Communication: SSL: Certificates Done Right Service level certificate management should have nothing to do with front- end certificates
  55. 55. Inter-Node Communication internode_authenticator: Stronger node identity verification
  56. 56. Inter-Node Communication:Authenticator Available commercially in DSE
  57. 57. Inter-Node Communication:Authenticator Pretty easy to roll your own
  58. 58. Inter-Node Communication:Authenticator o.a.c.auth.IInternodeAuthenticator boolean authenticate(InetAddress remoteAddress, int remotePort); void validateConfiguration() throws ConfigurationException;
  59. 59. Encryption at Rest Inter-Node Communication Client-Server Communication Authentication and Authorization Management andTooling
  60. 60. Client-Server Communication Same CA setup as Server-to-Server (same limitations, too)
  61. 61. Client-Server Communication client_encryption_options: enabled: true keystore: conf/client-keystore.jks keystore_password: clientkeypass truststore: conf/client-truststore.jks truststore_password: clienttrustpass protocol: TLS algorithm: SunX509 store_type: JKS cipher_suites: [TLS_RSA_WITH_AES_256_CBC_SHA] require_client_auth: true
  62. 62. Client-Server Communication client_encryption_options: enabled: false keystore: conf/client-keystore.jks keystore_password: clientkeypass truststore: conf/client-truststore.jks truststore_password: clienttrustpass protocol: TLS algorithm: SunX509 store_type: JKS cipher_suites: [TLS_RSA_WITH_AES_256_CBC_SHA] require_client_auth: true If you don't set this to true, you might as well turn encryption off
  63. 63. Encryption at Rest Inter-Node Communication Client-Server Communication Authentication and Authorization Management andTooling
  64. 64. Authentication and Authorization: Best practices are the same as for RDBMS
  65. 65. Authentication and Authorization: Best Practices Segment users: - Admin - Per-service CRUD - CRUD (reporting)
  66. 66. Authentication and Authorization: Best Practices Limit access to tables and keyspaces
  67. 67. Authentication and Authorization: Role-based access control (RBAC) in 2.2
  68. 68. Authentication and Authorization:Authentication authenticator: PasswordAuthenticator Tip: keep your read-only cqlsh credentials in $HOME/.cassandra/cqlshrc of the system's admin account
  69. 69. Authentication and Authorization:Authorization authorizer: CassandraAuthorizer TIP: turn permissions_validity_in_ms WAY UP default is 2000 (2 seconds!!) can use permissions_update_interval_in_ms for async refresh if needed
  70. 70. Authentication and Authorization:Authorization These tables have default read permission for every authenticated user: system.schema_keyspace system.schema_columns system.schema_columnfamilies system.local system.peers
  71. 71. Authentication and Authorization:Authentication IMPORTANT: "Please increase system_auth keyspace replication factor if you use this..." RF=Number of Nodes
  72. 72. Encryption at Rest Inter-Node Communication Client-Server Communication Authentication and Authorization Management and Tooling
  73. 73. Mangement andTooling: Securing JMX
  74. 74. Management andTooling: Securing JMX Options as of 2.1.8 will look familiar: JVM_OPTS="$JVM_OPTS -Djavax.net.ssl.keyStore=/path/to/keystore" JVM_OPTS="$JVM_OPTS -Djavax.net.ssl.keyStorePassword=<keystore-password>" JVM_OPTS="$JVM_OPTS -Djavax.net.ssl.trustStore=/path/to/truststore" JVM_OPTS="$JVM_OPTS -Djavax.net.ssl.trustStorePassword=<truststore-password>" JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.ssl.need.client.auth=true" JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.registry.ssl=true" JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.ssl.enabled.protocols=<enabled-protocols>" JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.ssl.enabled.cipher.suites=<enabled-cipher- suites>"
  75. 75. Management andTooling: Securing JMX Thus: as of 2.1.8, nodetool can use SSL* * https://issues.apache.org/jira/browse/CASSANDRA-9090
  76. 76. Management andTooling: Securing JMX Configure authentication for JMX* Now you can do: nodetool status -u monitor -pw monitorpass * http://docs.datastax.com/en/cassandra/2.1/cassandra/security/secureJmxAuthentication.html
  77. 77. Management andTooling: Securing JMX JMX user/password and role configuration is flexible For details, take a look at the defaults: $JAVA_HOME/jre/lib/management/jmxremote.access $JAVA_HOME/jre/lib/management/jmxremote.password.template
  78. 78. Thanks.
  79. 79. Nate McCall @zznate Co-Founder & Sr.Technical Consultant www.thelastpickle.com Seattle Cassandra Users

×