Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Securing kafka with 500 billion messages a day

45 views

Published on

Silicon Valley Code Camp 2019 Slides

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Securing kafka with 500 billion messages a day

  1. 1. Securing Kafka at PayPal with 500 Billion Messages a Day
  2. 2. Agenda © 2019 PayPal Inc. Confidential and proprietary. Kafka @ PayPal Today TLS on Kafka Overview Enabling mTLS Authentication Future Work Conclusion
  3. 3. Kafka @ PayPal Today © 2019 PayPal Inc. Confidential and proprietary. Overview 500+ Billion messages per day 50+ Clusters ~7PB Disk 5000+ Topics 0.9 0.10 1.10.8Kafka Journey 2.2
  4. 4. Kafka @ PayPal Today © 2019 PayPal Inc. Confidential and proprietary. Tech stack Language s Gimel Application Frameworks Multi-Tenant Multiple Security & Availability Zones
  5. 5. Data Pipelines © 2019 PayPal Inc. Confidential and proprietary. Use Cases User behavioral tracking Experimental Merchant monitoring Risk & compliance Business Events Application logs Application metrics Gimel Frameworks & Platforms Real-Time Streaming Batch Processing Kafka
  6. 6. Enabling mTLS
  7. 7. SSL & TLS Terminology • SSL Key • SSL Certificate • CA • Trusted CAs © 2019 PayPal Inc. Confidential and proprietary.
  8. 8. TLS Introduction Encryption and Authenticating using SSL in Open Source Kafka • Generate SSL Key and Certs • Create CA • Sign the certificates • Configure Kafka Properties © 2019 PayPal Inc. Confidential and proprietary.
  9. 9. © 2019 PayPal Inc. Confidential and proprietary. keytool -keystore server.keystore.jks -alias localhost -validity {validity} -genkey -keyalg RSA openssl req -new -x509 -keyout ca-key -out ca-cert -days 365 keytool -keystore client.truststore.jks -alias CARoot -import -file ca-cert keytool -keystore server.truststore.jks -alias CARoot -import -file ca-cert keytool -keystore server.keystore.jks -alias localhost -certreq -file cert-file openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-file -out cert-signed -days {validity} -CAcreateserial -passin pass:{ca- password} keytool -keystore server.keystore.jks -alias CARoot -import -file ca-cert keytool -keystore server.keystore.jks -alias localhost -import -file cert-signed CLI Commands
  10. 10. KeyStore & TrustStore © 2019 PayPal Inc. Confidential and proprietary. KeyStore TrustStore
  11. 11. Configuration Sample Broker Configurations listeners=PLAINTEXT://host.name:port,SSL://host.name:port ssl.keystore.location=/var/private/ssl/server.keystore.jks ssl.keystore.password=test1234 [Optional] ssl.key.password=test1234 ssl.truststore.location=/var/private/ssl/server.truststore.jks ssl.truststore.password=test1234 [Optional] © 2019 PayPal Inc. Confidential and proprietary.
  12. 12. Configuration Sample Client Configuration security.protocol=SSL ssl.truststore.location=/var/private/ssl/client.truststore.jks ssl.truststore.password=test1234 [Optional] [Optional with mTLS] ssl.keystore.location=/var/private/ssl/client.keystore.jks ssl.keystore.password=test1234 [Optional] ssl.key.password=test1234 © 2019 PayPal Inc. Confidential and proprietary.
  13. 13. © 2019 PayPal Inc. Confidential and proprietary. Kafka Broker File-Based KeyStore Client Application Server Authentication File-Based TrustStore TLS with One-way Auth
  14. 14. © 2019 PayPal Inc. Confidential and proprietary. Kafka Broker File-Based KeyStore Client Application Server Authentication File-Based TrustStore File-Based TrustStore File-Based KeyStore Client Authentication TLS with Mutual Auth
  15. 15. Challenges to enable Kafka TLS @ PayPal InfoSec/AppSec Restrictions • File-based security material/credentials are not allowed • Regular key rotations Scalability • Deploying the key/trust store files to the thousands of brokers • Deploying the key/trust store files to 10x/100x Kafka Client boxes • Keep all those key/trust store files and their passwords secure • Lack of unified way to distribute KeyStore and TrustStore for different languages © 2019 PayPal Inc. Confidential and proprietary.
  16. 16. PayPal’s Key Management Service In-house Key Management Service [Like Vault / AWS KMS] • A company internal CA authority issuing the certificates • Rotate the Keys © 2019 PayPal Inc. Confidential and proprietary.
  17. 17. Work Flow without SSL in PayPal © 2019 PayPal Inc. Confidential and proprietary. Kafka Client Kafka Client Config Service Config Service Send Topic Config Request Return Topic Config: {Topic1: Bootstrap Servers, client configs Topic 2: Bootstrap Servers, client configs } Connect to Kafka Broker
  18. 18. Solutions: Kafka Source Code Change – Client & Broker Change Kafka Source Code to allow KeyStore/TrustStore plug-in • Provide KeyStoreLoader & TrustStoreLoader Interface for customized KeyStore and TrustStore Loading © 2019 PayPal Inc. Confidential and proprietary.
  19. 19. Work Flow with SSL in PayPal © 2019 PayPal Inc. Confidential and proprietary. Kafka Client Kafka Client Config Service Config Service Send Topic Config Request Return Topic Config Connect to Kafka Broker KMS KMS Send Key/Trust Store Request Get KeyStore and TrustStore
  20. 20. Configuration Sample with Custom Key/Trust Store Broker Configurations listeners=SSL://host.name:port ssl.keystore.loader=com.paypal.kafka.broker.KeyStoreLoaderImpl ssl.keystore.location= ssl.keystore.password= ssl.key.password= ssl.truststore.loader=com.paypal.kafka.broker.TrustStoreLoaderImpl ssl.truststore.location= ssl.truststore.password= © 2019 PayPal Inc. Confidential and proprietary.
  21. 21. Client Configurations security.protocol=SSL ssl.truststore.loader= com.paypal.kafka.client.TrustStoreLoaderImpl ssl.keystore.location= ssl.keystore.password= ssl.key.password= [Optional with mTLS] ssl.keystore.loader=com.paypal.kafka.client.KeyStoreLoaderImpl ssl.truststore.location= ssl.truststore.password= © 2019 PayPal Inc. Confidential and proprietary. Configuration Sample with Custom Key/Trust Store
  22. 22. Kafka Improvement Proposal KIP-486 – Support custom way to load key and trust stores (Work-In-Progress) KIP-519 – Make SSL context/engine configuration extensible (Work-In-Progress) © 2019 PayPal Inc. Confidential and proprietary.
  23. 23. Kafka SASL Authentication
  24. 24. Authentication MTLS SASL • GSSAPI (Kerberos) • SCRAM • Delegation Tokens • PLAIN • OAUTHBEARER Authentication Overview © 2019 PayPal Inc. Confidential and proprietary.
  25. 25. Authentication KIP-12 - Kafka Sasl/Kerberos and SSL implementation • 0.9.0.0 KIP-85: Dynamic JAAS configuration for Kafka clients • 0.10.2.0 KIP-86: Configurable SASL callback handlers • 2.0.0 KIP-255: OAuth Authentication via SASL/OAUTHBEARER • 2.0.0 KIP 368: Allow SASL Connections to Periodically Re-Authenticate • 2.2.0 Authentication Overview – evolution © 2019 PayPal Inc. Confidential and proprietary.
  26. 26. Authentication KafkaClient (Procuder/Consumer) KafkaChannel Authenticator Authentication Overview – Auth Components © 2019 PayPal Inc. Confidential and proprietary. Kafka (Server) KafkaServer Processor Authenticator KafkaRequestHandler KafkaApis Authorizer
  27. 27. Authentication sasl.login.class LoginModule sasl.login.callback.handler.class sasl.client.callback.handler.class sasl.server.callback.handler.class principal.builder.class Authentication Overview - Pluggable Components © 2019 PayPal Inc. Confidential and proprietary.
  28. 28. Authentication KafkaClient LoginManager.acquireLoginManager DefaultLogin.login() LoginContext.login() LoginModule .configure() .login() AuthenticateCallbackHandler.handle(Callback) .commit() Set Credentials to Subject KafkaChannel.prepare() Authenticator.authenticate sendSaslClientToken SaslClient.evaluateChallenge AuthenticateCallbackHandler.handle(Callback) retrieve Credentials from Subject Authentication Overview – Client Auth Pluggable Components © 2019 PayPal Inc. Confidential and proprietary. <- sasl.login.class <- KafkaClient Jaas config <- sasl.login.callback.handler.class <- sasl.client.callback.handler.class
  29. 29. Authentication Kafka Server LoginManager.acquireLoginManager DefaultLogin.login() LoginContext.login() LoginModule .configure() .login() AuthenticateCallbackHandler.handle(Callback) .commit() Processor(1).run() poll() KafkaChannel.prepare() SaslServerAuthenticator.authenticate … AuthenticateCallbackHandler.handle(Callback) validate() Authentication Overview - Server Auth Pluggable Components © 2019 PayPal Inc. Confidential and proprietary. <- sasl.login.class <- KafkaServer jaas config <- sasl.login.callback.handler.class <- sasl.server.callback.handler.class
  30. 30. Authentication … LoginManager.acquireLoginManager DefaultLogin.login() LoginContext.login() LoginModule .configure() .login() AuthenticateCallbackHandler.handle(Callback) .commit() … Authentication Overview – Pluggable Components © 2019 PayPal Inc. Confidential and proprietary. <- sasl.login.class <- KafkaClient Jaas config <- sasl.login.callback.handler.class
  31. 31. Authentication sasl.login.class • An implementation of org.apache.kafka.common.security.auth.Login that performs login for each LoginModule specified through login context. • LoginManager initializes Login class by passing appropriate login handler configured through sasl.login.callback.handler.class • In most cases out of box Login Implementations are sufficient. DefaultLogin, KerberosLogin, OAuthBearerRefreshingLogin • Applicable to client and server Authentication Overview - Pluggable Components © 2019 PayPal Inc. Confidential and proprietary.
  32. 32. Authentication … LoginManager.acquireLoginManager DefaultLogin.login() LoginContext.login() LoginModule .configure() .login() AuthenticateCallbackHandler.handle(Callback) .commit() … Authentication Overview – Pluggable Components © 2019 PayPal Inc. Confidential and proprietary. <- sasl.login.class <- KafkaClient/KafkaServer Jaas config <- sasl.login.callback.handler.class
  33. 33. Authentication LoginModule • Interface to implement SASL specific Authentication • Java security LoginContext invokes LoginModule to perform implementation specific Authentication and initializes public and private credentials for a Subject • LoginModule should be provided in jaas config format • KIP-86 enables clients to load jaas config dynamically, As a jvm parameter export KAFKA_OPTS="-Djava.security.auth.login.config=$KAFKA_HOME/config/kafka_server_jaas.conf” Through Configuration property sasl.jaas.config=org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required • In most cases out of box LoginModule implementations are sufficient for each type of authentication. OAuthBearerLoginModule, PlainLoginModule, ScramLoginModule, Krb5LoginModule • Applicable to client and server Authentication Overview - Pluggable Components © 2019 PayPal Inc. Confidential and proprietary.
  34. 34. Authentication … LoginManager.acquireLoginManager DefaultLogin.login() LoginContext.login() LoginModule .configure() .login() AuthenticateCallbackHandler.handle(Callback) .commit() … Authentication Overview – Pluggable Components © 2019 PayPal Inc. Confidential and proprietary. <- sasl.login.class <- KafkaClient/KafkaServer Jaas config <- sasl.login.callback.handler.class
  35. 35. Authentication sasl.login.callback.handler.class • An implementation of AuthenticateCallbackHandler that should retrieve credentials from external services and set it to Callback, these credentials are used by LoginModule to set private and public credentials on the Subject. • For Production use cases DON’T use default handlers, it is recommended to write custom implementation of AuthenticateCallbackHandler compatible to respective SASL mechanisms LoginModule. • Out of box Handlers are • OAuthBearerUnsecuredLoginCallbackHandler, DefaultLoginCallbackHandler • Applicable to client and server Authentication Overview - Pluggable Components © 2019 PayPal Inc. Confidential and proprietary.
  36. 36. Authentication KafkaClient LoginManager.acquireLoginManager DefaultLogin.login() LoginContext.login() LoginModule .configure() .login() AuthenticateCallbackHandler.handle(Callback) .commit() KafkaChannel.prepare() Authenticator.authenticate sendSaslClientToken SaslClient.evaluateChallenge AuthenticateCallbackHandler.handle(Callback) Authentication Overview – Pluggable Components © 2019 PayPal Inc. Confidential and proprietary. <- sasl.login.class <- KafkaClient Jaas config <- sasl.login.callback.handler.class <- sasl.client.callback.handler.class
  37. 37. Authentication sasl.client.callback.handler.class • An implementation of AuthenticateCallbackHandler that should retrieve credentials from Subject that were created during LoginModule.login() and return it to Authenticator. • In most cases out of box Handlers are sufficient • SaslClientCallbackHandler, OAuthBearerSaslClientCallbackHandler, KerberosClientCallbackHandler • Applicable to client Authentication Overview - Pluggable Components © 2019 PayPal Inc. Confidential and proprietary.
  38. 38. Authentication Kafka Server LoginManager.acquireLoginManager DefaultLogin.login() LoginContext.login() LoginModule .configure() .login() AuthenticateCallbackHandler.handle(Callback) .commit() Processor(1).run() poll() KafkaChannel.prepare() SaslServerAuthenticator.authenticate … AuthenticateCallbackHandler.handle(Callback) Authentication Overview - Server Auth Pluggable Components © 2019 PayPal Inc. Confidential and proprietary. <- sasl.login.class <- KafkaServer jaas config <- sasl.login.callback.handler.class <- sasl.server.callback.handler.class
  39. 39. Authentication sasl.server.callback.handler.class • An implementation of AuthenticateCallbackHandler that should validate credentials passed by client through external service or by some means. • If you are using Plan or Oauth SASL mechanisum it is recommended to write custom implementation of AuthenticateCallbackHandler to validate credentials. • Out of box Handlers are • SaslServerCallbackHandler, OAuthBearerUnsecuredValidatorCallbackHandler, PlainServerCallbackHandler, ScramServerCallbackHandler Authentication Overview - Pluggable Components © 2019 PayPal Inc. Confidential and proprietary.
  40. 40. Kafka Security at PayPal - OAUTHBEARER
  41. 41. OAuth Authentication at PayPal © 2019 PayPal Inc. Confidential and proprietary. Kafka Client Request Access to Kafka KMS Kafka KafkaKMSKafka Client Creates a JWS token with Kafka Public Key Get Access Token Validate User Encrypted JWS token Authenticate(JWS token) Decrypt JWS token and extract Principal and Scopes OfflineOnline
  42. 42. Authentication Kafka Security at PayPal - OAUTHBEARER © 2019 PayPal Inc. Confidential and proprietary. 1. Broker Loading Keymaterial (PayPalServerLoginCallbackHandler) 1. On startup it authenticates with KMS 2. Loads required key material from KMS that is required to decrypt jws token provided by client. 2. Get Client Access Token (PayPalClientLoginCallbackHandler) 1. On startup it authenticates with KMS 2. Loads jws token 3. Set the token to Channel Subject Public and Private Credentials 3. Authenticate Client (PayPalServerTokenValidatorCallbackHandler) 1. SASL Authenticator uses Client Callback handler to retrieve credentials from Channel Subject 2. Pass it to Server for Authentication, 3. Server Callback handler receives the request for validation 4. It uses the private key to decrypt the token and extract the principal & scopes 5. Update this information to Channel Subject public and private credentials for later use by Principal extractor to Authorizing the resource.
  43. 43. Authentication Kafka Security at PayPal - OAUTHBEARER © 2019 PayPal Inc. Confidential and proprietary. sasl.login.class: OAuthBearerRefreshingLogin LoginModule: OAuthBearerLoginModule sasl.login.callback.handler.class: PayPalClientLoginCallbackHandler, PayPalServerLoginCallbackHandler sasl.client.callback.handler.class: OAuthBearerSaslClientCallbackHandler sasl.server.callback.handler.class: PayPalServerTokenValidatorCallbackHandler principal.builder.class: DefaultKafkaPrincipalBuilder
  44. 44. Authentication Kafka Security at PayPal © 2019 PayPal Inc. Confidential and proprietary. Connectio n Service Definition 1,4 Kafka SSL + SASL OAuth 2 Kafka SSL 3 ZK SASL:DIGEST -MD5 6 ZK Admin ports for follower if this instance is Leader 7 ZK Admin ports for Leader election
  45. 45. Future Work © 2019 PayPal Inc. Confidential and proprietary. • Automating ACL management
  46. 46. Conclusion © 2019 PayPal Inc. Confidential and proprietary.
  47. 47. Thank you!

×