Practical Kerberos

1 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Practical Kerberos
with Apache Accumulo
Josh Elser
Accumulo Summit 2017
2017/10/16

2 © Hortonworks Inc. 2011 – 2017. All Rights Reserved2 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Engineer at Hortonworks, Member of the Apache Software Foundation
Top-Level Projects
• Apache Accumulo®
• Apache CalciteTM
• Apache CommonsTM
• Apache HBase®
• Apache PhoenixTM
ASF Incubator
• Apache FluoTM
• Apache GossipTM
• Apache RyaTM
• Apache SliderTM
These names are trademarks or registered trademarks
of the Apache Software Foundation.

… but today we’re talking about Kerberos!
- “The Madness beyond the Gate” [1]
- An exploration in black magic and voodoo
- The word most accompanied with expletives
1: https://steveloughran.gitbooks.io/kerberos_and_hadoop/content/sections/kerberos_the_madness.html

What this talk won’t be...
3dom via https://www.flickr.com/photos/steve_l/6042206137/in/album-72157629289333057/, CC-BY-NC

Introduction to Kerberos
⬢ “Kerberos is a network authentication protocol. It is designed to provide strong
authentication for client/server applications by using secret-key cryptography” [1]
⬢ MIT Kerberos is one implementation
– Heimdal is another
– We’re talking about MIT Kerberos
⬢ Authentication over a computer network
– Not authorization
– No data privacy
1: http://web.mit.edu/kerberos/

⬢ Key Distribution Center (KDC)
– Centralized server which grants Kerberos “tickets”
– The “trusted third party” of the security model
⬢ Users are defined by a ”principal”
– primary[/instance]@REALM
– A human: elserj@HORTONWORKS.COM
– A service: accumulo/tserver1.hortonworks.com@HORTONWORKS.COM
– elserj@HORTONWORKS.COM is unique with elserj/login.accumulo.hwx.com@HORTONWORKS.COM

⬢Principals are identified by a secret shared with the KDC
– A normal password
– A keytab file (non-plaintext “password”, suitable for non-interactive logins)
⬢ Kerberos Ticket obtained from the KDC by using your secret
– Tickets expire
– Tickets are renewable*
Client Server
KDC
Password/Keytab Keytab
Authenticated RPC

Interacting with Kerberos
⬢ kadmin (or kadmin.local)
– Command-line interface for administrators to create, modify, delete principals.
⬢ kinit
– A command-line tool to obtain a ticket for a principal
– Places the ticket in a file on disk in a well-known location called a “ticket cache”
• Default location on Linux: /tmp/krb5cc_$(id –u `whoami`)
– The ticket cache is read-write protected for the user only (e.g. chmod 600)
– Can obtain a ticket for any principal using a password or keytab
– Ticket caches can hold multiple tickets
⬢ klist
– Lists the contents of the current user’s ticket cache
– Can list the keys in a keytab file

Benefits of Kerberos
⬢ Building a secure, network-based authentication system is very hard
⬢ Functions on non-trusted networks
– Security for multi-tenant systems, protect against malicious and non-malicious users
⬢ Leveraged across the Apache Hadoop “Stack”
⬢ Widely integrated externally
– Operating systems and programming languages
⬢Can integrate with Active Directory
Apache Hadoop is a registered trademark of the Apache Software Foundation

1
0
© Hortonworks Inc. 2011 – 2017. All Rights Reserved
Promises
It’s simple: you get your Kerberos ticket, use Accumulo, and it knows who you are!
[elserj@localhost] $ kinit elserj
Password for elserj@HORTONWORKS.COM:
[elserj@localhost] $ accumulo com.hortonworks.accumulo.MyMapReduceJob
…
Success!
[elserj@localhost] $
( ^ _ ^ )

1
1
Reality
[elserj@localhost] $ kinit elserj
Password for elserj@HORTONWORKS.COM:
[elserj@localhost] $ accumulo com.hortonworks.accumulo.MyMapReduceJob
...
2016-10-16 14:03:11,549 [security.UserGroupInformation] ERROR:
PrivilegedActionException as:accumulo/server.com (auth:KERBEROS)
cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by
GSSException: No valid credentials provided (Mechanism level: Failed to
find any Kerberos tgt)]
[elserj@localhost] $
(╯°□°）╯︵ ┻━┻

1
2
Ok, let’s figure out what went wrong?
What should I search for?
Thrift
SASL
GSSAPI
JGSSUGI
JAAS
KDC
JCE
Token
Ticket
Voldemort
“Bars near me
open now”
Cthulhu
Kerberos

1
3
How JVM-based applications can obtain Kerberos tickets
⬢ Extract a ticket from the local ticket cache for a principal
– hbase shell or hdfs dfs –ls /
⬢ UserGroupInformation Hadoop API (UGI)
– UserGroupInformation.loginUserFromKeytab(String, String)
– UserGroupInformation.loginUserFromKeytabAndReturnUGI(String, String)
⬢ javax.security.auth.Subject with Krb5LoginModule
– The APIs which UserGroupInformation uses under the covers
⬢ Automatic login via JAAS
– “Java Authentication and Authorization Service”, implementation of PAM (RFC 86.0)
– Configuration file, specified via Java system properties.
– Each “block” uses an identifier to denote login details for a specific system

1
4
Accumulo Service Logins
⬢ Accumulo services are daemons; they always use a keytab to login
⬢ Principal and keytab are specified in accumulo-site.xml for each service
⬢ A JAAS configuration file can be provided for Apache ZooKeeper client authentication
– Accumulo currently uses ZooKeeper DIGEST authentication, not SASL
⬢ Accumulo services automatically perform logins/renewals as necessary
– Anyone who tells you that they need to ”kinit for Accumulo to work” doesn’t know what they’re
talking about.
Apache ZooKeeper is a trademark of the Apache Software Foundation

1
5
Accumulo Clients
⬢ Accumulo clients will use a variety of mechanism for authentication
– Interactive use: ticket-cache
– Automated tasks/Daemons: UGI with keytab
⬢ Reminder: Kerberos tickets expire
– Clients must implement renewal logic
– UGI provides an API to do this
⬢ Typically, UGI is the way to go
–Concise and well-understood

1
6
On using UserGroupInformation correctly
⬢ We mentioned two different method calls earlier for logins
– void loginUserFromKeytab(String, String)
– UserGroupInformation loginUserFromKeytabAndReturnUGI(String, String)
⬢ loginUserFromKeytab is “global”
– Syntactic-sugar to make your life easier
– Works great when the application only acts as one user
⬢ loginUserFromKeytabAndReturnUGI is “localized”
– Requires invoking “doAs(...)”
– Allows for concurrent execution as different users in one JVM
⬢ KerberosToken tries to encapsulate this logic
– Better to own it yourself

1
7
Enter SASL: authentication framework over a transport
⬢ SASL is a framework for building RPC systems with authentication
⬢ “Simple Authentication and Security Layer” RFC-4422
– “A framework for authentication and data security in Internet protocols” [1]
– “decouples authentication mechanisms from application protocols”[1]
• Generic Security Services Application Program Interface (GSSAPI) speaks Kerberos
• DIGEST-MD5 an HTTP Digest authentication-like method (“delegation tokens”)
– Data security aka Quality of Protection (QoP)
• auth: Authentication only (default)
• auth-int: Previous, and integrity check of message content
• auth-conf: Previous, and encryption of message content
[1] https://en.wikipedia.org/wiki/Simple_Authentication_and_Security_Layer

1
8
Trust on an untrusted network
⬢ A Kerberos ticket implies a valid identity, not necessarily the identity you wanted
⬢ Kerberos relies on accurate/consistent DNS as the basis for a secure RPC model
– Secure your DNS as much as your KDC
⬢ Recall the service principal from earlier
– accumulo/tabletserver1.accumulo.hwx.com@HORTONWORKS.COM
⬢ The instance must be a fully-qualified domain name
⬢ Clients need to know primary and instance must match DNS
– “Caused by: KrbException: Identifier doesn't match expected value (906)”
– “error Message is Server not found in Kerberos database”

1
9
Trust on an untrusted network
Client Trusted ServiceGood
DNS
Rogue Service
Bad
DNS
service/svc1.hwx.com@HORTONWORKS.COMservice/svc18.hwx.com@HORTONWORKS.COM
Sends RPC “service” at
svc1.hwx.com
Without enforcement of DNS naming via SASL, a
client could be maliciously sent to a rogue service
without the client realizing it happened.

2
0
Harping on DNS
⬢ DNS must be correct, consistent, and secure
⬢ Hostnames are advertised for discovery
– Also benefits multi-homed networks
⬢ Forward and Reverse DNS mappings must be accurate on every node
– `nslookup tabletserver1.accumulo.hwx.com` returns 10.0.0.1
– `nslookup 10.0.0.1` returns tabletserver1.accumulo.hwx.com
⬢Check /etc/resolv.conf for quick troubleshooting

2
1
Recap: Kerberos authentication for Accumulo RPCs
⬢ Client and Server both obtain Kerberos ticket
– Password or Keytab via UGI/JAAS
– Tickets must be renewed before they expire
⬢ Accumulo leverages Thrift for RPCs
⬢ SASL is the framework which Thrift leverages for Kerberos-authenticated RPCs
– GSSAPI as the SASL mechanism which can “speak” Kerberos
– QoP defines the security of the RPC data (minimum of authentication)
⬢ Fully-qualified hostnames everywhere
– Forward and reverse DNS must be consistent across all clients and servers

2
2
The edge cases
⬢ Exceptions to how authentication works
– YARN jobs
– Accumulo Thrift Proxy Server
⬢ Not the traditional client/server model Kerberos was designed to fit
– 100-1000’s of tasks concurrently requiring a ticket
– Talk to Accumulo as a user without having that user’s credentials
⬢Two approaches introduced to address these problems

2
3
Delegation Tokens
⬢ Earlier mentioned, SASL supports a variety of mechanisms
– DIGEST-MD5 allows a digest-token style authentication scheme
⬢ Delegation token is a temporary ”password” which can authenticate a user
– Slight compromise of security for performance
⬢ Circumvents authentication to the KDC, instead handled by HDFS or Accumulo
⬢ Automatically obtained during job submission and added to the job cache
– We must rely on YARN to do the right thing
If you thought Kerberos documentation for Hadoop/Accumulo was sparse…

2
4
Delegation Tokens
Client
Accumulo
Master
KDCPassword/Keytab Keytab
Obtain DT
YARN
Containers
Accumulo
TabletServers
YARN
ResourceManager
Client Ticket
and DT YARN Ticket
and DT
DT

2
5
Proxy Users
⬢ Proxy: a service that provides access to another backend service
⬢ The proxy has its own Kerberos principal and keytab used to communicate with
Accumulo
⬢ These services are accessing Accumulo on behalf of another user.
– The ticket is for the service, but we want it to appear as if it is elserj@HORTONWORKS.COM
⬢ Defined for Accumulo in accumulo-site.xml
– instance.rpc.sasl.allowed.user.impersonation
– instance.rpc.sasl.allowed.host.impersonation
⬢ Configuration-based approach to allow services to “pretend” to be a user without
actually having that user’s credentials

2
6
Proxy Users
Client
KDC
Password/Keytab
AccumuloProxy Server
Client Ticket
Server Ticket
(Client principal)
Keytab
Keytab

2
7
Kerberos authentication for HTTP-based services (SPNEGO)
⬢ Not yet implemented in Accumulo Monitor
⬢ The need to protect services using HTTP
–Don’t want to reuse SASL
⬢ Simple and Protected GSSAPI Negotiation Mechanism (SPNEGO) RFC-4178
– The Negotiate HTTP header
– Built into cURL (--negotiate), most Java-based HTTP libraries, and web-browsers
⬢ Web-browsers often need special configuration to properly authenticate.
– Firefox: network.negotiate-auth.delegation-uris, network.negotiate-auth.trusted-uris
– Chrome: --auth-server-whitelist="*.domain" --auth-negotiate-delegate-whitelist="*.domain"

2
8
Troubleshooting: Prerequisites
⬢ Ensure a recent version of your JVM and Hadoop
– Bugs exist in UserGroupInformation for certain JVMs (vendor+version)
⬢ Ensure that the unlimited strength Java Cryptographic Extensions (JCE) are installed on
all nodes in the cluster
– And that clients/servers are using that JVM installation!
– Required for AES-256 encryption type on Kerberos keys (which you will likely get by default)
⬢ Ensure that you have DEBUG or TRACE logging enabled
– Server package: org.apache.accumulo.server.rpc
– Clients package: org.apache.accumulo.core.client.impl
⬢ Set the sun.security.krb5.debug system property to true in your application

2
9
Troubleshooting: Tips
⬢ Remember that DNS is the cornerstone
– When reading logs, make sure that you see the expected fully-qualified domain names
– Do not assume that DNS is correct: verify it.
⬢ Determine if an RPC issue is authentication or authorization
– If you see an Accumulo-level error, it is likely an authorization issue
– If you only see transport/connection-setup errors, it is likely an authentication issue
⬢ Remember that tickets expire
– Cross-reference ticket lifetimes with application logs
⬢ Read the logs.
⬢ No. Actually read them.
– A vast majority of errors can be solved with appropriate logging JVM-debugging

3
0
Reference Material
⬢ “Hadoop and Kerberos: The Madness beyond the Gate”
– https://steveloughran.gitbooks.io/kerberos_and_hadoop/content/index.html
⬢ Oracle documentation
– http://docs.oracle.com/javase/7/docs/technotes/guides/security/jaas/tutorials/GeneralAcnOnly.html
– https://docs.oracle.com/javase/7/docs/jre/api/security/jaas/spec/com/sun/security/auth/module/Krb5
LoginModule.html
⬢ MIT Kerberos documentation
– http://web.mit.edu/kerberos/
⬢ “Explain like I’m 5: Kerberos” (great low-level Kerberos write-up)
– http://www.roguelynn.com/words/explain-like-im-5-kerberos/
⬢KDiag: “Kerberos diagnostics for Hadoop”
–Apache Hadoop >=2.8 or https://github.com/steveloughran/kdiag

3
1
Developing with Kerberos
⬢ Apache Directory’s Kerby project
– Great for Kerberos authentication without Hadoop in the picture
– http://directory.apache.org/kerby/downloads.html
⬢ Apache Hadoop’s MiniKDC
– Built on top of Apache Directory
– https://github.com/apache/hadoop/blob/release-2.7.3-RC2/hadoop-common-project/hadoop-
minikdc/src/main/java/org/apache/hadoop/minikdc/MiniKdc.java
⬢ Can be used with HDFS, YARN, and Accumulo MiniClusters!
Kerberos is not an excuse to skip testing!
Apache Directory is a trademark of the Apache Software Foundation

3
2
© Hortonworks Inc. 2011 – 2017. All Rights Reserved3
2
Thanks!
Email: elserj@apache.org
Twitter: @joshelser
3dom via https://www.flickr.com/photos/steve_l/6674480535/in/album-72157629289333057/, CC-BY-NC
Thanks to those who gave feedback along the way: Brandon Wilson, Bryan Bende,
Michael Stack, Randy Gelhausen, Steve Loughran.

Practical Kerberos

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Practical Kerberos

Similar to Practical Kerberos (20)

Recently uploaded

Recently uploaded (20)

Practical Kerberos