In this session I will share our experience of deploying Secure NFS throughout a large enterprise with hundreds of NFS servers and thousands of hosts. I will be talking about challenges of managing Kerberos credentials for thousands of non-interactive service accounts running a multitude of business applications. I will explore how we are overcoming these hurdles and our approach to securing the NFS protocol.
5. 4
Updated thousands of hosts to support Secure NFS
Migrated thousands of users' home directories
Migrated hundreds of applications
Up to now
This is a journey – we are not yet done
12. 11
It works
Unified Name Space is the biggest initial hurdle
Must have Kerberos well established and understood
We need a better way to provide non-interactive users with credentials.
Secure NFS
Recap
How are we / Why are we doing it?
Thousands of hosts
Thousands NFS shares
Hundreds of filers
Thousands of users
Thousands of applications
Front office / back office / infrastructure / everything
- NFS is everywhere
- NFS v3 with auth_sys is insecure
- nothing new, we last tried introducing Secure NFS 15 years ago
Auditors are your friends – they just see the world through a different lens
Key Reasons:
auth_sys is weak, trusts the client hosts, clients can no longer be trusted
introduce authentication
alternatives CIFS, AFS, local (SAN) -> all dismissed
decided to leverage the existing Kerberos infrastructure and attempt Secure NFS again
- we’re lucky. We started doing Kerberos 10 years ago
- single realm across the globe, stringed together with our global name space
- this might be the biggest obstacle for a lot of people, adopting Kerberos and it prevents us from using Secure NFS everywhere!
- AD integration is a good option for many today. Good commercial products available.
- more on this later
- security mode is not visible. Security Negotiation takes they guessing out of it.
- SECINFO call in protocol
- badly implemented. Solaris was the only one when we started. As of RHEL 6.4 it is ok as well. NetApp implementation had bugs but better in Clustered Data ONTAP. FreeBSD doesn’t seem to have it implemented.
- it would make adoption so much easier if it was just there, no adding of -sec=krb5 in all the automounter maps
- Data ONTAP 7-mode only supports DES. weak cipher. bad. 3DES available in clustered Data ONTAP and AES next year.
- credential cache location.
- fixed in Solaris, SunSSH accounts for it.
- not fixed in Linux, OpenSSH uses mktemp(). rpc.gssd goes looking for it. buggy.
-> looking for the cc with the latest time stampe != valid cc!
- rpc.gssd is fix in FreeBSD but SSH creates it using mktemp() ...
needs a root/host.name principal
RHEL 5 only uses nfs/host.name in root context, the user root is always mapped to the user nfs! Access as root is not possible from RHEL 5
User Home Directories - Are easy
They all have credentials to start with
Logging in with Kerberos password, creates a cc
Users adapt
krenew comes in handy to long running jobs
or cron/at with ‘kinit -R’
- are hard
- they don’t have credentials. They don’t log in.
- how to give them credentials?
- no standard
- Microsoft got that right
- there is too many ways to do it in UNIX
- too many ways to authenticate users to start with! A lot of which do not result in a CC
- derive a credential cache from a keytab
- how to get a keytab in the first place when you have 100’000 to deploy?!
- problems of keytabs as long living credential? How to mange/rotate it?
- Kharon solves the keytab problem.
- uses the host keytab to get valid CC for the applications
- S4U - Microsoft Constrained Delegation