IBM Spectrum Scale Authentication for File Access - Deep Dive

IBM Spectrum Scale™
Authentication (for File Access)
Shradha Thakare
Spectrum Scale Auth Development
Kaustubh Katruwar
Lead, Spectrum Scale Auth Development

Acknowledgement
• Christof Schmitt
• Sandeep Patil
• Chetan Kulkarni
• Varun Mittal
2

Agenda and Flow
• Overview of Authentication and ID Mapping
• Authentication and ID Mapping in IBM Spectrum Scale™
• Administration
• Problem Determination Guide
• Question and Answers
3

Agenda and Flow
• Administration
4

Basics of Authentication
What is authentication?
• The objective of authentication is to verify the claimed identity of users and components.
• Usually it is done by having the user enter a valid user name and credential before access
is granted.
What is UNIX Authentication?
• A user uses their user name and a credential (such as password, private SSH key) to log
on to a UNIX workstation.
• The workstation looks up the user’s UID in local files or an external directory service such
as LDAP and then verifies the received credential.
• After a user is successfully logged on to a UNIX server, they are trusted on this server and
also by all other servers that trust this server.
What is Windows Authentication?
• A user logs on using their user name to log on to its Windows workstation.
• The workstation looks up the user's SID in the local Windows registry or on the Windows
domain controller and then verifies the received credential.
• After a user is successfully logged on to a Windows server, that user is trusted on this
server. 5
Overview of Authentication and ID Mapping

Identification
• The objective of identification is to identify users and infrastructure components. Identification methods include
unique user IDs (such as different persons use different user IDs).
• Other methods include keys and finger prints (such as a public ssh key) and digital certificates (such as a certificate
of the web server).
User names and user IDs (UIDs):
• UNIX system and UNIX based appliances use user names and user identifiers (UID) to represent users of the
system.
• The user name is typically a human readable sequence of alphanumeric characters and the UID is a positive integer
value.
• When a user logs on to a UNIX system, the operating systems looks up their UID and then uses this UID for further
representation of the user.
• User names, UIDs, and the mapping of user names to UIDs are stored locally in the /etc/passwd file. Or, they can be
stored on an external directory service such as AD, LDAP, or NIS.
Group names and Group IDs (GIDs)
• UNIX systems use groups to maintain sets of users which have the same permissions to access certain system
resources.
• Similar to user names and UIDs, a UNIX system also maintains group names and group identifiers (GIDs).
• A UNIX user might be a member of one or more groups, where one group is the primary or default group.
• Group names, GIDs, the mapping of group names to GIDs, and the member ships of users to groups are stored
locally in the /etc/group file. Or they can be stored on an external directory service such as AD, LDAP, or NIS. 6

Directory Service
Directory services allows to store and maintain user and group information centrally on an external server.
Microsoft Active Directory (AD) - Active Directory (AD) is a Microsoft created technology introduced from Windows
2000 onwards. It provides the following key network services:
• Directory Service
• Kerberos based authentication
• DNS naming
Services for UNIX (SFU) - Services for UNIX (SFU) is a Microsoft Windows component that provides interoperability
between Microsoft Windows and UNIX environments.
Lightweight Directory Access Protocol (LDAP) - Lightweight Directory Access Protocol (LDAP) is a directory service
access protocol over a TCP/IP network. LDAP is a lightweight alternative to traditional Directory Access Protocol (DAP),
therefore it is called LDAP.
An LDAP directory is usually structured hierarchically, as a tree of nodes. Each node represents an “entry” within the
LDAP database. A single LDAP entry can consist of multiple attribute-value pairs, and is uniquely identified by a
distinguished name.
Network Information Service (NIS) - Network Information Service is a directory service protocol for centrally storing
configuration data of a computer network. NIS protocols and commands were originally defined by Sun Microsystems,
but the service is now widely implemented. Originally called Yellow Pages or YP, some of the binary names still start
with “yp”.
The NIS information is stored in NIS maps, typically providing the following information:
- Password-related data similar to data stored in /etc/passwd
- Group related data similar to data stored in /etc/group
- Network configuration such as netgroups
7

Other Authentication elements for file access
• Netgroups: Groups of hosts are used to restrict access for mounting NFS
exports on a set of hosts, and deny mounting on the remainder of the
hosts. The IBM Spectrum Scale™ system supports only the netgroups that
are stored in NIS and in Lightweight Directory Access Protocol (LDAP).
• Kerberos: Kerberos is a network authentication protocol client/server
applications that provides secured communication by ensuring passwords
are not sent over the network to the system. It uses symmetric key
cryptography. The IBM Spectrum Scale™ system supports Kerberos with
both AD and LDAP-based authentication.
• Transport Level Security (TLS): The TLS protocol is primarily used to
increase the security and integrity of data that is sent over the network.
These protocols are based on public key cryptography and use digital
certificates based on X.509 for identification.
8

Integrating UNIX and Windows authentication
• Typically Windows authentication is used for incoming CIFS
connections and UNIX authentication for incoming NFS, HTTP, SFTP,
and SCP connections.
• To provide heterogeneous file sharing for UNIX and Windows the
system should support authentication methods for UNIX and
Windows.
• For Heterogeneous file access authentication servers like AD +
RFC2307, LDAP are used.
9

ID mapping
• As seen earlier the authentication of the user or groups of users is
associated with the identification of their unique identifiers.
• To support data access to Microsoft Windows clients (SMB protocol)
and to allow interoperability, that is, to share data among UNIX and
Windows clients (SMB and NFS protocols), the system must map
Windows SID to UNIX UID/GID.
• This process is referred to as ID mapping and the map is referred to
as ID map.
• The ID mapping can be done either internally in the system or in an
external authentication server.
10

Agenda and Flow
• Administration
11

Authentication in IBM Spectrum Scale™
• To enable read and write access to directories and files for the users on
the IBM Spectrum Scale™ system, you must configure user authentication
on the system.
• Only one user authentication method, and only one instance of that
method, can be supported.
• The following authentication services can be configured with the IBM
Spectrum Scale™ system for file protocol access:
• Microsoft Active Directory (AD)
• Lightweight Directory Access Protocol (LDAP)
• Network Information Service (NIS) for NFS client access
• User defined
12
Authentication and ID Mapping in IBM Spectrum Scale™

Authentication for Protocols
• Authentication for NFS/SMB/Object involves user credential validation and user identity management which
helps define ownership of data and is the foundational base for ACL for NFS and SMB
• Figure below shows the Flow of Authentication for File based protocols.
13
IBM Spectrum Scale CES Node Serving The Protocols (NFS and SMB)
User
Directory server
•Active Directory
•LDAP servers
3. Get user/group ID MAP
2.Validate user
4.Grant Access
user
ID Map server
•NIS
•LDAP
•RFC 2307
• NAS (internal – auto
generated)
1. Connect w/
credentials

Authentication considerations for File access
The following steps are involved in the user authentication for file
access:
• User tries to connect to the IBM Spectrum Scale™ system by using
their credentials.
• The IBM Spectrum Scale™ system contacts the authentication server
to validate the user.
• The IBM Spectrum Scale™ system contacts the ID map server that
provides UIDs and GIDs of the user and user group to verify the
identity of the user.
• If the user credentials are valid, the user gains access to the system.
14

High Level Component Diagram
15

Components that will be used for Authentication
16
SSSD
winbind
Keystone
(with LDAP/AD
middleware)
Auth Component for LDAP Authentication and respective ID mapping
(Component from Linux Distro) – Used by FILE (NFS/SMB)
Auth component for proxying request to NIS component
Auth Component for AD Authentication and respective ID mapping
(Component from SAMBA) ) – Used by FILE (NFS/SMB)
Auth Component for Object Authentication and Token Management
ypbind
Auth Component for NIS used by NFS protocol for ID Mapping
(Component from Linux Distro)

UID/GID/SID mapping in IBM Spectrum Scale™
• IBM Spectrum Scale™ stores all user data on GPFS file systems, which use UIDs
and GIDs for access control.
• For CIFS access, IBM Spectrum Scale™ needs to map SIDs to UIDs and GIDs to
enforce access control. NFSv3 auth sys clients send the UID and GID of a user who
requests access to a file.
• IBM Spectrum Scale™ uses Linux default access control mechanism by comparing
the received UID and GID with the UIDs and GIDs stored in GPFS.
• The UIDs and GIDs used by the NFS clients must match the UIDs and UIDs stored
inside GPFS.
17

ID Mapping methods in IBM Spectrum Scale™
The following methods are used to map Windows SID to UNIX UID and GID:
• External ID mapping methods
• A UID or GID of a user or group is created and stored in an external server.
• The external server administrator is responsible for creating or populating the UID/GID for the user/group in
their respective servers.
• The IBM Spectrum Scale™ system supports the following servers for external ID mapping:
• LDAP server, where the UID or GID is stored in a dedicated field in the user or group object on the LDAP server.
• AD server with RFC2307 schema extension defined. The UID or GID of a user or group that is defined in AD server is stored in a
dedicated field of the user or group object.
• Internal ID mapping method
• Automatic ID mapping when AD-based authentication is used.
• Automatic ID mapping method uses a reserved ID range to allocate ID based on the following logic.
• A user or group in AD is identified by SID, which includes a component that is called RID. Whenever a user or group from an AD
domain accesses IBM Spectrum Scale™, a range is allocated per AD domain. UID or GID is then allocated depending upon this
range and the RID of the user/group.
• For Example: S-1-5-21-3922795712-4076380459-2191511802-1304. Here:
• S – String is SID
• 1 – Revision Level
• 5 – Identifier Authority Value
• 21-3922795712-4076380459-2191511802 – Domain or Local Computer Identifier
• 1304 – Relative ID (RID)
18

Authentication + ID map schemes Support Matrix
19
Configuring Authentication with Plain AD
Configuring Authentication with NIS

Authentication + ID map schemes Support
Matrix
Configuring Authentication with LDAP
20

Authentication + ID map schemes Support Matrix
Note:
• The ID mapping option that is given in this table is only applicable for file
access. Ignore the ID mapping details if you are looking for the supported
configurations for object access.
• In User-defined mode, customer is free to choose the authentication and
ID mapping methods for file and object and manage on their own. That is,
the authentication needs to be configured by the administrator outside of
the IBM Spectrum Scale™ commands and ensure that it is common and
consistent across the cluster.
• NFS V3 with SMB protocol access on the same export (multiprotocol
access) is not supported.
21

Prerequisites for configuring Authentication
• Before you start configuring authentication for protocol access, the system administrator
needs to ensure that the authentication server is set up properly and the connection
between the IBM Spectrum Scale™ system and authentication server is established
properly.
• Depending on the requirement, the IBM Spectrum Scale™ system administrator needs to
set up the following servers:
• Microsoft Active Directory (AD) for file and object access
• Lightweight Directory Access Protocol server for file and object access
• Keystone server to configure local, AD, or LDAP-based authentication for object access.
Configuring Keystone is a mandatory requirement if you need to have Object access.
• Ensure the server details such as IP address or host name, admin user name, password,
base dn, and user dn are known.
• ID map role of the system must be identified. For stand alone systems the role would be
as “master”. In case of AFM-based deployments, one of them will be the “master” while
the rest will be “subordinate”
Note: The ID map role of master and subordinate systems are the following:
Master: System creates ID maps on its own.
Subordinate: System does not create ID maps on its own. ID maps must be exported
from the master to the subordinate. 22

Understanding AD-based authentication
• IBM Spectrum Scale™ can be configured with AD Based authentication.
• You can configure AD-based authentication with the following ID mapping methods:
• Automatic - This method is typically used if you have SMB only access and do not plan to deploy multiprotocol access.
• RFC2307- Supports multiprotocol access.
• LDAP - Supports multiprotocol access.
• AD admin user ID and password is required to join the IBM Spectrum Scale system as machine account into
the AD domain.
• During the AD join process, a computer account having the same name as the netbios name is searched
within the AD domain that will be joined. If the name is not found, a new computer entry is created in the
standard location (CN=Computers). If the user chooses to pre-create computer accounts for IBM Spectrum
Scale™ in the AD domain within a particular organizational unit, the computer account must be created with
a valid name and it must be passed as the netbios name while configuring the IBM Spectrum Scale™ system.
• The Authentication configuration provides a mechanism to set the UID and GID range. By default, it is set to
10000000-299999999. However, you can choose the minimum and maximum range value based on your
setup. The minimum range should be greater than 1000 to avoid an overlap of IDs used by end users,
administrative users, and operating system component users of the IBM Spectrum Scale™ system.
• Range size determines the total no. of UIDs and GIDs that are assigned per domain. For example, if range is
10000000-299999999 and range size is 1000000, then 290 domains are allowed. Choose a value for range
size that allows for the highest anticipated RID value among all of the anticipated AD users and AD groups in
all of the anticipated AD domains. Choose the range size value carefully because range size cannot be
changed after the first AD domain is defined on the IBM Spectrum Scale™ system.
23

Understanding AD-based authentication
Important:
• To achieve high-availability, you can configure multiple AD domain
controllers.
• While configuring AD-based authentication, you do not need to specify
multiple AD servers in the command line to achieve high-availability.
• The IBM Spectrum Scale™ system queries the specified AD server for
relevant details and configures itself for the AD-based authentication.
• The IBM Spectrum Scale™ system relies on the DNS server to identify the
set of available AD servers that are currently available in the environment
serving the same domain system.
24

Prerequisites for AD based Authentication
• User credentials with domain join privileges to join the Spectrum
Scale to AD domain should be available.
• Define the ID map range and size depending upon the maximum RID
(sum of allocated and expected growth).
• Ensure that range size (max range / min range) is defined such that at
least three domains can be mapped.
• Primary DNS is added in the /etc/resolv.conf file on all the protocol
nodes. It resolves the authentication server system with which
the IBM Spectrum Scale™ system is configured.
• The ID map role description
25

Understanding AD with RFC2307 Schema based ID
Mapping
• You can configure IBM Spectrum Scale™ system authentication with
Active Directory (AD) and RFC2307
and
Active Directory (AD) with Kerberized NFS and RFC2307 ID mapping.
• This is useful when you are planning multiprotocol access.
• In these authentication methods, Active Directory is used to store user credentials and UIDs and
GIDs are stored in the UNIX attributes that are populated in the RFC2307 schema extension of AD
server.
• Here we need to specify the UNIX Domain Map for those AD Domains for which ID mapping using
RFC2307 are to be configured along with the ID Map range.
• Any users or groups, from this domain, with missing UID/GID attributes or if not in specified
range are denied access.
26

Prerequisite for AD + RFC2307
• Define the ID Map range size based on the values entered in the UNIX
Attributes tab.
• Also need to make sure that the User’s Primary group has a valid GID
in the UNIX Attribute of the RFC2307 schema.
• The specified range should not intersect with the range specified by
using the --idmap-range option of the command.
• Using UIDs and GIDs greater than 1000 can avoid an overlap of IDs
used by end users, administrative users, and operating system
component users of the IBM Spectrum Scale™ system.
27

Understanding AD with LDAP ID mapping
• AD authentication with LDAP ID mapping provides a way for IBM
Spectrum Scale™ to read ID mappings from an LDAP server as defined
in RFC 2307.
• The LDAP server must be a stand-alone LDAP server.
• Need to specify the Domain name for which ID mapping service is to
be configured using the ldapmap-domains option along with the
range of ID Map.
• If user’s UID or GID is not within the range, access will be denied.
28

Prerequisite for AD + LDAP ID mapping
• Define the ID Map range size based on the values in the LDAP server.
• The specified range must not intersect with the --idmap-range of the
command.
• Using UIDs and GIDs greater than 1000 can avoid an overlap of IDs
used by end users, administrative users, and operating system
component users of the IBM Spectrum Scale™ system.
29

Understanding LDAP-based authentication
• Using LDAP-based authentication can be useful when you use an external LDAP server to store
user information and user passwords.
• In this authentication method, you can use LDAP as the authentication as well as the ID mapping
server for both NFS and SMB.
• Appropriate SMB schema needs to be uploaded in the LDAP if you plan to have SMB access.
• Based on the level of security, the following configurations are possible:
• LDAP with TLS
• LDAP with Kerberos
• LDAP with TLS and Kerberos
• LDAP
• TLS secures the communication between the IBM Spectrum Scale™ system and the LDAP server
• Kerberos provides secured communication by ensuring passwords are not sent over the network
to the system.
30

Prerequisites for LDAP-based authentication
• The LDAP server must already be configured.
• Enable TLS encryption on the LDAP server, if you need to secure
communication between the IBM Spectrum Scale™ system and LDAP
server. Details on configuring SSL or TLS encryption on the server can
be obtained from the OpenLDAP Administrator's Guide.
• To access SMB shares, LDAP user information must be updated with
unique Samba attributes in addition to the attributes that are stored
for a normal LDAP user. Ensure that these required Samba attributes
are present in the LDAP user entries.
31

Ensure that you have the following details before configuring LDAP based authentication:
• Domain details such as base dn, and dn prefixes of groups and users, else default values are used.
Default user group suffix is <ou=Groups, <base dn> and default user suffix
is ou=People, <base dn>.
• IP address or host name of LDAP server.
• Admin user ID and password of LDAP server that is used during LDAP simple bind and for LDAP
searches.
• The secret key to be provided for encrypting/decrypting passwords unless you have disabled
prompting for the key.
• NetBIOS name that is to be assigned for the IBM Spectrum Scale™ system.
• If you need to have secure communication between the IBM Spectrum Scale™ system and LDAP,
the CA signed certificate that is used by the LDAP server for TLS communication must be placed at
the specified location in the system.
• If you are using LDAP with Kerberos, create a Kerberos keytab file by using the MIT KDC
infrastructure.
• Primary DNS is added in the /etc/resolv.conf file on all the protocol nodes. It resolves the
authentication server system with which the IBM Spectrum Scale™ system is configured.
32
Prerequisites for LDAP-based authentication

Updating LDAP user information with Samba
attributes
• If you need to support SMB data access, LDAP schema must be extended to store more
attributes such as SID, Windows password hash to the POSIX user object. To use Samba accounts,
update LDAP user information with unique Samba attributes.
• The following sample LDIF file shows the minimum required samba attributes:
dn: cn=SMBuser,ou=People,dc=ibm,dc=com
changetype: modify
add : objectClass
objectClass: sambaSamAccount
-
add: sambaSID
sambaSID: S-1-5-21-1528920847-3529959213-2931869277-1102
-
add:sambaPasswordHistory
sambaPasswordHistory:
00000000000000000000000000000000000000000000000000000000
-
add:sambaNTPassword
sambaNTPassword: (valid samba password hash )
-
add:sambaPwdLastSet
sambaPwdLastSet: 1263386096
-
add:SambaAcctFlags
sambaAcctFlags: [U ]
33

attributes
Perform the following steps to create the values for sambaNTPassword, sambaPwdLastSet,
and SambaAcctFlags, which must be generated from a PERL module:
1. Download the module from http://search.cpan.org/~bjkuit/Crypt-SmbHash-0.12/SmbHash.pm. Create and
install the module by following the readme file.
2. Use the following PERL script to generate the LM and NT password hashes:
# cat /tmp/Crypt-SmbHash-0.12/gen_hash.pl
#!/usr/local/bin/perl
use Crypt::SmbHash;
$username = $ARGV[0];
$password = $ARGV[1];
if ( !$password ) {
print "Not enough argumentsn";
print "Usage: $0 username passwordn";
exit 1;
}
$uid = (getpwnam($username))[2];
my ($login,undef,$uid) = getpwnam($ARGV[0]);
ntlmgen $password, $lm, $nt;
printf "%s:%d:%s:%s:[%-11s]:LCT-%08Xn", $login, $uid, $lm, $nt, "U", time;
3. Generate the password hashes for any user as in the following example for the user test01:
# perl gen_hash.pl SMBuser test01
:0:47F9DBCCD37D6B40AAD3B435B51404EE:82E6D500C194BA5B9716495691FB7DD6:
[U ]:LCT-4C18B9FC
34

attributes
4. Use the information from step 3 to update the LDIF file in the format that
is provided in the example at the beginning of this topic.
• To generate the sambaPwdLastSet value, use the hexadecimal time value from step 3
after the dash character and convert it into decimal.
• A valid samba SID is required for a user to enable that user’s access to an IBM
Spectrum Scale™ share. To generate the samba SID, multiply the user's UID by 2 and
add 1000. The users's SID must contain the samba SID from the sambaDomainName,
which is either generated or picked up from the LDAP server, if it exists. The following
attributes for sambaDomainName LDIF entry are required:
dn: sambaDomainName=<IBM Spectrum Scale Netbios Name>,dc=ibm,dc=com
sambaDomainName: <IBM Spectrum Scale Netbios name>
sambaSID: S-1-5-21-1528920847-3529959213-2931869277
sambaPwdHistoryLength: 0
sambaMaxPwdAge: -1
sambaMinPwdAge: 0
35

attributes
This LDIF entry can be created by the LDAP server administrator by using either of the
following two methods:
A. Write and run a bash script similar to the following example:
sambaSID=
for num in 1 2 3 ;do
randNum=$(od -vAn -N4 -tu4 < /dev/urandom | sed -e 's/ //g')
if [ -z "$sambaSID" ];then
sambaSID="S-1-5-21-$randNum“
else
sambaSID="${sambaSID}-$ {randNum}“
fi
done
echo $sambaSID
Then, use the samba SID generated to create the LDIF file. The sambaDomainName must
match the IBM Spectrum Scale™ system Netbios name.
B. When you run the mmuserauth service create command or configure auth via GUI, the
system creates the sambaDomainName, if it does not exist.
C. The sambaSID for every user must have the following format: (samba SID for the domain)-
(userID*2+1000). For example: S-1-5-21-1528920847-3529959213-2931869277-
1102
5. Submit the ldapmodify command as shown in the following example to update the
user's information:
# ldapmodify -h localhost -D cn=Manager,dc=ibm,dc=com -W -x -f /tmp/samba_user.ldif
36

Prerequisite for LDAP with TLS for file access
• In addition to the pre-requisites for LDAP based authentication
scheme, Ensure that the CA certificate copied from the LDAP server is
placed under /var/mmfs/tmp directory with the
name ldap_cacert.pem; specifically, on the protocol node where the
command is run.
• For File LDAP TLS configuration; remember the Common Name for
your LDAP server which you set while creating Certificate. Clients
refer this Common Name while connecting to LDAP server over TLS.
You need to provide the same name to mmuserauth cli command
(otherwise File LDAP TLS Auth configuration fails).
37

Prerequisite for LDAP with Kerberos
1. In addition to the pre-requisites for LDAP based authentication scheme, Ensure that the keytab
file copied from the kerebos server is placed under the /var/mmfs/tmp directory with the
name as krb5.keytab on the node where the command is run.
2. Keytab file should contain CIFS and NFS service principal for enabling kerberized CIFS
and kerberized NFS access.
3. The service principal shoule be of the format
nfs/<NetBIOS name>@<REALM>
cifs/<NetBIOS name>@<REALM>
4. For example, if the NetBIOS name is FOO and the realm is EXAMPLE.COM, the service
principal name should be nfs/FOO@EXAMPLE.COM and cifs/FOO@EXAMPLE.COM
5. The NetBIOS name is the value specified for the option --netbios_name in the
mmuserauth command.
6. The realm name is the value specified for option --kerberos-realm in the
mmuserauth command.
38

Prerequisite for LDAP with TLS and Kerberos
Ensure that Pre-requisites for each are met:
1. LDAP based authentication scheme
2. LDAP with TLS based communciation
3. LDAP with Kerberos authentication
39

Prerequisite for Kerberos-based SMB access
• The time must be synchronized across the KDC server, the IBM Spectrum Scale™ cluster, and
the SMB clients, or else access to an SMB share could be denied.
• In MIT KDC configurations for the SMB services, the service principal name must use the
NetBIOS name and the realm name.
• The clients should use only the NetBIOS name when accessing an SMB share. Using any other
name or IP address might either cause a failure to connect or fallback to NTLM
authentication.
• With Active Directory KDC, you can use DNS alias (CNAME) for Kerberized SMB access. To use
the alias, you must register the DNS alias (CNAME) record for the NetBIOS name (system
account name) using the SetSPN tool available on Active Directory server. Not registering the
DNS alias record for the NetBIOS name might cause access to the SMB shares to be denied
with the error code, KDC_ERR_S_SPRINCIPAL_UNKNOWN.
• On Linux clients, to use Kerberized SMB access for IBM Spectrum Scale™ configured with MIT
KDC, you must at least have the 3.5.9 version of Samba client installed. The Linux clients
having an older Samba client might encounter the following error, while trying to access SMB
shares: ads_krb5_mk_req: krb5_get_credentials failed for foo$@KDC.COM (Server not found in
Kerberos database)
cli_session_setup_kerberos: spnego_gen_negTokenTarg failed: Server not found in Kerberos database
40

Understanding NIS Base Authentication
• The NIS-based authentication is useful in NFS only environment
where NIS acts as an ID mapping server and also used for netgroups.
• When file access is configured with NIS, SMB access cannot be
enabled.
41

Pre-requisite for NIS Based Authentication
The following details should be available:
• NIS server name. This is case-specific
• IP address or host name of the NIS server
• Primary DNS is added in the /etc/resolv.conf file on all the protocol
nodes. It resolves the authentication server system with which
the IBM Spectrum Scale™ system is configured.
42

Understanding User Defined Authentication
• The user is free to select the authentication and ID mapping methods
of their choice.
• It is the responsibility of the administrator of the client system to
manage the authentication and ID mapping for file (NFS and SMB)
access to the IBM Spectrum Scale™ system.
• The IBM Spectrum Scale system administrators are not allowed use
any of the GPFS™ commands to manage authentication.
• Refer following link for more details:
https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.3/com.ibm.spect
rum.scale.v4r23.doc/bl1adm_userdefinedauth.htm
43

Agenda and Flow
• Administration
44

Administration commands for Authentication
As discussed earlier, IBM Spectrum Scale™ can be configured with the
following external authentication servers for file access:
• Active Directory (AD)
• Light Weight Directory Access Protocol (LDAP)
• Network Information Service (NIS)
There are two methods available for setting up and also
Managing/Administering Authentication on the IBM Spectrum Scale™.
• Using CLI
• Installation toolkit
• In this session, we will look at administration using CLI.
45
Administration

Administration using the CLI commands
46

mmuserauth service Suite
• This command suite manages the authentication configuration of file and
object access protocols.
• The configuration allows protocol access methods to authenticate users
who need to access data that is stored on the system over these protocols.
• The different commands in the mmuserauth service suite are:
• mmuserauth service create - Configures authentication for file and object
access protocols.
• mmuserauth service list - Displays the details of the authentication
method that is configured for both file and object access protocols.
• mmuserauth service check - Verifies the authentication method
configuration details for file and object access protocols. Validates the connectivity to
the configured authentication servers. It also supports corrections to the
configuration details on the erroneously configured protocol nodes.
• mmuserauth service remove - Removes the authentication method
configuration of file and object access protocols and ID maps if any.
47
Administration

Life Cycle of Authentication CLI
48
Configure File and Object Auth
“mmuserauth service create”
Based on Auth type
- update the respective config
files
- restart the services
Check Auth config across the cluster
“mmuserauth service check”
Enable Required Protocols
List Configuration
(mmuserauth service list)
List File and Object Auth
config (separately)
Check if the authentication
is consistent across the
cluster (protocol nodes)
with an option for
rectification (optional)
•Allow Export Creation for
NFS/SMB only when auth configured
•Allow Object IO
Cleanup authentication
(mmuserauth service remove)
Start the protocols
For Object, when it is being enabled admin will
be prompted if he wants to use external keystone
or host internal keystone and likewise it will do keystone
initialization
Object and File auth config
have to be done separately
because of semantic
differences
Note: For Object when the protocol is enabled it is automatically configured with keystone with local auth (if internal keystone was selected).
mmuserauth service create is required only if you want to configure object with AD/LDAP - this is unlike FILE where there is no local auth.
Disable Protocols
Administration

Configuring Authentication with Plain AD
49
Administration

Configuring Authentication with automatic ID
mapping
• When the IBM Spectrum Scale™ system is configured for AD-based
authentication, automatic ID mapping method can be used to create UID or GID
of a user or group respectively.
• The ID maps are stored within the IBM Spectrum Scale™ system.
• Issue the mmuserauth service create command as shown:
• Note 1: In the command example shown in the next page, only a few/mandatory
parameters are shown. For details on other parameters that can be used, please
check the manpage or Knowledge Center.
• Note 2: The specified domain controller myADserver is only relevant for joining
the domain. After that step the configured DCs for the domain are queried from
DNS and an available one is automatically chosen.
The user account administrator is also only used for joining the domain
and creating or updating the machine account. After that, the protocol nodes use
the machine account to access AD.
50
Administration

Specified parameter is the
unique identifier of the
resources on a network
that are running NetBIOS
Specifies the ID map role of
the IBM Spectrum
Scale™ system. ID map role
of a stand-alone or singular
system deployment must be
selected "master".
Specified username is
used to join the cluster to
AD domain.
Authentication with automatic ID mapping
# mmuserauth service create --type ad
--data-access-method file --netbios-name
specscale --user-name administrator --
idmap-role master --servers myADserver --
password Passw0rd --idmap-range-size
1000000 --idmap-range 10000000-299999999
51
Specifies the pool from
which UIDs and GIDs are
assigned by the IBM
Spectrum Scale™ system
to AD users and group of
users.
The ID map range size
specifies the total
number of UIDs and
GIDs that are assignable
per domain.
Administration

1. Verify the authentication configuration by issuing the mmuserauth
service list command as shown in the following example:
# mmuserauth service list
2. Issue the mmuserauth service check command to check for the
authentication details.
# mmuserauth service check -–data-access-method file --nodes
clusternode1 --server-reachability
3. The system displays the following output:
Userauth file check on node: clusternode1
Checking nsswitch file: OK
AD servers status
NETLOGON connection: OK
Domain join status: OK
Machine password status: OK
Service 'gpfs-winbind' status: OK
52
Administration

FILE access configuration : AD
PARAMETERS VALUES
-------------------------------------------------
ENABLE_NFS_KERBEROS false
SERVERS myADserver
USER_NAME administrator
NETBIOS_NAME specscale
IDMAP_ROLE master
IDMAP_RANGE 10000000-299999999
IDMAP_RANGE_SIZE 1000000
UNIXMAP_DOMAINS none
LDAPMAP_DOMAINS none
OBJECT access not configured
PARAMETERS VALUES
-------------------------------------------------
5. Validate users can successfully authenticate.
53
Administration

Authentication with AD + RFC2307 ID mapping
• Note 1: In the command example shown in the next page, only a
few/mandatory parameters are shown. For details on other parameters
that can be used, please check the manpage or Knowledge Center.
• Note 2: The specified domain controller myADserver is only relevant for
joining the domain. After that step the configured DCs for the domain are
queried from DNS and an available one is automatically chosen.
The user account administrator is also only used for joining the
domain and creating or updating the machine account. After that, the
protocol nodes use the machine account to access AD.
54
Administration

AD domain.
The ID map range size
specifies the total
number of UIDs and
GIDs that are assignable
per domain.
Specifies the pool from
which UIDs and GIDs are
assigned by the IBM
Spectrum Scale™ system
to AD users and group of
users.
the IBM Spectrum
selected "master".
# mmuserauth service create --type ad --data-access-method
file --netbios-name specscale --user-name administrator
--idmap-role master --servers myADserver --password
Passw0rd --idmap-range-size 1000000 --idmap-range
10000000-299999999 --unixmap-domains 'DOMAIN(5000-20000)’
55
Specifies the list of domains from
where users and groups are read
from the UNIX attributes that are
populated in the RFC2307 schema
extension of AD server.
Administration

1. Issue the mmuserauth service list to verify the authentication configuration as
shown in the following example:
PARAMETERS VALUES
-------------------------------------------------
SERVERS myADserver
IDMAP_ROLE master
IDMAP_RANGE 10000000-299999999
UNIXMAP_DOMAINS DOMAIN(5000-20000)
PARAMETERS VALUES
56
Administration

Configuring AD using Kerberos with RFC2307 ID mapping
# mmuserauth service create --data-access-method file -
-type ad --netbios-name specscale --servers myADserver -
-user-name administrator --password Passw0rd --idmap-
role master --enable-nfs-kerberos --unixmap-domains
"DOMAIN(10000-200000)“
57
Enables Kerberized based access to exports. Kerberized
NFS-based access is only supported for users from AD
domains which are configured for fetching UID / GID
information from Active Directory (RFC2307 schema
attributes). Such an AD domain definition is specified via
the --unixmap-domains option.
Note: In the command example shown above, only the Kerberos parameter is explained. Rest are as described in
previous example. For details on other parameters that can be used, please check the manpage or Knowledge
Center.
Administration

3. Issue the mmuserauth service list to verify the authentication
configuration as shown in the following example:
PARAMETERS VALUES
-------------------------------------------------
ENABLE_NFS_KERBEROS true
SERVERS myADserver
IDMAP_ROLE master
IDMAP_RANGE 10000000-299999999
UNIXMAP_DOMAINS DOMAIN(1000-200000)
PARAMETERS VALUES
58
Administration

Authentication with AD + LDAP ID mapping
59
Administration

Defines the ID Map
range for user or group
from DOMAIN that
needs access to exports.
Name or IP address of
the LDAP server to fetch
the UID or GID for of a
user or group records in
RFC2307 schema .
Specifies the list of
domains from where
users and groups are read
from the UNIX attributes
that are populated in the
LDAP server in RFC2307
schema attributes
the IBM Spectrum
selected "master".
# mmuserauth service create --data-access-method file --type ad
--servers myADserver --user-name administrator --password Passw0rd
--netbios-name specscale --idmap-role master --ldapmap-domains
"DOMAIN1(type=stand-alone:range=1000-
100000:ldap_srv=myLDAPserver:usr_dn=ou=People,dc=example,dc=com:
grp_dn=ou=Groups,dc=example,dc=com:bind_dn=cn=manager,dc=example,dc
=com:bind_dn_pwd=password)“
60
AD domain.
Defines the bind tree on
the LDAP server where
user and groups objects
shall be found.
Administration

1. Issue the mmuserauth service list to verify the authentication configuration
as shown in the following example:
PARAMETERS VALUES
-------------------------------------------------
SERVERS myADserver
IDMAP_ROLE master
IDMAP_RANGE 10000000-299999999
UNIXMAP_DOMAINS none
LDAPMAP_DOMAINS DOMAIN1(type=stand-alone: range=1000-100000:
ldap_srv=myLDAPserver:usr_dn=ou=People,dc=example,dc=com:
grp_dn=ou=Groups,dc=example,dc=com:bind_dn=cn-manager,dc=example,dc=com)
61
Administration

Authentication with LDAP
62
Administration

• Issue the mmuserauth service create command as shown in
the following example.
few/mandatory parameters are shown. For details on other
parameters that can be used, please check the manpage or
Knowledge Center.
• Note 2: The specified domain controller myADserver is only
relevant for joining the domain. After that step the configured DCs for
the domain are queried from DNS and an available one is
automatically chosen.
63
Administration

the LDAP server to
authenticate against and
also fetch the UID or
GID.
Specifies the user name
to be used to perform
operations against the
authentication server.
Specifies the LDAP base
DN of the authentication
server.
# mmuserauth service create --type ldap
--data-access-method file –server myLDAPserver
--base-dn dc=example,dc=com --user-name
cn=manager,dc=example,dc=com --password secret
--netbios-name specscale
64
Administration

1. To verify the authentication configuration, issue the mmuserauth service list command as
shown in the following example:
PARAMETERS VALUES
-------------------------------------------------
ENABLE_SERVER_TLS false
ENABLE_KERBEROS false
USER_NAME cn=manager,dc=example,dc=com
SERVERS myLDAPserver
BASE_DN dc=example,dc=com
USER_DN none
GROUP_DN none
NETGROUP_DN none
USER_OBJECTCLASS posixAccount
GROUP_OBJECTCLASS posixGroup
USER_NAME_ATTRIB cn
USER_ID_ATTRIB uid
KERBEROS_SERVER none
KERBEROS_REALM none
PARAMETERS VALUES
-------------------------------------------------
65
Administration

3. To check whether the authentication configuration is consistent
across the cluster and the required services are enabled and
running, issue this command:
# mmuserauth service check --server-reachability
4. The system displays output similar to this:
Userauth file check on node: clusternode1
Checking SSSD_CONF: OK
Checking nsswitch file: OK
LDAP servers status
LDAP server myLDAPserver : OK
Service 'sssd' status: OK
Object not configured
66
Administration

Authentication with LDAP with TLS
• Issue the mmuserauth service create command as shown in the
following example.
67
Administration

Specifies the user name
to be used to perform
operations against the
Specifies the LDAP base
DN of the authentication
server.
the LDAP server to
authenticate against and
also fetch the UID or
GID.
--data-access-method file --servers myLDAPserver
--base-dn dc=example,dc=com --user-name
cn=manager,dc=example,dc=com --password secret
--netbios-name specscale --enable-server-tls
68
Enables TLS
communication with the
This option is disabled
by default.
Administration

1. Issue the mmuserauth service list command to see the current authentication
configuration as shown in the following example:
FILE access configuration : LDAP
PARAMETERS VALUES
-------------------------------------------------
ENABLE_SERVER_TLS true
ENABLE_KERBEROS false
USER_DN none
GROUP_DN none
NETGROUP_DN none
USER_NAME_ATTRIB cn
USER_ID_ATTRIB uid
KERBEROS_SERVER none
KERBEROS_REALM none
PARAMETERS VALUES
-------------------------------------------------
69
Administration

Authentication with LDAP with Kerberos
following example.
70
Administration

Specifies the Kerberos
server.
Indicates whether to
enable Kerberos in the
user authentication.
--data-access-method file --servers
myLDAPserver --base-dn dc=example,dc=com
--user-name cn=manager,dc=example,dc=com
--password secret --netbios-name specscale
--enable-kerberos --kerberos-server
myKerberosServer --kerberos-realm example.com
71
Note: In the command example shown above, only the Kerberos parameters are explained. For details on other parameters
Indicates the Kerberos server
authentication administrative
domain. The realm name is usually
the all-uppercase version of the
domain name. This option is case
sensitive.
Administration

1. Issue the mmuserauth service list command to see the current
authentication configuration as shown in the following example:
PARAMETERS VALUES
-------------------------------------------------
ENABLE_SERVER_TLS false
ENABLE_KERBEROS true
USER_DN none
GROUP_DN none
NETGROUP_DN none
USER_NAME_ATTRIB cn
USER_ID_ATTRIB uid
KERBEROS_SERVER myKerberosServer
KERBEROS_REALM example.com
PARAMETERS VALUES
-------------------------------------------------
72
Administration

Authentication with LDAP with TLS and Kerberos
following example.
73
Administration

Enables TLS
communication with the
This option is disabled
by default.
Specifies the Kerberos
server.
Indicates whether to
enable Kerberos in the
user authentication.
Authentication with LDAP with TLS and Kerberos
# mmuserauth service create --type ldap --data-access-
method file --servers myLDAPserver --base-dn
dc=example,dc=com --user-name
cn=manager,dc=example,dc=com --password secret --
netbios-name specscale --enable-server-tls --enable-
kerberos --kerberos-server myKerberosServer --
kerberos-realm example.com
74
Note: In the command example shown above, only the TLS and Kerberos parameters are explained. For details on other
parameters that can be used, please check the manpage or Knowledge Center.
Indicates the Kerberos server
authentication administrative
domain. The realm name is usually
the all-uppercase version of the
domain name. This option is case
sensitive.
Administration

Configuring LDAP with TLS and Kerberos for file access (Contd.)
1. To verify the authentication configuration, issue the mmuserauth service list command as shown
in the following example:
PARAMETERS VALUES
-------------------------------------------------
ENABLE_SERVER_TLS true
ENABLE_KERBEROS true
USER_DN none
GROUP_DN none
NETGROUP_DN none
USER_NAME_ATTRIB cn
USER_ID_ATTRIB uid
KERBEROS_SERVER myKerberosServer
KERBEROS_REALM example.com
PARAMETERS VALUES
-------------------------------------------------
75
Administration

Note: In the command example shown in the next page, only a
76
Administration

# mmuserauth service create --type nis
--data-access-method file --servers
myNISserver --domain nisdomain
77
Specifies the NIS Server
name. This is case specific.
You can also pass IP
Address.
The Domain Name for
NIS configuration
Administration

Configuring User Defined Authentication
# mmuserauth service create --type
userdefined --data-access-method file
78
Type set to Userdefined
Administration

Verifying the authentication services
configured in the system
• mmuserauth service check [--data-access-
method {file|object|all}] [-N|--nodes] {node-
list|cesNodes} [--server-reachability] [-r|--
rectify ]
• The mmuserauth service
check command helps to check whether the
authentication configuration is consistent
across the cluster and the required services are
enabled and running.
• This command validates and corrects the
authentication configuration files and starts any
associated services if needed.
79
Administration

Deleting authentication and ID mapping
configuration
• Deleting the authentication and ID mapping
configuration results in loss of access to data.
• Before you remove or edit ID mappings, determine
how access to data is going to be maintained.
• Issue the mmuserauth service remove
command to remove the authentication configuration
as shown in the following example:
# mmuserauth service remove –data-acess-method file
mmcesuserauth service remove: Command successfully
completed.
80
Administration

Modifying the authentication method
IMPORTANT:
• If data already exists or is created with the existing authentication and
ID mapping method, it is not recommended to change the
authentication or the ID mapping modes.
• Changing the authentication method also might invalidate the
existing ACLs that are applicable to files and directories.
• ACLs depend on the preexisting users and group IDs.
81
Administration

Modifying the authentication method (Contd.)
To modify the authentication method, perform the following steps:
1. List the existing authentication configuration for file and object
authentication method by using the mmuserauth service
list command.
2. Identify the parameters that you need to change. If an authentication
method and ID maps are already existing, you must not plan to change
the authentication type or ID mapping schemes.
When you remove the existing authentication method and ID maps, the
user and group of users who were accessing the data cannot access the
data anymore.
The following list provides the parameters that can be modified in each
authentication configuration:
82
Administration

For file authentication:
• With LDAP authentication, all attributes of the configuration can be modified. When
changing authentication servers, ensure that the newly specified servers are the replica
of the original servers, otherwise, it might result in loss of access to data.
• With AD authentication, most attributes (Check command manpage/Info center for
more information) of the configuration can be modified. When changing the
authentication server, ensure that the newly specified server is a domain controller in
the same AD domain that is being served by the original server, otherwise, it might result
in loss of access to data. If UNIX ID maps / LDAP ID maps are specified in current
configuration and more new AD domains are to be added, it is vital to specify the current
list of domains along with the new domains.
• With NIS authentication, all attributes of the configuration can be modified. When
changing servers, ensure that the newly specified servers are serving the same NIS
domain as the original servers; otherwise, it might result in loss of access to data.
83
Administration

3. Clean up the existing authentication by using the mmuserauth service
remove command. Do not specify the --idmapdelete option as it
results in loss of access to data.
4. Issue the mmuserauth service create with the required parameter
change; ensuring that you use the same authentication, ID mapping scheme,
and associated authentication servers.
5. List the authentication configuration by using the mmuserauth service
list to verify the change.
6. Ensure that the authentication is consistent across the cluster by using
the mmuserauth service check command.
84
Administration

Verifying that data can be accessed
• After successfully configuring AD, like seen in earlier slide, we can check for any issues by running, “mmuserauth service
check” command.
• Before we check if data is accessible, we must check for ACLs for the export as well as the File system Object/path.
• You can check using commands:
• Export ACLs: # mmsmb exportacl list
[smbexp1]
ACL:Everyone:ALLOWED/FULL
ACL:S-1-5-21-733047736-3426338400-2963614976-1321:ALLOWED/FULL
ACL:S-1-5-21-733047736-3426338400-2963614976-35291:ALLOWED/FULL
• File System ACLs: # mmgetacl /ibm/gpfs0/smbexp1
#NFSv4 ACL
#owner:root
#group:root
special:owner@:rwxc:allow
(X)READ/LIST (X)WRITE/CREATE (X)APPEND/MKDIR (X)SYNCHRONIZE (X)READ_ACL (X)READ_ATTR (X)READ_NAMED
(-)DELETE (X)DELETE_CHILD (X)CHOWN (X)EXEC/SEARCH (X)WRITE_ACL (X)WRITE_ATTR (X)WRITE_NAMED
special:group@:r-x-:allow
(X)READ/LIST (-)WRITE/CREATE (-)APPEND/MKDIR (X)SYNCHRONIZE (X)READ_ACL (X)READ_ATTR (X)READ_NAMED
(-)DELETE (-)DELETE_CHILD (-)CHOWN (X)EXEC/SEARCH (-)WRITE_ACL (-)WRITE_ATTR (-)WRITE_NAMED
special:everyone@:r-x-:allow
(X)READ/LIST (-)WRITE/CREATE (-)APPEND/MKDIR (X)SYNCHRONIZE (X)READ_ACL (X)READ_ATTR (X)READ_NAMED
(-)DELETE (-)DELETE_CHILD (-)CHOWN (X)EXEC/SEARCH (-)WRITE_ACL (-)WRITE_ATTR (-)WRITE_NAMED
user:DOMAIN1testuser1:rwxc:allow
(X)READ/LIST (X)WRITE/CREATE (X)APPEND/MKDIR (X)SYNCHRONIZE (X)READ_ACL (X)READ_ATTR (X)READ_NAMED
(-)DELETE (X)DELETE_CHILD (X)CHOWN (X)EXEC/SEARCH (X)WRITE_ACL (X)WRITE_ATTR (X)WRITE_NAMED
85
Administration

Verifying that data can be accessed (Contd…)
• We can then connect to the export. In this example, we will connect via smb client like so:
#smbclient //testcluster/smbexp1 -m smb2 –UDOMAIN1testuser1%Passw0rd -c "ls“
Domain=[DOMAIN1] OS=[] Server=[]
. D 0 Wed Jul 5 18:42:50 2017
.. D 0 Wed Jul 5 18:42:50 2017
4194304 blocks of size 1024. 364544 blocks available
• Now we try to write to the export:
#smbclient //testcluster/smbexp1 -m smb2 –
DOMAIN1testuser1%Passw0rd -c "lcd /tmp/; put testfile“
Domain=[DOMAIN1] OS=[] Server=[]
putting file testfile as testfile (1.6 kb/s) (average 1.6 kb/s)
• So we can see both READ and WRITE to the Export was successful. This mean that authentication
is working correctly and users were indeed Authorized to access data.
• Now let’s check from GPFS side:
# ls -l /ibm/gpfs0/smbexp1/
total 4
-rwxr--r-- 1 DOMAIN1testuser1 DOMAIN1crgroup1 8 Jul 5 18:48 testfile
• And checking for the UID
# ls -n /ibm/gpfs0/smbexp1/
total 4
-rwxr--r-- 1 12001321 12001322 8 Jul 5 18:48 testfile 86
Administration

Remote Cluster Mounts
• IBM Spectrum Scale™ allows you to create
NFS and SMB exports on remotely
mounted file systems.
• This allows you to separate the tasks
performed by each cluster. Storage cluster
owns the file systems and the storage.
• Protocol clusters contain the protocol node
that provides access to the remotely
mounted file system through NFS or SMB.
• In this configuration, each cluster is
managed independently.
• Refer the Infocenter for more information
https://www.ibm.com/support/knowledge
center/STXKQY_4.2.3/com.ibm.spectrum.sc
ale.v4r23.doc/bl1adv_protocoloverremotec
lu.htm
87
Administration

Remote Cluster Mounts (Contd…)
Configuring protocols for remote mounts:
• Install IBM Spectrum Scale on all nodes that are in the storage and protocol
clusters.
• Create the storage and protocol clusters.
• Create file systems on the storage cluster, taking the following into consideration:
• CES shared root file system – Each protocol cluster requires its own CES shared root file
system.
• Data file systems – At least one file system is required for each protocol cluster configured for
Cluster Export Services.
• Consider Authentication and ID Mapping requirements
• Configure clusters for remote mount.
• Once SMB and/or NFS is enabled, new exports can be created on the remotely
mounted data file system.
• Refer
https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.3/com.ibm.spectru
m.scale.v4r23.doc/bl1adv_protocoloverremoteclu.htm for more details. 88
Administration

Agenda and Flow
• Administration
89

Problem Determination Guide
This section describes the following:
• Monitoring IBM Spectrum Scale™
• Collecting details of issues using available methods
• Limitations of Authentication
• Known Use Cases
90

Monitoring IBM Spectrum Scale™
Monitoring using GUI
• Monitoring -> Events
• List the events that are reported in the IBM Spectrum Scale™ system
• Filter events on various criteria’s
• Events by severity – Error, Warning, Info
• Monitor and troubleshoot errors from the Events page.
• Settings -> Event Notifications
• Event Notifications via
• SNMP traps
• Email notifications
91

Monitoring using CLI
• The mmhealth command monitors health status
• Cluster health
• Node health
• Internal node role defines components to be monitored
• CES Node role
• This node role is active on the CES nodes that are listed by
mmlscluster --ces.
• Status of the CES is an aggregation of the status of its sub-services – AUTH,
AUTH_OBJ, BLOCK, CESNETWORK, NFS, SMB, OBJECT,
• Authentication sub service monitored: (Refer the knowledge center for more sub-
services)
a. AUTH – Tasks: Monitors LDAP, AD and or NIS-based authentication services.
b. AUTH_OBJ – Tasks: Monitoring the OpenStack identity service functionalities.
92

The following are the possible status of services:
• UNKNOWN - Status of the node or the service hosted on the node is not known.
• HEALTHY - The node or the service hosted on the node is working as expected. There are no active error events.
• CHECKING - The monitoring of a service or a component hosted on the node is starting at the moment. This state is a transient state and
is updated when the startup is completed.
• TIPS - There might be an issue with the configuration and tuning of the components. This status is only assigned to a Tip event.
• DEGRADED - The node or the service hosted on the node is not working as expected. That is, a problem occurred with the component
but it did not result in a complete failure.
• FAILED - The node or the service hosted on the node failed due to errors or cannot be reached anymore.
• DEPEND - The node or the services hosted on the node have failed due to the failure of some components. For example, an NFS or SMB
service shows this status if authentication has failed.
https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.3/com.ibm.spectrum.scale.v4r23.doc/bl1pdg_rasevents.htm#reference_bbt
_4nz_qt__auth
https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20(GPFS)/page/P
rotocols%20Problem%20Determination 93

Collecting details of the issue
Likely scenarios
• Authentication configuration failures
• Access failures
• SMB access is failing
• NFS access is failing
• Data ownership display problems
94

Data collection (First Time Data Capture)
• To diagnose the problem it is necessary to gather relevant information
from the cluster.
• Collection of debugging information, such as configuration files and
logs, can be achieved by using the gpfs.snap command.
• This command gathers data about GPFS, operating system
information, and information for each of the protocols.
• It also collects AUTHENTICATION related data like Authentication
configuration and logs.
• To collect only authentication traces use the following command,
gpfs.snap -–protocol authentication
95

Authentication details captured by gpfs.snap
The following authentication data is always obtained by the gpfs.snap command,
• The output of these commands:
mmuserauth service list
mmuserauth service check --data-access-method all --nodes cesNodes
mmuserauth service check --data-access-method all --nodes cesNodes --server-reachability
systemctl status ypbind
systemctl status sssd
ps aux | grep keystone
lsof -i
sestatus
systemctl status firewalld
systemstl status iptables
/usr/lpp/mmfs/bin/net ads info
• The following files:
/etc/nsswitch.conf
/etc/ypbind.conf
/etc/idmapd.conf
/etc/sssd/*
/etc/krb5.conf
/etc/krb5.keytab
/etc/firewalld/*
/var/log/sssd/*
96

SSSD ( LDAP or NIS based authentication )
1. Configuration Files are:
a) /etc/sssd/ sssd.conf
b) /etc/krb5.conf ( if LDAP Kerberos authentication configured )
c) /etc/krb5.keytab ( if LDAP Kerberos authentication configured )
2. Log files are:
a) /var/log/sssd/sssd.log
b) /var/log/sssd/sssd_nss.log
c) /var/log/sssd/sssd_LDAPDOMAIN.log (if LDAP authentication configured )
d) /var/log/sssd/sssd_NISDOMAIN.log (if NIS authentication configured)
Note: For more information on SSSD log files, see Red Hat Linux documentation
97

Winbind ( AD based authentication schemes )
1. Configuration Files are:
a) /etc/krb5.conf
b) /etc/krb5.keytab ( if AD with kerberized NFS is configured )
2. Log files are:
a) /var/adm/ras/log.wb-<DOMAIN>
[Depends upon available AD domains in the infrastructure]
b) /var/adm/ras/log.winbindd-dcconnect
c) /var/adm/ras/log.winbindd-idmap
d) /var/adm/ras/log.winbindd
98

Authentication configuration failures
• Pre configuration
• Pre-requisites are not meet and hence CLI fails
• Environment related failures
• Network related
• Administrative credentials requirements
• Post configuration
• Verify if the validation command outputs,
• mmuserauth service check --data-access-method all --nodes cesNodes
• mmuserauth service check --data-access-method all --nodes cesNodes --server-reachability
• Verify user can authentication over the SMB protocol
99

Access failures
• SMB
https://www.ibm.com/support/knowledgecenter/en/STXKQY_4.2.3/com.ibm.spect
rum.scale.v4r23.doc/bl1pdg_SMBrelatedissues.htm
• NFS
https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.3/com.ibm.spectru
m.scale.v4r23.doc/bl1pdg_pbmnfsv4.htm
• Tracing
• Logging at high levels
• The command mmprotocoltrace for starting and stopping tracing supports SMB,
Winbind, Network and Object tracing.
• NFS tracing can be done with a combination of commands.
100

SMB Tracing
1. Before you start the trace, you can check the configuration settings for the
type of trace that you plan to run
mmprotocoltrace config smb
The response to this command displays the current settings from the trace
configuration file. For more information about this file, see the “Trace
configuration file”
2. Clear the trace records from the previous trace of the same type:
mmprotocoltrace clear smb
This command responds with an error message if the previous state of a
trace node is something other than DONE or FAILED. If this error occurs,
follow the instructions in the “Resetting the trace system ”
101

3. Start the new trace:
mmprotocoltrace start smb -c <clientIP>
The following response is observed :
Trace ’3f36dbed-b567-4566-9beb-63b6420bbb2d’ created successfully for ’smb’
4. Check the status of the trace to verify that tracing is active on all the configured nodes:
mmprotocoltrace status smb
The following response is typical:
Trace ID: fcb7cb07-c45e-43f8-8f1f-2de50cf15062
State: Active
User ID: root
Protocol: smb
Start Time: 10:57:43 04/03/2016
End Time: 11:07:43 04/03/2016
Client IPs: 10.0.100.42, 10.0.100.43
Origin Node: ch-42.localnet.com
Syscall: False
Syscall Only:False
Nodes:
Node Name: ch-41.localnet.com
State: ACTIVE
Trace Location: /tmp/mmfs/smb.20160304_105742.trc
State: ACTIVE
Trace Location: /tmp/mmfs/smb.20160304_105742.trc
State: ACTIVE
Trace Location: /tmp/mmfs/smb.20160304_105742.trc 102

5. If all the nodes started successfully, perform the actions that you want to
trace. For example, if you are tracing a client IP address, enter commands
that create traffic on that client.
6. Stop the trace:
mmprotocoltrace stop smb
The following response is typical. The last line gives the location of the trace
log file:
Stopping traces Trace ’01239483-be84-wev9-a2d390i9ow02’ stopped for smb
Waiting for traces to complete
Waiting for node ’clusternode1’
Waiting for node ’clusternode2’
Finishing trace ’01239483-be84-wev9-a2d390i9ow02’
Trace tar file has been written to
’/tmp/mmfs/smb.20150513_162322.trc/smb.trace.20150513_162542.tar.gz
7. If you do not stop the trace, it continues until the trace duration expires.
8. Look in the trace log files for the results of the trace.
103

NFS tracing
1. NFS tracing is achieved by increasing the log level, repeating the issue,
capturing the log file, and then restoring the log level.
2. To increase the log level, use the command mmnfs configuration
change LOG_LEVEL=FULL_DEBUG.
3. You can set the log level to the following values: NULL, FATAL, MAJ, CRIT, WARN,
EVENT, INFO, DEBUG, MID_DEBUG, and FULL_DEBUG.
NOTE: FULL_DEBUG is the most useful for debugging purposes.
4. After the issue is recreated by running the gpfs.snap command either with no
arguments or with the --protocol nfs argument, the NFS logs are captured.
5. The logs can then be used to diagnose any issues.
6. To return the log level to normal, use the same command but with a lower
logging level (the default is EVENT).
104

Data ownership display problems
• Check data ownership on protocol server
• Check data ownership on client mounting the export
• Validate the user and its group resolves to the same UIDNumber and
GIDNumber on both – protocol server and client.
• Check if appropriate group memberships are returned
• Based on the variant of protocol access ( eg: NFSv3 Vs NFSv4 ) ensure
the necessary pre-requisite setup is done on protocol server and client
105

File Access Limitations in Authentication
AD based Authentication
• When the SMB service is stopped on a protocol node, the NFS-based
access is also affected on that protocol node.
• Following considerations apply to configuring an NFS server to look up
group membership information for an accessing NFS user:
• The server-side group lookup functionality, which is enabled by setting
the MANAGE_GIDS flag in the NFS configuration, works only after the user makes a
valid authentication connection over CIFS.
• You must make a valid authentication connection to the protocol node that serves
the public IP from which the NFS export is to be mounted.
• If the group membership of the user on an AD server changes, you must make a new
valid CIFS connection to the protocol node that serves the public IP from which the
NFS export is to be mounted. This new connection reflects the changes on the
protocol node of the CES cluster.
• It is a good practice to make a valid authentication connection over CIFS to all the
protocol nodes that participate in group membership evaluations. This practice
results in uniform membership evaluations on all the protocol nodes of the CES
cluster.
106

File Access Limitations in Authentication ( Contd. )
AD with automatic ID mapping
• Migrating the internally generated user and group ID maps to an external ID
mapping server.
AD with RFC2307 schema ID mapping
• Enabling RFC2307 for a trusted domain requires a two-way trust between the
native and the trusted domains. The authentication configuration command does
not check the two-way trust between the native domain and the RFC2307
domain.
• To access the IBM Spectrum Scale™ system, users and groups must have a valid
UID/GID assigned to them in AD. Therefore, the user's primary Microsoft
Windows group must be assigned a valid GID.
• Multiprotocol access to FILE exports is only allowed between NFSV4 and
SMB. Thus one cannot access the same FILE export by using both NFSV3 and SMB
protocols.
AD with LDAP ID mapping
• Only LDAP servers that implement RFC2307 schema are supported
107

File Access Limitations in Authentication (Contd.)
LDAP-based authentication
• Only LDAP servers that implement RFC2307 schema are supported
• SMB access shall fail for users with the same name even if present within
different organizational units under the specified baseDN in the LDAP server.
The LDAP user suffix and LDAP group suffix values configured on the system
are not effective for SMB access.
• LDAP referrals are not supported.
• ACL management through windows clients is not supported.
108

File Access Limitations in Authentication (Contd.)
• General Limitations
a) Authentication configuration commands restart the IBM Spectrum
Scale™ protocol services – SMB, NFS, OBJECT.
b) For file data access, switching or migrating from one authentication
method to another is not supported, because it might lead to loss of
access to the data on the system.
c) The IBM Spectrum Scale™ system does not support authentication servers
(AD, LDAP, and NIS) that are running on virtual machines that are stored on
an SMB or NFS export of same system.
109

Known Use Cases – 1
• Issue: AD + RFC 2307 based ID mapping authentication successfully configured.
1. Yet some users get ACCESS_DENIED when accessing over SMB
2. Some users / groups do not get resolved on protocol nodes
3. When trying to set ACLs for user / group ACL update successfully completes. But on listing ACL’s for the updated data, applied ACLs are
not seen
• Probable Cause/s:
1. Affected users do not have UIDNumber set / Affected users do not have UIDNumber set within the configured ID range for AD
domain
2. Windows Primary group of the affected users do not have GIDNumber set. / Windows Primary group of the affected users do not
have GIDNumber set within the configured ID range for AD domain
3. GIDNumber of the affected group is not set / GIDNumber of the affected group is not set within the configured ID range for AD
domain
• How to debug this issue:
• Assuming sufficient rights / ACL’s exist for accessing user and still the failure is observed in the deployment.
• Check if the UIDNumber is assigned to the affected user and is within defined range
# mmadquery list uids --filter=administrator
Password:
UIDS from server 1.2.3.4 (domain NASDOMAIN.COM)
User SID UID UIDNumber
------------- --------------------------------------------- ------------- ---------
administrator S-1-5-21-733047736-3426338400-2963614976-1218 administrator 20000 110

Known Use Cases – 1 ( Contd. )
• If UIDNumber is within range, check if the user’s Windows Primary group defined on Active directory has a valid GIDNumber value set
and is within the range.
# mmadquery list gids --filter=testgroup
Password:
User SID UID GIDNumber
------------- --------------------------------------------- ------------- ---------
testgroup S-1-5-21-733047736-3426338400-2963614976-2128 testgroup 23456
• Check if the GIDNumber is assigned to the affected group and is within defined range
• Conclusion:
• Access for those users and groups will be denied if UIDNumber or GIDNumber are not set correctly.
• In case of RFC2307, if a user’s Windows Primary group in Active Directory has a missing GIDNumber, access is denied for the
respective user.
• How to correct this issue:
• For that corresponding user, check for its Windows Primary group in Active Directory. ( On AD server, within the user properties,
Memberof tab displays group membership of the user and its Windows primary group defined )
• Check that the Group has a valid GIDNumber set.
• Best practice that should be followed:
• It is mandatory that the user’s UIDNumber and GIDNUmber of the Windows Primary Group of the user in Active Directory are
correctly set.
• It is also mandatory for every group planned to be utilized in the access and authorization GIDNumber is correctly set
• The best practice is appropriately plan for the ID range value to be configured during authentication configuration for UNIX domain
mapping. Post configuration verify these steps are followed before trying to store data or access data.
111

Known Use Cases - 2
• Issue: AD authentication is configured successfully. NFS users on UNIX / LINUX clients are unable to access data generated over SMB by the
same user’s identity defined on AD
• Probable Cause
• The UIDNumber and GIDNumber is not same on Unix client and protocol server In this case, access is denied and this is expected behavior.
• Check for the UIDNumber or GIDNumber for the user or group, respectively, on Windows which has access to the file.
• Check the UIDNumber or GIDNumber for the UNIX user which is denied access.
• The UIDNumber and GIDNumber for users on the Unix clients is defined on client or somewhere centrally in the infrastructure. The
UIDNumber or GIDNumber is generated by the IBM Spectrum Scale ™ system.
• Conclusion: If you have Unix users who want to access data, plain AD is not the correct authentication. You should implement AD + RFC2307 ID
mapping or AD + LDAP ID mapping.
• The only way to proceed is to clean up authentication and cleanup ID maps generated by running mmuserauth service
remove command.
• Then mmuserauth service remove --idmapDelete option to erase the id mapping that was created when actions were tried
out.
• Re-run the configuration command after choosing the correct method for the environment.
• Remember that old data will be inaccessible because it will have authorization defined with the UIDNumber and GIDNumber generated by
the IBM Spectrum Scale ™ system in earlier configuration.
• Plan for the appropriate authentication scheme based on the need of access by the customer.
112

Known Use Cases - 3
• Issue: Data ownership on NFSv4 mounted exports shows up as nobody : nobody
• Probable cause
• The NFSv4 user name mapping is not configured on NFS server
• The NFSv4 user name mapping is not configured on NFS client
• To use NFSV4 , set the NFS user map domain on the IBM Spectrum Scale protocol nodes and also ensure the same
NFS user map domain on every accessing NFSv4 client is set.
• On NFSv4 client the NFS user mapping service should also be running
• Check if the NFS user map domain is configured on IBM Spectrum Scale ™ system Issue the mmnfs configuration
list command. The system displays the following output, which shows that the ID map domain is not set:
# mmnfs configuration list
………….
………………..
……………………..
Idmapd Configuration
====================
====================
• Also check if the /etc/idmap.conf ( on RHEL / SLES / Ubuntu ) or relevant ID map configuration file NFS user name
mapping on NFSv4 client is also updated with the same value of NFS user name map domain as set on IBM Spectrum
Scale ™ system. The NFS user name mapping service should also be running on the NFSv4 client 113

Known Use Cases – 3 ( Contd. )
• Conclusion:
• To use NFSV4 ID mapping, you must set the NFS ID map domain on the IBM Spectrum Scale protocol nodes and you
must configure the same NFS ID map domain on every NFSv4 client.
• Enter the following command to set the NFS user map domain:
# mmnfs configuration change IDMAPD_DOMAIN=MY_IDMAP_DOMAIN
• Issue the mmnfs configuration list command to verify that the ID map domain is set.The system displays this output:
# mmnfs configuration list
………….
………………..
……………………..
Idmapd Configuration
====================
DOMAIN: MY_IDMAP_DOMAIN
====================
• To use NFSV4 ID mapping, you must set the NFS ID map domain on the IBM Spectrum Scale protocol nodes and you
must configure the same NFS ID map domain on every NFS client.
114

mmadquery Tool
• This tool was developed to query and validate Active Directory (AD)
server settings.
• This mainly works when AD server is configured with the system.
• It is available on all IBM Spectrum Scale™ editions.
• You can run mmadquery command to query an AD Server for:
• Users and Groups
• User IDs and Group IDs
• Known domain controller and trusts
• To run consistency checks.
• We will see a couple of example in next slide. For more information
on the command, check out
https://www.ibm.com/support/knowledgecenter/en/STXKQY_4.2.3/c
om.ibm.spectrum.scale.v4r23.doc/bl1adm_mmadquery.htm
115

mmadquery Tool
• Check if users / groups on AD server have RFC2307 attributes populated
# mmadquery list uids --server 1.2.3.4 --domain NASDMAIN .COM
Password:
User SID UID UIDNumber
---------------------------- ----------------------------------------------------------------------- ------------------- -----------
user1 S-1-5-21-733047736-3426338400-2963614976-1172 - -
user2 S-1-5-21-733047736-3426338400-2963614976-1173 user2 12345
Administrator S-1-5-21-733047736-3426338400-2963614976-500 Administrator 20021
Guest S-1-5-21-733047736-3426338400-2963614976-501 - -
# mmadquery list gids --server 1.2.3.4 --domain NASDMAIN .COM
Password:
GIDS from server 1.2.3.4 (domain NASDOMAIN.COM)
Group SID GIDNumber
--------------------------------------- ----------------------------------------------------------------------- ------------
Domain Computers S-1-5-21-733047736-3426338400-2963614976-515 -
Domain Controllers S-1-5-21-733047736-3426338400-2963614976-516 -
Domain Admins S-1-5-21-733047736-3426338400-2963614976-512 22000
Domain Users S-1-5-21-733047736-3426338400-2963614976-513 3300
116

Agenda and Flow
• Administration
117

Questions and Answers
Following are some of the questions asked during the session:
1. With Plain AD setup and AUTO ID Map, will NFS be supported?
>> No, with Plain AD Setup, NFS access is not supported. However
this does not mean you cannot create shares. You can if the NFS
service is enabled. However, you will not be able to provide
access to both Unix users and Windows users.
2. Can we change Authentication later if we plan to bring in NFS clients or Unix users?
>> This is a very important question. Many customers in the past
have reconfigured Authentication assuming its OK to add
additional RFC2307 or LDAP along with Auto ID Mapping. However,
the answer to this question is WE DO NOT SUPPORT MODIFYING
AUTHENTICATION that has been once setup. Hence, its IMPORTANT
that you consider all scenarios, foresee whether customer plans
to add UNIX clients, take into consider the growth of company
and hence number of users and groups which means decide on the
right range for the AUTO IDMaps and RFC2307/LDAP Ranges.
Yes, we have different schemes for Authentication and ID Mapping
that we support. But once chosen you need to stick to it. Hence,
before migrating data, you can test different methods and choose
the one that fits in your environment.
118
Question and Answers

3. How do we get the same UIDs on the DR system for AUTO ID Mapping setup?
>> In this case we need to configure the Primary System as
“Master” and the DR site as “Subordinate”. Once we have
configured AUTO ID Mapping or Plain AD on the Master, we can
import the ID Mappings using the commands available. On the DR
Site we then export the ID Mappings.
After this, we can copy data from Master to Subordinate.
When NEW Users are added on Master, its OK to not copy on the
subordinate. This is because subordinate system now has the
details of ID Mappings. It can calculate and generate the Same
Deterministic ID Map as on the Master.
The only time you will need to re-export the ID Maps from Master
is when there is a NEW Domain trusting the domain configured.
Once the ID Maps are exported into subordinate, it will
automatically generate deterministic ID Maps in future.
Its IMPORTANT to note that it is the responsibility of the
System Admin to export the ID Maps on the subordinate the first
time and whenever a new Domain is in the environment.
119

4. What happens if the UID or GID value exceeds the range configured?
>> If for example, for an RFC2307 setting the range provided
is 3000-4000, it means only 1000 users are eligible to
access the system. A user with UID 4001 or more will be
denied access.
Hence, its important to consider growth of company and in
turn the number of Users and Groups that will access the
system.
Similarly the range must be considered for AUTO ID Maps and
LDAP settings.
5. How to check details of users and groups on the AD Server?
>> You can now check different attributes of Users, Groups
and Domain from the IBM Spectrum Scale system using the tool
“mmadquery” which was developed to query the AD server for
such details.
Currently the tool only works for AD environment and not NIS
or LDAP.
120

6. What if the AD server with which Authentication is configured is DOWN? Will access be
affected?
>> The AD Server details that is provided during the
configuration of authentication is only used to communicate with
the server and create the Machine Account. Once authentication
is successfully configured, all communication to the Domain
Controller happens via the DNS.
The DNS is queried for the IP address of the closest DC and is
then communicated.
7. What is the typical use case for AD + LDAP setup?
>> When customer has both AD server and LDAP sever in the
environment. For heterogenous access of Windows and NFS clients,
we can use AD + LDAP where AD is used for user Authentication
for SMB Access and LDAP is used for ID Mapping.
Instead of introducing all user and group information into
RFC2307 we can re-use the data already existing in the LDAP
server.
121

8. Is there any specific setting to be done on AD server for successful configuration with
IBM Spectrum Scale?
>> No, there is no specific setting or changes that needs
to be done on the AD server. Any user that can create a
machine account on the Server, read trust information and
machine accounts if already created before hand, can be
used for configuring authentication.
The user need not be administrator.
9. Customers are not comfortable passing the --password parameter during configuration
of authentication? What can be done?
>> You can leave the --password parameter when configuring
authentication. In case of AD and LDAP, the user password
will then be prompted. System admin can then type the
password themselves.
Note: From the 4.2.4 release, we will be deprecating the -
-password parameter and you will always be required to
provide the password on the command prompt.
122

IBM Spectrum Scale Authentication for File Access - Deep Dive

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to IBM Spectrum Scale Authentication for File Access - Deep Dive

Similar to IBM Spectrum Scale Authentication for File Access - Deep Dive (20)

Recently uploaded

Recently uploaded (20)

IBM Spectrum Scale Authentication for File Access - Deep Dive