Galera Cluster for MySQL vs MySQL (NDB) Cluster: A High Level Comparison Severalnines
Galera Cluster for MySQL, Percona XtraDB Cluster and MariaDB Cluster (the three “flavours” of Galera Cluster) make use of the Galera WSREP libraries to handle synchronous replication.MySQL Cluster is the official clustering solution from Oracle, while Galera Cluster for MySQL is slowly but surely establishing itself as the de-facto clustering solution in the wider MySQL eco-system.
In this webinar, we will look at all these alternatives and present an unbiased view on their strengths/weaknesses and the use cases that fit each alternative.
This webinar will cover the following:
MySQL Cluster architecture: strengths and limitations
Galera Architecture: strengths and limitations
Deployment scenarios
Data migration
Read and write workloads (Optimistic/pessimistic locking)
WAN/Geographical replication
Schema changes
Management and monitoring
Virtualization with KVM (Kernel-based Virtual Machine)Novell
As a technical preview, SUSE Linux Enterprise Server 11 contains KVM, which is the next-generation virtualization software delivered with the Linux kernel. In this technical session we will demonstrate how to set up SUSE Linux Enterprise Server 11 for KVM, install some virtual machines and deal with different storage and networking setups.
To demonstrate live migration we will also show a distributed replicated block device (DRBD) setup and a setup based on iSCSI and OCFS2, which are included in SUSE Linux Enterprise Server 11 and SUSE Linux Enterprise 11 High Availability Extension.
The Network File System (NFS) Version 4 is a distributed file system similar to previous versions of NFS in its straightforward design, simplified error recovery, and independence of transport protocols and operating systems for file access in a heterogeneous network.
NFS, was developed by Sun Microsystems to provide distributed transparent file access in a heterogeneous network. It achieves this by being relatively simple in design and not relying too heavily on any particular file system model.
This presentation is based on the paper of “The NFS Version 4 Protocol” written by Brian Pawlowski, Spencer Shepler, Carl Beame, Brent Callaghan, Michael Eisler, David Noveck, David Robinson and Robert Thurlow.
IBM Spectrum Scale Best Practices for Genomics Medicine WorkloadsUlf Troppens
Genomics medicine requires physicians, data scientists and researchers to analyze huge amounts of genomics data quickly. The IBM Spectrum Scale Best Practices for Genomics Medicine Workload provides composable infrastructure that enables IT architects to customize deployments for varying functional and performance needs. The described scale-out architecture is capable to store, access and manage genomics data from a few 100 TB to tens of PB. The solution integrates compute resources and an easy-to-use Web User Interface to submit high-throughput batch jobs to analyze genomics data sets. While the best practices are optimized for genomics medicine workloads, most of the settings are generic and applicable to other workloads and industries.
KVM High Availability Regardless of Storage - Gabriel Brascher, VP of Apache ...ShapeBlue
Having High Availability enabled for KVM Hosts can improve greatly the QoS by handling (fence/recover) a problematic Host as well as re-starting its stopped VMs on healthy hosts. However, there is a limitation on CloudStack HA for KVM; it relies mainly on NFS heartbeat script checks. This Talk illustrates how CloudStack HA works for KVM hosts and it presents a way of improving its implementation in a way that KVM HA works with any storage system pluggable on KVM, not just NFS.
About Gabriel Brasher - https://blogs.apache.org/cloudstack/
------------------------------------------
CloudStack European User Group Virtual happened on May 27th. The first CSEUG Virtual proved to be a huge success. It collected people from 23 countries – Germany, the United Kingdom, Switzerland, India, Bulgaria, Greece, Poland, Serbia, Brazil, Chile, Russia, USA, Canada, Japan, France, Uruguay, Korea …
We also had a record number of registrations and attendees for a CloudStack User Group Event. The physical distance was not a stopper for our speakers, who joined the event from 6 different countries.
------------------------------------------
About CloudStack: https://cloudstack.apache.org/
Domino Server Health - Monitoring and ManagingGabriella Davis
If you're a Domino administrator how do you decide what to monitor on your servers and how to manage them ? What are the key things to monitor? How do good practice management tools such as statistics reporting, DDM, cluster symmetry, database repair and policy settings make your work lighter and faster. Finally we’ll talk about some of the “must dos” in the day, week and month of a Domino admin.
Presented at Engage.ug in Brussels May 2019
Galera Cluster for MySQL vs MySQL (NDB) Cluster: A High Level Comparison Severalnines
Galera Cluster for MySQL, Percona XtraDB Cluster and MariaDB Cluster (the three “flavours” of Galera Cluster) make use of the Galera WSREP libraries to handle synchronous replication.MySQL Cluster is the official clustering solution from Oracle, while Galera Cluster for MySQL is slowly but surely establishing itself as the de-facto clustering solution in the wider MySQL eco-system.
In this webinar, we will look at all these alternatives and present an unbiased view on their strengths/weaknesses and the use cases that fit each alternative.
This webinar will cover the following:
MySQL Cluster architecture: strengths and limitations
Galera Architecture: strengths and limitations
Deployment scenarios
Data migration
Read and write workloads (Optimistic/pessimistic locking)
WAN/Geographical replication
Schema changes
Management and monitoring
Virtualization with KVM (Kernel-based Virtual Machine)Novell
As a technical preview, SUSE Linux Enterprise Server 11 contains KVM, which is the next-generation virtualization software delivered with the Linux kernel. In this technical session we will demonstrate how to set up SUSE Linux Enterprise Server 11 for KVM, install some virtual machines and deal with different storage and networking setups.
To demonstrate live migration we will also show a distributed replicated block device (DRBD) setup and a setup based on iSCSI and OCFS2, which are included in SUSE Linux Enterprise Server 11 and SUSE Linux Enterprise 11 High Availability Extension.
The Network File System (NFS) Version 4 is a distributed file system similar to previous versions of NFS in its straightforward design, simplified error recovery, and independence of transport protocols and operating systems for file access in a heterogeneous network.
NFS, was developed by Sun Microsystems to provide distributed transparent file access in a heterogeneous network. It achieves this by being relatively simple in design and not relying too heavily on any particular file system model.
This presentation is based on the paper of “The NFS Version 4 Protocol” written by Brian Pawlowski, Spencer Shepler, Carl Beame, Brent Callaghan, Michael Eisler, David Noveck, David Robinson and Robert Thurlow.
IBM Spectrum Scale Best Practices for Genomics Medicine WorkloadsUlf Troppens
Genomics medicine requires physicians, data scientists and researchers to analyze huge amounts of genomics data quickly. The IBM Spectrum Scale Best Practices for Genomics Medicine Workload provides composable infrastructure that enables IT architects to customize deployments for varying functional and performance needs. The described scale-out architecture is capable to store, access and manage genomics data from a few 100 TB to tens of PB. The solution integrates compute resources and an easy-to-use Web User Interface to submit high-throughput batch jobs to analyze genomics data sets. While the best practices are optimized for genomics medicine workloads, most of the settings are generic and applicable to other workloads and industries.
KVM High Availability Regardless of Storage - Gabriel Brascher, VP of Apache ...ShapeBlue
Having High Availability enabled for KVM Hosts can improve greatly the QoS by handling (fence/recover) a problematic Host as well as re-starting its stopped VMs on healthy hosts. However, there is a limitation on CloudStack HA for KVM; it relies mainly on NFS heartbeat script checks. This Talk illustrates how CloudStack HA works for KVM hosts and it presents a way of improving its implementation in a way that KVM HA works with any storage system pluggable on KVM, not just NFS.
About Gabriel Brasher - https://blogs.apache.org/cloudstack/
------------------------------------------
CloudStack European User Group Virtual happened on May 27th. The first CSEUG Virtual proved to be a huge success. It collected people from 23 countries – Germany, the United Kingdom, Switzerland, India, Bulgaria, Greece, Poland, Serbia, Brazil, Chile, Russia, USA, Canada, Japan, France, Uruguay, Korea …
We also had a record number of registrations and attendees for a CloudStack User Group Event. The physical distance was not a stopper for our speakers, who joined the event from 6 different countries.
------------------------------------------
About CloudStack: https://cloudstack.apache.org/
Domino Server Health - Monitoring and ManagingGabriella Davis
If you're a Domino administrator how do you decide what to monitor on your servers and how to manage them ? What are the key things to monitor? How do good practice management tools such as statistics reporting, DDM, cluster symmetry, database repair and policy settings make your work lighter and faster. Finally we’ll talk about some of the “must dos” in the day, week and month of a Domino admin.
Presented at Engage.ug in Brussels May 2019
Presentation done at the November meeting of the Sudoers Barcelona group (https://www.meetup.com/sudoersbcn/).
HashiCorp Vault (https://www.vaultproject.io/)
"Vault és una eina per emmagatzemar i gestionar secrets. Veurem què ofereix, com instal·lar-la, utilitzar-la i operar-la, i la nostra experiència."
Securing Microservices using Play and Akka HTTPRafal Gancarz
Going down the microservices route makes a lot of things around creating and maintaining large systems easier but it comes at a cost too, particularly associated with challenges around security. While securing monolithic applications was a relatively well understood area, the same can't be said about microservice based architectures.
This presentation covers how implementing microservices affects the security of distributed systems, outlines pros and cons of several standards and common practices and offers practical suggestions for securing microservice based systems using Play and Akka HTTP.
Part 5 of the REAL Webinars on Oracle Cloud Native Application Development - ...Lucas Jellema
Focus on Cloud Operations - on monitoring, automation through infrastructure as code and on secure management of keys and secrets. We discuss Audit and Log, Monitor and Healthcheck, Alarms and Notifications, Vaults with Keys and Secrets and the Terraform OCI provider and Resource Manager with custom and pre built stacks.
Introduction to security implications while writing application code for public facing gateway services.
Key points:
1. How to build secure APIs
2. Understand security challenges while building a BFF (Backend For Frontend)
3. Security as a part of development
Identity service keystone | what is openstack | how to make openstack commands | connect ubuntu with database | install ubuntu on virtualbox | commands for identity service keystone in cloud computing
Geek Sync | Deployment and Management of Complex Azure EnvironmentsIDERA Software
You can watch the replay of this Geek Sync webinar in the IDERA Resource Center: http://ow.ly/pg7N50A4svf.
Today's data management professional is finding their landscape changing. They have multiple database platforms to manage, multi-OS environments and everyone wants it now.
Join IDERA and Kellyn Pot’Vin-Gorman as she discusses the power of auto deployment in Azure when faced with complex environments and tips to increase the knowledge you need at the speed of light. Kellyn will cover scripting basics, advanced Portal features, opportunities to lessen the learning curve and how multi-platform and tier doesn't have to mean multi-cloud.
Attendees can expect to learn how to build automation scripts efficiently, even if you have little scripting experience, and how to work with Azure automation deployments. This session will allow you to begin building a repository of multi-platform development scripts to use as needed.
About Kellyn: Kellyn Pot’Vin-Gorman is a member of the Oak Table Network and an IDERA ACE and Oracle ACE Director alumnus. She is the newest Technical Solution Professional in Power BI with AI in the EdTech group at Microsoft. Kellyn is known for her extensive work with multi-database platforms, DevOps, cloud migrations, virtualization, visualizations, scripting, environment optimization tuning, automation, and architecture design. She has spoken at numerous technical conferences for Oracle, Big Data, DevOps, Testing and SQL Server. Her blog, http://dbakevlar.com and social media activity under her handle, DBAKevlar is well respected for her insight and content.
Building microservices sample applicationAnil Allewar
The slides provide details on how to build the sample Microservices application that covers the whole distributed system paradigm.
Please refer to the introduction to Microservices before following the contents in this slide
https://www.slideshare.net/anilallewar/introduction-to-microservices-78270318
Scott Rea - IoT: Taking PKI Where No PKI Has Gone BeforeDigiCert, Inc.
Presentation by Scott Rea, DigiCert's Sr. PKI Architect, at AppSec California 2015.
Abstract:
Traditional PKI focuses on binding a public key to the keyholder’s identity, which is implicitly assumed to be a well-defined, relatively static thing (such as individual’s full name or email address, or the hostname of a public webserver). However, in the envisioned smart grid, for example, the relevant properties of the keyholder are not just the device’s identity (i.e. this is a meter made by ACME or this is a refrigerator made by GE) but its context: This is a refrigerator in the apartment rented by Alice, who buys power from X.
This context information will not necessarily be known until device installation and also may change dynamically. What if Alice sells her fridge on Craigslist or sublets her apartment to Bob? What if repair personnel replace Alice’s meter? This information may also not be particularly simple. What if Alice’s landlord owns many apartment buildings, and changes power vendors to get a better rate?
If our cryptographic infrastructure is going to enable relying parties to make the right judgments about IoT devices (such as the example provided using Smart Grid), this additional contextual information needs to be available. We can try to modify a traditional identity-based PKI to attest to these more dynamic kinds of identities, and we can also try to adapt the largely experimental world of attribute certificates to supplement the identity certificates in the smart-grid PKI. Either of these approaches will break new ground.
Alternatively, we can leave the identity PKI in place and use some other method of maintaining and distributing this additional data; which would require supplementing our scalable PKI with a non-scalable database.
In any of these approaches, we also need to think about who is authorized to make these dynamic updates or who is authoritative for making these types of attestations. Who witnesses that Alice has sold her refrigerator? Thinking about this organizational structure IoT devices also complicates the revocation problem. If we can’t quite figure out who it is that speaks for where a device currently lives, how will we figure out who it is who is authorized to say it has been compromised?
In this presentation, all of these issues and more will be explored and actionable guidelines will be proposed to build a secure and scalable system of IDs and attributes for the complex networked world that awaits us all.
The AWS platform offers a rich set of capabilities that can be leveraged by the customer to better control applications state, configuration, and supporting infrastructure throughout the service lifecycle – all while operating with security best practices such as audit and accountability, access control, change review and governance, and systems integrity. We will showcase and discuss design patterns for using these capabilities in synergy with fast-paced and agile application development methodologies – such as DevOps – to achieve an integrated security operations program.
Building IAM for OpenStack, presented at CIS (Cloud Identity Summit) 2015.
Discuss Identity Sources, Authentication, Managing Access and Federating Identities
Similar to IBM Spectrum Scale Authentication For Object - Deep Dive (20)
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...UiPathCommunity
💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™:
See how to accelerate model training and optimize model performance with active learning
Learn about the latest enhancements to out-of-the-box document processing – with little to no training required
Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages
This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath.
Speakers:
👨🏫 Andras Palfi, Senior Product Manager, UiPath
👩🏫 Lenka Dulovicova, Product Program Manager, UiPath
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Tobias Schneck
As AI technology is pushing into IT I was wondering myself, as an “infrastructure container kubernetes guy”, how get this fancy AI technology get managed from an infrastructure operational view? Is it possible to apply our lovely cloud native principals as well? What benefit’s both technologies could bring to each other?
Let me take this questions and provide you a short journey through existing deployment models and use cases for AI software. On practical examples, we discuss what cloud/on-premise strategy we may need for applying it to our own infrastructure to get it to work from an enterprise perspective. I want to give an overview about infrastructure requirements and technologies, what could be beneficial or limiting your AI use cases in an enterprise environment. An interactive Demo will give you some insides, what approaches I got already working for real.
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf91mobiles
91mobiles recently conducted a Smart TV Buyer Insights Survey in which we asked over 3,000 respondents about the TV they own, aspects they look at on a new TV, and their TV buying preferences.
Essentials of Automations: Optimizing FME Workflows with ParametersSafe Software
Are you looking to streamline your workflows and boost your projects’ efficiency? Do you find yourself searching for ways to add flexibility and control over your FME workflows? If so, you’re in the right place.
Join us for an insightful dive into the world of FME parameters, a critical element in optimizing workflow efficiency. This webinar marks the beginning of our three-part “Essentials of Automation” series. This first webinar is designed to equip you with the knowledge and skills to utilize parameters effectively: enhancing the flexibility, maintainability, and user control of your FME projects.
Here’s what you’ll gain:
- Essentials of FME Parameters: Understand the pivotal role of parameters, including Reader/Writer, Transformer, User, and FME Flow categories. Discover how they are the key to unlocking automation and optimization within your workflows.
- Practical Applications in FME Form: Delve into key user parameter types including choice, connections, and file URLs. Allow users to control how a workflow runs, making your workflows more reusable. Learn to import values and deliver the best user experience for your workflows while enhancing accuracy.
- Optimization Strategies in FME Flow: Explore the creation and strategic deployment of parameters in FME Flow, including the use of deployment and geometry parameters, to maximize workflow efficiency.
- Pro Tips for Success: Gain insights on parameterizing connections and leveraging new features like Conditional Visibility for clarity and simplicity.
We’ll wrap up with a glimpse into future webinars, followed by a Q&A session to address your specific questions surrounding this topic.
Don’t miss this opportunity to elevate your FME expertise and drive your projects to new heights of efficiency.
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Albert Hoitingh
In this session I delve into the encryption technology used in Microsoft 365 and Microsoft Purview. Including the concepts of Customer Key and Double Key Encryption.
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualityInflectra
In this insightful webinar, Inflectra explores how artificial intelligence (AI) is transforming software development and testing. Discover how AI-powered tools are revolutionizing every stage of the software development lifecycle (SDLC), from design and prototyping to testing, deployment, and monitoring.
Learn about:
• The Future of Testing: How AI is shifting testing towards verification, analysis, and higher-level skills, while reducing repetitive tasks.
• Test Automation: How AI-powered test case generation, optimization, and self-healing tests are making testing more efficient and effective.
• Visual Testing: Explore the emerging capabilities of AI in visual testing and how it's set to revolutionize UI verification.
• Inflectra's AI Solutions: See demonstrations of Inflectra's cutting-edge AI tools like the ChatGPT plugin and Azure Open AI platform, designed to streamline your testing process.
Whether you're a developer, tester, or QA professional, this webinar will give you valuable insights into how AI is shaping the future of software delivery.
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...Ramesh Iyer
In today's fast-changing business world, Companies that adapt and embrace new ideas often need help to keep up with the competition. However, fostering a culture of innovation takes much work. It takes vision, leadership and willingness to take risks in the right proportion. Sachin Dev Duggal, co-founder of Builder.ai, has perfected the art of this balance, creating a company culture where creativity and growth are nurtured at each stage.
Connector Corner: Automate dynamic content and events by pushing a buttonDianaGray10
Here is something new! In our next Connector Corner webinar, we will demonstrate how you can use a single workflow to:
Create a campaign using Mailchimp with merge tags/fields
Send an interactive Slack channel message (using buttons)
Have the message received by managers and peers along with a test email for review
But there’s more:
In a second workflow supporting the same use case, you’ll see:
Your campaign sent to target colleagues for approval
If the “Approve” button is clicked, a Jira/Zendesk ticket is created for the marketing design team
But—if the “Reject” button is pushed, colleagues will be alerted via Slack message
Join us to learn more about this new, human-in-the-loop capability, brought to you by Integration Service connectors.
And...
Speakers:
Akshay Agnihotri, Product Manager
Charlie Greenberg, Host
Connector Corner: Automate dynamic content and events by pushing a button
IBM Spectrum Scale Authentication For Object - Deep Dive
1. IBM Spectrum Scale™
Authentication (for Object
Access)
Smita Raut
Spectrum Scale Cloud and Object
Sandeep Patil
STSM, Spectrum Scale
Deepak Ghuge
Spectrum Scale Cloud and Object
2. Agenda and Flow
• Object Authentication in IBM Spectrum Scale™
• Administration – Prerequisites and Overview
• Administration – Install Toolkit Method
• Administration – Using CLI
• Administration of Unified File and Object
• Validating Object Authentication
• Creating projects, roles and setting ACLs
• Problem Determination Guide
2
3. Agenda and Flow
• Object Authentication in IBM Spectrum Scale™
• Administration – Prerequisites and Overview
• Administration – Install Toolkit Method
• Administration – Using CLI
• Administration of Unified File and Object
• Validating Object Authentication
• Creating projects, roles and setting ACLs
• Problem Determination Guide
3
4. Introduction to OpenStack Keystone
• Identity service used by OpenStack for authentication and high-level
authorization
• Supports token based authentication and user-service authorization
• Implements OpenStack’s identity API
• OpenStack Keystone packages bundled and shipped with Spectrum
Scale
• When configured , Keystone runs on all the spectrum scale protocol
nodes ensuring HA .
• Requests coming to keystone can be load balanced using DNS round
robin or HAProxy with Spectrum Scale
• Spectrum Scale supports Keystone V2.0 and V3 4
5. Spectrum Scale Object Authentication Flow
• Swift clients make a request
to keystone to get the auth
token
• Auth token is valid for a
configured duration of time,
typically 24 hrs
• Swift clients pass on this
token to Swift service to
perform Object IO
• Swift validates this token with
keystone
5
6. Supported Types for Object Authentication
• LDAP/AD(Active Directory)
• Users from LDAP(RFC2307) or AD can be used for authentication
• Support for single domain for Active Directory
• TLS supported for communication with LDAP/AD
• Kerberos, Trust in AD, LDAP referral is not supported
• Local Authentication
• Users are stored in postgresql database
• Userdefined Authentication
• If External keystone needs to be used(Advance Functionality)
• Support v2.0 and v3 keystone api
6
7. Agenda and Flow
• Object Authentication in IBM Spectrum Scale™
• Administration – Prerequisites and Overview
• Administration – Install Toolkit Method
• Administration – Using CLI
• Administration of Unified File and Object
• Validating Object Authentication
• Creating projects, roles and setting ACLs
• Problem Determination Guide
7
8. Object Authentication Prerequisites
• The system administrator needs to ensure that the authentication
server is set up properly and the connection between the IBM
Spectrum Scale™ system and authentication server is established
properly.
• Depending on the requirement, the IBM Spectrum Scale™ system
administrator needs to set up the following servers:
• Microsoft Active Directory (AD) for file and object access
• Lightweight Directory Access Protocol server for file and object access
• If external keystone is to be used, then keystone server must be configured
• Ensure the server details such as IP address or host name, admin user
name, password, base dn, and user dn are known.
8
9. Administration commands for Authentication
IBM Spectrum Scale™ can be configured with the following authentication
servers for object access:
• Active Directory (AD)
• Light Weight Directory Access Protocol (LDAP)
• Local Authentication Server (Postgres)
• User Defined Authentication (External Keystone)
Two methods available for Managing/Administering
• Spectrum Scale Installation toolkit
• Using CLI
o During Object deploy (mmobj swift base)
o After Object deploy (mmuserauth service create)
Administration
9
10. Agenda and Flow
• Object Authentication in IBM Spectrum Scale™
• Administration – Prerequisites and Overview
• Administration – Install Toolkit Method
• Administration – Using CLI
• Administration of Unified File and Object
• Validating Object Authentication
• Creating projects, roles and setting ACLs
• Problem Determination Guide
10
11. Configuring Object Authentication Using Install
Toolkit
• During First time Object Install/Enable
• Four authentication options:
- Local Authentication
- Active Directory
- LDAP
- User Defined (External Keystone)
• By default object is configured with local auth.
• Object configuration with SSL-enabled external keystone is not
supported using the install toolkit
• Cant be used for changing authentication
Administration
11
12. spectrumscale auth object
• To setup object authentication, run the
installer command-
spectrumscale auth object [-h]
[--https] [--pki]
{local,external,ldap,ad}
• This will automatically open a template file
for you to fill with the required auth
settings. TLS and SSL related settings can
also be done here. Save the file and close
it.
• If this install toolkit auth command has
been run, authentication will automatically
be enabled by the installer.
• This command must be run before running
“spectrumscale deploy”. After
deploy, object gets configured with these
authentication settings.
• This command can only be used during
initial deployment. It cannot be used on a
cluster with object deployed to configure
or change object auth.
Administration
12
Sample AD auth configuration file
[object]
remote_keystone = False
[object_auth]
enable_object_auth = True
backend_server = ad
# mandatory settings for object
authentication:
# Specifies the host name or IP address of
the authentication server.
servers =
# Specifies the base DN of the authentication
server.
base_dn =
# Specifies the DN for user search base.
user_dn =
# Specifies the user which will be assigned
the administrator role
# in Keystone.
admin_user =
# Specifies the AD user which will be used as
the swift service user.
# This user's details will be updated in
proxy-server.conf.
swift_user =
# Specifies the password of the swift_user.
# Leave as [prompt] to be prompted for the
password in
# a secure manner.
swift_password = [prompt]
Sample external keystone
auth configuration file
[object_auth]
# This installer will not
configure your external
keystone server
enable_object_auth = False
backend_server = external
[object]
remote_keystone = True
# Set to True to create swift
service, user and endpoint in
remote keystone
configure_remote_keystone =
False
# Supply the full URL for
your external keystone server
keystone_url =
http://extserver.com
13. Agenda and Flow
• Object Authentication in IBM Spectrum Scale™
• Administration – Prerequisites and Overview
• Administration – Install Toolkit Method
• Administration – Using CLI
• Administration of Unified File and Object
• Validating Object Authentication
• Creating projects, roles and setting ACLs
• Problem Determination Guide
13
14. mmobj swift base
• Used for initial configuration of object protocol when Spectrum Scale install toolkit is not used for
object deployment.
• Supports configuring local authentication or user defined authentication. It is mandatory to select
either of the authentication option.
• AD or LDAP authentication configuration is not supported through this command.
• Sample command:
mmobj swift base -g /gpfs1 -o swift --cluster-hostname
c6f1c1p1v1 --local-keystone --admin-password Passw0rd --admin-
user keystone
Note:
- admin-password parameter can be skipped on the command line if desired for security
reasons. It will be prompted in that case.
- If AD or LDAP authentication must be used, earlier auth configuration done via mmobj must
be removed and new auth with AD/LDAP to be configured using mmuserauth.
Administration
14
15. mmuserauth service Suite
• This command suite manages the authentication configuration of file and object
access protocols.
• The configuration allows protocol access methods to authenticate users who
need to access data that is stored on the system over these protocols.
• The different commands in the mmuserauth service suite are:
• mmuserauth service create - Configures authentication for file and object access
protocols.
• mmuserauth service list - Displays the details of the authentication method that is
configured for both file and object access protocols.
• mmuserauth service check - Verifies the authentication method configuration details
for file and object access protocols. Validates the connectivity to the configured
authentication servers. It also supports corrections to the configuration details on the
erroneously configured protocol nodes.
• mmuserauth service remove - Removes the authentication method configuration of
file and object access protocols and ID maps if any.
Note : use option --data-access-method object in all mmuserauth service
<operation> command for object authentication
Administration
15
16. Configuring Object with Local Authentication
Administration
mmuserauth service create –data-access-method object –type local –ks-
dns-name cesobjnode –ks-admin-user admin –ks-admin-pwd Password –ks-
swift-user swift –ks-swift-pwd Password
Verify the configuration by running:
mmuserauth service list --data-access-
method object
OBJECT access configuration : LOCAL
PARAMETERS VALUES
————————————————-
ENABLE_KS_SSL false
ENABLE_KS_CASIGNING false
KS_ADMIN_USER admin
The openrc file should look like:
export OS_AUTH_URL=”https://127.0.0.1:35357/v3
export OS_IDENTITY_API_VERSION=3
export OS_AUTH_VERSION=3
export OS_USERNAME=”admin”
export OS_PASSWORD=”Password”
export OS_USER_DOMAIN_NAME=Default
export OS_PROJECT_NAME=admin
export OS_PROJECT_DOMAIN_NAME=Default
16
Note:
- ks-admin-user is the keystone administrative user. If using local auth, this user is automatically created in postgres
database and appropriate role assigned.
- ks-swift-user is the user to be used by swift services to communicate with keystone. If using local auth, this user is
automatically created in postgres database and appropriate role assigned.
17. 17
Protocol Node 1 Protocol Node 2 Protocol Node n
CesSharedRoot
Postgresql
Keystone DB
Swift
Keystone
Swift
Keystone
Swift
Keystone
Postgres
Object Authentication with Local Authentication Postgres Keystone DB
Username-Password
Domain
Role
Project
User-Project-Role
Token
18. 18
Protocol Node 1 Protocol Node 2 Protocol Node n
CesSharedRoot
Postgresql
Keystone DB
Swift
Keystone
Swift
Keystone
Swift
Keystone
Postgres
Object Authentication with Local Authentication Postgres Keystone DB
Username-Password
Domain
Role
Project
User-Project-Role
Token
1
2
3
4
5
6
7
19. 19
Protocol Node 1 Protocol Node 2 Protocol Node n
CesSharedRoot
Postgresql
Keystone DB
Swift
Keystone
Swift
Keystone
Swift
Keystone
Postgres
Object Authentication with Local Authentication
--enable-ks-ssl
Postgres Keystone DB
Username-Password
Domain
Role
Project
User-Project-Role
Token
Secure Communication
20. Configuring Object with LDAP Authentication
Administration
mmuserauth service create –type ldap –data-access-method object –user-name
“cn=manager,dc=essldapdomain” –password “Passw0rd” –base-dn
dc=isst,dc=aus,dc=stglabs,dc=ibm,dc=com –ks-dns-name 192.168.6.99 –ks-admin-user
user1 –servers 192.168.101.55 –user-dn “ou=People,dc=essldapdomain” –ks-swift-user
swift –ks-swift-pwd Passw0rd
Verify the configuration by running:
mmuserauth service list –data-access-method object
OBJECT access configuration : LDAP
PARAMETERS VALUES
————————————————-
ENABLE_ANONYMOUS_BIND false
ENABLE_SERVER_TLS false
ENABLE_KS_SSL false
USER_NAME cn=manager,dc=essldapdomain
SERVERS 192.168.101.55
BASE_DN dc=isst,dc=aus,dc=stglabs,dc=ibm,dc=com
USER_DN ou=people,dc=essldapdomain
USER_OBJECTCLASS posixAccount
USER_NAME_ATTRIB cn
USER_ID_ATTRIB uid
USER_MAIL_ATTRIB mail
USER_FILTER none
ENABLE_KS_CASIGNING false
KS_ADMIN_USER user1
20
Note: Both the –ks-admin-user and the –
ks-swift-user specified in the command
must already exist in LDAP.
21. Configuring Object with AD Authentication
Administration
mmuserauth service create –type ad –data-access-method object –user-name
“cn=Administrator,cn=Users,dc=adcons,dc=spectrum” –password “Passw0rd3” –base-dn
“dc=adcons,dc=spectrum” –ks-dns-name 192.168.6.99 –ks-admin-user Administrator –ks-swift-user
swift –ks-swift-pwd Passw0rd2 –servers 192.168.76.50 –user-id-attrib cn –user-name-attrib
sAMAccountName –user-objectclass organizationalPerson –user-dn “cn=Users,dc=adcons,dc=spectrum”
Verify the configuration by running:
mmuserauth service list –data-access-method object
OBJECT access configuration : AD
PARAMETERS VALUES
————————————————-
ENABLE_ANONYMOUS_BIND false
ENABLE_SERVER_TLS false
ENABLE_KS_SSL false
USER_NAME cn=Administrator,cn=Users,dc=adcons,dc=spectrum
SERVERS 192.168.76.50
BASE_DN dc=adcons,dc=spectrum
USER_DN cn=users,dc=adcons,dc=spectrum
USER_OBJECTCLASS organizationalPerson
USER_NAME_ATTRIB sAMAccountName
USER_ID_ATTRIB cn
USER_MAIL_ATTRIB mail
USER_FILTER none
ENABLE_KS_CASIGNING false
KS_ADMIN_USER Administrator 21
Note: Both the –ks-admin-user and
the –ks-swift-user specified in the
command must already exist in AD
22. 22
Protocol Node 1 Protocol Node 2 Protocol Node n
CesSharedRoot
Postgresql
Keystone DB
Swift
Keystone
Swift
Keystone
Swift
Keystone
Postgres
Active Directory /
LDAP
Object Authentication with Active Directory or LDAPPostgres Keystone DB
Username-Password
Domain
Role
Project
User-Project-Role
Token
23. 23
Protocol Node 1 Protocol Node 2 Protocol Node n
CesSharedRoot
Postgresql
Keystone DB
Swift
Keystone
Swift
Keystone
Swift
Keystone
Postgres
Active Directory /
LDAP
Object Authentication with Active Directory or LDAPPostgres Keystone DB
Username-Password
Domain
Role
Project
User-Project-Role
Token
1
2
3
4
5
6
7
8
24. Configuring Object Authentication with TLS
Administration
–enable-server-tls needs to be specified in the mmuserauth
command in order to configure server TLS.
E.g. the command to configure AD-TLS would look like:
mmuserauth service create –type ad –data-
access-method object –user-name
“cn=Administrator,cn=Users,dc=adcons,dc=spectrum
” –password “Passw0rd3” –base-dn
“dc=adcons,dc=spectrum” –ks-dns-name
192.168.6.99 –ks-admin-user Administrator –ks-
swift-user swift –ks-swift-pwd Passw0rd2 –
servers 192.168.76.50 –user-id-attrib cn –user-
name-attrib sAMAccountName –user-objectclass
organizationalPerson –user-dn
“cn=Users,dc=adcons,dc=spectrum” –enable-server-
tls
Verify the configuration by running:
mmuserauth service list –data-access-method object
OBJECT access configuration : AD
PARAMETERS VALUES
————————————————-
ENABLE_ANONYMOUS_BIND false
ENABLE_SERVER_TLS true
ENABLE_KS_SSL false
USER_NAME cn=Administrator,cn=Users,dc=a
dcons,dc=spectrum
SERVERS 192.168.76.50
BASE_DN dc=adcons,dc=spectrum
USER_DN cn=users,dc=adcons,dc=spectrum
USER_OBJECTCLASS organizationalPerson
USER_NAME_ATTRIB sAMAccountName
USER_ID_ATTRIB cn
USER_MAIL_ATTRIB mail
USER_FILTER none
ENABLE_KS_CASIGNING false
KS_ADMIN_USER Administrator
In order to configure Object with AD-TLS or LDAP-TLS, copy the TLS certificate to local ces node
from where CLI will be run. The TLS certificate should be named object_ldap_cacert.pem and
copied to /var/mmfs/tmp
24
25. 25
Protocol Node 1 Protocol Node 2 Protocol Node n
CesSharedRoot
Postgresql
Keystone DB
Swift
Keystone
Swift
Keystone
Swift
Keystone
Postgres
Active Directory /
LDAP
Object Authentication with Active Directory or LDAP
--enable-server-tls
Secure Communication
Postgres Keystone DB
Username-Password
Domain
Role
Project
User-Project-Role
Token
26. Configuring Object Authentication with
Keystone https (SSL)
Administration
mmuserauth service create –data-access-method object –type local
–ks-dns-name cesobjnode –enable-ks-ssl –ks-admin-user admin –ks-
admin-pwd Password –ks-swift-user swift –ks-swift-pwd Password
Verify the configuration by running:
mmuserauth service list --data-
access-method object
OBJECT access configuration : LOCAL
PARAMETERS VALUES
————————————————-
ENABLE_KS_SSL true
ENABLE_KS_CASIGNING false
KS_ADMIN_USER admin
The openrc file should look like:
export OS_CACERT=”/etc/keystone/ssl/certs/ssl_cacert.pem”
export OS_AUTH_URL=”https://cesobjnode:35357/v3;
export OS_IDENTITY_API_VERSION=3
export OS_AUTH_VERSION=3
export OS_USERNAME=”admin”
export OS_PASSWORD=”Password”
export OS_USER_DOMAIN_NAME=Default
export OS_PROJECT_NAME=admin
export OS_PROJECT_DOMAIN_NAME=Default
26
Pre-requisite: Get set of ssl certificate(Private key, Certificate and Cacert) and copy these files to /var/mmfs/tmp
27. 27
Protocol Node 1 Protocol Node 2 Protocol Node n
CesSharedRoot
Postgresql
Keystone DB
Swift
Keystone
Swift
Keystone
Swift
Keystone
Postgres
Active Directory /
LDAP
Object Authentication with Active Directory or LDAP
--enable-ks-ssl
Postgres Keystone DB
Username-Password
Domain
Role
Project
User-Project-Role
Token
Secure Communication
28. 28
Protocol Node 1 Protocol Node 2 Protocol Node n
CesSharedRoot
Postgresql
Keystone DB
Swift
Keystone
Swift
Keystone
Swift
Keystone
Postgres
Active Directory /
LDAP
Object Authentication with Active Directory or LDAP
--enable-server-tls & --enable-ks-ssl
Secure Communication
Postgres Keystone DB
Username-Password
Domain
Role
Project
User-Project-Role
Token
29. Configuring User Defined Object Authentication with
External Keystone Server
Administration
mmuserauth service create --data-access-method object --type
userdefined --ks-ext-endpoint http://192.168.126.156:35357/v3 --ks-
swift-user swift --ks-swift-pwd password
Verify the configuration by running:
mmuserauth service list --data-access-
method object
OBJECT access configuration : USERDEFINED
PARAMETERS VALUES
------------------------------------------
-------
The openrc file should look like:
# Mon May 2 13:58:12 IST 2016
export OS_AUTH_URL=http://192.168.126.156:35357/v3
export OS_IDENTITY_API_VERSION=3
export OS_AUTH_VERSION=3
export OS_USERNAME=”admin”
export OS_PASSWORD=
export OS_USER_DOMAIN_NAME=Default
export OS_PROJECT_NAME=admin
export OS_PROJECT_DOMAIN_NAME=Default
29
30. 30
Protocol Node 1 Protocol Node 2 Protocol Node n
CesSharedRoot
Swift Swift
Keystone
Swift
Object Authentication with External Keystone
31. 31
Protocol Node 1 Protocol Node 2 Protocol Node n
CesSharedRoot
Swift Swift
Keystone
Swift
Object Authentication with External Keystone
1
3
4
5
2
32. 32
Protocol Node 1 Protocol Node 2 Protocol Node n
CesSharedRoot
Swift Swift
Keystone
Swift
Secure Communication
Object Authentication with External Keystone : --enable-ks-ssl
33. Verifying the authentication services
configured in the system
• mmuserauth service check --data-access-method object [-
N|--nodes] {node-list|cesNodes} [--server-reachability]
[-r|--rectify ]
• The mmuserauth service check command helps to
check whether the authentication configuration is consistent
across the cluster and the required services are enabled and
running.
• This command validates and corrects the authentication
configuration files and starts any associated services if
needed.
Administration
33
34. Deleting authentication and ID mapping
configuration
• Deleting the authentication and ID mapping configuration results
in loss of access to data.
• Object IDMapping = Relationship { user-project-role }
(mmuserauth service remove –data-access-method object—
idmapdelete - delete this relationship)
• Issue the mmuserauth service remove command to
remove the authentication configuration as shown in the
following example:
# mmuserauth service remove –data-acess-method object
mmcesuserauthrmservice: Command successfully completed.
• # mmuserauth service remove –data-acess-method object --
idmapdelete
Administration
34
35. Modifying the authentication method
IMPORTANT:
• Modification = remove + create
• Modifying authentication method should only be done during pre-production phase where
customer trying to see which mechanism really suite his requirement.
• If data already exists or is created with the existing authentication, it is not recommended to
change the authentication. It might result in access to data loss or unauthorized access.
• There is support for changing authentication parameter but limited to only set parameter(refer
documentation). Eg : Change of ldap/ad server ip, password change,ldap filter change.
• Object Authentication parameter should only be changed via mmobj config change command.
Note – mmobj config change is object only command, it does not applies to file authentication
• Parameter updated via mmobj config change command does not reflect in mmuser service list
command.
Administration
35
36. Agenda and Flow
• Object Authentication in IBM Spectrum Scale™
• Administration – Prerequisites and Overview
• Administration – Install Toolkit Method
• Administration – Using CLI
• Administration of Unified File and Object
• Validating Object Authentication
• Creating projects, roles and setting ACLs
• Problem Determination Guide
36
37. Configuring Object Authentication for
Unified File and Object
Administration
Local_mode - Separate identity between object and file
(Default mode)
• Object authentication setup is independent of file
authentication setup
Unified_mode - Shared identity between object and file
• Supported only with Active Directory (AD) with UNIX-
mapped domains and LDAP authentication
configurations
• Authentication for both file and object access must be
configured and the authentication schemes must be
the same and configured with the same server
37
Ref: Video of presentation done on this topic at OpenStack summit April 2016 in Austin-
https://www.youtube.com/watch?v=6ovLb6aktbM&t=93s
38. Unified File and Object – unified_mode of ID Mapping
Administration
38
• Users from object and file are expected to be common and coming from the same directory service (only
AD+RFC 2307 or LDAP)
• Object created from the object interface is owned by the user doing the object PUT operation
• If the object already exists, existing ownership of the corresponding file is retained if retain_owner is set to yes
in object-server-sof.conf
• Object access follows the object ACL semantics and file access follows the file ACL semantics
• If the object is created or updated over existing file then existing file ACL, xattrs, and winattrs are retained if
retain_acl, retain_xattr, and retain_winattr are set to yes in object-server-sof.conf
• Security or system extended attributes and other IBM Spectrum Scale extended attributes such as immutability,
pcache, etc. are not retained
• Swift metadata (user.swift.metadata) is also not retained and it is replaced according to object semantics
• Change id_mgmt in the object-server-sof.conf file using the mmobj config change command as follows
mmobj config change --ccrfile object-server-sof.conf --section DEFAULT --
property id_mgmt --value unified_mode
• If object authentication is configured with AD, set ad_domain in the object-server-sof.conf file
mmobj config change --ccrfile object-server-sof.conf --section DEFAULT --
property ad_domain --value POLLUX
• List the currently configured id_mgmt mode using the mmobj config list command as follows
mmobj config list --ccrfile object-server-sof.conf --section DEFAULT --
property id_mgmt
39. Agenda and Flow
• Object Authentication in IBM Spectrum Scale™
• Administration – Prerequisites and Overview
• Administration – Install Toolkit Method
• Administration – Using CLI
• Administration of Unified File and Object
• Validating Object Authentication
• Creating projects, roles and setting ACLs
• Problem Determination Guide
39
40. Validating Object Authentication Using
Openstack client
• Swift and
openstack
clients are
installed on
CES nodes by
default
• They use
environment
variables from
openrc if not
specified in
commandline
• Keystone
AD/LDAP
interface is
read-only and
new users
cannot be
created
through
keystone
40
List current keystone endpoints:
List current projects: List current users
Show current defined roles:
41. Agenda and Flow
• Object Authentication in IBM Spectrum Scale™
• Administration – Prerequisites and Overview
• Administration – Install Toolkit Method
• Administration – Using CLI
• Administration of Unified File and Object
• Validating Object Authentication
• Creating projects, roles and setting ACLs
• Problem Determination Guide
41
42. Creating Users, Projects, Roles and Setting ACLS
42
Create a new role: Create a new project:
Assign new role to the user:
Setting ACLs on container:
All these operations can also
be performed through GUI
Create a new user (only for local auth):
43. Agenda and Flow
• Object Authentication in IBM Spectrum Scale™
• Administration – Prerequisites and Overview
• Administration – Install Toolkit Method
• Administration – Using CLI
• Administration of Unified File and Object
• Validating Object Authentication
• Creating projects, roles and setting ACLs
• Problem Determination Guide
43
44. Problem Determination Guide
This section describes the following:
• Monitoring IBM Spectrum Scale™
• Collecting details of issues using available methods
• Usecases/Common Problems
• Debugging
44
45. Monitoring IBM Spectrum Scale™
Monitoring GUI:
• Monitoring -> Events Page in the GUI allows you to review the set of events that are
reported in the IBM Spectrum Scale™ system.
• You can filter the Events as Current Issues, Unread Issues and All Events.
• You can also determine if the event is Informational, Warning or an Error.
• You can mark the even as Read and also resolve some issues by Running a Fixed
Procedure. “Run Fix Procedure” Action helps to do so.
• The system can also use SMTP traps and email to notify you of an event.
• Settings -> Event Notifications Page allows to do so.
• Notifications are usually sent immediately after an event is raised.
• Reports of all events can also be sent. Reports are sent once a day.
• You can configure Email Notification for receiving emails for Authentication Events.
• To create email recipients, select Email Recipients from the Event Notifications page,
and then click Create Recipient
• Refer the Knowledge center to know more on how to setup up SMTP Manager.
Problem Determination Guide
45
46. Monitoring IBM Spectrum Scale™
Monitoring using CLI
• The mmhealth command is used to monitor the health status of the system and the
services running on the nodes.
• The sub-components of CES service such as NFS, SMB, Object, and authentication have
their own health monitors.
• The mmhealth command gets the health details from these monitoring services.
• Monitoring health of CES Node:
• Node role: This node role is active on the CES nodes that are listed by
mmlscluster --ces.
• Once a node obtains this role, all corresponding CES sub-services are activated on that node.
• The CES service does not have its own monitoring service or events. The status of the CES is an
aggregation of the status of its sub-services.
• The following few sub-services are monitored: (Refer the knowledge center for more sub-services)
a. AUTH – Tasks: Monitors LDAP, AD and or NIS-based authentication services.
b. AUTH_OBJ – Tasks: Monitoring the OpenStack identity service functionalities.
c. OBJECT – Tasks: Monitors the IBM Spectrum Scale™ for object functionality. Especially, the status of relevant
system services and accessibility to ports are checked.
Problem Determination Guide
46
47. Monitoring IBM Spectrum Scale™
The following are the possible status of nodes and services:
• UNKNOWN - Status of the node or the service hosted on the node is not known.
• HEALTHY - The node or the service hosted on the node is working as expected. There are no
active error events.
• CHECKING - The monitoring of a service or a component hosted on the node is starting at the
moment. This state is a transient state and is updated when the startup is completed.
• TIPS - There might be an issue with the configuration and tuning of the components. This status is
only assigned to a Tip event.
• DEGRADED - The node or the service hosted on the node is not working as expected. That is, a
problem occurred with the component but it did not result in a complete failure.
• FAILED - The node or the service hosted on the node failed due to errors or cannot be reached
anymore.
• DEPEND - The node or the services hosted on the node have failed due to the failure of some
components. For example, an NFS or SMB service shows this status if authentication has failed.
Problem Determination Guide
47
48. Collecting details of the issue
Collecting details of the issue involves collecting data using gpfs.snap for:
1. Authentication
2. Object Protocol
Authentication Related – For such issues, gpfs.snap command would
collect all Authentication configuration and error logs. Also the different log
files for the authentication components can be checked.
Object Protocol Related – For such issues, gpfs.snap command would
collect keystone and http server related configuration and logs.
Problem Determination Guide
48
49. LDAP Attributes related issuesScenario : Object authentication with LDAP. The default values of mmuserauth option are not matching with actual values on LDAP
[root@c1n4 ~]# mmuserauth service create --data-access-method object --type ldap --servers 192.168.122.27 --
user-name administrator@sonas.com --password Passw0rd --user-dn dc=sonas,dc=com --base-dn dc=sonas,dc=com --ks-
admin-user administrator --ks-swift-user longnameuser --ks-swift-pwd Passw0rd --ks-dns-name c1ces
[E] Didn't find entry for user administrator with ldap search mmuserauth service create: Command failed. Examine
previous error messages to determine cause.
The command failed because it is not able to find user administrator using the option specified and default options. One need to specify the option explicitly on
command line if default values are not matching with the LDAP server environment.
49
Default values of mmuserauth when –type=ldap
and –data-access-method=object
--user-objectclass= posixAccount
--user-name-attrib= cn
--user-id-attrib=uid
[root@c1n4 ~]# ldapsearch -x -h 192.168.122.27 -b dc=sonas,dc=com -D administrator@sonas.com -w Passw0rd
cn=administrator
# extended LDIF
# Administrator, Users, SONAS.COM
dn: CN=Administrator,CN=Users,DC=SONAS,DC=COM
objectClass: top
objectClass: person
objectClass: organizationalPerson
objectClass: user
cn: Administrator
distinguishedName: CN=Administrator,CN=Users,DC=SONAS,DC=COM
name: Administrator
objectGUID:: gWYgEijUAkG6rDsjQ6fC7A==
sAMAccountName: Administrator
sAMAccountType: 805306368
uid: Administrator
mail: administrator@sonas.com
uidNumber: 20021
gidNumber: 21000
unixHomeDirectory: /home/Administrator
loginShell: /bin/sh
50. 50
Executing mmuserauth again with specifying the default option with correct values on command line itself
[root@c1n4 ~]# /usr/lpp/mmfs/bin/mmuserauth service create --data-access-method object --type ldap --servers 192.168.122.27
--user-name administrator@sonas.com --password Passw0rd --user-dn dc=sonas,dc=com --base-dn dc=sonas,dc=com --ks-admin-user
administrator --ks-swift-user longnameuser --ks-swift-pwd Passw0rd --ks-dns-name c1ces --user-objectclass
organizationalPerson --user-id-attrib CN --user-name-attrib uid
mmcesobjcrbase: Validating execution environment.
mmcesobjcrbase: Performing SELinux configuration.
mmcesobjcrbase: Configuring Keystone server in /ibm/gpfs0/ces/object/keystone.
mmcesobjcrbase: Initiating action (start) on postgres in the cluster.
mmcesobjcrbase: Validating Keystone environment.
mmcesobjcrbase: Validating Swift values in Keystone.
mmcesobjcrbase: Configuration complete.
Object configuration with LDAP as the identity backend has completed successfully.
Object authentication configuration completed successfully.
[root@c1n4 ~]# /usr/lpp/mmfs/bin/mmuserauth service list --data-access-method object
OBJECT access configuration : LDAP
PARAMETERS VALUES
-------------------------------------------------
ENABLE_ANONYMOUS_BIND false
ENABLE_SERVER_TLS false
ENABLE_KS_SSL false
USER_NAME administrator@sonas.com
SERVERS 192.168.122.27
BASE_DN dc=sonas,dc=com
USER_DN dc=sonas,dc=com
USER_OBJECTCLASS organizationalPerson
USER_NAME_ATTRIB uid
USER_ID_ATTRIB CN
USER_MAIL_ATTRIB mail
USER_FILTER none
ENABLE_KS_CASIGNING false
KS_ADMIN_USER administrator
[root@c1n4 ~]# /usr/lpp/mmfs/bin/mmuserauth service check
--data-access-method object -N cesNodes
Userauth object check on node: c1n3
Checking keystone.conf: OK
Checking wsgi-keystone.conf: OK
Checking /etc/keystone/ssl/certs/signing_cert.pem: OK
Checking /etc/keystone/ssl/private/signing_key.pem: OK
Checking /etc/keystone/ssl/certs/signing_cacert.pem: OK
Service 'httpd' status: OK
Userauth object check on node: c1n4
Checking keystone.conf: OK
Checking wsgi-keystone.conf: OK
Checking /etc/keystone/ssl/certs/signing_cert.pem: OK
Checking /etc/keystone/ssl/private/signing_key.pem: OK
Checking /etc/keystone/ssl/certs/signing_cacert.pem: OK
Service 'httpd' status: OK
[root@c1n4 ~]#
LDAP Attributes related issues…Continue
51. External Keystone – Consideration and Issues
• api supported by external keystone V2.0 or v3?
• ssl/non-ssl - In case of ssl CN is ssl certificate should match with hostname is keystoneURL
• <swiftuser> must exist in external keystone, and it should have ‘admin’ role in ‘service’ project in ’Default’
domain
• Validating external keystone before configuration
For v3:
export OS_AUTH_URL=“<keystoneURL>”
export OS_IDENTITY_API_VERSION=3
export OS_AUTH_VERSION=3
export OS_USERNAME=“<swift user>”
export OS_PASSWORD=“<swift Password>”
export OS_USER_DOMAIN_NAME=Default
export OS_PROJECT_NAME=service
export OS_PROJECT_DOMAIN_NAME=Default
openstack --insecure role list --user <swiftUser> --project service -f value -c Name
In case of ssl - openstack --os-cacert <cacert path> role list --user <swiftUser> --project service -f value -c Name
Command should return the ‘admin’ role
Once pre-validation is successful use following command to configure authentication service
# mmuserauth service create --data-access-method object --type userdefined --ks-ext-endpoint
http://192.168.126.156:35357/v3 --ks-swift-user swift --ks-swift-pwd password
For v2.0:
/usr/bin/keystone [--os-cacert <cacert path>] --os-username <swiftUser> --os-password <swiftPassword> --os-tenant-name
service --os-auth-url <keystoneURL> user-role-list --user <swiftUser>
Command should return the ‘admin’ role
Once pre-validation is successful use following command to configure authentication service
#mmuserauth service create --data-access-method object --type userdefined --ks-ext-endpoint
http://192.168.126.156:35357/v2.0 --ks-swift-user swift --ks-swift-pwd password
Note : mmuserauth try to find the api version by querying the external keystone if not specified in keystoneURL. It is recommended to have api version in keystoneURL
51
52. SSL Certificate related issue
52
Scenario : Object authentication is configured with SSL using certificate having CN that does not match hostname(Endpoint)
[root@c1n4 ~]# openstack user list
Discovering versions from the identity service failed when creating the password plugin. Attempting to determine version from
URL.
SSL exception connecting to https://192.168.126.180:35357/v3/auth/tokens: hostname ’192.168.126.180' doesn't match u'c1ces’
Issue : CN in the ssl certificate(c1ces) is not matching with hostname(192.168.126.180)
Check the CN used in certificate by using following command
[root@c1n4 ~]# openssl x509 -in /var/mmfs/tmp/ssl_cert.pem -noout -purpose -text
Certificate purposes:
SSL client : Yes
SSL client CA : No
. . . . .
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 1 (0x1)
Signature Algorithm: sha1WithRSAEncryption
Issuer: C=US, ST=Unset, L=Unset, O=Unset, CN=c1ces
Validity
Not Before: May 11 10:03:12 2017 GMT
Not After : May 9 10:03:12 2027 GMT
Subject: C=US, ST=Unset, O=Unset, CN=c1ces
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
[root@c1n4 tmp]# /usr/lpp/mmfs/bin/mmuserauth service create --type local --data-access-method object --ks-admin-user deepak --
ks-admin-pwd password --enable-ks-ssl --ks-dns-name c1ces
53. LDAP/AD - TLS certificate Related issue
53
Scenario : Object authentication with TLS enabled LDAP. The CN used in TLS certificate on LDAP server and the IP/Hostname being used in mmuserauth are different.
[root@c1n4 tmp]# /usr/lpp/mmfs/bin/mmuserauth service create --data-access-method object --type ldap --servers
192.168.122.27 --user-name administrator@sonas.com --password Passw0rd --user-dn dc=sonas,dc=com --base-dn
dc=sonas,dc=com --ks-admin-user administrator --ks-swift-user longnameuser --ks-swift-pwd Passw0rd --ks-dns-name
c1ces --user-objectclass organizationalPerson --user-id-attrib CN --user-name-attrib uid --enable-server-tls
[E] Failed to execute command ldapsearch
mmuserauth service create: Command failed. Examine previous error messages to determine cause.
# Check using ldapsearch command whether LDAP communication succeeds with provided certificate
[root@c1n4 tmp]# export LDAPTLS_CACERT=/var/mmfs/tmp/object_ldap_cacert.pem; ldapsearch -x -h 192.168.122.27 -b
dc=sonas,dc=com -D administrator@sonas.com -w Passw0rd cn=administrator -ZZZ
ldap_start_tls: Connect error (-11)
additional info: TLS error -8179:Peer's Certificate issuer is not recognized.
# Above error shows the certificate is invalid
# Get the correct certificate for LDAP/AD admin and try same command.
export LDAPTLS_CACERT=/var/mmfs/tmp/object_ldap_cacert.pem; ldapsearch -x -h 192.168.122.27 -b dc=sonas,dc=com -
D administrator@sonas.com -w Passw0rd cn=administrator -ZZZ
ldap_start_tls: Connect error (-11)
additional info: TLS: hostname does not match CN in peer certificate
#Above error shows Certificate is correct but the CN of certificate is not matching with the LDAP hostname
# Get the correct name from LDAP/AD using which TLS certificate is issued and try ldapsearch again
[root@c1n4 tmp]# export LDAPTLS_CACERT=/var/mmfs/tmp/object_ldap_cacert.pem; ldapsearch -x -h w2k8-phy-
sonas.sonas.com -b dc=sonas,dc=com -D administrator@sonas.com -w Passw0rd cn=administrator -ZZZ
# extended LDIF
# Administrator, Users, SONAS.COM
dn: CN=Administrator,CN=Users,DC=SONAS,DC=COM
objectClass: top
objectClass: person
……
54. 54
Executing mmuserauth with valid tls certificate and CN of LDAP server
[root@c1n4 tmp]# /usr/lpp/mmfs/bin/mmuserauth service create --data-access-method object --type ldap --servers w2k8-phy-sonas.sonas.com --user-name
administrator@sonas.com --password Passw0rd --user-dn dc=sonas,dc=com --base-dn dc=sonas,dc=com --ks-admin-user administrator --ks-swift-user longnameuser --ks-
swift-pwd Passw0rd --ks-dns-name c1ces --user-objectclass organizationalPerson --user-id-attrib CN --user-name-attrib uid --enable-server-tls
mmcesobjcrbase: Validating execution environment.
mmcesobjcrbase: Performing SELinux configuration.
mmcesobjcrbase: Configuring Keystone server in /ibm/gpfs0/ces/object/keystone.
mmcesobjcrbase: Initiating action (start) on postgres in the cluster.
mmcesobjcrbase: Validating Keystone environment.
mmcesobjcrbase: Validating Swift values in Keystone.
mmcesobjcrbase: Configuration complete.
Object configuration with LDAP as the identity backend has completed successfully.
Object authentication configuration completed successfully.
[root@c1n4 tmp]# /usr/lpp/mmfs/bin/mmuserauth service list --data-access-method object
OBJECT access configuration : LDAP
PARAMETERS VALUES
-------------------------------------------------
ENABLE_ANONYMOUS_BIND false
ENABLE_SERVER_TLS true
ENABLE_KS_SSL false
USER_NAME administrator@sonas.com
SERVERS w2k8-phy-sonas.sonas.com
BASE_DN dc=sonas,dc=com
USER_DN dc=sonas,dc=com
USER_OBJECTCLASS organizationalPerson
USER_NAME_ATTRIB uid
USER_ID_ATTRIB CN
USER_MAIL_ATTRIB mail
USER_FILTER none
ENABLE_KS_CASIGNING false
KS_ADMIN_USER administrator
[root@c1n4 tmp]# /usr/lpp/mmfs/bin/mmuserauth service check --data-access-
method object -N cesNodes
Userauth object check on node: c1n3
Checking keystone.conf: OK
Checking wsgi-keystone.conf: OK
Checking /etc/keystone/ssl/certs/signing_cert.pem: OK
Checking /etc/keystone/ssl/private/signing_key.pem: OK
Checking /etc/keystone/ssl/certs/signing_cacert.pem: OK
Checking /etc/keystone/ssl/certs/object_ldap_cacert.pem: OK
Service 'httpd' status: OK
Userauth object check on node: c1n4
Checking keystone.conf: OK
Checking wsgi-keystone.conf: OK
Checking /etc/keystone/ssl/certs/signing_cert.pem: OK
Checking /etc/keystone/ssl/private/signing_key.pem: OK
Checking /etc/keystone/ssl/certs/signing_cacert.pem: OK
Checking /etc/keystone/ssl/certs/object_ldap_cacert.pem: OK
Service 'httpd' status: OK
LDAP/AD - TLS certificate Related issue…Continue
55. 55
Scenario : Object authentication is configured LDAP and LDAP server is not reachable from one or more protocol nodes.
[root@c1n3 ~]# openstack user list
An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-237a98d6-9973-4251-9ae7-
f118eb214804)
[root@c1n3 ~]# /usr/lpp/mmfs/bin/mmuserauth service check --data-access-method object --server-reachability
Userauth object check on node: c1n3
Checking keystone.conf: OK
Checking wsgi-keystone.conf: OK
Checking /etc/keystone/ssl/certs/signing_cert.pem: OK
Checking /etc/keystone/ssl/private/signing_key.pem: OK
Checking /etc/keystone/ssl/certs/signing_cacert.pem: OK
Checking /etc/keystone/ssl/certs/object_ldap_cacert.pem: OK
LDAP servers status
LDAP server w2k8-phy-sonas.sonas.com : ERROR
Service 'httpd' status: OK
[root@c1n3 ~]# /usr/lpp/mmfs/bin/mmuserauth service check --data-access-method object --server-reachability
Userauth object check on node: c1n3
Checking keystone.conf: OK
Checking wsgi-keystone.conf: OK
Checking /etc/keystone/ssl/certs/signing_cert.pem: OK
Checking /etc/keystone/ssl/private/signing_key.pem: OK
Checking /etc/keystone/ssl/certs/signing_cacert.pem: OK
Checking /etc/keystone/ssl/certs/object_ldap_cacert.pem: OK
LDAP servers status
LDAP server w2k8-phy-sonas.sonas.com : OK
Service 'httpd' status: OK
[root@c1n3 ~]# openstack user list
+------------------------------+---------------+
| ID | Name |
+------------------------------+---------------+
| Administrator | Administrator |
LDAP/AD server is not reachable
56. Commonly occurring issues
AD/LDAP
• LDAP/AD is not reachable – Network issue/Server Down/Firewall issue
• LDAP/AD bind password changed/User deleted/Lockef/Permission changed
• LDAP/AD TLS certificate expired
• Swift user password changed/expired
• Swift user account locked/disabled
• Role of swift user removed from service account
External Keystone(Userdefined)
• External keystone is not rechable - Network issue/Server Down/Firewall issue
• Swift user deleted/Password Changed/Account locked
• Role changes on External Keystone
Local
• Postgresql is not running
• Password of swift user changed / User deleted
• Role change
56
57. Debugging
Check output of following commands
$mmuserauth service list
$mmuserauth service check --data-access-method object --nodes cesNodes
$mmuserauth service check --data-access-method object --nodes cesNodes --server-reachability
$mmces service list –v –a
$mmces events active
Enable Debugging :
1. CLI debugging - $mmces log level 3
2. Enabling debugging of keystone - /usr/lpp/mmfs/bin/mmobj config change --ccrfile keystone.conf --section DEFAULT
--property debug --value true
Note : Disable the debugging once problem is resolved. Debugging create lots of logs.
****** Do not modify any configuration file manually ******
Log Files to check if issue with object authentication
1. /var/adm/ras/mmfs.log*
2. /var/log/keystone/*
3. /var/log/message
4. /var/log/secure
5. /var/adm/ras/mmsysmonitor.log
Problem Determination Guide
57
Spectrum Scale object uses this keystone service for authentication
Packages are bundled with spectrum-scale-object rpm
Various swift clients available, e.g. cyberduck, openstack swift client, write your own client using CURL (tool for transferring data to and from server using supported protocols like http, https etc)
Auth token is generated for the user and stored in portgres database
Keystone token has info of user, role, expiry time, endpoint
In case of PKI these params are encrypted. This secures ks to swift communication.
For object AD/LDAP are same.
We support single domain not multi domain.
Trust concept in AD not supported in object (i.e. keystone).
in case of multi-cluster configurations one can be local but other should be external to this
Keystone token has info of user, role, expiry time, endpoint. In case of PKI these params are encrypted. This secures ks to swift communication
AD auth configuration and LDAP auth configuration file templates are same. Specify appropriate backend_server.
For local auth configuration, no config file template is needed
This command is also used when object is disabled and needs to be re-enabled again
- If using AD or LDAP, the ks-admin-user and ks-swift user refer to AD or LDAP users and must exist on the server.
Swift clients communicate with keystone and swift services running on protocol nodes
Swift services communicate with keystone services running on local node for requests like token verification
Keystone service from every protocol node communicates with the portgres service running on singleton node (designation can be found using mmces address list or mmces node list)
Postgres service deals with the data stored in postgres database on cesSharedRoot
SwiftClient sends the Username,Password,Project etc to Keystone
Keystone connect with Postgres running on one of the protocol node
Postgres validate the user, password Project, role etc by looking into database which is in cessharedroot and issue the TOKEN
SwiftClient receives the TOKEN
SwiftClient send request to swift for object/container/account with TOKEN
Swift Validate the TOKEN
SwiftClient receive the data from swift
In this case credentials are not stored in postgres DB
Keystone service running on every protocol node communicates with the AD/LDAP server to perform authentication
SwiftClient sends the Username,Password,Project etc to Keystone
Keystone connect to Active Directory or LDAP to validate the username and password
Keystone connect with Postgres running on one of the protocol node
Postgres validate the user, Project, role etc by looking into database which is in cessharedroot and issue the TOKEN
SwiftClient receives the TOKEN
SwiftClient send request to swift for object/container/account with TOKEN
Swift Validate the TOKEN
SwiftClient receive the data from swift
Keystone is not setup on the cluster but an external keystone server is used
SwiftClient sends the Username,Password,Project etc to Keystone
Keystone provide the TOKEN by validating username,password,project etc
SwiftClient send request to swift for object/container/account with TOKEN
Swift Validate the TOKEN
SwiftClient receive the data from swift
In case of local auth, users can be created using openstack user create command
Openstack user create cannot be used to create users on AD/LDAP. AD/LDAP interface for keystone is read-only