Pre-Con Ed: Who's minding the SSO store?

World®
’16
Who’s Minding your SSO Store?
Jason Wilcox Sr. Services Architect
SCX16E
SECURITY

2 © 2016 CA. ALL RIGHTS RESERVED.@CAWORLD #CAWORLD
© 2016 CA. All rights reserved. All trademarks referenced herein belong to their respective companies.
The content provided in this CA World 2016 presentation is intended for informational purposes only and does not form any type of
warranty. The information provided by a CA partner and/or CA customer has not been reviewed for accuracy by CA.
For Informational Purposes Only
Terms of this Presentation

Abstract
At the heart of your enterprise security infrastructure is your CA Single Sign On web access
management environment. Some take its simplicity for granted, the silent workhorse that
provides a great user experience across your apps with high performance and reliability. But
who and what is keeping tabs on your CA SSO application?
In this session, we will explore best practices, methods and tools that can be deployed to
monitor the health of your mission critical web access management solution. We will cover:
The key aspects of what impacts the performance and stability of your CA Single Sign On
solution How to trace a problem to root cause using tools like CA Trace Log Reader, APM for SSO,
Spylogix, and Splunk. How to create a monitoring and alerting strategy for your CA Single Sing On
Solution. How to use monitoring data to tune and optimize your CA Single Sign On Solution
components
Jason Wilcox
CA technologies
Sr. Services Architect

Agenda
WHO’S TALKING TO MY POLICY SERVER, AND WHY DO I CARE?
MST = T/ATT(S), YES MATH MATTERS
ESTABLISHING A PROACTIVE MONITORING PROGRAM
WHAT ARE THE SSO KPI’S
HOW DO I MONITOR THOSE KPI’S
BUY WHY DOES IT MATTER?
1
2
3
4
5
6

Agents Connection
§ Web Agent opens the TCP
connection to the Policy Server
– By default, under load, the web
agent will open a max of 20 sockets
– Connections are “long-lived”
§ Policy Server closes connections
– Idle Timeout (minutes) in
SMConsole
– Non-Idle Timeout via
AgentConnectionMaxLifetime in XPS
Object
Agents Open the Connection, the Policy Server Can Close the Connection
Authorization
Authentication
Administration
Accounting
Administrative UIUser Store
Policy Server
Protected Resources
Policy Store
Web Server
Web Agent

Agent Conversation
§ Individual sockets are
synchronous connections
§ Once the Web Agent “asks a
question” to the policy server,
that socket connection is “busy”
until the Policy Server responds
The Conversation is a Request/Response Model, No Interruptions Allowed
Authorization
Authentication
Administration
Accounting
Policy Server
Protected Resources
Policy Store
Web Server
Web Agent

Agent Security
§ Agent API initiates a connection
with the Trusted Host Name and
Shared Secret
– Handshake includes establishing
encrypted channel
– Retrieves operational parameters
from the HCO
– Retrieves configuration parameters
from the ACO (Or LocalConfig)
The Conversation Starts with Authentication, and Created an Encrypted Channel
Authorization
Authentication
Administration
Accounting
Policy Server
Protected Resources
Policy Store
Web Server
Web Agent

Topics they Discuss
§ Over this TCP connection the
Web Agent sends the following
Agent API commands:
– isProtected()
– isAuthenticated()
– isAuthorized()
§ Not the only commands but
those are the primary functions
of the Agent.
Agent to Policy Server Communication is Primarily Request/Response
Authorization
Authentication
Administration
Accounting
Policy Server
Protected Resources
Policy Store
Web Server
Web Agent

The Policy Server responds
§ We can see the reactor
thread taking these
requests and putting
them in the queue
§ Now on the other end
of the queue, we can
see what are called
“Worker threads” to
handle the work
When the Agent Comes Knocking, Who Answers?
Web
Agent
Reactor
Thread
Worker
Thread
Policy Server

Configuring Worker Threads
§ The worker threads are
configured here in the
management console
– default is for 20 worker
threads in CA SiteMinder
R12.5x
§ How many do you need?
Threads Never Die, Once Reached They Will Always At Max

What Do Worker Threads Do?
§ Worker threads do the work!
§ The worker threads take the item off the queue, and go to the
User Store, Policy Store, Session Store, cache, etc…
§ Worker threads generate assertions
§ Worker threads process xml
§ Worker threads do everything the policy server needs to do
Before I Can tell You How Many You Need, We Need to Know What they are Doing?

Thread Locking
§ Similar to the Agent socket request, the worker thread will
continue handling the request until it is complete.
– For example, if the worker threads need to do an isAuthenticate() call,
it will go out to the LDAP directory server. The worker thread will be
blocked until the ldapsearch and bind is complete.
– If an individual worker thread needs to make multiple LDAP calls, those
calls are processed in a synchronous manner within that thread
– This thread cannot be used for anything else while blocked
Worker Threads Start a Task and are Busy Until the Task is Completed or Times Out

Control
We have control over the agent
connections, the policy server
sockets, and the number and
lifetime of each of them.
We don’t have control over
how long a task takes to
complete.
Agents
Agents initiate a secure
connection to the policy server.
Each agent connection takes up
a socket on the policy server.
Each agent connection
performs a task, and is busy
until that task is completed.
Policy Server
Policy Servers receive requests
from the agents with a reactor
thread.
The reactor thread puts
requests in the queue for
worker threads to work.
Worker threads perform a task
and are blocked until the task is
completed
Checkpoint
What Do We Know So Far?

MST = t/att(s), Yes Math Matters
§ With the information we have, we can build a predictive
model for performance and capacity.
§ We must also understand the impacts of throughput on that
model, and the impacts of latency on throughput.
§ Using this we can identify Key Performance Indicators that
should be proactively monitored, managed, and reported on.
Remember that teacher who said someday this will save your life? Yeah it won’t but math still matters

Building Our Model
§ Throughput
– Total transactions per second the policy server is fulfilling (per second)
§ Latency
– How long does each transaction take to be processed
§ Thread Latency – how long before a worker thread pulls the request from
the queue
§ Execution Latency – how much time does that worker thread take in
processing the request

The Relationship of Threads, Throughput and Latency
§ On any system with a set number of threads, throughput
and latency are interrelated
– As latency goes up the throughput goes down
– As throughput goes down additional requests are queued causing
increased latency
Maximum Server Throughput =
!"#$%&'
()* !,- !./$ ('$1)

We Control Threads, but Not Thread
or Execution Latency
§ There are two primary reasons for Policy Server slowdown
1. Too many Agent API requests coming in for the Policy Server
2. Response time from the user directory
§ If too many Agent API requests are coming in, the thread latency will
increase if there aren’t enough threads to service them in a timely
manner.
§ If the response time for the user directory increases execution latency
increases, which in turn causes thread latency to increase.

Too Many Agent API Requests
§ A single web page doesn’t mean a single request.
§ HCO settings limiting the number of agent connections aren’t applicable
depending on the apache threading model.
§ If the application teams have configured to allow 2000 max clients, but
you are saying 20 max connections….it will be 2000 max connections.
§ At peak times, if not properly managed, your web servers can overload
your policy server and significantly increase thread latency.

Directory Latency Affects Throughput which
Affects Single Sign-On Performance
§ Assumptions
– 15 threads
– Average of 7 LDAP queries per transaction
– Average LDAP (including the network) latency is 10ms
– Goal: 125 transactions / sec
§ Average transaction time must be at least 70Ms (LDAP) + 30ms processing = 100
ms (0.1 sec)
§ 15 threads / 0.1 seconds = 150 transactions / sec maximum
§ When LDAP goes to 15ms the maximum throughput drops to 111 txns/sec
Using: Maximum Server Throughput =

Keeping Low Transactional Latency
§ Right size connections, threads, and agent ratio’s for load and hardware.
§ Minimize custom / third party code and optimize any callouts that code
makes to remote systems
§ Use smart LDAP searches and optimized database queries
§ Keep Agent to Policy server and Policy server to user directory
connections over fast connections – possibly in same datacenter
§ Work with user directory teams to ensure the directories are performing
as required

Throughput
We can model throughput to
properly plan for capacity.
We can model what even a
small change in the network or
directory performance will do.
We can use these models to
become proactive and
predictive.
Thread Latency
Can be affected by a large
volume of requests
Can be affected by high
execution latency
You must do the math in
advance and right size for the
expected peak loads
Execution Latency
Poor Policy Design can increase
calls to the directory
SSO is usually the victim, but
unless you can prove it, that
doesn’t matter.
Increasing the number of
threads is not always the
answer.
Checkpoint
What Do We Know So Far?

What are the SSO KPI’s
§ With the information we have, we can build a predictive
model for performance and capacity.
§ We must also understand the impacts of throughput on that
model, and the impacts of latency on throughput.
§ Using this we can identify Key Performance Indicators that
should be proactively monitored, managed, and reported on.
Knowledge is half the battle, but you will still lose if that’s all you have.

What do you Need Track?
User Store Access Policy Store Access Session Store Access
Cache Success Rate Cache Miss Rate Socket Counts
Max Queue Length Current Normal Queue Length High Priority Queue Length
Avg Authorization Time Avg Authentication Time Avg Validation Time
Avg IsProtected Time Max Sockets Transaction Counts
Agent Transaction Times Agent Cache settings

How Can We Monitor SSO KPI’s
§ CA SM Trace Tool
§ CA OneView Monitor
§ CA APM for SSO
§ SNMP (Splunk, UIM, Any tool that can issue SNMP GET)
§ Spylogix for CA SSO
It’s not about how you get the data, but what you do with it

SM Trace Tool
Unlocking the Data in Your CA SSO Logs
§ Load and parse logs from all
CA SSO components
§ Generates reports detailing
performance for that point
in time
§ Identifies potential
bottlenecks
§ Let’s take a look!!!

OneView Monitor
§ Centrally reports KPI’s
§ Gathers data from agents and policy servers
§ You need to record and gather the data
The Data is there, if You Go and Get it

CA APM
When You Prefer those KPI’s Wrapped up in a Bow
§ Built in CA SSO Dashboards
§ Know instantly if there is a
problem, drill down to RCA
§ Data and analysis comes to
you.

SNMP
Integrate with Your Preferred Tool
§ 60 Objects accessible by
SNMP GET
§ 17 Events available to be
sent via SNMP Trap
§ Present the data how you
want it presented
§ Let’s look at some samples
from Splunk!

SpyLogix for CA SSO
Certified CA Solution
§ Full Analytics platform for
CA SSO
§ Built in dashboards for
performance, systems
management and utilization
§ Focused on MTTR and
MTBSI

Why Does it Matter? How will this Data Help Me?
§ User’s are complaining about ‘slow’ SSO performance
– What is the definition of slow?
– Can you show what the performance of SSO is to combat that
impression?
– Does their definition of slow include the load time for the application
page? How do you show if you are affecting that?
– Can you show historical evidence of performance and stability?
§ Maybe SSO is having an issue, can you pinpoint where it is?
Real world scenario’s

Having Access to the Data Improves Your Response
§ Here are graphs showing
the customer queue depth
§ From these charts we can
see the queue keeps
growing
– Either the load has increased,
or
– The backend cannot keep up

Now I Know there is an Issue, What’s Causing it?
§ Need to answer two questions
– How do we determine which of the two conditions the Policy Server is
in?
§ Are we queuing the request because there are too many incoming
requests?
§ Are we queuing the request because the transactions are taking too long to
process?
– How can we easily answer those questions?

SM Trace Tool is Able to Identify Slow
LDAP Response Times
§ We can see the distribution
of LDAP responses which
have a major influence on
CA SiteMinder throughput
§ We had to go get this data,
what if we had been alerted
when the average started
increasing, before users
complained?

APM for CA SSO Shows the Same Type of Slowdowns
§ We see a different
representation of the same
problem.
§ But instead of being alerted
to the problem by users,
APM for SSO can alert us
before becomes a problem.

Establishing a Proactive Monitoring Program
§ We cannot rely on user errors to tell us when there is a
problem, its often too late
§ You must have the data readily available to advertise SSO’s
success
§ You must have the data readily available to act when SSO has
an issue.
Users are not your first line of alerting

Single Sign-On is Mission Critical
§ Single Sign-On touches many applications across the
enterprise
– Both internal employee and Consumer transactions
§ If Single Sign-On stops, the applications stop as well
§ When a problem occurs we must know why so action can be
taken
– Need to identify problems that are intermittent
– Need to identify possible problems before they cause outages

Organizations Need to Identify Problems Quickly
§ Single Sign-On can cross many organizations
– Application teams
– Directory teams
– Single Sign-On teams
§ When a problem occurs we tend to play organizational blame
games
§ Since Single Sign-On touches many components it often gets
blamed even if it is not at fault

Many Different Ways to “Monitor”
§ “Monitor” can mean many different things
– Components Up / Down
– System “health”
– User activity
– Administrative Activity
§ Your program must incorporate elements of all to be most
effective

Synthetic Transactions (CA App Synthetic Monitor)
§ Tools to Automatically “login” and access a page
§ Sees the site from an end user perspective
§ Be careful when geographically distributed
– The monitor becomes the failure point
§ Be careful of monitoring becoming the biggest user of the
system.

Synthetic Transactions (CA App Synthetic Monitor)
§ What it tells you
– Is your website responding to logins
– Login transaction and first page load times.
§ Benefits
– Looks across entire site
§ Drawbacks
– Unknown what the path is for the transaction
§ Failover, round robin, internal component failures are hidden
– Can create extra load on system
Tip: a single website on each policy server with a single agent that only communicates to that policy server

ServerErrorFile ACO Setting
§ Existing ACO setting to either display a friendly HTML page or
redirect on a WebAgent Error
§ Use the redirect ability to redirect users to a friendly page on
a separate web server
– Create a separate log for errors for all agents in a single spot
– Collect the error code (Querystring)
– Collect the referrer (HTTP headers)
§ Log these and analyze weekly.

ServerErrorFile ACO Setting
– Has a Web Agent encountered an error
§ What the error code is
§ Which website
§ Benefits
– Real time information – can trigger an alert
– Useful in calculating intermittent issues
– Can also display a friendly error page
§ Drawbacks
– If you aren’t analyzing the data, there is no value

Network Visualization
(CA Application Delivery Analysis (ADA))
§ Network Layer Monitoring tool
§ Plugs into network switches and looks at TCP Traffic
§ Can examine communications to/from multiple systems and
understand latency of these components

Network Visualization
(CA Application Delivery Analysis (ADA))
– Latency of communications between multiple components
§ Benefits
– Can quickly identify component have trouble
– Can identify if it is the network or the application
§ Drawbacks
– Not included in Core Single Sign-On License
– Not a Single Sign-On specific solution
– Overkill in small environments

Key Performance Indicators
§ Oneview/APM/SNMP/SpyLogix
§ One component of a comprehensive solution
§ Often this is the missing component in a comprehensive
monitoring solution.

Step 1 – Baseline Your Environment
§ Once you have chosen your toolset identify your baseline
– Capture data without alerting for 2 – 4 weeks
§ Focus on a timeframe that spans key peak usage periods
– Monthly or quarterly spikes
§ Discuss the goals with your customers and stakeholders
– They often have insights into their usage that you may not know.

Step 2 – Create habits to review data
§ Create habits and carve out time for data to be reviewed
– This helps identify new area’s for coverage
– Identifies previously unknown and unmonitored errors
– The more you review a system, the better you know it
§ As new errors are found, create a knowledge base and keep it updated
– Share that data and steps to resolve. The more people that know the better.
This is about ensuring a good customer experience.
§ Incentivize finding new items. Reward your team for finding new issues
and their resolution, make it cultural

Step 3 – Create alerts based on the baseline data
§ When performance degraded more than x% warn
§ When performance degrades more than XX% alert
§ Warn your team so they can act before a problem
§ Alert broad and wide. Transparency builds trust, trust builds
confidence.
§ Continue to fine tune alerts

Step 4 – Create Dashboards
§ Create dashboards to help your team stay on top of the
solution.
§ Identify the most critical items and put them in first
§ Identify the most troublesome and put them in second
§ Make sure everyone knows how to get to them and how to
read them.

Step 5 – Advertise, Engage, and Sell
§ Create dashboards for your ‘customer’ to see
– General ones that are across the whole solution
– Specific application based dashboards for larger apps
§ Create an executive report and send it out regularly to your customers,
their chain of command and your chain of comment.
– Advertise your success.
§ Consider a subscription model to your successes and challenges
– Internal twitter feeds where customers can subscribe and just see what is
going on.

Recommended Sessions
SESSION # TITLE DATE/TIME
SCX12E
Pre-Con Ed: Five Easy Steps for Migrating to
CA Directory
11/15/2016 at 3:30 pm
SCT44T
Web Access Management and Federation –
Two Great Tastes that Taste Good Together
11/16/2016 at 11:30 am
SCX20S
CA Roadmap: Authentication, Single Sign-
On, Directory
11/17/2016 at 01:45 pm

We Want to Hear From You!
§ IT Central is a leading technology review site. CA has them to
help generate product reviews for our Security products.
§ ITCS staff will be at most sessions. If you would like to offer a
product review, please ask them after the class, or go by their
booth.
Note:
§ Only takes 5-7 mins
§ You have total control over the review
§ It can be anonymous, if required

Questions?

Thank you.
Stay connected at communities.ca.com

Security
For more information on Security, please visit:
http://cainc.to/EtfYyw

Pre-Con Ed: Who's minding the SSO store?

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Pre-Con Ed: Who's minding the SSO store?

Similar to Pre-Con Ed: Who's minding the SSO store? (20)

More from CA Technologies

More from CA Technologies (20)

Recently uploaded

Recently uploaded (20)

Pre-Con Ed: Who's minding the SSO store?