executive white paper
Best Practices in Siebel CRM
Monitor, Measure and Manage the End User Experience
Best Practices in Siebel CRM Performance Management: Monitor, Measure and Manage the End User Experience 2
3 Executive Summary
Knoa® Software is Why End User Performance Management is needed to optimize
leading the performance Siebel CRM apps
management industry 5 ROI Isn’t Real Until It’s Realized
with end user experience The ROI impact of issues with adoption and efficient and effective use
6 Traditional Performance Management Approaches Aren’t Enough
that reach beyond Monitoring system and application health is no longer enough
infrastructure and for service of business apps
application performance 7 Overview of Knoa Experience and Performance Manager
to the monitor, measure A review of the architecture and functionality of Knoa EPM
and manage how end
9 Knoa EPM Use Cases
users are utilizing the Knoa EPM delivers results to the entire application team. This section
application to optimize details use case examples compiled from Knoa’s customer base
business process 14 You Can Manage What You Measure
execution. The impact that real end user experience and performance metrics
can have on the application management team
15 Could You Benefit From Knoa EPM?
Four questions to help you evaluate the potential value of Knoa EPM
for your Siebel deployment
Best Practices in Siebel CRM Performance Management: Monitor, Measure and Manage the End User Experience 3
CRM solutions, like Siebel, automate complex business processes where high
levels of application performance are crucial. Yet many frustrated execs find
that despite the small fortune they have spent on products to manage servers,
“An end user performance
networks and applications, the business constituencies still complain about the management package
performance, usability and availability of the Siebel CRM application.
from Knoa Software is
What is often overlooked is that application performance is just one of three key capturing the CIO’s
drivers of return on investment (ROI) from your Siebel application. An efficient
infrastructure is critical. It’s important to optimize the Siebel application for your
attention for its ability to
own unique business environment, user needs and usage patterns. But, it’s all boost worker productivity
for naught if you do not build and sustain adoption and efficient and effective
use of the Siebel application by your end users.
and optimize application
How can you tell how if your Siebel solution performs for each person who
uses it? How can you identify which users have difficulty executing which
transactions? What does it mean when your end users say the system is “slow”? Performance
How can you pinpoint the actual issues with the real data you need to take
corrective action? Knoa can give you the answers.
Management suite lets
IT departments measure
Enterprises with mission critical Siebel deployments do not have accurate
measurements of adoption, efficient use, and effective use of the application by
response times, distinguish
the end users. Without these measurements, companies are unable to identify and between system and user
resolve critical issues that impact the user’s ability to optimize business process
execution. The result is significantly reduced ROI, and significantly increased cost
error and monitor work
of application support. flows – all in real time…”
Knoa® Software is leading the performance management industry with end
user experience management solutions that reach beyond infrastructure and – Brian Gillooly
application performance to monitor, measure and manage how end users are Optimize IT
utilizing the application. Knoa customers not only know if core enterprise
applications are delivering an acceptable user experience, but also, if
application users are executing key processes effectively and efficiently –
the key to achieving business value and ROI.
Best Practices in Siebel CRM Performance Management: Monitor, Measure and Manage the End User Experience 4
Knoa Experience and Performance Manager (EPM) monitors enterprise
applications from the perspective of the end user to capture the actual end user
experience. Knoa EPM captures transaction response times as well as ecosystem
and application errors – along with a time-synchronized workflow – from all
For all the investment users, at all locations, at all times, across the entire application. This information
in application is used for alerting and proactive remediation of end user issues (whether they
are system errors, user errors or performance problems). Knoa EPM also
management solutions, analyzes and aggregates information to provide actionable insight into process
nearly two-thirds of compliance, training requirements, adoption issues and a number of other
companies (66 percent)
rely upon help desk calls Knoa EPM provides a complete picture of end user experience by focusing
on both the experience the end user receives from the application and the
as the main indicator performance the end user achieves working with the application. You can think
that end users are of Knoa EPM as the “third leg” to the performance management stool –
infrastructure, applications, and now, end user metrics. Think about it! The
experiencing response infrastructure is monitored and measured and optimized; application execution
time and/or availability across that infrastructure is monitored and measured. In a recent study
conducted by Forrester Consulting an overwhelming 87 percent of IT executives
problems. either strongly agreed (50%) or agreed (37%) that the best way to manage the
service of business critical applications is by measuring performance at the end
user level and rapidly resolving significant performance problems.
Clearly, there’s a need for a change. In fact, the Forrester Consulting report
found that change is imminent. Sixty seven percent of the surveyed
organizations are either much more focused (27%) or more focused (40%)
on the end user experience than they were two years earlier.
Forrester concluded that best practices companies are intent on
collecting metrics on the end user experience.
Best Practices in Siebel CRM Performance Management: Monitor, Measure and Manage the End User Experience 5
ROI Isn’t Real Until It’s Realized
The initial investment in CRM applications is high. Software licensing costs can
range from $1,000 to $2,800 per end user. Implementation services are 2-3
times the software costs and there is investment in back-end infrastructure.
One Version of the Truth.
Ongoing costs are high, as well. Software maintenance averages 20%, annually.
Gartner reports that 60% of ongoing costs of a typical deployment are for
support staff. ROI is only realized from this investment if the end users adopt
Knoa EPM gives the
the application and use it efficiently and effectively to improve the execution enterprise real metrics for
of core business processes – which is very difficult to achieve.
adoption, efficient and
Consider this all too common scenario that afflicts many CRM deployments. effective use of the
Months after deployment a high percentage of the functionality is not being
used. End users complain that the application is “slow” and “doesn’t work”.
application. With end user
Some users find the software cumbersome to adopt into their routines and metrics, companies are
create their own non-compliant work-arounds. Employee turnover slowly
erodes the user base proficiency.
able to identify and
resolve critical issues that
Flying Blind impact the users’ ability
Here’s how one of Knoa’s customers described the problem that led them to
deploy Knoa EPM. to effectively execute, and
hamper ROI realization.
“User adoption of our CRM application has been stalled by a number of factors
affecting field productivity. There are complaints about poor response time, remote
users can’t access the system, and it’s not clear if there are system problems or
if they are just using the wrong procedure. And across the board there are
complaints that things take “twice as long” with the new system. In the absence of
comprehensive and representative data, many discussions regarding application
performance are based on reported user impressions, limited visual observations,
and occasional stopwatch timings. This has resulted in extrapolations about the
performance of the whole system that are inaccurate, and lead to decisions that
are not in the best interest of the business.”
Enterprises with mission critical CRM deployments do not have accurate
measurements of adoption, efficient use, and effective use of the application by
the end users. Without these measurements, companies are unable to identify,
and resolve critical issues that impact the user’s ability to effectively execute.
The result is significantly reduced ROI, and significantly increased cost of
Best Practices in Siebel CRM Performance Management: Monitor, Measure and Manage the End User Experience 6
Traditional Approaches to Performance Management
Aren’t Enough Anymore
CRM applications like Siebel are complex. It’s the flip-side of the flexibility they provide to meet a wide variety of customer
needs. Each Siebel customer has their unique business environment – user needs, usage patterns, customizations, system
components, architectures and platform environments vary widely. In the face of this complexity, it is becoming increasingly
difficult for measurement and metrics taken at the infrastructure level (system, network or transaction management) to provide
a true perspective of the end user experience with the application.
In a recent study conducted by Forrester Consulting an overwhelming 87 percent of IT executives either strongly agreed
(50%) or agreed (37%) that the best way to manage the service of business critical applications is by measuring performance
at the end user level and rapidly resolving significant performance problems.
Knoa believes that the application is “not performing” – from the end user’s Received Error Messages 40%
perspective – whenever the end user is unable to effectively execute the key
processes required for their role. In our vision, application performance, from Site Navigation Issues 37%
the end user perspective includes:
• Slow transactions Transaction Execution Issues 31%
• Cumbersome Navigation
Login Issues 31%
• System Errors
• Non-intuitive user interface which causes excessive user errors
Content Issues 31%
Knoa EPM goes beyond the response-time dimension to monitor and capture Snapshot of the End User Experience
metrics on the entire end user experience – including system and user errors with online transactions
as well as the user navigation and interaction with the application (workflow). (Source: USA Today, 2006)
The monitoring capabilities of traditional performance management products provide a significant amount of insight
about application health (e.g. transactions that are failing; aggregate transaction response times, application availability).
Knoa EPM extends performance management to a whole new level by monitoring and managing the “health” of the end-user
population. Has the application been adopted? Are the end users using the application proficiently. Is the application enabling
the end users to efficiently and effectively? After all, adoption and efficient and effective use are the keys to ROI from the
Best Practices in Siebel CRM Performance Management: Monitor, Measure and Manage the End User Experience 7
Knoa Experience and Performance Manager
Knoa EPM gives the Siebel support team a comprehensive "fact-based"
perspective about the end user's experience with the application. Who's using it?
What's the response time? Who's getting errors? How many transactions are
“The combination of
being abandoned? the user-base coverage
Knoa EPM is the only end user experience management solution that monitors
and application depth
the entire end user experience from the perspective of the end user. Knoa EPM provided by this
monitors the Siebel application from the perspective of the end user to capture
the actual end user experience, as well as ecosystem and application errors,
approach is tough
and a time-synchronized workflow – from all users, at all locations, at all times. to beat. With this
Knoa EPM uniquely captures a complete picture of the end user experience
and behavior: can manage the
• Actual User-Experienced Response Time for key system transactions,
such as navigating between application screens, save operations, execute
operations, etc. by end users, as well
• Complete Quality of Experience, including system and application errors
and user created errors
as the experience that
• Comprehensive perspective of application utilization – which transactions they receive.”
are used, in what sequence, for how long, etc.
Knoa EPM is based upon a passive monitoring engine that is deployed to – Bernd Harzog
the client workstation along with a template specifically designed for the CEO, APM Experts
target application. This client engine monitors end users’ interaction with the
application. Knoa EPM selectively captures the actual steps, and the sequence
of events in the user’s interaction with the application. In addition to this time-
synchronized workflow, Knoa EPM captures a number of metrics around both
the application’s performance and the end user’s performance:
• All user errors
• All system errors
• Transaction response times
This information, called track data, is transferred to the Knoa EPM analysis
server which prepares a series of targeted analyses and reports. This
information is viewed through the Knoa EPM Console, which presents a series
of management dashboards and drill downs for analyzing and querying the
Best Practices in Siebel CRM Performance Management: Monitor, Measure and Manage the End User Experience 8
A Unique Architecture for End User Experience
“The insight into actual Knoa EPM is based upon a unique architecture that was designed, from the
ground up for user-centric monitoring and experience management. Key
end user experience capabilities the architecture delivers are:
gives us benefits across
the board, helping The ‘out of box’ Knoa EPM solution automatically ‘sees’ every Siebel transaction
target response time and the steps that the user is taking before, during and after the transaction.
Knoa EPM’s self-discovery capabilities eliminate, completely, the need for
issues, solve end user customers to design a data collection strategy, define transactions to the system
problems faster, identify or train the system to recognize new transactions. Automated Discovery means:
• No up-front requirements definition or project scoping
usage and adoption • Faster implementation using out-of-the-box templates
issues and even make • lower overall cost of ownership
• Adaptable technology to easily support the most aggressive release/update
priority decisions on schedules
investments.” The ‘out of box’ Knoa EPM solution gives you global coverage of the entire
Siebel application, across all instances and systems in the field. This global
– Stuart Smith, coverage also gives you a comprehensive perspective on adoption and usage
Director, across the entire user population.
CRM Performance, • Application usage
• Transaction usage
• Screen usage
• Function usage
• Active/idle time
• User behavior
Non-Intrusive to Siebel Application
The Knoa EPM system does not insert tags or instrumentation into the Siebel
application, create any system events, or require any back-end processes
that impact application performance. The Knoa components do not require
integration with any components in the application landscape, whether front-
end or back-end.
Small Footprint/Low Impact
Knoa EPM uses a Seibel-specific template to selectively record the significant
user events that will produce actionable insight, without any discernable
system overhead or impact. It is this light operational footprint that enables
you to monitor and measure the user experience across the entire application
Best Practices in Siebel CRM Performance Management: Monitor, Measure and Manage the End User Experience 9
See the forest and the trees. Knoa EPM delivers results to the entire Siebel support team.
Here are some real use case examples submitted by Knoa customers:
Knoa Use Case: Performance Management
Improve Siebel Application Performance
Excessive application response time is one of the most critical performance issues encountered by the users of today’s complex
mission critical applications. It’s also a huge support problem. With Knoa EPM you know the response time that was actually
experienced by the end-user – for every transaction, for every user, every time.
A leading consumer packaged goods company had put millions of dollars over 3 years in a new Sales Force automation solution.
The number one complaint from users was long response times for key functions, such as creating and saving CRM records. The
QA/Performance team was only able to act on these performance issues once they escalated to critical levels.
The company implemented the Knoa EPM system to detect performance fluctuations for key functions in real-time. Using an
alert system that notified the IT team whenever response times in the field exceeded set tolerance thresholds, the team was able to
preempt the escalation of performance problems in many cases. In addition, by being able to isolate performance problems to
specific areas of the application, the team also reduced the time to resolve critical issues.
Best Practices in Siebel CRM Performance Management: Monitor, Measure and Manage the End User Experience 10
Knoa Use Case: Application Utilization
Link investment and effort to business impact
With Knoa EPM you gain valuable, actionable insights into application utilization: Who is using the application and for how
long? Which areas and functions are most used? Which areas and functions are least used? What usage patterns are creating
strains on IT infrastructure? Knowing the usage profile for your Siebel application and your user base is the most fundamental
step towards application and user performance management.
The IT Support and Development team of a leading global financial services company received requests from their field users
to implement a costly and extensive functionality enhancement for their enterprise sales application. The IT team dedicated
considerable resources to implement this request and launch the new capability.
After the enhancement was implemented, using the Knoa application usage data, the IT Team discovered that few employees
actually used the new functionality. Based on this fact, and a detailed review of Business Operations based on Knoa data, they
made a decision to discontinue support. Eliminating the module from production saved the organization tens of thousands of
dollars in development time and resources.
Best Practices in Siebel CRM Performance Management: Monitor, Measure and Manage the End User Experience 11
Knoa Use Case: End User Proficiency
Drive efficient and effective use by managing errors
With Knoa EPM solution you can identify 100% of all user-experienced errors, whether they are caused by user actions or by
any of the application layers. Knoa EPM also gives you the context in which each error was encountered and the path that the
user took before encountering the error. This information is available in real-time for the end user support team.
By implementing Knoa EPM, the IT department of a leading computer services company realized that a large percentage of
the errors encountered by users were never reported to the Support Desk. In particular, a single error type, generated while
executing searches, accounted for 25% of all errors generated by the company’s CRM application.
The productivity losses and infrastructure impact of repeated and failed queries executed by users amounted to thousands
of dollars each day.
Based on the Knoa data, the IT department promptly identified the issue, which was then addressed with a two-fold strategy:
1) provide targeted user training in order to avoid the error, in the short run, and 2) re-engineer the search functionality of the
application to eliminate the error altogether, in the long run.
Best Practices in Siebel CRM Performance Management: Monitor, Measure and Manage the End User Experience 12
Knoa Use Case: End User Support
Accelerate Problem Resolution with end user metrics and workflow
Knoa EPM captures a complete record of end user behavior, before, during and after the transaction. This end user workflow data
transforms the end user support process.
A major electronics company was well into their deployment of a CRM solution. An inside sales reps in the Raleigh NC call center
was experiencing problems. The information being communicated to the support team from the help desk was very sketchy and
incomplete. In the past the CRM support team would spend hours on the phone with the inside rep, and often had to make a cost-
ly on-site support call to deal with issues like this. However, Knoa EPM enabled a dramatically different approach. Through the
management console, they reviewed this ISRs’ recorded experience with the application. First the team looked at the metrics and
found that the response time she was experiencing was fine – well within the sub-3 second goal. Next they found that she was
generating four times more user errors than her peers. Investing further, by looking at a recorded record of her actual interaction
with the application, they found she was using an incorrect process to validate the customer information (a common process step
in many different transaction types). With this insight, her supervisor was able to instruct her in the proper customer validation
process and “the problems with the system” were quickly resolved.
Best Practices in Siebel CRM Performance Management: Monitor, Measure and Manage the End User Experience 13
Knoa Use Case: Application Support/Engineering
Key Performance Indicators (KPI) that report user errors on an aggregate level were useful in resolving another vexing issue
for that electronics provider. There were grumblings in the outside sales force that the opportunity management screen was
really difficult, slow and cumbersome.
When the KPI information for errors was reviewed, the application team noticed that virtually every end user was making the
same errors on these screens. The first instinct was to schedule an e-learning retraining session. However, a closer review of the
recorded work flow for a representative sample of users revealed that the errors were being caused when the sales reps were
trying to put data in extraneous fields that were on the screen but not required for the transaction to execute. The fix was
simple. No retraining was needed. The application team removed those fields from the screen and the errors were eliminated.
The end-to-end transaction time improved by 25% as well.
Best Practices in Siebel CRM Performance Management: Monitor, Measure and Manage the End User Experience 14
You Can’t Manage What You Can’t Measure.
It’s rule #1 in the management handbook, isn’t it? And the rule that is the driving
force behind the investment in CRM applications in the first place. But, without a
Questions Knoa Answers strategy to measure end user adoption, and without a strategy to gather insight
for British Telecom about end user’s experience and actual performance with the application, many
IT organizations are left with the reality that they can’t manage and improve end
user adoption and build end user proficiency.
Is the application
performing effectively? Knoa Experience and Performance Manager accurately determines how each
employee actually uses the application and provides input to comprehensive
reports that you can use to pinpoint training needs, adjust business processes,
Are end users able and ensure that your end users are using the application to deliver all the value
to effectively and you expect from the application. As an added benefit, Knoa also measures and
efficiently execute reports critical information about how your users experience the application’s
processes – in other • Application Support teams have real-time metrics to proactively address
words, has the training response time issues problems
program led to real • Help Desk personnel have immediate visibility user into actual user
results? interaction with the application, so they can quickly resolve issues with
minimal disruption to the end users
Can we provide objective, • Training organizations can pinpoint which specific application areas (and
real-time data on the which specific users) require additional training
user experience to allow
• Application and Process Engineering can spot cumbersome work flows and
the help desk and key errors that are affecting end user satisfaction and efficiency
users to resolve user’s
questions rapidly? • The CIO has comprehensive metrics from the entire application landscape
to support reporting to the business on SLAs and insight into application
usage and performance to make intelligent decisions about where to deploy
Can we maintain an resources to improve the application performance and effectiveness
electronic audit trail that
• The business executives have actionable insight to drive the highest possible
tracks the completion of levels of adoption, efficient and effective use of the application which is the
key transactions? key to a business return on the investment
What is the impact of
Best Practices in Siebel CRM Performance Management: Monitor, Measure and Manage the End User Experience 15
Can Knoa Experience And Performance
Management System Help You?
Are you certain that all of your end users are actually using the functions Sixty seven percent of
of the application required to execute their role?
Knoa EPM captures and reports real metrics about application usage. Who is surveyed organizations
using which transactions? How long are they active? How long are they idle? are either much more
Knoa EPM gives you the ability to pinpoint and analyze user adoption issues
that are hindering the achievement of business results. focused (27%) or more
focused (40%) on the end
Do your end users ever use system performance as a key reason why
critical objectives are not met on time and on budget? user experience than they
One of the benefits of Knoa EPM is that it gives you the ability to correlate were two years earlier.
business outcomes with efficient, effective application usage. Knoa EPM system
captures and reports real metrics about the end users experience with the
application, you will now know if the users were experiencing long transaction Forrester Consulting
response times or if there were a lot of system errors presented to the end user. 2006
You will also know if the end users themselves were causing a lot of errors
by their own behavior. You will then know where to apply corrective action.
Perhaps it's a user training issue, or perhaps the errors are being caused by a
cumbersome or confusing user interface that can be corrected.
Do you suspect that there are a lot of problems and issues that are never
surfaced to the help desk?
Well, you’re probably right. It is often reported in surveys of the service
management process that 70-80% of all end user problems are not reported
because the end users ignore the errors, abandon the transaction and/or find
workarounds. In fact, Forrester Consulting recently reported that an end user
typically experiences a problem 6 times before they call the help desk. Knoa
EPM gives you the ability to pinpoint and proactively remediate problems which
impact the efficiency or the effectiveness of the end user population using the
application to accomplish business goals. An EPM system will make sure you
have visibility into all errors – for all users, in all locations, all the time.
When your end users do report problems to the help desk, do you find
they are wasting a lot of time trying to answer questions and document or
recreate the problems?
Knoa EPM captures the actual work flow that the user experienced – before,
during and after the transaction. This information is immediately available to
help desk personnel so there is minimal business disruption from issues with the
application. Employee productivity impact is kept to a minimum when there is
an issue (users spend less time on the phone with help desk and problems are
Best Practices in Siebel CRM Performance Management: Monitor, Measure and Manage the End User Experience 16
Expertise and Performance Management systems are becoming a priority for
the business executive and CIO who need to drive the highest possible levels of
adoption and efficient and effective use of core enterprise applications to drive
a business return on investment. Only with the insight into end user metrics
About Knoa that an EPM system can provide can you really know if your core enterprise
applications are delivering an acceptable user experience and if application
Knoa® Software is leading the users have adopted the applications and are executing key processes effectively
performance management and efficiently – the key to achieving business value and ROI from the
industry with end user experience application investment.
monitoring solutions that reach Training
beyond infrastructure and
application performance to the
monitor, measure and manage
how end users are utilizing the
application to optimize business
process execution. Knoa
customers not only know if
core enterprise applications
are delivering an acceptable
user experience, but also, if
application users are executing
key processes effectively and
efficiently – the key to achieving
business value and ROI. Knoa is
headquartered in New York, NY.
For more information about
Knoa, please visit www.knoa.com.
5 Union Square West
New York, NY 10003
Copyright, 2007 Knoa Software
All rights reserved.