Using IBM Rational Change as an Enterprise-Wide Error Management Solution – Case Nokia publication date: Jun 6, 2010 IBM Innovate Software Conference 2010 Orlando, US

1. The premiere software and product delivery event.
June 6–10 Orlando, Florida
IBM Innovate 2010
Session Track
Template
Presenter’s Name
Principal Consultant and CEO, Contribyte
Lasse.Mikkonen@contribyte.fi
CCM-1738

2. Using IBM Rational Change as an Enterprise-Wide Error
Management Solution – Case Nokia
Introduction of Presenter
Lasse Mikkonen, CEO Contribyte
Contribyte IBM Business Partner, Finland
– ALM focused process improvement company
– 20 people total – 15 consultants
Starring: Chris Parkyn, Nokia, Fault Management Solutions, Architect
Nokia

3. 3
how it started – Case Using Change Enterprise Scale
Incentive to improve visibility
ALM scope
narrowed down to EM only – DOORS XT planned, but no real concept
Merge distributed EM solutions into 1
Time to market
System renewal
Replications issues
Cost savings
Capacity
Maintenance
Performance

4. Original requirements
The problem to be solved – functional requirements
Create a system to handle large amount of EM data with web UI
One global repository
High Availability
Automated traceability to SCM
Code changes
Release
Central Repository approach Rational Change
legacy problems with replication
Claimed to be World's largest single R&D unit
Advanced integration requirements
Company internal lifecycle management tools
3rd party suppliers

5. Non functional requirements
Distance and time zones
All continents
Off shoring
How to define Load
current system users
peak time, peak projects hit rate * # systems to replace * something
New use cases
The other way around – Time consumption/Use Case
Usability
Started with usability study
Lot to improve
No pop-ups
Tree-type browsing
Company UI guidelines

6. Concepting
Error Mgmt Concept
General Error Management Theory
Different types of objects
General process
DB level integration to SCM - still
missing core parts
Baseline content
Release information for error tracking
Release structure information
Release Planning
Test Management integration
Customer Issue Management
integration
Release subsystem
Organizes EM data
Structural data – requires tree view
100's of thousands of categories
General Product, Feature, release
structure
Steers Processing responsibilities
Access Mgmt
Based on current policy
Roles
Access
Single Repository
no DB 'boundaries'
Advanced visibility rules based on
releases

7. Example Screenshot of customized layout

8. The premiere software and product delivery event.
June 6–10 Orlando, Florida
Building the Rational Change Environment

9. Building the Rational Change Environment
Solution Hardware Requirements
Only concerned with adding Rational Change to the existing Change
environment
Current Synergy environment consisted of thousands of users, distributed
over many geographical sites.
0% unplanned downtime
Need to have resilience in the solution case of Hardware failure
No interruptions from backups, though need to ensure data integrity
Solution needed to be distributed across sites
Hardware needs to be able to scale to handle the planned load.

10. Building the Rational Change Environment
Testing and Tuning Requirements
Need Clear Requirements on usage amounts
Know how your system will fit in the corporate environment.
Authentication – such as LDAP
Initial and future integrations
Know how your users are distributed
Understand the availability requirements and concurrent user load
Plan for the worst data amounts
From business representatives, know the expected performance

11. Building the Rational Change Environment
Testing and Tuning Requirements
Automated Testing
Ensure that you have the tools and resources needed to perform automated testing for the
solution you are building
LoadRunner, IBM Rational Performance tester for example
Have light weight testing tools available for the development
Jmeter
When developing in-house tools, ensure that you have the initial requirements laid down to
ensure that they can be tested and tuned – standard languages, debugging and
configuration
Cost to support, when everything moves in different magical ways
Know how and where you are going to store the testing results
Think about who has access to the results....
Know where the breakpoints are in the system you are building
If you don’t – you’ll find out when the system is actually running
Try and get needed specialists involved at the start – such as Informix DBA’s

12. Building the Rational Change Environment
Metrics Requirements
One aspect often over looked until a Manager comes a
knocking
Plan what information is going to be needed
How will performance be measured
End user expected performance
Server statistics – CPU, memory, network, swap and the alike
Thresholds – do you know and monitor when things are about to break

13. Building the Rational Change Environment
What Did we Do .....
Consisted of 12 servers in the production environment alone split up
into multiple Firewall segments
Concentrated on getting the production ready
Production used initially as a QA environment
Testing of official internal releases only
Fine Tuning of every aspect – often changing a single item at a time.
Ensure every component has a failover from web servers to scripts
– OS based monitoring and clustering
Got everything working before firewalls were configured
IT Environment
Used for Training and feature evaluation
Used for functional testing
Ensured support were learning new functionality and features

14. Building the Rational Change Environment
What Did we Do (cont.)
QA Environment
Once the production environment was configured we ensured the
same configuration was mirrored over to the QA environment.
Development Environment
Standalone environment maintained by the build manager
WARNING:
Do as much testing in the production environment – as once you are live
you will be restricted on what you can do.

15. Building the Rational Change Environment
What Did we Do (cont.)
Backups
Evolved the backup solution we developed for the Synergy
Environment
Makes use of filers and Informix blocking
Added additional oncheck commands for online checking
Ended up having full database backups lasting around ~5 seconds with no
interruption to service
For performance we added update_statistics x3 times p/day
Reduced the need to perform weekly/ adhoc index rebuilds and
database dumps/ backups which cause serve service disruption

16. Building the Rational Change Environment
What Did we Do (cont.)
Metrics
We setup a number of tools to monitor the system
Cacti for server performance – Memory, Swap, Disk etc
AWStats for basic web analysis
– Needed to be able to extended and convert the Jetty logging format for this to
work.
– Gives a good break-down on browser information, OS’s and distribution across
web servers.
In-house Data Warehouse
– Allowed us to plug the holes missing from Change and the above, whilst being
able to view past events and plan future activities (such as performance testing)
Ensure support is aware of what is where and how the data can be
interpreted.

17. Metrics
Building the Rational Change Environment
What Did we Do (cont.)

18. Building the Rational Change Environment
What Did we Do (cont.)
Data-Warehouse Metrics – what we collected
Web throughput p/web server – customer facing
Throughput of integrations
User operations – such as login, Creates, Reports etc
JVM Performance – garbage collections
Attachment sizes and when they were added
Lucene index creation times
License events from FlexLM license log file
DCM Transfer information
Synergy session information
Synergy database roles

19. Building the Rational Change Environment
What Did we Do (cont.)
Performance Testing
Testing the individual components
Tests cases developed to test and tune specific areas, such as
– TDS (now RDS)
– Single and Multiple web servers
– Database transactions (Informix)
– Failover testing
– Collect performance metrics as the testing was being carried out
Developed a number of baseline tests
Easily to compare the performance between releases

20. Building the Rational Change Environment
Building up the Support Organization
Support Requirements (24x5 application, 24x7 solution)
Support’s role
Handle end-user queries
Be able to monitor the system and alert upon failure/s
Perform basic debugging capabilities and address problems as they arise
Handle release deployments to the various environments
Training Support
Ensure that they are involved from the start in all aspects from concept, create and
deployment/ installation
Ensure competence gaps are plugged quickly
Task Forces
Ensure that competent resources are available when critical issues arise or long-term
debugging is needed.

21. The premiere software and product delivery event.
June 6–10 Orlando, Florida
Lessons Learnt

22. Lessons Leant
Deployments
Training and piloting was not enough to prepare us
Life-cycle
We were over ambitious as to what we wanted for the initial stages
Caused additional deployment efforts syncing legacy systems
Makes upgrades ever more difficult
Performance
Performance is the number one issue
We’ve made real progress on the overall performance, such as
– Login times
– Constantly tuning Jetty JVM
– Improving metrics collected
– Reviewing queries by end-users to use indexed column
Ensure you communicate effectively performance testing results, as otherwise you’ll end up
spending time explaining more than you bargained for.

23. Lessons Leant
Support
Make sure it’s clear from the onset what is expected from them
Still ramping up the competence of support
Environments
Ended up with too many environments to please everyone:
Testing, Training, Performance testing, Integration testing, development, support testing, deployment.
Need a cleaner way to deploy releases to the various environments
Rational Change as an Enterprise Application
Missing full integration to Rational Synergy – a lot of work is needed to get the full
release information synchronised to Rational Change.
Data loss problems in TDS gave Rational Change a bad name
Database schema is using old Informix functionality – problems with the 63 char limit
being the biggest complaint we’ve had from end-users.

24. The Way Forward
Continue adding more and more users
Change 5.2 Upgrade
Improving Performance
Reviewing the life-cycle
Moving to central / remote
Introducing a common approach to deploy Change
to all environments
Hardware replacement – simplifying the solution

25. 2
5

26. 2
6
© Copyright IBM Corporation 2010. All rights reserved. The information contained in these materials is provided for informational purposes only, and is provided AS IS without warranty of any kind,
express or implied. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, these materials. Nothing contained in these materials is intended to, nor shall have
the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM
software. References in these materials to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or capabilities
referenced in these materials may change at any time at IBM’s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future
product or feature availability in any way. IBM, the IBM logo, Rational, the Rational logo, Telelogic, the Telelogic logo, and other IBM products and services
are trademarks of the International Business Machines Corporation, in the United States, other countries or both. Other company, product,
or service names may be trademarks or service marks of others.
www.ibm/software/rational