Liberty Mutual Information Systems uses open source tools to help Liberty Mutual Group exceed their business objectives by delivering high-value, market-responsive IT solutions. Richard Thompson discusses why open source tools are useful during various phases of development like unit testing, configuration management, and continuous integration. He provides examples of specific open source tools used for tasks like test reporting, static analysis, performance testing, and more. Thompson also outlines lessons for successfully implementing open source tools, like considering community size and support when selecting tools.

1. “As a part of Liberty Mutual Information Systems enable Liberty Mutual Group to exceed their
business objectives by delivering high value, market responsive IT solutions”
Testing During The Development Phase
Using Open Source Tools
March 2008
Richard Thompson
(Verification Technologist)
1

2. Agenda
Introduction
Why Open Source
Examples Through the Lifecycle
Lessons in Implementing Open Source
Successfully
Questions
2

3. Overview of Liberty-IT
Belfast based software development & testing house
with ~230 staff
Our parent company is Liberty Mutual Group
a leading global insurer
Liberty Mutual ranks 95th on the Fortune 500 list of largest
U.S. corporations (based on 2006 revenue).
$85.5 billion in consolidated assets
Technologies include
Java, .Net, Struts, IBM Portal
Hibernate, Spring, Web Services -> SOA
Multiple database platforms
Dedicated Testing practice focusing on technical testing
solutions
3

4. Disclaimer
The views expressed within this
presentation belong solely to Richard
Thompson and do not necessarily
represent those of Liberty Mutual or
Liberty-IT.
4

5. Why Open Source?
5

6. Business Demands
Faster
Strategic advantage to be first to market
Quarterly -> Monthly -> Fortnightly Releases
Better
Lower and lower tolerance for poor quality
Expect higher productivity
Demand more visibility into project status
Cheaper
Continually looking to reduce costs
Everything must have a ROI (Return on Investment)
6

7. Growing Role of Open Source
Perception of high vendors prices
Growing number of high quality, mature open source
products
Which are highly reliable
Have large world wide support
More and more large companies are using Open
Source tools/ components
Amazon
Google
IBM
Conducted by Boston-based Forrester Research Inc. in
February 2004, a study revealed that 65 of the 140
companies questioned -- almost 50 percent -- are using
open source applications.
7

8. Examples Through the Lifecycle
8

9. Open Source Solutions
Unit Testing JUnit, HttpUnit, Configuration CVS, Subversion
HTMLUnit, MaxQ Management
Automated Watir, Samie, Build Management ANT
Functional/ Pamie, Jiffie
Regression Testing
Performance OpenSTA, JMeter, Project Management XPlanner,
Testing WebInject, Grinder dotProject
Web Service Testing SoapUI, HttpUnit, Application Big Brother, Nagios
XMLUnit Monitoring
Code Coverage Defect Tracking Mantis, Bugzilla
Emma, Cobertura
Test Management TestLink,
Test Case Web
Static Analysis CheckStyle, PMD,
FindMoreBugs
Continuous Cruise Control
Integration
9

10. Testing Through the Lifecycle
Requirements Development IS Testing Business Testing Production
Post
Requirements Unit/Component Smoke Acceptance
Implementation
Review & Testability Testing Testing Testing
Validation
Code Review & System Regression Application
Static Analysis Testing Testing Monitoring
Performance
Code Coverage Testing
Defect Tracking
Reports
10

11. Automated Unit Test Reporting
JUnit
• Simple framework for writing
automated tests
• Written in Java to test Java, so
Developers only need to know 1
language
• Most widely used automated unit
testing framework for Java
• Provides fast/ repeated feedback on
quality of code
11

12. PMD – Static Analysis
PMD scans Java source code and looks for potential problems like:
•Possible bugs - empty try/catch/finally/switch statements
•Dead code - unused local variables, parameters and private methods
•Suboptimal code - wasteful String/StringBuffer usage
•Overcomplicated expressions - unnecessary if statements, for loops that could
be while loops
•Duplicate code - copied/pasted code means copied/pasted bugs
12

13. Continuous Integration Process
Cruise Control
Change Code Server
Request Repository (Continuous Build
& Test)
Developer
Build
(Automated Build
Script)
Reporting
DB Automated Code Automated
Analysis Unit Test
1. Code Coverage,
2. Static Analysis,
3.Coding Standards
13

14. Code Coverage
14

15. Cruise Control Dashboard
15

16. Continuous Integration – Daily builds
16

17. CVS Check-In Process
CVS
Comment OK
Mantis
Comment
Bugnote
Developer Invalid
Checks In code
Pre-commit validation Post-commit validation
Ensures that comment contains Writes out the following
a) BUGID: information to a DB table
b) BUGNOTE: a) BUGID:
c) TEST PLAN/DESCRIPTION b) BUGNOTE:
c) TEST PLAN/DESCRIPTION
d) CVS Committer
e) Files updated (with full path)
17

18. Watir – Browser Automation
Watir
•Web Application Testing in Ruby
•Watir is a toolkit used to automate
browser-based tests in Internet Explorer
using IE’s COM interface
18

19. Performance Testing
OpenSTA
•Open Source alternative to LoadRunner/
Rational Performance Manager
•Large community, excellent online
documentation
•Option to purchase guaranteed support
(for a price ☺)
19

20. Lessons In Implementing Open
Source Successfully
20

21. Factors to consider when selecting
Main Places to look
Sourceforge (http://sourceforge.net/ )
Freshmeat (http://freshmeat.net/ )
What to look for
Active community
Regular releases
> 90% Activity
Documentation
Do the technologies used match those used within the company?
Perform a Pilot
Try on a single project, review lessons learned, move forward
Determining ROI
Measure cost to implement on project verses benefits gained
Also factor in ongoing maintenance
Implementation
Typically requires a higher level of technical ability than Vendor solutions
21

22. Pilot & Measure The Value
• Perform a pilot implementation and measure the setup time, learning curve
& value added
• For example: Piloting Watir on a particular project resulted in:
• 38% less defects identified during Business Testing
• Testing Coverage significantly increased
• Developers free to perform other activities
• Automated tests reused within Regression & Smoke Testing
• Reduction in Defects found in Production
90
80
70
60
50 May
%
Aug
40
30
20
10
0
Requirements Development Testing Production 22

23. Questions?
23