This document summarizes a QA report for 2011 from Luis Barreiro of Mobicents. It discusses achievements including community releases maintaining jobs and adding value to the community. It also discusses four JBCP releases between versions 1.2.9 to 5.1.2. It outlines the goals and challenges of a performance testing tool including collecting useful data with low overhead and detecting problems early. It describes using OS tools to collect data and parsing output files to analyze performance and track regressions against goals.

1. Copyright 2011 Red Hat Inc.
Mobicents
JBoss Communications Platform
QA Report for 2011
Luis Barreiro
Sochi – December 2011
Copyright 2009 Red Hat Inc.

2. Copyright 2011 Red Hat Inc.
Achievements

Comunity Releases

Maintain and extend existing jobs in JBoss Jenkins / Move to cloudbees

Added value to comunity (docs, mobicents.org, maven snapshots, etc.)

JBCP 1.2.x Releases

1.2.9 to 1.2.11 (3 Releases + 4 patch releases)

JBCP 5.x Releases

5.0.1 to 5.1.2 (4 Releases + 4 patch releases)

3. Copyright 2011 Red Hat Inc.
Performance Test Tool – Design goals

Minimum overhead

Ability to parse big files

Useful presentation

Track progression

Early detection of problems

Regression analysis
Performance Test Tool – Challengers

What to collect

What to compare

Variability of parameters

4. Copyright 2011 Red Hat Inc.
Performance Tool – Data collection

Black box / Offline analisys
– vmstat -a -n 1 > vmstat-slee-scspt.csv &

Rely on OS tools, components themseves

They already have the data !!!

Parsing of *.csv (or similar) files

Implemented for vmstat, jstat, SIPp
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free inact active si so bi bo in cs us sy id wa st
0 1 168192 3134020 1165704 5189392 0 0 0 2 0 1 5 0 95 0 0
3 0 168192 3114756 1165164 5209344 0 0 0 0 1139 10696 33 3 63 1 0
2 0 168192 3112524 1165160 5211672 0 0 0 0 1005 10637 19 2 80 0 0

5. Copyright 2011 Red Hat Inc.
Performance Tool – Data analysis

Graph

PDF report

Junit compatible XML

Statistics file

Sample count, Sum, Min, Min, Average, Std Deviation
RetransmissionsP SAMPLES IS 16726.000 samples;
RetransmissionsP SUM IS 1279.000 message*second⁻¹;
RetransmissionsP MIN IS 0.000 message*second⁻¹;
RetransmissionsP MAX IS 211.000 message*second⁻¹;
RetransmissionsP AVG IS 0.076 message*second⁻¹;
RetransmissionsP STD IS 2.579 message*second⁻¹;

6. Copyright 2011 Red Hat Inc.
Performance Tool – Regression analysis

Goals file
SuccessfulCallP SUM IS AT_LEAST 99.9999 % OF 10000000 call*second⁻¹;
RetransmissionsP SUM IS AT_MOST 0.1 % OF 10000000 message*second⁻¹;
ResponseTime1P MAX IS AT_MOST 1.5 second;
ResponseTime1P AVG IS AT_MOST 0.025 second;
OldUsed AVG IS AT_MOST 5000000 kB;
GCTime AVG IS AT_MOST 15 % OF 1 second⁻¹;
CPUIdle AVG IS MORE_THAN 43 percent;

Graphs for category and global

7. Copyright 2011 Red Hat Inc.
Questions?
Copyright 2009 Red Hat Inc.