Software functional testing can unveil a wide range of potential malfunctions in applications. However, there is a significant fraction of errors that will be hardly detected through a traditional testing process. Problems such as memory corruptions, memory leaks, performance bottlenecks, low-level system call failures and I/O errors might not surface any symptoms in a tester’s machine while causing disasters in production. On the other hand, many handy tools have been emerging in all popular platforms allowing a tester or an analyst to monitor the behavior of an application with respect to these dark areas in order to identify potential fatal problems that would go unnoticed otherwise. Unfortunately, these tools are not yet in widespread use due to few reasons. First, the usage of tools requires a certain amount of expertise on system internals. Furthermore, these monitoring tools generate a vast amount of data even with elegant filtering and thereby demand a significant amount of time for an analysis even from experts. As the end result, using monitoring tools to improve software quality becomes a costly operation. Another facet of this problem is the lack of infrastructure to automate recurring analysis patterns. This paper describes the current state of an ongoing research in developing a framework that automates a significant part of the process of monitoring various quality aspects of a software application with the utilization of tools and deriving conclusions based on results. According to our knowledge this is the first framework to do this. It formulates infrastructure for analysts to extract relevant data from monitoring tool logs, process those data, make inferences and present analysis results to a wide range of stakeholders in a project.

1. JCSSE 2012
A SOFTWARE MONITORING FRAMEWORK
FOR QUALITY VERIFICATION
MANAGED BLACK BOX TESTING
PRESENTED BY:
DILEEPA JAYATHILAKE
99X TECHNOLOGY

2. QUALITY IS PARAMOUNT
Operational Overhead Poor Quality
• Constant re-planning Kills
• Derailing deadlines
• Higher costs
Through-life Costs
• Recalls / Updates
• Warranty Claims
• Litigation
Business Value Decline
• Loss of market share
• Dropped brand equity

3. FUNCTIONAL TESTING
Focuses on spotting any observable deviations of the product
with respect to the desired features and behavior
 Both manual and automated
Covers most feature-related malfunctions
Limited to only the behaviors observable directly

4. WHITE BOX TESTING
 Supplements functional testing
 Directly examines code
 Can test cyclometric
complexity, testability, unit test
coverage, potential memory leaks, class
and function level
complexity, deprecated API usage, code
duplication, undocumented code, styling
errors
 Many tools exist for this
 Does not cover operational errors

5. BLACK BOX TESTING
 Certain errors are not captured either by  Memory
functional testing or by white box testing corruptions, memory leaks
 Disk access errors
 They need monitoring application in  Failures in certain low level
operation operating system calls
 Pitfalls due to insufficient
 Black box testing is required for a testing user access rights
process to be completed  Unjustifiable performance
bottlenecks
 Wide spectrum of tools is available in
each platform
 Some tools provide a UI while others
generate a log

6. BLACK BOX TESTING TOOLS
Tool Remarks
Apache - Used for server load testing in client-server applications
JMeter - Supports files, servlets, Perl scripts, Java objects, database queries, ftp servers
- Works with HTTP, HTTPS, SOAP, Database via JDBC, LDAP, JMS, POP3 and IMAP
protocols
- Portable across platforms
- Can replay test results
- Highly extendable via plugins
- Provides simple and intuitive UI
- Does not support client-side scripts
Microsoft - Captures system level problems that occur when a program is run
Application - Can monitor Virtual memory usage, First chance access violation exceptions, Input-
Verifier output transfers, Synchronization objects (e.g. locks), Handle usage, Thread pool
usage, Thread local storage, Dll handling, Interactive service creation, Dangerous
API calls, Driver installations and User privilege issues
- Writes results to an XML log
- Optionally a debugger can be attached

7. BLACK BOX TESTING TOOLS CONTD.
Tool Remarks
LeakDiag & - A pair of tools that can be used in conjunction to detect memory leaks in an
LDGrapher application
- Can monitor Virtual memory allocator, Windows heap allocator, COM allocator, C
runtime allocator and TLS slot allocator
- LeakDiag generates an XML log, which can be graphically viewed through
LDGrapher
Process - Windows logging utility, which collects and records most of the system activity
Monitor regarding file system, registry and process/thread activity
- Comes with a rich GUI that displays information in real time
- Offers advanced filtering options to trace specific activity in the interest
- Provides a boot-time logger
- Should be run only for a limited time

8. BLACK BOX TESTING TOOLS CONTD.
Tool Remarks
XPerf - Performance-profiling tool for Windows applications
- Can be used for tracking performance bottlenecks in applications as well as for
comparing time taken for various operations in an application
- Writes monitored data into an XML log which can be later explored via XPerfView
- Comes with a rich collection of preset monitoring profiles
Application - This is a Windows application lifecycle management toolset
Compatibility - Toolset comprises Standard User Analyzer, Internet Explorer Test Tool and Setup
Toolkit Analysis Tool
- Can be utilized to
• Analyze portfolio of applications, web sites and computers
• Centrally manage compatibility evaluators and settings, Rationalize and
organize applications, web sites and computers
• Prioritize application compatibility with filtered reporting
• Add and manage issues and solutions for enterprise- computing
environment
• Deploy automated mitigations to known compatibility issues
• Send and receive compatibility information from the Microsoft
Compatibility Exchange

9. BLACK BOX TESTING TOOLS CONTD.
Tool Remarks
God - Process-monitoring tool for Linux, BSD and Darwin systems
- Provides watchdog functionality on processes with respect to their CPU and
memory usage
- Can be configured to perform certain actions on a process depending on its CPU
and memory consumption
- Has both a polling mode and an event responder
- Generates a line log
- Capable of sending alerts as email, chat or Twitter messages
Instruments - Standard tool for profiling processes that run under Mac OS X or iOS
- Uses monitoring templates called instruments to collect data on various aspects
- Monitoring areas include CPU consumption, memory usage, memory leaks, I/O
operations, power usage and network traffic
- Data viewers are built in
- Capable of recording a sequence of actions to replay later

10. Tool usage
requires
platform
expertise
Analysis
Generated logs
knowledge not
are bulky
shared
Black box
testing
pitfalls
Difficult to
communicate Difficult to
results to correlate results
different from multiple
stakeholders in tools
a project
Lot of manual
work
Framework is needed to integrate tools and automate analysis

11. FRAMEWORK ARCHITECTURE

12. CONCLUSIONS
LDEL significantly reduces code for log data extraction
LDEL improves readability of the data extraction code
Introduction of SQLite into data manager enabled
handling high data volumes and persistence
Need a better database for efficiently handling
heterogeneous log data
MML is not appropriate for UI generation