Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

A software monitoring framework for quality verification


Published on

Software functional testing can unveil a wide range of potential malfunctions in applications. However, there is a significant fraction of errors that will be hardly detected through a traditional testing process. Problems such as memory corruptions, memory leaks, performance bottlenecks, low-level system call failures and I/O errors might not surface any symptoms in a tester’s machine while causing disasters in production. On the other hand, many handy tools have been emerging in all popular platforms allowing a tester or an analyst to monitor the behavior of an application with respect to these dark areas in order to identify potential fatal problems that would go unnoticed otherwise. Unfortunately, these tools are not yet in widespread use due to few reasons. First, the usage of tools requires a certain amount of expertise on system internals. Furthermore, these monitoring tools generate a vast amount of data even with elegant filtering and thereby demand a significant amount of time for an analysis even from experts. As the end result, using monitoring tools to improve software quality becomes a costly operation. Another facet of this problem is the lack of infrastructure to automate recurring analysis patterns.
This paper describes the current state of an ongoing research in developing a framework that automates a significant part of the process of monitoring various quality aspects of a software application with the utilization of tools and deriving conclusions based on results. According to our knowledge this is the first framework to do this. It formulates infrastructure for analysts to extract relevant data from monitoring tool logs, process those data, make inferences and present analysis results to a wide range of stakeholders in a project.

Published in: Technology
  • Be the first to like this

A software monitoring framework for quality verification

  2. 2. QUALITY IS PARAMOUNTOperational Overhead Poor Quality • Constant re-planning Kills • Derailing deadlines • Higher costs Through-life Costs • Recalls / Updates • Warranty Claims • Litigation Business Value Decline • Loss of market share • Dropped brand equity
  3. 3. FUNCTIONAL TESTINGFocuses on spotting any observable deviations of the product with respect to the desired features and behavior Both manual and automatedCovers most feature-related malfunctionsLimited to only the behaviors observable directly
  4. 4. WHITE BOX TESTING Supplements functional testing Directly examines code Can test cyclometric complexity, testability, unit test coverage, potential memory leaks, class and function level complexity, deprecated API usage, code duplication, undocumented code, styling errors Many tools exist for this Does not cover operational errors
  5. 5. BLACK BOX TESTING Certain errors are not captured either by  Memory functional testing or by white box testing corruptions, memory leaks  Disk access errors They need monitoring application in  Failures in certain low level operation operating system calls  Pitfalls due to insufficient Black box testing is required for a testing user access rights process to be completed  Unjustifiable performance bottlenecks Wide spectrum of tools is available in each platform Some tools provide a UI while others generate a log
  6. 6. BLACK BOX TESTING TOOLS Tool RemarksApache - Used for server load testing in client-server applicationsJMeter - Supports files, servlets, Perl scripts, Java objects, database queries, ftp servers - Works with HTTP, HTTPS, SOAP, Database via JDBC, LDAP, JMS, POP3 and IMAP protocols - Portable across platforms - Can replay test results - Highly extendable via plugins - Provides simple and intuitive UI - Does not support client-side scriptsMicrosoft - Captures system level problems that occur when a program is runApplication - Can monitor Virtual memory usage, First chance access violation exceptions, Input-Verifier output transfers, Synchronization objects (e.g. locks), Handle usage, Thread pool usage, Thread local storage, Dll handling, Interactive service creation, Dangerous API calls, Driver installations and User privilege issues - Writes results to an XML log - Optionally a debugger can be attached
  7. 7. BLACK BOX TESTING TOOLS CONTD. Tool RemarksLeakDiag & - A pair of tools that can be used in conjunction to detect memory leaks in anLDGrapher application - Can monitor Virtual memory allocator, Windows heap allocator, COM allocator, C runtime allocator and TLS slot allocator - LeakDiag generates an XML log, which can be graphically viewed through LDGrapherProcess - Windows logging utility, which collects and records most of the system activityMonitor regarding file system, registry and process/thread activity - Comes with a rich GUI that displays information in real time - Offers advanced filtering options to trace specific activity in the interest - Provides a boot-time logger - Should be run only for a limited time
  8. 8. BLACK BOX TESTING TOOLS CONTD. Tool RemarksXPerf - Performance-profiling tool for Windows applications - Can be used for tracking performance bottlenecks in applications as well as for comparing time taken for various operations in an application - Writes monitored data into an XML log which can be later explored via XPerfView - Comes with a rich collection of preset monitoring profilesApplication - This is a Windows application lifecycle management toolsetCompatibility - Toolset comprises Standard User Analyzer, Internet Explorer Test Tool and SetupToolkit Analysis Tool - Can be utilized to • Analyze portfolio of applications, web sites and computers • Centrally manage compatibility evaluators and settings, Rationalize and organize applications, web sites and computers • Prioritize application compatibility with filtered reporting • Add and manage issues and solutions for enterprise- computing environment • Deploy automated mitigations to known compatibility issues • Send and receive compatibility information from the Microsoft Compatibility Exchange
  9. 9. BLACK BOX TESTING TOOLS CONTD. Tool RemarksGod - Process-monitoring tool for Linux, BSD and Darwin systems - Provides watchdog functionality on processes with respect to their CPU and memory usage - Can be configured to perform certain actions on a process depending on its CPU and memory consumption - Has both a polling mode and an event responder - Generates a line log - Capable of sending alerts as email, chat or Twitter messagesInstruments - Standard tool for profiling processes that run under Mac OS X or iOS - Uses monitoring templates called instruments to collect data on various aspects - Monitoring areas include CPU consumption, memory usage, memory leaks, I/O operations, power usage and network traffic - Data viewers are built in - Capable of recording a sequence of actions to replay later
  10. 10. Tool usage requires platform expertise Analysis Generated logs knowledge not are bulky shared Black box testing pitfalls Difficult to communicate Difficult to results to correlate results different from multiple stakeholders in tools a project Lot of manual workFramework is needed to integrate tools and automate analysis
  12. 12. CONCLUSIONSLDEL significantly reduces code for log data extractionLDEL improves readability of the data extraction codeIntroduction of SQLite into data manager enabledhandling high data volumes and persistenceNeed a better database for efficiently handlingheterogeneous log dataMML is not appropriate for UI generation