Your SlideShare is downloading. ×
A software monitoring framework for quality verification
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.

Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

A software monitoring framework for quality verification


Published on

Software functional testing can unveil a wide range of potential malfunctions in applications. However, there is a significant fraction of errors that will be hardly detected through a traditional …

Software functional testing can unveil a wide range of potential malfunctions in applications. However, there is a significant fraction of errors that will be hardly detected through a traditional testing process. Problems such as memory corruptions, memory leaks, performance bottlenecks, low-level system call failures and I/O errors might not surface any symptoms in a tester’s machine while causing disasters in production. On the other hand, many handy tools have been emerging in all popular platforms allowing a tester or an analyst to monitor the behavior of an application with respect to these dark areas in order to identify potential fatal problems that would go unnoticed otherwise. Unfortunately, these tools are not yet in widespread use due to few reasons. First, the usage of tools requires a certain amount of expertise on system internals. Furthermore, these monitoring tools generate a vast amount of data even with elegant filtering and thereby demand a significant amount of time for an analysis even from experts. As the end result, using monitoring tools to improve software quality becomes a costly operation. Another facet of this problem is the lack of infrastructure to automate recurring analysis patterns.
This paper describes the current state of an ongoing research in developing a framework that automates a significant part of the process of monitoring various quality aspects of a software application with the utilization of tools and deriving conclusions based on results. According to our knowledge this is the first framework to do this. It formulates infrastructure for analysts to extract relevant data from monitoring tool logs, process those data, make inferences and present analysis results to a wide range of stakeholders in a project.

Published in: Technology

1 Comment
  • Be the first to like this

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide
  • I think that every presentation regarding finance or investments has to be grounded in some type of metaphor. The right metaphor allows the audience to feel immediately comfortable with the presentation even though they may not have much prior knowledge on the subject matter. To me, sailing has always seemed to provide just the right terms to describe both the planning and patience required of a successful investor. Whether it be “charting the course” of an investment plan or having the patience required to navigate choppy markets, sailing and investing can make for a compelling presentation. Disclaimer: You understand that Microsoft does not endorse or control the content provided in the following presentation. Microsoft provides this content to you for informational purposes only; it is not intended to be relied upon as business or financial advice. Microsoft does not guarantee or otherwise warrant the accuracy or validity of this information and encourages you to consult with a business or financial professional as appropriate.
  • Tools are two fold 1. white box testing tools 2. Result integration tools
  • Tools are two fold 1. white box testing tools 2. Result integration tools
  • Tools are two fold 1. white box testing tools 2. Result integration tools
  • Tools are two fold 1. white box testing tools 2. Result integration tools
  • Tools are two fold 1. white box testing tools 2. Result integration tools
  • Tools are two fold 1. white box testing tools 2. Result integration tools
  • This slide lays out the overall objective of the presentation; which is how to achieve or start working towards a “successful retirement”. It starts with a “written financial plan”, and moves clock-wise – all centered around the main goal. On this continuum, buttons can be added or removed, depending upon the financial professional giving the presentation.A successful retirement plan has many moving parts and each of them must serve a unique function in carrying you to retirement. Regardless of your situation, the journey to retirement is likely dotted with some storms and choppy seas along the way. Here, I have laid out what I believe to be the very basic principles necessary for any successful retirement plan.
  • Tools are two fold 1. white box testing tools 2. Result integration tools
  • Transcript

    • 2. QUALITY IS PARAMOUNTOperational Overhead Poor Quality • Constant re-planning Kills • Derailing deadlines • Higher costs Through-life Costs • Recalls / Updates • Warranty Claims • Litigation Business Value Decline • Loss of market share • Dropped brand equity
    • 3. FUNCTIONAL TESTINGFocuses on spotting any observable deviations of the product with respect to the desired features and behavior Both manual and automatedCovers most feature-related malfunctionsLimited to only the behaviors observable directly
    • 4. WHITE BOX TESTING Supplements functional testing Directly examines code Can test cyclometric complexity, testability, unit test coverage, potential memory leaks, class and function level complexity, deprecated API usage, code duplication, undocumented code, styling errors Many tools exist for this Does not cover operational errors
    • 5. BLACK BOX TESTING Certain errors are not captured either by  Memory functional testing or by white box testing corruptions, memory leaks  Disk access errors They need monitoring application in  Failures in certain low level operation operating system calls  Pitfalls due to insufficient Black box testing is required for a testing user access rights process to be completed  Unjustifiable performance bottlenecks Wide spectrum of tools is available in each platform Some tools provide a UI while others generate a log
    • 6. BLACK BOX TESTING TOOLS Tool RemarksApache - Used for server load testing in client-server applicationsJMeter - Supports files, servlets, Perl scripts, Java objects, database queries, ftp servers - Works with HTTP, HTTPS, SOAP, Database via JDBC, LDAP, JMS, POP3 and IMAP protocols - Portable across platforms - Can replay test results - Highly extendable via plugins - Provides simple and intuitive UI - Does not support client-side scriptsMicrosoft - Captures system level problems that occur when a program is runApplication - Can monitor Virtual memory usage, First chance access violation exceptions, Input-Verifier output transfers, Synchronization objects (e.g. locks), Handle usage, Thread pool usage, Thread local storage, Dll handling, Interactive service creation, Dangerous API calls, Driver installations and User privilege issues - Writes results to an XML log - Optionally a debugger can be attached
    • 7. BLACK BOX TESTING TOOLS CONTD. Tool RemarksLeakDiag & - A pair of tools that can be used in conjunction to detect memory leaks in anLDGrapher application - Can monitor Virtual memory allocator, Windows heap allocator, COM allocator, C runtime allocator and TLS slot allocator - LeakDiag generates an XML log, which can be graphically viewed through LDGrapherProcess - Windows logging utility, which collects and records most of the system activityMonitor regarding file system, registry and process/thread activity - Comes with a rich GUI that displays information in real time - Offers advanced filtering options to trace specific activity in the interest - Provides a boot-time logger - Should be run only for a limited time
    • 8. BLACK BOX TESTING TOOLS CONTD. Tool RemarksXPerf - Performance-profiling tool for Windows applications - Can be used for tracking performance bottlenecks in applications as well as for comparing time taken for various operations in an application - Writes monitored data into an XML log which can be later explored via XPerfView - Comes with a rich collection of preset monitoring profilesApplication - This is a Windows application lifecycle management toolsetCompatibility - Toolset comprises Standard User Analyzer, Internet Explorer Test Tool and SetupToolkit Analysis Tool - Can be utilized to • Analyze portfolio of applications, web sites and computers • Centrally manage compatibility evaluators and settings, Rationalize and organize applications, web sites and computers • Prioritize application compatibility with filtered reporting • Add and manage issues and solutions for enterprise- computing environment • Deploy automated mitigations to known compatibility issues • Send and receive compatibility information from the Microsoft Compatibility Exchange
    • 9. BLACK BOX TESTING TOOLS CONTD. Tool RemarksGod - Process-monitoring tool for Linux, BSD and Darwin systems - Provides watchdog functionality on processes with respect to their CPU and memory usage - Can be configured to perform certain actions on a process depending on its CPU and memory consumption - Has both a polling mode and an event responder - Generates a line log - Capable of sending alerts as email, chat or Twitter messagesInstruments - Standard tool for profiling processes that run under Mac OS X or iOS - Uses monitoring templates called instruments to collect data on various aspects - Monitoring areas include CPU consumption, memory usage, memory leaks, I/O operations, power usage and network traffic - Data viewers are built in - Capable of recording a sequence of actions to replay later
    • 10. Tool usage requires platform expertise Analysis Generated logs knowledge not are bulky shared Black box testing pitfalls Difficult to communicate Difficult to results to correlate results different from multiple stakeholders in tools a project Lot of manual workFramework is needed to integrate tools and automate analysis
    • 12. CONCLUSIONSLDEL significantly reduces code for log data extractionLDEL improves readability of the data extraction codeIntroduction of SQLite into data manager enabledhandling high data volumes and persistenceNeed a better database for efficiently handlingheterogeneous log dataMML is not appropriate for UI generation