RTF - Prasad bhatt

IBM Rational Software Conference 2009


Prasad Bhat
Product Consultant, Rational Services
prasad.bhat@in.ibm.com

© 2009 IBM Corporation
1


Industry Axiom?

Quality Time
to
Market

2


Embedded Software Development Challenges
Application Complexity Process Complexity
Strong timing constraints Requirements shift
Low memory footprints Design translation errors
Concurrent/Distributed/ Lack of understanding
Networked
Difficult to maintain
Environment Complexity Poor performance
Multiple RTOS vendors Late discovery
Multiple chip vendors Incomplete integration
Multiple IDEs
Limited host-target
connectivity
Low built-in
debugging capabilities

3


Experience Tells Us…
Debugging is not testing!
Systematic testing tools and
methods are necessary to ensure
proactive and early problem
discovery
A debugger details symptoms,
but debugging requires a
diagnosis
Runtime analysis is a key
practice for simplifying the
diagnosis of issues impacting
reliability, scalability and durability

4


Would This Accelerate Quality?
Automated component testing
at all levels of complexity
From the simplest function to
distributed systems
Extensive runtime analysis
capabilities
Memory and performance
profiling, code coverage, runtime
tracing
Static metrics calculation
Assist with test prioritization
Dynamic links between code,
test results, and visual model
Full regression testing abilities

5


IBM Rational Test RealTime
Automated component testing at all levels of complexity
– from the simplest function to distributed systems
Memory and performance profiling, code coverage
measurement, runtime tracing and thread profiling
Static metrics calculation to assist with test prioritization
Dynamic links between code, test, and visual model
Full regression testing capabilities

All in one tool
All in one tool
For any IDE - Hosted on any target!
For any IDE - Hosted on any target!
6


Fix Your Code Before It Breaks
Rational Solutions and the Embedded Systems Market
Serves and Empowers Any Test and Debug Process
Delivers a Unified Component Testing and Runtime Analysis Solution
Delivers Total Environment Adaptability
What Do You Need?

7


Rational Platform: Supporting Embedded Industry

Medical Telecom Devices
Scanners
Rational Solution: Pagers
Surgical Lasers Addressing your Phones
Pace-makers embedded Switches
design, test, and Routers
management
needs

Aerospace / Military Automotive
Aircraft Body Electronics
Spacecraft Power train
Missiles Chassis

8


What Do You Need?

9


What We Have Seen
Every developer knows testing ensures quality
code…but few do more than debug

“I’ll test in the next
project – I’ve got
too much code to
write and fix”

But is there room for risk in
the embedded world?
10


A Day in the Life
Fewer tests lead
to less stability

Pressure leads Less stability leads to
to fewer tests work under pressure

11


Serves and Empowers any Test and Debug Process
Test as you code
Automatic component test template
and data generation
code code code code

Black- and white-
code code code code
code code code code
code code code code
Test
Test
box testing
code code code ode
code code code code
code code code code
code code code code

All levels of complexity:
code code code code
code

From single functions
to distributed systems
Static metric calculation for: Resolve Analyze
Resolve Analyze
• tests prioritization
• complexity estimation
Full regression testing

12


Test as you code
Analyze while you test
Code coverage code code code code

analysis
code code code code
code code code code
code code code code
Test
Test
code code code ode
code code code code

Memory profiling code code code code
code code code code
code code code code
code

Performance profiling
Runtime tracing
Resolve
Resolve Analyze
Analyze

13


Test as you code
Resolve what you have uncovered code code code code
code code code code
Test
code code code code
code code code code
Test
Test execution integrated with your code code code ode
code code code code
code code code code
debugger code code code code
code code code code
code

Consolidated, detailed
to-the-point test
reporting
Resolve
Resolve Analyze
Analyze
Test data hyperlinked
to runtime analysis results
and code

14


Test as you code
Resolve what you have uncovered code code code code
code code code code
Test
code code code code
code code code code
Test
code code code ode
code code code code
code code code code
code code code code
code code code code
code

Resolve
Resolve Analyze
Analyze

Now fix the defects, enhance your tests
Now fix the defects, enhance your tests
And move on!
And move on!
15


Eases transition from manual testing to automation
Source-code aware and thus easily adoptable
Powerful test languages drive robust data-driven tests
Creates shared debug and test vocabulary for your team
Flexible to minimally impact preexisting processes
Process agnostic, so you can stay in your comfort zone
Shortens time-to-problem-resolution, maximizes time-to-code
Mix and match functionality to accommodate your needs
Built explicitly for the rigors of embedded development
Manages target environment so you can focus on good test creation
Handles all test enablement activities so you test early and often
Guarantees test reuse when the environment changes to ensure your
testing effort will pay off in regression testing dividends

16


What Do You Need?

17


IBM Rational Test RealTime - Overview
Component Testing
Component Testing Code
Code Memory Performance Runtime Thread
Memory Performance Runtime Thread
Coverage
Coverage Profiling Profiling
Profiling Profiling Tracing Profiling
Tracing Profiling

C, C++, Ada, Java C, C++ C, C++ C, C++ C, C++ C, C++
Ada, Java Java Java Java Java

Unit, Integration, Validation, Regression and System Testing
System Testing
System Testing
C

Target Deployment Port
Built to achieve standards compliance
DO-178B
MISRA
Defense Standard 00-55
18


A Unified Component Testing and Runtime Analysis Solution
Combines entire features set into a single, unified tool
Optimizes test effort through addition of runtime analysis functionality
Accelerates problem resolution through shared team assets
Simplifies refactoring verification
Enables quality verification for all test granularity levels and
certification standards
Focuses your efforts on a single toolset
Produces information required for code certification
Provides a means for achieving pervasive quality
Accommodates the larger IBM Rational lifecycle solution
Alleviates friction generated by use of non-integrated tools
Improves team and asset stability through traceability
Single vendor relationship ensures future integration improvement

19


What Do You Need?

20



Target Deployment Technology
A low-overhead, versatile target deployment technology
• Compiler-independent high level scripting API
• Debugger-independent test harness deployment
• Target-independent results upload & report creation

Full Target Independence!
Full Target Independence!

21



4-Bit to 64-Bit
Cross-Development Environments Languages
Used By Our Customers
C
WindRiver Montavista Tasking
C++
GreenHills TI CAD-UL Ada
J2ME/J2SE
ARM NEC Cosmic

Enea Hitachi Hiware
Platforms

Windows CE Apex Hitex Windows
Solaris
LynuxWorks Sun Symbian Linux
HP-UX
Lauterbach Microtec ……
AIX

22


Customizable to support a complete range of embedded targets
Assures tool adoption
Reduces your ramp-up time when target configuration changes
Guarantees the reuse of test assets despite target constraints
Host, Build and Target Environment Agnostic
Ensures portability of test and runtime analysis processes
Simplifies multiple team deployment
Optimizes ROI in comparison to home-grown test solutions
Size and Speed Optimized to Limit Target Impact
Enables full control to minimize instrumentation overhead
Frees your tests from having to compensate for target restraints
Avoids overtasking your system

23


What Do You Need?

24


Component Testing
Coverage
Coverage Profiling
Profiling Profiling
Profiling Tracing Profiling
Tracing Profiling

System Testing
System Testing
C

Rational Rhapsody
Rational ClearCase
Rational ClearQuest
Rational Test Manager
RUP

25


Component Testing
Coverage
Coverage Profiling
Profiling Profiling
Tracing Profiling
C, C++, Ada, Java C/C++, Ada83/95, C++
C/C++, Ada83/95,
C, C++ C, C, C++ C, C++ C, C++
Java Primitives UML-based Runtime Thread
Java Java
Primitives UML-based Runtime Thread
Ada, PerformanceCode coverage
Performance Active
Java coverage
Java
Active Java Java
Code Memory Leak
Memory Leak
(J2ME/J2SE)
(J2ME/J2SE) measurements Detection
Measurement Tracing Representation
Measurement Tracing Representation
measurements Detection
System Testing
System Testing
C Message-based C
Message-based C Target-based Test
Target-based Test
Threads, Tasks,
Threads, Tasks, Execution
Execution
Processes & Nodes
Processes & Nodes Model-driven
Model-driven
Target Deployment Port Development
Development
Rational Rhapsody Test Asset Versioning
Test Asset Versioning
Rational ClearCase Change Management
Change Management
Test Asset
Test Asset
Rational ClearQuest Management
Development
Management
Development
Rational Test Manager Best Practices
Best Practices
RUP

26


27


IBM Rational Test RealTime: Test Script

28


IBM Rational Test RealTime: Metrics Calculation - 1

Metrics-derived graphical matrix
representing units complexity
used for test prioritization
Calculation of static metrics
for tested classes/functions
from selected components

29


IBM Rational Test RealTime: Metrics Calculation - 2

Calculation of static
metrics for analyzed
functions/methods
from instrumented Metrics information is
files accessed directly from
the project browser

30


IBM Rational Test RealTime: Code Coverage

31


IBM Rational Test RealTime: Memory Profiling

32


IBM Rational Test RealTime: Performance Profiling

33


IBM Rational Test RealTime: Runtime Tracing

34


IBM Rational Test RealTime: Test Report

35



Component Testing for C and Ada Presentation

Session# iSem08 36


Test RealTime: Component Testing for C & Ada
Source code-driven, data-intensive functional and structural testing
via function calls
Automatic
Test template generation from source code
Test data generation from pattern language
Stub creation
Regression Testing
Detailed reporting
Use of static metrics for test prioritization
Software Complexity Level
Works with Test RealTime Runtime Analysis features
Memory and Performance Profiling
Code Coverage and Runtime Tracing

37


Test Harness Architecture and Responsibilities
1. Test driver
Calls the function-under-test with
the desired range of parameters
Checks returned parameters
Accesses global variables
2. Stubs
Check input parameters
Return desired parameters

38


Component Testing for C & Ada: Process
Code Fix
Test Enhancement

Coding Test Script Test Result
Design Analysis
Software
Engineer

Source
Code

IBM Test RealTime Test Script Template Test Harness Test Execution Test Results
Component Testing Generation Generation Reporting
for C and Ada

Compilers /
Linkers /
Debuggers
Simulators /
Emulators /
Targets

39


Component Testing for C & Ada: Test Script Pattern

Test Script Tested Function:
Template a + b returns c
Default Automatically 3 simple integers
Test Default Generated to be checked!
Structure Values for
Tested
Tested Variables
Function
Invocation

40


Component Testing for C & Ada: Test Language
TEST 1 Input values initialization: Expected values definition:
- Multiple, - According to requirements
ELEMENT - Ranges, etc. - Using Range, delta, etc.
The
function #ret_val=myfunction(y,a,z,b,c);
under test
VAR glob, init=0 , ev==
VAR y, init in {-1,glob,0} , ev in {-1,0,0}
VAR a[1..10],init from 1 to 1000 step 1 , ev(y) in {0,2,3}
A single VAR z.field1,init=a[2] , ev=ret_val
instruction VAR b, init== , min=y, max=y*10
to define all
VAR c, init=b , ev=10, delta=10%
test data
VAR ret_val, init=MY_DEFINE , ev=init

STUB alloc_block, 0=>(100)&a, OTHERS=>()NIL In 2 lines,
3x1000=3000
END ELEMENT STUBs: test cases are
- Check parameters, generated!
- Return values
END TEST

41


Component Testing for C & Ada: Reporting
Easy to understand report
Passed and failed test
cases at a glance
Initial, expected, obtained
values for all managed
variables and stubs
Source code coverage
information from Rational
Test RealTime Code
Coverage feature
Exports to HTML great for
Distributed development
Test subcontracting

42


Component Testing for C & Ada: Tests Comparison
Between:
2 iterations of same software component
2 different development environments
Instrumented vs. non-instrumented code COMPARISON of 2
Different Executions
Generated code vs. manual code

Coding Test Script Test Result Test Result
Software Definition Analysis Analysis
Engineer

Test Re-execution
Test Execution Test Comparison
Test RealTime

Backup of previous
dynamic test results

43



Component Testing for C++ Presentation

Session# iSem08 44


Test RealTime: Component Testing for C++
Source code–driven, functional and structural testing
via method invocation scenarios
Automatic
Test template generation from source code
Assertion checking
Stub creation
Regression Testing
Detailed reporting
Use of static metrics for test prioritization
Software Complexity Level
Works with Test RealTime Runtime Analysis features

45


1. Test driver
Invokes a sequence of methods
in the set of classes under test
Assesses returned values
Assesses cluster properties
2. Stubs
Check input parameters
Return desired parameters
3. Class assertion checks
Based on user-defined contracts

46


Component Testing for C++: Process
Code Fix
Test Enhancement

Classes Test Script Test Result
Under Design Analysis
Software Development
Engineer

Source
Code

IBM Test RealTime Test Script Template Test Harness Test Execution Test Results
Component Testing Generation Generation Reporting
for C++

Compilers /
Linkers /
Debuggers
Simulators /
Emulators /
Targets

47


Component Testing for C++: Cluster Properties

Test class

Preamble
Create test context

Cluster property
definition

Test case

Cluster property check

<<Cluster>>

<<Property>>

48


Component Testing for C++: Cluster Properties

pop()
1: push(i)
2: j=pop() s:Stack
{i==j}

s:Stack

isEmpty

Cluster under test
stimulation using C++ call

Checks
Called methods
Boolean expressions
Postamble:
Clean test context

Test class instantiation

49


Component Testing for C++: Class Assertion
A set of conditions expected to always be true
Stack

pop():Item
push(I:Item)
clear()

Class context

Class invariant

Method pre-condition

Method post-condition

State definitions

Transitions
between
states

50


Component Testing for C++: Test Report
Easy to understand report:
Passed and failed test cases and assertion at a glance

Status for all executed
test cases and test suites

Summary of test and
contract-check

Listing of failed test cases

Targeted
Status
Method

Assertion

51



System Testing for C Presentation

Session# iSem08 52


Test RealTime: System Testing for C
Component Testing
Coverage
Coverage Profiling
Profiling Profiling
Tracing Profiling

System Testing
System Testing
C Message-based C
Message-based C
Threads, Tasks,
Threads, Tasks,
Processes & Nodes
Processes & Nodes
Rational Rose RealTime
Rational ClearCase
Rational ClearQuest
Rational Test Manager
RUP

53


System Testing for C: Message-Passing Testing
Integration and validation testing from:
Single thread … up to
Task(s) … up to
Node(s) … up to
Large networked system

Functional, load, and performance testing via message-passing API
Powerful scripting language
Detailed reporting
Regression testing
Works with Test RealTime runtime analysis features

54


1. Virtual Testers
Simulates external systems
Stubs internal actors
2. Each Virtual Tester
Sends events to the SUT
Controls the event flow
Checks event data and timing
Can be duplicated for load testing
3. System Testing supervisor
Monitoring services for virtual
tester distribution, communication
and synchronization

55


System Testing for C: Message Adaptation Layer
Full support of any
communication interface
1. Adaptation layer Virtual Tester
− Built with C procedures to send System Testing Script
and receive events (C-structs) SEND WAITTIL
− Symbolic management of events
Adaptation Layer
− Enables test script independence
of the messaging API API
2. Message API
− Part of the system-under-test
− Provided by a communication card
− Defines how to send and receive
events S UT

56


System Testing for C: Test Script Structuring
INITIALIZATION init_proc() Preamble, Postamble and
TERMINATION end_proc() Error recovery blocks
EXCEPTION recover_proc()

SCENARIO main
Test script composed
SCENARIO test_case1 of SCENARIOs and
sub-SCENARIOs …
END SCENARIO -- test_case1

SCENARIO test_case2
A scenario can be split into
INSTANCE Virtual_Tester1 INSTANCE blocks to define
asynchronous behaviors
END INSTANCE – Virtual_Tester1 (Virtual Testers)

INSTANCE Virtual_Tester2

END INSTANCE – Virtual_Tester2

END SCENARIO -- test_case2

END SCENARIO -- main 57


System Testing for C: Test Script Behavior
CHANNEL MY_COMMTYPE:mylink

SCENARIO example
Loops
WHILE (TIME(mytimer) < 100)
Synchronizations between
RENDEZVOUS start_example
Virtual Testers
IF (sync == 0) THEN
IF statements

Init of outgoing Events with VAR creq, INIT={send=>…,neg=>{opt=> …, }}
high level instructions
SEND (mylink, creq)
Send an Event via a defined
communication channel END IF

Set up a timer TIMER mytimer

Definition of constraints on DEF_MESSAGE cresp, EV= { … }
incoming Events
WAITTIL (MATCHED(cresp)||MATCHING(cack),WTIME>15)
Wait for complex conditions
CALL myexternal_func()
Call to external C code
END WHILE
END SCENARIO -- example

58


System Testing for C: Reporting
Dynamically jump between sequence diagram,
test report, and source code
Export to HTML Results Summary Test Summary

Test details

Virtual Testers

59


Probe Feature: Show Exchanges and Replay Them
Deployment Process
Define probes location in applications/simulators for sent and received messages
Execute normally using simulators/Hardware/GUIs to create/populate the test script
After execution, a UML/SD report shows all exchanged messages or events and
generates the script to replay

SUT Application Dynamic Results

Protocol Dynamic Results

External External
simulators equipments ENV L1 L2

Virtual Testers
Bus
Generated Script
UML/SD report

60



Runtime Analysis Features Presentation

Session# iSem08 61


Benefits of Runtime Analysis Features
Get insights into program execution
See how various components of a running application may affect
each other during execution
Get an overall picture of an application's execution behavior
over a period of time
Detect hard-to-find problems
Memory leaks
Performance bottlenecks
Unused and/or untested code
Directly correlate all analysis results to
Test cases and code

62


Runtime Analysis Features: Code Coverage

Up to nine incremental
Up to nine incremental
coverage levels, incl.
coverage levels, incl.
MC/DC
MC/DC
Linkage between
Linkage between
Coverage Viewer and
Coverage Viewer and
Select test case
Select test case
contribution to code Covered test cases
Covered test cases
Uncovered
contribution to code Uncovered
code
coverage
coverage code
code
code
Partially
Partially
Loops behavior
Loops behavior covered
covered
analysis
analysis code
code

Source component
Source component
Detailed
Detailed for
browser
coverage for
browser
coverage
detailed/aggregated
detailed/aggregated
statistics (%) Highlight executed/
statistics (%) rates
code coverage Highlight executed/
code coverage rates
non-executed Boolean
non-executed Boolean
conditions (C & Ada)
conditions (C & Ada)

63


Runtime Analysis Features: Memory Profiling

Memory error
statistics per
test case

Type of Call Stack
Call Stack
memory error linked to
linked to
source code
source code

Memory leak
highlighted in the
source code editor

64


Runtime Analysis Features: Performance Profiling

Graphical depiction of
slowest three performers

Performance data Function or
for one or more Function + Descendant
test cases time measurements

Sortable list of
functions and
methods

65


Runtime Analysis Features: Runtime Tracing

On-the-fly or post-mortem
On-the-fly or post-mortem
UML Sequence Diagram
UML Sequence Diagram
Powerful search,
Powerful search,
filtering &
filtering &
triggering
triggering
Track data
Track data
Source component
Source component values
values
browser for
browser for
expansion/collapse
expansion/collapse
of UML-based
of UML-based
tracing diagram
tracing diagram Timing
Timing
information
information Selected trace
Selected trace
event highlighted
event highlighted
in the source
in the source
code editor
code editor
Catch raised
Catch raised
exceptions
exceptions

66


Runtime Analysis Features: Runtime Tracing Bars

Dynamic Heap
Dynamic Heap
Memory Usage Bar
Memory Usage Bar
(Java Only)
(Java Only)

Click Threads Bar to
Active Thread
Click Threads Bar to
Active Thread
Dynamic Messages
Dynamic Messages get their properties
Representation
get their properties
Representation
Coverage Bar
Coverage Bar (C/C++/Java)
(C/C++/Java)
(C/C++/Java)
(C/C++/Java)
(C/C++/Java)
(C/C++/Java) While runtime
While runtime
tracing activity
tracing activity
is executing
is executing

67


Test RealTime: Integration with Test Manager

Test Plans are
Test Plans are
attached to Test
attached to Test
RealTime artifacts
RealTime artifacts

A direct association can be
A direct association can be
established between test
established between test
input ,, a test case and a
input a test case and a
test implementing it
test implementing it

Test Plans are built
Test Plans are built
and organized within
and organized within
TestManager
TestManager

68


Test RealTime: Integration with Test Manager

Run and monitor the tests built with
Run and monitor the tests built with
Test RealTime from TestManager
Test RealTime from TestManager

Finally, inspect from Test RealTime
Finally, inspect from Test RealTime
GUI the detailed results when
GUI the detailed results when
TestManager gives a failed status
TestManager gives a failed status

69


Test RealTime: Integration with ClearCase

Get access to
Get access to
Clearcase for all
Clearcase for all
artifacts from
artifacts from
Test RealTime
Test RealTime

70


Test RealTime 7.5 features
Integrated with new Rational Quality Manager Express Edition and Rational
Quality Manager Standard Edition
Integrated with new Rational Software Architect Standard Edition v7.5
Updated support for Eclipse v3.4 integration
New integration with Telelogic Rhapsody v7.4 and TestConductor (7.5)
Updated integration with Wind River Workbench 3.0
New out-of-the-box support for Symbian OS
New support of IPv6 network infrastructure

71


72

RTF - Prasad bhatt

More Related Content

What's hot

Similar to RTF - Prasad bhatt

More from Roopa Nadkarni

RTF - Prasad bhatt