This document summarizes a project using model-based testing (MBT) to test the charging subsystem (CS) of a mobile operator. Key points: - An MBT approach was used to generate automated test cases from a state machine model of the CS to test it more quickly and thoroughly than manual testing. - Over 12,000 test cases covering all transition triples were generated and executed in under 6 hours, finding 15 new errors. This significantly improved test coverage compared to manual testing. - The mobile operator was convinced of the benefits of MBT for reducing testing time and costs while improving quality compared to their previous approach. They plan to use MBT for testing in the future.

1. Model-based TTCN-3 testing of a
mobile operator charging subsystem
T3UC, Beijing
July 2010
Dr. Andres Kull, Elvior
Dr. Kullo Raiend, Elvior

2. System Under Test
► Customer: Estonian mobile operator EMT (www.emt.ee)
► SUT: post-paid data charging subsystem in EMT
► CS: provided by Ericsson (www.ericsson.com)
► CS is customized using EMT business rules
Billing Subsystem
Charging Provisioning
accumulators
GSM/WCDMA
SUT: Charging Subsystem
Network Data session

3. CS core functionality
► Subscribers are provisioned by billing subsystem
► Provisioning the charging rules to the subscribers
► According to the data sessions the subscriber
account is credited by CS
Billing Subsystem
Billing Sybsystem
Charging Provisioning
Charging reports Provisioning
accumulators
GSM/WCDMA
GSM/WCDMA Data usage
SUT: Charging Subsystem
Charging Subsystem
Network
Network Data session

4. CS testing bottlenecks
► Ericsson upgrades CS software from time to time
► Customer introduces new subscription packages or
changes existing ones from time to time
► Manual testing is time-consuming and error-prone
► Time for testing the updates is usually very short

5. Customer’s hopes from MBT
► Shorten the regression testing time
► Increase the test coverage
► Avoiding risks caused by the human factor in
executing huge amount of boring tests manually
► Reducing the amount of manual tests significantly

6. MBT workflow used
CS model in Poseidon
for UML CASE tool
Generator
TTCN3- test scripts
Develops
Defines
coverage
CS Requirements Analyses
► System requirements are modelled TTCN-3 test tool
► Tests are generated from models
► Resulting tests have predefined coverage SUT: CS

7. CS features under test
► Subscribers provisioning
► Changing subscription profiles
► Changing month and day
► Bonus data amount usage
► Free data amount usage
► Priced data usage within HPLMN limit
► Priced data usage in the case of unlimited HPLMN
► Data usage if HPLMN limit is exceeded

8. Main test scenario TTCN-3 test tool
Network Billing subsystem
SUT: Charging Subsystem (CS)
TTCN-3 test tool
Billing subsystem Network
Charging Subsystem (CS)
Provision a subcsriber Generate data session
Ask charging accumulators for the subscriber
Charging accumulators
Test if the subscriber account is credited as described by charging rules

9. CS model
► State machine models the behaviour of CS
► Only few model states (forBonus, forFree, noCharge,
Priced, LimitExceeded)
► 16 context variables
► Properties
of subscriber profiles
► CS accumulators for different purposes
► 106 transitions
► Express CS and environment transactions
► Used for modelling the charging rules

10. Tests generation and execution
Coverage Generation Test lengh Amount of Execution
time (transitions) TTCN-3 (LOC) time
All transitions 2 min 213 9 213 5 min
All transition 57 min 1 672 22 765 24 min
pairs
All transition 18 h 49 min 12 807 89 191 5h
triples

11. Detected errors
► Note: CS was quite well manually tested before the
MBT started
► In total 15 new errors were detected
► Detected charging errors could spoil the reputation of
the operator

12. Project process, time spent
► Beginning (94 h, 31%)
► Introduction to problem domain
► Setting up test environment
► Executing 1st generated test case
► Incremental development to cover the scope
(142 h, 47%)
► Model updates Results Results Model
Results Model Model
► Test generation analysis analysis uddate
analysis uddate uddate
► Test execution
► Analysing results Test Test Test Test Test Test
executionexecutiongeneration
execution generationgeneration
► Refactoring and test generation for different
subscription profiles (32 h, 11%)
► Creating of documentation (32 h, 11%)

13. Test automation specialists feedback
► Higher test coverage than manually scripted
► The tests building productivity increases significantly
► Long generated test cases detected otherwise hard
to find errors
► Significant tests maintenance costs decrease is
foreseen
► Requirements traceability and results analysis is the
key issue that should be improved in MBT

14. Conclusions
► Manual to MBT  reduces test quality decrease
caused by human factor
► All transition test coverage tests allow quickly (5min)
verify in regression test the past functionality
► MBT revealed significant amount of errors that were
not disclosed in manual tests
► At the end of project the testing staff was convinced
that CS is ready for taking into real use
► MBT is the technology that EMT will start to use

15. Thank you!
More information:
www.elvior.com
Cut your sofware testing expenses
Functional black-box tests automation