EuroSTAR Software Testing Conference 2012 presentation on Model-Based Tested for Integration Tested in Real Production by Olli-Pekka Puolitaival. See more at: http://conference.eurostarsoftwaretesting.com/past-presentations/

2. Olli-Pekka Puolitaival
• Lead quality engineer test
automation
• 5 yrs test automation
• 4 yrs research
Protecting the irreplaceable | f-secure.com

3. Index
• Idea of model-based testing
• Story of OSMO tester
• How we test
• Experiences & tips

4. MBT is new way to think
• Traditional way to think
• People do testing
• Regression automation reduce peoples work time
• New way to think
• Smart people want to do testing in smart way
• Computers are fast and hard worker, let them also try
“stupid” things

5. 5

6. 6

7. 7

8. 8

9. 9

10. 10

11. 11

12. 12

13. 13

14. Why we made osmo?
Traditional MBT tools
• Hard to take in use
• Closed
• Limited extending
• Advanced algorithms
• Expensive
Osmo tester
• Easy start
• Flexible
• Easy to extend
• Simple algorithms
• Free (open source)
14

15. Osmo tester idea
public class ExampleModel{
int i = 0;
@Guard(“increase")
public guardfunction() {
return true; //Return true if this transition is allowed to execute
}
@Transition(“example")
public void transitionfunction() {
System.out.println(i++); //This code is executed when transition is selected
}
…
public static void main(String[] args) {
OSMOTester tester = new OSMOTester(new ExampleModel());
tester.generate();
}
}
15
“Model is pure java code,
Which makes possible to use
Any java library”

16. Many additional features
• Support online and offline MBT
• Requirements tagging
• Coverage calculations
• Many test design algorithms, also manual drive
• Coverage, time and length based stopping criteria
• Self written adapters to the SUT
• Custom test output formats
• Easy to integrate with Jenkins
• Model reuse and combining
• Probability numbers support
• Available: http://code.google.com/p/osmo/
16
Easy to start and
custom for your needs

17. Tips
• Modelling
• Good model is high level, modular and cover core functionality
• Start from small and continue iteratively
• Adaptation
• Reuse code from test automation or development if possible
• Running
• Run tests always automatically
• Fixed run length for verifying and ”unlimited” for finding some new
• Reports
• Report what you need to know, visual is mostly better
• OSMO is not good if you need very sophisticated algorithms, but most people
don’t need those
17

18. The domain is content cloud

19. Client smoke
Our continuous integration
Backend
components
Backend
components
Building
Clients
19
Backend
components
Clients
Client
Integration
test
environment
Unittest
Clients
Smoke test
Regression test
Backend
integration
Test
Environment
Performance
test
Performance
Test
Environment
Client regression
Chain of Jenkins jobs runs everything

20. Jenkins
Android phone
Android client
CAN
Solution
Backend
Current test setup
“run once per hour”
Selenium webdriver
Firefox
Web adapter
Test status
radiator
Nativedriver client
Nativedriver
server
Model
OSMO
Android adapter

21. What kind of model?
• Idea of this testing
• Do actions and verify that other clients data is same
• Model includes actions which can be seen in other client like
• Upload a file
• Remove a file
• Rename a file
• Check quota
• Check that file exist
• Check that all files exist as supposed
21

22. Example test run
22
Osmo test engine Web page Android Client
Login
Upload file1.jpg
Login
Check file1.jpg
Upload file2.jpg
Remove file1.jpg
Check that only
file2.jpg exist
Logout
Logout

23. What are changing in test runs
• New file set each time
• New test accounts each time
• Random amount of uploaded files
• Randomly removed files
• Randomly renamed files
• Random checks
23

24. Experiences
• Useful testing because
• Tells well the status of core functionality
• Is very close the new user use case
• Test generation is easy to do with OSMO
• Modeling is easy and support well iterative development
• Test parameterization is easy
• The challenges
• Adapters are not easy to do and maintain
• I keep my testing enough simple, then I don’t need too much maintenance
• Automating test run was challenging in first time
• Developers was helping me
24

25. Next steps
• Look for new adapter for android
• Extend testing on other platforms
• Extend the power
• Cover many environments
• Many machines for testing
• Run continuously several kind of test sets:
• Short run for radiator
• “infinite run” for finding weakest point
25

26. My MBT cooking receipt
1. Find out the core functionality
2. Make a simple model of that
3. Make adaptation for test running
4. Automate test runs
5. Run model continuously against
SUT
6. Put the results visible in your room
for seeing the “temperature”
7. Keep the radiator GREEN
26