Experience paper presentation from XP2010. Paper itself can be found from: http://gosei.fi/material/xp2010-final-public-v1.pdf

1. Automated Acceptance Testing of High
Capacity Network Gateway
Ran Nyman, Ismo Aro, Ronald Wagner
RANNICON
NSN

2. Content
Background
Initial Phase
Agile Acceptance
Testing
Growing
CI
Scaling Now
Challenges with
ATDD
Conclusions
RANNICON
NSN

3. Who are We
Ran Nyman
Ismo Aro
Agile Coach
Agile Coach
Consultant
Scrum Master
Developer
Tester
Developer
RANNICON
NSN

4. Background
Product development started in 2007
The product

High capacity GGSN and SAE
Gateway
Two teams and cultures

Waterfall

Agile
RANNICON
NSN

5. The Starting Phase
Two teams
Mixing teams
No legacy code
First steps:

PBL

Initial architecture and

Development environment
RANNICON
NSN

6. Growing First Wave
Two more teams from traditional
background


C team grasped new development ideas
fast
D team had difficulties in changing
RANNICON
NSN

7. Growing Second Wave
More local teams
Subcontractor on different continent

Same rules apply

CI challenges

Customs problems

Lot's of traveling
And it slowly started to work
RANNICON
NSN

8. Current Setup
More than 20 teams
Located in four countries
Development staff is still growing

More teams on existing sites
RANNICON
NSN

9. How Does it Work
Release at least every other sprint
“As a result of our agility in developing
products, we have been able to demonstrate the
product capabilities at an early stage of the
development cycle itself, helping us to win new
customers cases.”

Head of the product line
RANNICON
NSN

10. Agile Acceptance Testing
RANNICON
NSN

11. Agile Acceptance Testing
Acceptance test are decided in beginning
of sprint
Tests are automated during development
All acceptance test are automated
Test scripts are only documentation that is
written from tests
RANNICON
NSN

12. Tools
CI servers: Bamboo, BuildBot
Automation Framework: Robot
Framework
Protocol tester: Catapult
RANNICON
NSN

13. Continuous Integration
CI

Build (compilation and Unit Tests)

Sanity

Nightly regression
RANNICON
NSN

14. CI Initial
RANNICON
NSN

15. CI Current
RANNICON
NSN

16. Test Growth
RANNICON
NSN

17. RANNICON
NSN

18. Challenges with ATDD
Limited e2e testing:
No visibility what is ready for shipping
Hard to troubleshoot
Changing functionality leads to
unpredictable failures in tests
ATDD is done on to low level
RANNICON
NSN

19. E2E Tests in Future
Test cases are created before code and they
work as a tool of communication.
Test Cases are going to be written using
BDD. eg.
Given activate customer license
When customized packet is send to A interface
Then packet is delivered from A interface to B interface.
And no counters are increased.
RANNICON
NSN

20. Conclusions
Automated acceptance testing pays off
100% test automation is the target
Real life is more complex than you think
Do not forget the CI challenges
Splitting requirements and tracking
progress
RANNICON
NSN

21. Questions
RANNICON
NSN

22. Contact Us
Ran Nyman

ran@rannicon.com

www: http://rannicon.com
Ismo Aro

ismo.aro@nsn.com

Twitter: ismoaro

www: http://www.graceland.fi
Roland Wagner

ro.wagner@gmx.net
RANNICON
NSN