Why use Specflow (Gherkin) and Selenium ? Unit tests, functional tests, integration tests, acceptance tests ...

1. Software Testing &
Quality Assurance
Why using Specflow (Gherkin) and Selenium ?
Benjamin Baumann
@Zentiltoutou
October 21st 2016

2. How much costs a bug?
• 1962 NASA - Mariner 1
Omission of a hyphen in instructions. Auto destruct. $18M
• 1996 Ariane 5
Integer Overflow on old legacy code that is not needed. No flight
simulator to spare money. $370M for satellites alone

3. Coder punchline #1

4. How much costs a bug?
• 2009 Toyota
Lag in the anti-lock-brake system. Recall 9M cars. $3G estimated
• 2012 Knight Capital Group
Code deployed to all but one server. This server sent out 4M+ wrong
stock orders in one hour. $440M

6. But we’re no Spacecraft or trading firm!
• 2015 Eurosport Player V6
Video lags, fps drop, overheating PC, angry clients.
Due to incompatibility between technology and requirements.
Detected by beta testing but gone live before the test results

7. Ok. Let’s test! White-box? Black-box?
• White-box testing
Tests the internal structure or workings of a program :
• Unit tests
• Functional testing
• Integration testing
• Regression testing
• Load & Stress testing
• Sanity & Smoke testing
• Black-box testing
Software is a black box. Tester knows nothing about how it works.
• Acceptance testing
• Beta testing

8. Unit tests are idiots.

9. Unit tests are idiots.

10. Unit tests are idiots but legion!
• Think about code coverage
• Assess code quality, not software’s
• Think about edge cases, error handling

11. Unit tests are idiots, legion and hard to
maintain
• Each code change/writing begin with test change/writing
• « I’m a developer, these tests are idiots! Why do I have to write
them? »
• « The tests failed, such a pain. I’ll just comment them out »

12. Break #1 : That’s not me. All internet is saying it!

13. That’s not me. All internet is saying it!

14. That’s not me. All internet is saying it!

15. That’s not me. All internet is saying it!

16. Still, I want to do unit testing
• Please, do unit testing only if :
• You have a decent number of developers in your team
• You have time to do it and to maintain it
• You can run them and validate them automatically (through CI)
• You run a business critical project
• You have detailled technical specs
• Coding unit test is still coding, treat your tests like your code
• Don’t write the code AND the tests, make someone else write the tests.
• Golden rule : « Each unit test case should be very limited in scope »

17. Ok I have a ton of unit tests. Am I OK?

18. Ok I have a ton of unit tests. Am I OK?

19. Focus on functional,regression and
acceptance tests
• Testing software features above code methods
• Less detailed, less to do, less to maintain
• Focus on testing the requirements/specs
• Safeguard for developers

20. Ok let’s test a feature !
• Describe the feature : Who? What? Why?
• « In order to watch the player
As a Eurosport Player user
I want to be able to connect»
• Help describing the need
• Feature-centric
• Technology agnostic
• Need for testing scenarios

21. Ok let’s write a scenario !
Given I go to the Eurosport Player home page
And I open the connection modal
And I enter my credentials
When I submit the form
Then I must be logged in
White-box? Black-box?

22. Ok let’s write a scenario!
Given I go to the Eurosport Player home page
And I open the connection modal
And I enter my credentials
When I submit the form
Then I must be logged in
White-box! Describing a feature should be blackboxed.

23. Ok let’s write a better scenario!
Given I go to the Eurosport Player
When I login
Then I must be able to watch Eurosport Player
No tech. No design stuff. Just the feature.
Feature does not change when code is refactored.
You should rewrite the code behind, but not the test scenario

24. Break #2: Understanding others

25. Break #2: Understanding others
• « Hey, this is not the behaviour I asked for !», « But that’s what you
told me in your mail… », « You did not understand, that’s not what I
wanted »
• « Domain, Network, AD, I mix up all these, it’s gibberish to me »
• Solutions
• Learn the other language so that you understand them
• Everyone learns a new language so that we understand each other
(AWS)
• Don’t go into details, just do high level
• Speak like you’d speak to the average man in the street

26. What about Gherkin?
• We’ve already used Gherkin
• Gherkin is the language that Specflow understands. It is a Business
Readable, Domain Specific Language that lets you describe software’s
behaviour without detailing how that behaviour is implemented.
• Gherkin serves two purposes — documentation and automated tests.
The third is a bonus feature — when it yells in red it’s talking to you,
telling you what code you should write.

27. Documentation by Gherkin
• List features
• List testing scenarios
• Readable by EVERYONE
• Hard to write, easy to understand
• A heaven for maintenance
• http://qa.sti.eurosport.intra/Home/SpecflowList?path=d%3A%5Csites
%5CQA%5CProjects%5CEurosport%20Player%20PROD%5CScenarios&
projectName=Eurosport%20Player%20PROD

28. Automated tests with Specflow
• Each scenario line is linked to a code method
• Pass scenario data to the method
• Scenario Context (browser, language, environment…)
• Runnable in classic test runners (NUnit, MsTest …)
• Explicit errors
• When failed, need to go deeper with unit tests or by hand

29. And what about Selenium?
• Selenium WebDriver is a library that can drive a web browser
• Method on the webdriver results on « real user » action
• Let’ code!

30. Merci!