The document discusses the challenges and criticisms of using Cucumber for automated testing, emphasizing its frequent misuse, high costs, and slow test suites. It highlights the disconnect between business and development teams regarding Gherkin files and suggests that developers often end up writing subpar specifications. Alternatives such as RSpec and RSpec-Gherkin are presented as potentially better solutions depending on the context and stakeholder involvement.

1. Cucumber
WE are Doing it Wrong
name = 'Dmitry Sharkov'
twitter = "@#{name.gsub(' ','')}"
company = 'CareWorks Tech'

2. Is Cucumber Bad?
teammate92.deviantart.com

3. No. But it is frequently used
inappropriately and ends up
not as valuable as expected.
http://whynne.deviantart.com/

4. What we will talk about:
Cucumber's value proposition
Challenges that I have seen teams experience with it
Various developers hating on Cucumber
Cucumber's developer bias
High cost of Cucumber as a functional spec
The compromises made
Alternatives to Cucumber
Selecting the right tool for the situation

5. • Devs understanding requirements is necessary
• Business-legible specification is valuable
• Acceptance-level testing is valuable
• Automated testing is key to agility
• Traceability from specification to tests is valuable
• Green Arrow is basically a "budget" Batman
• Capturing preconditions in specs is valuable
Let's agree that…

6. Sounds a lot like…
www.effenvodka.com/
and many teams struggle with it.

7. Cukes gone bad
Business does not contribute to or
consume Feature (Gherkin) files.
Developers write and use them.
The Gherkin produced is poor and
difficult to elegantly map to code.
Tests are taking up almost as much
time/money as implementation code.
Test suites take too long to complete.
Test-first approach doesn't last long.
http://plantsvszombies.wikia.com/wiki/File:Nuclear_Cucumber.png

8. Are devs writing the Gherkin?
Business creates requirements with JIRA or a
wiki and shows little interest in Gherkin. Devs
write it since they're using Cucumber.
Or the business generates poor Gherkin. Devs
re-write it since they're using Cucumber.
This can lead to frustration and wasted time.

9. Does the Cucumber test suite take
too long to run outside of a CI grid?
Does it not get run as code is
written, prior to integration?
Are the tests too slow in CI to
provide a good feedback loop?
Regressions are being caught later.
http://pokemon.wikia.com/wiki/Slowpoke

10. Does Cucumber feel too costly?
Estimates are inflated due to Cucumber.
Developers see it as a burden and don't feel a
positive R.O.I.
Do developers often joke about writing lots of
test code to test a minor change?
Low velocity, low morale. Cucumber is at risk.

11. Too expensive, too difficult, too slow.
Over-using Cucumber.
Not capturing as much
as possible in unit and
integration tests.
Possibly over-testing
altogether.
Look at what questions remain unanswered and
answer them in the simplest, fastest way possible.
http://i.stack.imgur.com/U1mq6.jpg

12. Business and development
having trouble with the Gherkin
IS unique to Cucumber.
Why?
Problems of cost, speed, and complexity are
not unique to Cucumber.

13. Cucumber is actually NOT great at this
• Devs understanding requirements is necessary
• Business-legible specification is valuable
• Acceptance-level testing is valuable
• Automated testing is key to agility
• Traceability from specification to tests is valuable
• Green Arrow is basically a "budget" Batman
• Capturing preconditions in specs is valuable

14. By the way, this isn't news.
It's easy to find
developers
critiquing
Cucumber online.
https://www.flickr.com/photos/sabrewer_1/3767863496/

15. –James Shore
(author of The Art Of Agile Development)
http://www.jamesshore.com/Blog/The-Problems-With-Acceptance-Testing.html
“These two problems--that customers don't participate,
which eliminates the purpose of acceptance testing,
and that they create a significant maintenance burden,
means that acceptance testing isn't worth the cost. I no
longer use it or recommend it.”

16. –Kevin Liddle
http://blog.8thlight.com/kevin-liddle/2013/09/18/a-case-against-cucumber.html
“In theory, this is a valuable thing, a bridge between
the divergent worlds of developers and managers. In
practice, however, I’ve never seen Cucumber used
this way… Quit writing [Gherkin] unless you can
honestly say that there is someone reading them
who would not understand pure Ruby.”

17. –David Hansson (creator of RoR)
https://signalvnoise.com/posts/3159-testing-like-the-tsa
“Don’t use Cucumber unless you live in the magic
kingdom of non-programmers-writing-tests (and
send me a bottle of fairy dust if you’re there!)”

18. –Robert "Uncle Bob" Martin
http://blog.8thlight.com/uncle-bob/2013/09/26/AT-FAIL.html
“...the programmers… decided to write the acceptance
tests themselves….
...the point was to get the business to provide a formal
specification of the system so that the programmers could
understand it. What… is the point of having the
programmers write the formal specification so that the
programmers can then understand it?
But then... the Cucumber folks decided to turn Cucumber
into a programmer tool! …They made rampant use of
regular expressions. They did everything they could to
make it easy for programmers to write these tests.”
Never offers a solution, but does clarify the problem.

19. Strategies are at odds.
Art by Jim Lee, Alex Sinclair, Scott Williams
Steps should be
business-oriented
and focused.
Steps are reusable and
support composition.
Cucumber is helpful if you want to do this

20. Cucumber is a great developer-oriented tool.
Regular expressions
Nested steps
Hooks and Tags
Transforms
http://cucumber.gigidigi.com/
Reuse, maintainability,
general "code quality"
Harder for non-devs to
write the Gherkin.

21. Using Cucumber's developer-friendly
features can yield a poor functional spec:
@exchangeUser
Scenario: user resets password
Given I am not logged in
When I go to the login page
And I click on the forgot password link
And I am on the forgot password page
And I enter my email address
And click the reset password button
Then I get an email with a reset password link
When I click on the link in the email
Then I go to the reset password page
And I enter a new password
…

22. Scenario: user resets password
Given I forgot my password
When I reset it with an email
Then I can log in with the new password
A business-oriented functional spec
reads better but may have other issues.
• Steps are rarely reusable.
• Steps can turn out to be no-ops.
• Important interactions are hidden.
• Maintainability is not actually improved, either.
(Code is likely less modular and more
repetitive.)

23. Write legible, focused
business specification
and not much of the Cucumber
framework is taken advantage of.
Cucumber forces a compromise if
we want non-technical people to
author the spec.

24. Gherkin can
make for great
specification.
But Cucumber's
stepwise automation
of Gherkin is a
double-edged
sword.
Gherkin != Cucumber

25. Gherkin != Cucumber
Gherkin's value for a
specification is to force
consideration of
preconditions.
Cucumber's value is to make
it a living, executable spec.
Costs:value curves are linear
vs logarithmic.
Cucumber needs Gherkin. But
not vice-versa.
http://thebigfatnoodle.com/top-nosh/the-gherkins-4030-restaurant-and-bar/

26. I have always argued in favor of
But at times I have felt the tight coupling
between steps and code to be a hindrance.

27. RSpec-Gherkin
Introducing…
https://rubygems.org/gems/rspec-gherkin
http://www.savills.co.uk/blog/article/185219/commercial-property/how-savills-sold-the-gherkin.aspx

28. Not quite ready for prime time…
…but may become a useful option.

29. An updated, functioning
version is in development.
(quick demo)

30. Let's revisit what our
NEEDS are.
Does Cucumber help meet them? Does
Gherkin? Might something else?

31. • Devs understanding requirements is necessary
• Business-legible specification is valuable
• Acceptance-level testing is valuable
• Automated testing is key to agility
• Traceability from specification to tests is valuable
• Capturing preconditions in specs is valuable

32. Developers understanding requirements
Gherkin done well is best due to
specification by example. Done
poorly it is probably worse than a
simple list of acceptance criteria.
JIRA (or the like) can link to
supporting docs (e.g. wireframes).
Whatever tool helps facilitate
conversation between devs and the
business (or BA) is the one to use.

33. Business-legible specification
Cucumber allows for very legible
specs with a bit of work.
rspec-gherkin allows for them with a
bit less work (steps are not mapped
directly).
RSpec can produce legible reports but
they will probably be much less precise.
JIRA puts no constraints on a BA so
you can have maximum legibility but
absolutely no coupling to tests.

34. Automated, acceptance-level testing
Cucumber can do it.
RSpec can do it.
rspec-gherkin can do it.
Any test runner that can interface
with an app driver can.
If Gherkin isn't valued, then anything
but RSpec forces extra work.
You need good test engineers to get
it done in any case.

35. Traceability from Specification to Test
Cucumber does it well but
mapping of individual steps puts a
burden on spec authors.
rspec-gherkin has less strict
translation at lower cost.
RSpec does not capture
specification as well.

36. Capturing preconditions in specs
Gherkin is just about the only way
to guarantee this occurs.

37. Seems like Gherkin has
clear advantages.
But if the business does not collaborate
in its creation, and does not consume it
otherwise, then it is mostly waste.
And it is more difficult to write if steps
are mapped to code rather than whole
scenarios.

38. Is rspec-gherkin superior to Cucumber?
• No coupling of individual steps to code blocks.
• Less of a burden on Gherkin authors.
• Less of a burden of communication and
collaboration.
• Particularly in cases where legacy software has
no tests, no spec, and no dedicated business
stakeholder, rspec-gherkin might work well.

39. Is there a case
where Cucumber
is clearly superior?
Yes… when there are
many manual testers
who cannot transition
to programming.
http://blog.eat24hours.com/12-hidden-meanings-of-popular-food-phrases/

40. What I wish we had…
A platform connected to multiple test runners
that provided an option to have tight coupling
(of steps via Cucumber), loose coupling (of
Scenarios via rspec-gherkin), or meta reporting
(from lower-level tests) on a per-Scenario basis.

41. If the business does not consume Gherkin, just
use RSpec.
If the business wants to use Gherkin, consider
Cucumber or rspec-gherkin based on
circumstances.
Maybe we need a new tool altogether.
If it ain't broke, don't fix it.
:-/
CONCLUSIONS

42. Thanks!
Questions, Comments, Criticism?
@DmitrySharkov
dsharkov@gmail.com
dmitry.sharkov@careworkstech.com