Presentation from Quest 2015 - Covers building a new QA Team that matters, how to approach Agile Testing, and how to present the message to renovate your existing QA team for Agile.

1. MAURIZIO MANCINI • DIRECTOR OF QA • YELLOW PAGES CANADA • @QAANDPROCESSGUY
How to build a QA Team that matters
in an
Agile or Incremental and Iterative (IID) world.
GIANCARLO BISCEGLIA • SENIOR QA MANAGER • YELLOW PAGES CANADA

2. Who are we?
• Head of QA for Yellow Pages Canada (YP)
• Also responsible for Software Process Deployment at YP
• 25 years of software engineering experience
Maurizio Mancini
Giancarlo Bisceglia
• Senior QA Manager for Yellow Pages Canada (YP)
• 15 years of software engineering experience

3. About Our Team
• YP has ~ 2300 employees and ~ 1 Billion in Revenue
• Our team of 100 QA are embedded into 15 development teams
• Our team is geographically disbursed in Montreal (4 locations) and in Toronto (2
locations)
• Our team works on projects using all types of software development process
(Agile, Incremental & Iterative (IID), Waterfall)

4. Agenda
• Yellow Pages Yesterday and Today
• QA “State of the Nation” from our perspective
• How to hire and build a QA Team that will succeed in any SDLC
• Agile Testing – What’s that about?
• Test Automation – Fundamental Building Block
• The Testing Pyramid – Where does QA fit in?
• Different Layers of Testing
• Where else does QA fit in an Agile or IID World?

5. Yellow Pages Canada
• 3 years ago no real QA Team existed
• A few scattered QA souls left to fend
for themselves
• A lot of consultants
• Almost all testing was manual, very
little automation
• A lot of “End to End” testing
• Most QA was done by BA’s and The
Business
Yesterday Today
• Team of 100 QA Engineers, Test
Automation Developers, and QA
Analysts
• A full QA practice that uses common
tools and QA approaches
• On average our testing is 50-50
Automated/Manual
• QA Teams focus on System Testing and
BA/Product Owners focus on UAT
• QA teams are Embedded with the Dev
Teams

6. What do our Teams Test?

7. Releases in 2014

8. QA “State of the Nation”
• Over the last 30 to 40 years, the QA software industry has gone from being a
“nice to have”
“a must”
“developers will do it all” in the Agile world

9. QA “State of the Nation”
Like all extremes, we believe that “pendulum of sensibility” is starting to settle in
the middle

10. What is the role of QA in today’s
Agile or IID world?

12. The Death of QA…
One of the heated debates in the industry is that QA is dead with Agile becoming
the predominant software development process…
We like to say that Agile has finally brought balance to the QA world…

15. Agile has brought balance to the QA world
More developers than testers, too much testing,
not enough time for QA to test everything…
Agile has helped rebalance the inequities that existed when it came to testing
software….

16. Agile has also helped reinforce that you cannot just
throw the code over the fence!!

17. Agile has brought balance to the QA world
• Agile has helped refocus development teams on the importance of testing.
• Agile has re-asserted what Software Engineering 101 has said all along…
Unit Testing is a MUST!

18. TESTING PYRAMID
Traditional Goal
End to End GUI Tests
Functional Tests
Integration
Tests
Unit Tests
GUI Tests
Functional
Tests
Integration Tests
Unit Tests
Q
A
D
E
V

19. UNIT Testing is not the

20. Donald Knuth
He once warned in a letter to one of his fans,
"Beware of bugs in the above code; I have only proved it correct, not tried it."
Knuth has been called the "father of the analysis of algorithms"
‘To tweak what Knuth said,
"Beware of the above code. I have only tested that it works. I haven't actually tried it.“’
http://jjinux.blogspot.ca/2012/05/unit-tests-dont-find-bugs-death-of-qa.html
Shannon Behrens

21. UNIT Testing is great but you
need other types of testing!

22. Where do we go from here?

23. Let’s build that QA Team!

25. Hiring
Which Roles should you be hiring for the current world of Software Development
that includes Agile, IID and sometimes waterfall?
• QA Engineer
• Also known as QA Specialist, QA Developer, Engineer in Test
• Test Automation Engineer
• QA Analyst

26. Hiring
What should the mix be?
• QA Engineer (75%)
• Also known as QA Specialist, QA Developer, Engineer in Test
• Test Automation Engineer (10%)
• QA Analyst (15%)

27. Candidate Profiles
Early Career – Look for people that think they want to be a developer.
Mid Career – Look for someone that may have coded before and has
discovered the love of testing.
Late Career – Look for someone with a passion for testing and breaking
software.

28. QA Engineer

29. QA Engineer
Also known as a QA Developer, QA Specialist, or Engineer in Test
Key skills and qualities you should be looking for:
 Coding and/or scripting – Experience in Record and Playback  not good enough
 API and Web services background – ex: REST and SOAP
 SQL and database background
 Networking background – Should have the basic knowledge of networking
 Linux/UNIX and/or Server background
 Passion for testing
 Solid understanding of QA and how it differs from Dev
 Understands that there is more to testing than “executing a test”
 Someone that likes to get under the hood  QA Mechanic

30. Test Automation
Developer

31. Test Automation Developer
This will be the person that will put in place your test automation framework
• Helps to implement test automation best practices for QA Engineers
• Will support the team as they use the framework in their project
• Helps conduct QA Teams code reviews

32. Test Automation Developer
Key skills and qualities you should be looking for:
 This is a pure developer that also has a passion for testing
 Object Oriented coding skills
 Scripting language skills
 Deep understanding of QA Automation Tools
 API and Web services background
 SQL and database background
 Networking background
 Linux/UNIX and/or Server background
 Solid understanding of QA and how it differs from Dev
 Understands that there is more to testing than “executing a test”
 Someone that likes to get under the hood  QA Mechanic

33. QA Analyst

34. QA Analyst
This is your manual tester
• This is the person that will also drive Exploratory Testing
• This resource can play other roles – More on this later

35. QA Analyst
 API and Web services background
 SQL and database background
 Networking background
 Solid understanding of QA and how
it differs from Dev
 Understands that there is more to
testing than “executing a test”
 Has a natural tendency to want to
understand how something works
Key skills you should be looking for:
 Passion for testing
 Not just a GUI tester, someone that likes
to get under the hood  QA Mechanic
 Drives Technology in his/her home
(Hacks home technology)
Examples:
• Uses Opensource firmware on their
home router
• Pushes usage of his/her smart phone
beyond everyday usage
• Able to clean a computer, infected with
a virus, without the help of a tool

36. Hiring Team

37. Hiring Team
• You should form a Hiring Team
• Formulate a hiring plan – Who you are looking for, what are the criteria
• Each member of the team focuses on a set of skills to look for
• Have a battery of questions that the team uses. For example:
 Linux/UNIX
 SQL
 Networking
 Database
 Coding
 Actually test something live with bugs
• This will ensure that you are hiring consistently and that you build that awesome
QA Team

39. The Interview
• QA Engineers and Test Automation Developers should be interviewed by a
Developer too
• Questions must include coding questions
• Your questions must include “white boarding”
• Include multi-dimensional questions
• Multi-System
• Many Technologies
• System Architecture Questions – Are they able to visualize it?
• Goal is to reveal if the candidate has a QA Mechanic mentality
• Do they have a QA mindset? More on this later.
• Do they have a desire to break software?
• Are they technically sound in the technical skills that you are looking for?
• Do they have the passion for QA?

40. QA Mechanic

41. Team Structure

44. Embed QA with the Dev teams!!

46. Central QA but with an Embedded Focus
• Common set of Automation Tools
• Shared Automation Framework
• Common standards
• Common Hiring practices
• Able to shift a QA resource when that resource wants to advance their career
• Helps to avoid unproductive Agile Teams
Benefits of a Central QA Team
Benefits of an Embedded QA Team
• QA involved in the daily team interactions
• QA testing early and often
• QA helps to keep a process focus within the team
• Get the benefits of a true agile team

48. How do we approach Testing?

49. Agile Testing

50. Agile Testing

51. TESTING PYRAMID
Traditional Goal
End to End GUI Tests
Functional Tests
Integration
Tests
Unit Tests
GUI Tests
Functional
Tests
Integration Tests
Unit Tests
Q
A
D
E
V

52. UNIT Testing is great but you need other types of
testing!
• Integration Testing
• Stubbing external dependencies to facilitate UI automation
• Exploratory Testing
• Performance Testing
• UAT using BDD’s

53. Unit and
Integration
Stubbed
Functional Testing
System
Integration
Performance UAT
Owner Dev QA Engineers QA Engineers
QA Analysts
QA Engineers Product Owner
QA Analyst
Type White Box Black Box Black Box Black Box Black Box
Environment None Local Deployment
or CI Server
“Prod-like”
QA
Environment
Performance
Environment
Staging or
Training
Environment
Complexity
Focus
Low Level
Correctness
Maintainable
Negative
inputs
Mid Level
Extreme cases
Negative tests
UI Permutations
Failure Tests
Mid Level
Positive Tests
Exploratory
tests
System
Integration
High Level
Scalability
Robustness
Reliability
High Level
Business
Flows
Layers of Testing

54. Test Automation

55. What tools we chose

56. • Many paid tools are based on open source libraries nowadays.
• Don’t think that because you pay for a tool, it’s best of breed.
• Beware of Scriptless tools. You still need people that can code
reusable classes. Yes you get a library but how complete is it?
• Record and playback just encourages end to end testing.
• Automation project needs to be run like a dev project, including
code reviews
• Invest in a custom framework which uses libraries or frameworks
that can be swapped out.
Don’t tie yourself to one technology

57. Leverage Continuous Integration

58. Ideal for helping Product Owners and QA Analysts define business
requirements in an English like syntax that can be automated.
Limitations of BDD’s
• Good to use for top of pyramid tests, flows that don’t change often
• Not a replacement for unit testing or automated functional tests
• Best if it’s a layer you add on top of your existing automation
framework
• Syntactically discourages code reuse
• Test Steps are expensive to maintain
BDD Usage in Automation
Behavior – Driven Development

59. • Time for Automation is considered expendable
• Environment Deployment Automation
• Coding Standards for the software being developed
Test Automation Challenges

60. A Different Way of Thinking

61. DON’T IGNORE WHAT’S IN BETWEEN
TEST WEB UI, MOBILE APPS, API, DATABASE, REPORTS
INDEPENDENTLY

62. Break up the Test Effort

63. • Most expensive testing you can do
• Too late in the Cycle
• Hard to coordinate and manage
• Don’t make this your primary method of testing
• Use it as a last resort
End to End Testing

64. • No more excuses, QA unblocks itself
• Think outside the box, to find ways to test partial
implementations (UI not implemented)
• Test system components in isolation using stubs
• Test Early, Test Often
• Push for continuous integration
• Deploy software and maintain QA environments
• Work with DevOps on deployment Automation
The New QA Mindset

65. Renovating an existing QA Team for an
Agile or IID world!

66. Skills Assessment

67. Skills Assessment
What you need
to improve
NOW
What
you have
today
What you
need
tomorrow

68. Now What?

69. Now What?
• Train Existing Staff – On the Job, Classroom, Self Study
• Hiring new people – QA Engineers, Test Automation Developers
• New Approaches to Testing – Testing Pyramid
• Embed your team(s) if you haven’t already
• Change in Roles and New Roles
How do you present your new approach to your existing staff?

70. Here is a sample presentation …

72. Quality is Free!

73. If Developers created
perfect code then you
wouldn’t need any
testing…

74. What costs is the
incorrect
implementation of a
software requirement…

75. QA is there to make developers and
development better.
We don’t ensure better software – we
enable developers to build better
software.
James Whittaker

76. Is there a role for QA
in an Agile World?

77. Yes!!!

78. TESTING PYRAMID
Traditional Goal
End to End GUI Tests
Functional Tests
Integration
Tests
Unit Tests
GUI Tests
Functional
Tests
Integration Tests
Unit Tests
Q
A
D
E
V

79. What is the difference between QA and QC?
Quality Assurance – Is focused on the development, implementation and
measurement of software processes in an organization.
Focused on Defect Prevention
Quality Control – Is focused on the comparison of a product against the
requirements and ensuring that the requirements have been met.
Reactive approach focused on defect detection and removal
L
E
S
S
M
O
R
E

80. So where does QA fit
in this new way of
thinking?

82. QA people should do
more QA and less QC!

83. QA Coaching Role
QA can play a vital role coaching the
Development Team on how to test…
…so that Developers can create better
Unit and Integration tests.

84. Automated Tests
QA can play a role in real automation…
Not Record and Playback…

85. But Manual Testing is still important…
Automated Tests are great but
look around, it is still humans
using the software…
Exploratory Testing is
essential… and QA is key in
this role!

86. Let’s not forget System Integration Testing
Most systems today integrate
with many other systems
SIT is essential…
and QA is key in this role!

87. User Community of Testers
If your systems and applications allow it , then why not create a
User Community of real testers?
Have QA manage your user
community of testers!
Employees
Early Adopters

88. There is plenty of
testing to go around!

90. QA Engineers at the Core of QA Team Agile Testing
Key Takeaways

91. New QA Mindset Use of Automation must be
at the Core of your QA
Key Takeaways

92. Thank you!
MAURIZIO MANCINI • DIRECTOR OF QA • YELLOW PAGES CANADA • @QAANDPROCESSGUY
GIANCARLO BISCEGLIA • SENIOR QA MANAGER • YELLOW PAGES CANADA