Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
MAURIZIO MANCINI • DIRECTOR OF QA • YELLOW PAGES CANADA • @QAANDPROCESSGUY
How to build a QA Team that matters
in an
Agile...
Who are we?
• Head of QA for Yellow Pages Canada (YP)
• Also responsible for Software Process Deployment at YP
• 25 years ...
About Our Team
• YP has ~ 2300 employees and ~ 1 Billion in Revenue
• Our team of 100 QA are embedded into 15 development ...
Agenda
• Yellow Pages Yesterday and Today
• QA “State of the Nation” from our perspective
• How to hire and build a QA Tea...
Yellow Pages Canada
• 3 years ago no real QA Team existed
• A few scattered QA souls left to fend
for themselves
• A lot o...
What do our Teams Test?
Releases in 2014
QA “State of the Nation”
• Over the last 30 to 40 years, the QA software industry has gone from being a
“nice to have”
“a ...
QA “State of the Nation”
Like all extremes, we believe that “pendulum of sensibility” is starting to settle in
the middle
What is the role of QA in today’s
Agile or IID world?
The Death of QA…
One of the heated debates in the industry is that QA is dead with Agile becoming
the predominant software...
Agile has brought balance to the QA world
More developers than testers, too much testing,
not enough time for QA to test e...
Agile has also helped reinforce that you cannot just
throw the code over the fence!!
Agile has brought balance to the QA world
• Agile has helped refocus development teams on the importance of testing.
• Agi...
TESTING PYRAMID
Traditional Goal
End to End GUI Tests
Functional Tests
Integration
Tests
Unit Tests
GUI Tests
Functional
T...
UNIT Testing is not the
Donald Knuth
He once warned in a letter to one of his fans,
"Beware of bugs in the above code; I have only proved it corre...
UNIT Testing is great but you
need other types of testing!
Where do we go from here?
Let’s build that QA Team!
Hiring
Which Roles should you be hiring for the current world of Software Development
that includes Agile, IID and sometim...
Hiring
What should the mix be?
• QA Engineer (75%)
• Also known as QA Specialist, QA Developer, Engineer in Test
• Test Au...
Candidate Profiles
Early Career – Look for people that think they want to be a developer.
Mid Career – Look for someone th...
QA Engineer
QA Engineer
Also known as a QA Developer, QA Specialist, or Engineer in Test
Key skills and qualities you should be lookin...
Test Automation
Developer
Test Automation Developer
This will be the person that will put in place your test automation framework
• Helps to impleme...
Test Automation Developer
Key skills and qualities you should be looking for:
 This is a pure developer that also has a p...
QA Analyst
QA Analyst
This is your manual tester
• This is the person that will also drive Exploratory Testing
• This resource can pl...
QA Analyst
 API and Web services background
 SQL and database background
 Networking background
 Solid understanding o...
Hiring Team
Hiring Team
• You should form a Hiring Team
• Formulate a hiring plan – Who you are looking for, what are the criteria
• E...
The Interview
• QA Engineers and Test Automation Developers should be interviewed by a
Developer too
• Questions must incl...
QA Mechanic
Team Structure
Embed QA with the Dev teams!!
Central QA but with an Embedded Focus
• Common set of Automation Tools
• Shared Automation Framework
• Common standards
• ...
How do we approach Testing?
Agile Testing
Agile Testing
TESTING PYRAMID
Traditional Goal
End to End GUI Tests
Functional Tests
Integration
Tests
Unit Tests
GUI Tests
Functional
T...
UNIT Testing is great but you need other types of
testing!
• Integration Testing
• Stubbing external dependencies to facil...
Unit and
Integration
Stubbed
Functional Testing
System
Integration
Performance UAT
Owner Dev QA Engineers QA Engineers
QA ...
Test Automation
What tools we chose
• Many paid tools are based on open source libraries nowadays.
• Don’t think that because you pay for a tool, it’s best of...
Leverage Continuous Integration
Ideal for helping Product Owners and QA Analysts define business
requirements in an English like syntax that can be automa...
• Time for Automation is considered expendable
• Environment Deployment Automation
• Coding Standards for the software bei...
A Different Way of Thinking
DON’T IGNORE WHAT’S IN BETWEEN
TEST WEB UI, MOBILE APPS, API, DATABASE, REPORTS
INDEPENDENTLY
Break up the Test Effort
• Most expensive testing you can do
• Too late in the Cycle
• Hard to coordinate and manage
• Don’t make this your primary...
• No more excuses, QA unblocks itself
• Think outside the box, to find ways to test partial
implementations (UI not implem...
Renovating an existing QA Team for an
Agile or IID world!
Skills Assessment
Skills Assessment
What you need
to improve
NOW
What
you have
today
What you
need
tomorrow
Now What?
Now What?
• Train Existing Staff – On the Job, Classroom, Self Study
• Hiring new people – QA Engineers, Test Automation D...
Here is a sample presentation …
Quality is Free!
If Developers created
perfect code then you
wouldn’t need any
testing…
What costs is the
incorrect
implementation of a
software requirement…
QA is there to make developers and
development better.
We don’t ensure better software – we
enable developers to build bet...
Is there a role for QA
in an Agile World?
Yes!!!
TESTING PYRAMID
Traditional Goal
End to End GUI Tests
Functional Tests
Integration
Tests
Unit Tests
GUI Tests
Functional
T...
What is the difference between QA and QC?
Quality Assurance – Is focused on the development, implementation and
measuremen...
So where does QA fit
in this new way of
thinking?
QA people should do
more QA and less QC!
QA Coaching Role
QA can play a vital role coaching the
Development Team on how to test…
…so that Developers can create bet...
Automated Tests
QA can play a role in real automation…
Not Record and Playback…
But Manual Testing is still important…
Automated Tests are great but
look around, it is still humans
using the software…
E...
Let’s not forget System Integration Testing
Most systems today integrate
with many other systems
SIT is essential…
and QA ...
User Community of Testers
If your systems and applications allow it , then why not create a
User Community of real testers...
There is plenty of
testing to go around!
QA Engineers at the Core of QA Team Agile Testing
Key Takeaways
New QA Mindset Use of Automation must be
at the Core of your QA
Key Takeaways
Thank you!
MAURIZIO MANCINI • DIRECTOR OF QA • YELLOW PAGES CANADA • @QAANDPROCESSGUY
GIANCARLO BISCEGLIA • SENIOR QA MANA...
Building QA Team that matters for an Agile World
Building QA Team that matters for an Agile World
Building QA Team that matters for an Agile World
Building QA Team that matters for an Agile World
Building QA Team that matters for an Agile World
Building QA Team that matters for an Agile World
Building QA Team that matters for an Agile World
Building QA Team that matters for an Agile World
Building QA Team that matters for an Agile World
Building QA Team that matters for an Agile World
Building QA Team that matters for an Agile World
Building QA Team that matters for an Agile World
Upcoming SlideShare
Loading in …5
×

Building QA Team that matters for an Agile World

1,579 views

Published on

Presentation from Quest 2015 - Covers building a new QA Team that matters, how to approach Agile Testing, and how to present the message to renovate your existing QA team for Agile.

Published in: Software
  • Earn $500 for taking a 1 hour paid survey! read more... ➤➤ https://tinyurl.com/realmoneystreams2019
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • You can now be your own boss and get yourself a very generous daily income. START FREE...●●● https://tinyurl.com/make2793amonth
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Building QA Team that matters for an Agile World

  1. 1. MAURIZIO MANCINI • DIRECTOR OF QA • YELLOW PAGES CANADA • @QAANDPROCESSGUY How to build a QA Team that matters in an Agile or Incremental and Iterative (IID) world. GIANCARLO BISCEGLIA • SENIOR QA MANAGER • YELLOW PAGES CANADA
  2. 2. Who are we? • Head of QA for Yellow Pages Canada (YP) • Also responsible for Software Process Deployment at YP • 25 years of software engineering experience Maurizio Mancini Giancarlo Bisceglia • Senior QA Manager for Yellow Pages Canada (YP) • 15 years of software engineering experience
  3. 3. About Our Team • YP has ~ 2300 employees and ~ 1 Billion in Revenue • Our team of 100 QA are embedded into 15 development teams • Our team is geographically disbursed in Montreal (4 locations) and in Toronto (2 locations) • Our team works on projects using all types of software development process (Agile, Incremental & Iterative (IID), Waterfall)
  4. 4. Agenda • Yellow Pages Yesterday and Today • QA “State of the Nation” from our perspective • How to hire and build a QA Team that will succeed in any SDLC • Agile Testing – What’s that about? • Test Automation – Fundamental Building Block • The Testing Pyramid – Where does QA fit in? • Different Layers of Testing • Where else does QA fit in an Agile or IID World?
  5. 5. Yellow Pages Canada • 3 years ago no real QA Team existed • A few scattered QA souls left to fend for themselves • A lot of consultants • Almost all testing was manual, very little automation • A lot of “End to End” testing • Most QA was done by BA’s and The Business Yesterday Today • Team of 100 QA Engineers, Test Automation Developers, and QA Analysts • A full QA practice that uses common tools and QA approaches • On average our testing is 50-50 Automated/Manual • QA Teams focus on System Testing and BA/Product Owners focus on UAT • QA teams are Embedded with the Dev Teams
  6. 6. What do our Teams Test?
  7. 7. Releases in 2014
  8. 8. QA “State of the Nation” • Over the last 30 to 40 years, the QA software industry has gone from being a “nice to have” “a must” “developers will do it all” in the Agile world
  9. 9. QA “State of the Nation” Like all extremes, we believe that “pendulum of sensibility” is starting to settle in the middle
  10. 10. What is the role of QA in today’s Agile or IID world?
  11. 11. The Death of QA… One of the heated debates in the industry is that QA is dead with Agile becoming the predominant software development process… We like to say that Agile has finally brought balance to the QA world…
  12. 12. Agile has brought balance to the QA world More developers than testers, too much testing, not enough time for QA to test everything… Agile has helped rebalance the inequities that existed when it came to testing software….
  13. 13. Agile has also helped reinforce that you cannot just throw the code over the fence!!
  14. 14. Agile has brought balance to the QA world • Agile has helped refocus development teams on the importance of testing. • Agile has re-asserted what Software Engineering 101 has said all along… Unit Testing is a MUST!
  15. 15. TESTING PYRAMID Traditional Goal End to End GUI Tests Functional Tests Integration Tests Unit Tests GUI Tests Functional Tests Integration Tests Unit Tests Q A D E V
  16. 16. UNIT Testing is not the
  17. 17. Donald Knuth He once warned in a letter to one of his fans, "Beware of bugs in the above code; I have only proved it correct, not tried it." Knuth has been called the "father of the analysis of algorithms" ‘To tweak what Knuth said, "Beware of the above code. I have only tested that it works. I haven't actually tried it.“’ http://jjinux.blogspot.ca/2012/05/unit-tests-dont-find-bugs-death-of-qa.html Shannon Behrens
  18. 18. UNIT Testing is great but you need other types of testing!
  19. 19. Where do we go from here?
  20. 20. Let’s build that QA Team!
  21. 21. Hiring Which Roles should you be hiring for the current world of Software Development that includes Agile, IID and sometimes waterfall? • QA Engineer • Also known as QA Specialist, QA Developer, Engineer in Test • Test Automation Engineer • QA Analyst
  22. 22. Hiring What should the mix be? • QA Engineer (75%) • Also known as QA Specialist, QA Developer, Engineer in Test • Test Automation Engineer (10%) • QA Analyst (15%)
  23. 23. Candidate Profiles Early Career – Look for people that think they want to be a developer. Mid Career – Look for someone that may have coded before and has discovered the love of testing. Late Career – Look for someone with a passion for testing and breaking software.
  24. 24. QA Engineer
  25. 25. QA Engineer Also known as a QA Developer, QA Specialist, or Engineer in Test Key skills and qualities you should be looking for:  Coding and/or scripting – Experience in Record and Playback  not good enough  API and Web services background – ex: REST and SOAP  SQL and database background  Networking background – Should have the basic knowledge of networking  Linux/UNIX and/or Server background  Passion for testing  Solid understanding of QA and how it differs from Dev  Understands that there is more to testing than “executing a test”  Someone that likes to get under the hood  QA Mechanic
  26. 26. Test Automation Developer
  27. 27. Test Automation Developer This will be the person that will put in place your test automation framework • Helps to implement test automation best practices for QA Engineers • Will support the team as they use the framework in their project • Helps conduct QA Teams code reviews
  28. 28. Test Automation Developer Key skills and qualities you should be looking for:  This is a pure developer that also has a passion for testing  Object Oriented coding skills  Scripting language skills  Deep understanding of QA Automation Tools  API and Web services background  SQL and database background  Networking background  Linux/UNIX and/or Server background  Solid understanding of QA and how it differs from Dev  Understands that there is more to testing than “executing a test”  Someone that likes to get under the hood  QA Mechanic
  29. 29. QA Analyst
  30. 30. QA Analyst This is your manual tester • This is the person that will also drive Exploratory Testing • This resource can play other roles – More on this later
  31. 31. QA Analyst  API and Web services background  SQL and database background  Networking background  Solid understanding of QA and how it differs from Dev  Understands that there is more to testing than “executing a test”  Has a natural tendency to want to understand how something works Key skills you should be looking for:  Passion for testing  Not just a GUI tester, someone that likes to get under the hood  QA Mechanic  Drives Technology in his/her home (Hacks home technology) Examples: • Uses Opensource firmware on their home router • Pushes usage of his/her smart phone beyond everyday usage • Able to clean a computer, infected with a virus, without the help of a tool
  32. 32. Hiring Team
  33. 33. Hiring Team • You should form a Hiring Team • Formulate a hiring plan – Who you are looking for, what are the criteria • Each member of the team focuses on a set of skills to look for • Have a battery of questions that the team uses. For example:  Linux/UNIX  SQL  Networking  Database  Coding  Actually test something live with bugs • This will ensure that you are hiring consistently and that you build that awesome QA Team
  34. 34. The Interview • QA Engineers and Test Automation Developers should be interviewed by a Developer too • Questions must include coding questions • Your questions must include “white boarding” • Include multi-dimensional questions • Multi-System • Many Technologies • System Architecture Questions – Are they able to visualize it? • Goal is to reveal if the candidate has a QA Mechanic mentality • Do they have a QA mindset? More on this later. • Do they have a desire to break software? • Are they technically sound in the technical skills that you are looking for? • Do they have the passion for QA?
  35. 35. QA Mechanic
  36. 36. Team Structure
  37. 37. Embed QA with the Dev teams!!
  38. 38. Central QA but with an Embedded Focus • Common set of Automation Tools • Shared Automation Framework • Common standards • Common Hiring practices • Able to shift a QA resource when that resource wants to advance their career • Helps to avoid unproductive Agile Teams Benefits of a Central QA Team Benefits of an Embedded QA Team • QA involved in the daily team interactions • QA testing early and often • QA helps to keep a process focus within the team • Get the benefits of a true agile team
  39. 39. How do we approach Testing?
  40. 40. Agile Testing
  41. 41. Agile Testing
  42. 42. TESTING PYRAMID Traditional Goal End to End GUI Tests Functional Tests Integration Tests Unit Tests GUI Tests Functional Tests Integration Tests Unit Tests Q A D E V
  43. 43. UNIT Testing is great but you need other types of testing! • Integration Testing • Stubbing external dependencies to facilitate UI automation • Exploratory Testing • Performance Testing • UAT using BDD’s
  44. 44. Unit and Integration Stubbed Functional Testing System Integration Performance UAT Owner Dev QA Engineers QA Engineers QA Analysts QA Engineers Product Owner QA Analyst Type White Box Black Box Black Box Black Box Black Box Environment None Local Deployment or CI Server “Prod-like” QA Environment Performance Environment Staging or Training Environment Complexity Focus Low Level Correctness Maintainable Negative inputs Mid Level Extreme cases Negative tests UI Permutations Failure Tests Mid Level Positive Tests Exploratory tests System Integration High Level Scalability Robustness Reliability High Level Business Flows Layers of Testing
  45. 45. Test Automation
  46. 46. What tools we chose
  47. 47. • Many paid tools are based on open source libraries nowadays. • Don’t think that because you pay for a tool, it’s best of breed. • Beware of Scriptless tools. You still need people that can code reusable classes. Yes you get a library but how complete is it? • Record and playback just encourages end to end testing. • Automation project needs to be run like a dev project, including code reviews • Invest in a custom framework which uses libraries or frameworks that can be swapped out. Don’t tie yourself to one technology
  48. 48. Leverage Continuous Integration
  49. 49. Ideal for helping Product Owners and QA Analysts define business requirements in an English like syntax that can be automated. Limitations of BDD’s • Good to use for top of pyramid tests, flows that don’t change often • Not a replacement for unit testing or automated functional tests • Best if it’s a layer you add on top of your existing automation framework • Syntactically discourages code reuse • Test Steps are expensive to maintain BDD Usage in Automation Behavior – Driven Development
  50. 50. • Time for Automation is considered expendable • Environment Deployment Automation • Coding Standards for the software being developed Test Automation Challenges
  51. 51. A Different Way of Thinking
  52. 52. DON’T IGNORE WHAT’S IN BETWEEN TEST WEB UI, MOBILE APPS, API, DATABASE, REPORTS INDEPENDENTLY
  53. 53. Break up the Test Effort
  54. 54. • Most expensive testing you can do • Too late in the Cycle • Hard to coordinate and manage • Don’t make this your primary method of testing • Use it as a last resort End to End Testing
  55. 55. • No more excuses, QA unblocks itself • Think outside the box, to find ways to test partial implementations (UI not implemented) • Test system components in isolation using stubs • Test Early, Test Often • Push for continuous integration • Deploy software and maintain QA environments • Work with DevOps on deployment Automation The New QA Mindset
  56. 56. Renovating an existing QA Team for an Agile or IID world!
  57. 57. Skills Assessment
  58. 58. Skills Assessment What you need to improve NOW What you have today What you need tomorrow
  59. 59. Now What?
  60. 60. Now What? • Train Existing Staff – On the Job, Classroom, Self Study • Hiring new people – QA Engineers, Test Automation Developers • New Approaches to Testing – Testing Pyramid • Embed your team(s) if you haven’t already • Change in Roles and New Roles How do you present your new approach to your existing staff?
  61. 61. Here is a sample presentation …
  62. 62. Quality is Free!
  63. 63. If Developers created perfect code then you wouldn’t need any testing…
  64. 64. What costs is the incorrect implementation of a software requirement…
  65. 65. QA is there to make developers and development better. We don’t ensure better software – we enable developers to build better software. James Whittaker
  66. 66. Is there a role for QA in an Agile World?
  67. 67. Yes!!!
  68. 68. TESTING PYRAMID Traditional Goal End to End GUI Tests Functional Tests Integration Tests Unit Tests GUI Tests Functional Tests Integration Tests Unit Tests Q A D E V
  69. 69. What is the difference between QA and QC? Quality Assurance – Is focused on the development, implementation and measurement of software processes in an organization. Focused on Defect Prevention Quality Control – Is focused on the comparison of a product against the requirements and ensuring that the requirements have been met. Reactive approach focused on defect detection and removal L E S S M O R E
  70. 70. So where does QA fit in this new way of thinking?
  71. 71. QA people should do more QA and less QC!
  72. 72. QA Coaching Role QA can play a vital role coaching the Development Team on how to test… …so that Developers can create better Unit and Integration tests.
  73. 73. Automated Tests QA can play a role in real automation… Not Record and Playback…
  74. 74. But Manual Testing is still important… Automated Tests are great but look around, it is still humans using the software… Exploratory Testing is essential… and QA is key in this role!
  75. 75. Let’s not forget System Integration Testing Most systems today integrate with many other systems SIT is essential… and QA is key in this role!
  76. 76. User Community of Testers If your systems and applications allow it , then why not create a User Community of real testers? Have QA manage your user community of testers! Employees Early Adopters
  77. 77. There is plenty of testing to go around!
  78. 78. QA Engineers at the Core of QA Team Agile Testing Key Takeaways
  79. 79. New QA Mindset Use of Automation must be at the Core of your QA Key Takeaways
  80. 80. Thank you! MAURIZIO MANCINI • DIRECTOR OF QA • YELLOW PAGES CANADA • @QAANDPROCESSGUY GIANCARLO BISCEGLIA • SENIOR QA MANAGER • YELLOW PAGES CANADA

×