SlideShare a Scribd company logo
Performance and Load
Testing Process.
How to plan it
www.qaexperts.pro
22
12+ years general working experience in IT/QA (40+ projects)
----------
•QA Consultant / QA Manager at Ciklum Interactive
•Load Testing Manager (3+ years, 15+ projects)
•Automated Testing Manager
•Co-owner at www.qaexperts.pro
----------
Skype: vladimir.primakov
Linkedin: ua.linkedin.com/in/vladimirprimakov/
Email: v.v.primakov@gmail.com
Some Words About Me
Volodymyr Prymakov (Vladimir Primakov)
www.qaexperts.pro
33
• Introduction - 5 min
• Main Part - 30 min
• Questions - 5 min
Presentation’s Structure
Presentation Plan
www.qaexperts.pro
44
Assumptions
We assume that:
• Load & Performance testing tools are already chosen and used.
• Load testing framework and infrastructure is build and fully functional.
• Load testing team is already formed
www.qaexperts.pro
55
Load Testing Process
1. Planning
2. Evaluating Technical Risks
3. Prerequisites solving
4. Script implementation & adjustment
5. Test execution, analysis, and reporting
6. Summary reporting and conclusions
www.qaexperts.pro
66
1. PLANNING
www.qaexperts.pro
77
PRODUCT AND SCOPE
www.qaexperts.pro
88
Product Idea:
Business domain, main specialties
PRODUCT AND SCOPE
Solution Architecture.
The underlying technology.
Pointer to Weak areas.
www.qaexperts.pro
99
Product and product parts
IN and OUT of scope.
Priority
PRODUCT AND SCOPE
3D-Party services
IN and OUT of scope.
Mockups. Impact?
www.qaexperts.pro
1010
PRODUCT AND SCOPE
www.qaexperts.pro
60%
30%
2%8%
CLIENT PLATFORMS?
1111 www.qaexperts.pro
PRODUCT AND SCOPE
COMMUNICATION TECHNOLOGY
• Communication model and implementation for
a) client <-> server side
b) server-side <-> server-side
• Protocols/exchange formats used in the communication
• If there is any repetitive calls available in the communication?
HTTP(S)
(85%)
Web-sockets
(10%)
TCP/IP
(5%)
TLS
HTML
Strings
Binary
XML
JSON
1212
PRODUCT AND SCOPE
USAGE FLOWS / USER ACTIONS
www.qaexperts.pro
1313
PRODUCT AND SCOPE
USAGE FLOWS / USER ACTIONS
www.qaexperts.pro
•Types of flows
• Logged in and anonymous
• Main Operations
• Navigational
•Approach (Stability and Traceability):
•Try to isolate testable operation in a single flow
•Make flows as shorter as you can
•Avoid dependencies between flows
1414
PRODUCT AND SCOPE
USAGE FLOWS / USER ACTIONS
www.qaexperts.pro
Flow Type Flow %
Anonymous
Registration 3%
Main Pages Navigation 10%
Categories Navigation 15%
Search and Navigate Item 20%
Search and Buy item 15%
Search and Add to Cart / Remove from
cart 15%
Compare Items 5%
Logged in
Add comment to / Ask question about
an item 5%
Add item to my wish list 5,00%
Navigate to wish list 5,00%
Edit Profile 2%
Etc… …
SUM 100,00%
Registration Flow
1. Open main page
2. Go to the registration page
3. Fill in all the required data and
click Register
4. Get the corresponding
registration activation email -
follow the registration activation
link and login
Example:
1515
TEST ENVIRONMENTS
www.qaexperts.pro
Test environments for:
1. Evaluating the product / Technical Risk
2. Script implementation
3. Load test execution
Difference between environment for test
execution and target production environment?
The corresponding risks.
Possibility to minimize the difference.
1616
LOAD PROFILES
TYPES OF LOAD TESTS
www.qaexperts.pro
•10-100% Usual Load
•Stress Load
•Capacity Testing
•Spike Stress Load
•Soak Testing (24h-7d)
•Volume Testing
Load profiles Types of load tests
1717
EXPECTED LOAD
PASS-FAIL CRITERIA
www.qaexperts.pro
•Client-side criteria
•Server-side criteria
•Test Data
1818
EXPECTED LOAD
PASS-FAIL CRITERIA
www.qaexperts.pro
Client
Which is actually:
a) Number of registered users or
b) Number of users per day or
d) Correct number of concurrent users 
Response time for a operation or a request?
Number of concurrent users
=100000
Response time: 2 Sec
1919
EXPECTED LOAD
PASS-FAIL CRITERIA
www.qaexperts.pro
CLIENT-SIDE CRITERIA
•Max number of concurrent users (anonymous and logged-in, active
and passive)
•% of Mobile/web/desktop/terminal clients
•Max rate of main actions/operations per min, e.g.:
100 logins/min, 10 registration/min
•Max response times for key pages and operations, e.g.:
login – 3 sec, registration – 5 sec
•Acceptable % of fails, e.g.: 2%
Availability of such statistics from web-logs, Google analytics, and other tools/places?
2020
EXPECTED LOAD
PASS-FAIL CRITERIA
www.qaexperts.pro
SERVER-SIDE CRITERIA
•Max Resource Consumption:
CPU (e.g. 80%), RAM (e.g. 80%), Disk in/out, Network, etc.
These parameters should be monitored within every node of a certain
component
•Max Errors Rate
(in logs, etc.)
e.g. 0.5%
2121
EXPECTED LOAD
PASS-FAIL CRITERIA
www.qaexperts.pro
TEST DATA
•Users
•User’s Data
•Product’s Data
---------------------------
1. Type of Data,
2. Size and Amount
of Data
2222
OTHER NUANCES
www.qaexperts.pro
•Scheduling services?
•Client’s internet
connection speed?
•Static content hosting:
CDN or own servers?
•Target region (USA,
Europe, etc.)?
2323
ORGANIZATIONAL MOMENTS
www.qaexperts.pro
•Project Milestones?
•Date and Time frames for
load testing runs?
•Stakeholders?
•Organizational, Business,
Technical, and Server-side
support contacts
2424
• Difference between Target
and Test environments
• Risk of using service-
mockups
• Risk of negatively impacting
the production environment
or 3d party services
• Wrong target load
expectations
• …
www.qaexperts.pro
TYPICAL RISKS
2525
• Product & Solution
successfully passed UAT
• All prerequisites are solved
• No unexpected deploys
• …
www.qaexperts.pro
DEPENDENCIES / ASSUMPTIONS
2626
2. EVALUATING
TECHNICAL
RISKS
www.qaexperts.pro
2727
3.
PREREQUISITES
SOLVING
www.qaexperts.pro
2828 www.qaexperts.pro
PREREQUISITES SOLVING
PLANNING IS DONE!
Non-functional requirements
and user flows are defined.
2929 www.qaexperts.pro
PREREQUISITES SOLVING
PREPARE TEST ENVIRONMENT!
• Test environments availability
and access. Whitelisting load
injectors.
• Test environment and product
stability.
• Sever-side monitoring tools.
• Test environment alignment to
the production environment.
• Test users and test data.
3030
4.
SCRIPT
IMPLEMENTATION
www.qaexperts.pro
3131 www.qaexperts.pro
SCRIPT IMPLEMENTATION
1. Type of scripts:
• Logged-in/Anonymous
• Main flows scripts,
• Navigational Scripts,
• Repetitive calls scripts
2. Adjust scripts to the target
environments
3. Regularly Update scripts to
product changes
4. Define and follow script
readiness criteria
3232
5.
TEST EXECUTION,
ANALYSIS, AND
REPORTING
www.qaexperts.pro
3333
www.qaexperts.pro
PLAN TEST EXECUTION
a) Plan load gradually (to eliminate Risks):
10%
25%
50%
75%
100%
b) Or start load from Capacity test with long ramp-up
period (to get fast results)
Define load schedule: Types of load tests, dates, and
timeframes for test runs
3434 www.qaexperts.pro
• Type of test/Goal
• Date & Time of test
• Server sider conditions and configuration; product
version.
• % of load
• Ramp-up, Sustain, Ramp-down
• Scripts (Flows) to run
• Number of concurrent users in general and per script
• Expected operations rate
• Responsible specialists and their contacts in case of
emergency.
PLAN A RUN
PLAN TEST EXECUTION
3535 www.qaexperts.pro
TEST EXECUTION
• Make sure the server side support’s specialists are available
• Do smoke test
• Run Test:
• Monitor Test product health:
• Client
• Response time growth
• Error rate growth
• Error types
• Server
• Resource consumptions
• Errors
• Servers availability
• Communicate the results in real time
• Be ready to Stop Test in case of Emergency!
• Monitor Load Injectors health
• If possible, play manually with the product under load
3636 www.qaexperts.pro
TEST RESULTS ANALYSIS
AND REPORTING
Client Side:
• Response time-based capacity point
• Errors-based capacity point
• Failed scripts and transactions
• Slow scripts and slow transactions
• Errors details: types and amount
• Max transactions/operations rates reached.
• Timely trends and correlation with concurrent
users number
• Main bottlenecks concussions
Server-side:
• Resources consumptions
• Errors and log-messages
• Slow DB transactions
• Correlation with client side metrics
3737 www.qaexperts.pro
Execute
Load Test
Meet
Pass
Criteria?
Do
Profiling
Load Test
Fix
Solution/
Product
TEST EXECUTION PROCESS
Ready For
Release
YES
NO
If possible, the final step of load testing is testing on the prod environment under
conditions similar to the real ones.
Update
Scripts
Analyze
Results
3838
6.
SUMMARY
REPORTING AND
CONCLUSIONS
www.qaexperts.pro
3939
www.qaexperts.pro
SUMMARY REPORTING
AND CONCLUSIONS
1. Compare test runs: types of tests,
server-side conditions, client and
servers-side metrics
2. Conclusion about readiness to release.
3. Recommendations on further steps:
a. Next load tests required.
b. How to survive the expected load in
case of not meeting the PASS criteria.
4141
GENERAL SCHEDULE & ESTIMATION
Weeks 1 2 3 4 5 6 7 8 9
Planning
Evaluating Technical Risks
Prerequisites solving
Script implementation &
adjustment
Test execution, analysis, and
reporting
Summary reporting and
conclusions
80%
50%
EFFORTS AND DURATION
www.qaexperts.pro
4242
LOAD TESTING IN SDLS
• In ongoing development – Verifying and validating
component, queue of components, and integration
related performance & robustness.
• Before Release – Verifying and validating the whole
product performance & robustness before release.
• Maintenance – Verify and validating architectural,
configurational, capacity-related, db-related, and
integration-related changes.
WHEN TO PERFORM
www.qaexperts.pro
4343
Thank you! ;)
QUESTIONS?

More Related Content

What's hot

QA Club Kiev #18 - Test Management in Google Sheets
QA Club Kiev #18 - Test Management in Google SheetsQA Club Kiev #18 - Test Management in Google Sheets
QA Club Kiev #18 - Test Management in Google Sheets
QA Club Kiev
 
Performance Testing
Performance TestingPerformance Testing
Performance Testing
QA Club Kiev
 
QA Club Kiev #17 Measuring quality by Volodymyr Prymakov
QA Club Kiev #17 Measuring quality by Volodymyr PrymakovQA Club Kiev #17 Measuring quality by Volodymyr Prymakov
QA Club Kiev #17 Measuring quality by Volodymyr Prymakov
QA Club Kiev
 
QA Process Overview
QA Process OverviewQA Process Overview
QA Process Overview
Deepak Rathod
 
РАМЕЛЛА БАСЕНКО ISTQB® Streams “Core&Agile”
РАМЕЛЛА БАСЕНКО ISTQB® Streams “Core&Agile”РАМЕЛЛА БАСЕНКО ISTQB® Streams “Core&Agile”
РАМЕЛЛА БАСЕНКО ISTQB® Streams “Core&Agile”
QADay
 
Eclipse Day India 2015 - Eclipse RCP testing using Jubula based automation
Eclipse Day India 2015 - Eclipse RCP testing using Jubula based automationEclipse Day India 2015 - Eclipse RCP testing using Jubula based automation
Eclipse Day India 2015 - Eclipse RCP testing using Jubula based automation
Eclipse Day India
 
Navyug Process Flow QA
Navyug Process Flow QANavyug Process Flow QA
Navyug Process Flow QA
Sadhvi Singh
 
Test link
Test linkTest link
Test link
DialogWebdesign
 
QA Process Overview for Firefox OS 2014
QA Process Overview for Firefox OS 2014QA Process Overview for Firefox OS 2014
QA Process Overview for Firefox OS 2014
Anthony Chung
 
Defect root cause analysis, Андрей Титаренко
Defect root cause analysis, Андрей ТитаренкоDefect root cause analysis, Андрей Титаренко
Defect root cause analysis, Андрей Титаренко
Sigma Software
 
How to organize qa process in agile speed
How to organize qa process in agile speedHow to organize qa process in agile speed
How to organize qa process in agile speed
Svitlana Dubyk
 
ReportPortal.io - Open Source experience. Showcase, benefits
ReportPortal.io - Open Source experience. Showcase, benefits ReportPortal.io - Open Source experience. Showcase, benefits
ReportPortal.io - Open Source experience. Showcase, benefits
COMAQA.BY
 
Qa process 2012
Qa process 2012Qa process 2012
Qa process 2012
ashokack
 
Meetup TestingUy 2017: Automating the Viewer: a cross-functional team approach
Meetup TestingUy 2017: Automating the Viewer: a cross-functional team approachMeetup TestingUy 2017: Automating the Viewer: a cross-functional team approach
Meetup TestingUy 2017: Automating the Viewer: a cross-functional team approach
Claudia Badell
 
Roman iovlev battle - JDI vs Selenide - Selenium Camp
Roman iovlev battle - JDI vs Selenide - Selenium CampRoman iovlev battle - JDI vs Selenide - Selenium Camp
Roman iovlev battle - JDI vs Selenide - Selenium Camp
Роман Иовлев
 
Meetup TestingUY 2016: Adopting testing in a cross-functional team
Meetup TestingUY 2016: Adopting testing in a cross-functional teamMeetup TestingUY 2016: Adopting testing in a cross-functional team
Meetup TestingUY 2016: Adopting testing in a cross-functional team
Claudia Badell
 
Quality Management Introduction
Quality Management IntroductionQuality Management Introduction
Quality Management Introduction
Duy Tan Geek
 
product Qa workflow
product Qa workflowproduct Qa workflow
product Qa workflow
tanvir afzal
 
ProductSavvy - Scrum and QA
ProductSavvy - Scrum and QAProductSavvy - Scrum and QA
ProductSavvy - Scrum and QA
Jonathan Chashper
 
QA and scrum
QA and scrumQA and scrum
QA and scrum
clintonbosch
 

What's hot (20)

QA Club Kiev #18 - Test Management in Google Sheets
QA Club Kiev #18 - Test Management in Google SheetsQA Club Kiev #18 - Test Management in Google Sheets
QA Club Kiev #18 - Test Management in Google Sheets
 
Performance Testing
Performance TestingPerformance Testing
Performance Testing
 
QA Club Kiev #17 Measuring quality by Volodymyr Prymakov
QA Club Kiev #17 Measuring quality by Volodymyr PrymakovQA Club Kiev #17 Measuring quality by Volodymyr Prymakov
QA Club Kiev #17 Measuring quality by Volodymyr Prymakov
 
QA Process Overview
QA Process OverviewQA Process Overview
QA Process Overview
 
РАМЕЛЛА БАСЕНКО ISTQB® Streams “Core&Agile”
РАМЕЛЛА БАСЕНКО ISTQB® Streams “Core&Agile”РАМЕЛЛА БАСЕНКО ISTQB® Streams “Core&Agile”
РАМЕЛЛА БАСЕНКО ISTQB® Streams “Core&Agile”
 
Eclipse Day India 2015 - Eclipse RCP testing using Jubula based automation
Eclipse Day India 2015 - Eclipse RCP testing using Jubula based automationEclipse Day India 2015 - Eclipse RCP testing using Jubula based automation
Eclipse Day India 2015 - Eclipse RCP testing using Jubula based automation
 
Navyug Process Flow QA
Navyug Process Flow QANavyug Process Flow QA
Navyug Process Flow QA
 
Test link
Test linkTest link
Test link
 
QA Process Overview for Firefox OS 2014
QA Process Overview for Firefox OS 2014QA Process Overview for Firefox OS 2014
QA Process Overview for Firefox OS 2014
 
Defect root cause analysis, Андрей Титаренко
Defect root cause analysis, Андрей ТитаренкоDefect root cause analysis, Андрей Титаренко
Defect root cause analysis, Андрей Титаренко
 
How to organize qa process in agile speed
How to organize qa process in agile speedHow to organize qa process in agile speed
How to organize qa process in agile speed
 
ReportPortal.io - Open Source experience. Showcase, benefits
ReportPortal.io - Open Source experience. Showcase, benefits ReportPortal.io - Open Source experience. Showcase, benefits
ReportPortal.io - Open Source experience. Showcase, benefits
 
Qa process 2012
Qa process 2012Qa process 2012
Qa process 2012
 
Meetup TestingUy 2017: Automating the Viewer: a cross-functional team approach
Meetup TestingUy 2017: Automating the Viewer: a cross-functional team approachMeetup TestingUy 2017: Automating the Viewer: a cross-functional team approach
Meetup TestingUy 2017: Automating the Viewer: a cross-functional team approach
 
Roman iovlev battle - JDI vs Selenide - Selenium Camp
Roman iovlev battle - JDI vs Selenide - Selenium CampRoman iovlev battle - JDI vs Selenide - Selenium Camp
Roman iovlev battle - JDI vs Selenide - Selenium Camp
 
Meetup TestingUY 2016: Adopting testing in a cross-functional team
Meetup TestingUY 2016: Adopting testing in a cross-functional teamMeetup TestingUY 2016: Adopting testing in a cross-functional team
Meetup TestingUY 2016: Adopting testing in a cross-functional team
 
Quality Management Introduction
Quality Management IntroductionQuality Management Introduction
Quality Management Introduction
 
product Qa workflow
product Qa workflowproduct Qa workflow
product Qa workflow
 
ProductSavvy - Scrum and QA
ProductSavvy - Scrum and QAProductSavvy - Scrum and QA
ProductSavvy - Scrum and QA
 
QA and scrum
QA and scrumQA and scrum
QA and scrum
 

Similar to QA Fest 2015. Владимир Примаков. Процесс нагрузочного тестирования и его планирование

JMeter
JMeterJMeter
Load testing with Visual Studio and Azure - Andrew Siemer
Load testing with Visual Studio and Azure - Andrew SiemerLoad testing with Visual Studio and Azure - Andrew Siemer
Load testing with Visual Studio and Azure - Andrew Siemer
Andrew Siemer
 
Software Project Management lecture 10
Software Project Management lecture 10Software Project Management lecture 10
Software Project Management lecture 10
Syed Muhammad Hammad
 
Holiday Readiness: Best Practices for Successful Holiday Readiness Testing
Holiday Readiness: Best Practices for Successful Holiday Readiness TestingHoliday Readiness: Best Practices for Successful Holiday Readiness Testing
Holiday Readiness: Best Practices for Successful Holiday Readiness Testing
Apica
 
A Three-Tier Load Testing Program Saved Our Bacon
A Three-Tier Load Testing Program Saved Our BaconA Three-Tier Load Testing Program Saved Our Bacon
A Three-Tier Load Testing Program Saved Our Bacon
TechWell
 
QA metrics in Agile (GUIDE)
QA metrics in Agile (GUIDE)QA metrics in Agile (GUIDE)
QA metrics in Agile (GUIDE)
Vladimir Primakov (Volodymyr Prymakov)
 
Amita_Kashyap1_CV
Amita_Kashyap1_CVAmita_Kashyap1_CV
Amita_Kashyap1_CV
Amita Kashyap
 
Aakash shah performance tester
Aakash shah  performance testerAakash shah  performance tester
Aakash shah performance tester
anandkayalmatrix
 
Neev Load Testing Services
Neev Load Testing ServicesNeev Load Testing Services
Neev Load Testing Services
Neev Technologies
 
Spm lecture-3
Spm lecture-3Spm lecture-3
Spm lecture-3
Sulman Ahmed
 
Lect3
Lect3Lect3
reddythippa ETL 8Years
reddythippa ETL 8Yearsreddythippa ETL 8Years
reddythippa ETL 8Years
Thippa Reddy
 
Webinar: Automated Load Testing Results Analysis
Webinar:  Automated Load Testing Results AnalysisWebinar:  Automated Load Testing Results Analysis
Webinar: Automated Load Testing Results Analysis
John Emmitt
 
Priyaranjan Testing CV
Priyaranjan Testing CVPriyaranjan Testing CV
Priyaranjan Testing CV
Priyaranjan Dash
 
6 Steps to Performance Testing like a Pro
6 Steps to Performance Testing like a Pro6 Steps to Performance Testing like a Pro
6 Steps to Performance Testing like a Pro
LogiGear Corporation
 
Introduction to Software Testing Part 2
Introduction to Software Testing Part 2Introduction to Software Testing Part 2
Introduction to Software Testing Part 2
Murageppa-QA
 
#NEOTYSPAC performance testing shift left
#NEOTYSPAC performance testing shift left#NEOTYSPAC performance testing shift left
#NEOTYSPAC performance testing shift left
Amir Rozenberg
 
Types of Testing
Types of TestingTypes of Testing
Types of Testing
Murageppa-QA
 
Small is Beautiful- Fully Automate your Test Case Design
Small is Beautiful- Fully Automate your Test Case DesignSmall is Beautiful- Fully Automate your Test Case Design
Small is Beautiful- Fully Automate your Test Case Design
Georgina Tilby
 
Technical Without Code
Technical Without CodeTechnical Without Code
Technical Without Code
Caitlin Cassidy
 

Similar to QA Fest 2015. Владимир Примаков. Процесс нагрузочного тестирования и его планирование (20)

JMeter
JMeterJMeter
JMeter
 
Load testing with Visual Studio and Azure - Andrew Siemer
Load testing with Visual Studio and Azure - Andrew SiemerLoad testing with Visual Studio and Azure - Andrew Siemer
Load testing with Visual Studio and Azure - Andrew Siemer
 
Software Project Management lecture 10
Software Project Management lecture 10Software Project Management lecture 10
Software Project Management lecture 10
 
Holiday Readiness: Best Practices for Successful Holiday Readiness Testing
Holiday Readiness: Best Practices for Successful Holiday Readiness TestingHoliday Readiness: Best Practices for Successful Holiday Readiness Testing
Holiday Readiness: Best Practices for Successful Holiday Readiness Testing
 
A Three-Tier Load Testing Program Saved Our Bacon
A Three-Tier Load Testing Program Saved Our BaconA Three-Tier Load Testing Program Saved Our Bacon
A Three-Tier Load Testing Program Saved Our Bacon
 
QA metrics in Agile (GUIDE)
QA metrics in Agile (GUIDE)QA metrics in Agile (GUIDE)
QA metrics in Agile (GUIDE)
 
Amita_Kashyap1_CV
Amita_Kashyap1_CVAmita_Kashyap1_CV
Amita_Kashyap1_CV
 
Aakash shah performance tester
Aakash shah  performance testerAakash shah  performance tester
Aakash shah performance tester
 
Neev Load Testing Services
Neev Load Testing ServicesNeev Load Testing Services
Neev Load Testing Services
 
Spm lecture-3
Spm lecture-3Spm lecture-3
Spm lecture-3
 
Lect3
Lect3Lect3
Lect3
 
reddythippa ETL 8Years
reddythippa ETL 8Yearsreddythippa ETL 8Years
reddythippa ETL 8Years
 
Webinar: Automated Load Testing Results Analysis
Webinar:  Automated Load Testing Results AnalysisWebinar:  Automated Load Testing Results Analysis
Webinar: Automated Load Testing Results Analysis
 
Priyaranjan Testing CV
Priyaranjan Testing CVPriyaranjan Testing CV
Priyaranjan Testing CV
 
6 Steps to Performance Testing like a Pro
6 Steps to Performance Testing like a Pro6 Steps to Performance Testing like a Pro
6 Steps to Performance Testing like a Pro
 
Introduction to Software Testing Part 2
Introduction to Software Testing Part 2Introduction to Software Testing Part 2
Introduction to Software Testing Part 2
 
#NEOTYSPAC performance testing shift left
#NEOTYSPAC performance testing shift left#NEOTYSPAC performance testing shift left
#NEOTYSPAC performance testing shift left
 
Types of Testing
Types of TestingTypes of Testing
Types of Testing
 
Small is Beautiful- Fully Automate your Test Case Design
Small is Beautiful- Fully Automate your Test Case DesignSmall is Beautiful- Fully Automate your Test Case Design
Small is Beautiful- Fully Automate your Test Case Design
 
Technical Without Code
Technical Without CodeTechnical Without Code
Technical Without Code
 

More from QAFest

QA Fest 2019. Сергій Короленко. Топ веб вразливостей за 40 хвилин
QA Fest 2019. Сергій Короленко. Топ веб вразливостей за 40 хвилинQA Fest 2019. Сергій Короленко. Топ веб вразливостей за 40 хвилин
QA Fest 2019. Сергій Короленко. Топ веб вразливостей за 40 хвилин
QAFest
 
QA Fest 2019. Анна Чернышова. Self-healing test automation 2.0. The Future
QA Fest 2019. Анна Чернышова. Self-healing test automation 2.0. The FutureQA Fest 2019. Анна Чернышова. Self-healing test automation 2.0. The Future
QA Fest 2019. Анна Чернышова. Self-healing test automation 2.0. The Future
QAFest
 
QA Fest 2019. Doug Sillars. It's just too Slow: Testing Mobile application pe...
QA Fest 2019. Doug Sillars. It's just too Slow: Testing Mobile application pe...QA Fest 2019. Doug Sillars. It's just too Slow: Testing Mobile application pe...
QA Fest 2019. Doug Sillars. It's just too Slow: Testing Mobile application pe...
QAFest
 
QA Fest 2019. Катерина Спринсян. Параллельное покрытие автотестами и другие и...
QA Fest 2019. Катерина Спринсян. Параллельное покрытие автотестами и другие и...QA Fest 2019. Катерина Спринсян. Параллельное покрытие автотестами и другие и...
QA Fest 2019. Катерина Спринсян. Параллельное покрытие автотестами и другие и...
QAFest
 
QA Fest 2019. Никита Галкин. Как зарабатывать больше
QA Fest 2019. Никита Галкин. Как зарабатывать большеQA Fest 2019. Никита Галкин. Как зарабатывать больше
QA Fest 2019. Никита Галкин. Как зарабатывать больше
QAFest
 
QA Fest 2019. Сергей Пирогов. Why everything is spoiled
QA Fest 2019. Сергей Пирогов. Why everything is spoiledQA Fest 2019. Сергей Пирогов. Why everything is spoiled
QA Fest 2019. Сергей Пирогов. Why everything is spoiled
QAFest
 
QA Fest 2019. Сергей Новик. Между мотивацией и выгоранием
QA Fest 2019. Сергей Новик. Между мотивацией и выгораниемQA Fest 2019. Сергей Новик. Между мотивацией и выгоранием
QA Fest 2019. Сергей Новик. Между мотивацией и выгоранием
QAFest
 
QA Fest 2019. Владимир Никонов. Код Шредингера или зачем и как мы тестируем н...
QA Fest 2019. Владимир Никонов. Код Шредингера или зачем и как мы тестируем н...QA Fest 2019. Владимир Никонов. Код Шредингера или зачем и как мы тестируем н...
QA Fest 2019. Владимир Никонов. Код Шредингера или зачем и как мы тестируем н...
QAFest
 
QA Fest 2019. Владимир Трандафилов. GUI automation of WEB application with SV...
QA Fest 2019. Владимир Трандафилов. GUI automation of WEB application with SV...QA Fest 2019. Владимир Трандафилов. GUI automation of WEB application with SV...
QA Fest 2019. Владимир Трандафилов. GUI automation of WEB application with SV...
QAFest
 
QA Fest 2019. Иван Крутов. Bulletproof Selenium Cluster
QA Fest 2019. Иван Крутов. Bulletproof Selenium ClusterQA Fest 2019. Иван Крутов. Bulletproof Selenium Cluster
QA Fest 2019. Иван Крутов. Bulletproof Selenium Cluster
QAFest
 
QA Fest 2019. Николай Мижигурский. Миссия /*не*/выполнима: гуманитарий собесе...
QA Fest 2019. Николай Мижигурский. Миссия /*не*/выполнима: гуманитарий собесе...QA Fest 2019. Николай Мижигурский. Миссия /*не*/выполнима: гуманитарий собесе...
QA Fest 2019. Николай Мижигурский. Миссия /*не*/выполнима: гуманитарий собесе...
QAFest
 
QA Fest 2019. Володимир Стиран. Чим раніше – тим вигідніше, але ніколи не піз...
QA Fest 2019. Володимир Стиран. Чим раніше – тим вигідніше, але ніколи не піз...QA Fest 2019. Володимир Стиран. Чим раніше – тим вигідніше, але ніколи не піз...
QA Fest 2019. Володимир Стиран. Чим раніше – тим вигідніше, але ніколи не піз...
QAFest
 
QA Fest 2019. Дмитрий Прокопук. Mocks and network tricks in UI automation
QA Fest 2019. Дмитрий Прокопук. Mocks and network tricks in UI automationQA Fest 2019. Дмитрий Прокопук. Mocks and network tricks in UI automation
QA Fest 2019. Дмитрий Прокопук. Mocks and network tricks in UI automation
QAFest
 
QA Fest 2019. Екатерина Дядечко. Тестирование медицинского софта — вызовы и в...
QA Fest 2019. Екатерина Дядечко. Тестирование медицинского софта — вызовы и в...QA Fest 2019. Екатерина Дядечко. Тестирование медицинского софта — вызовы и в...
QA Fest 2019. Екатерина Дядечко. Тестирование медицинского софта — вызовы и в...
QAFest
 
QA Fest 2019. Катерина Черникова. Tune your P’s: the pop-art of keeping testa...
QA Fest 2019. Катерина Черникова. Tune your P’s: the pop-art of keeping testa...QA Fest 2019. Катерина Черникова. Tune your P’s: the pop-art of keeping testa...
QA Fest 2019. Катерина Черникова. Tune your P’s: the pop-art of keeping testa...
QAFest
 
QA Fest 2019. Алиса Бойко. Какнезапутаться в коммуникативных сетях IT
QA Fest 2019. Алиса Бойко. Какнезапутаться в коммуникативных сетях ITQA Fest 2019. Алиса Бойко. Какнезапутаться в коммуникативных сетях IT
QA Fest 2019. Алиса Бойко. Какнезапутаться в коммуникативных сетях IT
QAFest
 
QA Fest 2019. Святослав Логин. Как найти уязвимости в мобильном приложении
QA Fest 2019. Святослав Логин. Как найти уязвимости в мобильном приложенииQA Fest 2019. Святослав Логин. Как найти уязвимости в мобильном приложении
QA Fest 2019. Святослав Логин. Как найти уязвимости в мобильном приложении
QAFest
 
QA Fest 2019. Катерина Шепелєва та Інна Оснач. Що українцям потрібно знати пр...
QA Fest 2019. Катерина Шепелєва та Інна Оснач. Що українцям потрібно знати пр...QA Fest 2019. Катерина Шепелєва та Інна Оснач. Що українцям потрібно знати пр...
QA Fest 2019. Катерина Шепелєва та Інна Оснач. Що українцям потрібно знати пр...
QAFest
 
QA Fest 2019. Антон Серпутько. Нагрузочное тестирование распределенных асинхр...
QA Fest 2019. Антон Серпутько. Нагрузочное тестирование распределенных асинхр...QA Fest 2019. Антон Серпутько. Нагрузочное тестирование распределенных асинхр...
QA Fest 2019. Антон Серпутько. Нагрузочное тестирование распределенных асинхр...
QAFest
 
QA Fest 2019. Петр Тарасенко. QA Hackathon - The Cookbook 22
QA Fest 2019. Петр Тарасенко. QA Hackathon - The Cookbook 22QA Fest 2019. Петр Тарасенко. QA Hackathon - The Cookbook 22
QA Fest 2019. Петр Тарасенко. QA Hackathon - The Cookbook 22
QAFest
 

More from QAFest (20)

QA Fest 2019. Сергій Короленко. Топ веб вразливостей за 40 хвилин
QA Fest 2019. Сергій Короленко. Топ веб вразливостей за 40 хвилинQA Fest 2019. Сергій Короленко. Топ веб вразливостей за 40 хвилин
QA Fest 2019. Сергій Короленко. Топ веб вразливостей за 40 хвилин
 
QA Fest 2019. Анна Чернышова. Self-healing test automation 2.0. The Future
QA Fest 2019. Анна Чернышова. Self-healing test automation 2.0. The FutureQA Fest 2019. Анна Чернышова. Self-healing test automation 2.0. The Future
QA Fest 2019. Анна Чернышова. Self-healing test automation 2.0. The Future
 
QA Fest 2019. Doug Sillars. It's just too Slow: Testing Mobile application pe...
QA Fest 2019. Doug Sillars. It's just too Slow: Testing Mobile application pe...QA Fest 2019. Doug Sillars. It's just too Slow: Testing Mobile application pe...
QA Fest 2019. Doug Sillars. It's just too Slow: Testing Mobile application pe...
 
QA Fest 2019. Катерина Спринсян. Параллельное покрытие автотестами и другие и...
QA Fest 2019. Катерина Спринсян. Параллельное покрытие автотестами и другие и...QA Fest 2019. Катерина Спринсян. Параллельное покрытие автотестами и другие и...
QA Fest 2019. Катерина Спринсян. Параллельное покрытие автотестами и другие и...
 
QA Fest 2019. Никита Галкин. Как зарабатывать больше
QA Fest 2019. Никита Галкин. Как зарабатывать большеQA Fest 2019. Никита Галкин. Как зарабатывать больше
QA Fest 2019. Никита Галкин. Как зарабатывать больше
 
QA Fest 2019. Сергей Пирогов. Why everything is spoiled
QA Fest 2019. Сергей Пирогов. Why everything is spoiledQA Fest 2019. Сергей Пирогов. Why everything is spoiled
QA Fest 2019. Сергей Пирогов. Why everything is spoiled
 
QA Fest 2019. Сергей Новик. Между мотивацией и выгоранием
QA Fest 2019. Сергей Новик. Между мотивацией и выгораниемQA Fest 2019. Сергей Новик. Между мотивацией и выгоранием
QA Fest 2019. Сергей Новик. Между мотивацией и выгоранием
 
QA Fest 2019. Владимир Никонов. Код Шредингера или зачем и как мы тестируем н...
QA Fest 2019. Владимир Никонов. Код Шредингера или зачем и как мы тестируем н...QA Fest 2019. Владимир Никонов. Код Шредингера или зачем и как мы тестируем н...
QA Fest 2019. Владимир Никонов. Код Шредингера или зачем и как мы тестируем н...
 
QA Fest 2019. Владимир Трандафилов. GUI automation of WEB application with SV...
QA Fest 2019. Владимир Трандафилов. GUI automation of WEB application with SV...QA Fest 2019. Владимир Трандафилов. GUI automation of WEB application with SV...
QA Fest 2019. Владимир Трандафилов. GUI automation of WEB application with SV...
 
QA Fest 2019. Иван Крутов. Bulletproof Selenium Cluster
QA Fest 2019. Иван Крутов. Bulletproof Selenium ClusterQA Fest 2019. Иван Крутов. Bulletproof Selenium Cluster
QA Fest 2019. Иван Крутов. Bulletproof Selenium Cluster
 
QA Fest 2019. Николай Мижигурский. Миссия /*не*/выполнима: гуманитарий собесе...
QA Fest 2019. Николай Мижигурский. Миссия /*не*/выполнима: гуманитарий собесе...QA Fest 2019. Николай Мижигурский. Миссия /*не*/выполнима: гуманитарий собесе...
QA Fest 2019. Николай Мижигурский. Миссия /*не*/выполнима: гуманитарий собесе...
 
QA Fest 2019. Володимир Стиран. Чим раніше – тим вигідніше, але ніколи не піз...
QA Fest 2019. Володимир Стиран. Чим раніше – тим вигідніше, але ніколи не піз...QA Fest 2019. Володимир Стиран. Чим раніше – тим вигідніше, але ніколи не піз...
QA Fest 2019. Володимир Стиран. Чим раніше – тим вигідніше, але ніколи не піз...
 
QA Fest 2019. Дмитрий Прокопук. Mocks and network tricks in UI automation
QA Fest 2019. Дмитрий Прокопук. Mocks and network tricks in UI automationQA Fest 2019. Дмитрий Прокопук. Mocks and network tricks in UI automation
QA Fest 2019. Дмитрий Прокопук. Mocks and network tricks in UI automation
 
QA Fest 2019. Екатерина Дядечко. Тестирование медицинского софта — вызовы и в...
QA Fest 2019. Екатерина Дядечко. Тестирование медицинского софта — вызовы и в...QA Fest 2019. Екатерина Дядечко. Тестирование медицинского софта — вызовы и в...
QA Fest 2019. Екатерина Дядечко. Тестирование медицинского софта — вызовы и в...
 
QA Fest 2019. Катерина Черникова. Tune your P’s: the pop-art of keeping testa...
QA Fest 2019. Катерина Черникова. Tune your P’s: the pop-art of keeping testa...QA Fest 2019. Катерина Черникова. Tune your P’s: the pop-art of keeping testa...
QA Fest 2019. Катерина Черникова. Tune your P’s: the pop-art of keeping testa...
 
QA Fest 2019. Алиса Бойко. Какнезапутаться в коммуникативных сетях IT
QA Fest 2019. Алиса Бойко. Какнезапутаться в коммуникативных сетях ITQA Fest 2019. Алиса Бойко. Какнезапутаться в коммуникативных сетях IT
QA Fest 2019. Алиса Бойко. Какнезапутаться в коммуникативных сетях IT
 
QA Fest 2019. Святослав Логин. Как найти уязвимости в мобильном приложении
QA Fest 2019. Святослав Логин. Как найти уязвимости в мобильном приложенииQA Fest 2019. Святослав Логин. Как найти уязвимости в мобильном приложении
QA Fest 2019. Святослав Логин. Как найти уязвимости в мобильном приложении
 
QA Fest 2019. Катерина Шепелєва та Інна Оснач. Що українцям потрібно знати пр...
QA Fest 2019. Катерина Шепелєва та Інна Оснач. Що українцям потрібно знати пр...QA Fest 2019. Катерина Шепелєва та Інна Оснач. Що українцям потрібно знати пр...
QA Fest 2019. Катерина Шепелєва та Інна Оснач. Що українцям потрібно знати пр...
 
QA Fest 2019. Антон Серпутько. Нагрузочное тестирование распределенных асинхр...
QA Fest 2019. Антон Серпутько. Нагрузочное тестирование распределенных асинхр...QA Fest 2019. Антон Серпутько. Нагрузочное тестирование распределенных асинхр...
QA Fest 2019. Антон Серпутько. Нагрузочное тестирование распределенных асинхр...
 
QA Fest 2019. Петр Тарасенко. QA Hackathon - The Cookbook 22
QA Fest 2019. Петр Тарасенко. QA Hackathon - The Cookbook 22QA Fest 2019. Петр Тарасенко. QA Hackathon - The Cookbook 22
QA Fest 2019. Петр Тарасенко. QA Hackathon - The Cookbook 22
 

Recently uploaded

FEELINGS AND EMOTIONS INSIDE OUT MOVIE.ppt
FEELINGS AND EMOTIONS INSIDE OUT MOVIE.pptFEELINGS AND EMOTIONS INSIDE OUT MOVIE.ppt
FEELINGS AND EMOTIONS INSIDE OUT MOVIE.ppt
JenezarieTarra1
 
RDBMS Lecture Notes Unit4 chapter12 VIEW
RDBMS Lecture Notes Unit4 chapter12 VIEWRDBMS Lecture Notes Unit4 chapter12 VIEW
RDBMS Lecture Notes Unit4 chapter12 VIEW
Murugan Solaiyappan
 
C Interview Questions PDF By Scholarhat.pdf
C Interview Questions PDF By Scholarhat.pdfC Interview Questions PDF By Scholarhat.pdf
C Interview Questions PDF By Scholarhat.pdf
Scholarhat
 
How to Manage Early Receipt Printing in Odoo 17 POS
How to Manage Early Receipt Printing in Odoo 17 POSHow to Manage Early Receipt Printing in Odoo 17 POS
How to Manage Early Receipt Printing in Odoo 17 POS
Celine George
 
How to Empty a One2Many Field in Odoo 17
How to Empty a One2Many Field in Odoo 17How to Empty a One2Many Field in Odoo 17
How to Empty a One2Many Field in Odoo 17
Celine George
 
NAEYC Code of Ethical Conduct Resource Book
NAEYC Code of Ethical Conduct Resource BookNAEYC Code of Ethical Conduct Resource Book
NAEYC Code of Ethical Conduct Resource Book
lakitawilson
 
SD_Integrating 21st Century Skills in Classroom-based Assessment.pptx
SD_Integrating 21st Century Skills in Classroom-based Assessment.pptxSD_Integrating 21st Century Skills in Classroom-based Assessment.pptx
SD_Integrating 21st Century Skills in Classroom-based Assessment.pptx
elwoodprias1
 
How to Create a New Article in Knowledge App in Odoo 17
How to Create a New Article in Knowledge App in Odoo 17How to Create a New Article in Knowledge App in Odoo 17
How to Create a New Article in Knowledge App in Odoo 17
Celine George
 
Genetics Teaching Plan: Dr.Kshirsagar R.V.
Genetics Teaching Plan: Dr.Kshirsagar R.V.Genetics Teaching Plan: Dr.Kshirsagar R.V.
Genetics Teaching Plan: Dr.Kshirsagar R.V.
DrRavindrakshirsagar1
 
Parent PD Design for Professional Development .docx
Parent PD Design for Professional Development .docxParent PD Design for Professional Development .docx
Parent PD Design for Professional Development .docx
AntonioJarligoCompra
 
JavaScript Interview Questions PDF By ScholarHat
JavaScript Interview  Questions PDF By ScholarHatJavaScript Interview  Questions PDF By ScholarHat
JavaScript Interview Questions PDF By ScholarHat
Scholarhat
 
BÀI TẬP BỔ TRỢ 4 KỸ NĂNG TIẾNG ANH LỚP 12 - GLOBAL SUCCESS - FORM MỚI 2025 - ...
BÀI TẬP BỔ TRỢ 4 KỸ NĂNG TIẾNG ANH LỚP 12 - GLOBAL SUCCESS - FORM MỚI 2025 - ...BÀI TẬP BỔ TRỢ 4 KỸ NĂNG TIẾNG ANH LỚP 12 - GLOBAL SUCCESS - FORM MỚI 2025 - ...
BÀI TẬP BỔ TRỢ 4 KỸ NĂNG TIẾNG ANH LỚP 12 - GLOBAL SUCCESS - FORM MỚI 2025 - ...
Nguyen Thanh Tu Collection
 
SEQUNCES Lecture_Notes_Unit4_chapter11_sequence
SEQUNCES  Lecture_Notes_Unit4_chapter11_sequenceSEQUNCES  Lecture_Notes_Unit4_chapter11_sequence
SEQUNCES Lecture_Notes_Unit4_chapter11_sequence
Murugan Solaiyappan
 
CTD Punjab Police Past Papers MCQs PPSC PDF
CTD Punjab Police Past Papers MCQs PPSC PDFCTD Punjab Police Past Papers MCQs PPSC PDF
CTD Punjab Police Past Papers MCQs PPSC PDF
hammadmughal76316
 
What is Packaging of Products in Odoo 17
What is Packaging of Products in Odoo 17What is Packaging of Products in Odoo 17
What is Packaging of Products in Odoo 17
Celine George
 
matatag curriculum education for Kindergarten
matatag curriculum education for Kindergartenmatatag curriculum education for Kindergarten
matatag curriculum education for Kindergarten
SarahAlie1
 
How To Update One2many Field From OnChange of Field in Odoo 17
How To Update One2many Field From OnChange of Field in Odoo 17How To Update One2many Field From OnChange of Field in Odoo 17
How To Update One2many Field From OnChange of Field in Odoo 17
Celine George
 
DANH SÁCH THÍ SINH XÉT TUYỂN SỚM ĐỦ ĐIỀU KIỆN TRÚNG TUYỂN ĐẠI HỌC CHÍNH QUY N...
DANH SÁCH THÍ SINH XÉT TUYỂN SỚM ĐỦ ĐIỀU KIỆN TRÚNG TUYỂN ĐẠI HỌC CHÍNH QUY N...DANH SÁCH THÍ SINH XÉT TUYỂN SỚM ĐỦ ĐIỀU KIỆN TRÚNG TUYỂN ĐẠI HỌC CHÍNH QUY N...
DANH SÁCH THÍ SINH XÉT TUYỂN SỚM ĐỦ ĐIỀU KIỆN TRÚNG TUYỂN ĐẠI HỌC CHÍNH QUY N...
thanhluan21
 
How to Manage Large Scrollbar in Odoo 17 POS
How to Manage Large Scrollbar in Odoo 17 POSHow to Manage Large Scrollbar in Odoo 17 POS
How to Manage Large Scrollbar in Odoo 17 POS
Celine George
 
formative Evaluation By Dr.Kshirsagar R.V
formative Evaluation By Dr.Kshirsagar R.Vformative Evaluation By Dr.Kshirsagar R.V
formative Evaluation By Dr.Kshirsagar R.V
DrRavindrakshirsagar1
 

Recently uploaded (20)

FEELINGS AND EMOTIONS INSIDE OUT MOVIE.ppt
FEELINGS AND EMOTIONS INSIDE OUT MOVIE.pptFEELINGS AND EMOTIONS INSIDE OUT MOVIE.ppt
FEELINGS AND EMOTIONS INSIDE OUT MOVIE.ppt
 
RDBMS Lecture Notes Unit4 chapter12 VIEW
RDBMS Lecture Notes Unit4 chapter12 VIEWRDBMS Lecture Notes Unit4 chapter12 VIEW
RDBMS Lecture Notes Unit4 chapter12 VIEW
 
C Interview Questions PDF By Scholarhat.pdf
C Interview Questions PDF By Scholarhat.pdfC Interview Questions PDF By Scholarhat.pdf
C Interview Questions PDF By Scholarhat.pdf
 
How to Manage Early Receipt Printing in Odoo 17 POS
How to Manage Early Receipt Printing in Odoo 17 POSHow to Manage Early Receipt Printing in Odoo 17 POS
How to Manage Early Receipt Printing in Odoo 17 POS
 
How to Empty a One2Many Field in Odoo 17
How to Empty a One2Many Field in Odoo 17How to Empty a One2Many Field in Odoo 17
How to Empty a One2Many Field in Odoo 17
 
NAEYC Code of Ethical Conduct Resource Book
NAEYC Code of Ethical Conduct Resource BookNAEYC Code of Ethical Conduct Resource Book
NAEYC Code of Ethical Conduct Resource Book
 
SD_Integrating 21st Century Skills in Classroom-based Assessment.pptx
SD_Integrating 21st Century Skills in Classroom-based Assessment.pptxSD_Integrating 21st Century Skills in Classroom-based Assessment.pptx
SD_Integrating 21st Century Skills in Classroom-based Assessment.pptx
 
How to Create a New Article in Knowledge App in Odoo 17
How to Create a New Article in Knowledge App in Odoo 17How to Create a New Article in Knowledge App in Odoo 17
How to Create a New Article in Knowledge App in Odoo 17
 
Genetics Teaching Plan: Dr.Kshirsagar R.V.
Genetics Teaching Plan: Dr.Kshirsagar R.V.Genetics Teaching Plan: Dr.Kshirsagar R.V.
Genetics Teaching Plan: Dr.Kshirsagar R.V.
 
Parent PD Design for Professional Development .docx
Parent PD Design for Professional Development .docxParent PD Design for Professional Development .docx
Parent PD Design for Professional Development .docx
 
JavaScript Interview Questions PDF By ScholarHat
JavaScript Interview  Questions PDF By ScholarHatJavaScript Interview  Questions PDF By ScholarHat
JavaScript Interview Questions PDF By ScholarHat
 
BÀI TẬP BỔ TRỢ 4 KỸ NĂNG TIẾNG ANH LỚP 12 - GLOBAL SUCCESS - FORM MỚI 2025 - ...
BÀI TẬP BỔ TRỢ 4 KỸ NĂNG TIẾNG ANH LỚP 12 - GLOBAL SUCCESS - FORM MỚI 2025 - ...BÀI TẬP BỔ TRỢ 4 KỸ NĂNG TIẾNG ANH LỚP 12 - GLOBAL SUCCESS - FORM MỚI 2025 - ...
BÀI TẬP BỔ TRỢ 4 KỸ NĂNG TIẾNG ANH LỚP 12 - GLOBAL SUCCESS - FORM MỚI 2025 - ...
 
SEQUNCES Lecture_Notes_Unit4_chapter11_sequence
SEQUNCES  Lecture_Notes_Unit4_chapter11_sequenceSEQUNCES  Lecture_Notes_Unit4_chapter11_sequence
SEQUNCES Lecture_Notes_Unit4_chapter11_sequence
 
CTD Punjab Police Past Papers MCQs PPSC PDF
CTD Punjab Police Past Papers MCQs PPSC PDFCTD Punjab Police Past Papers MCQs PPSC PDF
CTD Punjab Police Past Papers MCQs PPSC PDF
 
What is Packaging of Products in Odoo 17
What is Packaging of Products in Odoo 17What is Packaging of Products in Odoo 17
What is Packaging of Products in Odoo 17
 
matatag curriculum education for Kindergarten
matatag curriculum education for Kindergartenmatatag curriculum education for Kindergarten
matatag curriculum education for Kindergarten
 
How To Update One2many Field From OnChange of Field in Odoo 17
How To Update One2many Field From OnChange of Field in Odoo 17How To Update One2many Field From OnChange of Field in Odoo 17
How To Update One2many Field From OnChange of Field in Odoo 17
 
DANH SÁCH THÍ SINH XÉT TUYỂN SỚM ĐỦ ĐIỀU KIỆN TRÚNG TUYỂN ĐẠI HỌC CHÍNH QUY N...
DANH SÁCH THÍ SINH XÉT TUYỂN SỚM ĐỦ ĐIỀU KIỆN TRÚNG TUYỂN ĐẠI HỌC CHÍNH QUY N...DANH SÁCH THÍ SINH XÉT TUYỂN SỚM ĐỦ ĐIỀU KIỆN TRÚNG TUYỂN ĐẠI HỌC CHÍNH QUY N...
DANH SÁCH THÍ SINH XÉT TUYỂN SỚM ĐỦ ĐIỀU KIỆN TRÚNG TUYỂN ĐẠI HỌC CHÍNH QUY N...
 
How to Manage Large Scrollbar in Odoo 17 POS
How to Manage Large Scrollbar in Odoo 17 POSHow to Manage Large Scrollbar in Odoo 17 POS
How to Manage Large Scrollbar in Odoo 17 POS
 
formative Evaluation By Dr.Kshirsagar R.V
formative Evaluation By Dr.Kshirsagar R.Vformative Evaluation By Dr.Kshirsagar R.V
formative Evaluation By Dr.Kshirsagar R.V
 

QA Fest 2015. Владимир Примаков. Процесс нагрузочного тестирования и его планирование

  • 1. Performance and Load Testing Process. How to plan it www.qaexperts.pro
  • 2. 22 12+ years general working experience in IT/QA (40+ projects) ---------- •QA Consultant / QA Manager at Ciklum Interactive •Load Testing Manager (3+ years, 15+ projects) •Automated Testing Manager •Co-owner at www.qaexperts.pro ---------- Skype: vladimir.primakov Linkedin: ua.linkedin.com/in/vladimirprimakov/ Email: v.v.primakov@gmail.com Some Words About Me Volodymyr Prymakov (Vladimir Primakov) www.qaexperts.pro
  • 3. 33 • Introduction - 5 min • Main Part - 30 min • Questions - 5 min Presentation’s Structure Presentation Plan www.qaexperts.pro
  • 4. 44 Assumptions We assume that: • Load & Performance testing tools are already chosen and used. • Load testing framework and infrastructure is build and fully functional. • Load testing team is already formed www.qaexperts.pro
  • 5. 55 Load Testing Process 1. Planning 2. Evaluating Technical Risks 3. Prerequisites solving 4. Script implementation & adjustment 5. Test execution, analysis, and reporting 6. Summary reporting and conclusions www.qaexperts.pro
  • 8. 88 Product Idea: Business domain, main specialties PRODUCT AND SCOPE Solution Architecture. The underlying technology. Pointer to Weak areas. www.qaexperts.pro
  • 9. 99 Product and product parts IN and OUT of scope. Priority PRODUCT AND SCOPE 3D-Party services IN and OUT of scope. Mockups. Impact? www.qaexperts.pro
  • 11. 1111 www.qaexperts.pro PRODUCT AND SCOPE COMMUNICATION TECHNOLOGY • Communication model and implementation for a) client <-> server side b) server-side <-> server-side • Protocols/exchange formats used in the communication • If there is any repetitive calls available in the communication? HTTP(S) (85%) Web-sockets (10%) TCP/IP (5%) TLS HTML Strings Binary XML JSON
  • 12. 1212 PRODUCT AND SCOPE USAGE FLOWS / USER ACTIONS www.qaexperts.pro
  • 13. 1313 PRODUCT AND SCOPE USAGE FLOWS / USER ACTIONS www.qaexperts.pro •Types of flows • Logged in and anonymous • Main Operations • Navigational •Approach (Stability and Traceability): •Try to isolate testable operation in a single flow •Make flows as shorter as you can •Avoid dependencies between flows
  • 14. 1414 PRODUCT AND SCOPE USAGE FLOWS / USER ACTIONS www.qaexperts.pro Flow Type Flow % Anonymous Registration 3% Main Pages Navigation 10% Categories Navigation 15% Search and Navigate Item 20% Search and Buy item 15% Search and Add to Cart / Remove from cart 15% Compare Items 5% Logged in Add comment to / Ask question about an item 5% Add item to my wish list 5,00% Navigate to wish list 5,00% Edit Profile 2% Etc… … SUM 100,00% Registration Flow 1. Open main page 2. Go to the registration page 3. Fill in all the required data and click Register 4. Get the corresponding registration activation email - follow the registration activation link and login Example:
  • 15. 1515 TEST ENVIRONMENTS www.qaexperts.pro Test environments for: 1. Evaluating the product / Technical Risk 2. Script implementation 3. Load test execution Difference between environment for test execution and target production environment? The corresponding risks. Possibility to minimize the difference.
  • 16. 1616 LOAD PROFILES TYPES OF LOAD TESTS www.qaexperts.pro •10-100% Usual Load •Stress Load •Capacity Testing •Spike Stress Load •Soak Testing (24h-7d) •Volume Testing Load profiles Types of load tests
  • 17. 1717 EXPECTED LOAD PASS-FAIL CRITERIA www.qaexperts.pro •Client-side criteria •Server-side criteria •Test Data
  • 18. 1818 EXPECTED LOAD PASS-FAIL CRITERIA www.qaexperts.pro Client Which is actually: a) Number of registered users or b) Number of users per day or d) Correct number of concurrent users  Response time for a operation or a request? Number of concurrent users =100000 Response time: 2 Sec
  • 19. 1919 EXPECTED LOAD PASS-FAIL CRITERIA www.qaexperts.pro CLIENT-SIDE CRITERIA •Max number of concurrent users (anonymous and logged-in, active and passive) •% of Mobile/web/desktop/terminal clients •Max rate of main actions/operations per min, e.g.: 100 logins/min, 10 registration/min •Max response times for key pages and operations, e.g.: login – 3 sec, registration – 5 sec •Acceptable % of fails, e.g.: 2% Availability of such statistics from web-logs, Google analytics, and other tools/places?
  • 20. 2020 EXPECTED LOAD PASS-FAIL CRITERIA www.qaexperts.pro SERVER-SIDE CRITERIA •Max Resource Consumption: CPU (e.g. 80%), RAM (e.g. 80%), Disk in/out, Network, etc. These parameters should be monitored within every node of a certain component •Max Errors Rate (in logs, etc.) e.g. 0.5%
  • 21. 2121 EXPECTED LOAD PASS-FAIL CRITERIA www.qaexperts.pro TEST DATA •Users •User’s Data •Product’s Data --------------------------- 1. Type of Data, 2. Size and Amount of Data
  • 22. 2222 OTHER NUANCES www.qaexperts.pro •Scheduling services? •Client’s internet connection speed? •Static content hosting: CDN or own servers? •Target region (USA, Europe, etc.)?
  • 23. 2323 ORGANIZATIONAL MOMENTS www.qaexperts.pro •Project Milestones? •Date and Time frames for load testing runs? •Stakeholders? •Organizational, Business, Technical, and Server-side support contacts
  • 24. 2424 • Difference between Target and Test environments • Risk of using service- mockups • Risk of negatively impacting the production environment or 3d party services • Wrong target load expectations • … www.qaexperts.pro TYPICAL RISKS
  • 25. 2525 • Product & Solution successfully passed UAT • All prerequisites are solved • No unexpected deploys • … www.qaexperts.pro DEPENDENCIES / ASSUMPTIONS
  • 28. 2828 www.qaexperts.pro PREREQUISITES SOLVING PLANNING IS DONE! Non-functional requirements and user flows are defined.
  • 29. 2929 www.qaexperts.pro PREREQUISITES SOLVING PREPARE TEST ENVIRONMENT! • Test environments availability and access. Whitelisting load injectors. • Test environment and product stability. • Sever-side monitoring tools. • Test environment alignment to the production environment. • Test users and test data.
  • 31. 3131 www.qaexperts.pro SCRIPT IMPLEMENTATION 1. Type of scripts: • Logged-in/Anonymous • Main flows scripts, • Navigational Scripts, • Repetitive calls scripts 2. Adjust scripts to the target environments 3. Regularly Update scripts to product changes 4. Define and follow script readiness criteria
  • 33. 3333 www.qaexperts.pro PLAN TEST EXECUTION a) Plan load gradually (to eliminate Risks): 10% 25% 50% 75% 100% b) Or start load from Capacity test with long ramp-up period (to get fast results) Define load schedule: Types of load tests, dates, and timeframes for test runs
  • 34. 3434 www.qaexperts.pro • Type of test/Goal • Date & Time of test • Server sider conditions and configuration; product version. • % of load • Ramp-up, Sustain, Ramp-down • Scripts (Flows) to run • Number of concurrent users in general and per script • Expected operations rate • Responsible specialists and their contacts in case of emergency. PLAN A RUN PLAN TEST EXECUTION
  • 35. 3535 www.qaexperts.pro TEST EXECUTION • Make sure the server side support’s specialists are available • Do smoke test • Run Test: • Monitor Test product health: • Client • Response time growth • Error rate growth • Error types • Server • Resource consumptions • Errors • Servers availability • Communicate the results in real time • Be ready to Stop Test in case of Emergency! • Monitor Load Injectors health • If possible, play manually with the product under load
  • 36. 3636 www.qaexperts.pro TEST RESULTS ANALYSIS AND REPORTING Client Side: • Response time-based capacity point • Errors-based capacity point • Failed scripts and transactions • Slow scripts and slow transactions • Errors details: types and amount • Max transactions/operations rates reached. • Timely trends and correlation with concurrent users number • Main bottlenecks concussions Server-side: • Resources consumptions • Errors and log-messages • Slow DB transactions • Correlation with client side metrics
  • 37. 3737 www.qaexperts.pro Execute Load Test Meet Pass Criteria? Do Profiling Load Test Fix Solution/ Product TEST EXECUTION PROCESS Ready For Release YES NO If possible, the final step of load testing is testing on the prod environment under conditions similar to the real ones. Update Scripts Analyze Results
  • 39. 3939 www.qaexperts.pro SUMMARY REPORTING AND CONCLUSIONS 1. Compare test runs: types of tests, server-side conditions, client and servers-side metrics 2. Conclusion about readiness to release. 3. Recommendations on further steps: a. Next load tests required. b. How to survive the expected load in case of not meeting the PASS criteria.
  • 40. 4141 GENERAL SCHEDULE & ESTIMATION Weeks 1 2 3 4 5 6 7 8 9 Planning Evaluating Technical Risks Prerequisites solving Script implementation & adjustment Test execution, analysis, and reporting Summary reporting and conclusions 80% 50% EFFORTS AND DURATION www.qaexperts.pro
  • 41. 4242 LOAD TESTING IN SDLS • In ongoing development – Verifying and validating component, queue of components, and integration related performance & robustness. • Before Release – Verifying and validating the whole product performance & robustness before release. • Maintenance – Verify and validating architectural, configurational, capacity-related, db-related, and integration-related changes. WHEN TO PERFORM www.qaexperts.pro