Anastasia Dovgan - practical tips and pitfalls of passing an external audit

1. ПРАКТИЧЕСКИЕ СОВЕТЫ И
ПОДВОДНЫЕ КАМНИ
ПРОХОЖДЕНИЯ ВНЕШНЕГО АУДИТА
Anastasiya Dovhan, GlobalLogic, Ukraine 1

2.  6+ лет опыта работы в тестировании
 3+ лет на позиции QA Lead
 Тестирование, планирование, эстимирование
преимущественно десктопных приложений в
том числе enterprise решений
 Активное участие в развитии QA Community
родного города
 Организация QA-конференций и митапов
 LinkedIn: www.linkedin.com/in/anastasiyadovgan/
 Email: dovhanastasia@gmail.com
Обо мне
2

3. AGENDA
 А нужен ли Аудит?
 Для каких проектов и продуктов аудит необходим?
 Почему внешний?
 Подготовка и процесс проходжения внешнего аудита
 Советы на основе личного опыта
 Выводы
3

4. 4
История одного неудачного релиза…
 Только ОН знал как это работает…
 Только ОНА знала как это
протестировать…
 Только НЕ ЭТОТ СЕРВЕР! Только
не он…

5. 5
История другого неудачного релиза…
 Почему этот проклятый баг вдруг
вылез на продакшине?
 Зачем они обновили спецификацию
через 3 дня после нашего релиза?
 А что уже вышел Android 9.0??? (iOS
11 / SQL Server 2017 / whatever) Ой…

6. 6

7. Независимая экспертиза
продуктов, процессов или системы
с целью определения ее
соответствия спецификациям,
нормам или стандартам
7

8. Квалифицированные специалисты
в области области проведения
аудитов
Непредвзятость, независимое
мнение
Возможности по улучшению
процессов
Возможность получить сертификат
8

9.  Продукт имеет повышенные требования к качеству (ПО для
медицинской, военной промышленности, управление техникой и
высокоточным оборудованием, финансовая сфера и т.д.)
 Большим проектам состоящим из нескольких связанных под-проектов
для:
 Понимания “состояния дел”
 Введения общих стандартов если это необходимо
 Продукт (или часть его функционала) разрабатывается согласно
утверждённым спецификациям, нормативам, системам качества
 Кардинальные изменения в процессах, подходах разработки или
тестирования продукта
9

10. 10

11. 11
Прогон регрессии:
~ 2000 man/hours
~ 285 man/days
~ 14 man/months
~ 1.2 man/years
Время релиза:
1-2 years

12. Подготовка
 Разработка стратегии
и тактики,
согласование с
клиентом
 График проведения,
план и программа
аудита
 Налаживание
коммуникации
Заключение
 Отчёт от аудитора с
указанием
отклонений и их
критичности
 План по устранению
недочетов с
дедлайнами
 Планирование
повторного
аудирования
 Сбор и анализ
данных аудитором
согласно плану
Аудит
12

13. Lead Auditor: <Name>
Audit dates: November 22 and 23, 2016
Notes:
 Regulation/Guidance: 21 CFR Part 11 and ISPE GAMP-5 will be used as a
reference for this audit.
 Focus on the following software: <Product name and version> Label
Printing Software
 The Following Agenda will be used as a guide to complete the audit. The
agenda may be altered at any time during the audit at the discretion of the
auditor(s).
13

14. Day 1 Areas audited
08:30 to 9:00 Opening Meeting
9:00-12:30
Quality Management System Review
- Quality Manual & Quality Policy
- Quality System Approach, Quality SOPs
- Management Responsibility
- Last audit results review
12:30 to 13:30 Lunch
13:30-17:00
Development Process controls
- Document Controls and control of records
- Software/System Development Methodology and Documentation
- Verification and Validation Testing Approach and Procedures
- Change Management, Configuration Management
- Documentation and Records Management
- Application/Product Security Practices
- Hardware Support Services
- Software Update Processes / Updates Release Processes
- Release Management 14

15.  Quality Policy Manual
 Company Overview
 Label Printing Software Literature, including a history/timeline of its
development
 Master list of Procedures - Applicable Operational Procedures for: SDLC,
Validation, Change Control, Document Control, Training, Defect Tracking,
Complaint Handling
 Organizational Structure Chart (including Management, Product Support &
Training Teams, Software Development, Software Testing and QA)
 Software Validation Documentation
 Development Records
 Training records for software development, testing & quality personnel
15

16. Required items Responsible Covered By Status ToDo
Quality Policy Manual QA Lead
INT_PD_100_Software_Qualit
y_Assurance_Handbook.docx
Need
update
To update
section 1, 2 and
7
Label Printing
Software Literature,
including a
history/timeline of its
development
PM
Product Release and License
compatibility.docx
N/A
To create
document
Software Validation
Documentation
PM
<Product_version> Release
Activities Checklist.docx
OK
Refer to our
document
QA Lead
<Product_version> Test
Summary Report.docx
OK
Refer to our
document
PM
<Product_version> Release
Memo.docx
OK
Refer to our
document 16

17. 17

18. 18

19. Categories of the deviations:
 Critical – a serious deviation from recognized quality system
and/or regulatory standard that has great impact to the quality
 Major – a deviation from recognized quality system and/or
regulatory standard that may impact quality
 Minor – a deviation from recognized quality system and/or
regulatory standard that does not impact the quality
 OFI (Opportunity for improvement) – recommendation or
suggestion aimed at continuously improving the quality system
19

20.  Critical deviations
- Information provided with the releases does not allow customers to identify
any incompatibilities with the labels created with different versions
 Major deviations
- Test execution is not documented in a way that allows an independent
review of the results obtained
 Minor deviations
- New joiner integration and training to internal processes and procedures is
not formalized
 OFI (Opportunity for improvement)
- Activities for the management of complaints are in place, but it is
recommended that a dedicated SOP (standard operating procedure) be
implemented
20

21.  Critical deviation: Information provided with the releases does not allow
customers to identify any incompatibilities with the labels created with
different versions
Action item Description Responsible
Update document
"QA Processes and
Procedures.doc”
Add section which will describe the approach and
process for old versions compatibility testing
QA Lead
Update Product
Release Notes
Updated Product Release Notes with appropriate
section: Old versions compatibility
Product
Manager
21

22. 22
Preventive actions – “must have” item

23. 23
Anything you say can and
will be used against you

24. 24
 Clarify unclear items
 It's normal to say:
 "did I understand you properly
that..."
 "does it mean that..."
 "could you rephrase the question"

25. Quality Policy Manual
Test processes
and procedures
Development
processes and
procedures
New joiner
training
procedure
Customer
support
process
25
Master list of
Procedures and Processes

26. 26

27.  References to
Standards
 References to
documents
SQA Plan
IEEE Standard 730-2014
Standard for Software Quality Assurance
Plans
Test Plan IEEE Standard 829-2008
Standard for Software Test
Documentation
Test Summary
Report
Reference documents
# Document ID-Number
[1] Test_processes_and_procedures.docx INT_PD_100
[2]
Development_processes_and_procedure
s.docx
INT_PD_101
27

28.  Modification list
 Documents
approval and
signature
Modification List
Anastasiya Dovhan 1.03 17-Nov-17
Section 4.15 updated – Bug Triage Meeting (BTM) was updated
according to the process with new TFS template
Document Information
Original Document Approval Signature:
Project Manager: <name>
R&D Manager: <name>
SQA Manager: Anastasiya Dovhan
28

29. 29
 Auditor = Friend
 Audit = Opportunities for improvement
 Participation in the audit = Skills improvement
 Regular audits = Confidence in the processes and quality

30. 30
Preparation checklist
for passing QA Audit

31. 31