Testers want to be responsible and professional. However, they often come under pressure to comply with rules, standards, and processes that aren't always helpful. It's the price of keeping your auditors happy. But do you really know what auditors want? Are they all simply rule-obsessed, pedantic “little dictators”? James Christie shows why good auditors worry about risk—not rules. They want to explain the important risks to the people who lose sleep over them. James explains auditors' and regulators' attitudes toward risk and evidence. He shows that auditors' standards and governance models do have useful advice—knowledge that can help you choose the right testing approach for your project. James shows how to enlist smart auditors as valuable allies—and how to challenge the poor ones. Understanding auditors' needs will help you do better testing, at less cost. Wouldn't senior management and your stakeholders be interested in that?
Is Agile the Prescription for the Public Sector’s IT Woes?TechWell
Information technology (IT) projects are notorious for exceeding budget and schedule estimates, and high visibility failures are common. IT projects in the public sector are particularly challenging. State, provincial, and federal governments worldwide have sponsored noteworthy disasters in the past twenty years. As agile methods have evolved, become more mainstream, and demonstrated their value in the private sector in the past decade, they are often cited as a remedy for the public sector’s IT misery. Payson Hall examines the gap between current public sector IT project challenges and the often-suggested agile solution. Payson explores the challenges to effective vendor-delivered public sector agile projects and possible responses to those challenges. He answers the questions: Is agile ready for large public sector projects? Is the public sector ready for agile? Leave with a better understanding of the problems public sector entities and vendors face and ideas for overcoming some of those barriers.
Executives’ Influence on Agile: The Good, the Bad, and the UglyTechWell
The evidence is in—and it's compelling. Well-executed agile practices can shorten software project schedules by 30 percent while cutting defects by 75 percent. However, many organizations struggle with agile adoption. And some of these struggles can be attributed to the executive leadership. In many cases, the "lead, follow, or get out of the way" attitude causes executives to try to lead when they should be following or getting out of the way. Drawing on his experiences with agile adoption at Synacor as it implements agile on an enterprise scale, Steve Davi illustrates how the executives on the ground can help or hurt agile adoption. Steve shares ways to turn those executive wolves into agile enablers as he describes the four critical actions that executives should take to support agile within their organization―define the vision, boundaries, and constraints; gain support and remove impediments; ensure openness and trust; and hold teams accountable.
Innovation Thinking: Evolve and Expand Your CapabilitiesTechWell
Innovation is a word frequently tossed around in organizations today. The standard clichés are do more with less and be creative. Companies want to be innovative but often struggle with how to define, implement, prioritize, and track their innovation efforts. Using the Innovation to Types model, Jennifer Bonine will help you transform your thinking regarding innovation and understand if your team and company goals match their innovation efforts. Learn how to classify your activities as "core" (to the business) or "context" (essential, but non-revenue generating). Once you understand how your innovation activities are related to revenue generating activities, you can better decide how much of your effort should be spent on core or context activities. Take away tools including an Innovation to Types model for classifying innovation, a Core and Context model to classify your activities, and a way to map your innovation initiatives to different contexts.
Getting Your Message Across: Communication Skills for TestersTechWell
Communication is at the heart of our profession. No matter how advanced our testing capabilities are, if we can’t convey our concerns in ways that connect with key members of the project team, our contribution is likely to be ignored. Because we act solely in an advisory capacity, rather than being in command, our power to exert influence is almost entirely based on our communication skills. With people suffering information overload and deluged with emails, it is more important than ever that we craft succinct and effective messages, using a range of communication modalities. Join Thomas McCoy as he draws on techniques from journalism, public relations, professional writing, psychology, and marketing to help you get your message across. Key themes include: non-verbal communication, presentation skills, persuasive writing, influencing skills, graphic communication, and communicating in teams and meetings. We will use a range of hands-on exercises to practice the concepts being discussed.
Whether you are new to testing or looking for a better way to organize your test practices, understanding risk is essential to successful testing. Dale Perry describes a general risk-based framework—applicable to any development lifecycle model—to help you make critical testing decisions earlier and with more confidence. Learn how to focus your testing effort, what elements to test, and how to organize test designs and documentation. Review the fundamentals of risk identification, analysis, and the role testing plays in risk mitigation. Develop an inventory of test objectives to help prioritize your testing and translate them into a concrete strategy for creating tests. Focus your tests on the areas essential to your stakeholders. Execution and assessing test results provide a better understanding of both the effectiveness of your testing and the potential for failure in your software. Take back a proven approach to organize your testing efforts and new ways to add more value to your project and organization.
The Doctor Is In: Diagnosing Test Automation DiseasesTechWell
Este documento presenta una charla sobre el diagnóstico y tratamiento de enfermedades en la automatización de pruebas. La charla cubre temas como problemas comunes en automatización de pruebas, patrones para solucionar dichos problemas y una metodología de diagnóstico similar a la de un médico. El objetivo es ayudar a los asistentes a identificar problemas en sus propios procesos de automatización y aplicar los patrones recomendados para cada caso.
EARS: The Easy Approach to Requirements SyntaxTechWell
One key to specifying effective functional requirements is minimizing misinterpretation and ambiguity. By employing a consistent syntax in your requirements, you can improve readability and help ensure that everyone on the team understands exactly what to develop. John Terzakis provides examples of typical requirements and explains how to improve them using the Easy Approach to Requirements Syntax (EARS). EARS provides a simple yet powerful method of capturing the nuances of functional requirements. John explains that you need to identify two distinct types of requirements. Ubiquitous requirements state a fundamental property of the software that always occurs; non-ubiquitous requirements depend on the occurrence of an event, error condition, state, or option. Learn and practice identifying the correct requirements type and restating those requirements with the corresponding syntax. Join John to find out what’s wrong with the requirements statement—“The software shall warn of low battery”—and how to fix it.
Is Agile the Prescription for the Public Sector’s IT Woes?TechWell
Information technology (IT) projects are notorious for exceeding budget and schedule estimates, and high visibility failures are common. IT projects in the public sector are particularly challenging. State, provincial, and federal governments worldwide have sponsored noteworthy disasters in the past twenty years. As agile methods have evolved, become more mainstream, and demonstrated their value in the private sector in the past decade, they are often cited as a remedy for the public sector’s IT misery. Payson Hall examines the gap between current public sector IT project challenges and the often-suggested agile solution. Payson explores the challenges to effective vendor-delivered public sector agile projects and possible responses to those challenges. He answers the questions: Is agile ready for large public sector projects? Is the public sector ready for agile? Leave with a better understanding of the problems public sector entities and vendors face and ideas for overcoming some of those barriers.
Executives’ Influence on Agile: The Good, the Bad, and the UglyTechWell
The evidence is in—and it's compelling. Well-executed agile practices can shorten software project schedules by 30 percent while cutting defects by 75 percent. However, many organizations struggle with agile adoption. And some of these struggles can be attributed to the executive leadership. In many cases, the "lead, follow, or get out of the way" attitude causes executives to try to lead when they should be following or getting out of the way. Drawing on his experiences with agile adoption at Synacor as it implements agile on an enterprise scale, Steve Davi illustrates how the executives on the ground can help or hurt agile adoption. Steve shares ways to turn those executive wolves into agile enablers as he describes the four critical actions that executives should take to support agile within their organization―define the vision, boundaries, and constraints; gain support and remove impediments; ensure openness and trust; and hold teams accountable.
Innovation Thinking: Evolve and Expand Your CapabilitiesTechWell
Innovation is a word frequently tossed around in organizations today. The standard clichés are do more with less and be creative. Companies want to be innovative but often struggle with how to define, implement, prioritize, and track their innovation efforts. Using the Innovation to Types model, Jennifer Bonine will help you transform your thinking regarding innovation and understand if your team and company goals match their innovation efforts. Learn how to classify your activities as "core" (to the business) or "context" (essential, but non-revenue generating). Once you understand how your innovation activities are related to revenue generating activities, you can better decide how much of your effort should be spent on core or context activities. Take away tools including an Innovation to Types model for classifying innovation, a Core and Context model to classify your activities, and a way to map your innovation initiatives to different contexts.
Getting Your Message Across: Communication Skills for TestersTechWell
Communication is at the heart of our profession. No matter how advanced our testing capabilities are, if we can’t convey our concerns in ways that connect with key members of the project team, our contribution is likely to be ignored. Because we act solely in an advisory capacity, rather than being in command, our power to exert influence is almost entirely based on our communication skills. With people suffering information overload and deluged with emails, it is more important than ever that we craft succinct and effective messages, using a range of communication modalities. Join Thomas McCoy as he draws on techniques from journalism, public relations, professional writing, psychology, and marketing to help you get your message across. Key themes include: non-verbal communication, presentation skills, persuasive writing, influencing skills, graphic communication, and communicating in teams and meetings. We will use a range of hands-on exercises to practice the concepts being discussed.
Whether you are new to testing or looking for a better way to organize your test practices, understanding risk is essential to successful testing. Dale Perry describes a general risk-based framework—applicable to any development lifecycle model—to help you make critical testing decisions earlier and with more confidence. Learn how to focus your testing effort, what elements to test, and how to organize test designs and documentation. Review the fundamentals of risk identification, analysis, and the role testing plays in risk mitigation. Develop an inventory of test objectives to help prioritize your testing and translate them into a concrete strategy for creating tests. Focus your tests on the areas essential to your stakeholders. Execution and assessing test results provide a better understanding of both the effectiveness of your testing and the potential for failure in your software. Take back a proven approach to organize your testing efforts and new ways to add more value to your project and organization.
The Doctor Is In: Diagnosing Test Automation DiseasesTechWell
Este documento presenta una charla sobre el diagnóstico y tratamiento de enfermedades en la automatización de pruebas. La charla cubre temas como problemas comunes en automatización de pruebas, patrones para solucionar dichos problemas y una metodología de diagnóstico similar a la de un médico. El objetivo es ayudar a los asistentes a identificar problemas en sus propios procesos de automatización y aplicar los patrones recomendados para cada caso.
EARS: The Easy Approach to Requirements SyntaxTechWell
One key to specifying effective functional requirements is minimizing misinterpretation and ambiguity. By employing a consistent syntax in your requirements, you can improve readability and help ensure that everyone on the team understands exactly what to develop. John Terzakis provides examples of typical requirements and explains how to improve them using the Easy Approach to Requirements Syntax (EARS). EARS provides a simple yet powerful method of capturing the nuances of functional requirements. John explains that you need to identify two distinct types of requirements. Ubiquitous requirements state a fundamental property of the software that always occurs; non-ubiquitous requirements depend on the occurrence of an event, error condition, state, or option. Learn and practice identifying the correct requirements type and restating those requirements with the corresponding syntax. Join John to find out what’s wrong with the requirements statement—“The software shall warn of low battery”—and how to fix it.
Balancing the Crusty and Old with the Shiny and NewTechWell
In his journeys, Bob Galen has discovered that testing takes on many forms. Some organizations have no automated tests and struggle to run massive manual regression tests within very short iterative releases. Other organizations are going “all in”―writing thousands of acceptance tests in Gherkin. The resulting imbalance in their testing approaches undermines an organization’s efficiency, effectiveness, and delivery nimbleness. Bob shares ideas to bring balance to testing. He explores the choices: manual vs. automated testing, designed and scripted test cases vs. exploratory tests, and thoroughly planned test projects vs. highly iterative reactive ones. Bob describes how to balance traditional test leadership with an iterative and whole team view to add value. And finally, he explores the balance of the gatekeeper vs. leading the collaboration with stakeholders to find the right requirements that solve their problems. Take away a strategic approach to structure your testing and a renewed understanding of how testing fits into a healthy and balanced culture.
Test Automation Strategies for the Agile WorldTechWell
With the adoption of agile practices in many organizations, the test automation landscape has changed. Bob Galen explores current disruptors to traditional automation strategies, and discusses relevant and current adjustments you need to make when developing your automation business case. Open source tools are becoming incredibly viable and beat their commercial equivalents in many ways―not only in cost, but also in functionality, creativity, evolutionary speed, and developer acceptance. Agile methods have fundamentally challenged our traditional automation strategies. Now we must keep up with incremental and emergent systems and architectures and their high rates of change. Bob explores new automation strategies, examining strategies for both greenfield applications and those pesky legacy projects. Learn how to wrap a business case and communication plan around them so you get the support you need. Leave the workshop with a serious game-plan for delivering on the promise of agile test automation.
Docker Containers in the Enterprise DevOps JourneyTechWell
As technology moves from being a cost-center to a revenue generator in nearly every business, technologists are expected to deliver more with fewer resources. DevOps enables this efficiency through improved collaboration between product management, development, release management, quality assurance, information security, and operations. However, Aater Suleman says that the challenge of incorporating DevOps into a business is no small task. Improving this collaboration requires cross-functional technologies that benefit all departments. By this definition, Docker may well be the most important tool in the DevOps toolbox as it allows empowering and permeable interfaces to be built between different departments throughout the DevOps loop. Aater explores both the Dev and Ops tracks of three companies and examines advantages that were achieved using Docker containers. He shows how Docker containers can work in environments from development to production and shares how this effort can be empirically tracked using five key performance indicators.
Playwriting, Imagination, and Agile Software Development … Oh My!TechWell
The document summarizes a presentation by Tania Katan on applying principles from theatre and storytelling to software development and work culture. Katan believes in using imagination and narrative structure to engage audiences and solve problems. She discusses how agile software development and narrative story arcs both involve initiating a project or story, encountering obstacles, getting feedback, iterating, and resolving issues. Katan provides exercises for developing one's point of view, understanding one's audience, dealing with critics, and maintaining momentum through continued practice of storytelling skills.
Business analysts, developers, and testers are sometimes not on the same page when it comes to test automation. When there is no transparency in test cases, execution, coverage, and data, review of automation by all stakeholders is difficult. Making automation scripts easily readable and writable allows stakeholders to better participate. Subodh Parulekar describes how his team dealt with these issues. Learn how they leveraged behavior-driven development (BDD) concepts and put a wrapper around their existing automation framework to make it more user-friendly with the easy to understand Given-When-Then format. Subodh discusses how his team implemented the new approach in four months to automate 700+ test cases. Now, test reports contain the actual Gherkin test step that passed or failed allowing any stakeholder to evaluate the outcome. Learn how stakeholders can rerun a failed test case from the reporting dashboard to determine if the failure is related to a synchronization, environmental, functional, or test data problem.
Build Your Open Source Performance Testing Platform in the CloudTechWell
Proprietary performance testing platforms can be complex, expensive, and difficult to scale. With the right approach, everything from continuous integration, to continuous deployment pipelines, to full-scale production loads can be supported, but a dizzying array of platforms, services, and approaches available in AWS and the open-source community must be navigated to arrive at solutions that work. Join Gopal Brugalette and explore how to build a performance testing platform in the cloud using open source tools. Gopal shares what he has learned from his failures and successes, explains why he's made the technical decisions he did, what he might have done differently, and how to create a roadmap for success. Attendees will gain insights into building a cloud-based performance testing platform using open-source and cloud tools to improve capabilities, increase efficiency, and reduce costs.
With the drive for continuous integration and delivery, the implications and approaches for designing more testable software are receiving substantial discussion and debate. What does testability really mean in practice? How do you take the idea of testability—how easy it is to test software—and put it into action through the different dimensions of designing and testing a real-world product? Nir Szilagyi recognizes that the challenges of difficult-to-test software can transform a testing cycle from a small automation and exploratory effort to a long struggle of test preparation, execution, and debugging. He says testability starts with software design, goes through implementation, and encompasses building modular software, abstraction, simplicity, clear data interface, separation of business logic into self-sustained entities, and more. On the technical side of testability, Nir explores ways quality engineers and leaders can influence testability from early development through deployment. From his experiences Nir shares real-life testability examples which touch on the human process of building software including the relationship between testers and developers.
IoT and Embedded Testing: A Roku Case StudyTechWell
With big hitters like Time Warner and HBO selectively testing Roku releases, testing these little boxes of joy is becoming more of a necessity in the IoT tester’s playbook. Join Rick Faulise as he shares the secrets of testing on a Roku device including how to get into the Roku interface and make it respond to your commands, how to select a broadcast environment for testing, and how to measure streaming performance. Take your IoT testing to the next level by understanding what special types of testing are unique to the Roku and other important considerations to keep in mind as you journey through the Brightscript SDK and Developer program, Telnet command prompts, and jailbreaking/hacking the Roku OS. Rick presents examples of testing on Roku devices and discusses how to decide what to test and in what order to test it. Take away two handouts: 1] how to jailbreak your Roku device, and 2] a comparison and contrast of testing on a Roku box, a Chromecast device, and an Amazon Fire TV stick.
This document provides a summary of security metrics using analogies from Ice Cube's music and movies. It discusses the importance of speed in various stages from detection of a breach to remediation. It also covers quality metrics to measure success rates and avoid mistakes. Coverage metrics ensure monitoring of all potential attack vectors. Charts are provided showing costs of incidents at different stages of the cyber kill chain as well as most common detection tools. The conclusion emphasizes quality of training over just purchasing new tools.
The document summarizes key principles from the theory of variation:
1. Variation exists in all systems and performance will naturally vary over time.
2. Understanding sources of variation allows managers to set appropriate targets and expectations.
3. The majority of variation is caused by the system, not individual performers. Improving the system design and processes can reduce variation.
4. Statistical process control methods help distinguish common from special causes of variation and determine when meaningful changes have occurred.
Meet TransmogrifAI, Open Source AutoML That Powers Einstein PredictionsMatthew Tovbin
Despite huge progress in machine learning over the past decade, building production-ready machine learning systems is still hard. Three years ago when we set out to build machine learning capabilities into the Salesforce platform we learned that building enterprise-scale machine learning systems is even harder.To solve the problems we encountered, we built TransmogrifAI (https://transmogrif.ai) (pronounced trans-mog-ri-phi), an end-to-end automated machine learning library for structured data, that is used in production today to help power our Salesforce Einstein AI platform. This talk highlights key capabilities of TransmogrifAI library and demonstrates them in action on a real-life machine learning application.
NBTC 2014 - Startup Analytics PresentationSean Power
This document provides a guide for startups on various practical considerations for building a startup. It discusses whether the reader has the right attributes to start a startup, finding the right team members, identifying a product idea, legal requirements, prototyping, fundraising, and using analytics to measure progress towards goals. Key aspects of analytics covered include measuring viral growth, infrastructure performance, customer and community sentiment, user engagement over time, funnel abandonment, costs, and determining the minimum resources needed to sustain the business. The guide emphasizes iterating based on analytics to achieve product-market fit.
The document provides guidance on writing internal audit reports. It discusses what an audit report is, its key features and contents. An audit report summarizes an audit's work, findings and recommendations. It follows a four-step process - identifying issues, analyzing root causes, recommending corrections, and obtaining management responses. The document outlines the different sections and details that should be included in an audit report, as well as the process of drafting, finalizing and communicating the report to relevant stakeholders. It also describes the different types of reports and users that may need to be considered when writing internal audit reports.
This document provides guidance on writing internal audit reports. It states that an audit report summarizes the audit work and findings/recommendations. Key aspects of an audit report include disclosing findings, describing findings and their root causes, providing suggestions for corrective actions, and including management's responses. The report should have a clear written expression of observations, risks identified, controls assessed, and suggestions for improvement. There are also guidelines provided around obtaining management comments, reaching agreement on findings, and finalizing the report.
The document provides guidance on writing internal audit reports. It discusses what an audit report is, its purpose, and key components. An audit report summarizes an audit's work, findings, and recommendations. It follows a four-step process: identifying issues, analyzing root causes, recommending corrections, and obtaining management responses. The document outlines the typical structure and contents of an audit report, including disclosing findings, describing findings in detail, and making suggestions. It also discusses report drafting, communication, and distribution processes to ensure reports reach the intended users.
The audit cycle performance consists of planning, preliminary survey, understanding the entity, identifying key areas and objectives, determining audit criteria, collecting evidence, generating working papers, and preparing audit findings. The initial stages involve obtaining general information on the company and identifying important areas to focus on. Later stages are testing evidence, documenting results, and communicating findings to the audited entity in a clear and understandable manner.
Project_CharterProject Title:GO HOME!!Black BeltProject ChampionExecutive SponsorMBB/MentorProblem StatementBusiness CaseProject GoalsProject ScopeHigh Level Project TimelineConstraints & DependenciesProject RisksAdditional InformationPhaseStartFinishIdentify ValueMap the Value StreamCreate FlowEstablish PullSeek PerfectionApproval/Steering CommitteeStakeholders & AdvisorsProject Team & SME'sNameOrganizationNameOrganizationName Organization
VILLANOVA UNIVERSITY
High Level Process MapPlease add in extra processes and symbols as neededGO HOME!!
Icing transfers to Kettle 2
Icing in Kettle 1
Check viscosity
Transfer to final assembly
Final quality check
Shortening Quality Check
Package product
SIPOCS.I.P.O.C. TemplateGO HOME!!SuppliersInputsProcessOutputsCustomers
VILLANOVA UNIVERSITY
Start
Step 1
Step 2
Step 3
Step 4
End
VSMValue Stream MapGO HOME!!SupplierCustomerInformationInformation
&"Arial,Bold"&10Value Stream Map &"Arial,Regular"&8v3.0
&"Arial,Regular"&8&G
Copyright 2016 GoLeanSixSigma.com. All Rights Reserved.
7 min
5 min
8 min
8 min
10 min
3 min
4 min
1 min
min
min
TOTAL
LT
PT
% C&A
Cause_Effect_DiagramGO HOME!!Instructions:STEP 1 : Define the problem. What is the product, process or service that has failed.STEP 2 : Starting with 'Materials' or any other label, ask: is there anything about materials that might contribute to the problem. Record it next to one of the arrows under Materials.STEP 3 : Repeat asking "is there anything about materials that might contribute to the problem"Problem Record each result next to an arrow.STEP 4 : Repeat Step 2 & 3 for each successive category.STEP 5 : Identify the candidates that are the most likely Root CauseSTEP 6 : If further "screening" is necessary, assess the likely Root Causes using the "Impact" and "Implement" matrix, selecting items marked 1, then 2 . . . 4 as priorities.
ParetoCategories# of OccurrencesIssue 125GO HOME!!Issue 220Issue 315Issue 410Issue 55The focus area should be on the water in the pipe followed by contacting the supplier to understand why their materials are out of spec.
5-Why5-WHY ANALYSIS SHEETGO HOME!!Note: Continue on separate page if 5-Whys are not enough to determine root cause.
WHY #1:
Adjusting pH
WHY #5
Control valve not tunned properly.
WHY #2:
Stoichiometry out of wack
WHY #4
Control valve not shutting off quick enough.
WHY #3
Overshooting pH
TEMPORARY Date 6.8.20
COUNTERMEASURES
Tune the valve and shut off in time.
FINAL COUNTERMEASURE Name Johnny Jones
- PERMANENT CORRECTIVE ACTION Date 6.15.20
Smart transmitter to be installed that provides warning when valve tunning is outside of control limits.
VERIFICATION:
No Recurrence in Three Months? TBD Date 6.7.20
Single-Point Lesson? _________ Date ________
DO THE ...
This document provides an overview of the Theory of Constraints (TOC). It discusses key concepts in TOC like constraints, bottlenecks, throughput, inventory, operating expenses, and exploiting constraints. It explains that the goal is to make money now and in the future through sales rather than production. Bottlenecks limit the system and floating bottlenecks are unpredictable and should be avoided. It also introduces TOC tools like the three global measures (throughput, inventory, operating expense), Drum-Buffer-Rope, and how TOC differs from just-in-time approaches.
This document discusses the value of DevOps and monitoring tools in improving collaboration between development and operations teams and justifying investments in automation. It notes that traditionally, dev teams focused on features while ops focused on incidents, but both were measured by vague business metrics like revenue and uptime. New tools can help baseline current problems, measure progress over time, and demonstrate business impact to obtain support for further investments. The document advocates for monitoring the full customer experience rather than individual system components.
The nature of exploration, coupled with the ability of testers to rapidly apply their skills and experience, make exploratory testing a widely used test approach—especially when time is short. Unfortunately, exploratory testing often is dismissed by project managers who assume that it is not reproducible, measurable, or accountable. If you have these concerns, you may find a solution in a technique called session-based test management (SBTM), developed by Jon Bach and his brother James to specifically address these issues. In SBTM, testers are assigned areas of a product to explore, and testing is time boxed in “sessions” that have mission statements called “charters” to create a meaningful and countable unit of work. Jon discusses—and you practice—the skills of exploration using the SBTM approach. He demonstrates a freely available, open source tool to help manage your exploration and prepares you to implement SBTM in your test organization.
This document provides information about ISO 9001 audits and documentation. It discusses the basic responsibilities during an audit, the purpose of documentation and ensuring procedures are followed. It outlines the audit process, including auditor roles and responsibilities, types of audits, and how to prepare for and respond during an audit. Key points emphasized are knowing applicable documentation, following documented procedures, being honest and not guessing during questioning.
Balancing the Crusty and Old with the Shiny and NewTechWell
In his journeys, Bob Galen has discovered that testing takes on many forms. Some organizations have no automated tests and struggle to run massive manual regression tests within very short iterative releases. Other organizations are going “all in”―writing thousands of acceptance tests in Gherkin. The resulting imbalance in their testing approaches undermines an organization’s efficiency, effectiveness, and delivery nimbleness. Bob shares ideas to bring balance to testing. He explores the choices: manual vs. automated testing, designed and scripted test cases vs. exploratory tests, and thoroughly planned test projects vs. highly iterative reactive ones. Bob describes how to balance traditional test leadership with an iterative and whole team view to add value. And finally, he explores the balance of the gatekeeper vs. leading the collaboration with stakeholders to find the right requirements that solve their problems. Take away a strategic approach to structure your testing and a renewed understanding of how testing fits into a healthy and balanced culture.
Test Automation Strategies for the Agile WorldTechWell
With the adoption of agile practices in many organizations, the test automation landscape has changed. Bob Galen explores current disruptors to traditional automation strategies, and discusses relevant and current adjustments you need to make when developing your automation business case. Open source tools are becoming incredibly viable and beat their commercial equivalents in many ways―not only in cost, but also in functionality, creativity, evolutionary speed, and developer acceptance. Agile methods have fundamentally challenged our traditional automation strategies. Now we must keep up with incremental and emergent systems and architectures and their high rates of change. Bob explores new automation strategies, examining strategies for both greenfield applications and those pesky legacy projects. Learn how to wrap a business case and communication plan around them so you get the support you need. Leave the workshop with a serious game-plan for delivering on the promise of agile test automation.
Docker Containers in the Enterprise DevOps JourneyTechWell
As technology moves from being a cost-center to a revenue generator in nearly every business, technologists are expected to deliver more with fewer resources. DevOps enables this efficiency through improved collaboration between product management, development, release management, quality assurance, information security, and operations. However, Aater Suleman says that the challenge of incorporating DevOps into a business is no small task. Improving this collaboration requires cross-functional technologies that benefit all departments. By this definition, Docker may well be the most important tool in the DevOps toolbox as it allows empowering and permeable interfaces to be built between different departments throughout the DevOps loop. Aater explores both the Dev and Ops tracks of three companies and examines advantages that were achieved using Docker containers. He shows how Docker containers can work in environments from development to production and shares how this effort can be empirically tracked using five key performance indicators.
Playwriting, Imagination, and Agile Software Development … Oh My!TechWell
The document summarizes a presentation by Tania Katan on applying principles from theatre and storytelling to software development and work culture. Katan believes in using imagination and narrative structure to engage audiences and solve problems. She discusses how agile software development and narrative story arcs both involve initiating a project or story, encountering obstacles, getting feedback, iterating, and resolving issues. Katan provides exercises for developing one's point of view, understanding one's audience, dealing with critics, and maintaining momentum through continued practice of storytelling skills.
Business analysts, developers, and testers are sometimes not on the same page when it comes to test automation. When there is no transparency in test cases, execution, coverage, and data, review of automation by all stakeholders is difficult. Making automation scripts easily readable and writable allows stakeholders to better participate. Subodh Parulekar describes how his team dealt with these issues. Learn how they leveraged behavior-driven development (BDD) concepts and put a wrapper around their existing automation framework to make it more user-friendly with the easy to understand Given-When-Then format. Subodh discusses how his team implemented the new approach in four months to automate 700+ test cases. Now, test reports contain the actual Gherkin test step that passed or failed allowing any stakeholder to evaluate the outcome. Learn how stakeholders can rerun a failed test case from the reporting dashboard to determine if the failure is related to a synchronization, environmental, functional, or test data problem.
Build Your Open Source Performance Testing Platform in the CloudTechWell
Proprietary performance testing platforms can be complex, expensive, and difficult to scale. With the right approach, everything from continuous integration, to continuous deployment pipelines, to full-scale production loads can be supported, but a dizzying array of platforms, services, and approaches available in AWS and the open-source community must be navigated to arrive at solutions that work. Join Gopal Brugalette and explore how to build a performance testing platform in the cloud using open source tools. Gopal shares what he has learned from his failures and successes, explains why he's made the technical decisions he did, what he might have done differently, and how to create a roadmap for success. Attendees will gain insights into building a cloud-based performance testing platform using open-source and cloud tools to improve capabilities, increase efficiency, and reduce costs.
With the drive for continuous integration and delivery, the implications and approaches for designing more testable software are receiving substantial discussion and debate. What does testability really mean in practice? How do you take the idea of testability—how easy it is to test software—and put it into action through the different dimensions of designing and testing a real-world product? Nir Szilagyi recognizes that the challenges of difficult-to-test software can transform a testing cycle from a small automation and exploratory effort to a long struggle of test preparation, execution, and debugging. He says testability starts with software design, goes through implementation, and encompasses building modular software, abstraction, simplicity, clear data interface, separation of business logic into self-sustained entities, and more. On the technical side of testability, Nir explores ways quality engineers and leaders can influence testability from early development through deployment. From his experiences Nir shares real-life testability examples which touch on the human process of building software including the relationship between testers and developers.
IoT and Embedded Testing: A Roku Case StudyTechWell
With big hitters like Time Warner and HBO selectively testing Roku releases, testing these little boxes of joy is becoming more of a necessity in the IoT tester’s playbook. Join Rick Faulise as he shares the secrets of testing on a Roku device including how to get into the Roku interface and make it respond to your commands, how to select a broadcast environment for testing, and how to measure streaming performance. Take your IoT testing to the next level by understanding what special types of testing are unique to the Roku and other important considerations to keep in mind as you journey through the Brightscript SDK and Developer program, Telnet command prompts, and jailbreaking/hacking the Roku OS. Rick presents examples of testing on Roku devices and discusses how to decide what to test and in what order to test it. Take away two handouts: 1] how to jailbreak your Roku device, and 2] a comparison and contrast of testing on a Roku box, a Chromecast device, and an Amazon Fire TV stick.
This document provides a summary of security metrics using analogies from Ice Cube's music and movies. It discusses the importance of speed in various stages from detection of a breach to remediation. It also covers quality metrics to measure success rates and avoid mistakes. Coverage metrics ensure monitoring of all potential attack vectors. Charts are provided showing costs of incidents at different stages of the cyber kill chain as well as most common detection tools. The conclusion emphasizes quality of training over just purchasing new tools.
The document summarizes key principles from the theory of variation:
1. Variation exists in all systems and performance will naturally vary over time.
2. Understanding sources of variation allows managers to set appropriate targets and expectations.
3. The majority of variation is caused by the system, not individual performers. Improving the system design and processes can reduce variation.
4. Statistical process control methods help distinguish common from special causes of variation and determine when meaningful changes have occurred.
Meet TransmogrifAI, Open Source AutoML That Powers Einstein PredictionsMatthew Tovbin
Despite huge progress in machine learning over the past decade, building production-ready machine learning systems is still hard. Three years ago when we set out to build machine learning capabilities into the Salesforce platform we learned that building enterprise-scale machine learning systems is even harder.To solve the problems we encountered, we built TransmogrifAI (https://transmogrif.ai) (pronounced trans-mog-ri-phi), an end-to-end automated machine learning library for structured data, that is used in production today to help power our Salesforce Einstein AI platform. This talk highlights key capabilities of TransmogrifAI library and demonstrates them in action on a real-life machine learning application.
NBTC 2014 - Startup Analytics PresentationSean Power
This document provides a guide for startups on various practical considerations for building a startup. It discusses whether the reader has the right attributes to start a startup, finding the right team members, identifying a product idea, legal requirements, prototyping, fundraising, and using analytics to measure progress towards goals. Key aspects of analytics covered include measuring viral growth, infrastructure performance, customer and community sentiment, user engagement over time, funnel abandonment, costs, and determining the minimum resources needed to sustain the business. The guide emphasizes iterating based on analytics to achieve product-market fit.
The document provides guidance on writing internal audit reports. It discusses what an audit report is, its key features and contents. An audit report summarizes an audit's work, findings and recommendations. It follows a four-step process - identifying issues, analyzing root causes, recommending corrections, and obtaining management responses. The document outlines the different sections and details that should be included in an audit report, as well as the process of drafting, finalizing and communicating the report to relevant stakeholders. It also describes the different types of reports and users that may need to be considered when writing internal audit reports.
This document provides guidance on writing internal audit reports. It states that an audit report summarizes the audit work and findings/recommendations. Key aspects of an audit report include disclosing findings, describing findings and their root causes, providing suggestions for corrective actions, and including management's responses. The report should have a clear written expression of observations, risks identified, controls assessed, and suggestions for improvement. There are also guidelines provided around obtaining management comments, reaching agreement on findings, and finalizing the report.
The document provides guidance on writing internal audit reports. It discusses what an audit report is, its purpose, and key components. An audit report summarizes an audit's work, findings, and recommendations. It follows a four-step process: identifying issues, analyzing root causes, recommending corrections, and obtaining management responses. The document outlines the typical structure and contents of an audit report, including disclosing findings, describing findings in detail, and making suggestions. It also discusses report drafting, communication, and distribution processes to ensure reports reach the intended users.
The audit cycle performance consists of planning, preliminary survey, understanding the entity, identifying key areas and objectives, determining audit criteria, collecting evidence, generating working papers, and preparing audit findings. The initial stages involve obtaining general information on the company and identifying important areas to focus on. Later stages are testing evidence, documenting results, and communicating findings to the audited entity in a clear and understandable manner.
Project_CharterProject Title:GO HOME!!Black BeltProject ChampionExecutive SponsorMBB/MentorProblem StatementBusiness CaseProject GoalsProject ScopeHigh Level Project TimelineConstraints & DependenciesProject RisksAdditional InformationPhaseStartFinishIdentify ValueMap the Value StreamCreate FlowEstablish PullSeek PerfectionApproval/Steering CommitteeStakeholders & AdvisorsProject Team & SME'sNameOrganizationNameOrganizationName Organization
VILLANOVA UNIVERSITY
High Level Process MapPlease add in extra processes and symbols as neededGO HOME!!
Icing transfers to Kettle 2
Icing in Kettle 1
Check viscosity
Transfer to final assembly
Final quality check
Shortening Quality Check
Package product
SIPOCS.I.P.O.C. TemplateGO HOME!!SuppliersInputsProcessOutputsCustomers
VILLANOVA UNIVERSITY
Start
Step 1
Step 2
Step 3
Step 4
End
VSMValue Stream MapGO HOME!!SupplierCustomerInformationInformation
&"Arial,Bold"&10Value Stream Map &"Arial,Regular"&8v3.0
&"Arial,Regular"&8&G
Copyright 2016 GoLeanSixSigma.com. All Rights Reserved.
7 min
5 min
8 min
8 min
10 min
3 min
4 min
1 min
min
min
TOTAL
LT
PT
% C&A
Cause_Effect_DiagramGO HOME!!Instructions:STEP 1 : Define the problem. What is the product, process or service that has failed.STEP 2 : Starting with 'Materials' or any other label, ask: is there anything about materials that might contribute to the problem. Record it next to one of the arrows under Materials.STEP 3 : Repeat asking "is there anything about materials that might contribute to the problem"Problem Record each result next to an arrow.STEP 4 : Repeat Step 2 & 3 for each successive category.STEP 5 : Identify the candidates that are the most likely Root CauseSTEP 6 : If further "screening" is necessary, assess the likely Root Causes using the "Impact" and "Implement" matrix, selecting items marked 1, then 2 . . . 4 as priorities.
ParetoCategories# of OccurrencesIssue 125GO HOME!!Issue 220Issue 315Issue 410Issue 55The focus area should be on the water in the pipe followed by contacting the supplier to understand why their materials are out of spec.
5-Why5-WHY ANALYSIS SHEETGO HOME!!Note: Continue on separate page if 5-Whys are not enough to determine root cause.
WHY #1:
Adjusting pH
WHY #5
Control valve not tunned properly.
WHY #2:
Stoichiometry out of wack
WHY #4
Control valve not shutting off quick enough.
WHY #3
Overshooting pH
TEMPORARY Date 6.8.20
COUNTERMEASURES
Tune the valve and shut off in time.
FINAL COUNTERMEASURE Name Johnny Jones
- PERMANENT CORRECTIVE ACTION Date 6.15.20
Smart transmitter to be installed that provides warning when valve tunning is outside of control limits.
VERIFICATION:
No Recurrence in Three Months? TBD Date 6.7.20
Single-Point Lesson? _________ Date ________
DO THE ...
This document provides an overview of the Theory of Constraints (TOC). It discusses key concepts in TOC like constraints, bottlenecks, throughput, inventory, operating expenses, and exploiting constraints. It explains that the goal is to make money now and in the future through sales rather than production. Bottlenecks limit the system and floating bottlenecks are unpredictable and should be avoided. It also introduces TOC tools like the three global measures (throughput, inventory, operating expense), Drum-Buffer-Rope, and how TOC differs from just-in-time approaches.
This document discusses the value of DevOps and monitoring tools in improving collaboration between development and operations teams and justifying investments in automation. It notes that traditionally, dev teams focused on features while ops focused on incidents, but both were measured by vague business metrics like revenue and uptime. New tools can help baseline current problems, measure progress over time, and demonstrate business impact to obtain support for further investments. The document advocates for monitoring the full customer experience rather than individual system components.
The nature of exploration, coupled with the ability of testers to rapidly apply their skills and experience, make exploratory testing a widely used test approach—especially when time is short. Unfortunately, exploratory testing often is dismissed by project managers who assume that it is not reproducible, measurable, or accountable. If you have these concerns, you may find a solution in a technique called session-based test management (SBTM), developed by Jon Bach and his brother James to specifically address these issues. In SBTM, testers are assigned areas of a product to explore, and testing is time boxed in “sessions” that have mission statements called “charters” to create a meaningful and countable unit of work. Jon discusses—and you practice—the skills of exploration using the SBTM approach. He demonstrates a freely available, open source tool to help manage your exploration and prepares you to implement SBTM in your test organization.
This document provides information about ISO 9001 audits and documentation. It discusses the basic responsibilities during an audit, the purpose of documentation and ensuring procedures are followed. It outlines the audit process, including auditor roles and responsibilities, types of audits, and how to prepare for and respond during an audit. Key points emphasized are knowing applicable documentation, following documented procedures, being honest and not guessing during questioning.
5 Steps to Getting Organizational Buy-In for Your Enterprise Software ProjectJeff Carr
When looking for a new enterprise software system, your organization must begin your journey by making a business case for enterprise software. This involves numerous steps, including determining an expected return on investment, anticipating total costs, and fully documenting the business case for investing in a new or upgraded system.
Find out how manufacturing and distribution companies can drive an effective justification process with this presentation deck.
Explore five critical areas needed to justify one of the most complex and resource-intensive initiatives your company will face:
- Assess your internal environment
- Fully document your current state
- Clearly map your future state
- Get a handle on total costs of upgrade or new enterprise system
- Accurately calculate ROI
How to Own a Really Big Complex Product v3Mike Cottmeyer
The document discusses how to effectively manage product ownership for large, complex products developed by multiple teams. It notes that having a single product owner does not scale in these situations. Instead, it advocates developing organizational capabilities around business analysis, engineering, and leadership/coordination. These capabilities can then be applied differently depending on the organizational level, such as across individual teams, projects, or the entire product portfolio. The key is creating situation-specific strategies that recognize it takes collaboration between teams to deliver value from complex products.
The document discusses how to effectively manage product ownership of large, complex products across multiple teams. It describes how the responsibilities of a product owner scale up for enterprise-level products involving many interdependent components. These responsibilities require developing organizational capabilities for business analysis, engineering coordination, and leadership across teams and projects rather than relying on a single product owner role. The document provides examples and strategies for defining features and coordinating roadmaps across teams to deliver value over time for complex enterprise products.
How to own a really big complex product v3Mike Cottmeyer
The document discusses how to effectively manage the role of a product owner for large, complex products developed by multiple teams. It describes how the product owner role expands beyond a single person to encompass business analysis, engineering, and leadership capabilities across teams, projects, and the overall product portfolio. Different expressions of product ownership are explored, including scrum of scrums, product owner teams, integration teams, and lean/kanban approaches.
This document provides information about ISO 9001 audits. It discusses the basic responsibilities during an audit, such as knowing job duties and documentation. It describes the documentation structure and typical types of documentation like policies, procedures, instructions, forms and records. It explains what an auditor's role is and that they will verify that documentation matches actual job performance. The document emphasizes that all employees are subject to audit and that audits are meant to provide assurance of compliance to standards.
Isabel Evans stopped drawing and painting after being told she was not very good at it, which led to a loss of confidence in her creative and professional abilities. However, she realized that attempting creative activities is important for cognitive and emotional development, and that making mistakes and learning from failures allows for growth. By reengaging with failure through art and with support from others, Isabel was able to regain confidence in her abilities and reboot her career. The document discusses different perspectives on failure and the importance of learning from mistakes.
Instill a DevOps Testing Culture in Your Team and Organization TechWell
The DevOps movement is here. Companies across many industries are breaking down siloed IT departments and federating them into product development teams. Testing and its practices are at the heart of these changes. Traditionally, IT organizations have been staffed with mostly manual testers and a limited number of automation and performance engineers. To keep pace with development in the new “you build it, you own it” environment, testing teams and individuals must develop new technical skills and even embrace coding to stay relevant and add greater value to the business. DevOps really starts with testing. Join Adam Auerbach as he explains what DevOps is and how it relates to testing. He describes how testing must change from top to bottom and how to access your own environment to identify improvement opportunities. Adam dives into practices like service virtualization, test data management, and continuous testing so you can understand where you are now and identify steps needed to instill a DevOps testing culture in your team and organization.
Test Design for Fully Automated Build ArchitectureTechWell
This document summarizes a half-day tutorial on test design for fully automated build architectures presented by Melissa Benua of mParticle at STAREAST 2018. The tutorial covered guiding principles for test design including prioritizing important and reliable tests, structuring automated pipelines around components, packages, and releases, and monitoring test results through code coverage, flaky test handling, and logging versus counters. It also included exercises mapping test cases to functional boundaries and categories of tests to pipeline stages.
System-Level Test Automation: Ensuring a Good StartTechWell
Many organizations invest a lot of effort in test automation at the system level but then have serious problems later on. As a leader, how can you ensure that your new automation efforts will get off to a good start? What can you do to ensure that your automation work provides continuing value? This tutorial covers both “theory” and “practice”. Dot Graham explains the critical issues for getting a good start, and Chris Loder describes his experiences in getting good automation started at a number of companies. The tutorial covers the most important management issues you must address for test automation success, particularly when you are new to automation, and how to choose the best approaches for your organization—no matter which automation tools you use. Focusing on system level testing, Dot and Chris explain how automation affects staffing, who should be responsible for which automation tasks, how managers can best support automation efforts to promote success, what you can realistically expect in benefits and how to report them. They explain—for non-techies—the key technical issues that can make or break your automation effort. Come away with your own clarified automation objectives, and a draft test automation strategy to use to plan your own system-level test automation.
Build Your Mobile App Quality and Test StrategyTechWell
Let’s build a mobile app quality and testing strategy together. Whether you have a web, hybrid, or native app, building a quality and testing strategy means (1) knowing what data and tools you have available to make agile decisions, (2) understanding your customers and your competitors, and (3) testing your app under real-world conditions. Jason Arbon guides you through the latest techniques, data, and tools to ensure the awesomeness of your mobile app quality and testing strategy. Leave this interactive session with a strategy for your very own app—or one you pretend to own. The information Jason shares is based on data from Appdiff’s next-gen mobile app testing platform, lessons from Applause/uTest’s crowd, text mining hundreds of millions of app store reviews, and in-depth discussions with top mobile app development teams.
Testing Transformation: The Art and Science for SuccessTechWell
Technologies, testing processes, and the role of the tester have evolved significantly in the past few years with the advent of agile, DevOps, and other new technologies. It is critical that we testing professionals evaluate ourselves and continue to add tangible value to our organizations. In your work, are you focused on the trivial or on real game changers? Jennifer Bonine describes critical elements that help you artfully blend people, process, and technology to create a synergistic relationship that adds value. Jennifer shares ideas on mastering politics, maneuvering core vs. context, and innovating your technology strategies and processes. She explores how new processes can be introduced in an organization, what the role of organizational culture is in determining the success of a project, and how you can know what tools will add value vs. simply adding overhead and complexity. Jennifer reviews critically needed tester skills and discusses a continual learning model to evolve your skills and stay relevant. This discussion can lead you to technologies, processes, and skills you can stake your career on.
We’ve all been there. We work incredibly hard to develop a feature and design tests based on written requirements. We build a detailed test plan that aligns the tests with the software and the documented business needs. And when we put the tests to the software, it all falls apart because the requirements were changed without informing everyone. Mary Thorn says help is at hand. Enter behavior-driven development (BDD), and Cucumber and SpecFlow, tools for running automated acceptance tests and facilitating BDD. Mary explores the nuances of Cucumber and SpecFlow, and shows you how to implement BDD and agile acceptance testing. By fostering collaboration for implementing active requirements via a common language and format, Cucumber and SpecFlow bridge the communication gap between business stakeholders and implementation teams. In this workshop, practice writing feature files with the best practices Mary has discovered over numerous implementations. If you experience developers not coding to requirements, testers not getting requirements updates, or customers who feel out of the loop and don’t get what they ask for, Mary has answers for you.
Develop WebDriver Automated Tests—and Keep Your SanityTechWell
Many teams go crazy because of brittle, high-maintenance automated test suites. Jim Holmes helps you understand how to create a flexible, maintainable, high-value suite of functional tests using Selenium WebDriver. Learn the basics of what to test, what not to test, and how to avoid overlapping with other types of testing. Jim includes both philosophical concepts and hands-on coding. Testers who haven't written code should not be intimidated! We'll pair you up to make sure you're successful. Learn to create practical tests dealing with advanced situations such as input validation, AJAX delays, and working with file downloads. Additionally, discover when you need to work together with developers to create a system that's more easily testable. This tutorial focuses primarily on automating web tests, but many of the same concepts can be applied to other UI environments. Demos and labs will be in C# and Java using WebDriver. Leave this tutorial having learned how to write high-value WebDriver tests—and stay sane while doing so.
DevOps is a cultural shift aimed at streamlining intergroup communication and improving operational efficiency for development and operations groups. Over time, inclusion of other IT groups under the DevOps umbrella has become the norm for many organizations. But even broadening the boundaries of DevOps, the conversation has been largely devoid of the business units’ place at the table. A common mistake organizations make while going through the DevOps transformation is drawing a line at the IT boundary. If that occurs, a larger, more inclusive silo within the organization is created, operating in an informational vacuum and causing operational inefficiency and goal misalignment. Sharing his experiences working on both sides of the fence, Leon Fayer describes the importance of including business units in order to align technology decisions with business goals. Leon discusses inclusion of business units in existing agile processes, benefits of cross-departmental monitoring, and a business-first approach to technology decisions.
Eliminate Cloud Waste with a Holistic DevOps StrategyTechWell
Chris Parlette maintains that renting infrastructure on demand is the most disruptive trend in IT in decades. In 2016, enterprises spent $23B on public cloud IaaS services. By 2020, that figure is expected to reach $65B. The public cloud is now used like a utility, and like any utility, there is waste. Who's responsible for optimizing the infrastructure and reducing wasted expenses? It’s DevOps. The excess expense, known as cloud waste, comprises several interrelated problems: services running when they don't need to be, improperly sized infrastructure, orphaned resources, and shadow IT. There are a few core tenets of DevOps—holistic thinking, no silos, rapid useful feedback, and automation—that can be applied to reducing your cloud waste. Join Chris to learn why you should include continuous cost optimization in your DevOps processes. Automate cost control, reduce your cloud expenses, and make your life easier.
Transform Test Organizations for the New World of DevOpsTechWell
With the recent emergence of DevOps across the industry, testing organizations are being challenged to transform themselves significantly within a short period of time to stay meaningful within their organizations. It’s not easy to plan and approach these changes considering the way testing organizations have remained structured for ages. These challenges start from foundational organizational structures and can cut across leadership influence, competencies, tools strategy, infrastructure, and other dimensions. Sumit Kumar shares his experience assisting various organizations to overcome these challenges using an organized DevOps enablement framework. The framework includes radical restructuring, turning the tools strategy upside down, a multidimensional workforce enablement supported by infrastructure changes, redeveloped collaborations models, and more. From his real world experiences Sumit shares tips for approaching this journey and explains the roadmap for testing organizations to transform themselves to lead the quality in DevOps.
The Fourth Constraint in Project Delivery—LeadershipTechWell
All too often, the triple constraints—time, cost, and quality—are bandied about as if they are the be-all, end-all. While they are important, leadership—the fourth and larger underpinning constraint—influences the first three. Statistics on project success and failure abound, and these measurements are usually taken against the triple constraints. According to the Project Management Institute, only 53 percent of projects are completed within budget, and only 49 percent are completed on time. If so many projects overrun budget and are late, we can’t really say, “Good, fast, or cheap—pick two.” Rob Burkett talks about leadership at every level of a team. He shares his insights and stories gleaned from his years of IT and project management experience. Rob speaks to some of the glaring difficulties in the workplace in general and some specifically related to IT delivery and project management. Leave with a clearer understanding of how to communicate with teams and team members, and gain a better understanding of how you can be a leader—up and down your organization.
Resolve the Contradiction of Specialists within Agile TeamsTechWell
As teams grow, organizations often draw a distinction between feature teams, which deliver the visible business value to the user, and component teams, which manage shared work. Steve Berczuk says that this distinction can help organizations be more productive and scale effectively, but he recognizes that not all shared work fits into this model. Some work is best handled by “specialists,” that is people with unique skills. Although teams composed entirely of T-shaped people is ideal, certain skills are hard to come by and are used irregularly across an organization. Since these specialists often need to work closely with teams, rather than working from their own backlog, they don’t fit into the component team model. The use of shared resources presents challenges to the agile planning model. Steve Berczuk shares how teams such as those providing infrastructure services and specialists can fit into a feature+component team model, and how variations such as embedding specialists in a scrum team can both present process challenges and add significant value to both the team and the larger organization.
Pin the Tail on the Metric: A Field-Tested Agile GameTechWell
Metrics don’t have to be a necessary evil. If done right, metrics can help guide us to make better forward-looking decisions, rather than being used for simply managing or monitoring. They can help us identify trade-offs between options for what to do next versus punitive or worse, purely managerial measures. Steve Martin won’t be giving the Top Ten List of field-tested metrics you should use. Instead, in this interactive mini-workshop, he leads you through the critical thinking necessary for you to determine what is right for you to measure. First, Steve explores why you want to measure something—whether it’s for a team, a portfolio, or even an agile transformation. Next, he provides multiple real-life metrics examples to help drive home concepts behind characteristics of good and bad metrics. Finally, Steve shows how to run his field-tested agile game—Pin the Tail on the Metric. Take back this activity to help you guide metrics conversations at your organization.
Agile Performance Holarchy (APH)—A Model for Scaling Agile TeamsTechWell
A hierarchy is an organizational network that has a top and a bottom, and where position is determined by rank, importance, and value. A holarchy is a network that has no top or bottom and where each person’s value derives from his ability, rather than position. As more companies seek the benefits of agile, leaders need to build and sustain delivery capability while scaling agile without introducing unnecessary process and overhead. The Agile Performance Holarchy (APH) is an empirical model for scaling and sustaining agility while continuing to deliver great products. Jeff Dalton designed the APH by drawing from lessons learned observing and assessing hundreds of agile companies and teams. The APH helps implement a holarchy—a system composed of interacting organizational units called holons—centered on a series of performance circles that embody the behaviors of high performing agile organizations. Jeff describes how APH provides guidelines in the areas of leadership, values, teaming, visioning, governing, building, supporting, and engaging within an all-agile organization. Join Jeff to see what the APH is all about and how you can use it in your team and organization.
A Business-First Approach to DevOps ImplementationTechWell
DevOps is a cultural shift aimed at streamlining intergroup communication and improving operational efficiency for development and operations groups. Over time, inclusion of other IT groups under the DevOps umbrella has become the norm for many organizations. But even broadening the boundaries of DevOps, the conversation has been largely devoid of the business units’ place at the table. A common mistake organizations make while going through the DevOps transformation is drawing a line at the IT boundary. If that occurs, a larger, more inclusive silo within the organization is created, operating in an informational vacuum and causing operational inefficiency and goal misalignment. Sharing his experiences working on both sides of the fence, Leon Fayer describes the importance of including business units in order to align technology decisions with business goals. Leon discusses inclusion of business units in existing agile processes, benefits of cross-departmental monitoring, and a business-first approach to technology decisions.
Databases in a Continuous Integration/Delivery ProcessTechWell
The document summarizes a presentation about including databases in a continuous integration/delivery process. It discusses treating database code like application code by placing it under version control and integrating databases into the DevOps software development pipeline. This allows databases to be built, tested, and released like other software through continuous integration, delivery, and deployment.
Mobile Testing: What—and What Not—to AutomateTechWell
Organizations are moving rapidly into mobile technology, which has significantly increased the demand for testing of mobile applications. David Dangs says testers naturally are turning to automation to help ease the workload, increase potential test coverage, and improve testing efficiency. But should you try to automate all things mobile? Unfortunately, the answer is not always clear. Mobile has its own set of complications, compounded by a wide variety of devices and OS platforms. Join David to learn what mobile testing activities are ripe for automation—and those items best left to manual efforts. He describes the various considerations for automating each type of mobile application: mobile web, native app, and hybrid applications. David also covers device-level testing, types of testing, available automation tools, and recommendations for automation effectiveness. Finally, based on his years of mobile testing experience, David provides some tips and tricks to approach mobile automation. Leave with a clear plan for automating your mobile applications.
Cultural Intelligence: A Key Skill for SuccessTechWell
Diversity is becoming the norm in everyday life. However, introducing global delivery models without a proper understanding of intercultural differences can lead to difficulty, frustration, and reduced productivity. Priyanka Sharma and Thena Barry say that in our diverse world, we need teams with people who can cross these boundaries, communicate effectively, and build the diverse networks necessary to avoid problems. We need to learn about cultural intelligence (CI) and cultural quotient (CQ). CI is the ability to relate and work effectively across cultures. CQ is the cognitive, motivational, and behavioral capacity to understand and respond to beliefs, values, attitudes, and behaviors of individuals and groups. Together, CI and CQ can help us build behavioral capacities that aid motivation, behavior, and productivity in teams as well as individuals. Priyanka and Thena show how to build a more culturally intelligent place with tools and techniques from Leading with Cultural Intelligence, as well as content from the Hofstede cultural model. In addition, they illustrate the model with real-life experiences and demonstrate how they adapted in similar circumstances.
Turn the Lights On: A Power Utility Company's Agile TransformationTechWell
Why would a century-old utility with no direct competitors take on the challenge of transforming its entire IT application organization to an agile methodology? In an increasingly interconnected world, the expectations of customers continue to evolve. From smart meters to smart phones, IoT is creating a crisis point for industries not accustomed to rapid change. Glen Morris explains that pizzas can be tracked by the minute and packages at every stop, and customers now expect this same customer service model should exist for all industries—including power. Glen examines how to create momentum and transform non-IT-focused industries to an agile model. If you are struggling with gaining traction in your pursuit of agile within your business, Glen gives you concrete, practical experiences to leverage in your pursuit. Finally, he communicates how to gain buy-in from business partners who have no idea or concern about agile or its methodologies. If your business partners look at you with amusement when you mention the need for a dedicated Product Owner, join Glen as he walks you through the approaches to overcoming agile skepticism.
Infrastructure Challenges in Scaling RAG with Custom AI modelsZilliz
Building Retrieval-Augmented Generation (RAG) systems with open-source and custom AI models is a complex task. This talk explores the challenges in productionizing RAG systems, including retrieval performance, response synthesis, and evaluation. We’ll discuss how to leverage open-source models like text embeddings, language models, and custom fine-tuned models to enhance RAG performance. Additionally, we’ll cover how BentoML can help orchestrate and scale these AI components efficiently, ensuring seamless deployment and management of RAG systems in the cloud.
Removing Uninteresting Bytes in Software FuzzingAftab Hussain
Imagine a world where software fuzzing, the process of mutating bytes in test seeds to uncover hidden and erroneous program behaviors, becomes faster and more effective. A lot depends on the initial seeds, which can significantly dictate the trajectory of a fuzzing campaign, particularly in terms of how long it takes to uncover interesting behaviour in your code. We introduce DIAR, a technique designed to speedup fuzzing campaigns by pinpointing and eliminating those uninteresting bytes in the seeds. Picture this: instead of wasting valuable resources on meaningless mutations in large, bloated seeds, DIAR removes the unnecessary bytes, streamlining the entire process.
In this work, we equipped AFL, a popular fuzzer, with DIAR and examined two critical Linux libraries -- Libxml's xmllint, a tool for parsing xml documents, and Binutil's readelf, an essential debugging and security analysis command-line tool used to display detailed information about ELF (Executable and Linkable Format). Our preliminary results show that AFL+DIAR does not only discover new paths more quickly but also achieves higher coverage overall. This work thus showcases how starting with lean and optimized seeds can lead to faster, more comprehensive fuzzing campaigns -- and DIAR helps you find such seeds.
- These are slides of the talk given at IEEE International Conference on Software Testing Verification and Validation Workshop, ICSTW 2022.
AI 101: An Introduction to the Basics and Impact of Artificial IntelligenceIndexBug
Imagine a world where machines not only perform tasks but also learn, adapt, and make decisions. This is the promise of Artificial Intelligence (AI), a technology that's not just enhancing our lives but revolutionizing entire industries.
OpenID AuthZEN Interop Read Out - AuthorizationDavid Brossard
During Identiverse 2024 and EIC 2024, members of the OpenID AuthZEN WG got together and demoed their authorization endpoints conforming to the AuthZEN API
HCL Notes and Domino License Cost Reduction in the World of DLAUpanagenda
Webinar Recording: https://www.panagenda.com/webinars/hcl-notes-and-domino-license-cost-reduction-in-the-world-of-dlau/
The introduction of DLAU and the CCB & CCX licensing model caused quite a stir in the HCL community. As a Notes and Domino customer, you may have faced challenges with unexpected user counts and license costs. You probably have questions on how this new licensing approach works and how to benefit from it. Most importantly, you likely have budget constraints and want to save money where possible. Don’t worry, we can help with all of this!
We’ll show you how to fix common misconfigurations that cause higher-than-expected user counts, and how to identify accounts which you can deactivate to save money. There are also frequent patterns that can cause unnecessary cost, like using a person document instead of a mail-in for shared mailboxes. We’ll provide examples and solutions for those as well. And naturally we’ll explain the new licensing model.
Join HCL Ambassador Marc Thomas in this webinar with a special guest appearance from Franz Walder. It will give you the tools and know-how to stay on top of what is going on with Domino licensing. You will be able lower your cost through an optimized configuration and keep it low going forward.
These topics will be covered
- Reducing license cost by finding and fixing misconfigurations and superfluous accounts
- How do CCB and CCX licenses really work?
- Understanding the DLAU tool and how to best utilize it
- Tips for common problem areas, like team mailboxes, functional/test users, etc
- Practical examples and best practices to implement right away
Fueling AI with Great Data with Airbyte WebinarZilliz
This talk will focus on how to collect data from a variety of sources, leveraging this data for RAG and other GenAI use cases, and finally charting your course to productionalization.
“An Outlook of the Ongoing and Future Relationship between Blockchain Technologies and Process-aware Information Systems.” Invited talk at the joint workshop on Blockchain for Information Systems (BC4IS) and Blockchain for Trusted Data Sharing (B4TDS), co-located with with the 36th International Conference on Advanced Information Systems Engineering (CAiSE), 3 June 2024, Limassol, Cyprus.
Threats to mobile devices are more prevalent and increasing in scope and complexity. Users of mobile devices desire to take full advantage of the features
available on those devices, but many of the features provide convenience and capability but sacrifice security. This best practices guide outlines steps the users can take to better protect personal devices and information.
Ivanti’s Patch Tuesday breakdown goes beyond patching your applications and brings you the intelligence and guidance needed to prioritize where to focus your attention first. Catch early analysis on our Ivanti blog, then join industry expert Chris Goettl for the Patch Tuesday Webinar Event. There we’ll do a deep dive into each of the bulletins and give guidance on the risks associated with the newly-identified vulnerabilities.
Ocean lotus Threat actors project by John Sitima 2024 (1).pptxSitimaJohn
Ocean Lotus cyber threat actors represent a sophisticated, persistent, and politically motivated group that poses a significant risk to organizations and individuals in the Southeast Asian region. Their continuous evolution and adaptability underscore the need for robust cybersecurity measures and international cooperation to identify and mitigate the threats posed by such advanced persistent threat groups.
Your One-Stop Shop for Python Success: Top 10 US Python Development Providersakankshawande
Simplify your search for a reliable Python development partner! This list presents the top 10 trusted US providers offering comprehensive Python development services, ensuring your project's success from conception to completion.
Monitoring and Managing Anomaly Detection on OpenShift.pdfTosin Akinosho
Monitoring and Managing Anomaly Detection on OpenShift
Overview
Dive into the world of anomaly detection on edge devices with our comprehensive hands-on tutorial. This SlideShare presentation will guide you through the entire process, from data collection and model training to edge deployment and real-time monitoring. Perfect for those looking to implement robust anomaly detection systems on resource-constrained IoT/edge devices.
Key Topics Covered
1. Introduction to Anomaly Detection
- Understand the fundamentals of anomaly detection and its importance in identifying unusual behavior or failures in systems.
2. Understanding Edge (IoT)
- Learn about edge computing and IoT, and how they enable real-time data processing and decision-making at the source.
3. What is ArgoCD?
- Discover ArgoCD, a declarative, GitOps continuous delivery tool for Kubernetes, and its role in deploying applications on edge devices.
4. Deployment Using ArgoCD for Edge Devices
- Step-by-step guide on deploying anomaly detection models on edge devices using ArgoCD.
5. Introduction to Apache Kafka and S3
- Explore Apache Kafka for real-time data streaming and Amazon S3 for scalable storage solutions.
6. Viewing Kafka Messages in the Data Lake
- Learn how to view and analyze Kafka messages stored in a data lake for better insights.
7. What is Prometheus?
- Get to know Prometheus, an open-source monitoring and alerting toolkit, and its application in monitoring edge devices.
8. Monitoring Application Metrics with Prometheus
- Detailed instructions on setting up Prometheus to monitor the performance and health of your anomaly detection system.
9. What is Camel K?
- Introduction to Camel K, a lightweight integration framework built on Apache Camel, designed for Kubernetes.
10. Configuring Camel K Integrations for Data Pipelines
- Learn how to configure Camel K for seamless data pipeline integrations in your anomaly detection workflow.
11. What is a Jupyter Notebook?
- Overview of Jupyter Notebooks, an open-source web application for creating and sharing documents with live code, equations, visualizations, and narrative text.
12. Jupyter Notebooks with Code Examples
- Hands-on examples and code snippets in Jupyter Notebooks to help you implement and test anomaly detection models.
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slackshyamraj55
Discover the seamless integration of RPA (Robotic Process Automation), COMPOSER, and APM with AWS IDP enhanced with Slack notifications. Explore how these technologies converge to streamline workflows, optimize performance, and ensure secure access, all while leveraging the power of AWS IDP and real-time communication via Slack notifications.
Programming Foundation Models with DSPy - Meetup SlidesZilliz
Prompting language models is hard, while programming language models is easy. In this talk, I will discuss the state-of-the-art framework DSPy for programming foundation models with its powerful optimizers and runtime constraint system.
TrustArc Webinar - 2024 Global Privacy SurveyTrustArc
How does your privacy program stack up against your peers? What challenges are privacy teams tackling and prioritizing in 2024?
In the fifth annual Global Privacy Benchmarks Survey, we asked over 1,800 global privacy professionals and business executives to share their perspectives on the current state of privacy inside and outside of their organizations. This year’s report focused on emerging areas of importance for privacy and compliance professionals, including considerations and implications of Artificial Intelligence (AI) technologies, building brand trust, and different approaches for achieving higher privacy competence scores.
See how organizational priorities and strategic approaches to data security and privacy are evolving around the globe.
This webinar will review:
- The top 10 privacy insights from the fifth annual Global Privacy Benchmarks Survey
- The top challenges for privacy leaders, practitioners, and organizations in 2024
- Key themes to consider in developing and maintaining your privacy program
Satisfying Auditors: Plans and Evidence in a Regulated Environment
1. MM
PM Tutorial
10/13/2014 1:00:00 PM
"Satisfying Auditors: Plans and
Evidence in a Regulated Environment"
Presented by:
James Christie
Claro Testing
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com
2. James Christie
Claro Testing
James Christie is a testing consultant with thirty-one years of IT experience. Before moving
into testing, James spent six years as an IT auditor, so he has experience on both sides of the
fence. With experience in information security management, project management, business
analysis, and development, he is particularly interested in links between testing, auditing,
governance, and compliance. James spent fourteen years working for a large UK insurance
company, then nine years with IBM working with large clients in the UK and Finland. A member
of the Information Systems Audit and Control Association, James has been self-employed for
the past eight years.
3. 9/11/2014
1
Satisfying Auditors:
Plans and Evidence in a
Regulated Environment
James Christie
How I ended up in software
testing via auditing.
Why Alice in Wonderland was
relevant to my attempts to
understand what goes on in
big companies.
An introduction – to me and the tutorial 1a
4. 9/11/2014
2
“The chief difficulty Alice
found at first was in
managing her flamingo”
An introduction – to me and the tutorial 1a
“When I use a word,”
Humpty Dumpty said in rather a scornful tone,
“it means just what I choose it to mean —
neither more nor less”.
1b
Nothing seemed to make sense
6. 9/11/2014
4
Y2K – a testing time 1d
Image courtesy Stuart Miles & FreeDigitalPhotos.net
Information security management – the IBM way 1d
7. 9/11/2014
5
Some internal audit departments have an image problem
Ambiguous? I’m not sure exactly what this means,
but it’s not good.
1d
Back to testing again 1d
9. 9/11/2014
7
“External auditors are watchdogs not bloodhounds”
2b
Providing an opinion to the shareholders
about whether the accounts are true and fair.
Images courtesy Artur84/FreeDigitalPhotos.net
Providing an opinion to the shareholders
about whether the accounts are true and fair.
External auditor independence
Such a big problem it’s more than just a problem.
2b
10. 9/11/2014
8
“Commercial suicide”, alleged quote from current
chair of UK Financial Conduct Authority.
John Griffith-Jones
External auditor independence
Challenging client management?
2b
Images courtesy Artur84/FreeDigitalPhotos.net
Problem #1 - up or out
Images courtesy Stuart Miles, Renjith Krishnanur84/FreeDigitalPhotos.net
”It would seem that about 50% of newly qualified ACAs”It would seem that about 50% of newly qualified ACAs”It would seem that about 50% of newly qualified ACAs”It would seem that about 50% of newly qualified ACAs
do not have enough practical experience to continuedo not have enough practical experience to continuedo not have enough practical experience to continuedo not have enough practical experience to continue
their careers astheir careers astheir careers astheir careers as auditors”auditors”auditors”auditors”
MichaelMichaelMichaelMichael IzzaIzzaIzzaIzza, ICAEW Chief Executive, 2009, ICAEW Chief Executive, 2009, ICAEW Chief Executive, 2009, ICAEW Chief Executive, 2009
2b
11. 9/11/2014
9
””””Most internal auditors would join me in assessing theMost internal auditors would join me in assessing theMost internal auditors would join me in assessing theMost internal auditors would join me in assessing the
external audit partners and senior managers as arrogantexternal audit partners and senior managers as arrogantexternal audit partners and senior managers as arrogantexternal audit partners and senior managers as arrogant
beyond their competencebeyond their competencebeyond their competencebeyond their competence””””
NormanNormanNormanNorman Marks, 2010 (Marks, 2010 (Marks, 2010 (Marks, 2010 (chief audit exec at major global
corporations for 20+ years))))
Problem #2 – quality of people
2b
Problem #3 - sampling
Auditors can’t checkAuditors can’t checkAuditors can’t checkAuditors can’t check
all theall theall theall the figures. Thatfigures. Thatfigures. Thatfigures. That
would make auditswould make auditswould make auditswould make audits
far too expensive.far too expensive.far too expensive.far too expensive.
But they can’t justBut they can’t justBut they can’t justBut they can’t just
take figures on trust.take figures on trust.take figures on trust.take figures on trust.
So theySo theySo theySo they sample.sample.sample.sample.
How much do theyHow much do theyHow much do theyHow much do they
samplesamplesamplesample????
How do they chooseHow do they chooseHow do they chooseHow do they choose
the sample?the sample?the sample?the sample?
Cartoons courtesy Scott Adams
2b
12. 9/11/2014
10
Add up everything that moves through the books;Add up everything that moves through the books;Add up everything that moves through the books;Add up everything that moves through the books;
all revenue plus all costs, to get turnover=t. Let’sall revenue plus all costs, to get turnover=t. Let’sall revenue plus all costs, to get turnover=t. Let’sall revenue plus all costs, to get turnover=t. Let’s
say t=£25 million.say t=£25 million.say t=£25 million.say t=£25 million.
Problem #3 – sampling on the
Internal Controls Basis
Apply the accounts total & control score to theApply the accounts total & control score to theApply the accounts total & control score to theApply the accounts total & control score to the
sampling formula to get the sampling interval,sampling formula to get the sampling interval,sampling formula to get the sampling interval,sampling formula to get the sampling interval,
e.g. interval = t/(100*s). So interval = £50,000.e.g. interval = t/(100*s). So interval = £50,000.e.g. interval = t/(100*s). So interval = £50,000.e.g. interval = t/(100*s). So interval = £50,000.
Assess the internal controls and assign a score, s,Assess the internal controls and assign a score, s,Assess the internal controls and assign a score, s,Assess the internal controls and assign a score, s,
egegegeg 1= atrocious & 5=excellent. Assume s=5.1= atrocious & 5=excellent. Assume s=5.1= atrocious & 5=excellent. Assume s=5.1= atrocious & 5=excellent. Assume s=5.
2b
Problem #3 – sampling on the
Internal Controls Basis
Pull out a bank note. Take the last three digits.Pull out a bank note. Take the last three digits.Pull out a bank note. Take the last three digits.Pull out a bank note. Take the last three digits.
Express that as a decimal fraction of 1,000.Express that as a decimal fraction of 1,000.Express that as a decimal fraction of 1,000.Express that as a decimal fraction of 1,000.
Let’s say 0.851.Let’s say 0.851.Let’s say 0.851.Let’s say 0.851.
Sample your way through the accounts, examiningSample your way through the accounts, examiningSample your way through the accounts, examiningSample your way through the accounts, examining
every transaction you hit at the sample interval,every transaction you hit at the sample interval,every transaction you hit at the sample interval,every transaction you hit at the sample interval,
hitting every single transaction of £50,000+.hitting every single transaction of £50,000+.hitting every single transaction of £50,000+.hitting every single transaction of £50,000+.
Apply the fraction to the interval to get theApply the fraction to the interval to get theApply the fraction to the interval to get theApply the fraction to the interval to get the
starting point, i.e. £42,550.starting point, i.e. £42,550.starting point, i.e. £42,550.starting point, i.e. £42,550.
2b
13. 9/11/2014
11
Problem #3 – sampling on the
Internal Controls Basis; gaming the system
(aka cheating)
Why might you want to manipulate the method,Why might you want to manipulate the method,Why might you want to manipulate the method,Why might you want to manipulate the method,
and how would you do it?and how would you do it?and how would you do it?and how would you do it?
You can rig the internal controls score to get theYou can rig the internal controls score to get theYou can rig the internal controls score to get theYou can rig the internal controls score to get the
result you want. The higher the score, the higherresult you want. The higher the score, the higherresult you want. The higher the score, the higherresult you want. The higher the score, the higher
the sampling interval, and the less work thethe sampling interval, and the less work thethe sampling interval, and the less work thethe sampling interval, and the less work the
auditors have to do.auditors have to do.auditors have to do.auditors have to do.
You can rig the formula, but that’s obvious andYou can rig the formula, but that’s obvious andYou can rig the formula, but that’s obvious andYou can rig the formula, but that’s obvious and
you’d have to justify it.you’d have to justify it.you’d have to justify it.you’d have to justify it.
REMEMBER THE FEE IS SET BEFORE THE INTERNALREMEMBER THE FEE IS SET BEFORE THE INTERNALREMEMBER THE FEE IS SET BEFORE THE INTERNALREMEMBER THE FEE IS SET BEFORE THE INTERNAL
CONTROLCONTROLCONTROLCONTROL SCORE IS CALCULATED.SCORE IS CALCULATED.SCORE IS CALCULATED.SCORE IS CALCULATED.
2b
Internal Audit
a totally different perspective
“Internal auditing is an“Internal auditing is an“Internal auditing is an“Internal auditing is an independentindependentindependentindependent, objective, objective, objective, objective
assurance and consulting activity designed toassurance and consulting activity designed toassurance and consulting activity designed toassurance and consulting activity designed to
add value and improveadd value and improveadd value and improveadd value and improve an organization'san organization'san organization'san organization's
operations.operations.operations.operations.
It helps an organization accomplish itsIt helps an organization accomplish itsIt helps an organization accomplish itsIt helps an organization accomplish its
objectives by bringing a systematic, disciplinedobjectives by bringing a systematic, disciplinedobjectives by bringing a systematic, disciplinedobjectives by bringing a systematic, disciplined
approachapproachapproachapproach to evaluate and improve theto evaluate and improve theto evaluate and improve theto evaluate and improve the
effectiveness of risk management, control, andeffectiveness of risk management, control, andeffectiveness of risk management, control, andeffectiveness of risk management, control, and
governance processesgovernance processesgovernance processesgovernance processes.”.”.”.”
Global InstituteGlobal InstituteGlobal InstituteGlobal Institute of Internalof Internalof Internalof Internal AuditorsAuditorsAuditorsAuditors
2c
14. 9/11/2014
12
Internal Audit
The people are different
Image courtesy cooldesign & FreeDigitalPhotos.net
2c
Top six qualities internal auditors need
IIA’s 2013 Global Pulse of the Profession survey
Critical thinking
Communication
skills
Risk-management
IT knowledge
Data mining/analytics
Accounting
2c
15. 9/11/2014
13
1 - Critical thinking
2 - Communication skills
3 - Risk-management
4 - IT knowledge
5 - Data mining/analytics (frauds! ☺)
6 – Accountancy knowledge
Communications
Data mining/analytics
Risk-management assurance
IT knowledge
Accountancy knowledge
Top six qualities internal auditors need
IIA’s 2013 Global Pulse of the Profession survey
2c
Internal auditors know more
Deeper business knowledge
Greater tacit knowledge
Greater nous (streetwise)
More mature & stronger
characters?
Image courtesy Krormrathog & FreeDigitalPhotos.net
2c
16. 9/11/2014
14
Are internal auditors stronger?
You can’t bully good internal
auditors.
(If you can bully them then
they don’t last long).
2c
The internal audit hothouse
Internal audit is used as a
training ground for high
quality staff.
There is a potential
downside to staff rotation.
Where do they go next?
Image courtesy Chanpipat & FreeDigitalPhotos.net
2c
17. 9/11/2014
15
Risk and the financial crash
Risk is a tricky
concept and auditors
didn’t handle it well.
Image courtesy cooldesign & FreeDigitalPhotos.net
3
“...the chance, high or low, of somebody
being harmed by the hazard, and how
serious the harm could be”
(UK Health & Safety Executive)
Image courtesy jscreationzs & FreeDigitalPhotos.net
What is risk anyway?
“the effect of uncertainty on objectives”
(ISO 31000)
“a set of circumstances that hinder the
achievement of objectives”
(David Griffiths)
3a
18. 9/11/2014
16
UK HSE risk matrix
What is risk anyway?
3a
Enrico Fermi – the brilliant nuclear physicist
who worked on the project to develop the
atomic bomb.
What is risk anyway?
1939. The probability that nuclear fission
could be controlled for power or weapons?
10%
1945. The probability that the atomic bomb
would set the atmosphere on fire and wipe
out life on earth?
10%
1950. The probability that humans would
develop the technology to travel faster
than the speed of light by 1960? 10%
3a
19. 9/11/2014
17
Tim O’Riordan & Patrick Cox, 2001.
Science, Risk, Uncertainty & Precaution.
University of Cambridge.
3a
What is risk anyway?
Simple, understandable
and totally misleading?
Complex, accurate(?) and
totally uninformative?
Risk – the big dilemma?
or
Images courtesy Luigi Diamanti, Mr Brightman & FreeDigitalPhotos.net
3a
20. 9/11/2014
18
Rick Buy – Chief Risk Officer.
His stated aim was to ”condense”condense”condense”condense all the risks ofall the risks ofall the risks ofall the risks of
the corporation into a single metricthe corporation into a single metricthe corporation into a single metricthe corporation into a single metric”.”.”.”.
Risk – the big dilemma?
3b
Risk – and how we lost sight of it
Image courtesy of Just2shutter / FreeDigitalPhotos.net
“With half a decade’s
hindsight, it is clear the
crisis had multiple causes.
The most obvious is the
financiers themselves –
especially the irrationally
exuberant Anglo-Saxon sort,
who claimed to have found a
way to banish risk when in
fact they had simply lost
track of it.”
The Economist
Image courtesy pakorn / FreeDigitalPhotos.net
3c
21. 9/11/2014
19
Risk – and how we lost sight of it
Image courtesy of Just2shutter / FreeDigitalPhotos.net
“The weaknesses of
group risk in HBOS
were a matter of
design, not accident.”
Parliamentary
Commission on Banking
Standards;
“An Accident Waiting To
Happen: The Failure of
HBOS”
3c
Image courtesy pakorn / FreeDigitalPhotos.net
* Fixed probability
* Time period
* Amount at risk
Eg, 95% probability that the
maximum loss in a week
will not exceed £1m.
Definitely not 5% probability
of losing just £1m in a week.
Value at Risk - or losing sight of risk
Image courtesy pakorn / FreeDigitalPhotos.net
3c
22. 9/11/2014
20
Value at Risk – ignoring Black Swans
Decision makers and
auditors lost sight of
what VaR actually means.
Above the “VaR break”
all bets are off – we’re
into Black Swan
territory.
And that’s pretty much
what happened.
3c
Big 4 audit fees for 2007
““““…fees are now coming before independence, objectivity
(and sometimes, even competence) in important parts of
the accounting profession.””””
Paul Moore (ex partner KPMG, ex Head of Group Regulatory
Risk, HBOS - 2013)
3d
23. 9/11/2014
21
Big 6 foul ups in US
US PCAOB Audit
Failures 2012 (2011)
Grant Thornton 65% (43%)
BDO 55% (39%)
Ernst & Young 48% (36%)
PWC 39% (41%)
KPMG 34% (23%)
Deloitte 25% (42%)
(% of audits inspected deemed to
be “audit failures” by regulator)
3d
Image courtesy Stuart Miles & FreeDigitalPhotos.net
Has external audit had its day?
“External audit is now largely out-
dated. The binary nature of the
opinion renders it useless.”
Richard Anderson chairman of the
Institute of Risk Management,
2011
“With or without new rules, the
main worry for auditors may be
that people wonder whether
their reports are worth a bean.”
The Economist, April 2014
3e
24. 9/11/2014
22
Has external audit had its day?
“The fact that the audit process
failed to highlight developing
problems in the banking sector
does cause us to question exactly
how useful audit currently is.”
House of Commons Treasury
Committee “Banking Crisis”, 2009
“The problem is that there's not
a lot of evidence that (external)
auditors are very good at
assessing risk.”
Charles Cullinan, Bryant College,
USA
3e
Is internal audit better placed?
PeoplePeoplePeoplePeople
TimeTimeTimeTime
Business knowledgeBusiness knowledgeBusiness knowledgeBusiness knowledge
IndependenceIndependenceIndependenceIndependence
Business modelBusiness modelBusiness modelBusiness model
None of these are guaranteed advantagesNone of these are guaranteed advantagesNone of these are guaranteed advantagesNone of these are guaranteed advantages
Greater scrutiny of corporate culture?Greater scrutiny of corporate culture?Greater scrutiny of corporate culture?Greater scrutiny of corporate culture?
3f
25. 9/11/2014
23
Evidence and Opinion
How do we know anything?
What matters? Who cares?
“To know that we know what
we know, and to know that
we do not know what we do
not know, that is true
knowledge.” Copernicus
4a
4a
Ontology
What is the nature of
reality?
What isisisis a windmill?
What is real?
4a
26. 9/11/2014
24
How can Don Quijote know
anything?
Epistemology
What can he know about
windmills?
From Sancho Panza?
From his senses?
From books!
How can he know about
windmills?
Trigger’s Broom
4a
27. 9/11/2014
25
A positivist worldview?
Have we treated testing, and
auditing, as if they are like
scientific experiments where
we know and control all the
variables?
Have we been too keen to
assume the world we are
investigating is a neater and
more ordered place than it
really is?
4a
Is an interpretivist
worldview more helpful?
A dangerous extreme for
testers?
Certainly for auditors!
There is no single, fixed
reality. Everything is a social
construct so we have to
understand what we are
looking at rather than
criticising or condemning.
4a
28. 9/11/2014
26
A balanced approach?
(just doing the best we can)
We might not know things with
certainty, but we can make
statements based on evidence
& keep refining our opinion.
Positivists might think that
certainty is out there and we
can know it.
Interpretivists might not say
anything useful; they’re all
features, not bugs!
4a
They areThey areThey areThey are overoveroverover----simplifiers. Theysimplifiers. Theysimplifiers. Theysimplifiers. They
take a complicated issue andtake a complicated issue andtake a complicated issue andtake a complicated issue and
deliver a simplistic anddeliver a simplistic anddeliver a simplistic anddeliver a simplistic and
superficially plausible answer.superficially plausible answer.superficially plausible answer.superficially plausible answer.
They offer clear, actionable adviceThey offer clear, actionable adviceThey offer clear, actionable adviceThey offer clear, actionable advice
but…but…but…but…
Thanks toThanks toThanks toThanks to StianStianStianStian Westlake for this.Westlake for this.Westlake for this.Westlake for this.
Berks
4b
Don’t be a berk or a wanker
29. 9/11/2014
27
A wanker (that’s me)
They want to be robustThey want to be robustThey want to be robustThey want to be robust
and comprehensive, andand comprehensive, andand comprehensive, andand comprehensive, and
forget about clarity &forget about clarity &forget about clarity &forget about clarity &
brevity.brevity.brevity.brevity.
4b
Don’t be a berk or a wanker
Rikard Edgren
“Reality isn’t binary… we“Reality isn’t binary… we“Reality isn’t binary… we“Reality isn’t binary… we
don’t know everything indon’t know everything indon’t know everything indon’t know everything in
advance.advance.advance.advance.
We should observe theWe should observe theWe should observe theWe should observe the
software without asoftware without asoftware without asoftware without a
hypothesis to nullify.”hypothesis to nullify.”hypothesis to nullify.”hypothesis to nullify.”
Image courtesy digitalart binary / FreeDigitalPhotos.net
4c
The binary trap
30. 9/11/2014
28
The questions we can answer yes/no with most
certainty are probably those that don't matter.
The danger is that we focus on them because the
light is better there.
The binary trap
4c
It’s not meant to be easy, it’s meant to be valuable.
Test scripts are not testing.
Checklists are not auditing.
4c
The binary trap
31. 9/11/2014
29
Relying on scripts and checklists
assumes that the information we
want is under the streetlight.
It assumes that we can know in
advance what matters, what we
need to look for.
It assumes that the important
questions can be answered with
a “yes” or “no”.
The relevance to testers
4c
If we focus only on what was
specified we will not see what
was needed but neither
specified nor built (5).
And we won’t see what was
not specified or needed, but
was built (6).
Thanks to James Lyndsay, Iain McCowatt,
James Bach & Michael Bolton.
and auditors want to know too
Either could be damaging.
The relevance to testers
4c
32. 9/11/2014
30
Good auditors learn by listening.
Bad auditors don’t listen. Their
checklist tells them the “right
answers”.
UK & US regulators are pushing
auditors away from binary
opinions. EU???
An auditor – “one who hears, a listener
4c
Risk Based Auditing
What is it? How do they do it?
Image courtesy David Castillo Dominici/FreeDigitalPhotos.net
We don’t understand risk well.
We don’t understand auditing.
So do we really know what Risk
Based Auditing means?
5
33. 9/11/2014
31
Risk Based Auditing – what is it?
2- RBA provides
assurance that risks
are being managed
effectively.
3- RBA focuses
effort on the areas
most likely to suffer
problems.
1- RBA identifies
risks so that
management can
eliminate them.
4- RBA focuses on
the risks that pose
the greatest threat to
company objectives.
5a
1- RBA identifies
risks so that
management can
eliminate them.
Risk Based Auditing – what is it?
5a
3- RBA focuses
effort on the areas
most likely to suffer
problems.
2- RBA provides
assurance that risks
are being managed
effectively.
4- RBA focuses on
the risks that pose
the greatest threat to
company objectives.
34. 9/11/2014
32
Controls based auditor; “how can I be
sure no-one will steal bricks while the
house is being built?”
Script driven tester; “what tests should I
write for using these bricks to build a nice
house?”
Risk based auditor; “could someone hit the
cashier over the head with a brick and
steal the payroll? Is that significant?”
The exploratory tester?
Image courtesy Piyachok Thawornmat/FreeDigitalPhotos.net
Risk Based Auditing – what is it?
5a
There’s compliance,
and then there’s compliance
Big difference between the
cops and mere processes!
5a
35. 9/11/2014
33
Reasonable assurance about risks
not absolute
5a
Appropriate…
sufficient…
reasonable…
material
Auditors are looking for
reasonable assurance,
not absolute assurance.
Risks that matter
“Audit priorities (should) align with
those of the board and executive
management. Risks that keep our
stakeholders up at night also should be
of concern to us.”
Richard Chambers, CEO & President of
Institute of Internal Auditors
“The problem is that there's not a lot of
evidence that (external) auditors are
very good at assessing risk.”
(reminder!)
Charles Cullinan, Bryant College
Image courtesy digitalart /FreeDigitalPhotos.net
5a
36. 9/11/2014
34
Attitude of the Institute
of Internal Auditors
Compliance auditing; “tipping
out the pieces of a jigsaw puzzle
on to the Audit Committee table
rather than turning those pieces
into a picture.”
Sarah Blackburn, ex President of
IIA UK
Image courtesy Stuart Miles/FreeDigitalPhotos.net
5a
Image courtesy Stuart Miles/FreeDigitalPhotos.net
“In a risk-based approach to security,
compliance is provided by security –
security is not necessarily provided
by compliance.”
John Wheeler, Gartner Inc
Moving this way?
5a
Attitude of the Institute
of Internal Auditors
““““Many organizations look at
compliance as a set of check boxes…
but compliance is not the goal, it’s a
result.”
Mike Rothman, Security Incite
37. 9/11/2014
35
Risk Based Auditing - doing it
Image courtesy Stuart Miles/FreeDigitalPhotos.net
There are no right answers
(probably).
The checklist is not the audit. It’s
just a tool.
Auditors who rely on checklists are
unprofessional compliance monkeys. It
demeans and deskills the profession.
5b
Risk Based Auditing - planning it
A development that is under way
RequirementsRequirementsRequirementsRequirements
Test strategyTest strategyTest strategyTest strategy Test resultsTest resultsTest resultsTest resultsDetailed testDetailed testDetailed testDetailed test
plan/scriptsplan/scriptsplan/scriptsplan/scripts
DesignDesignDesignDesign
documentsdocumentsdocumentsdocuments
DevelopmentDevelopmentDevelopmentDevelopment
standardsstandardsstandardsstandards
Problem recordsProblem recordsProblem recordsProblem records
Other?Other?Other?Other?
Handover packHandover packHandover packHandover packProject planProject planProject planProject plan
Change recordsChange recordsChange recordsChange records
A development that went live two months ago
A live system that's been running for four years
5b
38. 9/11/2014
36
A development that is under way
RequirementsRequirementsRequirementsRequirements
Test strategyTest strategyTest strategyTest strategy Test resultsTest resultsTest resultsTest resultsDetailed testDetailed testDetailed testDetailed test
plan/scriptsplan/scriptsplan/scriptsplan/scripts
DesignDesignDesignDesign
documentsdocumentsdocumentsdocuments
DevelopmentDevelopmentDevelopmentDevelopment
standardsstandardsstandardsstandards
Problem recordsProblem recordsProblem recordsProblem records
Other?Other?Other?Other?
Handover packHandover packHandover packHandover packProject planProject planProject planProject plan
Change recordsChange recordsChange recordsChange records
5b
Risk Based Auditing - planning it
A development that went live two months ago
RequirementsRequirementsRequirementsRequirements
Test strategyTest strategyTest strategyTest strategy Test resultsTest resultsTest resultsTest resultsDetailed testDetailed testDetailed testDetailed test
plan/scriptsplan/scriptsplan/scriptsplan/scripts
DesignDesignDesignDesign
documentsdocumentsdocumentsdocuments
DevelopmentDevelopmentDevelopmentDevelopment
standardsstandardsstandardsstandards
Problem recordsProblem recordsProblem recordsProblem records
Other?Other?Other?Other?
Handover packHandover packHandover packHandover packProject planProject planProject planProject plan
Change recordsChange recordsChange recordsChange records
5b
Risk Based Auditing - planning it
39. 9/11/2014
37
A live system that's been running for four years
RequirementsRequirementsRequirementsRequirements
Test strategyTest strategyTest strategyTest strategy Test resultsTest resultsTest resultsTest resultsDetailed testDetailed testDetailed testDetailed test
plan/scriptsplan/scriptsplan/scriptsplan/scripts
DesignDesignDesignDesign
documentsdocumentsdocumentsdocuments
DevelopmentDevelopmentDevelopmentDevelopment
standardsstandardsstandardsstandards
Problem recordsProblem recordsProblem recordsProblem records
Other?Other?Other?Other?
Handover packHandover packHandover packHandover packProject planProject planProject planProject plan
Change recordsChange recordsChange recordsChange records
5b
Risk Based Auditing - planning it
Conway’s Law – a personal
hobby horse.
“Organizations which
design systems ... are
constrained to produce
designs which are copies
of the communication
structures of these
organizations”
Melvin Conway
Image courtesy jscreationzs/FreeDigitalPhotos.net
5b
Risk Based Auditing - planning it
40. 9/11/2014
38
The communications and
organisational structure
are a useful guide to
where the worst flaws
will be in the project and
system.
My auditor’s corollary (or
heuristic) to Conway’s
Law.
Risk Based Auditing - planning it
5b
Conway’s Law – a personal
hobby horse.
Image courtesy jscreationzs/FreeDigitalPhotos.net
Risk Based Auditing
IDEF0 & decomposing an application
5b
41. 9/11/2014
39
Risk Based Auditing
Exploratory testing?
Breaking the application
Image courtesy Stuart Miles/FreeDigitalPhotos.net
5b
Don’t tell me, show me (auditor’s mantra)
“Don’t tell me the moon
is shining, show me the
glint of light on broken
glass”
Anton Chekhov
5b
42. 9/11/2014
40
And why does it matter?
Different parts of the world
have different models –
with different outcomes.
6
What is Governance?
κυβερνάω [kubernáo] – to steer?
6a
What is Governance?
43. 9/11/2014
41
Corporate governance is the board’s job
Should not involve day to day operational
management by full-time executives
Supervising management & reporting to
shareholders
Setting the strategic aims & values
Leadership to put them into effect
Values based on principles of transparency,
accountability, probity and long term
sustainability
Paraphrased from the UK Financial Reporting
Council’s “UK Corporate Governance Code”
6a
IT governance is the responsibility of corporate
management
Evaluates stakeholders’ needs and sets
objectives to satisfy them
Directs and sets priorities
Monitors performance
Paraphrased from ISACA’s definition
6a
44. 9/11/2014
42
IT management
Plans
Builds
Runs
Monitors
All in alignment with the strategic
direction set by the governance body
Paraphrased from ISACA’s definition
6a
Why governance is a good thing
If we get
governance
wrong then
we suffer
Images courtesy Scott Adams & Stuart Miles/FreeDigitalPhotos.net
6b
45. 9/11/2014
43
Governance - Risk Management
Three Lines of Defence
Functions that own and manage risks;
operational management (the front line)
Functions that oversee risks; risk
management and compliance function
Functions that provide independent assurance;
internal audit
IIA strongly recommended guidance
6c
Governance – comply or explain
“Comply or explain” is the UK approach.
Also Germany and Netherlands.
UK Corporate Governance Code, Deutscher Corporate
Governance Codex & Code Tabaksblat
US style
Comply or else!
(my experience)
Images courtesy Stuart Miles & FreeDigitalPhotos.net
6d
46. 9/11/2014
44
Governance – different countries,
different models, different outcomes
etc
6e
ISACA
Information Systems Audit & Control
Association
ISACA and COBIT 5
Why they matter
7
7
47. 9/11/2014
45
ISO/IEC 38500:2008 Model for
Corporate Governance of IT
7a
COBIT 5 interpretation of IT governance
Control Objectives for Information
and Related Technology
7b
48. 9/11/2014
46
COBIT 5 interpretation of IT governance
7c
A Quality Management System with quality
standards.
AP011 Manage QualityAP011 Manage QualityAP011 Manage QualityAP011 Manage Quality
“Best practices” to be used as a “reference when
improving and tailoring”.
Based on industry “goodgoodgoodgood practices”.
No mention of specific standards (or even the
need to go looking for standards to adapt).
ISACA expect
the following
COBIT 5 interpretation of IT governance
7c
49. 9/11/2014
47
“Validate all requirements through approaches
such as peer review, model validation or
operational prototyping”.
BA102 ManageBA102 ManageBA102 ManageBA102 Manage
Requirements DefinitionRequirements DefinitionRequirements DefinitionRequirements Definition
“If appropriate, implement the selected option
as a pilot to determine possible improvements”.
“Review the alternative solutions… and select the
most appropriate one based on feasibility… risk
and cost.”
ISACA expect
the following
7c
COBIT 5 interpretation of IT governance
“… using agreed-on and appropriate phased or
rapid agile development techniques”.
BA103 Manage SolutionsBA103 Manage SolutionsBA103 Manage SolutionsBA103 Manage Solutions
Identification & BuildIdentification & BuildIdentification & BuildIdentification & Build
ISACA expect
the following
“Proactively evaluate for design weaknesses (e.g.,
inconsistencies, lack of clarity, potential flaws)
throughout the life cycle”.
BA103.02 Design detailedBA103.02 Design detailedBA103.02 Design detailedBA103.02 Design detailed
solution componentssolution componentssolution componentssolution components
7c
COBIT 5 interpretation of IT governance
50. 9/11/2014
48
“Develop, resource and execute a QA plan
aligned with the QMS to obtain the quality
specified in the requirements definition and the
enterprise’s quality policies and procedures.”
BA103.06 Perform qualityBA103.06 Perform qualityBA103.06 Perform qualityBA103.06 Perform quality
assuranceassuranceassuranceassurance
ISACA expect
the following
BA103 Manage SolutionsBA103 Manage SolutionsBA103 Manage SolutionsBA103 Manage Solutions
Identification & BuildIdentification & BuildIdentification & BuildIdentification & Build
7c
COBIT 5 interpretation of IT governance
“1. Define a QA plan & practices including, e.g.,
specification of quality criteria, validation and
verification processes, definition of how quality
will be reviewed, necessary qualifications of
quality reviewers, and roles and responsibilities
for the achievement of quality.”
BA103.06 Perform qualityBA103.06 Perform qualityBA103.06 Perform qualityBA103.06 Perform quality
assuranceassuranceassuranceassurance
ISACA expect
the following
BA103 Manage SolutionsBA103 Manage SolutionsBA103 Manage SolutionsBA103 Manage Solutions
Identification & BuildIdentification & BuildIdentification & BuildIdentification & Build
7c
COBIT 5 interpretation of IT governance
51. 9/11/2014
49
“2. Frequently monitor the solution quality
based on project requirements, enterprise
policies, adherence to development
methodologies, quality management
procedures and acceptance criteria.”
BA103 Manage SolutionsBA103 Manage SolutionsBA103 Manage SolutionsBA103 Manage Solutions
Identification & BuildIdentification & BuildIdentification & BuildIdentification & Build
BA103.06 Perform qualityBA103.06 Perform qualityBA103.06 Perform qualityBA103.06 Perform quality
assuranceassuranceassuranceassurance
ISACA expect
the following
7c
COBIT 5 interpretation of IT governance
“3. Employ code inspection, test-driven
development practices, automated testing,
continuous integration, walk-throughs and
testing of applications as appropriate.”
ISACA expect
the following
BA103 Manage SolutionsBA103 Manage SolutionsBA103 Manage SolutionsBA103 Manage Solutions
Identification & BuildIdentification & BuildIdentification & BuildIdentification & Build
BA103.06 Perform qualityBA103.06 Perform qualityBA103.06 Perform qualityBA103.06 Perform quality
assuranceassuranceassuranceassurance
7c
COBIT 5 interpretation of IT governance
52. 9/11/2014
50
“Establish a test plan and required environments
to test the individual and integrated solution
components, including the business processes
and supporting services, applications and
infrastructure.”
ISACA expect
the following
BA103 Manage SolutionsBA103 Manage SolutionsBA103 Manage SolutionsBA103 Manage Solutions
Identification & BuildIdentification & BuildIdentification & BuildIdentification & Build
BAI03.07 Prepare forBAI03.07 Prepare forBAI03.07 Prepare forBAI03.07 Prepare for
solution testingsolution testingsolution testingsolution testing
7c
COBIT 5 interpretation of IT governance
“Execute testing continually during development.”
ISACA expect
the following
BA103 Manage SolutionsBA103 Manage SolutionsBA103 Manage SolutionsBA103 Manage Solutions
Identification & BuildIdentification & BuildIdentification & BuildIdentification & Build
BAI03.08 ExecuteBAI03.08 ExecuteBAI03.08 ExecuteBAI03.08 Execute
solution testingsolution testingsolution testingsolution testing
Not “keep busy writing scripts till the testing
phase”.
7c
COBIT 5 interpretation of IT governance
53. 9/11/2014
51
“1. Undertake testing of solutions and their
components in accordance with the testing plan.
Include testers independent from the solution
team…”
ISACA expect
the following
BA103 Manage SolutionsBA103 Manage SolutionsBA103 Manage SolutionsBA103 Manage Solutions
Identification & BuildIdentification & BuildIdentification & BuildIdentification & Build
BAI03.08 ExecuteBAI03.08 ExecuteBAI03.08 ExecuteBAI03.08 Execute
solution testingsolution testingsolution testingsolution testing
7c
COBIT 5 interpretation of IT governance
“2. Use clearly defined test instructions, as
defined in the test plan, and consider the
appropriate balance between automated scripted
tests and interactive user testing.”
ISACA expect
the following
BA103 Manage SolutionsBA103 Manage SolutionsBA103 Manage SolutionsBA103 Manage Solutions
Identification & BuildIdentification & BuildIdentification & BuildIdentification & Build
BAI03.08 ExecuteBAI03.08 ExecuteBAI03.08 ExecuteBAI03.08 Execute
solution testingsolution testingsolution testingsolution testing
7c
COBIT 5 interpretation of IT governance
54. 9/11/2014
52
“3. Undertake all tests in accordance with the
test plan and practices including the integration
of business processes & IT solution components
and of non-functional requirements (e.g.,
security, interoperability, usability).”
ISACA expect
the following
BA103 Manage SolutionsBA103 Manage SolutionsBA103 Manage SolutionsBA103 Manage Solutions
Identification & BuildIdentification & BuildIdentification & BuildIdentification & Build
BAI03.08 ExecuteBAI03.08 ExecuteBAI03.08 ExecuteBAI03.08 Execute
solution testingsolution testingsolution testingsolution testing
7c
COBIT 5 interpretation of IT governance
“4. Identify, log and classify (e.g., minor,
significant and mission-critical) errors during
testing... Ensure that an audit trail of test results
is maintained.”
ISACA expect
the following
BA103 Manage SolutionsBA103 Manage SolutionsBA103 Manage SolutionsBA103 Manage Solutions
Identification & BuildIdentification & BuildIdentification & BuildIdentification & Build
BAI03.08 ExecuteBAI03.08 ExecuteBAI03.08 ExecuteBAI03.08 Execute
solution testingsolution testingsolution testingsolution testing
7c
COBIT 5 interpretation of IT governance
55. 9/11/2014
53
“5. Record testing outcomes and communicate
results of testing to stakeholders in accordance
with the test plan.”
ISACA expect
the following
BA103 Manage SolutionsBA103 Manage SolutionsBA103 Manage SolutionsBA103 Manage Solutions
Identification & BuildIdentification & BuildIdentification & BuildIdentification & Build
BAI03.08 ExecuteBAI03.08 ExecuteBAI03.08 ExecuteBAI03.08 Execute
solution testingsolution testingsolution testingsolution testing
7c
COBIT 5 interpretation of IT governance
“2. Ensure that the test plan reflects an
assessment of risk from the project.”
BA107.03 Plan acceptanceBA107.03 Plan acceptanceBA107.03 Plan acceptanceBA107.03 Plan acceptance
teststeststeststests
BA107 Manage ChangeBA107 Manage ChangeBA107 Manage ChangeBA107 Manage Change
Acceptance and TransitioningAcceptance and TransitioningAcceptance and TransitioningAcceptance and Transitioning
Not in BA103 surprisingly.
ISACA expect
the following
7c
COBIT 5 interpretation of IT governance
56. 9/11/2014
54
“3. Ensure that the test plan addresses the
potential need for internal or external
accreditation of outcomes of the test process
(e.g., financial regulatory requirements).”
BA107.03 Plan acceptanceBA107.03 Plan acceptanceBA107.03 Plan acceptanceBA107.03 Plan acceptance
teststeststeststests
BA107 Manage ChangeBA107 Manage ChangeBA107 Manage ChangeBA107 Manage Change
Acceptance and TransitioningAcceptance and TransitioningAcceptance and TransitioningAcceptance and Transitioning
ISACA expect
the following
7c
COBIT 5 interpretation of IT governance
“5. Ensure that the test plan identifies testing
phases appropriate to the operational
requirements and environment.”
BA107.03 Plan acceptanceBA107.03 Plan acceptanceBA107.03 Plan acceptanceBA107.03 Plan acceptance
teststeststeststests
BA107 Manage ChangeBA107 Manage ChangeBA107 Manage ChangeBA107 Manage Change
Acceptance and TransitioningAcceptance and TransitioningAcceptance and TransitioningAcceptance and Transitioning
ISACA expect
the following
7c
COBIT 5 interpretation of IT governance
57. 9/11/2014
55
“6. Confirm that the test plan considers test
preparation … training requirements, … test
environment, planning/performing/documenting/
retaining test cases, error and problem handling,
correction and escalation, and formal approval.”
BA107.03 Plan acceptanceBA107.03 Plan acceptanceBA107.03 Plan acceptanceBA107.03 Plan acceptance
teststeststeststests
BA107 Manage ChangeBA107 Manage ChangeBA107 Manage ChangeBA107 Manage Change
Acceptance and TransitioningAcceptance and TransitioningAcceptance and TransitioningAcceptance and Transitioning
ISACA expect
the following
7c
COBIT 5 interpretation of IT governance
“6. Consider using clearly defined test
instructions (scripts) to implement the tests.”
BA107.05 PerformBA107.05 PerformBA107.05 PerformBA107.05 Perform
acceptance testsacceptance testsacceptance testsacceptance tests
That’s the end of testing in COBIT 5
BA107 Manage ChangeBA107 Manage ChangeBA107 Manage ChangeBA107 Manage Change
Acceptance and TransitioningAcceptance and TransitioningAcceptance and TransitioningAcceptance and Transitioning
ISACA expect
the following
7c
COBIT 5 interpretation of IT governance
58. 9/11/2014
56
COBIT 5 – big lessons for testers
No insistence on “best practice”
Countless references to ISO standards for;
- Risk management
- Security
- Release management
- Configuration management
- Service level management
- Incident management
- Problem management
- Business continuity
- etc No mention of testing standards
No insistence on detailed scripts or test cases.
None at all!
7d
Institute of Internal Auditors
IIA standards - good news
(seriously!)
8
59. 9/11/2014
57
The Snowflake Theory of IT Audit
“Every IT environment is unique
and represents a unique set of
risks. The differences make it
increasingly difficult to take a
generic or checklist approach
to auditing.”
Institute of Internal Auditors
Global Technology Audit Guide,
Management of IT Audit, 1st edition,
2006
8a
IIA IT Audit Management Standard
“Frameworks and StandardsFrameworks and StandardsFrameworks and StandardsFrameworks and Standards
One challenge auditors face when executing
IT audit work is knowing what to audit
against. Most organizations have not fully
developed IT control baselines for all
applications and technologies. The rapid
evolution of technology could likely render
any baselines useless after a short period of
time.”
Institute of Internal Auditors
Global Technology Audit Guide,
Management of IT Audit, 2nd edition,
2013
Image courtesy Salvatore Vuono & FreeDigitalPhotos.net
8b
60. 9/11/2014
58
ISO standards are not mentioned except in an
appendix “… for consideration”.
COBIT 5 is a recommended source of “control
objectives” against which auditors can work. It
offers “robust and generally accepted IT-
specific control objectives… that helps
management to conceptualize an approach for
measuring and managing IT risk”.
Institute of Internal Auditors
Global Technology Audit Guide,
Management of IT Audit, 2nd edition, 2013
IIA IT Audit Management Standard
8b
IIA Auditing IT Projects Standard
A basic primer in software development (not a
criticism – humility is not a bad thing).
Every organisation uses a different mix of
methods, standards & tools. Auditors must
understand these. They’re the ones that matter.
Institute of Internal Auditors
Global Technology Audit Guide,
Auditing IT Projects, 2009
8c
61. 9/11/2014
59
Mentions ISO project management standards,
but not testing standards.
Favourably disposed towards Agile (one of the
top ten factors for project success).
Institute of Internal Auditors
Global Technology Audit Guide,
Auditing IT Projects, 2009
Importance of COBIT 5 is stressed – though the
IIA does think it’s mainly about project
management.
8c
IIA Auditing IT Projects Standard
“Internal auditors should not expect
organizations to fully implement PMBOK,
PRINCE2, COBIT, or any other large set of best
practices. Rather, they should expect to see
that these practices have been customized and
integrated into the organization’s project
management methodology.”
Institute of Internal Auditors
Global Technology Audit Guide,
Auditing IT Projects, 2009
IIA Auditing IT Projects Standard
8c
62. 9/11/2014
60
Sarbanes Oxley
Does Sarbox deserve its
scary reputation?
Yes, but…
No, but…
9
Is Sarbanes Oxley scary?
Yes, especially section 404.
That’s the requirement that
management and the external
auditors must report on internal
control over financial reporting.
It’s a lot of work and it scares
people who can make life difficult.
Image courtesy Simon Howden & FreeDigitalPhotos.net
But, it’s only for US
companies, but… but…
9a
63. 9/11/2014
61
No, so long as you don’t
have Wally in charge of
compliance.
Comply with COBIT 5
and Sarbox need not
be a problem for
testers.
That’s one of the
reasons COBIT 5 is so
important.
Cartoon courtesy Scott Adams
Is Sarbanes Oxley scary?
9b
“Documentation is never required
‘for the auditors’.
If it is required it is because
it is needed to manage the project,
or it is a requirement of the project
that has to be justified like any
other requirement.”
James Christie
“Do standards keep testers in
the kindergarten?”
Testing Experience, Dec 2009
http://clarotesting.com/page20.htmhttp://clarotesting.com/page20.htmhttp://clarotesting.com/page20.htmhttp://clarotesting.com/page20.htm
Image courtesy Simon Howden & FreeDigitalPhotos.net
Is Sarbanes Oxley scary?
9b
64. 9/11/2014
62
US Food & Drugs Administration
What does the FDA expect?
10
What does the FDA expect?
Strong Credible
Images courtesy Simon Howden, Stuart Miles & FreeDigitalPhotos.net
10a
US Food & Drugs Administration
65. 9/11/2014
63
What does the FDA expect?
“Test procedures, test
data, and test results
should be documented in a
manner permitting
objective pass/fail
decisions to be reached.”
Image courtesy Stuart Miles & FreeDigitalPhotos.net
General Principles of
Software Validation,
FDA 2002
General Principles of
Software Validation,
FDA 2002
10b
US Food & Drugs Administration
What does the FDA expect?
10c
US Food & Drugs Administration
“The FDA is open to agile
processes and realizes that
the current approach to
software validation is not
working”
Griffin Jones
CAST 2011
66. 9/11/2014
64
AAMI TIR45:2012
“Guidance on the use
of AGILE practices in
the development of
medical device
software”
What does the FDA expect?
10c
US Food & Drugs Administration
“Agile can be adapted to the
unique needs of medical
device software… … and (can
satisfy) regulatory requirements.”
AAMI TIR45:2012
Shows how Agile maps
onto IEC 62304 (the
standard specifying
lifecycle requirements
for developing medical
software).
What does the FDA expect?
10c
US Food & Drugs Administration
67. 9/11/2014
65
“The exploratory stage of
clinical device development is intended to
allow for any iterative improvement of
the design of the device, advance the
understanding of how the device works
and its safety, and to set the stage for the
pivotal study.”
Image courtesy digitalart & FreeDigitalPhotos.net
FDA draft guidance
2011
http://www.fda.gov/MedicalDevices/
DeviceRegulationandGuidance/Guida
nceDocuments/ucm265553.htm
10d
US Food & Drugs Administration
What does the FDA expect?
Image courtesy Master & FreeDigitalPhotos.net
What does the FDA expect?
10e
US Food & Drugs Administration
68. 9/11/2014
66
Image courtesy Master & FreeDigitalPhotos.net
Clear
Objective (not requiring
interpretation)
Authentic
Demonstrable integrity
Readable & available
Evidence that will stand up in court
What does the FDA expect?
10e
US Food & Drugs Administration
Check out Griffin Jones’ work.
See his talk on YouTube.
http://www.youtube.com/watch?v=i8he7Rejn5s
Image courtesy Master & FreeDigitalPhotos.net
Attributable and not repudiable
Full record & audit trail for changes
Contemporary
Evidence that will stand up in court
Check out Griffin Jones’ work.
See his talk on YouTube.
http://www.youtube.com/watch?v=i8he7Rejn5s
What does the FDA expect?
10e
US Food & Drugs Administration
Seriously consider filming testing.
69. 9/11/2014
67
Image courtesy Master & FreeDigitalPhotos.net
Evidence that will stand up in court
Check out Griffin Jones’ work.
See his talk on YouTube.
http://www.youtube.com/watch?v=i8he7Rejn5s
What does the FDA expect?
10e
US Food & Drugs Administration
The evidence has to be sufficient
(quality and quantity) so that 3rd
parties will have to come to the
same conclusion if they review it,
without interpretation by the
testers.
What does the FDA expect?
10f
US Food & Drugs Administration
“the more energy put in
to preparation, the less
likely direct observations
are captured”
Griffin Jones
on Twitter
70. 9/11/2014
68
What does the FDA expect?
10f
US Food & Drugs Administration
Evidence of planning is
emphatically not evidence of
what was done.
Detailed test script
documentation is not
evidence of test execution.
Is a beautifully constructed
project plan evidence that the
project finished on time?
Image courtesy Stuart Miles & FreeDigitalPhotos.net
10f
US Food & Drugs Administration
Image courtesy digitalart & FreeDigitalPhotos.net
Get help
71. 9/11/2014
69
Test Strategy & Planning
What does a good auditor expect?
Images courtesy Stuart Miles & Master/FreeDigitalPhotos.net
StrategyStrategyStrategyStrategy not form filling
Relevance, not boiler-plate
11a
Test Strategy & Planning
Images courtesy Stuart Miles & Master/FreeDigitalPhotos.net
11a
Thoughtfulness, not massive documentation
Honesty, not spurious confidence
What does a good auditor expect?
72. 9/11/2014
70
The strategy is not
the process.
The strategy isn’t part
of the plan, it shapes
the plan.
We’re hopeless at strategy
Cartoon courtesy Scott Adams
Test Strategy & Planning 11a
My experience - we randomly mix up
processes, strategy & planning.
James Bach talking like an auditor sensation!
Test Strategy & Planning 11a
73. 9/11/2014
71
James Bach talking like an auditor sensation!
Test Strategy & Planning 11a
Brainless optimism.
Problems are not removed
with a stroke of the pen.
Problems do not disappear
if they are ignored.
Budding auditor
or tester?
James Bach talking like an auditor sensation!
Test Strategy & Planning 11a
74. 9/11/2014
72
Kopimism - “the act of
copying is sacred”.
Copy/pasting is not
cool. It’s evidence of a
lack of thought.
Writing a strategy is not
a matter of fleshing out
a template, or recycling
an old strategy.
The Kopimism Heresy
Test Strategy & Planning 11a
“Strategies” running to 50+ pages.
“Assumptions” & “risks” that are
just wishes that bad things won’t
happen (ifififif they’re even stated).
Failure to learn from experience.
Go live dates announced before
work is sized or staff secured.
Successive draft versions of project
plans that get more optimistic
without obvious plausible reasons.
Images courtesy digitalart/FreeDigitalPhotos.net
11b
Test Strategy & Planning
More warning signs (a personal list)
75. 9/11/2014
73
Requirements can’t be traced
through to testing.
Images courtesy digitalart/FreeDigitalPhotos.net
11b
Test Strategy & Planning
“Testing must be traceable to
requirements”.
Vague defect management process.
Environments?
Conflicting demands on resources.
Conway’s Law.
More warning signs (a personal list)
A better
way?
Test Strategy & Planning
RST Heuristic Test Strategy Model
11c
76. 9/11/2014
74
Really good, but…
it’s not a template,
it won’t think for you,
it won’t stop you
making blunders I’ve
seen with traditional
approaches,
and you have to follow
the spirit, not the letter,
and THINK.
Test Strategy & Planning
RST Heuristic Test Strategy (Plan? Model
11c
The strategy has to show
how you’ve thought your way
through from the problem to
a plausible answer.
Image courtesy David Castillo Dominici & FreeDigitalPhotos.net
The plan should show
how you’ll implement the
strategy.
Test Strategy & Planning
“Plan =strategy + logistics”
11d
77. 9/11/2014
75
Images courtesy Simon Howden, Stuart Miles, Artur84 & stockimages/FreeDigitalPhotos.net
What does a
good auditor
expect?
12
Test Execution
Test Execution
COBIT 5
Remember COBIT 5. That says it all.
Record & communicate everything
you said you’d do.
Exploratory testing?
Rapid Software
Testing?
What does a
good auditor
expect?
Images courtesy Simon Howden, Stuart Miles, Artur84 & stockimages/FreeDigitalPhotos.net
12a
78. 9/11/2014
76
Test execution deviating from the
plan. Hmm!
Changes to defect management &
reporting and test priorities
during the test execution.
Lack of an audit trail for defects/fixes
& a lack of reliable, contemporary
evidence.
Image courtesy digitalart/FreeDigitalPhotos.net
Test Execution
Warning signs (an official list from COBIT 5)
12b
Image courtesy digitalart/FreeDigitalPhotos.net
Test Execution
Warning signs (an official list from COBIT 5)
12b
In summary, auditors expect the
plan to be relevant.
There are good reasons to change
plans and schedules during
testing, but auditors will be very
suspicious of anything that looks
like winging the testing because
the plan was rubbish, or rigging
the testing schedule to hit the
implementation date.
79. 9/11/2014
77
Image courtesy digitalart/FreeDigitalPhotos.net
Test Execution
Warning signs (a personal list)
12c
Reporting that implies a link between
test case passes & progress.
Confusion between defect fix
priority & defect severity.
Massaging defect severity down
and up.
Treating usability issues as cosmetic.
Image courtesy digitalart/FreeDigitalPhotos.net
Test Execution
Warning signs (a personal list)
12c
Large numbers of defects being
rejected.
Defects rejected because there’s no
matching test case or requirement.
Defects rejected because the
requirements are assumed to be
correct.
Failure to write reusable automated
tests.
80. 9/11/2014
78
Test Reports
Images courtesy Stuart Miles & David Castillo Dominici/FreeDigitalPhotos.net
13a
What does a good
auditor expect?
The same as a good
test manager.
What does a good
auditor expect?
Images courtesy Stuart Miles & David Castillo Dominici/FreeDigitalPhotos.net
13b
Test Reports
(with thanks to Rapid Software Testing)
Learning about the
product
Learning about
how the product
was tested.
Learning about
how good the
testing was.
81. 9/11/2014
79
Images courtesy Stuart MilesFreeDigitalPhotos.net
Putting the jigsaw together.
Don’t empty the box
onto the table.
Put the pieces together to
assemble a clear picture, to
tell a compelling story.
Test Reports
13c
Images courtesy Stuart Miles & David Castillo Dominici/FreeDigitalPhotos.net
Auditors live and die by evidence.
Opinions are not casual observations.
They must be backed by evidence.
Test Reports
13d
82. 9/11/2014
80
Images courtesy Stuart Miles & David Castillo Dominici/FreeDigitalPhotos.net
Finally, say what you mean and
mean what you say.
Auditors will take your statements at
face value.
Test Reports
13d
Testing Standards
14
83. 9/11/2014
81
Wrap Up
Image courtesy Stuart Miles/FreeDigitalPhotos.net
Never follow the letter of the
law and ignore the spirit.
Never do something just
because “that’s what the
auditors will expect”.
Do the right thing and be ready
to justify it.
Go and speak to the auditors.
Say what you mean and mean
what you say. And never lie!
15a
Image courtesy Stuart Miles/FreeDigitalPhotos.net
Email: james@clarotesting.com
Twitter: @james_christie
www.clarotesting.wordpress.com
www.clarotesting.com
15b
Wrap Up