While testing in demo and stage is good (indeed, essential), testing in production is all too often overlooked. Deploying to production and hoping for the best is a gamble, not a strategy.
In this talk, we discuss
1) Better production deployment and testing strategies including dark pool testing, canary releases and feature switching.
2) After deployment, your work is still not done. We'll talk about Observability, including monitoring, tracing and metrics.
3) Finally, even with the best deployment strategies and monitoring techniques, your software WILL fail in production. It's a question of when, not if. So why not simulate those failures first? We'll finish with game days and chaos engineering.
This talk should be of interest to all developers, QA and Ops folks who are responsible for getting working software in front of users.
2017-07-11 GovLoop: Changing the Open Hybrid Cloud Game (Deploying OpenShift ...Shawn Wells
Microsoft and Red Hat have certified OpenShift Container Platform to run on Microsoft Azure. This talk steps through the reference architecture and ongoing work to accelerate government ATOs.
How GitLab and HackerOne help organizations innovate faster without compromis...HackerOne
In this webinar, GitLab’s Product Manager, Victor Wu, dives into how GitLab helps you ship secure code, the tools they use, and a few industry best practices they follow to protect data and secrets. Then, GitLab Security Lead, Brian Neel, will explain how they leverage their community using HackerOne to spot and prioritize security issues quickly.
How to make the agile team work with security requirements? To get secure coding practices into agile development is often hard work. A security functional requirement might be included in the sprint, but to get secure testing, secure architecture and feedback of security incidents working is not an easy talk for many agile teams. In my role as Scrum Master and security consultant I have developed a recipe of 7 steps that I will present to you. Where we will talk about agile secure development, agile threat modelling, agile security testing and agile workflows with security. Many of the steps can be made without costly tools, and I will present open source alternatives for all steps. This to make a test easier and to get a lower startup of your teams security process.
JSCONF 2018 - Baking security into DevOps - a tale of hunting down bugs befor...Wouter Bloeyaert
Do you know that 90% of all vulnerabilities can be prevented by introducing security in every step of your software development lifecycle (SDLC)? Get ready to join Wouter on his journey on how he introduced security into the SDLC at a company.
During his talk, Wouter will introduce you to how development, operations and security can be fitted together into “SecDevOps”.
The talk uses practical examples so that you will be able to experiment with “SecDevOps” yourself and know what you should pay attention to when implementing this into your own SDLC.
Application Security in an Agile World - Agile Singapore 2016Stefan Streichsbier
This document discusses application security in an agile development world. It begins with a brief history of application security and defines it as a quality aspect that contributes to business success like user experience and performance. Application security was traditionally handled by network teams but is now the responsibility of developers. The document advocates for adopting a DevSecOps approach where security is integrated into the development process through activities like threat modeling, design reviews, security testing, and monitoring. This allows catching issues earlier in the development cycle when they are cheaper to fix. The document provides examples of how to incorporate security into agile frameworks like Scrum.
Talk about application security in an agile world. How can security be integrated into agile and how can DevSecOps be leveraged to achieve security at scale at speed.
Why to DevSecOps - Introduction of Bangalore Azure Group meetupSébastien Paulet
The document discusses DevSecOps, which combines development, operations, and security to build security into the development lifecycle. It advocates for building security in from the start rather than bolting it on later. DevSecOps aims to automate security, empower development teams with security responsibilities, provide continuous security assurance across the development process, and establish operational security best practices. The document provides examples of DevSecOps tools and practices that can be used on the Microsoft Azure cloud platform, such as static analysis tools, security testing integrations, and managing open source usage.
When performing a security testing, I often sit in a room with other QA and Software testers.
During that time, it is likely I receive questions such as: "Roberto, are you hacking this? Are you breaking
this again? What exactly are you testing?"
Whi l e talking to them I realise there is an information gap between us, especially when they share
information which is essential for my testing and crucial to identify security vulnerabilities.
After a good number of security tests, I came to a conclusion that people in our industry do not realise that
software testing and security testing have a lot to share.
This talk intends to reduce that information gap and provides an introduction to security software testing,
methodologies, and most importantly offers some food for thought to stimulate synergy between security
and software testers
2017-07-11 GovLoop: Changing the Open Hybrid Cloud Game (Deploying OpenShift ...Shawn Wells
Microsoft and Red Hat have certified OpenShift Container Platform to run on Microsoft Azure. This talk steps through the reference architecture and ongoing work to accelerate government ATOs.
How GitLab and HackerOne help organizations innovate faster without compromis...HackerOne
In this webinar, GitLab’s Product Manager, Victor Wu, dives into how GitLab helps you ship secure code, the tools they use, and a few industry best practices they follow to protect data and secrets. Then, GitLab Security Lead, Brian Neel, will explain how they leverage their community using HackerOne to spot and prioritize security issues quickly.
How to make the agile team work with security requirements? To get secure coding practices into agile development is often hard work. A security functional requirement might be included in the sprint, but to get secure testing, secure architecture and feedback of security incidents working is not an easy talk for many agile teams. In my role as Scrum Master and security consultant I have developed a recipe of 7 steps that I will present to you. Where we will talk about agile secure development, agile threat modelling, agile security testing and agile workflows with security. Many of the steps can be made without costly tools, and I will present open source alternatives for all steps. This to make a test easier and to get a lower startup of your teams security process.
JSCONF 2018 - Baking security into DevOps - a tale of hunting down bugs befor...Wouter Bloeyaert
Do you know that 90% of all vulnerabilities can be prevented by introducing security in every step of your software development lifecycle (SDLC)? Get ready to join Wouter on his journey on how he introduced security into the SDLC at a company.
During his talk, Wouter will introduce you to how development, operations and security can be fitted together into “SecDevOps”.
The talk uses practical examples so that you will be able to experiment with “SecDevOps” yourself and know what you should pay attention to when implementing this into your own SDLC.
Application Security in an Agile World - Agile Singapore 2016Stefan Streichsbier
This document discusses application security in an agile development world. It begins with a brief history of application security and defines it as a quality aspect that contributes to business success like user experience and performance. Application security was traditionally handled by network teams but is now the responsibility of developers. The document advocates for adopting a DevSecOps approach where security is integrated into the development process through activities like threat modeling, design reviews, security testing, and monitoring. This allows catching issues earlier in the development cycle when they are cheaper to fix. The document provides examples of how to incorporate security into agile frameworks like Scrum.
Talk about application security in an agile world. How can security be integrated into agile and how can DevSecOps be leveraged to achieve security at scale at speed.
Why to DevSecOps - Introduction of Bangalore Azure Group meetupSébastien Paulet
The document discusses DevSecOps, which combines development, operations, and security to build security into the development lifecycle. It advocates for building security in from the start rather than bolting it on later. DevSecOps aims to automate security, empower development teams with security responsibilities, provide continuous security assurance across the development process, and establish operational security best practices. The document provides examples of DevSecOps tools and practices that can be used on the Microsoft Azure cloud platform, such as static analysis tools, security testing integrations, and managing open source usage.
When performing a security testing, I often sit in a room with other QA and Software testers.
During that time, it is likely I receive questions such as: "Roberto, are you hacking this? Are you breaking
this again? What exactly are you testing?"
Whi l e talking to them I realise there is an information gap between us, especially when they share
information which is essential for my testing and crucial to identify security vulnerabilities.
After a good number of security tests, I came to a conclusion that people in our industry do not realise that
software testing and security testing have a lot to share.
This talk intends to reduce that information gap and provides an introduction to security software testing,
methodologies, and most importantly offers some food for thought to stimulate synergy between security
and software testers
DevSecOps for Developers, How To Start (ETC 2020)Patricia Aas
How can you squeeze Security into DevOps? Security is often an understaffed function, so how can you leverage what you have in DevOps to improve your security posture?
Often the culture clash between Security and Development is even more prominent than between Development and Operations. Understanding the differences in how these functions work, and leveraging their similarities, will reveal processes already in place that can be used to improve security. This fine tuning of tools and processes can give you DevSecOps on a shoestring.
In un processo di sviluppo software vengono utilizzate migliaia di componenti open source e di proprietà. Sebbene questa pratica sia volta alla produzione di sviluppo software di alta qualità nel modo più efficiente possibile, basta dare un’occhiata alle statistiche per capire che la realtà è ben diversa.
Il webinar illustra lo stato attuale della supply chain del software secondo la ricerca condotta nel 2016 da Nexus Sonatype su 3.000 aziende e più di 25.000 applicazioni.
Beyond Agile Testing to Lean Development — Rakuten Technology ConferenceJames Coplien
The document discusses moving beyond traditional agile testing approaches to lean testing. It argues that most unit tests are unnecessary and test scenarios that will never occur. It promotes exploratory and experience-based testing over unit testing alone. The document also advocates for shipping tests with code to catch bugs in the field, using assertions to make code more readable, and taking a lean approach where fixing testing processes is prioritized over fixing individual bugs.
What Every Developer And Tester Should Know About Software SecurityAnne Oikarinen
The document discusses what software developers and testers should know about software security. It emphasizes the importance of threat modeling to understand potential threats, creating security requirements, and including security testing in the development process. It provides examples of security best practices like checking for vulnerabilities, conducting code reviews, and penetration testing applications to find issues before attackers. The goal is to integrate security practices into development rather than as an afterthought.
CyberSecurity - Future Risks, Zero Trust and the Optus Data Leak.pdfRoger Qiu
https://www.meetup.com/cybersecurity-digital-trust/
https://www.meetup.com/cybersecurity-digital-trust/events/289368916/
Hi All, Let's get together and talk Cyber Security + enjoy free pizza!
We're professionals in the technology space, and we're starting a new meetup to address modern cybersecurity challenges.
Through these events, we aim to:
- Share ideas, best practices, and case studies
- Present the latest risks and developments in cybersecurity
- Practice cybersecurity problems and solutions
- Support each other via networking opportunities and broadening our business understanding of related topics
For this initial meetup, we will be covering a technical analysis of the 2022 Optus data leak, and the corporate impact this is having on digital governance in Australia.
And we will expand the conversation to zero-trust systems, and some recent developments in the cybersecurity space such as webauthn, web3, secrets management... etc.
Potential topics for future events:
* centralisation/decentralisation of digital assets
* increasing risk due to secret sprawl
* software supply chain security
* zero-trust, trustless systems and how we got here from Snowden NSA leaks
* case studies like the 2022 Optus data leak
* identity fraud and developments in digital identity
We hope this will create a community for professionals to share ideas and tips on how companies can improve their capabilities and most importantly create a safe and fun environment for everyone.
The document outlines 12 crucial Windows security skills for 2018 according to security experts at CQURE. The skills are organized into 12 groups: 1) Platform Security & Internals, 2) Attacks On Credentials & Prevention Solutions, 3) PowerShell As A Hacking Tool, 4) Office 365 Security, 5) Raising the bar for malware, 6) Microsoft SQL Server Security, 7) Improving security with Azure, 8) Virtualization based security, 9) Machine Learning for Security, 10) Windows 2016 security and infrastructure improvements, 11) Practical Public Key Infrastructure, and 12) Advanced Monitoring and Auditing. The document provides brief descriptions of the types of skills covered in each group.
Applying formal methods to existing software by B.MonateMahaut Gouhier
"Applying formal methods to existing software: what can you expect?" Talk by Benjamin Monate, Co-founder and CTO of TrustInSoft, at the 2018 Sound Static Analysis for Security Workshop, in the NIST, USA, on June 27th.
This work has been supported by the Core Infrastructure Initiative of the Linux foundation.
Learn more about TrustInSoft
https://trust-in-soft.com/
Automated verification is becoming increasingly important. Getting a product from idea to customer as fast as possible in a Continuous Delivery, or a Deployment pipeline is crucial in more businesses than ever before. But how do we get a product through that pipe line, with high quality? Kristian will talk about how automated verification can get you there.
Multipoint Conferencing Unit Comparative StudyVideoguy
The Polycom MCU outperformed the other MCUs in the comparative study. It passed 63 out of 63 test cases, while the RADVISION viaIP 400 MCU passed 14 cases, the TANDBERG MCU 16+16 passed 5 cases, and the TANDBERG MPS passed 7 cases. The Polycom MCU demonstrated the most complete set of security, versatility, and administration features. It provided the largest and most flexible set of options to successfully complete three example use cases.
In the past 5 years Continuous Delivery has gained much attention. Its benefits of rapid, iterative change are well understood, all the way up to board level. However, CD often encounters an adversary; Security. Protection of data and computer systems seems to stand on concepts like infrequent change, segregation of duties and bureaucratic heavyweight process. But are CD and Security really at odds?
We don’t think so. Whilst we’ll show you the dangers of unfettered CD pipelines and the risk of letting security spread fear. We will also share ways in which we’ve managed to balance speed and security in our pipelines–considering both the technical and organisational aspects. In fact we hope you’ll see that not only is there a way, but it’s a far better way.
Foundations of Software Testing Lecture 4Iosif Itkin
This lecture is a part of the online course on Software Testing for Complex Intelligent Systems and Autonomous Vehicles. The course lectures provide the theoretical basics of testing autonomous systems based on artificial intelligence.
The fourth lecture of the course entitled Foundations of Software Testing reviews the ‘absence-of-errors fallacy’ and other principles of software testing, as well as the types and levels of software testing. The lecture also provides a fuller picture of the understanding of test objectives and methodologies by different schools of thought within the software testing domain.
Agile and Continuous Delivery for Audits and Exams - DC Continuous Delivery M...Simon Storm
The document provides tips for auditors and examiners to better prepare for audits when using agile and continuous delivery practices. It recommends socializing plans for process changes to avoid surprises. It also suggests demonstrating expertise in agile and continuous delivery through training and certifications. Continuous delivery practices like infrastructure as code, static code analysis, automated testing, and repository management make the development process more auditable. The tips emphasize digitizing documentation, logging pipeline activity, and capturing metrics to demonstrate maturity. Ensuring quality practices like keeping QA involved, logging deployments, and code reviews with pull requests can help pass audits. Finally, it recommends getting ahead of outstanding risks like access controls and separation of duties when using tools like Jenkins and Git
For many DevOps teams, their CI server has become just like the young shepherd-boy in the famous Aesop fable where he would cry wolf to falsely alarm the village to save their sheep. When the build is broken due to flaky automated tests, the team loses faith in the tests. How do we build the trust?
Are Your Continuous Tests Too Fragile for Agile?Parasoft
With a fragile test suite, the Continuous Testing that's vital to Agile just isn't feasible. If you truly want to automate the execution of a broad test suite—embracing unit, component, integration, functional, performance, and security testing—during continuous integration, you need to ensure that your test suite is up to the task. How do you achieve this? This presentation provides tips on ensuring that your tests are up to the task.
Fuzzing is a software testing technique that feeds random data to a program to test for crashes or security vulnerabilities. It can find bugs that other testing methods may miss by exploring unusual code paths. While fuzzing is effective at finding bugs, it only finds issues and does not evaluate the quality or reliability of the software. Code coverage metrics can be used alongside fuzzing to measure how thoroughly the code has been tested, but may still miss some bugs. Fuzzing works best when the tester has knowledge of the program's internal structure and algorithms.
This document discusses running acceptance tests as monitors in production environments. It describes why this can be useful for failure detection, capacity planning, and gaining business insights. However, care must be taken to avoid polluting data or sending unnecessary alerts. The author demonstrates a tool called atam4j that treats acceptance tests as a microservice, similar to other applications, making them easier to deploy and monitor in production.
How to Better Manage Technical Debt While Innovating on DevOpsDynatrace
Forget the “Unicorns.” There is a lot to learn from “DevOps Unicorns” such as Etsy or Facebook, but for enterprises dealing with technical debt in legacy systems developed by teams no longer with the company, copying the unicorns is not an option.
Richard Dominguez, Operations Developer at Prep Sportswear, needed to “keep the lights on” for their legacy systems, while enabling his DevOps teams to launch new features much faster. Today Prep Sportswear releases more updates to their legacy systems than ever before by reducing MTTR (Mean Time To Repair), giving them more time to innovate on DevOps and Continuous Delivery on their new platform. You’ll learn:
• Top metrics for an Ops dashboard to catch potential issues early
• Tips to manage technical debt in legacy code caused by dev teams long gone
• Efficient ways to close loops while providing input to DevOps so they can optimize innovation and releases
What is testing?
What is agile testing?
What is automated testing?
What is agile testing?
Unit testing
Mock testing
Functional testing
Acceptance testing
Integration testing
Performance/load/stress testing
Deployment testing
Methods of testing
White/black/grayboxtesting
GUI vs. businesslogictesting
Improving code testability
Codefacing vs. businessfacingtesting
Smoke testing
Automated testing strategies
Virtualization
Code coverage
Resources
File Can be downloaded from:
http://community.scmgalaxy.com/
This document discusses challenges with integrating security into agile development processes and proposes solutions. It notes that traditional security approaches like threat modeling and penetration testing don't work well in agile environments with short release cycles. The document recommends automating security scans and tests to run with each code change. It also suggests integrating security findings into existing bug tracking tools to streamline remediation. The overall goal is to make security practices more agile and collaborative to improve cycle times for fixing issues.
The DevOps movement provides guidance on better ways to deliver working software to production in a fast, safe and automated manner. Is releasing new features every few weeks still a good approach? How quickly can you get a critical bug fix into production? Are you continually learning and improving? This talked is based on books such as The DevOps Handbook, Release It, and from real world experience delivering "mission critical" microservices in high volume production environments. Come learn how to increase profitability, improve culture, and exceed productivity goals through DevOps practices. We've all messed up releases. Learn from it. Embrace. Improve!
Unit testing has entered the main stream. It is generally considered best practice to have a high level of unit test code coverage, and to ideally write tests before the code, via Test Driven Development.
However, some code is just plain difficult to test. The cost of effort of adding the tests may seem to outweigh the benefits. In this session, we will do a quick review of the benefits of unit tests, but focus on how to test tricky code, such as that static and private methods, and legacy code in general.
Examples are in Java, but the principals are language agnostic.
DevSecOps for Developers, How To Start (ETC 2020)Patricia Aas
How can you squeeze Security into DevOps? Security is often an understaffed function, so how can you leverage what you have in DevOps to improve your security posture?
Often the culture clash between Security and Development is even more prominent than between Development and Operations. Understanding the differences in how these functions work, and leveraging their similarities, will reveal processes already in place that can be used to improve security. This fine tuning of tools and processes can give you DevSecOps on a shoestring.
In un processo di sviluppo software vengono utilizzate migliaia di componenti open source e di proprietà. Sebbene questa pratica sia volta alla produzione di sviluppo software di alta qualità nel modo più efficiente possibile, basta dare un’occhiata alle statistiche per capire che la realtà è ben diversa.
Il webinar illustra lo stato attuale della supply chain del software secondo la ricerca condotta nel 2016 da Nexus Sonatype su 3.000 aziende e più di 25.000 applicazioni.
Beyond Agile Testing to Lean Development — Rakuten Technology ConferenceJames Coplien
The document discusses moving beyond traditional agile testing approaches to lean testing. It argues that most unit tests are unnecessary and test scenarios that will never occur. It promotes exploratory and experience-based testing over unit testing alone. The document also advocates for shipping tests with code to catch bugs in the field, using assertions to make code more readable, and taking a lean approach where fixing testing processes is prioritized over fixing individual bugs.
What Every Developer And Tester Should Know About Software SecurityAnne Oikarinen
The document discusses what software developers and testers should know about software security. It emphasizes the importance of threat modeling to understand potential threats, creating security requirements, and including security testing in the development process. It provides examples of security best practices like checking for vulnerabilities, conducting code reviews, and penetration testing applications to find issues before attackers. The goal is to integrate security practices into development rather than as an afterthought.
CyberSecurity - Future Risks, Zero Trust and the Optus Data Leak.pdfRoger Qiu
https://www.meetup.com/cybersecurity-digital-trust/
https://www.meetup.com/cybersecurity-digital-trust/events/289368916/
Hi All, Let's get together and talk Cyber Security + enjoy free pizza!
We're professionals in the technology space, and we're starting a new meetup to address modern cybersecurity challenges.
Through these events, we aim to:
- Share ideas, best practices, and case studies
- Present the latest risks and developments in cybersecurity
- Practice cybersecurity problems and solutions
- Support each other via networking opportunities and broadening our business understanding of related topics
For this initial meetup, we will be covering a technical analysis of the 2022 Optus data leak, and the corporate impact this is having on digital governance in Australia.
And we will expand the conversation to zero-trust systems, and some recent developments in the cybersecurity space such as webauthn, web3, secrets management... etc.
Potential topics for future events:
* centralisation/decentralisation of digital assets
* increasing risk due to secret sprawl
* software supply chain security
* zero-trust, trustless systems and how we got here from Snowden NSA leaks
* case studies like the 2022 Optus data leak
* identity fraud and developments in digital identity
We hope this will create a community for professionals to share ideas and tips on how companies can improve their capabilities and most importantly create a safe and fun environment for everyone.
The document outlines 12 crucial Windows security skills for 2018 according to security experts at CQURE. The skills are organized into 12 groups: 1) Platform Security & Internals, 2) Attacks On Credentials & Prevention Solutions, 3) PowerShell As A Hacking Tool, 4) Office 365 Security, 5) Raising the bar for malware, 6) Microsoft SQL Server Security, 7) Improving security with Azure, 8) Virtualization based security, 9) Machine Learning for Security, 10) Windows 2016 security and infrastructure improvements, 11) Practical Public Key Infrastructure, and 12) Advanced Monitoring and Auditing. The document provides brief descriptions of the types of skills covered in each group.
Applying formal methods to existing software by B.MonateMahaut Gouhier
"Applying formal methods to existing software: what can you expect?" Talk by Benjamin Monate, Co-founder and CTO of TrustInSoft, at the 2018 Sound Static Analysis for Security Workshop, in the NIST, USA, on June 27th.
This work has been supported by the Core Infrastructure Initiative of the Linux foundation.
Learn more about TrustInSoft
https://trust-in-soft.com/
Automated verification is becoming increasingly important. Getting a product from idea to customer as fast as possible in a Continuous Delivery, or a Deployment pipeline is crucial in more businesses than ever before. But how do we get a product through that pipe line, with high quality? Kristian will talk about how automated verification can get you there.
Multipoint Conferencing Unit Comparative StudyVideoguy
The Polycom MCU outperformed the other MCUs in the comparative study. It passed 63 out of 63 test cases, while the RADVISION viaIP 400 MCU passed 14 cases, the TANDBERG MCU 16+16 passed 5 cases, and the TANDBERG MPS passed 7 cases. The Polycom MCU demonstrated the most complete set of security, versatility, and administration features. It provided the largest and most flexible set of options to successfully complete three example use cases.
In the past 5 years Continuous Delivery has gained much attention. Its benefits of rapid, iterative change are well understood, all the way up to board level. However, CD often encounters an adversary; Security. Protection of data and computer systems seems to stand on concepts like infrequent change, segregation of duties and bureaucratic heavyweight process. But are CD and Security really at odds?
We don’t think so. Whilst we’ll show you the dangers of unfettered CD pipelines and the risk of letting security spread fear. We will also share ways in which we’ve managed to balance speed and security in our pipelines–considering both the technical and organisational aspects. In fact we hope you’ll see that not only is there a way, but it’s a far better way.
Foundations of Software Testing Lecture 4Iosif Itkin
This lecture is a part of the online course on Software Testing for Complex Intelligent Systems and Autonomous Vehicles. The course lectures provide the theoretical basics of testing autonomous systems based on artificial intelligence.
The fourth lecture of the course entitled Foundations of Software Testing reviews the ‘absence-of-errors fallacy’ and other principles of software testing, as well as the types and levels of software testing. The lecture also provides a fuller picture of the understanding of test objectives and methodologies by different schools of thought within the software testing domain.
Agile and Continuous Delivery for Audits and Exams - DC Continuous Delivery M...Simon Storm
The document provides tips for auditors and examiners to better prepare for audits when using agile and continuous delivery practices. It recommends socializing plans for process changes to avoid surprises. It also suggests demonstrating expertise in agile and continuous delivery through training and certifications. Continuous delivery practices like infrastructure as code, static code analysis, automated testing, and repository management make the development process more auditable. The tips emphasize digitizing documentation, logging pipeline activity, and capturing metrics to demonstrate maturity. Ensuring quality practices like keeping QA involved, logging deployments, and code reviews with pull requests can help pass audits. Finally, it recommends getting ahead of outstanding risks like access controls and separation of duties when using tools like Jenkins and Git
For many DevOps teams, their CI server has become just like the young shepherd-boy in the famous Aesop fable where he would cry wolf to falsely alarm the village to save their sheep. When the build is broken due to flaky automated tests, the team loses faith in the tests. How do we build the trust?
Are Your Continuous Tests Too Fragile for Agile?Parasoft
With a fragile test suite, the Continuous Testing that's vital to Agile just isn't feasible. If you truly want to automate the execution of a broad test suite—embracing unit, component, integration, functional, performance, and security testing—during continuous integration, you need to ensure that your test suite is up to the task. How do you achieve this? This presentation provides tips on ensuring that your tests are up to the task.
Fuzzing is a software testing technique that feeds random data to a program to test for crashes or security vulnerabilities. It can find bugs that other testing methods may miss by exploring unusual code paths. While fuzzing is effective at finding bugs, it only finds issues and does not evaluate the quality or reliability of the software. Code coverage metrics can be used alongside fuzzing to measure how thoroughly the code has been tested, but may still miss some bugs. Fuzzing works best when the tester has knowledge of the program's internal structure and algorithms.
This document discusses running acceptance tests as monitors in production environments. It describes why this can be useful for failure detection, capacity planning, and gaining business insights. However, care must be taken to avoid polluting data or sending unnecessary alerts. The author demonstrates a tool called atam4j that treats acceptance tests as a microservice, similar to other applications, making them easier to deploy and monitor in production.
How to Better Manage Technical Debt While Innovating on DevOpsDynatrace
Forget the “Unicorns.” There is a lot to learn from “DevOps Unicorns” such as Etsy or Facebook, but for enterprises dealing with technical debt in legacy systems developed by teams no longer with the company, copying the unicorns is not an option.
Richard Dominguez, Operations Developer at Prep Sportswear, needed to “keep the lights on” for their legacy systems, while enabling his DevOps teams to launch new features much faster. Today Prep Sportswear releases more updates to their legacy systems than ever before by reducing MTTR (Mean Time To Repair), giving them more time to innovate on DevOps and Continuous Delivery on their new platform. You’ll learn:
• Top metrics for an Ops dashboard to catch potential issues early
• Tips to manage technical debt in legacy code caused by dev teams long gone
• Efficient ways to close loops while providing input to DevOps so they can optimize innovation and releases
What is testing?
What is agile testing?
What is automated testing?
What is agile testing?
Unit testing
Mock testing
Functional testing
Acceptance testing
Integration testing
Performance/load/stress testing
Deployment testing
Methods of testing
White/black/grayboxtesting
GUI vs. businesslogictesting
Improving code testability
Codefacing vs. businessfacingtesting
Smoke testing
Automated testing strategies
Virtualization
Code coverage
Resources
File Can be downloaded from:
http://community.scmgalaxy.com/
This document discusses challenges with integrating security into agile development processes and proposes solutions. It notes that traditional security approaches like threat modeling and penetration testing don't work well in agile environments with short release cycles. The document recommends automating security scans and tests to run with each code change. It also suggests integrating security findings into existing bug tracking tools to streamline remediation. The overall goal is to make security practices more agile and collaborative to improve cycle times for fixing issues.
The DevOps movement provides guidance on better ways to deliver working software to production in a fast, safe and automated manner. Is releasing new features every few weeks still a good approach? How quickly can you get a critical bug fix into production? Are you continually learning and improving? This talked is based on books such as The DevOps Handbook, Release It, and from real world experience delivering "mission critical" microservices in high volume production environments. Come learn how to increase profitability, improve culture, and exceed productivity goals through DevOps practices. We've all messed up releases. Learn from it. Embrace. Improve!
Unit testing has entered the main stream. It is generally considered best practice to have a high level of unit test code coverage, and to ideally write tests before the code, via Test Driven Development.
However, some code is just plain difficult to test. The cost of effort of adding the tests may seem to outweigh the benefits. In this session, we will do a quick review of the benefits of unit tests, but focus on how to test tricky code, such as that static and private methods, and legacy code in general.
Examples are in Java, but the principals are language agnostic.
The microservice architectural style is an approach to developing an application as a suite of small services that each can be independently developed and deployed. In this talk, we will cover the pros and cons of microservices, including contrasting them with the more traditional 'monolithic' application. We will also dive into the most common mechanism used to expose the functionality of a microservice. REST is an architecture style for building scalable web services. You've at least heard of it, you may have contributed to or even created 'RESTful' applications, but are you familiar with the basic constraints that make up REST? We'll cover the theory behind REST before diving into pragmatic implementation styles and better practices.
Rest and Microservices at the Las Vegas Dot Net GroupShaun Abram
The microservice architectural style is an approach to developing an application as a suite of small services. Each can be independently developed and deployed. This presentation covers the pros and cons of microservices, including contrasting with the more traditional 'monolithic' application. We also dive into the most common mechanism used to expose their functionality: RESTful APIs, including a discussion of HTTP and its components.
1) The document discusses microservices and REST architectures. It defines microservices as small, focused pieces of software that are independently developed and deployed.
2) REST is described as an architectural style using HTTP as a stateless protocol and uniform interfaces to access resources. The key constraints of REST like client-server, statelessness and cacheability are explained.
3) The document advocates for building microservices that expose functionality through RESTful APIs and HTTP to allow independent development and deployment of services.
Software quality is critical to consistently and continually delivering new features to our users. This talk covers the importance of software quality and how to deliver it via unit testing, Test Driven Development and clean code in general.
This is the deck from a talk I gave at Desert Code Camp 2013.
The Microsoft 365 Migration Tutorial For Beginner.pptxoperationspcvita
This presentation will help you understand the power of Microsoft 365. However, we have mentioned every productivity app included in Office 365. Additionally, we have suggested the migration situation related to Office 365 and how we can help you.
You can also read: https://www.systoolsgroup.com/updates/office-365-tenant-to-tenant-migration-step-by-step-complete-guide/
Monitoring and Managing Anomaly Detection on OpenShift.pdfTosin Akinosho
Monitoring and Managing Anomaly Detection on OpenShift
Overview
Dive into the world of anomaly detection on edge devices with our comprehensive hands-on tutorial. This SlideShare presentation will guide you through the entire process, from data collection and model training to edge deployment and real-time monitoring. Perfect for those looking to implement robust anomaly detection systems on resource-constrained IoT/edge devices.
Key Topics Covered
1. Introduction to Anomaly Detection
- Understand the fundamentals of anomaly detection and its importance in identifying unusual behavior or failures in systems.
2. Understanding Edge (IoT)
- Learn about edge computing and IoT, and how they enable real-time data processing and decision-making at the source.
3. What is ArgoCD?
- Discover ArgoCD, a declarative, GitOps continuous delivery tool for Kubernetes, and its role in deploying applications on edge devices.
4. Deployment Using ArgoCD for Edge Devices
- Step-by-step guide on deploying anomaly detection models on edge devices using ArgoCD.
5. Introduction to Apache Kafka and S3
- Explore Apache Kafka for real-time data streaming and Amazon S3 for scalable storage solutions.
6. Viewing Kafka Messages in the Data Lake
- Learn how to view and analyze Kafka messages stored in a data lake for better insights.
7. What is Prometheus?
- Get to know Prometheus, an open-source monitoring and alerting toolkit, and its application in monitoring edge devices.
8. Monitoring Application Metrics with Prometheus
- Detailed instructions on setting up Prometheus to monitor the performance and health of your anomaly detection system.
9. What is Camel K?
- Introduction to Camel K, a lightweight integration framework built on Apache Camel, designed for Kubernetes.
10. Configuring Camel K Integrations for Data Pipelines
- Learn how to configure Camel K for seamless data pipeline integrations in your anomaly detection workflow.
11. What is a Jupyter Notebook?
- Overview of Jupyter Notebooks, an open-source web application for creating and sharing documents with live code, equations, visualizations, and narrative text.
12. Jupyter Notebooks with Code Examples
- Hands-on examples and code snippets in Jupyter Notebooks to help you implement and test anomaly detection models.
For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2024/06/temporal-event-neural-networks-a-more-efficient-alternative-to-the-transformer-a-presentation-from-brainchip/
Chris Jones, Director of Product Management at BrainChip , presents the “Temporal Event Neural Networks: A More Efficient Alternative to the Transformer” tutorial at the May 2024 Embedded Vision Summit.
The expansion of AI services necessitates enhanced computational capabilities on edge devices. Temporal Event Neural Networks (TENNs), developed by BrainChip, represent a novel and highly efficient state-space network. TENNs demonstrate exceptional proficiency in handling multi-dimensional streaming data, facilitating advancements in object detection, action recognition, speech enhancement and language model/sequence generation. Through the utilization of polynomial-based continuous convolutions, TENNs streamline models, expedite training processes and significantly diminish memory requirements, achieving notable reductions of up to 50x in parameters and 5,000x in energy consumption compared to prevailing methodologies like transformers.
Integration with BrainChip’s Akida neuromorphic hardware IP further enhances TENNs’ capabilities, enabling the realization of highly capable, portable and passively cooled edge devices. This presentation delves into the technical innovations underlying TENNs, presents real-world benchmarks, and elucidates how this cutting-edge approach is positioned to revolutionize edge AI across diverse applications.
Freshworks Rethinks NoSQL for Rapid Scaling & Cost-EfficiencyScyllaDB
Freshworks creates AI-boosted business software that helps employees work more efficiently and effectively. Managing data across multiple RDBMS and NoSQL databases was already a challenge at their current scale. To prepare for 10X growth, they knew it was time to rethink their database strategy. Learn how they architected a solution that would simplify scaling while keeping costs under control.
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectorsDianaGray10
Join us to learn how UiPath Apps can directly and easily interact with prebuilt connectors via Integration Service--including Salesforce, ServiceNow, Open GenAI, and more.
The best part is you can achieve this without building a custom workflow! Say goodbye to the hassle of using separate automations to call APIs. By seamlessly integrating within App Studio, you can now easily streamline your workflow, while gaining direct access to our Connector Catalog of popular applications.
We’ll discuss and demo the benefits of UiPath Apps and connectors including:
Creating a compelling user experience for any software, without the limitations of APIs.
Accelerating the app creation process, saving time and effort
Enjoying high-performance CRUD (create, read, update, delete) operations, for
seamless data management.
Speakers:
Russell Alfeche, Technology Leader, RPA at qBotic and UiPath MVP
Charlie Greenberg, host
Digital Marketing Trends in 2024 | Guide for Staying AheadWask
https://www.wask.co/ebooks/digital-marketing-trends-in-2024
Feeling lost in the digital marketing whirlwind of 2024? Technology is changing, consumer habits are evolving, and staying ahead of the curve feels like a never-ending pursuit. This e-book is your compass. Dive into actionable insights to handle the complexities of modern marketing. From hyper-personalization to the power of user-generated content, learn how to build long-term relationships with your audience and unlock the secrets to success in the ever-shifting digital landscape.
Programming Foundation Models with DSPy - Meetup SlidesZilliz
Prompting language models is hard, while programming language models is easy. In this talk, I will discuss the state-of-the-art framework DSPy for programming foundation models with its powerful optimizers and runtime constraint system.
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdfChart Kalyan
A Mix Chart displays historical data of numbers in a graphical or tabular form. The Kalyan Rajdhani Mix Chart specifically shows the results of a sequence of numbers over different periods.
FREE A4 Cyber Security Awareness Posters-Social Engineering part 3Data Hops
Free A4 downloadable and printable Cyber Security, Social Engineering Safety and security Training Posters . Promote security awareness in the home or workplace. Lock them Out From training providers datahops.com
In the realm of cybersecurity, offensive security practices act as a critical shield. By simulating real-world attacks in a controlled environment, these techniques expose vulnerabilities before malicious actors can exploit them. This proactive approach allows manufacturers to identify and fix weaknesses, significantly enhancing system security.
This presentation delves into the development of a system designed to mimic Galileo's Open Service signal using software-defined radio (SDR) technology. We'll begin with a foundational overview of both Global Navigation Satellite Systems (GNSS) and the intricacies of digital signal processing.
The presentation culminates in a live demonstration. We'll showcase the manipulation of Galileo's Open Service pilot signal, simulating an attack on various software and hardware systems. This practical demonstration serves to highlight the potential consequences of unaddressed vulnerabilities, emphasizing the importance of offensive security practices in safeguarding critical infrastructure.
HCL Notes and Domino License Cost Reduction in the World of DLAUpanagenda
Webinar Recording: https://www.panagenda.com/webinars/hcl-notes-and-domino-license-cost-reduction-in-the-world-of-dlau/
The introduction of DLAU and the CCB & CCX licensing model caused quite a stir in the HCL community. As a Notes and Domino customer, you may have faced challenges with unexpected user counts and license costs. You probably have questions on how this new licensing approach works and how to benefit from it. Most importantly, you likely have budget constraints and want to save money where possible. Don’t worry, we can help with all of this!
We’ll show you how to fix common misconfigurations that cause higher-than-expected user counts, and how to identify accounts which you can deactivate to save money. There are also frequent patterns that can cause unnecessary cost, like using a person document instead of a mail-in for shared mailboxes. We’ll provide examples and solutions for those as well. And naturally we’ll explain the new licensing model.
Join HCL Ambassador Marc Thomas in this webinar with a special guest appearance from Franz Walder. It will give you the tools and know-how to stay on top of what is going on with Domino licensing. You will be able lower your cost through an optimized configuration and keep it low going forward.
These topics will be covered
- Reducing license cost by finding and fixing misconfigurations and superfluous accounts
- How do CCB and CCX licenses really work?
- Understanding the DLAU tool and how to best utilize it
- Tips for common problem areas, like team mailboxes, functional/test users, etc
- Practical examples and best practices to implement right away
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAUpanagenda
Webinar Recording: https://www.panagenda.com/webinars/hcl-notes-und-domino-lizenzkostenreduzierung-in-der-welt-von-dlau/
DLAU und die Lizenzen nach dem CCB- und CCX-Modell sind für viele in der HCL-Community seit letztem Jahr ein heißes Thema. Als Notes- oder Domino-Kunde haben Sie vielleicht mit unerwartet hohen Benutzerzahlen und Lizenzgebühren zu kämpfen. Sie fragen sich vielleicht, wie diese neue Art der Lizenzierung funktioniert und welchen Nutzen sie Ihnen bringt. Vor allem wollen Sie sicherlich Ihr Budget einhalten und Kosten sparen, wo immer möglich. Das verstehen wir und wir möchten Ihnen dabei helfen!
Wir erklären Ihnen, wie Sie häufige Konfigurationsprobleme lösen können, die dazu führen können, dass mehr Benutzer gezählt werden als nötig, und wie Sie überflüssige oder ungenutzte Konten identifizieren und entfernen können, um Geld zu sparen. Es gibt auch einige Ansätze, die zu unnötigen Ausgaben führen können, z. B. wenn ein Personendokument anstelle eines Mail-Ins für geteilte Mailboxen verwendet wird. Wir zeigen Ihnen solche Fälle und deren Lösungen. Und natürlich erklären wir Ihnen das neue Lizenzmodell.
Nehmen Sie an diesem Webinar teil, bei dem HCL-Ambassador Marc Thomas und Gastredner Franz Walder Ihnen diese neue Welt näherbringen. Es vermittelt Ihnen die Tools und das Know-how, um den Überblick zu bewahren. Sie werden in der Lage sein, Ihre Kosten durch eine optimierte Domino-Konfiguration zu reduzieren und auch in Zukunft gering zu halten.
Diese Themen werden behandelt
- Reduzierung der Lizenzkosten durch Auffinden und Beheben von Fehlkonfigurationen und überflüssigen Konten
- Wie funktionieren CCB- und CCX-Lizenzen wirklich?
- Verstehen des DLAU-Tools und wie man es am besten nutzt
- Tipps für häufige Problembereiche, wie z. B. Team-Postfächer, Funktions-/Testbenutzer usw.
- Praxisbeispiele und Best Practices zum sofortigen Umsetzen
5th LF Energy Power Grid Model Meet-up SlidesDanBrown980551
5th Power Grid Model Meet-up
It is with great pleasure that we extend to you an invitation to the 5th Power Grid Model Meet-up, scheduled for 6th June 2024. This event will adopt a hybrid format, allowing participants to join us either through an online Mircosoft Teams session or in person at TU/e located at Den Dolech 2, Eindhoven, Netherlands. The meet-up will be hosted by Eindhoven University of Technology (TU/e), a research university specializing in engineering science & technology.
Power Grid Model
The global energy transition is placing new and unprecedented demands on Distribution System Operators (DSOs). Alongside upgrades to grid capacity, processes such as digitization, capacity optimization, and congestion management are becoming vital for delivering reliable services.
Power Grid Model is an open source project from Linux Foundation Energy and provides a calculation engine that is increasingly essential for DSOs. It offers a standards-based foundation enabling real-time power systems analysis, simulations of electrical power grids, and sophisticated what-if analysis. In addition, it enables in-depth studies and analysis of the electrical power grid’s behavior and performance. This comprehensive model incorporates essential factors such as power generation capacity, electrical losses, voltage levels, power flows, and system stability.
Power Grid Model is currently being applied in a wide variety of use cases, including grid planning, expansion, reliability, and congestion studies. It can also help in analyzing the impact of renewable energy integration, assessing the effects of disturbances or faults, and developing strategies for grid control and optimization.
What to expect
For the upcoming meetup we are organizing, we have an exciting lineup of activities planned:
-Insightful presentations covering two practical applications of the Power Grid Model.
-An update on the latest advancements in Power Grid -Model technology during the first and second quarters of 2024.
-An interactive brainstorming session to discuss and propose new feature requests.
-An opportunity to connect with fellow Power Grid Model enthusiasts and users.
7. Confidential
We need Observability in our systems
7
Everything is sometimes broken
Something is always broken
If nothing seems broken...
…your monitoring is broken
It’s impossible to predict the myriad states of partial failures we’ll see
15. Confidential 15
Testing in Production
Chaos
Engineering
Carefully planned experiments
designed to reveal weaknesses in our
systems
aka
Resilience Engineering
16. Confidential
Game Days
16
An exercise where we place systems
under stress to
learn and improve resilience
(And even just getting the team together to discuss resilience can be worthwhile)
17. Confidential
Chaos Engineering – a step by step guide
17
Hypothesis
(Steady state)
Minimize
Blast Radius
Run
Analyze
Increase
Repeat,
Automate
20. Confidential
Reading material
20
Chaos Engineering (free eBook)
https://www.oreilly.com/webops-perf/free/chaos-engineering.csp
Distributed Systems Observability (free eBook)
https://distributed-systems-observability-ebook.humio.com/
21. Confidential
Reading material
21
shaunabram.com
Principles of Chaos Engineering
principlesofchaos.org
How to run a Game Day
https://www.gremlin.com/community/tutorials/how-to-run-a-gameday/
Testing in production:
https://medium.com/@copyconstruct/testing-in-production-the-safe-way-18ca102d0ef1
Monitoring in the time of Cloud Native
https://medium.com/@copyconstruct/monitoring-in-the-time-of-cloud-native-c87c7a5bfa3e
Deploy != Release
https://blog.turbinelabs.io/deploy-not-equal-release-part-one-4724bc1e726b
Joke… or Good?
Non-prod = pale imitation, like mocks, or “it works on my machine”
Prod is different; 4th trimester…
“Testing in production”
You may have seen this meme before: The DosEquis guys saying
“I don’t always test, but when I do, I test in production”
“Testing in production” has been kind of a joke -> what you’re really saying is you don’t test anywhere.
And instead you’re just winging it: deploying to production and <CROSS FINGERS> hoping it all works.
But then I began to look at it differently.
The DosEquis guy usually says “I don’t always drink beer, but when I do, I drink DosEquis”
Meaning DosEquis is the best beer to drink.
So, the implication here is not that testing in production is a joke, but that Production is actually the BEST place to test.
And I’m increasingly believing that to be the case. Or, at least, that production is an environment we shouldn’t be ignoring for testing.
After all, prod is the only place your software has an impact on your customers is production.
But there has been this status quo of production being sacrosanct. Instead of testing there, it is common to keep a non-prod env, such as staging, as identical to production as possible, and test there.
Such environments are usually a pale imitation of production however.
Testing in staging is kind of like testing with mocks, an imitation, but not the real thing.
Saying “works in staging” is only one step better than “works on my machine”.
Production is a different beast!
I’ve heard of software being released to production as being like a baby’s 4th trimester.
When software leave’s it artificial environments and slams into the real world
But what makes the real world of production so special?
Serious question: In what ways in Production different from other environments?
Hardware & Cluster size, Data
Configuration, Traffic, Monitoring
Some things we can only test in production
As our architecture becomes more compilated (particularly with Microservices), we need to consider all options to allow us to test and deliver working software to our customer. Including testing in production.
So should we skip testing in non-prod first? No!
Testing in production is by no means a substitute for pre-production testing
I’ve given talks on
unit testing, integration testing
Mocks
About code coverage and Continuous Integration
I believe very firmly in all those things.
Testing in Production is an addition to all those.
Most production testing is really validation only – although there is at least one exception (A/B testing)
Respect production
Beware of unwanted side effects
Stateless services are good candidates
Think SAFE methods e.g. GET, HEAD
Consider testing using expected failures of others e.g. PUT that results in 400 error (still tells you something)
Or at least be able to tell the difference between test data and “real” prod data
Today we’re going to cover some of the different ways we can test in production
We’ll start with Observability, the foundation for any testing in production. Observability = Knowing what the heck your app is doing anyway. Going beyond just logs and alerting
Around deployment & release times
Chaos engineering. Perhaps the most advanced form of production testing, but I would argue its actually not that advanced. I talk about what is is, some basic rules for doing and how we’ve been starting to use it where I work.
Observability is
The ability…
Being able to answer questions that you have never thought of before
You can think of it as the next step beyond just monitoring and alerting
Systems have become more distributed, and in the case of containerization, more ephemeral. It is increasingly difficult to know what our software is doing
And Observability means bringing better visibility into systems
To have better visibility, we need to acknowledge that…
Everything is sometimes broken
Something is always broker
-> No complex system is ever fully healthy
If nothing broken…
Distributed systems are unpredictable. In particular, it’s impossible to predict all the ways a system might fail
Failure needs to be embraced at every phase (from design to implementation, testing, deployment, and operation)
Ease of debugging is of high importance
Logging:
Structured logging: plain text -> splunk friendly -> json
Eventlog can be a great source of logs for debugging too
Consider sampling rather than aggregation
Metrics: Time series metrics, like tracking system stats such as CPU and mem usage, stats like # logins
Tracing: Distributed traceability using Correlation ID lib; Zipkin etc
Alerting: Useful for about proactively learning about, typically, predictable issues
Tools: e.g. Splunk, NR, OverOps; Wavefront
EPX, TDA (Thread Dump Analyzer) UX
OverOps / HoneyComb
Stacktraces and exception trackers?
OK, so that was Observability
The ability to answer questions about our applications behavior in production. Questions we may have never even though of before.
And with Observability in place, what types of production testing can we do…
Let’s move onto…
Testing at Release time
Let’s start by defining some terms:
Deployment vs release
When talking with engineers, I usually use the term Chaos Engineering, because it sounds cool! When talking with management, I tend to use the term resilience engineering, since it sounds less scary. The terms are synonymous. In the past, terms such as Disaster Recovery and Contingency Planning have been used to describe somewhat similar processes.
Whatever term you use, it basically refers to
->
Conducting carefully planned experiments designed to reveal weaknesses in our systems.
In other words, CE is the practice of confirming that your applications work as you expect them to in production.
Despite the name, Chaos Engineering is not about introducing Chaos into your system! Instead it is about identifying any chaos already there, so that you can remediate.
For example, if you GIVE A GOOD CANONICAL EXAMPLE OF A CHAOS ENGINEERING EXPERIMENT HERE
if you believe your application will failover to something if x happens
or can handle x requests per second before failing.
What are Game Days?
If Chaos Engineering is the theory, Game days are the practice; the execution
Game days are where you start with Chaos engineering
->
Game days are “An exercise where we place systems under stress to learn and improve resilience”
Systems can be technology, people, process
They are like fire drills – an opportunity to practice a potentially dangerous scenario in a safer environment
To start with, what are we trying to test! Pick a hypothesis.
Typically in Chaos Engineering experiments, the hypothesis is that if I do X (take out a server, kill a region), everything should be OK
But we need to be specific about how to measure things are OK
If out hypothesis is “if we fail primary DB, everything should be ok,”
Then we need to define what OK is!
And a big part of defining OK is to define “Steady State”
Steady state is essentially what the key metrics are for you to monitor as part of your test. It could be things like:
Loan application remain constant
Or response times remain in an acceptable range
If you don’t define steady state, how do you know your test is working on not? How do you know if you are breaking things?
With a hypothesis in mind, and a way to test, but first think abut blast radius
2. Minimize the blast radius
The blast radius refers to to how much damage can be done by the experiment
If you take out a server, and everything is in fact NOT OK, how bad might it be
Try to ensure that you limit he possible damage
For example, if your hypothesis is that
When Foo service is running in a pool of 2 servers
And one of those servers dies, CPU and memory utilization should increase on the remain servers, but response time remain unaffected
That is a fine thing to test
But if you have 10 services depending on that service (even in non prod), and your wrong that response times will be unaffected, you may have caused 10 other services to have problems
So a way to limit the blast radius in that test would be to test using a pool of Foo Service that only one other service relies on. Hopefully a service that you also control and that is closely monitored as part of the test.
Another way to minimize possible damage is to make sure that you have the equivalent of a big red Stop Test button!
If you metrics aren’t looking good, have the ability to abort the test immediately.
Remember: our goal here is to build confidence in the resilience of the system and we so that with Controlled, contained experiments, that grow in scope incrementally.
3. Run the experiment
Figure out the best way to test your hypothesis
If you plan to take out a server, how do you do it?
ssh in and kill -9? Orderly shutdown? Have Ops do it for you? Do you simulate failure by using bogus IP addresses, or simply removing a server from a VIP pool?
And again, stop if metrics or alert dictate
4. Analyze the results
Were your expectations correct?
Did your system handle things correctly
Did you spot issue with you alerts, metrics that should be improved before any future tests
5. Increase scope
The idea is to start small
1 service, in non-prod, and gradually expand to prod.
And the goal should be prod. Prod is where’s it’s at!
That brings us to the end of the presentation
We have talked about Testing in production
No longer a joke, instead increasingly viewed as a best practice. It is not a replacement for the essential and high value non-prod testing we do, but instead an addition.
Observability: Testing in production, and indeed in all envs, requires being able to understand what our applications do. Conventional logs, monitoring and alerting are all good, but Observability is about more than that. It’s about the ability to answer complex questions about our apps at run time. Questions we may not have even thought of before like: why is my app slow. Is it me or a downstream service? Where is all my memory being used. We can use metrics, tracing, any tools at our disposal so that we can see what’s going on when things go wrong. Or better still, to proactively spot problems in advance.
And with Observability in place, we can actually start to test in production!
We ran through different types of
Testing at Release
We can do, including
after deployment (Config, smoke, load, shadow)
At release time: Canary and internal release
After release: Feature flags and A/B testing
Finally, even when everything is up and running in prod, customers are using it, and all looks good, there is still more testing we can do
Chaos Engineering
Not introducing chaos, but exposing the already present chaos!
Carefully planned experiments designed to reveal weaknesses in our systems