Andreas Grabner maintains that most performance and scalability problems don’t need a large or long running performance test or the expertise of a performance engineering guru. Don’t let anybody tell you that performance is too hard to practice because it actually is not. You can take the initiative and find these often serious defects. Andreas analyzed and spotted the performance and scalability issues in more than 200 applications last year. He shares his performance testing approaches and explores the top problem patterns that you can learn to spot in your apps. By looking at key metrics found in log files and performance monitoring data, you will learn to identify most problems with a single functional test and a simple five-user load test. The problem patterns Andreas explains are applicable to any type of technology and platform. Try out your new skills in your current testing project and take the first step toward becoming a performance diagnostic hero.
Embracing Failure - Fault Injection and Service Resilience at NetflixJosh Evans
A presentation given at AWS re:Invent on how Netflix induces failure to validate and harden production systems. Technologies discussed include the Simian Army (Chaos Monkey, Gorilla, Kong) and our next gen Failure Injection Test framework (FIT).
Engineering Netflix Global Operations in the CloudJosh Evans
Delivered at re:Invent 2015.
Operating a massively scalable, constantly changing, distributed global service is a daunting task. We innovate at breakneck speed to attract new customers and stay ahead of the competition. This means more features, more experiments, more deployments, more engineers making changes in production environments, and ever-increasing complexity. Simultaneously improving service availability and accelerating rate of change seems impossible on the surface. At Netflix, operations engineering is both a technical and organizational construct designed to accomplish just that by integrating disciplines like continuous delivery, fault injection, regional traffic management, crisis response, best practice automation, and real-time analytics. In this talk, designed for technical leaders seeking a path to operational excellence, we'll explore these disciplines in depth and how they integrate and create competitive advantages.
Application Security Assessments by the Numbers - A Whole-istic View - OWASP ...Denim Group
By analyzing the data from over 60 mobile application security assessments, we identify the typical types of mobile vulnerabilities, the system components that contain those vulnerabilities, the components where given types of vulnerabilities cluster, and how to test for each of these.
Attendees will learn in the session how to identify these vulnerabilities, how to create and implement an effective mobile security plan, and where to focus their limited testing resources to minimize mobile application portfolio risks. This is critical because automated web application testing tools are able to easily find vulnerabilities while today's mobile security industry does not offer automated testing tools that can effectively test web services (i.e. the interaction between mobile clients and back-end services.) As a result, best practices for mobile application testing must incorporate significant, often laborious, manual testing. At this point in the presentation, we will use the statistics from the research to define the appropriate manual testing that needs to be implemented.
A webinar hosted by Curiosity Software Ireland on November 10th 2020. Watch the on demand recording here: https://opentestingplatform.curiositysoftware.ie/broken-promise-test-automation-webinar
Let’s face it - in this day and age you can’t test everything. Our environments have become too complex, while the allocated time for the construction and execution of tests is shorter than ever. With these constraints, two pressing questions arise:
• Are we building tests that truly matter for the release?
• Are we optimizing our regression suites in light of the changing application?
Believe it or not, this webinar is NOT a pitch about how AI or ML will magically solve your problems. Instead, Huw Price, Managing Director of Curiosity and Daniel Howard, Senior Researcher at Bloor Research, will offer a definitive plan for evolving sustainable automation. They will map new and emerging techniques for achieving in-sprint testing, including:
1. Automation that extends far beyond test execution;
2. Optimisation techniques for targeting testing exactly where it’s needed;
3. Methods for capitalising on data created by integrated DevOps toolchains.
Curiosity Director of Technology, James Walker, will be on hand to provide demos of the key technologies identified by Huw and Daniel. You will come away with actionable guidance for optimizing your testing, while tackling the time-intensive processes that test automation has introduced.
Join Huw, Daniel and James to see how testing can move beyond hand-cranking tests!
Watch the on demand recording here: https://opentestingplatform.curiositysoftware.ie/broken-promise-test-automation-webinar
Curiosity and Sauce Labs present - When to stop testing: 3 dimensions of test...Curiosity Software Ireland
This webinar was co-hosted by Curiosity Software and Sauce Labs on the 28th of September, 2021. Watch the webinar on demand today: https://opentestingplatform.curiositysoftware.ie/stop-testing-test-coverage-webinar
A definition of “done” is one of the hardest and most valuable things to come by in testing. Faced with fast-changing, massively complex systems, there’s no time to test everything in short sprints. Even defining “everything” is hard enough, given the vast and often unknown system logic, user devices, and integrated technologies that must be factored into rigorous testing. Too often, a lack of measurability combines with unsystematic test design, forcing testers to guess or hope that testing is “done”. This introduces uncertainty with every rapid release. Tests leave logic exposed to costly bugs and performance issues, while untested devices warp UIs and user experiences.
This webinar will set out how testing can rapidly identify, generate, and run the tests needed to de-risk rapid software releases. It will define functional test coverage in three dimensions, considering the system logic and data that must be tested, the optimal device mix, and the need to test across different system tiers. James Walker, Curiosity’s Director of Technology, and Marcus Merrell, Senior Director of Technology Strategy at Sauce Labs, will then demonstrate how in-sprint testing can target tests based on this multifaceted measure. You will see how:
1. Generating optimised tests, data and scripts from visual flowcharts avoids slow test creation and maintenance, while testing system logic rigorously based on time and risk.
2. Pushing tests to cloud-based device labs minimises environment and device limitations, enabling the right mix for each stage of the testing lifecycle.
3. Updating central flows regenerates tests in-sprint, targeting impacted and risky logic across APIs, UIs and back-end systems.
Do You Have a Scanner or Do You Have a Scanning Program? (AppSecEU 2013)Denim Group
By this point, most organizations have acquired at least one code or application scanning technology to incorporate into their software security program. Unfortunately, for many organizations the scanner represents the entirety of that so-called “program” and often the scanners are not used correctly or on a consistent basis.
This presentation looks at the components of a comprehensive software security program, the role that automation plays in these programs and tools and techniques that can be used to help increase the value an organization receives from its application scanning activities. It starts by examining common traps organizations fall into where they fail to address coverage concerns – either breadth of scanning coverage across the application portfolio or depth of coverage issues where application scans do not provide sufficient insight into the security state of target applications. After discussing approaches to address these coverage issues, the presentation walks through metrics organizations can use to keep tabs on their scanning progress to better understand what is being scanned, how frequently and at what depth.
The presentation also contains a demonstration of how freely available tools such as the open source ThreadFix application vulnerability management platform and the OWASP Zed Attack Proxy (ZAP) scanner can be combined to create a baseline scanning program for an organization and how this approach can be generalized to use any scanning technology.
Andreas Grabner maintains that most performance and scalability problems don’t need a large or long running performance test or the expertise of a performance engineering guru. Don’t let anybody tell you that performance is too hard to practice because it actually is not. You can take the initiative and find these often serious defects. Andreas analyzed and spotted the performance and scalability issues in more than 200 applications last year. He shares his performance testing approaches and explores the top problem patterns that you can learn to spot in your apps. By looking at key metrics found in log files and performance monitoring data, you will learn to identify most problems with a single functional test and a simple five-user load test. The problem patterns Andreas explains are applicable to any type of technology and platform. Try out your new skills in your current testing project and take the first step toward becoming a performance diagnostic hero.
Embracing Failure - Fault Injection and Service Resilience at NetflixJosh Evans
A presentation given at AWS re:Invent on how Netflix induces failure to validate and harden production systems. Technologies discussed include the Simian Army (Chaos Monkey, Gorilla, Kong) and our next gen Failure Injection Test framework (FIT).
Engineering Netflix Global Operations in the CloudJosh Evans
Delivered at re:Invent 2015.
Operating a massively scalable, constantly changing, distributed global service is a daunting task. We innovate at breakneck speed to attract new customers and stay ahead of the competition. This means more features, more experiments, more deployments, more engineers making changes in production environments, and ever-increasing complexity. Simultaneously improving service availability and accelerating rate of change seems impossible on the surface. At Netflix, operations engineering is both a technical and organizational construct designed to accomplish just that by integrating disciplines like continuous delivery, fault injection, regional traffic management, crisis response, best practice automation, and real-time analytics. In this talk, designed for technical leaders seeking a path to operational excellence, we'll explore these disciplines in depth and how they integrate and create competitive advantages.
Application Security Assessments by the Numbers - A Whole-istic View - OWASP ...Denim Group
By analyzing the data from over 60 mobile application security assessments, we identify the typical types of mobile vulnerabilities, the system components that contain those vulnerabilities, the components where given types of vulnerabilities cluster, and how to test for each of these.
Attendees will learn in the session how to identify these vulnerabilities, how to create and implement an effective mobile security plan, and where to focus their limited testing resources to minimize mobile application portfolio risks. This is critical because automated web application testing tools are able to easily find vulnerabilities while today's mobile security industry does not offer automated testing tools that can effectively test web services (i.e. the interaction between mobile clients and back-end services.) As a result, best practices for mobile application testing must incorporate significant, often laborious, manual testing. At this point in the presentation, we will use the statistics from the research to define the appropriate manual testing that needs to be implemented.
A webinar hosted by Curiosity Software Ireland on November 10th 2020. Watch the on demand recording here: https://opentestingplatform.curiositysoftware.ie/broken-promise-test-automation-webinar
Let’s face it - in this day and age you can’t test everything. Our environments have become too complex, while the allocated time for the construction and execution of tests is shorter than ever. With these constraints, two pressing questions arise:
• Are we building tests that truly matter for the release?
• Are we optimizing our regression suites in light of the changing application?
Believe it or not, this webinar is NOT a pitch about how AI or ML will magically solve your problems. Instead, Huw Price, Managing Director of Curiosity and Daniel Howard, Senior Researcher at Bloor Research, will offer a definitive plan for evolving sustainable automation. They will map new and emerging techniques for achieving in-sprint testing, including:
1. Automation that extends far beyond test execution;
2. Optimisation techniques for targeting testing exactly where it’s needed;
3. Methods for capitalising on data created by integrated DevOps toolchains.
Curiosity Director of Technology, James Walker, will be on hand to provide demos of the key technologies identified by Huw and Daniel. You will come away with actionable guidance for optimizing your testing, while tackling the time-intensive processes that test automation has introduced.
Join Huw, Daniel and James to see how testing can move beyond hand-cranking tests!
Watch the on demand recording here: https://opentestingplatform.curiositysoftware.ie/broken-promise-test-automation-webinar
Curiosity and Sauce Labs present - When to stop testing: 3 dimensions of test...Curiosity Software Ireland
This webinar was co-hosted by Curiosity Software and Sauce Labs on the 28th of September, 2021. Watch the webinar on demand today: https://opentestingplatform.curiositysoftware.ie/stop-testing-test-coverage-webinar
A definition of “done” is one of the hardest and most valuable things to come by in testing. Faced with fast-changing, massively complex systems, there’s no time to test everything in short sprints. Even defining “everything” is hard enough, given the vast and often unknown system logic, user devices, and integrated technologies that must be factored into rigorous testing. Too often, a lack of measurability combines with unsystematic test design, forcing testers to guess or hope that testing is “done”. This introduces uncertainty with every rapid release. Tests leave logic exposed to costly bugs and performance issues, while untested devices warp UIs and user experiences.
This webinar will set out how testing can rapidly identify, generate, and run the tests needed to de-risk rapid software releases. It will define functional test coverage in three dimensions, considering the system logic and data that must be tested, the optimal device mix, and the need to test across different system tiers. James Walker, Curiosity’s Director of Technology, and Marcus Merrell, Senior Director of Technology Strategy at Sauce Labs, will then demonstrate how in-sprint testing can target tests based on this multifaceted measure. You will see how:
1. Generating optimised tests, data and scripts from visual flowcharts avoids slow test creation and maintenance, while testing system logic rigorously based on time and risk.
2. Pushing tests to cloud-based device labs minimises environment and device limitations, enabling the right mix for each stage of the testing lifecycle.
3. Updating central flows regenerates tests in-sprint, targeting impacted and risky logic across APIs, UIs and back-end systems.
Do You Have a Scanner or Do You Have a Scanning Program? (AppSecEU 2013)Denim Group
By this point, most organizations have acquired at least one code or application scanning technology to incorporate into their software security program. Unfortunately, for many organizations the scanner represents the entirety of that so-called “program” and often the scanners are not used correctly or on a consistent basis.
This presentation looks at the components of a comprehensive software security program, the role that automation plays in these programs and tools and techniques that can be used to help increase the value an organization receives from its application scanning activities. It starts by examining common traps organizations fall into where they fail to address coverage concerns – either breadth of scanning coverage across the application portfolio or depth of coverage issues where application scans do not provide sufficient insight into the security state of target applications. After discussing approaches to address these coverage issues, the presentation walks through metrics organizations can use to keep tabs on their scanning progress to better understand what is being scanned, how frequently and at what depth.
The presentation also contains a demonstration of how freely available tools such as the open source ThreadFix application vulnerability management platform and the OWASP Zed Attack Proxy (ZAP) scanner can be combined to create a baseline scanning program for an organization and how this approach can be generalized to use any scanning technology.
Il webinar è una finestra su IBM Requirements Quality Assistant.
Abbiamo mostrato come RQA ci aiuta a migliorare la qualità dei requisiti e quindi delle applicazioni che rilasciamo.
Abbiamo visto inoltre come IBM RQA può supportarci quando utilizziamo tool come IBM Jazz.
Il webinar è condotto da Profesia e un esperto di prodotto. Richiedi una demo personalizzata scrivendo a sales@profesia.it
*** Watch the on demand webinar recording here - https://curiositysoftware.ie/resources/test-data-development-webinar/ ***
A Curiosity Software and Windocks webinar, presented live on the 2nd of February, 2021. Now available to stream on demand!
Test data “provisioning” is lagging far behind the sophistication of today’s systems. Development has shifted to containerisation and microservices, rapidly ripping out and replacing reusable components. Testers must also rapidly rip-and-replace versioned components in their environments, while retaining complex data relationships between shifting technologies. The deployed data must furthermore be diverse, compliant and compact, fulfilling all positive and negative scenarios in the shortest test runs possible.
Sound like an impossible requirement? While it is, if you rely on making costly physical copies of low-variety production data. “Test data management” instead needs to embrace the world of containers and APIs, along with the pipelines that enable developers to deliver so rapidly. We need a new approach to testing massively complex systems in short sprints.
This webinar will showcase how Test Data Automation combines with containerised data cloning, automatically deploying versioned virtual databases as tests are created and run. Huw Price, Managing Director of Curiosity Software Ireland, and Paul Stanton, co-founder and Vice President of Windocks, will show you how:
1. Test Data Automation provides complete and compliant data on demand, delivering test-ready data that is masked and enhanced with synthetic data.
2. Parallel test teams and frameworks leverage fresh containers, without slow data provisioning or complex configuration.
3. Organisations regain full visibility and control over test data, while enjoying the added affordability of database virtualisation.
*** Watch the on demand webinar recording here - https://curiositysoftware.ie/resources/test-data-development-webinar/ ***
Given the changing nature of enterprise networking, Riverbed decided to survey attendees of the recent VMworld conference about their companies’ current plans for these emerging technologies. Riverbed surveyed 260 attendees face-to-face, from a variety of roles and with a median company size of 2,300 employees.
Customers and employees complaining about poor network performance or application delays? Want to put an end to the whining? Learn how combining visibility with WAN optimization delivers optimal performance for customers and employees regardless of location by watching this webinar from Riverbed. http://rvbd.ly/1OVbaQw
Is your company thinking about using Selenium to implement test automation in a joint development and operations environment? If your company has already started using Selenium, have you experienced execution or integration challenges? The path to a well-oiled and successful Selenium test automation program comes down to using the right techniques and development standards that incorporate modularity and flexibility. Jin Reck describes how to design effective web test automation development, and shares common challenges and solutions when implementing an automated testing framework in the real world. Jin shows how to incorporate Selenium with continuous integration platforms and discusses techniques, adjustments, lessons learned, and best practices from successful implementations. Leave with a better understanding of how to design and employ Selenium to create robust and reliable automated tests that increase the efficiency and productivity of test teams and make for a capable and successful testing program.
Static Application Security Testing Strategies for Automation and Continuous ...Kevin Fealey
Static Application Security Testing (SAST) introduces challenges with existing Software Development Lifecycle Configurations. Strategies at different points of the SDLC improve deployment time, while still improving the quality and security of the deliverable. This session will discuss the different strategies that can be implemented for SAST within SDLC—strategies catering to developers versus security analysts versus release engineers. The strategies consider the challenges each team may encounter, allowing them to incorporate security testing without jeopardizing deadlines or existing process.
Remediation Statistics: What Does Fixing Application Vulnerabilities Cost?Denim Group
For the security industry to mature more data needs to be available about the true cost of security vulnerabilities. Data and statistics are starting to be released, but most of this currently focuses on the prevalence of different types of vulnerabilities and incidents rather than the costs of addressing the underlying issues. This session presents statistics from the remediation of 15 web-based applications in order to provide insight into the actual cost of remediating application-level vulnerabilities.
The presentation begins by setting out a structured model for software security remediation projects so that time spent on tasks can be consistently tracked. It lays out possible sources of bias in the underlying data to allow for better-informed consumption of the final analysis. Also it discusses different approaches to remediating vulnerabilities such as fixing easy vulnerabilities first versus fixing serious vulnerabilities first.
Next, historical data from the fifteen remediation projects is presented. This data consists of the average cost to remediate specific classes of vulnerabilities – cross-site scripting, SQL injection and so on – as well as the overall project composition to demonstrate the percentage of time spent on actual fixes as well as the percentages of time spent on other supporting activities such as environment setup, testing and verification and deployment. The data on the remediation of specific vulnerabilities allows for a comparison of the relative difficulty of remediating different vulnerability types. The data on the overall project composition can be used to determine the relative “efficiency” of different projects.
Finally, analysis of the data is used to create a model for estimating remediation projects so that organizations can create realistic estimates in order to make informed remediate/do not remediate decisions. In addition, characteristics of the analyzed projects are mapped to project composition to demonstrate best practices that can be used to decrease the cost of future remediation efforts.
Taking AppSec to 11 - BSides Austin 2016Matt Tesauro
Curious how DevOps, Agile and CI/CD ideas can speed up your AppSec program? Here's how it can be done and an example where it lead to a 5x speed/flow improvement.
Building Your Application Security Data Hub - OWASP AppSecUSADenim Group
One of the reasons application security is so challenging to address is that it spans multiple teams within an organization. Development teams build software, security testing teams find vulnerabilities, security operations staff manage applications in production and IT audit organizations make sure that the resulting software meets compliance and governance requirements. In addition, each team has a different toolbox they use to meet their goals, ranging from scanning tools, defect trackers, Integrated Development Environments (IDEs), WAFs and GRC systems. Unfortunately, in most organizations the interactions between these teams is often strained and the flow of data between these disparate tools and systems is non-existent or tediously implemented manually.
In today’s presentation, we will demonstrate how leading organizations are breaking down these barriers between teams and better integrating their disparate tools to enable the flow of application security data between silos to accelerate and simplify their remediation efforts. At the same time, we will show how to collect the proper data to measure the performance and illustrate the improvement of the software security program. The challenges that need to be overcome to enable teams and tools to work seamlessly with one another will be enumerated individually. Team and tool interaction patterns will also be outlined that reduce the friction that will arise while addressing application security risks. Using open source products such as OWASP ZAP, ThreadFix, Bugzilla and Eclipse, a significant amount of time will also be spent demonstrating the kinds of interactions that need to be enabled between tools. This will provide attendees with practical examples on how to replicate a powerful, integrated Application Security program within their own organizations. In addition, how to gather program-wide metrics and regularly calculate measurements such as mean-time-to-fix will also be demonstrated to enable attendees to monitor and ensure the continuing health and performance of their Application Security program.
Leverage DevOps & Agile Development to Transform Your Application Testing Pro...Deborah Schalm
Discover how Sona Srinivasan, Senior Architect of Cisco IT’s Global Architecture and Technology Services group, helps transform an IT DevOps strategy to a Security DevOps strategy, with IBM Security's assistance. Cisco is presently implementing continuous security and agile methods throughout the software development lifecycle (SDLC), and specific examples of current initiatives will be reviewed in this session.
Today, organizations of all shapes and sizes depend on feature-packed application releases to keep end users productive and happy. In their new book, The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations, Gene Kim and his co-authors shared ways that high-performing organizations use DevOps principles to enable reliable deployments - and boring releases!
Gene Kim, CTO, DevOps researcher and co-author of the DevOps Handbook and The Phoenix Project, and Anders Wallgren, CTO of Electric Cloud shared their tips for overcoming the challenges of DevOps and Continuous Delivery at scale. During the webinar, they discussed:
- The business value of DevOps
- How to eliminate “deployment anxiety” and increase business agility
- Lessons learned from large scale DevOps transformations
- The advantages and disadvantages of practicing DevOps in large organizations
I created and delivered this presentation at the 2007 PLM World Conference. The topic was a client Teamcenter Community implementation project I managed.
Continuous Delivery Pipelines help developers safely iterate on and test new code in production-like environments. But what if your team is tasked with developing the CD Pipeline itself? How do you empower multiple SCM engineers to make changes in a controlled way without impacting each other or degrading developer productivity?
Learn how Sony has leveraged ElectricFlow and DevOps principles to construct a flexible, version-able, review-able, test-able, revert-able and resilient CD pipeline. This talk will focus on the infrastructure, tools and processes used to create and operate this pipeline, and the methods used to safely and rapidly onboard new business groups, developers and SCM engineers.
Modelling and Analysing Operation Processes for Dependability Liming Zhu
The 43rd Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN13) talk slides. June 27th, 2013. Full text here: http://www.nicta.com.au/pub?doc=7031
Il webinar è una finestra su IBM Requirements Quality Assistant.
Abbiamo mostrato come RQA ci aiuta a migliorare la qualità dei requisiti e quindi delle applicazioni che rilasciamo.
Abbiamo visto inoltre come IBM RQA può supportarci quando utilizziamo tool come IBM Jazz.
Il webinar è condotto da Profesia e un esperto di prodotto. Richiedi una demo personalizzata scrivendo a sales@profesia.it
*** Watch the on demand webinar recording here - https://curiositysoftware.ie/resources/test-data-development-webinar/ ***
A Curiosity Software and Windocks webinar, presented live on the 2nd of February, 2021. Now available to stream on demand!
Test data “provisioning” is lagging far behind the sophistication of today’s systems. Development has shifted to containerisation and microservices, rapidly ripping out and replacing reusable components. Testers must also rapidly rip-and-replace versioned components in their environments, while retaining complex data relationships between shifting technologies. The deployed data must furthermore be diverse, compliant and compact, fulfilling all positive and negative scenarios in the shortest test runs possible.
Sound like an impossible requirement? While it is, if you rely on making costly physical copies of low-variety production data. “Test data management” instead needs to embrace the world of containers and APIs, along with the pipelines that enable developers to deliver so rapidly. We need a new approach to testing massively complex systems in short sprints.
This webinar will showcase how Test Data Automation combines with containerised data cloning, automatically deploying versioned virtual databases as tests are created and run. Huw Price, Managing Director of Curiosity Software Ireland, and Paul Stanton, co-founder and Vice President of Windocks, will show you how:
1. Test Data Automation provides complete and compliant data on demand, delivering test-ready data that is masked and enhanced with synthetic data.
2. Parallel test teams and frameworks leverage fresh containers, without slow data provisioning or complex configuration.
3. Organisations regain full visibility and control over test data, while enjoying the added affordability of database virtualisation.
*** Watch the on demand webinar recording here - https://curiositysoftware.ie/resources/test-data-development-webinar/ ***
Given the changing nature of enterprise networking, Riverbed decided to survey attendees of the recent VMworld conference about their companies’ current plans for these emerging technologies. Riverbed surveyed 260 attendees face-to-face, from a variety of roles and with a median company size of 2,300 employees.
Customers and employees complaining about poor network performance or application delays? Want to put an end to the whining? Learn how combining visibility with WAN optimization delivers optimal performance for customers and employees regardless of location by watching this webinar from Riverbed. http://rvbd.ly/1OVbaQw
Is your company thinking about using Selenium to implement test automation in a joint development and operations environment? If your company has already started using Selenium, have you experienced execution or integration challenges? The path to a well-oiled and successful Selenium test automation program comes down to using the right techniques and development standards that incorporate modularity and flexibility. Jin Reck describes how to design effective web test automation development, and shares common challenges and solutions when implementing an automated testing framework in the real world. Jin shows how to incorporate Selenium with continuous integration platforms and discusses techniques, adjustments, lessons learned, and best practices from successful implementations. Leave with a better understanding of how to design and employ Selenium to create robust and reliable automated tests that increase the efficiency and productivity of test teams and make for a capable and successful testing program.
Static Application Security Testing Strategies for Automation and Continuous ...Kevin Fealey
Static Application Security Testing (SAST) introduces challenges with existing Software Development Lifecycle Configurations. Strategies at different points of the SDLC improve deployment time, while still improving the quality and security of the deliverable. This session will discuss the different strategies that can be implemented for SAST within SDLC—strategies catering to developers versus security analysts versus release engineers. The strategies consider the challenges each team may encounter, allowing them to incorporate security testing without jeopardizing deadlines or existing process.
Remediation Statistics: What Does Fixing Application Vulnerabilities Cost?Denim Group
For the security industry to mature more data needs to be available about the true cost of security vulnerabilities. Data and statistics are starting to be released, but most of this currently focuses on the prevalence of different types of vulnerabilities and incidents rather than the costs of addressing the underlying issues. This session presents statistics from the remediation of 15 web-based applications in order to provide insight into the actual cost of remediating application-level vulnerabilities.
The presentation begins by setting out a structured model for software security remediation projects so that time spent on tasks can be consistently tracked. It lays out possible sources of bias in the underlying data to allow for better-informed consumption of the final analysis. Also it discusses different approaches to remediating vulnerabilities such as fixing easy vulnerabilities first versus fixing serious vulnerabilities first.
Next, historical data from the fifteen remediation projects is presented. This data consists of the average cost to remediate specific classes of vulnerabilities – cross-site scripting, SQL injection and so on – as well as the overall project composition to demonstrate the percentage of time spent on actual fixes as well as the percentages of time spent on other supporting activities such as environment setup, testing and verification and deployment. The data on the remediation of specific vulnerabilities allows for a comparison of the relative difficulty of remediating different vulnerability types. The data on the overall project composition can be used to determine the relative “efficiency” of different projects.
Finally, analysis of the data is used to create a model for estimating remediation projects so that organizations can create realistic estimates in order to make informed remediate/do not remediate decisions. In addition, characteristics of the analyzed projects are mapped to project composition to demonstrate best practices that can be used to decrease the cost of future remediation efforts.
Taking AppSec to 11 - BSides Austin 2016Matt Tesauro
Curious how DevOps, Agile and CI/CD ideas can speed up your AppSec program? Here's how it can be done and an example where it lead to a 5x speed/flow improvement.
Building Your Application Security Data Hub - OWASP AppSecUSADenim Group
One of the reasons application security is so challenging to address is that it spans multiple teams within an organization. Development teams build software, security testing teams find vulnerabilities, security operations staff manage applications in production and IT audit organizations make sure that the resulting software meets compliance and governance requirements. In addition, each team has a different toolbox they use to meet their goals, ranging from scanning tools, defect trackers, Integrated Development Environments (IDEs), WAFs and GRC systems. Unfortunately, in most organizations the interactions between these teams is often strained and the flow of data between these disparate tools and systems is non-existent or tediously implemented manually.
In today’s presentation, we will demonstrate how leading organizations are breaking down these barriers between teams and better integrating their disparate tools to enable the flow of application security data between silos to accelerate and simplify their remediation efforts. At the same time, we will show how to collect the proper data to measure the performance and illustrate the improvement of the software security program. The challenges that need to be overcome to enable teams and tools to work seamlessly with one another will be enumerated individually. Team and tool interaction patterns will also be outlined that reduce the friction that will arise while addressing application security risks. Using open source products such as OWASP ZAP, ThreadFix, Bugzilla and Eclipse, a significant amount of time will also be spent demonstrating the kinds of interactions that need to be enabled between tools. This will provide attendees with practical examples on how to replicate a powerful, integrated Application Security program within their own organizations. In addition, how to gather program-wide metrics and regularly calculate measurements such as mean-time-to-fix will also be demonstrated to enable attendees to monitor and ensure the continuing health and performance of their Application Security program.
Leverage DevOps & Agile Development to Transform Your Application Testing Pro...Deborah Schalm
Discover how Sona Srinivasan, Senior Architect of Cisco IT’s Global Architecture and Technology Services group, helps transform an IT DevOps strategy to a Security DevOps strategy, with IBM Security's assistance. Cisco is presently implementing continuous security and agile methods throughout the software development lifecycle (SDLC), and specific examples of current initiatives will be reviewed in this session.
Today, organizations of all shapes and sizes depend on feature-packed application releases to keep end users productive and happy. In their new book, The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations, Gene Kim and his co-authors shared ways that high-performing organizations use DevOps principles to enable reliable deployments - and boring releases!
Gene Kim, CTO, DevOps researcher and co-author of the DevOps Handbook and The Phoenix Project, and Anders Wallgren, CTO of Electric Cloud shared their tips for overcoming the challenges of DevOps and Continuous Delivery at scale. During the webinar, they discussed:
- The business value of DevOps
- How to eliminate “deployment anxiety” and increase business agility
- Lessons learned from large scale DevOps transformations
- The advantages and disadvantages of practicing DevOps in large organizations
I created and delivered this presentation at the 2007 PLM World Conference. The topic was a client Teamcenter Community implementation project I managed.
Continuous Delivery Pipelines help developers safely iterate on and test new code in production-like environments. But what if your team is tasked with developing the CD Pipeline itself? How do you empower multiple SCM engineers to make changes in a controlled way without impacting each other or degrading developer productivity?
Learn how Sony has leveraged ElectricFlow and DevOps principles to construct a flexible, version-able, review-able, test-able, revert-able and resilient CD pipeline. This talk will focus on the infrastructure, tools and processes used to create and operate this pipeline, and the methods used to safely and rapidly onboard new business groups, developers and SCM engineers.
Modelling and Analysing Operation Processes for Dependability Liming Zhu
The 43rd Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN13) talk slides. June 27th, 2013. Full text here: http://www.nicta.com.au/pub?doc=7031
Using Collaborate Plan to set up your Blackboard Collaborate Session in advance of the arrival of your participants is a great way to focus on your audience and not the tools.
Bridging the Engagement Gap for Distance Students Through TeleroboticsMichael Griffith
Traditional telepresence classrooms are expensive to implement and often require technical and instructional support to successfully translate an instructor's pedagogy. There is often an engagement gap when comparing the level of participation and curiosity exhibited by the distant student and their resident peers. A second issue is that students enrolled in a fully supported telepresent class are often enrolled in other courses that don’t take place in rooms equipped for their needs. Our team is exploring a mix of technologies that will allow distance students to engage with their instructor, peers, and course material in a standard classroom using an inexpensive, portable telerobotics platform. We hope to close the engagement gap by placing these robots among the resident students and in the direct line-of-sight of the instructor thus reducing distraction for the resident students and allowing the instructor to react to non-verbal cues exhibited by the distance students.
Challenges in Practicing High Frequency Releases in Cloud Environments Liming Zhu
Talk at RELENG 2014
Full paper: http://www.nicta.com.au/pub?doc=7925
The continuous delivery trend is dramatically shortening release cycles from months into hours. Applications with high frequency releases often rely heavily on automated deployment tools using cloud infrastructure APIs. We report some results from experiments on reliability issues of cloud infrastructure and trade-offs between using heavily-baked and lightly-baked images. Our experiments were based on Amazon Web Service (AWS) OpsWorks APIs and configuration management tool Chef. As a result of our experiments, we then propose error handling practices that can be included in tailor-made continuous deployment facilities.
More related info at our DevOps book http://www.ssrg.nicta.com.au/projects/devops_book/
Dependable Operation - Performance Management and Capacity Planning Under Con...Liming Zhu
Talk at http://www.cmga.org.au/ Meet up
Modern large-scale applications experience sporadic changes due to operational activities such as upgrade, redeployment, on-demand scaling and interferences from other simultaneous operations. This poses new challenges in system monitoring, capacity planning, performance management, error detection and diagnosis. For example, the traditional anomaly-detection-based techniques are less effective during the “sporadic” operation period as a wide range of legitimate changes confound the situation and make performance baseline establishment for “normal” operation difficult. The increasing frequency of these sporadic operations (e.g. due to continuous deployment) is exacerbating the problem. In this talk, we will introduce a number of ongoing research activities at NICTA addressing these issues. For example, we propose the Process Oriented Dependability (POD) approach, an approach that explicitly models these sporadic operations as processes and uses the process context to filter logs, traverse fault trees and conduct adaptive monitoring.
Real World Problem Solving Using Application Performance Management 10CA Technologies
CA Application Performance Management 10 dramatically reduces the time needed to find and solve app problems. In this session you will learn about common problem-solving techniques used by experts to solve real-world app problems. You will get a chance to put these techniques to the test in a hands-on lab that mimics an interesting application performance problem.
For more information, please visit http://cainc.to/Nv2VOe
VMworld 2013: Moving Enterprise Application Dev/Test to VMware’s Internal Pri...VMworld
VMworld Europe 2013
Thirumalesh Reddy, Vmware
Venkat Gopalakrishnan, VMware
Learn more about VMworld and register at http://www.vmworld.com/index.jspa?src=socmed-vmworld-slideshare
FOSDEM 2024 - Deploy Fast, Without Breaking Things: Level Up APIOps With Open...SonjaChevre
For developers building distributed applications, few things are more frustrating than having an API break unexpectedly. When an API changes in a backwards-incompatible way, it can disrupt downstream consumers. Suddenly, applications start failing, integrations break, and developers are left scrambling to fix the issues.
How can developers deploy changes fast without breaking APIs and maintain stability for consumers? By adding modern observability techniques to their APIOps pipeline.
In this talk, we will start together from a traditional APIOps pipeline in ArgoCD and explore how incorporating modern observability techniques can help developers deploy changes quickly and efficiently while maintaining stability for API consumers.
In this presentation, Sonja (Group Product Manager in the API space) and Adnan (5 years in the observability space) are combining their expertise to present best practices for detecting and resolving API issues in production.
Nonfunctional Testing: Examine the Other Side of the CoinTechWell
Creating a highly available, scalable, and high-performing system requires a substantial amount of what we call nonfunctional testing. Developing nonfunctional testing skills is a must for many of today’s quality engineers (QEs). For the past several years, Balaji Arunachalam’s quality team for Intuit Core Services has experienced several highly available and disaster recovery buildup and testing challenges. Their journey includes the evolution of functional QEs into hybrid QEs who are capable of doing both functional and nonfunctional testing. Nonfunctional testing includes capacity, stability, benchmarking, FMEA/RAS, datacenter failover, and scalability testing. Balaji shares nonfunctional testing best practices, learnings, and mistakes they encountered on this journey. If you or your team is ready flip the coin and take a serious look at nonfunctional testing methods, opportunities, challenges, and solutions, this session is for you.
Case Study: Verizon Wireless: Chasing the Yellow Before They Turn RedCA Technologies
In the age of the application economy, Verizon shows one way they strive to earn user loyalty. By correlating network and application metrics and gaining network and application performance insights, Verizon is able to homogenize behaviors across organizations and focus on fault prevention rather than decreasing time to fix after the fault has already occurred.
For more information on DevOps solutions from CA Technologies, please visit: http://bit.ly/1wbjjqX
CPN208 Failures at Scale & How to Ride Through Them - AWS re: Invent 2012Amazon Web Services
At scale, rare and unexpected events will happen. Things eventually will go wrong. This talk dives into what can go wrong at scale and how to architect applications to ride through disaster obliviously. We’ll talk about AWS infrastructure design including Regions and Availability Zones and show how applications can be written and operated to best exploit this industry-unique infrastructure redundancy model. Believing that experience is one of the best teachers, we will go through some of the more interesting and educational industry post mortems including some experienced at AWS to motivate these application design decisions and show how they can mitigate the damage of the truly unexpected.
AppSphere 15 - How AppDynamics is Shaking up the Synthetic Monitoring Product...AppDynamics
Synthetic monitoring has been around for nearly two decades, but the innovation in this area has crawled to a trickle. Users are coping with complex and disjointed products driven by proprietary technology. This is about to change: AppDynamics Synthetic monitoring technology is driven by the leading-edge front end optimization open source technology WebPageTest and W3C standards like Webdriver. AppDynamics has embraced and combined them with changes in Cloud Computing to deliver a new generation of synthetic monitoring. These technologies allow not only for availability monitoring today, but hold a vast array of use cases and capabilities for the future which will create new innovation.
Key Takeaways:
- Learn about WebPageTest, and why it's the leading tool for front end optimization
- How AppDynamics leverages WebPageTest and Webdriver technologies
- How AppDynamics is leveraging changes in Cloud computing to deliver a new generation of synthetic -monitoring
- What future capabilities AppDynamics will leverage from these projects to create new use cases
This deck was originally presented at AppSphere 2015.
Eric Proegler Oredev Performance Testing in New ContextsEric Proegler
Virtualization, Cloud Deployments, and Cloud-Based Tools have challenged and changed performance testing practices. Today’s performance tester can summons tens of thousands of virtual users from the cloud in a few minutes at a cost far lower than the expensive on-premise installations of yesteryear.
Meanwhile, systems under test have changed more. Updated software stacks have increased the complexity of scripting and performance measurement, but the biggest changes are in the nature and quantities of resources powering the systems. Interpreting resource usage when resources are shared on a private virtualization platform is exceedingly difficult. Understanding resources when they live in a large public cloud is impossible.
Metrics Driven DevOps - Automate Scalability and Performance Into your PipelineAndreas Grabner
Continuous Delivery only works if you combine automation with automatic metrics driven quality gates focusing on architectural, scalabilty and performance metrics.
In this presentation I start with several dashboard examples explaining key metrics in production and explain how to automate these metrics into your delivery pipeline.
Moderator:
Chris Grundemann, Network Automation Forum
Speakers:
Jeff Loughridge, Konekti Systems
Mark Ciecior, Carrier Access IT
William Collins, Alkira
Ask The Architect: RightScale & AWS Dive Deep into Hybrid ITRightScale
With the increased use of cloud services, organizations are faced with finding the most efficient way to use existing IT infrastructure alongside cloud-based compute, storage and networking resources. This has resulted in the rise of hybrid IT whereby companies leverage both on-premises and cloud resources to drive increased agility, stability and accessibility.
Examine common application performance problems hiding in plain sight. See how you can quickly remove the noise, pinpoint root cause and fix these problems once and for all. Watch the webinar replay: http://rvbd.ly/1QGxMBs
Software Architecture for Foundation Model-Based SystemsLiming Zhu
With the successful implementation of Large Language Models (LLMs) in chatbots like ChatGPT, there is growing attention on foundation models, which are anticipated to serve as core components in the development of future AI systems. Yet, systematic exploration into the design of foundation model-based systems, particularly concerning risk management, trust, and trustworthiness, remains limited. In this talk, I propose the challenges and initial approaches in both architecting LLM-based systems and how LLM systems have an impact on software engineering. I point to some initial directions such as architecting as a process of understanding (rather than designing/building), setting and trade-offing guardrails (rather than quality attributes), and radical observability.
Responsible/Trustworthy AI in the Era of Foundation Models Liming Zhu
The emergence of large language models (LLM) such as GPT4 has garnered significant attention, placing foundation models at the forefront of AI systems. However, integrating foundation models raises concerns regarding responsible/trustworthy AI due to their opaque nature and rapidly moving capability boundaries. This talk addresses these challenges in the context of industry and defence and proposes a pattern-oriented reference architecture for responsible AI/trustworthy design in foundation model-based systems. It explores the evolution of AI systems architecture, transitioning from a many-model/module architecture to a increasingly monolithic architecture centered around foundation models.
ICSE23 Keynote: Software Engineering as the Linchpin of Responsible AILiming Zhu
From humanity’s existential risks to safety risks in critical systems to ethical risks, responsible AI, as the saviour, has become a massive research challenge with significant real-world consequences. However, achieving responsible AI remains elusive despite the plethora of high-level ethical principles, risk frameworks and progress in algorithmic assurance. In the meantime, software engineering (SE) is being upended by AI, grappling with building system-level quality and alignment from inscrutable ML models and code generated from natural language prompts. The upending poses new challenges and opportunities for engineering AI systems responsibly. This talk will share our experiences in helping the industry achieve responsible AI systems by inventing new SE approaches. It will dive into industry challenges (such as risk silos and principle-algorithm gaps) and research challenges (such as lack of requirements, emerging properties and inscrutable systems) and make the point that SE is the linchpin of responsible AI. But SE also requires some fundamental rethinking - shifting from building functions to AI systems to discovering and managing emerging functions from AI systems. Only by doing so can SE take on critical new roles, from understanding human intelligence to building a thriving human-AI symbiosis.
Responsible AI & Cybersecurity: A tale of two technology risksLiming Zhu
With the broader adoption of digital technologies and AI, organisations face the emerging risks of AI, the unfamiliar, and the intensified risk of cybersecurity, the familiar. AI and cybersecurity are intertwined, but risk silos are often created when they are dealt with at the technology and governance levels. This talk will explore the interactions between responsible AI and cybersecurity risks via industry case studies. It will show how we can break down the risk silos and use emerging trust-enhancing technologies, architecture and end-to-end software engineering/DevOps practices to connect the two worlds and uplift the risk management posture for both.
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdfMalak Abu Hammad
Discover how MongoDB Atlas and vector search technology can revolutionize your application's search capabilities. This comprehensive presentation covers:
* What is Vector Search?
* Importance and benefits of vector search
* Practical use cases across various industries
* Step-by-step implementation guide
* Live demos with code snippets
* Enhancing LLM capabilities with vector search
* Best practices and optimization strategies
Perfect for developers, AI enthusiasts, and tech leaders. Learn how to leverage MongoDB Atlas to deliver highly relevant, context-aware search results, transforming your data retrieval process. Stay ahead in tech innovation and maximize the potential of your applications.
#MongoDB #VectorSearch #AI #SemanticSearch #TechInnovation #DataScience #LLM #MachineLearning #SearchTechnology
Securing your Kubernetes cluster_ a step-by-step guide to success !KatiaHIMEUR1
Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster.
However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks.
In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.
Building RAG with self-deployed Milvus vector database and Snowpark Container...Zilliz
This talk will give hands-on advice on building RAG applications with an open-source Milvus database deployed as a docker container. We will also introduce the integration of Milvus with Snowpark Container Services.
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
UiPath Test Automation using UiPath Test Suite series, part 5DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 5. In this session, we will cover CI/CD with devops.
Topics covered:
CI/CD with in UiPath
End-to-end overview of CI/CD pipeline with Azure devops
Speaker:
Lyndsey Byblow, Test Suite Sales Engineer @ UiPath, Inc.
20 Comprehensive Checklist of Designing and Developing a WebsitePixlogix Infotech
Dive into the world of Website Designing and Developing with Pixlogix! Looking to create a stunning online presence? Look no further! Our comprehensive checklist covers everything you need to know to craft a website that stands out. From user-friendly design to seamless functionality, we've got you covered. Don't miss out on this invaluable resource! Check out our checklist now at Pixlogix and start your journey towards a captivating online presence today.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
In the rapidly evolving landscape of technologies, XML continues to play a vital role in structuring, storing, and transporting data across diverse systems. The recent advancements in artificial intelligence (AI) present new methodologies for enhancing XML development workflows, introducing efficiency, automation, and intelligent capabilities. This presentation will outline the scope and perspective of utilizing AI in XML development. The potential benefits and the possible pitfalls will be highlighted, providing a balanced view of the subject.
We will explore the capabilities of AI in understanding XML markup languages and autonomously creating structured XML content. Additionally, we will examine the capacity of AI to enrich plain text with appropriate XML markup. Practical examples and methodological guidelines will be provided to elucidate how AI can be effectively prompted to interpret and generate accurate XML markup.
Further emphasis will be placed on the role of AI in developing XSLT, or schemas such as XSD and Schematron. We will address the techniques and strategies adopted to create prompts for generating code, explaining code, or refactoring the code, and the results achieved.
The discussion will extend to how AI can be used to transform XML content. In particular, the focus will be on the use of AI XPath extension functions in XSLT, Schematron, Schematron Quick Fixes, or for XML content refactoring.
The presentation aims to deliver a comprehensive overview of AI usage in XML development, providing attendees with the necessary knowledge to make informed decisions. Whether you’re at the early stages of adopting AI or considering integrating it in advanced XML development, this presentation will cover all levels of expertise.
By highlighting the potential advantages and challenges of integrating AI with XML development tools and languages, the presentation seeks to inspire thoughtful conversation around the future of XML development. We’ll not only delve into the technical aspects of AI-powered XML development but also discuss practical implications and possible future directions.
A tale of scale & speed: How the US Navy is enabling software delivery from l...sonjaschweigert1
Rapid and secure feature delivery is a goal across every application team and every branch of the DoD. The Navy’s DevSecOps platform, Party Barge, has achieved:
- Reduction in onboarding time from 5 weeks to 1 day
- Improved developer experience and productivity through actionable findings and reduction of false positives
- Maintenance of superior security standards and inherent policy enforcement with Authorization to Operate (ATO)
Development teams can ship efficiently and ensure applications are cyber ready for Navy Authorizing Officials (AOs). In this webinar, Sigma Defense and Anchore will give attendees a look behind the scenes and demo secure pipeline automation and security artifacts that speed up application ATO and time to production.
We will cover:
- How to remove silos in DevSecOps
- How to build efficient development pipeline roles and component templates
- How to deliver security artifacts that matter for ATO’s (SBOMs, vulnerability reports, and policy evidence)
- How to streamline operations with automated policy checks on container images
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AIVladimir Iglovikov, Ph.D.
Presented by Vladimir Iglovikov:
- https://www.linkedin.com/in/iglovikov/
- https://x.com/viglovikov
- https://www.instagram.com/ternaus/
This presentation delves into the journey of Albumentations.ai, a highly successful open-source library for data augmentation.
Created out of a necessity for superior performance in Kaggle competitions, Albumentations has grown to become a widely used tool among data scientists and machine learning practitioners.
This case study covers various aspects, including:
People: The contributors and community that have supported Albumentations.
Metrics: The success indicators such as downloads, daily active users, GitHub stars, and financial contributions.
Challenges: The hurdles in monetizing open-source projects and measuring user engagement.
Development Practices: Best practices for creating, maintaining, and scaling open-source libraries, including code hygiene, CI/CD, and fast iteration.
Community Building: Strategies for making adoption easy, iterating quickly, and fostering a vibrant, engaged community.
Marketing: Both online and offline marketing tactics, focusing on real, impactful interactions and collaborations.
Mental Health: Maintaining balance and not feeling pressured by user demands.
Key insights include the importance of automation, making the adoption process seamless, and leveraging offline interactions for marketing. The presentation also emphasizes the need for continuous small improvements and building a friendly, inclusive community that contributes to the project's growth.
Vladimir Iglovikov brings his extensive experience as a Kaggle Grandmaster, ex-Staff ML Engineer at Lyft, sharing valuable lessons and practical advice for anyone looking to enhance the adoption of their open-source projects.
Explore more about Albumentations and join the community at:
GitHub: https://github.com/albumentations-team/albumentations
Website: https://albumentations.ai/
LinkedIn: https://www.linkedin.com/company/100504475
Twitter: https://x.com/albumentations
Maruthi Prithivirajan, Head of ASEAN & IN Solution Architecture, Neo4j
Get an inside look at the latest Neo4j innovations that enable relationship-driven intelligence at scale. Learn more about the newest cloud integrations and product enhancements that make Neo4j an essential choice for developers building apps with interconnected data and generative AI.
How to Get CNIC Information System with Paksim Ga.pptxdanishmna97
Pakdata Cf is a groundbreaking system designed to streamline and facilitate access to CNIC information. This innovative platform leverages advanced technology to provide users with efficient and secure access to their CNIC details.
Full-RAG: A modern architecture for hyper-personalizationZilliz
Mike Del Balso, CEO & Co-Founder at Tecton, presents "Full RAG," a novel approach to AI recommendation systems, aiming to push beyond the limitations of traditional models through a deep integration of contextual insights and real-time data, leveraging the Retrieval-Augmented Generation architecture. This talk will outline Full RAG's potential to significantly enhance personalization, address engineering challenges such as data management and model training, and introduce data enrichment with reranking as a key solution. Attendees will gain crucial insights into the importance of hyperpersonalization in AI, the capabilities of Full RAG for advanced personalization, and strategies for managing complex data integrations for deploying cutting-edge AI solutions.
Threats to mobile devices are more prevalent and increasing in scope and complexity. Users of mobile devices desire to take full advantage of the features
available on those devices, but many of the features provide convenience and capability but sacrifice security. This best practices guide outlines steps the users can take to better protect personal devices and information.
National Security Agency - NSA mobile device best practices
Cloud API Issues: an Empirical Study and Impact
1. NICTA Copyright 2012 From imagination to impact
Cloud API Issues: an Empirical
Study and Impact
Qinghua Lu, Liming Zhu, Len Bass, Xiwei Xu,
Zhanwen Li, Hiroshi Wada
Software Systems Research Group, NICTA
QoSA13, Vancouver
Slides at: http://www.slideshare.net/LimingZhu/
2. NICTA Copyright 2012 From imagination to impact
Motivation
• Cloud applications fail due to operation issues
– Gartner reports: 80% of outage caused by operations
• People/Process: replication/failover, auto-scaling, upgrade…
– Lessons from our own cloud DR product: Yuruware.com
– DevOps movement
• Operational causes of failures
– Infrastructure and processes, but,
– Most things are done through infrastructure API
• Highly dependable cloud applications require
– Architecting for not just the software but also its operation (thru API)
– Architecting for indirect control (thru API)
– Better understanding of Cloud API Issues
• reliability, performance, nature of failures and faults 2
3. NICTA Copyright 2012 From imagination to impact
Main Contributions
• Empirical study of cloud infrastructure API issues
– 922 failure/fault cases from Amazon EC2 forums (2010 to 2012)
• Around five most used API calls
• Fault analysis supplemented by other sources
– Classified the API failures and faults (causes of failures)
• Using the classic dependable computing taxonomy (Avizienis,04)
• Failures: content, late timing, halt, erratic
• Faults: development, physical, interaction
• Impact analysis through an initial proposal for tolerating
cloud API failures/faults
– Suggestions for tolerating content failures
– 11 patterns for tolerating timing failures
3
4. NICTA Copyright 2012 From imagination to impact
Some Empirical Findings
• Majority (60%) of the cases of API failures are related to stuck API
calls or unresponsive API calls
• 19% of the cases are related to the output issues of API calls
– Error messages, missing/wrong/unexpected contents
• 12% of the cases are about slow responsive API calls
• 9% cases are related to API calls that
– were pending for a certain time and then returned to the original state
without informing the caller properly
– were reported to be successful first but failed later
4
5. NICTA Copyright 2012 From imagination to impact
Methodology
5
• Amazon: EC2 forums and outage reports
• Netflix: technical blogs and GitHub OSS projects
• Yuruware.com: disaster recovery product which heavily relies on cloud
infrastructure APIs
6. NICTA Copyright 2012 From imagination to impact
Data Collected from Amazon EC2 Forum
6
Searched keywords and number of returned records
API of Interests Number of records
from inception to 2012
Number of records
from 2010 to 2012
describe instance 283 150
start instance 227 204
stop instance 349 348
detach volume 235 203
associate elastic IP 264 204
Total 1358 1109
Case type, number and percentage in the found cases
Case type Case number Percentage of all cases
from 2010-2012
API failures 922 83%
Enquiries 125 11%
API enhancements 62 6%
7. NICTA Copyright 2012 From imagination to impact
Classification of API Failures
7
Fault -> Error -> Failure
Failure: deviation from correct
service (external visible)
Error: internal erroneous state
Fault: adjudicated or hypothesized
causes of a failure
[13] A. Avizienis, J. C. Laprie, B. Randell, and C. Landwehr, "Basic concepts and
taxonomy of dependable and secure computing," Dependable and Secure
Computing, IEEE Transactions on, vol. 1, pp. 11-33, 2004.
8. NICTA Copyright 2012 From imagination to impact
Classification of API Failures
• Content failures (19%)
– With error messages; missing/wrong/unexpected content
• 61% of the times users understood the causes/solutions from the error message
• 39% of the times users could not pinpoint the causes from the error message
8
Posted on Jan 10, 2012 5:42 AM
Symptom: When a user tried to start an instance, the operation failed with an unclear error
message.
Error message: State Transition Reason - Server.InternalError: Internal error on launch
Root cause: Unknown.
Solution: AWS engineers advised detaching the EBS volume from the instance and attaching it to
another running instance.
Posted on Jun 14, 2012 9:57PM
Symptom: Failed API calls and receiving Request limit exceeded error message.
Error message: Client.RequestLimitExceeded: Request limit exceeded
Root cause: API calls exceeded limit.
Solution: N/A. There is no official information on the limit or the time span on which the limit is
calculated or suggested wait time.
Failed call where the error message is unclear.
Failed call where the error message is clear.
9. NICTA Copyright 2012 From imagination to impact
Classification of API Failures
• Late timing failures (12%)
– the arrival time of the delivered information deviates from the
expected time but they do eventually arrive
9
A late timing failure example.
Posted on Aug 27, 2012 11:57 AM
Symptom: It took 16 minutes for an instance
to stop.
Root cause: n/a.
Solution: The AWS engineer advised to try
“force stop” twice if this happens next time.
10. NICTA Copyright 2012 From imagination to impact
Classification of API Failures
• Halt failures (60%)
– The external state becomes constant.
– Most frequent failures!
10
A general halt failure example.
Posted on Jun 27, 2012 12:04 AM
Symptom: A user reported that the instance is stuck at stopping and “force stop” would not help.
Root cause: n/a.
Solution: The AWS engineer stopped the instance for the user on the AWS side (with some side
effect).
A silent failure example.
Posted on Oct 23, 2012 7:45 AM
Symptom: An instance was not accessible and the user could not stop/start it or create a snapshot
Root cause: AWS outage.
Solution: The AWS engineer advised that the user must launch a replacement instance from a pre-
existing backup (EBS AMI). Attempts to stop an inaccessible instance will likely result in an instance
becoming stuck in the stopping state. Customers that do not have a known good backup must wait
for the issue to be resolved for their instance connectivity to be restored.
11. NICTA Copyright 2012 From imagination to impact
Classification of API Failures
• Erratic failures (35%)
– When the delivered service is unpredictable: Two subtypes:
• the call is pending for a certain time and then returns to the original state
• the call is successfully executed first but failed eventually
11
Two erratic failure examples.
Posted on Feb 1, 2012 8:15 AM
Symptom: A user associated an elastic IP with an instance and could SSH into the instance with the
elastic IP. After a few minutes, the elastic IP was silently disassociated from the instance.
Root cause: An issue with the underlying host.
Solution: The AWS engineer advised that the quickest fix was to stop and then start the instance to
relocate to a different host.
Posted on Jan 14, 2011 1:43 PM
Symptom: A user tried to start the instance several times. It indicated that the status is pending and
it goes back to stop.
Root cause: n/a.
Solution: The AWS engineer returned the user’s EBS volume to the available state and believed this
would resolve the user’s problem.
12. NICTA Copyright 2012 From imagination to impact
Classifying of Faults (Causes of Failures)
• Development faults – software bugs
– User workarounds exist but may break after bug fixing
• Physical faults
– Stopping/Starting to move to a new physical machine but
problematic stopping
– Future work: classifying using virtual resource characteristics
• Interaction faults
– Misconfiguration faults count for 30%
• Accidental & purposeful misconfiguration
– Purposeful misconfiguration
• lack of knowledge (subjective uncertainty vs. stochastic uncertainty)
• Configuration and operation impact on availability 1,2
1. X. Xu, Q. Lu, L. Zhu, et al., "Availability Analysis of In-Cloud Applications," in ISARCS13 (11:30
tomorrow)
2. Q. Lu, X. Xu, L. Zhu, L. Bass, et al., "Incorporating Uncertainty into in-Cloud Application
Deployment Decisions for Availability," in IEEE Cloud 2013 12
13. NICTA Copyright 2012 From imagination to impact
Tolerating API Failures/Faults
13
• Perspective
– cloud consumer and application oriented
– limited visibility: e.g. may not know the root cause
– indirect control: e.g. solutions are thru APIs as well
• Different failures/faults require different approaches
– Failure/Fault classification dependent
– Suggestions, patterns and ad-hoc use of failure/fault characteristics:
• Content failure: alternative sources for content, defensive programming…
• Late timing failures: API call life cycle driven
14. NICTA Copyright 2012 From imagination to impact
API Call Life Cycle Driven Patterns
14
15. NICTA Copyright 2012 From imagination to impact
Pattern Examples
• Faster forced fail/complete
– force-fail-r or force-fail-s
• Netflix Hystrix: fail fast based on 95-99 percentile delay
– force-complete-r
• Yuruware: ignore some “describe” API calls
• Hedged requests or more sophisticated retry
– continue-request
• Common: send the same request to 2 places and cancel the slow one
– reallocate or reallocate-s
• Yuruware: attach the to-be-moved volume to different mover instances
after early mover failures
15
16. NICTA Copyright 2012 From imagination to impact 16
Conclusion and Future Work
• Empirical study of cloud infrastructure API issues
– Analysed & classified 922 failure/faults from Amazon EC2 forums
• Inform better architecting for operations (i.e. operator as a stakeholder)
– Future work (completed)
• Expanded to more cases from other sources (2087 issues)
• Proposed a new scheme for classifying faults
• Tolerating cloud API failures/faults
– Patterns for tolerating different types of API failures/faults
– Future work (ongoing)
• More actionable mechanisms/patterns and their implementation
• Use the characteristics of the faults and failures
– for smarter recovery and error diagnosis during operation
• What we need: more real world operation logs and collaborators
{Liming.Zhu, Len.Bass}@nicta.com.au
Slides available at http://www.slideshare.net/LimingZhu/