This document discusses the concept of an autonomic cloud platform and operations engineering. It proposes using CFEngine as a configuration management tool to allow machines to manage themselves autonomously based on constraint models defined by users. This would maximize hardware performance and scalability while allowing deep system introspection and flexibility. Key aspects are recommended technologies like SmartOS, Joyent cloud, and DTrace for monitoring. The goal is for machines and humans to work as a team through evolving communication, trust, and a theory of mind approach rather than imperative commands that do not suit autonomous machine operation.
Moving to Microservices with the Help of Distributed TracesKP Kaiser
Moving away from a monolith to a microservices architecture is a process fraught with hidden challenges. There's legacy code, infrastructure, and organizational processes that all need to change, in order to make the switch successful.
But microservices come with a huge increase in infrastructure complexity. We'll see how distributed traces empower developers to work with greater autonomy, in increasingly complex deployment environments.
Making Observability Actionable At Scale - DBS DevConnect 2019Squadcast Inc
Many organisations already possess a vast amount of existing data about production systems. As customer expectations evolve, organisations are often challenged to find more proactive ways of dealing with traditionally reactive incident response activity. In this talk, we discuss approaches to unlock value from this data by making it truly actionable. Understanding production failure modes better, enriching technical and business context effectively, decomposing response activity into shared primitives, actions and workflows, and overall, sharing and augmenting this active knowledge repository on a continuous basis are key takeaways. Through case studies, we'll discuss how we can accomplish this by engineering your observability processes and tooling to work for human-in-the-loop interpretation and response rather than a purely human-reliant strategy.
Putting Devs On-Call: How to Empower Your TeamVictorOps
A main tenet of DevOps is bridging the gap between the Dev team and the Ops team. One way to accomplish this is to include devs in the on-call rotation. While this may sound difficult, it’s not impossible to do…as our guide demonstrates.
We profile four companies that have successfully transitioned their dev team to being on-call and their stories can provide examples for how you too can do it.
Getting software released to users can be risky, time-consuming and painful. The solution is the ability to deliver reliable software continuously through build, test and deployment automation, and through improved collaboration between developers, testers and operations. In this tutorial we will present principles and technical practices that enable teams to incrementally deliver software of high quality and value into production whenever they want, and extremely fast. The size of the project or the complexity of its code base does not matter.
In the first half of the tutorial we will introduce the concepts of continuous delivery, through continuous integration; and automation of the build, test and deployment process. We will also go through som basic principles and patterns for building automatable applications (architecture). We will cover experiences on team collaboration patterns and lastly; techniques for solving tasks such as an easy and comprehendible version control strategy.
The second half of the tutorial we will be working with automated provisioning of agile infrastructure, including the use of tools (puppet) to automate the management of testing and production environments. We will go through some scripting lessons examplifying how to implement zero-downtime deploys (… and rollback – if something goes wrong!), with examples in both bash and Ruby. Along with controlling the start, stop, restart lifecycles during deploys, we will also show some simple techniques for backups, logging, error handling, monitoring and verification of application health that can make the automation more robust.
We will also use servers "in the cloud" to demonstrate different techniques, and we hope to make it a fun day and to deliver software (examples) several times throughout the workshop.
Required knowledge: Agile/Lean basics, Linux basics, version control basics, maven basics.
Moving to Microservices with the Help of Distributed TracesKP Kaiser
Moving away from a monolith to a microservices architecture is a process fraught with hidden challenges. There's legacy code, infrastructure, and organizational processes that all need to change, in order to make the switch successful.
But microservices come with a huge increase in infrastructure complexity. We'll see how distributed traces empower developers to work with greater autonomy, in increasingly complex deployment environments.
Making Observability Actionable At Scale - DBS DevConnect 2019Squadcast Inc
Many organisations already possess a vast amount of existing data about production systems. As customer expectations evolve, organisations are often challenged to find more proactive ways of dealing with traditionally reactive incident response activity. In this talk, we discuss approaches to unlock value from this data by making it truly actionable. Understanding production failure modes better, enriching technical and business context effectively, decomposing response activity into shared primitives, actions and workflows, and overall, sharing and augmenting this active knowledge repository on a continuous basis are key takeaways. Through case studies, we'll discuss how we can accomplish this by engineering your observability processes and tooling to work for human-in-the-loop interpretation and response rather than a purely human-reliant strategy.
Putting Devs On-Call: How to Empower Your TeamVictorOps
A main tenet of DevOps is bridging the gap between the Dev team and the Ops team. One way to accomplish this is to include devs in the on-call rotation. While this may sound difficult, it’s not impossible to do…as our guide demonstrates.
We profile four companies that have successfully transitioned their dev team to being on-call and their stories can provide examples for how you too can do it.
Getting software released to users can be risky, time-consuming and painful. The solution is the ability to deliver reliable software continuously through build, test and deployment automation, and through improved collaboration between developers, testers and operations. In this tutorial we will present principles and technical practices that enable teams to incrementally deliver software of high quality and value into production whenever they want, and extremely fast. The size of the project or the complexity of its code base does not matter.
In the first half of the tutorial we will introduce the concepts of continuous delivery, through continuous integration; and automation of the build, test and deployment process. We will also go through som basic principles and patterns for building automatable applications (architecture). We will cover experiences on team collaboration patterns and lastly; techniques for solving tasks such as an easy and comprehendible version control strategy.
The second half of the tutorial we will be working with automated provisioning of agile infrastructure, including the use of tools (puppet) to automate the management of testing and production environments. We will go through some scripting lessons examplifying how to implement zero-downtime deploys (… and rollback – if something goes wrong!), with examples in both bash and Ruby. Along with controlling the start, stop, restart lifecycles during deploys, we will also show some simple techniques for backups, logging, error handling, monitoring and verification of application health that can make the automation more robust.
We will also use servers "in the cloud" to demonstrate different techniques, and we hope to make it a fun day and to deliver software (examples) several times throughout the workshop.
Required knowledge: Agile/Lean basics, Linux basics, version control basics, maven basics.
Video link: https://vimeo.com/131377935
Talk for Monitorama 2015. The history (or quasi history) of microservices and how to begin to approach monitoring from the POV of a smaller startup/company.
Understanding Artificial Intelligence - Major concepts for enterprise applica...APPANION
Artificial Intelligence is a fundamental topic – for us as humans, as a society but also for businesses. For business executives and decision-makers, it is sometimes hard to keep up with rapidly evolving technologies as part of the day-to-day business. By providing this curated compilation of information about the fundamental aspects of AI, we want to captivate and inspire you to become more involved with the technology by better understanding the underlying concepts and value drivers of this technology
Discussion - Weeks 1–2COLLAPSETop of FormShared Practice—Rol.docxcuddietheresa
Discussion - Weeks 1–2
COLLAPSE
Top of Form
Shared Practice—Role of Business Information Systems
Note: This Discussion has slightly different due dates than what is typical for this program. Be mindful of this as you post and respond in the Discussion. Your post is due on Day 7 and your Response is due on Day 3 of Week 2.
As a manager, it is critical for you to understand the types of business information systems available to support business operations, management, and strategy. As of 2013, these include, but are certainly not limited to the following:
· Supply Chain Management (SCM)
· Accounting Information System
· Customer Relationship Management (CRM)
· Decision Support Systems (DSS)
· Enterprise Resource Planning (ERP)
· Human Resource Management
These types of systems support critical business functions and operations that every organization must manage. The effective manager understands the purpose of these types of systems and how they can be best used to manage the organization's data and information.
In this Discussion, you will share your knowledge and findings related to business information systems and the role they play in your organization. You will also consider your colleagues' experiences to explore additional ways business information systems might be applied in your colleagues' organizations, or an organization with which you are familiar.
By Day 7
· Describe two or three of the more important technologies or business information systems used in your organization, or in one with which you are familiar.
· Discuss two examples of how these business information systems are affecting the organization you selected. Be sure to discuss how individual behaviors and organizational or individual processes are changing and what you can learn from the issues encountered.
· Summarize what you have learned about the importance of business information systems and why managers need to understand how systems can be used to the organization's advantage.
You should find and use at least one additional current article from a credible resource, either from the Walden Library or the Internet. Please be specific, and remember to use citations and references as necessary.
General Guidance: Your initial Discussion post, due by Day 7, will typically be 3–4 paragraphs in length as a general expectation/estimate. Refer to the rubric for the Week 1 Discussion for grading elements and criteria. Your Instructor will use the rubric to assess your work.
Week 2
By Day 3
In your Week 1 Discussion you described how business information systems have been applied in an organization with which you are familiar. Read through your colleagues' posts and by Day 3 (Week 2), respond to two of your colleagues in one or more of the following ways:
· Examine how the business information systems described by your colleague could be or are being used by your organization. Offer additional ways either organization might take advantage of these systems.
· Examine how the b ...
No Silver Bullet Essence and Accidents ofSoftware Engineeri.docxcurwenmichaela
No Silver Bullet: Essence and Accidents of
Software Engineering
by Frederick P. Brooks, Jr.
Of all the monsters that fill the nightmares of our folklore, none terrify more than werewolves, because
they transform unexpectedly from the familiar into horrors. For these, one seeks bullets of silver that can
magically lay them to rest.
The familiar software project, at least as seen by the nontechnical manager, has something of this
character; it is usually innocent and straightforward, but is capable of becoming a monster of missed
schedules, blown budgets, and flawed products. So we hear desperate cries for a silver bullet--something
to make software costs drop as rapidly as computer hardware costs do.
But, as we look to the horizon of a decade hence, we see no silver bullet. There is no single
development, in either technology or in management technique, that by itself promises even one order-
of-magnitude improvement in productivity, in reliability, in simplicity. In this article, I shall try to show
why, by examining both the nature of the software problem and the properties of the bullets proposed.
Skepticism is not pessimism, however. Although we see no startling breakthroughs--and indeed, I
believe such to be inconsistent with the nature of software--many encouraging innovations are under
way. A disciplined, consistent effort to develop, propagate, and exploit these innovations should indeed
yield an order-of-magnitude improvement. There is no royal road, but there is a road.
The first step toward the management of disease was replacement of demon theories and humours
theories by the germ theory. That very step, the beginning of hope, in itself dashed all hopes of magical
solutions. It told workers that progress would be made stepwise, at great effort, and that a persistent,
unremitting care would have to be paid to a discipline of cleanliness. So it is with software engineering
today.
Does It Have to Be Hard?--Essential Difficulties
Not only are there no silver bullets now in view, the very nature of software makes it unlikely that there
will be any--no inventions that will do for software productivity, reliability, and simplicity what
electronics, transistors, and large-scale integration did for computer hardware. We cannot expect ever to
see twofold gains every two years.
First, one must observe that the anomaly is not that software progress is so slow, but that computer
hardware progress is so fast. No other technology since civilization began has seen six orders of
magnitude in performance price gain in 30 years. In no other technology can one choose to take the gain
in either improved performance or in reduced costs. These gains flow from the transformation of
computer manufacture from an assembly industry into a process industry.
Second, to see what rate of progress one can expect in software technology, let us examine the
difficulties of that technology. Following Aristotle, I divide them into essence, the difficulties inhe ...
Artificial Intelligence (AI) -> understanding what it is & how you can use it...Adela VILLANUEVA
The goal of this presentation is to provide you with a basic understanding of AI and to prepare you to think about how your organization might apply it.
Prometheus is a next-generation monitoring system. It lets you see you not just what your systems look like from the outside, but also gives visibility into the internals and business aspects of your systems. This allows everyone to benefit, including both operations and developers. This talk will look at the concepts behind monitoring with Prometheus, how it's designed, why it's suitable for Cloud Native environments and how you can get involved.
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
More Related Content
Similar to Velocity conf 2013 freedom in disguise - khushil dep
Video link: https://vimeo.com/131377935
Talk for Monitorama 2015. The history (or quasi history) of microservices and how to begin to approach monitoring from the POV of a smaller startup/company.
Understanding Artificial Intelligence - Major concepts for enterprise applica...APPANION
Artificial Intelligence is a fundamental topic – for us as humans, as a society but also for businesses. For business executives and decision-makers, it is sometimes hard to keep up with rapidly evolving technologies as part of the day-to-day business. By providing this curated compilation of information about the fundamental aspects of AI, we want to captivate and inspire you to become more involved with the technology by better understanding the underlying concepts and value drivers of this technology
Discussion - Weeks 1–2COLLAPSETop of FormShared Practice—Rol.docxcuddietheresa
Discussion - Weeks 1–2
COLLAPSE
Top of Form
Shared Practice—Role of Business Information Systems
Note: This Discussion has slightly different due dates than what is typical for this program. Be mindful of this as you post and respond in the Discussion. Your post is due on Day 7 and your Response is due on Day 3 of Week 2.
As a manager, it is critical for you to understand the types of business information systems available to support business operations, management, and strategy. As of 2013, these include, but are certainly not limited to the following:
· Supply Chain Management (SCM)
· Accounting Information System
· Customer Relationship Management (CRM)
· Decision Support Systems (DSS)
· Enterprise Resource Planning (ERP)
· Human Resource Management
These types of systems support critical business functions and operations that every organization must manage. The effective manager understands the purpose of these types of systems and how they can be best used to manage the organization's data and information.
In this Discussion, you will share your knowledge and findings related to business information systems and the role they play in your organization. You will also consider your colleagues' experiences to explore additional ways business information systems might be applied in your colleagues' organizations, or an organization with which you are familiar.
By Day 7
· Describe two or three of the more important technologies or business information systems used in your organization, or in one with which you are familiar.
· Discuss two examples of how these business information systems are affecting the organization you selected. Be sure to discuss how individual behaviors and organizational or individual processes are changing and what you can learn from the issues encountered.
· Summarize what you have learned about the importance of business information systems and why managers need to understand how systems can be used to the organization's advantage.
You should find and use at least one additional current article from a credible resource, either from the Walden Library or the Internet. Please be specific, and remember to use citations and references as necessary.
General Guidance: Your initial Discussion post, due by Day 7, will typically be 3–4 paragraphs in length as a general expectation/estimate. Refer to the rubric for the Week 1 Discussion for grading elements and criteria. Your Instructor will use the rubric to assess your work.
Week 2
By Day 3
In your Week 1 Discussion you described how business information systems have been applied in an organization with which you are familiar. Read through your colleagues' posts and by Day 3 (Week 2), respond to two of your colleagues in one or more of the following ways:
· Examine how the business information systems described by your colleague could be or are being used by your organization. Offer additional ways either organization might take advantage of these systems.
· Examine how the b ...
No Silver Bullet Essence and Accidents ofSoftware Engineeri.docxcurwenmichaela
No Silver Bullet: Essence and Accidents of
Software Engineering
by Frederick P. Brooks, Jr.
Of all the monsters that fill the nightmares of our folklore, none terrify more than werewolves, because
they transform unexpectedly from the familiar into horrors. For these, one seeks bullets of silver that can
magically lay them to rest.
The familiar software project, at least as seen by the nontechnical manager, has something of this
character; it is usually innocent and straightforward, but is capable of becoming a monster of missed
schedules, blown budgets, and flawed products. So we hear desperate cries for a silver bullet--something
to make software costs drop as rapidly as computer hardware costs do.
But, as we look to the horizon of a decade hence, we see no silver bullet. There is no single
development, in either technology or in management technique, that by itself promises even one order-
of-magnitude improvement in productivity, in reliability, in simplicity. In this article, I shall try to show
why, by examining both the nature of the software problem and the properties of the bullets proposed.
Skepticism is not pessimism, however. Although we see no startling breakthroughs--and indeed, I
believe such to be inconsistent with the nature of software--many encouraging innovations are under
way. A disciplined, consistent effort to develop, propagate, and exploit these innovations should indeed
yield an order-of-magnitude improvement. There is no royal road, but there is a road.
The first step toward the management of disease was replacement of demon theories and humours
theories by the germ theory. That very step, the beginning of hope, in itself dashed all hopes of magical
solutions. It told workers that progress would be made stepwise, at great effort, and that a persistent,
unremitting care would have to be paid to a discipline of cleanliness. So it is with software engineering
today.
Does It Have to Be Hard?--Essential Difficulties
Not only are there no silver bullets now in view, the very nature of software makes it unlikely that there
will be any--no inventions that will do for software productivity, reliability, and simplicity what
electronics, transistors, and large-scale integration did for computer hardware. We cannot expect ever to
see twofold gains every two years.
First, one must observe that the anomaly is not that software progress is so slow, but that computer
hardware progress is so fast. No other technology since civilization began has seen six orders of
magnitude in performance price gain in 30 years. In no other technology can one choose to take the gain
in either improved performance or in reduced costs. These gains flow from the transformation of
computer manufacture from an assembly industry into a process industry.
Second, to see what rate of progress one can expect in software technology, let us examine the
difficulties of that technology. Following Aristotle, I divide them into essence, the difficulties inhe ...
Artificial Intelligence (AI) -> understanding what it is & how you can use it...Adela VILLANUEVA
The goal of this presentation is to provide you with a basic understanding of AI and to prepare you to think about how your organization might apply it.
Prometheus is a next-generation monitoring system. It lets you see you not just what your systems look like from the outside, but also gives visibility into the internals and business aspects of your systems. This allows everyone to benefit, including both operations and developers. This talk will look at the concepts behind monitoring with Prometheus, how it's designed, why it's suitable for Cloud Native environments and how you can get involved.
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...Ramesh Iyer
In today's fast-changing business world, Companies that adapt and embrace new ideas often need help to keep up with the competition. However, fostering a culture of innovation takes much work. It takes vision, leadership and willingness to take risks in the right proportion. Sachin Dev Duggal, co-founder of Builder.ai, has perfected the art of this balance, creating a company culture where creativity and growth are nurtured at each stage.
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...UiPathCommunity
💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™:
See how to accelerate model training and optimize model performance with active learning
Learn about the latest enhancements to out-of-the-box document processing – with little to no training required
Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages
This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath.
Speakers:
👨🏫 Andras Palfi, Senior Product Manager, UiPath
👩🏫 Lenka Dulovicova, Product Program Manager, UiPath
Securing your Kubernetes cluster_ a step-by-step guide to success !KatiaHIMEUR1
Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster.
However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks.
In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.
Key Trends Shaping the Future of Infrastructure.pdfCheryl Hung
Keynote at DIGIT West Expo, Glasgow on 29 May 2024.
Cheryl Hung, ochery.com
Sr Director, Infrastructure Ecosystem, Arm.
The key trends across hardware, cloud and open-source; exploring how these areas are likely to mature and develop over the short and long-term, and then considering how organisations can position themselves to adapt and thrive.
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Tobias Schneck
As AI technology is pushing into IT I was wondering myself, as an “infrastructure container kubernetes guy”, how get this fancy AI technology get managed from an infrastructure operational view? Is it possible to apply our lovely cloud native principals as well? What benefit’s both technologies could bring to each other?
Let me take this questions and provide you a short journey through existing deployment models and use cases for AI software. On practical examples, we discuss what cloud/on-premise strategy we may need for applying it to our own infrastructure to get it to work from an enterprise perspective. I want to give an overview about infrastructure requirements and technologies, what could be beneficial or limiting your AI use cases in an enterprise environment. An interactive Demo will give you some insides, what approaches I got already working for real.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
Elevating Tactical DDD Patterns Through Object CalisthenicsDorra BARTAGUIZ
After immersing yourself in the blue book and its red counterpart, attending DDD-focused conferences, and applying tactical patterns, you're left with a crucial question: How do I ensure my design is effective? Tactical patterns within Domain-Driven Design (DDD) serve as guiding principles for creating clear and manageable domain models. However, achieving success with these patterns requires additional guidance. Interestingly, we've observed that a set of constraints initially designed for training purposes remarkably aligns with effective pattern implementation, offering a more ‘mechanical’ approach. Let's explore together how Object Calisthenics can elevate the design of your tactical DDD patterns, offering concrete help for those venturing into DDD for the first time!
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualityInflectra
In this insightful webinar, Inflectra explores how artificial intelligence (AI) is transforming software development and testing. Discover how AI-powered tools are revolutionizing every stage of the software development lifecycle (SDLC), from design and prototyping to testing, deployment, and monitoring.
Learn about:
• The Future of Testing: How AI is shifting testing towards verification, analysis, and higher-level skills, while reducing repetitive tasks.
• Test Automation: How AI-powered test case generation, optimization, and self-healing tests are making testing more efficient and effective.
• Visual Testing: Explore the emerging capabilities of AI in visual testing and how it's set to revolutionize UI verification.
• Inflectra's AI Solutions: See demonstrations of Inflectra's cutting-edge AI tools like the ChatGPT plugin and Azure Open AI platform, designed to streamline your testing process.
Whether you're a developer, tester, or QA professional, this webinar will give you valuable insights into how AI is shaping the future of software delivery.
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
2. This presentation is NOT …
A detailed description of how the MailOnline uses cfengine
Comparing cfengine to puppet/chef/salt/ansible etc
Tutoring you on configuration management - that’s easy and it’s well understood
Selling you any tool or product mentioned herein
Providing you the answers to all your questions
Telling you what tools you should/shouldn’t be using
Regurgitating the last five years of rigmarole
Ruby
3. This presentation is about …
FINDING a definition for an autonomic cloud platform
EXAMINING our definition of systems engineering
EVOLVING our understanding of operations engineering
LEARNING cfengine and how to use and abuse it
EXPLORING the promise of autonomic machine operation
4. What does an autonomic cloud platform look like?
Maximises the performance available from the underlying hardware to increase ROI.
Scales robustly and easily both vertically and horizontally, programmatically
Provides durable, secure, fast and reliable storage.
Allows deep, full stack introspection easily and without restriction.
Protects your instances from the effects or attacks of others on the same platform.
Provides a flexible service management toolset.
Defines an open set of API’s which allow access to the full range of capabilities and data.
5. Recommended Technology Stack
SmartOS operating system based on Illumos kernel with ZFS, DTrace, SMF and Zones.
Based off the Illumos OpenSolaris fork.
Joyent Public Cloud for a real alternative to Amazon Web Services.
More horsepower for your dollar/pound/euro than AWS in my experience. YMMV.
Engineering excellence - Bryan Cantrill, Brendan Gregg, Ben Rockwood to name a couple
of greats who work there.
6. The Halcyon Dream
User defined software managed by
release manager.
cf-engine allows the machine to
manage itself under our constraint
models.
DTrace provides diagnostic and
performance feedback across the
stack.
JoyentSDC APIs provide easy
functionality for all platform
operations.
7. Systems Engineering is the application
of technical expertise, diligence,
reflection, communication,
collaboration, patience and innovation
between multi-disciplinary teams to
create something of use.
8.
9. What’s wrong with DevOps?
DevOps (a portmanteau of development and operations) is a software
development method that stresses communication, collaboration and
integration between software developers and information technology
(IT) professionals. DevOps is a response to the interdependence of
software development and IT operations. It aims to help an
organisation rapidly produce software products and services.
10. What we do to our machines at the moment …
Imperative approach to machine operation
Obligatory behavioural model where we force or
coerce our machines into actions
Unsympathetic to real-time environmental events
Scales inefficiently introducing risk
You begin to doubt your machines
You being to doubt yourself
13. What we should be getting our machines to do …
“I’m sorry Dave, I can’t do that…. right now. It would be unwise. I
am aware of events in realtime that would affect the outcome of
your request adversely that you are not aware of.
Don’t worry, I promise to do it as soon as the probability of
success has improved and will orchestrate the dependant
activities so you reach your goals.
There is a lot else you could be doing right now - you don’t have
to wait for me Dave.
I’ve got you Dave. We are a team.
Did you see the game last night?”
14. What is Operations Engineering anyway …
Enable better machine/human relationships through
evolving theory of mind, communication and trust
15. Theory of Mind (ToM)
The ability to attribute mental states to oneself and others and to
understand that others also have mental states that may differ
from one’s own. The basis of empathy some might say.
16. Communication
The activity of conveying information through an exchange of
thoughts, messages or information as by speech, visual signals,
writing or by behaviour. It is the meaningful exchange of
information between two or more entities.
17. Meaningful Information
Knowledge communicated or received concerning a
particular fact or circumstance, expressed with clarity,
within context, concisely, in a timely manner,
significant and with purpose.
18. Trust
Socially we require trust when operating on, and often, beyond the edge of what is known
through practical experience and that which may arise from new possibilities.
!
Psychologically human trust is believing that an entity that is trusted will do what is expected of
it by you.
!
We must trust our machines to operate not only within known boundaries but those that we
cannot yet envisage.
19. The Machine
Electro-mechanical computation engine with mechanisms for input and output.
Capable of sustained activity without distraction nor deviation.
Fast, accurate, reliable and repeatable task handling.
No capacity for independent imagination.
Able to generate and analyse vast quantities of information.
Inefficient and communicating with humans.
20. I, Human
Biological system with facilities for input and output.
Incapable of sustained activity without distraction and deviation.
Slow, prone to error, unreliable and fragile task handling.
Excessive capacity for independent imagination.
Unable to generate and analyse vast quantities of information.
Inefficient and communicating with anyone or anything.
21. Why we fail our Machines
Human process is based on human understanding of events.
Human process is designed for human implementation.
Human process maps dangerously to machine computation and understanding of events.
Humans have hidden sanity checking which machines are unable to deduce or reproduce.
Human imperative command structures do not suit machines which are convergent by nature.
Humans are unable to maintain focus.
22. VIEW YOUR GOALS FROM THE MACHINES PERSPECTIVE
!
YOUR MACHINES CARRY YOUR LOGIC INTO ACTION
23. Promises
Promises are the foundations of trust between two entities - the machine and the human.
The promiser requires certain promises from the promisee.
Our need for trust in our machines is fulfilled by the machine promising to actuate the
promises we request from it.
We ask that the machine promises these actuations in a timely manner in the machines
context - not our own.
We declare goals and ask the machine to converge on that goal when it’s able to do so.
24. Promise Theory
Proposed by Mark Burgess in 2004.
Autonomy - we do not make assumptions about others behaviour. We only
document that behaviour on which we can speak authoritatively. This forces us to
more completely define what behaviour we are able to promise and this in turn
leads us to a more complete understanding of what are desired state entails.
Emergent Behaviour - when we behave in a model of voluntary cooperation as
independent and autonomous agents, certain behavioural patterns must naturally
emerge. The atomicity of promises enables us to better understand what we are
promising and thus find those contradictions which might else have been missed.
25. The Forgotten Orchestration
An operating system is a complex collection of software that orchestrates computer hardware
resources and provides common services for user defined software.
All user defined software sits within this orchestration framework and is already being orchestrated.
It is dangerous therefore to try to further orchestrate user defined software in an obligatory
manner.
Obligatory behaviour is where you force or coerce behaviour from your software or the operating
system with little or no regard to the underlying orchestration.
It’s a bit like the Universe really…
26. Universal Orchestration
QUARK - packets of energy with mass-like properties which exist in pairs or triplets (we think).
PROTON/NEUTRON - three or more quarks interacting and oh you know, orchestrating!
ATOM - one or more protons, zero or more neutrons and some electrons whizzing about orchestrating the hell out of a snazzy
number!
MOLECULE - made of atoms all orchestrating to their own private adagio!
ORGANELLE - molecules that orchestrate their way to things like a cell nucleus or ribosomes!
CELL - organelles just orchestrating away the Sunday afternoon with things like cytoplasm to make the little things, like the building
blocks of life.
TISSUE - a set of cells orchestrating their way out of boredom to make things like muscle tissue or heart tissue.
ORGAN - a set of at least two types of tissues orchestrating their way through a chorus to something like a heart or a pair of lungs.
28. Human Orchestration
Micro-management incurs great operational cost and entails an increased risk from unknowable events.
You neither force nor coerce your engineers into behavioural patterns which are alien to them.
You trust your engineers to adapt and use their own preferred behaviour in order to reach set goals.
Your goals might be decided for you by other people or organisations.
These people and organisations trust you to meet the goals and keep your promises.
Why then do you not apply the same trust to your machines?
31. What is it?
A systems engineering framework that enables autonomous behaviour of agents.
Created by Mark Burgess in the early 1990’s - the original DevOps tool.
What the hell is DevOps anyway - it’s called Engineering, stop making up words!
Written in C and runs on most unices and even Windows.
Small footprint, very fast execution.
Best of all …
33. Autonomic Operation
Machines are best placed to make decisions based on environmental conditions.
Machines do not require obligatory behaviour imposed upon them by humans.
Machines require logic and freedom to enable them to achieve the goals we set them.
This area needs more research and experimentation.
34. Components of CFengine
cf-execd - scheduling daemon which runs cf-agent, gathers output and
send reports.
cf-agent - evaluates policies and actuates changes to the machine.
cf-monitord - samples probes defined in policies and attempts to learn
normal system state.
cf-server - daemon which allows authorised access to policy files and
allow authorised access to cf-runagent.
cf-runagent - connects to a list of cf-server instances and is able to ask
for policy evaluation on these instances foregoing the normal cf-execd
scheduling on these instances
https://cfengine.com/docs/3.5/manuals-components.html
35. Components of CFengine
cf-hub - collects data about hosts managed by cfengine.
cf-promises - policy validation tool to aid development. Parses policies for
syntax errors. Validates policies composed on multiple files. Validates semantic
correctness of policies. Partially evaluates policies to expose any errors.
Makes NO CHANGES to the system.
cf-key - generates key pairs for remote authentication.
36. COMMUNITY TOOLS
The following from http://www.cfengineers.net/downloads/cfengine-tools-and-utilities/
§ cf-keycrypt - encrypt/decrypt arbitrary files using cfengine crypto keys for extra security.
§ cf-profile - parses verbose cf-agent execution and records timings and execution trees.
§ cf-runwrapper - cf-runagent wrapper for extra control over cf-runagent behaviour
!
The following from https://github.com/lpefferkorn/cfe-profiler
§ cfe-profiler - measures policy execution times to find top consumers inline with run.
!
The following from https://github.com/cfengine/design-center/tree/master/tools/hcgrep
§ hcgrep - make Hard Classes easier to view and search for
37. Promise Recap
Make a promise about something and cfengine will attempt to keep it.
Each promise is actuated three times to allow convergence to occur.
Everything is a promise and some have commitments:
§ A file exists.
§ It commits to being owned by root
§ A user is present on the system.
§ It commits to having a home directory at /home/khushil
§ The CPU load is below a certain value.
The policies in cfengine are comprised of promises
Convergence is about making the promises to get to the state we want to be in.
What do you need? What must you avoid? Define these promises.
Don’t get distracted by how you get there!
38. Language concepts
https://cfengine.com/docs/3.5/manuals-language-concepts.html
One grammatical form for all statements in the cfengine DSL.
It is a DOMAIN SPECIFIC LANGUAGE.
Everything in cfengine is made thus.
Promises
Bundles
Bodies
Classes / Classifiers
Variables
Datatypes
bundle bundle_type name
{
promise_type:
classes::
“promiser" -‐> { "promisee1", "promisee2", ... }
attribute_1 => value_1,
attribute_2 => value_2,
…
attribute_n => value_n;
}
39. How MailOnline thinks about promises
Discovery - examine the system and raise policy defined global classes to
augment hard classes discovered by cfengine.
Contract - select from a set of pre-defined bundles of promises which are
relevant to our desired state for the machine within it’s function context.
Actuation - selected contracts will ensure that our desired state is converged
upon as quickly and safely as possible.
40. An example desired state to converge upon
I want to deploy our snazzy new web application.
nginx is required on the machine.
An application specific nginx configuration is required for nginx.
Our application code must be on the machine
It must be accessible my the nginx service
It must be secure
nginx must be running to serve traffic
If any of these promises are not met we will not reach our goal.
41. Simple language rules
https://cfengine.com/docs/3.5/reference-syntax.html
Keywords, variable names, bundles, bodies and classes must be composed of (a-zA-Z-9_)
Literal data must be quoted
Promise bundles are declared thus:
bundle agent-type identifier { … }
Promise bodies are declared thus:
body constraint_type template_identifier { … }
Body attributes are declared thus:
LHS (cfengine word) => RHS (user defined data)
42. Classes / Classifiers
https://cfengine.com/docs/3.5/manuals-language-concepts-classes.html
HARD classes are discovered by cfengine upon cf-agent execution before any other policies are
converged.
27_0_0_1 64_bit 8_cpus Afternoon
SOFT classes are user defined and used to implement classification and logic.
Evaluated when bundles are evaluated.
Classes are LOCAL to the bundle they are defined in and are NOT accessible outside the bundle.
Classes defined in common bundles ARE accessible as they are GLOBAL classes within the
namespace.
Classes can be raised by promises upon promise outcomes to further aid classification and logic.
44. Scalar variables hold single values:
List variables hold several values:
!
!
vars:
Vars:
“little”
string
=>
“little”;
“strings”
slist
=>
{“this”,”is”,”a”,”list”};
“boy”
int
=>
“4”;
“ints”
ilist
=>
{“1”,”2”,”3”};
“blue”
rela
=>
“3.147”;
“reals”
rlist
=>
{“1.1”,”2.2”,”3.3”};
!
!
$(little) or $(bundle_name.little)
${little} or ${bundle_name.little}
@(strings) refers to the whole list
$(strings) will loop through each element
46. Promises
https://cfengine.com/docs/3.5/manuals-language-concepts-promises.html
Everything is a promise
Promises have types
files, commands, methods, reports, packages, processes, storage, services, databases, guest_environments, outputs
Promises have bodies
Promises are grouped into bundles
Bundles have types
agent, common, edit_line, server, knowledge, monitor
Bundles live in namespaces
Namespaces have access to global classes
Promises can call modules which are external scripts
Modules can be sent classes or variables on calling
Modules can send classes or variables on exit
47. Everything is a promise
Promises can be made about different
subjects such as command execution,
service control, ACL’s.
Promises have types depending on the
bundle you’re working in.
Promisers promise and can be any
object such as a file or network or even
a port.
Promises have attributes which affect the
behaviour of the promise.
Implicit promises such as reports or
commands have implicit behaviour.
https://cfengine.com/docs/3.5/reference-promise-types.html
49. Looping in CFengine
https://cfengine.com/docs/3.5/manuals-language-concepts-loops.html
No explicit loops anywhere to be found!
cfengine uses lists
Referencing a list as a scalar will cause implicit
looping through the list
Powerful behaviour
Demands a shift in thinking
bundle agent implicit_looping
{
vars:
“my_list”
slist
=>
{“a”,”b”,”c”,”d”};
!
}
reports:
“$(my_list)”;
50. Multiple list looping in CFengine
2013-‐11-‐12T15:34:11+0000
notice:
R:
Simple
list
element
is
a
2013-‐11-‐12T15:34:11+0000
notice:
R:
Simple
list
element
is
b
2013-‐11-‐12T15:34:11+0000
notice:
R:
Simple
list
element
is
c
bundle
common
control
{
bundlesequence
=>
{"simple_implicit_looping",
"stats_iteration"};
}
!
bundle
agent
simple_implicit_looping
{
vars:
"simple_list"
slist
=>
reports:
"Simple
list
element
is
";
}
2013-‐11-‐12T15:34:11+0000
notice:
R:
Simple
list
element
is
d
2013-‐11-‐12T15:34:11+0000
notice:
R:
mon.value_rootprocs
is
230.00
2013-‐11-‐12T15:34:11+0000
notice:
R:
mon.av_rootprocs
is
209.30
2013-‐11-‐12T15:34:11+0000
notice:
R:
mon.dev_rootprocs
is
150.77
{
"a","b","c","d"
};
!
bundle
agent
stats_iteration
{
vars:
"stats"
slist
=>
{
"value",
"av",
"dev"
};
"monvars"
slist
=>
{
"rootprocs",
"otherprocs",
"diskfree",
"loadavg"
};
reports:
"mon.$(stats)_$(monvars)
is
$(mon.$(stats)_$(monvars))";
}
2013-‐11-‐12T15:34:11+0000
notice:
R:
mon.value_otherprocs
is
17.00
2013-‐11-‐12T15:34:11+0000
notice:
R:
mon.av_otherprocs
is
15.47
2013-‐11-‐12T15:34:11+0000
notice:
R:
mon.dev_otherprocs
is
11.16
2013-‐11-‐12T15:34:11+0000
notice:
R:
mon.value_diskfree
is
93.00
2013-‐11-‐12T15:34:11+0000
notice:
R:
mon.av_diskfree
is
84.63
2013-‐11-‐12T15:34:11+0000
notice:
R:
mon.dev_diskfree
is
61.07
2013-‐11-‐12T15:34:11+0000
notice:
R:
mon.value_loadavg
is
1.36
2013-‐11-‐12T15:34:11+0000
notice:
R:
mon.av_loadavg
is
1.44
2013-‐11-‐12T15:34:11+0000
notice:
R:
mon.dev_loadavg
is
3.10
51. It’s not rocket science
bundle
agent
ensure_nginx_configuration_file()
{
!
#
The
files
section
deals
with
promising
things
about,
well,
files!
files:
!
#
We
only
want
to
do
the
following
if
we’re
on
a
Linux
box
indicated
by
the
‘linux’
hard
class
linux::
#
This
is
the
file
we
want
to
promise
will
be
there
“/opt/local/etc/nginx.conf”
!
#
These
are
the
properties
about
the
promise
create
=>
“true;
source
=>
local_dcp(“/var/cfengine/inputs/templates/linux-‐nginx.conf”);
perms
=>
mog(“755”,”root”,”root”);
handle
=>
“ensure_linux_nginx_conf_maintained”;
comment
=>
“Maintain
the
nginx.conf
file
for
Linux
machines”;
classes
=>
if_repaired(“nginx_file_changed”);
!
#
The
reports
section
allows
us
to
output
messages
to
the
user
reports:
!
#
We
only
want
to
say
something
if
something
has
changed
nginx_file_changed::
“WARNING:
NGINX
configuration
file
has
been
changed!
Restart
required!”;
!
}
52. Further Reading & Doing
www.cfengine.com - CFengine AS company site. Documentation and a good starting point.
www.daemondreams.co.uk - an updated blog site where I keep articles and notes of interest on cfengine.
www.cfengineers.net - a community focused site lead by a group of consultants in cfengineering.
www.watson-wilson.ca - a highly recommended cfengineering consultant.
www.normation.com - a commercial cfengineering company with a project called Rudder which is cool.
www.loicp.eu/blog - a cfengine centric blog exploring some newer functions in 3.5.
evolvethinking.com/evolve-thinkings-free-cfengine-library/ - good cfengine library to help you get started.