This presentation discusses two key components of our deployment pipeline: Continuous integration of Chef code and automated deployment of Java applications. CI jobs for Chef code run static analysis and then provision, configure and test EC2 instances. Release jobs publish new cookbook versions to the Chef server. Deployment jobs identify target EC2 and VMware nodes and orchestrate Chef client runs. The flexibility of Jenkins is essential to our overall delivery architecture.
How to do Test Driven Development to write Chef cookbooks?
This slide-deck explains how to use TDD and tools of the trade, to develop cookbooks, Unit Tests and Integration Tests.
Source code is available on GitHub: https://github.com/siso/cheftdd-cookbook
Drupal Continuous Integration with Jenkins - The BasicsJohn Smith
Please check out our new SlideShow of setting up and configuring a Jenkins Continuous Integration server for use within a Drupal development environment. We walk you through the steps of installing Ubuntu 10.04 LTS, Jenkins, Drush and several other PHP coding tools and Drupal Modules to help check your code against current Drupal standards. Then we walk you through creating a git post-receive script, and Jenkins job to pull it all together.
Introduction to Chef: Automate Your Infrastructure by Modeling It In CodeJosh Padnick
Presentation by Josh Padnick given at Desert Code Camp on April 5, 2014. Introduces OpsCode Chef with a special emphasis on learning the key Chef concepts. Also includes tips & tricks and references to best practices.
How to do Test Driven Development to write Chef cookbooks?
This slide-deck explains how to use TDD and tools of the trade, to develop cookbooks, Unit Tests and Integration Tests.
Source code is available on GitHub: https://github.com/siso/cheftdd-cookbook
Drupal Continuous Integration with Jenkins - The BasicsJohn Smith
Please check out our new SlideShow of setting up and configuring a Jenkins Continuous Integration server for use within a Drupal development environment. We walk you through the steps of installing Ubuntu 10.04 LTS, Jenkins, Drush and several other PHP coding tools and Drupal Modules to help check your code against current Drupal standards. Then we walk you through creating a git post-receive script, and Jenkins job to pull it all together.
Introduction to Chef: Automate Your Infrastructure by Modeling It In CodeJosh Padnick
Presentation by Josh Padnick given at Desert Code Camp on April 5, 2014. Introduces OpsCode Chef with a special emphasis on learning the key Chef concepts. Also includes tips & tricks and references to best practices.
This presentation was held at the DevOps Meetup Frankfurt on 2014/12/08. It describes some tools and practices for testing your infrastructure provisioned with Chef.
Community Cookbooks & further resources - Fundamentals Webinar Series Part 6Chef
Part 6 of a 6 part series introducing you to the fundamentals of Chef.
This session includes an introducing Community Cookbooks and some additional resources.
After viewing this webinar you will be able to:
- Find, preview, and download cookbooks from the Chef Community site
- Use knife to work with the Community Site API
- Download, extract, examine and implement cookbooks from the Community site
Video of this webinar can be found at the following URL
https://www.youtube.com/watch?v=ovTIeS3kx4g&list=PL11cZfNdwNyPnZA9D1MbVqldGuOWqbumZ
Overview of Chef - Fundamentals Webinar Series Part 1Chef
This is an Overview of Chef. After viewing this webinar you will be able to:
- Describe how Chef thinks about Infrastructure Automation
- Define the following terms:
- Resource
- Recipe
- Node
- Run List
- Search
- Login to Hosted Chef
- Run `knife` commands from your workstation
Video of this webinar can be found at the following URL
https://www.youtube.com/watch?v=S5lHUpzoCYo&list=PL11cZfNdwNyPnZA9D1MbVqldGuOWqbumZ
Chef vs Puppet vs Ansible vs Saltstack | Configuration Management Tools | Dev...Simplilearn
This presentation "Chef vs Puppet vs Ansible vs Saltstack" will compare the DevOps configuration management tools Chef, Puppet, Ansible and Saltstack in terms of their capabilities, architecture, performance, ease of setup, language, scalability and pros and cons. The chef is a configuration management tool written in Ruby and Erlang. Puppet is an open-source software configuration management tool that runs on many Unix-like systems and also Windows. Ansible is yet another tool that automates software provisioning, configuration management, and application deployment. Saltstack is a Python-based open-source configuration management tool. Now, let us get started and get to know which is the best configuration management platform among Chef, Puppet, Ansible and Saltstack.
Below are the contents of our "Chef vs Puppet vs Ansible vs Saltstack" configuration management tools comparison slides:
1) Need for Configuration Management Tools
2) Chef - Infrastructure, Architecture, Pros and Cons
3) Puppet- Infrastructure, Architecture, Pros and Cons
4) Ansible - Infrastructure, Architecture, Pros and Cons
5) Saltstack - Infrastructure, Architecture, Pros and Cons
6) Comparison on the basis of architecture, ease of setup, language, scalability, management and interoperability.
Why learn DevOps?
Simplilearn’s DevOps training course is designed to help you become a DevOps practitioner and apply the latest in DevOps methodology to automate your software development lifecycle right out of the class. You will master configuration management; continuous integration deployment, delivery and monitoring using DevOps tools such as Git, Docker, Jenkins, Puppet and Nagios in a practical, hands-on and interactive approach. The DevOps training course focuses heavily on the use of Docker containers, a technology that is revolutionizing the way apps are deployed in the cloud today and is a critical skillset to master in the cloud age.
After completing the DevOps training course you will achieve hands-on expertise in various aspects of the DevOps delivery model. The practical learning outcomes of this Devops training course are:
An understanding of DevOps and the modern DevOps toolsets
The ability to automate all aspects of a modern code delivery and deployment pipeline using:
1. Source code management tools
2. Build tools
3. Test automation tools
4. Containerization through Docker
5. Configuration management tools
6. Monitoring tools
Who should take this course?
DevOps career opportunities are thriving worldwide. DevOps was featured as one of the 11 best jobs in America for 2017, according to CBS News, and data from Payscale.com shows that DevOps Managers earn as much as $122,234 per year, with DevOps engineers making as much as $151,461.
Learn more at https://www.simplilearn.com/cloud-computing/devops-practitioner-certification-training
When most people talk about automating infrastructure, they focus on things like consistency, scalability, and flexibility. While fine goals, we recently converted several projects to Chef for both systems AND application deployment, and found that, with a little work, these tools could also help you enable better software quality assurance, load modeling, and even improve resource allocation.
By sharing cookbooks across projects, we were able to standardize practices and eliminate arbitrary differences, while using parameterization to perfectly isolate the special needs of each project. This allowed us to transfer knowledge among staff much more quickly. Pulling in and parameterizing application state – database contents, website assets, uploaded content – allowed us to spin up new environments with as much or as little state as needed. Integrating with Vagrant and Jenkins, we were then able to use chef to treat the entire image – system and application – as a test fixture. As each engineer (ops or dev) has visibility into the whole stack, we can more easily move people between dev and ops, or between projects.
This presentation was held at the DevOps Meetup Frankfurt on 2014/12/08. It describes some tools and practices for testing your infrastructure provisioned with Chef.
Community Cookbooks & further resources - Fundamentals Webinar Series Part 6Chef
Part 6 of a 6 part series introducing you to the fundamentals of Chef.
This session includes an introducing Community Cookbooks and some additional resources.
After viewing this webinar you will be able to:
- Find, preview, and download cookbooks from the Chef Community site
- Use knife to work with the Community Site API
- Download, extract, examine and implement cookbooks from the Community site
Video of this webinar can be found at the following URL
https://www.youtube.com/watch?v=ovTIeS3kx4g&list=PL11cZfNdwNyPnZA9D1MbVqldGuOWqbumZ
Overview of Chef - Fundamentals Webinar Series Part 1Chef
This is an Overview of Chef. After viewing this webinar you will be able to:
- Describe how Chef thinks about Infrastructure Automation
- Define the following terms:
- Resource
- Recipe
- Node
- Run List
- Search
- Login to Hosted Chef
- Run `knife` commands from your workstation
Video of this webinar can be found at the following URL
https://www.youtube.com/watch?v=S5lHUpzoCYo&list=PL11cZfNdwNyPnZA9D1MbVqldGuOWqbumZ
Chef vs Puppet vs Ansible vs Saltstack | Configuration Management Tools | Dev...Simplilearn
This presentation "Chef vs Puppet vs Ansible vs Saltstack" will compare the DevOps configuration management tools Chef, Puppet, Ansible and Saltstack in terms of their capabilities, architecture, performance, ease of setup, language, scalability and pros and cons. The chef is a configuration management tool written in Ruby and Erlang. Puppet is an open-source software configuration management tool that runs on many Unix-like systems and also Windows. Ansible is yet another tool that automates software provisioning, configuration management, and application deployment. Saltstack is a Python-based open-source configuration management tool. Now, let us get started and get to know which is the best configuration management platform among Chef, Puppet, Ansible and Saltstack.
Below are the contents of our "Chef vs Puppet vs Ansible vs Saltstack" configuration management tools comparison slides:
1) Need for Configuration Management Tools
2) Chef - Infrastructure, Architecture, Pros and Cons
3) Puppet- Infrastructure, Architecture, Pros and Cons
4) Ansible - Infrastructure, Architecture, Pros and Cons
5) Saltstack - Infrastructure, Architecture, Pros and Cons
6) Comparison on the basis of architecture, ease of setup, language, scalability, management and interoperability.
Why learn DevOps?
Simplilearn’s DevOps training course is designed to help you become a DevOps practitioner and apply the latest in DevOps methodology to automate your software development lifecycle right out of the class. You will master configuration management; continuous integration deployment, delivery and monitoring using DevOps tools such as Git, Docker, Jenkins, Puppet and Nagios in a practical, hands-on and interactive approach. The DevOps training course focuses heavily on the use of Docker containers, a technology that is revolutionizing the way apps are deployed in the cloud today and is a critical skillset to master in the cloud age.
After completing the DevOps training course you will achieve hands-on expertise in various aspects of the DevOps delivery model. The practical learning outcomes of this Devops training course are:
An understanding of DevOps and the modern DevOps toolsets
The ability to automate all aspects of a modern code delivery and deployment pipeline using:
1. Source code management tools
2. Build tools
3. Test automation tools
4. Containerization through Docker
5. Configuration management tools
6. Monitoring tools
Who should take this course?
DevOps career opportunities are thriving worldwide. DevOps was featured as one of the 11 best jobs in America for 2017, according to CBS News, and data from Payscale.com shows that DevOps Managers earn as much as $122,234 per year, with DevOps engineers making as much as $151,461.
Learn more at https://www.simplilearn.com/cloud-computing/devops-practitioner-certification-training
When most people talk about automating infrastructure, they focus on things like consistency, scalability, and flexibility. While fine goals, we recently converted several projects to Chef for both systems AND application deployment, and found that, with a little work, these tools could also help you enable better software quality assurance, load modeling, and even improve resource allocation.
By sharing cookbooks across projects, we were able to standardize practices and eliminate arbitrary differences, while using parameterization to perfectly isolate the special needs of each project. This allowed us to transfer knowledge among staff much more quickly. Pulling in and parameterizing application state – database contents, website assets, uploaded content – allowed us to spin up new environments with as much or as little state as needed. Integrating with Vagrant and Jenkins, we were then able to use chef to treat the entire image – system and application – as a test fixture. As each engineer (ops or dev) has visibility into the whole stack, we can more easily move people between dev and ops, or between projects.
Priming Your Teams For Microservice Deployment to the CloudMatt Callanan
You think of a great idea for a microservice and want to ship it to production as quickly as possible. Of course you'll need to create a Git repo with a codebase that reuses libraries you share with other services. And you'll want a build and a basic test suite. You'll want to deploy it to immutable servers using infrastructure as code that dev and ops can maintain. Centralised logging, monitoring, and HipChat notifications would also be great. Of course you'll want a load balancer and a CNAME that your other microservices can hit. You'd love to have blue-green deploys and the ability to deploy updates at any time through a Continuous Delivery pipeline. Phew! How long will it take to set all this up? A couple of days? A week? A month?
What if you could do all of this within 30 minutes? And with a click of a button soon be receiving production traffic?
Matt introduces "Primer", Expedia's microservice generation and deployment platform that enables rapid experimentation in the cloud, how it's caused unprecedented rates of learning, and explain tips and tricks on how to build one yourself with practical takeaways for everyone from the startup to the enterprise.
Video: https://www.youtube.com/watch?v=Xy4EkaXyEs4
Meetup: http://www.meetup.com/Devops-Brisbane/events/225050723/
This presentation walks through a Jenkins as Code approach that aims to fully automate and describe the creation of Infrastructure, Application and Configuration as Code.
We treat our applications with a strong 'as code' approach, but often forget about the critical operational tools. This presentation shows how it is possible to create a code first approach to creating and managing a Jenkins Service.
Working code repository is available at https://bitbucket.org/stevemac/dockerfiles
How EIS has solved some of the DevOps problems with Windows and design.
Worth noting the cookbook design in reality turns into a diagram like the following:
https://www.ebsco.com/files/blog/img/uploads/graph.png
JUC Europe 2015: From Virtual Machines to Containers: Achieving Continuous In...CloudBees
By Christian Lipphardt, Camunda Services
Camunda is an open source, Java-based framework process/business process automation. As a middleware technology, Camunda integrates with six different Java application servers (in different versions) and supports six different database products. The team at Camunda maintains five supported versions of Camunda itself, adding two versions every year. Maintaining the necessary continuous integration (CI) infrastructure based on virtual machines became increasingly problematic, with poor build reproducibility and limited scalability. Feedback cycles for developers were unacceptable. Recently Camunda switched from the virtual machine model to a container model based on Docker. The Camunda team now develops infrastructure as code and applies microservice-like separation of concerns. In the talk, Daniel will share the new CI architecture and present lessons learned.
An introduction the Job DSL plugin for the Jenkins continuous integration server. Learn how to treat job and view configuration as code, how to store the configuration in SCM and how to apply code reuse and refactoring. Learn how to extend the Job DSL for your favorite plugins.
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Tobias Schneck
As AI technology is pushing into IT I was wondering myself, as an “infrastructure container kubernetes guy”, how get this fancy AI technology get managed from an infrastructure operational view? Is it possible to apply our lovely cloud native principals as well? What benefit’s both technologies could bring to each other?
Let me take this questions and provide you a short journey through existing deployment models and use cases for AI software. On practical examples, we discuss what cloud/on-premise strategy we may need for applying it to our own infrastructure to get it to work from an enterprise perspective. I want to give an overview about infrastructure requirements and technologies, what could be beneficial or limiting your AI use cases in an enterprise environment. An interactive Demo will give you some insides, what approaches I got already working for real.
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
UiPath Test Automation using UiPath Test Suite series, part 4DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap.
The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies.
Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques
What will you get from this session?
1. Insights into SAP testing best practices
2. Heatmap utilization for testing
3. Optimization of testing processes
4. Demo
Topics covered:
Execution from the test manager
Orchestrator execution result
Defect reporting
SAP heatmap example with demo
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Jeffrey Haguewood
Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows.
We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases.
This video focuses on the notifications, alerts, and approval requests using Slack for Bonterra Impact Management. The solutions covered in this webinar can also be deployed for Microsoft Teams.
Interested in deploying notification automations for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.
PHP Frameworks: I want to break free (IPC Berlin 2024)Ralf Eggert
In this presentation, we examine the challenges and limitations of relying too heavily on PHP frameworks in web development. We discuss the history of PHP and its frameworks to understand how this dependence has evolved. The focus will be on providing concrete tips and strategies to reduce reliance on these frameworks, based on real-world examples and practical considerations. The goal is to equip developers with the skills and knowledge to create more flexible and future-proof web applications. We'll explore the importance of maintaining autonomy in a rapidly changing tech landscape and how to make informed decisions in PHP development.
This talk is aimed at encouraging a more independent approach to using PHP frameworks, moving towards a more flexible and future-proof approach to PHP development.
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
JMeter webinar - integration with InfluxDB and GrafanaRTTS
Watch this recorded webinar about real-time monitoring of application performance. See how to integrate Apache JMeter, the open-source leader in performance testing, with InfluxDB, the open-source time-series database, and Grafana, the open-source analytics and visualization application.
In this webinar, we will review the benefits of leveraging InfluxDB and Grafana when executing load tests and demonstrate how these tools are used to visualize performance metrics.
Length: 30 minutes
Session Overview
-------------------------------------------
During this webinar, we will cover the following topics while demonstrating the integrations of JMeter, InfluxDB and Grafana:
- What out-of-the-box solutions are available for real-time monitoring JMeter tests?
- What are the benefits of integrating InfluxDB and Grafana into the load testing stack?
- Which features are provided by Grafana?
- Demonstration of InfluxDB and Grafana using a practice web application
To view the webinar recording, go to:
https://www.rttsweb.com/jmeter-integration-webinar
Let's dive deeper into the world of ODC! Ricardo Alves (OutSystems) will join us to tell all about the new Data Fabric. After that, Sezen de Bruijn (OutSystems) will get into the details on how to best design a sturdy architecture within ODC.
UiPath Test Automation using UiPath Test Suite series, part 3DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 3. In this session, we will cover desktop automation along with UI automation.
Topics covered:
UI automation Introduction,
UI automation Sample
Desktop automation flow
Pradeep Chinnala, Senior Consultant Automation Developer @WonderBotz and UiPath MVP
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Search and Society: Reimagining Information Access for Radical FuturesBhaskar Mitra
The field of Information retrieval (IR) is currently undergoing a transformative shift, at least partly due to the emerging applications of generative AI to information access. In this talk, we will deliberate on the sociotechnical implications of generative AI for information access. We will argue that there is both a critical necessity and an exciting opportunity for the IR community to re-center our research agendas on societal needs while dismantling the artificial separation between the work on fairness, accountability, transparency, and ethics in IR and the rest of IR research. Instead of adopting a reactionary strategy of trying to mitigate potential social harms from emerging technologies, the community should aim to proactively set the research agenda for the kinds of systems we should build inspired by diverse explicitly stated sociotechnical imaginaries. The sociotechnical imaginaries that underpin the design and development of information access technologies needs to be explicitly articulated, and we need to develop theories of change in context of these diverse perspectives. Our guiding future imaginaries must be informed by other academic fields, such as democratic theory and critical theory, and should be co-developed with social science scholars, legal scholars, civil rights and social justice activists, and artists, among others.
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
Neuro-symbolic is not enough, we need neuro-*semantic*Frank van Harmelen
Neuro-symbolic (NeSy) AI is on the rise. However, simply machine learning on just any symbolic structure is not sufficient to really harvest the gains of NeSy. These will only be gained when the symbolic structures have an actual semantics. I give an operational definition of semantics as “predictable inference”.
All of this illustrated with link prediction over knowledge graphs, but the argument is general.
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...Ramesh Iyer
In today's fast-changing business world, Companies that adapt and embrace new ideas often need help to keep up with the competition. However, fostering a culture of innovation takes much work. It takes vision, leadership and willingness to take risks in the right proportion. Sachin Dev Duggal, co-founder of Builder.ai, has perfected the art of this balance, creating a company culture where creativity and growth are nurtured at each stage.
Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...
Jenkins and Chef: Infrastructure CI and Automated Deployment
1. Jenkins User Conference Boston #jenkinsconf
Jenkins and Chef
Infrastructure CI and Application Deployment
Dan Stine
Copyright Clearance Center
www.copyright.com
June 18, 2014
#jenkinsconf
2. Jenkins User Conference Boston #jenkinsconf
About Me
Software Architect
Library & Framework Developer
Infrastructure Lead & Product Owner
Enemy of Inefficiency and Needless
Inconsistency
dstine at copyright.com
sw at stinemail.com
github.com/dstine
4. Jenkins User Conference Boston #jenkinsconf
Agenda
• Context
• Primer
• Deployment Process Design
• Cookbook Builds
• Application Deployment
• Wrap Up
6. Jenkins User Conference Boston #jenkinsconf
Standard Software Platform
• Started platform definition in 2011
– Homogeneous by default
• Tools
– Java, Spring, Tomcat, Postgres
– Git/GitHub, Gradle, Jenkins, Artifactory, Liquibase
• Process
– Standard development workflow
– Standard application shape & operational profile
8. Jenkins User Conference Boston #jenkinsconf
Initial Delivery Pipeline
• Automated build process
• Publish build artifacts to Artifactory
– Application WARs
– Liquibase JARs
• Manual deploys
– (Many apps) x (many versions) x (multiple
environments) = TIME & EFFORT
– The more frequently a task is performed, the
greater the return from improved efficiency
9. Jenkins User Conference Boston #jenkinsconf
Improved Deployment Process
• Goals
– Reduce effort
– Improve speed, reliability, and frequency
– Handle app deploys and db schema updates
– Enable self-service
• Process Changes
– Manual Automated
– Prose instructions Infrastructure as code
12. Jenkins User Conference Boston #jenkinsconf
Layers of System Management
• Orchestration
– Processes collaborating in a distributed
system
• Configuration
– Install and configure packages and software
• Provisioning
– Hypervisor (VMware, EC2)
13. Jenkins User Conference Boston #jenkinsconf
Infrastructure as Code
• Develop and manage software
infrastructure with practices similar to
those used to develop software applications
• Examples
– Source Code
– Modularity
– Abstraction
– Testing
14. Jenkins User Conference Boston #jenkinsconf
Configuration Management
“Process for establishing and maintaining consistency
of a product’s performance, functional and physical
attributes with its requirements, design and operational
information throughout its life” (wikipedia)
• CM tools for managing software systems
– CFEngine, Puppet, Chef, Salt, Ansible
• Embody Infrastructure as Code principles
• Define desired state of machine
– Each run inspects state and makes necessary
changes, if any
15. Jenkins User Conference Boston #jenkinsconf
Chef
• Configuration management tool
• Exposes DSL hosted in Ruby
– Express “what” not “how”
– Clean, purposeful capture of intent
• Favor DSL when writing code
– Ruby is available, if required
16. Jenkins User Conference Boston #jenkinsconf
Chef Terminology (1)
• Chef client is installed on nodes
(machines) which are registered with
the Chef server
• Developers write code on workstations and
use tools such as knife to interact with server
• Chef models node configuration as a set of
DSL resources (e.g. package, service,
directory) which are mapped to internal
providers (actual code to execute)
– Can define custom resources
17. Jenkins User Conference Boston #jenkinsconf
Example Chef Code
This resource declaration
directory '/a/b/c' do
owner 'admin'
group 'admin
mode '0755'
action :create
recursive true
end
ensures that
$ ls -ld /a/b/c
drwxr-xr-x. 5 admin admin 4096 Feb 14 11:22 /a/b/c
18. Jenkins User Conference Boston #jenkinsconf
Chef Terminology (2)
• A recipe declares a set of resources with
desired configuration
• A cookbook contains a set of semantically-
related code and is the fundamental unit of
distribution for Chef code
– Compare to JAR for Java code
• A data bag holds JSON information in one or
more data bag items accessible from Chef
code
• Chef environments model deployed
environments
• Each node has a run list containing recipes
20. Jenkins User Conference Boston #jenkinsconf
Basic Approach
• Deploy custom applications with Chef
• Execute schema updates with Liquibase
• Coordinate everything with:
21. Jenkins User Conference Boston #jenkinsconf
Jenkins as Coordinator
• General purpose job executor
– Shell script steps for Chef API
– Gradle steps for Liquibase updates
– Arbitrary code at any point in lifecycle
• UI
– Smooth integration with Active Directory
– Authentication and authorization
• Administration
– Familiar with Jenkins from application build jobs
22. Jenkins User Conference Boston #jenkinsconf
CCC Application Group
• Set of deployable units that are
versioned and released together
• For example, a group might have
– UI
– REST service
– Message consumer
– DB
• Build with a single command
• Deploy with a single command
23. Jenkins User Conference Boston #jenkinsconf
Technical Design Goals
• Provide clean API
– Specify only essential differences between
apps
– Custom Chef resource is the interface
– Codify & enforce standards
• Balance consistency & flexibility
– Code in semantically-versioned cookbooks
– Configuration in data bags
• Controlled cookbook promotion
– Chef environment specifies cookbook version
constraint
24. Jenkins User Conference Boston #jenkinsconf
Cookbook Types
• Library Cookbooks
– Encapsulate common re-usable logic
– Define custom resource to install an app
• And the implementing provider
• Application Cookbooks
– Depend on library cookbooks
– One cookbook per application group
• One recipe per application
• Recipes use custom resource
– Lightweight
25. Jenkins User Conference Boston #jenkinsconf
Data Bags
• Contain application configuration
– Service endpoints, JAVA_OPTS, etc.
• One data bag per application group
– One data bag item per environment
• “Live” reflection of deployed configuration
– Edit push to Chef server deploy
– Master always matches state of Chef server
26. Jenkins User Conference Boston #jenkinsconf
Custom Resource Usage
Ensure my-app-ui WAR is deployed:
ccc_webapp "my-app-ui" do
provider :ccc_webapp
artifact_group 'com.copyright.myapp'
artifact 'my-app-ui'
container 'MY-APP-UI'
http_port '8080'
shutdown_port '8900'
properties_template 'ccc.properties.erb'
app_group_data_bag 'my_app'
end
27. Jenkins User Conference Boston #jenkinsconf
Custom Resource Actions
• Retrieves Java, Tomcat & WAR from
Artifactory
• Installs Java and Tomcat in standard
locations
• Creates and configures Tomcat container
• Installs WAR in the container
• Opens port in host firewall
• Generates application properties file
• Starts container
(Each action taken only if necessary)
28. Jenkins User Conference Boston #jenkinsconf
Data Bag Structure
data_bags/my_app/DEV.json (data bag item)
"version": "1.4.9",
"runtime": {
"my-app-ui": {
"java_opts": "-Xmx2G -XX:MaxPermSize=1024m"
}
},
"app_config": {
"db.url": "jdbc:postgresql://devdb:5432/myapp",
"svc.foo.url": "http://devsvc:9000/foo"
}
data_bags/my_app/TEST.json
...
data_bags/my_app/PROD.json
29. Jenkins User Conference Boston #jenkinsconf
Cookbook Data Bag Code *
ccc/providers/webapp.rb (library cookbook)
app_group_data = data_bag_item(app_group_data_bag,
node.chef_environment)
java_opts = app_group_data['runtime'][artifact]['java_opts']
// pass java_opts to Tomcat container
app_config = app_group_data['app_config']
// pass app_config to template resource declaration
my_app/templates/ccc.properties.erb (application cookbook)
db.url=<%= @app_config['db.url'] %>
svc.foo.url=<%= @app_config['svc.foo.url'] %>
* Included for future reference
30. Jenkins User Conference Boston #jenkinsconf
Roles
• Deployers
– Update data bags & environment files
– Initiate deployments
• Tech leads
– Maintain application cookbooks
• Framework developers
– Maintain library cookbooks
– Maintain framework
– Process improvement
33. Jenkins User Conference Boston #jenkinsconf
Jenkins Build Server
• For each application group
– Cookbook CI job
– Cookbook release job
– Same master as application build jobs
• New class of slaves
– Ruby with required gems
– Chef with credentials for Chef server
– EC2 credentials to create test nodes
34. Jenkins User Conference Boston #jenkinsconf
Cookbook CI Job
• Triggered when new Chef code is
merged
• Static analysis
– JSON syntax (json gem)
– Ruby syntax and style (Tailor)
– Chef syntax (Knife)
– Chef style and correctness (Foodcritic)
• Integration testing
– Test Kitchen with kitchen-ec2 plugin
35. Jenkins User Conference Boston #jenkinsconf
Integration Testing Lifecycle
• Spin up EC2 instance(s) to mimic
actual deployment topology of
application group
• Run Chef on each instance (node)
• Execute asserts – pass or fail
• Dispose of instance(s)
36. Jenkins User Conference Boston #jenkinsconf
Integration Testing Details
• Instances created from AMI
– Preconfigured with Ruby and Chef
• Using Chef Solo
– Avoid adding ephemeral nodes to Chef server
• Faux Chef environment “CHEFDEV”
– JSON for real environments is reserved
• Tag EC2 instances for traceability
• Troubleshoot by running Test Kitchen from
workstation
37. Jenkins User Conference Boston #jenkinsconf
Cookbook Release Job
• Triggered manually
• Runs same tests as CI job
• Uploads new cookbook version to Chef
server
• Tags Git repo
40. Jenkins User Conference Boston #jenkinsconf
Jenkins Deploy Server
• Separate master for deploys
• Slaves
– Ruby with required gems
– Chef with credentials for Chef server
– SSH keys for nodes
41. Jenkins User Conference Boston #jenkinsconf
Deploy Job Types
• Each app group has two deploy jobs
– DEV deploy for Development
– Non-DEV deploy for Operations
– Will provide more flavors over time
• Job parameters
– Environment (non-DEV jobs only)
– Application group version
42. Jenkins User Conference Boston #jenkinsconf
What Does a Deployer Do?
• Makes configuration changes
– Edits application data bag item
– Edits environment file (if necessary)
– Merges code
• Executes job in Jenkins
43. Jenkins User Conference Boston #jenkinsconf
Example Deploy Job Run
• Deployer enters parameters
– Application version = 1.4.9
– Environment = TEST
• Then automation takes over
– Confirms my_app data bag has TEST version 1.4.9
– Uploads TEST environment file and my_app data
bag item for TEST to Chef server
– Finds all nodes in TEST environment with run list
containing my_app recipes
– Runs Chef client on each found node
– Sends email notification
47. Jenkins User Conference Boston #jenkinsconf
Most Important Advice
• Beware of overly prescriptive “lessons
learned” and “best practices”
• Synthesize a solution for your context
• That said…
48. Jenkins User Conference Boston #jenkinsconf
Principles & Guidelines (1)
• Standardize wherever possible
– Technology, design, process
– Achieve economies of scale
– Exceptions are permissible but rare
• Every tool must have an API
– Avoid “hitting the wall” down the road
– Tradeoff some out-of-the-box capabilities
49. Jenkins User Conference Boston #jenkinsconf
Principles & Guidelines (2)
• Use multiple communication paths
– All-hands presentations
– Kickoff meetings with each team
– Developer walkthroughs
– Documentation
• Be opportunistic
– Find and nurture your early adopters
50. Jenkins User Conference Boston #jenkinsconf
Principles & Guidelines (3)
• Balance process and progress
– Must provide tangible results
– And also build foundation for future
– Just like with application development!
• Start with a big pain point
– Providing relief builds credibility going forward
– Hopefully recoups bandwidth to reinvest
51. Jenkins User Conference Boston #jenkinsconf
“When Will You Be Done?”
• DONE is a dangerous word
– Business won’t stop evolving
– Neither will the supporting applications
– Nor should the supporting infrastructure
• X is a journey, not a destination
– For many values of X
• Deployment automation
• Continuous delivery
• DevOps
52. Jenkins User Conference Boston #jenkinsconf
Thank You
• Copyright Clearance Center
Engineering Team
• Jenkins User Conference
Organizers and Sponsors
53. Jenkins User Conference Boston #jenkinsconf
Resources
• “Infrastructure in the Cloud Era”
Adam Jacob and Ezra Zygmuntowicz
http://www.slideshare.net/adamhjk/infrastructure-in-the-cloud-era
http://www.youtube.com/watch?v=HaABapTwQ2c
• Chef cookbook versioning policy
http://chef-community.github.io/cvp/
Exceptions are possible but rare
Just as sampling – many other technologies and standards and libraries and tools (project management) and processes
This presentation is a point-in-time-report
Java tech lead first comment: now the “UI guy” can do deploys
Everyone at this conference is familiar with Jenkins
May be less familiar with Chef, and we’ll be talking about it a lot, so here is a whirlwind introduction
No mkdir command & flags
There is a lot more to Chef – we covered the very basics
More and more people will / must learn Jenkins and Chef at some level
Compare to Java tools such as Checkstyle and FindBugs