Help students get familiar with the basic concepts of DevOps processes and technologies and the challenges facing companies who are looking to embrace scalable software deployment.
[This workshop was given to TAU CS students over the years 2015-2016]
2. Objective
2
Help students get familiar with the basic concepts
of DevOps processes and technologies and the
challenges facing companies who are looking to
embrace scalable software deployment.
3. Syllabus
3
Intro to DevOps
Intro Cloud Computing
Intro to Linux Basics
Intro to Version Control System (VCS)
o GIT
o GitHub
Intro to Web Development
o HTML
o JavaScript
o NodeJs
Demo App
Intro to Continuous Integration
o Jenkins
o CircelCI
Intro to Continuous Deployment
o Docker
o DockerHub
o AWS EC2
Continuous Delivery Demo
12. The purpose of DevOps
12
Create Value !
… is the purpose of your organization !
13. Creating Value
Dev Product Customer
$$$
Hey,
Show me the value !
We have lots
Of great ideas !
Get them to the
Customer fast !
QA Ops
How can
we help ?
14. Not the purpose of DevOps
Dev QA Ops Eng. Customer
Don’t know
if want…
15. Really Not the purpose of DevOps
Dev QA
Ops Eng.Product
Unrealistic project deadlines ! Sloppy coding practices !
Shetty developers ! I told ya’ !
Unhappy
Slow
Ineffective at delivery
Can’t learn from failures
Hard to make changes
Operational problems stack up
16. Impediments to release velocity
47%
46%
45%
40%
35%
31%
40%
47%
47%
50%
53%
53%
11%
6%
5%
6%
11%
14%
1%
1%
3%
4%
1%
1%
Provisioning of adequate development/test environments is problematic
Applications are monolithic/tightly coupled
Test processes are manual and effort laden
Business sponsors can't make themselves available more frequently
Deployment to production is manual/labor intensive
We lack application or technical expertise and fear unintended impact of change
"How do the following factors affect your ability to deliver working code to production faster?"
Large impact Small Impact No Impact Don't Know
Base: 255 North American and European IT decision-makers involved in application development and delivery at 12+ month intervals
Source: A commissioned study conducted by Forrester Consulting on behalf of HP, October 2013
17. The purpose of DevOps
IdeaFeature,
Bug Fix,
Enhancement
Deploy fast a/w quality
Get feedback
Customer
(production)
18. DevOps changes Org DNA
Dev+QA
(Engineering)
Dev+Ops
(DevOps)
Dev QA
Ops Eng. Ops Eng.
19. What is DevOps ?
19
A methodology, involving a spectrum of cultural
approaches, processes and technologies,
designed to help the business reduce it’s TTM
while also maintaining high quality releases
21. Processes
“A software engineering approach that help teams produce software in short
cycles, ensuring that the software can be reliably released at any time. It aims at
building, testing, and releasing software faster and more frequently. The approach
helps reduce the cost, time, and risk of delivering changes by allowing for
more incremental updates to applications in production”
Continuous Delivery
25. What is a DevOps engineer, anyway?
DevOps engineers are a pretty elite group, “Special Forces” in an organization, sharing the
following attributes:
Ability to use a wide variety of open source technologies and tools
Ability to code and script
Experience with systems (OS) and IT operations
Strong grasp of automation tools
A strong focus on business outcomes
Comfort with collaboration, open communication and reaching across functional
borders
26. Is DevOps Eng. wanted ?
https://puppet.com/resources/white-paper/2015-devops-salary-report/thank-you
30. Starting Point – Physical Machine
– Physical Hardware
– Processors, memory, chipset, I/O devices, etc.
– Resources often grossly underutilized
– Software
– Tightly coupled to physical hardware
– Single active OS instance
– OS controls hardware
31. Virtual Machine
– Software Abstraction
– Behaves like hardware
– Encapsulates all OS and application state
– Virtualization Layer
– Extra level of indirection
– Decouples hardware, OS
– Enforces isolation
– Multiplexes physical hardware across VMs
32. What is Cloud Computing ?
An analogy: Think of electricity service…
33. What is Cloud Computing ?
Cloud Commuting is also a utility service – giving you access to technology resources
managed by experts and available on-demand.
You simply access theses services over the internet, with no up-front costs and you pay only
for the resource you use.
34. Cloud Computing Stack
SaaS applications are designed for end-
users, delivered over the web.
PaaS is the set of tools and services
designed to make coding and deploying
those applications quick and efficient.
IaaS is the hardware and software that
powers it all – servers, storage, networks,
operating systems.
39. What is Linux ?
A free, open-source operating system.
Has been under active development since 1991.
It has evolved to be versatile and is used all over the world, from web servers to
cellphones.
41. Linux Distributions
Linux isn’t a complete OS — it’s just a kernel. Linux distributions take the Linux kernel and
combine it with other free software to create complete packages.
42. The Terminal Shell
The shell allows you to execute commands.
All administrative tasks can be accomplished through the terminal. This includes file
manipulation, package installation, and user management.
The terminal is interactive. You specify commands to run. The terminal outputs the results
of those commands. Executing any command is done by typing it and pressing Enter.
43. Basic Commands
pwd
ls
cd <folder>
cd ..
mkdir <folder>
rm <filefolder>
cat <file>
touch <file>
mv <file1> <file2>
cp <file1> <file2>
man <cmd>
45. The Problem
Lots of code (millions of lines)
and lots of developers spread
over the globe
46. Dealing With Change
Copying (Coursework_working.java, Coursework_tmp.java)
Copy & Paste code snippets
Copy entire directories
Emailing code to people
Using DropBox, Etc.
47. What is Version Control System ?
Category of software tools that help a software team manage changes to source code over
time.
Keeps track of every modification to the code in a special kind of database.
Essential part of the every-day of the modern software team‘s professional practices.
48. Benefits of Version Control Systems
History. A complete long-term change history of every file. This means every change made
by many individuals over the years.
Branching and merging. Having team members work concurrently is a no-brainer, but even
individuals working on their own can benefit from the ability to work on independent streams
of changes.
Traceability. Being able to trace each change made to the software and connect it to project
management and bug tracking software.
55. 55
What is GitHub
A PaaS based code hosting platform for version control and collaboration. It lets you and others
work together on projects from anywhere.
https://github.com/
59. First GIT Branch
Open the command line -
> git branch
> git checkout –b new_feature
> git branch
> echo "this is a test" > test
> git add test
> git commit –m “add new feature branch”
> git push origin new_feature
60. Using GutHub to host your website for FREE (!)
Open the command line -
> git branch
> git checkout –b gh-pages
From windows:
Go to C:cygwin64homelevyli
Create New file index.html
Add the string “<h1>Welcome to Liran’s website</h1>“
> git add index.html
> git commit –m “add index.html”
> git push origin gh-pages
https://liranlevy3.github.io/liran/
http://username.github.io/repository/
62. What is Web Development ?
The creation of dynamic web applications such as Facebook or e-commerce sites like Amazon.
Many argue it’s the best form of coding for beginners to learn. It’s easy to set up, you get
instant results and there’s plenty of online training available.
Learning web development It’s one of the smartest decisions you will ever make (!)
64. Web Development Overview
Front-End Development (a.k.a client-side development) – What a user sees when they
load a web application – things that runs in the browser.
HTML
CSS
JavaScript
Back-End Development (a.k.a server-side development) – What goes on behind the
scenes of a web application – things that runs in the server.
PHP
Ruby
Python
Java
NodeJs (JavaScript)
ASP.Net
Perl
65. How Front-End and Back-End interacts ?
Server-Rendered Apps
The browser sends a HTTP request and the server replies with a HTML page.
Between receiving the request and responding, the server usually queries the database and
feeds it into a template.
The browser then loads the HTML and defines what things are, CSS how they look and JS any
special interactions.
AJAX (Asynchronous JavaScript and XML)
AJAX is about updating parts of a web page, without reloading the whole page.
The browser sends a HTTP request from within the page and get’s back a JSON response.
This means that your server needs to have an endpoint which replies JSON to requests.
Two architectures for this are REST and SOAP.
67. What is HTML?
HTML is a markup language for describing web documents (web pages).
HTML documents are described by HTML tags.
Each HTML tag describes different document content.
68. HTML Tags
HTML tags are keywords (tag names) surrounded by angle brackets:
<tagname>
</tagname>
HTML tags normally come in pairs like <p> and </p>.
http://www.w3schools.com/tags/
70. HTML Elements
An HTML element usually consists of a start tag and end tag, with the content inserted in between
<tagname>content goes here...</tagname>
HTML elements can be nested (elements can contain elements).
All HTML elements can have attributes.
Attributes provide additional information about an element.
Attributes are always specified in the start tag.
Attributes usually come in name/value pairs like: name="value“.
HTML Attributes
71. Some Common Elements
HTML headings are defined with the <h1> to <h6> tags.
<h1>This is heading 1</h1>
HTML paragraphs are defined with the <p> tag.
<p>This is a paragraph</p>
HTML links are defined with the <a> tag.
<a href="http://www.hpe.com">This is a link</a>
HTML images are defined with the <img> tag.
<img src=“hpe.jpg" width="104" height="142">
72. Web Browsers
The purpose of a web browser is to read HTML documents and display them.
The browser does not display the HTML tags, but uses them to determine how to display the document.
73. First HTML Page
From windows:
Go to C:cygwin64homelevyli
Copy the file pic_bulboff.gif
Create new file first_html_page.html
<!DOCTYPE html>
<html>
<body>
<h1>My First HTML page</h1>
<p>Here is a lightbulb ! </p>
<img id="myImage" src="pic_bulboff.gif" style="width:100px">
</body>
</html>
75. Intro to Web Development
75
Front-End - JavaScript
76. What is JavaScript ?
JavaScript is the programming language of HTML and the Web.
JavaScript is easy to learn.
JavaScript Can Change HTML document.
78. JavaScript and HTML
JavaScript can be placed in the <body> and the <head> sections of an HTML page.
JavaScript code must be inserted between <script> and </script> tags.
http://www.w3schools.com/js/tryit.asp?filename=tryjs_whereto_head
79. Syntax
Statements are separated by ;
Strings are text, written within double (“) or single quotes (‘)
JavaScript uses the var keyword to declare variables
Code after double slashes // or between /* and */ is treated as a comment
All JavaScript identifiers are case sensitive
var x=1; // x is an identifier
var y=2; // y is an identifier
document.getElementById("demo").innerHTML = x + y ;
80. Keywords (Examples)
Keyword Description
break Terminates a switch or a loop
continue Jumps out of a loop and starts at the top
do ... while Executes a block of statements, and repeats the block, while a condition is true
for Marks a block of statements to be executed, as long as a condition is true
function Declares a function
if ... else Marks a block of statements to be executed, depending on a condition
return Exits a function
switch Marks a block of statements to be executed, depending on different cases
try ... catch Implements error handling to a block of statements
var Declares a variable
81. Data Types
Undefined, Number, Boolean, Array, Object
JavaScript has only one type of numbers.
Object properties are written as name:value pairs, separated by commas.
In JavaScript, the data type of null is an object.
var x; // Undefined
var size= 3; // Number
var x = true; // Boolean
var name = "Johnson"; // String
var cars = [“Honda", “Mazda", “Kia"]; // Array
var x = {firstName:“Liran", lastName:“Levy"}; // Object
var x = null; // Empty object
82. Objects
In JavaScript, all values, except primitive values, are objects.
Primitive are: strings, numbers, true, false, null and undefined.
Objects are variables that can contain many values.
The values are written as name:value pairs.
83. Objects Declreation
var person = {firstName:"John", lastName:"Doe", age:50, eyeColor:“green"};
var person = new Object();
person.firstName = "John";
person.lastName = "Doe";
person.age = 50;
person.eyeColor = “green";
function person(first, last, age, eye) {
this.firstName = first;
this.lastName = last;
this.age = age;
this.eyeColor = eye;
}
var myFather = new person("John", "Doe", 50, “green");
84. Objects Properties
Properties are the values associated with an object.
The syntax for accessing the property of an object is:
objectName.property // person.firstName
objectName["property"] // person["firstName"]
objectName[expression] // x = "firstName"; person[x]
85. Functions
A JavaScript function is a block of code designed to perform a particular task.
function myFunction(a, b) {
return a * b;
}
var myFunction = new Function("a", "b", "return a * b");
var x = myFunction(4, 3);
var x = function (a, b) {return a * b};
var y = x(4, 3);
86. HTML Events
An HTML event can be something the browser does, or something a user does.
JavaScript lets you execute code when events are detected.
HTML allows event handler attributes, with JavaScript code, to be added to HTML elements.
http://www.w3schools.com/js/tryit.asp?filename=tryjs_event_onclick1
87. HTML Events (Examples)
Event Description
onclick The user clicks an HTML element
onmouseover The user moves the mouse over an HTML element
onmouseout The user moves the mouse away from an HTML element
onkeydown The user pushes a keyboard key
88. JavaScript HTML DOM (Document Object Model)
When a web page is loaded, the browser creates a Document Object Model of the page.
90. First JavaScript Code
<!DOCTYPE html>
<html>
<body>
<h1>My First JS code</h1>
<p>Here is a lightbulb ! </p>
<button ???>Turn on the light</button>
<img id="myImage" src="pic_bulboff.gif" style="width:100px">
<button ???>Turn off the light</button>
</body>
</html>
From windows:
Go to C:cygwin64homelevyli
Copy the file pic_bulbon.gif
Create new file first_js_code.html
93. What is NodeJS?
A very powerful JavaScript-based framework built on Google Chrome's JavaScript V8 Engine.
Used to develop web applications like video streaming sites, single-page applications, and
other web applications.
Open source, completely free, and used by thousands of developers around the world.
Provides rich library of various JavaScript modules which simplifies the development of web
applications using Node.js to a great extent.
94. NodeJs is Asynchronous
Blocking Code Non-Blocking Code
var fs = require("fs");
var data = fs.readFileSync('input.txt');
console.log(data.toString());
console.log("Program Ended");
var fs = require("fs");
fs.readFile('input.txt', function (err, data) {
if (err) return console.error(err);
console.log(data.toString());
});
console.log("Program Ended");
A callback function is called at the completion of a given task.
Node makes heavy use of callbacks. All the APIs of Node are written in such a way that
they support callbacks.
95. NodeJs First Application
From windows:
Go to C:cygwin64homelevyli
Copy the file main.js
> node main.js
http://127.0.0.1:8080/
96. Node Package Manager
NPM makes it easy for JavaScript developers to share and reuse JS modules (packages).
A typical application, such as a website, will depend on dozens or hundreds of packages.
Developers can find packages to help build applications by browsing the NPM website.
All NPM packages are open source and can be downloaded for free.
https://www.npmjs.com/
97. Package.json
The best way to manage locally installed npm packages.
It serves as documentation for what packages your project depends on.
It allows you to specify the versions of a package that your project can use.
Makes your build reproducible which means that its way easier to share with other
developers.
Should be located on the project root folder.
105. Why Continuous Integration ?
105
Pain
Time
Observation #1
As longer as developers going
between merging their code
base, the more painful that
merge is going to be (known as
“Integration Hell”).
Integrate Faster
106. Why Continuous Integration ?
106
Quality
Time
Observation #2
𝑰𝒏𝒕𝒆𝒈𝒓𝒂𝒕𝒊𝒐𝒏 ≠ 𝑴𝒆𝒓𝒈𝒆
𝑰𝒏𝒕𝒆𝒈𝒓𝒂𝒕𝒊𝒐𝒏 = 𝑴𝒆𝒓𝒈𝒆 + 𝑸𝒖𝒂𝒍𝒊𝒕𝒚
Integrate Faster aw with Quality
107. What is it ?
107
Continuous Integration (CI) is a development practice that
requires developers to integrate code into a shared repository
several times a day. Each check-in is then verified by an
automated build and tests, allowing teams to detect problems
early. By integrating regularly, you can detect errors quickly,
and locate them more easily.
111. Essential Practices
111
Maintain a Single Source Repository
Automate the build
Make Your Build Self-Testing
Everyone Commits To the Mainline (Master)
Every Day
Every Commit Should Build the
Mainline (Master) on an Integration Machine
Fix Broken Builds Immediately
Keep the Build Fast
Test in a Clone of the Production Environment
118. 118
What is Jenkins?
Jenkins is an application and a framework that manages and monitor the execution of
repeated tasks.
Open source and under development since 2005
It’s primarily used as a Continuous Integration tool for building software
Supported by over hundreds plugins
http://jenkins-ci.org/
123. 123
What is CircelCI ?
A PaaS based Continuous Integration and Delivery platform.
Out of the box support for NodeJs, Java, Python, PHP, Scala, Go and more.
Out of the box integration with many testing's and deployment tools.
Fermium
https://circleci.com/
124. 124
CircelCI Flow
Infrastructure: Allocates the VM that will run the flow
Checkout: checking out and cloning your GIT repo
Machine: adjusting the VM to your preferences and requirements
Dependencies: setting up your project’s language-specific dependencies
Database: preparing the databases for your tests
Test: running your tests
Deployment: deploying your code to your web servers
TearDown: Collect metadata, disable VM
126. Webhooks
Webhooks allow external services to be notified when certain events happen within your
repository. When the specified events happen, GitHub sends a POST (HTTP) request to the
URLs provided.
130. What is it ?
130
Continuous deployment can be thought of as an extension of
Continuous Integration, aiming at minimizing lead time, the
time elapsed between development writing one new line of
code and this new code being used by live users, in
production.
134. Essential Practices
134
Compile only once (CI phase)
Deploy the same way to every environment
Smoke-Test your deployment
Deploy into a copy of production (Staging) first
136. CLT≈145m-325m
Deployment Frequency = 0m^-180m
^ - On demand self-service
Sanity
CI
• 20m
Full CI • 120m
CD • 5m
Idle time ≤ 180m
Idle time ≤ 0m
* CLT = The time from committing a change to deployment
Change Lead Time (CLT)
138. What are Linux Containers ?
Contain applications in a way that keep them isolated from the host system that they run on, by taking
advantage of Linux Kernel namespace and cgroup features.
Allow a developer to package up an application with all of the parts it needs, such as libraries and other
dependencies, and ship it all out as one package.
Designed to make it easier to provide a consistent experience as developers move code from
development environments into production in a fast and replicable way.
139. What is Docker ?
Docker is a command-line tool for programmatically defining the contents of a Linux container in code,
which can then be versioned, reproduced, shared, and modified easily just as if it were the source code
to a program.
Docker container can run anywhere (almost?), they’re lightweight and therefore can be shipped and
run, unchanged, on laptops, data center VMs and any cloud servers.
Docker is an open-source.
140. Virtual Machine Vs. Linux Containers
Virtual Machine contains not only the application and all needed bins and
libs, it also contains the entire guest operating system.
Docker container comprises just the application and its dependencies.
It runs as an isolated process on the host operating system, sharing the
kernel with other containers.
Thus, it enjoys the resource isolation and allocation benefits of VMs but is
much more portable and efficient.
143. 143
Back-End Front-EndSearch Queue DB
Development
VM
QA server Public Cloud Contributor’s
laptop
Production
Cluster
Customer Data
Center
…that can be manipulated
using standard operations and
run consistently on virtually any
hardware platform
An engine that enables any
payload to be encapsulated
as a lightweight, portable,
Self sufficient container…
And in our case
146. 146
Dockerfile
Docker can build images automatically by reading the instructions from a Dockerfile.
Dockerfile is a text document that contains all the commands a user could call on the command line to
assemble an image.
Dockerfile format -
Docker runs instructions in a Dockerfile in order.
The first instruction must be `FROM` in order to
specify the Base Image from which you are building.
The Docker build command takes a Dockerfile as input and output
A Docker image.
148. 148
Dockerfile Instructions (Examples)
FROM - Sets the base image for subsequent instructions. A valid Dockerfile must start with From.
RUN - Execute any commands in a new layer on top of the current image and commit the results. The
resulting committed image will be used for the next step in the dockerfile.
CMD – Provide default for executing container. There can only be one CMD instruction in a Docker file.
EXPOSE - Informs Docker that the container listens on the specified network ports at runtime.
ENV - Sets the environment variable the container has on runtime.
ADD/COPY - Copies new files, directories or remote file URLs from <src> to container file system.
WORKDIR- Sets the working directory for any instructions that follow it in the Dockerfile.
152. 152
What is DockerHub ?
A PaaS based Artifact Repository Manager (ARM) for Docker Images platform.
Similar to GitHub, but stores Docker binaries rather than source code.
Fermium
https://hub.docker.com/
157. What is Amazon Web Services (AWS) ?
157
Amazon Web Services (AWS) is a secure cloud services platform, offering compute power,
database storage, content delivery and other functionality to help businesses scale and grow.
AWS are both IaaS and PaaS provider.
161. AWS Elastic Compute Cloud (EC2)
161
AWS EC2 == Virtual Machine
AWS EC2 is on demand commuting power
Obtain and boot new virtual machines (servers) in seconds
Quickly scale capacity up and down
Servers from 0.02$ (2 cents) per hour
Support for Linux and Windows