SlideShare a Scribd company logo
Under the dome
(of failure driven pipeline)
Maciej Lasyk
4developers – Warsaw
2015-04-20
Join Fedora Infrastructure!
- learn Ansible
- learn Docker with Fedora Dockerfiles
http://fedoraproject.org/en/join-fedora
Agenda?
Don't run away ;)
[…]
Situations like this only reinforce my deep suspicion of
developers: They're often carelessly breaking things and
then disappearing, leaving Operations to clean up the
Mess.
[…]
“The Phoenix Project”
by Gene Kim, Kevin Behr and George Spafford
Under the Dome (of failure driven pipeline)
Under the Dome (of failure driven pipeline)
Under the Dome (of failure driven pipeline)
Conway's law (1968)
organizations which design systems ... are
constrained to produce designs which are copies
of the communication structures of these
organizations
http://en.wikipedia.org/wiki/Conway%27s_law
Ruth Malan (2008)
if the architecture of the system and the
architecture of the organization are at odds, the
architecture of the organization wins.
The organizational divides are going to drive the
true seams in the system.
http://traceinthesand.com/blog/2008/02/13/conways-law/
Yup, you're gut is telling truth...
Yup, you're gut is telling truth...
This will be another devops indoctrination
Yup, you're gut is telling truth...
This will be another devops indoctrination
What did you expect? ;)
Yup, you're gut is telling truth...
This will be another devops indoctrination
What did you expect? ;)
This presentation includes gentle product placement
Yup, you're gut is telling truth...
This will be another devops indoctrination
What did you expect? ;)
This presentation includes gentle product placement
DevOps Anti-Types & patterns
This is a copy/paste from
http://blog.matthewskelton.net/
w/my comments included
Great job Matthew! Thanks!
DevOps Anti-Types
http://blog.matthewskelton.net/2013/10/22/what-team-structure-is-right-for-devops-to-flourish/
DevOps Anti-Types
http://blog.matthewskelton.net/2013/10/22/what-team-structure-is-right-for-devops-to-flourish/
DevOps Anti-Types
http://blog.matthewskelton.net/2013/10/22/what-team-structure-is-right-for-devops-to-flourish/
DevOps Patterns
http://blog.matthewskelton.net/2013/10/22/what-team-structure-is-right-for-devops-to-flourish/
DevOps Patterns
http://blog.matthewskelton.net/2013/10/22/what-team-structure-is-right-for-devops-to-flourish/
DevOps Patterns
http://blog.matthewskelton.net/2013/10/22/what-team-structure-is-right-for-devops-to-flourish/
DevOps Patterns
http://blog.matthewskelton.net/2013/10/22/what-team-structure-is-right-for-devops-to-flourish/
DevOps Patterns
http://blog.matthewskelton.net/2013/10/22/what-team-structure-is-right-for-devops-to-flourish/
Ok let's CAMS
DevOPS ?== CAMS
(culture, automation, measurement, sharing)
DevOPS !== CAMS
DevOPS === people!
People
culture automation
measurement sharing
C for Culture
A for Automation
M for Monitoring
S for Sharing
Under the Dome (of failure driven pipeline)
Under the Dome (of failure driven pipeline)
Is there a need for change?
“agile” and “cloud”:
→ focus on delivery
→ close collaboration
→ lightweight environment and components
cultural change
modification of a society through innovation,
invention, discovery, or contact with other
societies
Dead sea effect
→ most talented evaporates
→ the residue
→ maintenance experts & bus factor == 1
http://brucefwebster.com/2008/04/11/the-wetware-crisis-the-dead-sea-effect/
→ talk. often. and get along
→ take responsibility - from beginning to the end
→ continuous improvement. seriously
→ be brave. don't be silent
→ it's better to be unpolite l/German than polite
l/Englishman
GTD? (getting things done)
GTD? (getting things done)
JFDI? (just fuckin' do it)
GTD? (getting things done)
JFDI? (just fuckin' do it)
MFBT? (move fast, break things)
GTD + JFDI + MFBT = FCH
GTD + JFDI + MFBT = FCH
(Fuckin' Customer Happy)
Under the Dome (of failure driven pipeline)
C for Culture
A for Automation
M for Monitoring
S for Sharing
Automation is big for most sysadmins. We’re
inherently lazy, so the idea of pushing a button
and making programs work for us? Appealing.
Standalone Sysadmin
http://www.standalone-sysadmin.com/blog/2011/04/view-from-the-other-side/
→ it has to be simple
→ don't reinvent the wheel. don't fabric
→ automate from very beginning
→ repeatable tasks leads to automation
→ repeatable tasks leads to automation
→ automation leads to consistency
→ repeatable tasks leads to automation
→ automation leads to consistency
→ consistency reduces errors
→ repeatable tasks leads to automation
→ automation leads to consistency
→ consistency reduces errors
→ reducing errors leads to stable environment
→ repeatable tasks leads to automation
→ automation leads to consistency
→ consistency reduces errors
→ reducing errors leads to stable environment
→ stable environment leads to less unplanned work
→ repeatable tasks leads to automation
→ automation leads to consistency
→ consistency reduces errors
→ reducing errors leads to stable environment
→ stable environment leads to less unplanned work
→ less unplanned work leads to focus on delivery
Remember?
http://blog.matthewskelton.net/2013/10/22/what-team-structure-is-right-for-devops-to-flourish/
Short story of Anti-Type C
“we don't need ops”
# it's madness with paths for different users and such option as:
# sudo su
# sudo -i
# su -
# su
# that is why we add variables to two places
ENVIRONMENT_FILE = '/etc/environment'
PROFILE_FILE = '/etc/profile'
INITIAL_PATH = '/usr/local/bin:/usr/bin:/bin'
# due to sudo issues (resetting PATH by /etc/sudoers)
# we have to add PATH to /root/.profile as well
Short story of Anti-Type C
“we don't need ops”
# it's madness with paths for different users and such option as:
# sudo su
# sudo -i
# su -
# su
# that is why we add variables to two places
ENVIRONMENT_FILE = '/etc/environment'
PROFILE_FILE = '/etc/profile'
INITIAL_PATH = '/usr/local/bin:/usr/bin:/bin'
# due to sudo issues (resetting PATH by /etc/sudoers)
# we have to add PATH to /root/.profile as well
Short story of Anti-Type C
“we don't need ops”
Shells:
→ login
→ non-login
→ interactive
→ non – interactive
Short story of Anti-Type C
“we don't need ops”
Shells:
→ login
→ non-login
→ interactive
→ non – interactive
→ su
→ sudo su: interactive, non-login, .bashrc
→ sudo su -: interactive, login, /etc/profile;/root/.profile;/root/.bashrc
→ sudo -i: interactive, login, /root/.profile;/root/.bashrc;/root/.login
→ sudo /bin/bash: interactive, non-login, ~/.bashrc
→ sudo -s: reads $SHELL and executes it
Short story of Anti-Type C
“we don't need ops”
Shells:
→ login
→ non-login
→ interactive
→ non – interactive
→ su
→ sudo su: interactive, non-login, .bashrc
→ sudo su -: interactive, login, /etc/profile;/root/.profile;/root/.bashrc
→ sudo -i: interactive, login, /root/.profile;/root/.bashrc;/root/.login
→ sudo /bin/bash: interactive, non-login, ~/.bashrc
→ sudo -s: reads $SHELL and executes it
Short story of Anti-Type C
“we don't need ops”
Shells:
→ login
→ non-login
→ interactive
→ non – interactive
→ su
→ sudo su: interactive, non-login, .bashrc
→ sudo su -: interactive, login, /etc/profile;/root/.profile;/root/.bashrc
→ sudo -i: interactive, login, /root/.profile;/root/.bashrc;/root/.login
→ sudo /bin/bash: interactive, non-login, ~/.bashrc
→ sudo -s: reads $SHELL and executes it
Short story of Anti-Type C
“we don't need ops”
Shells:
→ login
→ non-login
→ interactive
→ non – interactive
→ su
→ sudo su: interactive, non-login, .bashrc
→ sudo su -: interactive, login, /etc/profile;/root/.profile;/root/.bashrc
→ sudo -i: interactive, login, /root/.profile;/root/.bashrc;/root/.login
→ sudo /bin/bash: interactive, non-login, ~/.bashrc
→ sudo -s: reads $SHELL and executes it
Short story of Anti-Type C
“we don't need ops”
Shells:
→ login
→ non-login
→ interactive
→ non – interactive
→ su
→ sudo su: interactive, non-login, .bashrc
→ sudo su -: interactive, login, /etc/profile;/root/.profile;/root/.bashrc
→ sudo -i: interactive, login, /root/.profile;/root/.bashrc;/root/.login
→ sudo /bin/bash: interactive, non-login, ~/.bashrc
→ sudo -s: reads $SHELL and executes it
Short story of Anti-Type C
“we don't need ops”
Shells:
→ login
→ non-login
→ interactive
→ non – interactive
→ su
→ sudo su: interactive, non-login, .bashrc
→ sudo su -: interactive, login, /etc/profile;/root/.profile;/root/.bashrc
→ sudo -i: interactive, login, /root/.profile;/root/.bashrc;/root/.login
→ sudo /bin/bash: interactive, non-login, ~/.bashrc
→ sudo -s: reads $SHELL and executes it
def is_ubuntu():
return run("uname -a | grep Ubuntu | wc -l") == "1"
def install_apache_fix():
if is_ubuntu():
if exists("/lib/x86_64-linux-gnu/libssl.so.0.9.8"):
print "libssl.so.0.9.8 already installed - SKIPPING"
else:
sudo("apt-get -y install libssl0.9.8")
else:
#Debian
if exists("/usr/lib/libssl.so.0.9.8"):
print "libssl.so.0.9.8 already installed - SKIPPING"
else:
#download if necessary
url = "http://.../libssl0.9.8_0.9.8o-squeeze14_amd64.deb"
if download.sync_opt_download(_download_libssl_lock, url, store_file_path):
sudo('chmod ug+x %s' % store_file_path)
sudo("dpkg -i %s" % store_file_path)
def is_ubuntu():
return run("uname -a | grep Ubuntu | wc -l") == "1" /etc/issue maybe?
def install_apache_fix():
if is_ubuntu():
if exists("/lib/x86_64-linux-gnu/libssl.so.0.9.8"):
print "libssl.so.0.9.8 already installed - SKIPPING"
else:
sudo("apt-get -y install libssl0.9.8")
else:
#Debian
if exists("/usr/lib/libssl.so.0.9.8"):
print "libssl.so.0.9.8 already installed - SKIPPING"
else:
#download if necessary
url = "http://.../libssl0.9.8_0.9.8o-squeeze14_amd64.deb"
if download.sync_opt_download(_download_libssl_lock, url, store_file_path):
sudo('chmod ug+x %s' % store_file_path)
sudo("dpkg -i %s" % store_file_path)
def is_ubuntu():
return run("uname -a | grep Ubuntu | wc -l") == "1"
def install_apache_fix():
if is_ubuntu():
if exists("/lib/x86_64-linux-gnu/libssl.so.0.9.8"):
print "libssl.so.0.9.8 already installed - SKIPPING"
else:
sudo("apt-get -y install libssl0.9.8")
else:
#Debian
if exists("/usr/lib/libssl.so.0.9.8"):
print "libssl.so.0.9.8 already installed - SKIPPING"
else:
#download if necessary
url = "http://.../libssl0.9.8_0.9.8o-squeeze14_amd64.deb"
if download.sync_opt_download(_download_libssl_lock, url, store_file_path):
sudo('chmod ug+x %s' % store_file_path)
sudo("dpkg -i %s" % store_file_path)
def is_ubuntu():
return run("uname -a | grep Ubuntu | wc -l") == "1"
def install_apache_fix():
if is_ubuntu():
if exists("/lib/x86_64-linux-gnu/libssl.so.0.9.8"): ldconfig maybe?
print "libssl.so.0.9.8 already installed - SKIPPING"
else:
sudo("apt-get -y install libssl0.9.8")
else:
#Debian
if exists("/usr/lib/libssl.so.0.9.8"):
print "libssl.so.0.9.8 already installed - SKIPPING"
else:
#download if necessary
url = "http://.../libssl0.9.8_0.9.8o-squeeze14_amd64.deb"
if download.sync_opt_download(_download_libssl_lock, url, store_file_path):
sudo('chmod ug+x %s' % store_file_path)
sudo("dpkg -i %s" % store_file_path)
def is_ubuntu():
return run("uname -a | grep Ubuntu | wc -l") == "1"
def install_apache_fix():
if is_ubuntu():
if exists("/lib/x86_64-linux-gnu/libssl.so.0.9.8"):
print "libssl.so.0.9.8 already installed - SKIPPING"
else:
sudo("apt-get -y install libssl0.9.8")
else:
#Debian
if exists("/usr/lib/libssl.so.0.9.8"):
print "libssl.so.0.9.8 already installed - SKIPPING"
else:
#download if necessary
url = "http://.../libssl0.9.8_0.9.8o-squeeze14_amd64.deb"
if download.sync_opt_download(_download_libssl_lock, url, store_file_path):
sudo('chmod ug+x %s' % store_file_path)
sudo("dpkg -i %s" % store_file_path)
def is_ubuntu():
return run("uname -a | grep Ubuntu | wc -l") == "1"
def install_apache_fix():
if is_ubuntu():
if exists("/lib/x86_64-linux-gnu/libssl.so.0.9.8"):
print "libssl.so.0.9.8 already installed - SKIPPING"
else:
sudo("apt-get -y install libssl0.9.8")
else:
#Debian What about RHEL, Fedora, Slackware, Gentoo?
if exists("/usr/lib/libssl.so.0.9.8"):
print "libssl.so.0.9.8 already installed - SKIPPING"
else:
#downl. if necessary So whole this is for particular distro version?
url = "http://.../libssl0.9.8_0.9.8o-squeeze14_amd64.deb"
if download.sync_opt_download(_download_libssl_lock, url, store_file_path):
sudo('chmod ug+x %s' % store_file_path)
sudo("dpkg -i %s" % store_file_path)
def is_ubuntu():
return run("uname -a | grep Ubuntu | wc -l") == "1"
def install_apache_fix():
if is_ubuntu():
if exists("/lib/x86_64-linux-gnu/libssl.so.0.9.8"):
print "libssl.so.0.9.8 already installed - SKIPPING"
else:
sudo("apt-get -y install libssl0.9.8")
else:
#Debian
if exists("/usr/lib/libssl.so.0.9.8"):
print "libssl.so.0.9.8 already installed - SKIPPING"
else:
#downl. if necessary
url = "http://libssl0.9.8_0.9.8o-squeeze14_amd64.deb"
if download.sync_opt_download(_download_libssl_lock, url, store_file_path):
sudo('chmod ug+x %s' % store_file_path)
sudo("dpkg -i %s" % store_file_path)
def is_ubuntu():
return run("uname -a | grep Ubuntu | wc -l") == "1"
def install_apache_fix():
if is_ubuntu():
if exists("/lib/x86_64-linux-gnu/libssl.so.0.9.8"):
print "libssl.so.0.9.8 already installed - SKIPPING"
else:
sudo("apt-get -y install libssl0.9.8")
else:
#Debian
if exists("/usr/lib/libssl.so.0.9.8"):
print "libssl.so.0.9.8 already installed - SKIPPING"
else:
#downl. if necessary
url = "http://libssl0.9.8_0.9.8o-squeeze14_amd64.deb"
if download.sync_opt_download(_download_libssl_lock, url, store_file_path):
sudo('chmod ug+x %s' % store_file_path) # declarative madness
sudo("dpkg -i %s" % store_file_path)
Imperativeness vs declarativeness
Imperativeness vs declarativeness
def configure(dst_dir, config_properties, installer_file):
_copy_conf_file(dst_dir, properties)
def _copy_conf_file(dst_dir, properties):
sudo("cp %s %s" % (srcConfigPath, targetConfigPath))
change_directory_owner(targetConfigPath)
sudo('chmod ug+x %s' % store_file_path)
- name: configure this
hosts: all
tasks:
- name: copy conf file
file: >
src={{ some_source }}
dest={{ some_destination }}
perms=0750
Imperativeness vs declarativeness
def configure(dst_dir, config_properties, installer_file):
_copy_conf_file(dst_dir, properties)
def _copy_conf_file(dst_dir, properties):
sudo("cp %s %s" % (srcConfigPath, targetConfigPath))
change_directory_owner(targetConfigPath)
sudo('chmod ug+x %s' % store_file_path)
- name: configure this
hosts: all
tasks:
- name: copy conf file
file: >
src={{ some_source }}
dest={{ some_destination }}
perms=0750
Under the Dome (of failure driven pipeline)
→ flat learning curve
→ flat learning curve
→ doesn't required additional resources
→ flat learning curve
→ doesn't required additional resources
→ fit for maintenance jobs / procedures
→ flat learning curve
→ doesn't required additional resources
→ fit for maintenance jobs / procedures
→ great for any containers as non-daemon
→ flat learning curve
→ doesn't required additional resources
→ fit for maintenance jobs / procedures
→ great for any containers as non-daemon
→ deals with “deployment specs”
→ flat learning curve
→ doesn't required additional resources
→ fit for maintenance jobs / procedures
→ great for any containers as non-daemon
→ deals with “deployment specs”
→ might be easily adopted as universal language
Under the Dome (of failure driven pipeline)
Under the Dome (of failure driven pipeline)
→selinux enforcing i -rw-r--r--. stash stash
unconfined_u:object_r:mysqld_db_t:s0 authorized_keys
→/etc/ssh/sshd_config && /etc/network/interfaces
→ iptables-save nope?
→ broken _netfs ?
Under the Dome (of failure driven pipeline)
Under the Dome (of failure driven pipeline)
Under the Dome (of failure driven pipeline)
What if...
→ ./configure && make && make install → .zip
→ Dev & Ops have 2 different build & installation methods?
Plz..
→ pkg repos (or Nexus)
→ use fpm for creating pkgs if needed (demo)
C for Culture
A for Automation
M for Monitoring
S for Sharing
Under the Dome (of failure driven pipeline)
→ make developers create monitoring
→ find yourself between RRD and InfluxDB
→ will product team be able to query your monitoring DB?
→ Etsy case (Ganglia / Graphite)
→ make developers create monitoring
→ find yourself between RRD and InfluxDB
→ will product team be able to query your monitoring DB?
→ Etsy case (Ganglia / Graphite)
→ make developers create monitoring
→ find yourself between RRD and InfluxDB
→ will product team be able to query your monitoring DB?
→ Etsy case (Ganglia / Graphite)
→ make developers create monitoring
→ find yourself between RRD and InfluxDB
→ will product team be able to query your monitoring DB?
→ Etsy case (Ganglia / Graphite)
C for Culture
A for Automation
M for Monitoring
S for Sharing
→ learn on OPS mistakes
→ Major Incident Reports – source of improvement
→ Learn developers about change management
→ Make CM an easy process. Use simple tools.
→ learn on OPS mistakes
→ Major Incident Reports – source of improvement
→ Learn developers about change management
→ Make CM an easy process. Use simple tools.
→ learn on OPS mistakes
→ Major Incident Reports – source of improvement
→ Learn developers about change management
→ Make CM an easy process. Use simple tools.
→ learn on OPS mistakes
→ Major Incident Reports – source of improvement
→ Learn developers about change management
→ Make CM an easy process. Use simple tools.
Let's arch the infrastructure
Addressing the space
→ VLSM
→ DHCP & DDNS
→ KISS: flat networks!
→ stop /24!
Addressing the space
→ VLSM
→ DHCP & DDNS
→ KISS: flat networks!
→ stop /24!
Addressing the space
→ VLSM
→ DHCP & DDNS
→ KISS: flat networks!
→ stop /24!
Addressing the space
→ VLSM
→ DHCP & DDNS
→ KISS: flat networks!
→ stop /24!
Under the Dome (of failure driven pipeline)
What about DNS?
→ BIND roxx (views etc)
→ KISS: maybe decentralized w/Ansible?
view "internal-view" {
match-clients { internal; };
recursion yes;
zone "lasyk.info" IN {
type master;
file "internal.lasyk.info.conf";
allow-transfer { any; }
};
view "external-view" {
match-clients { any; };
recursion no;
zone "lasyk.info" IN {
type master;
file "external.lasyk.info.conf";
allow-transfer { none; };
};
view "internal-view" {
match-clients { internal; };
recursion yes;
zone "lasyk.info" IN {
type master;
file "internal.lasyk.info.conf";
allow-transfer { any; }
};
view "external-view" {
match-clients { any; };
recursion no;
zone "lasyk.info" IN {
type master;
file "external.lasyk.info.conf";
allow-transfer { none; };
};
Under the Dome (of failure driven pipeline)
Linux Containers = namespaces + cgroups + storage
Linux containers equation
Control Groups provide a mechanism for
aggregating/partitioning sets of tasks, and
all their future children, into hierarchical groups
with specialized behavior
control groups (cgroups)
→grouping processes
→allocating resources to particular groups
→memory
→network
→CPU
→storage bandwidth (I/O throttling)
→device whitelisting
control groups (cgroups)
→grouping processes
→allocating resources to particular groups
→memory
→network
→CPU
→storage bandwidth (I/O throttling)
→device whitelisting
control groups (cgroups)
→grouping processes
→allocating resources to particular groups
→memory
→network
→CPU
→storage bandwidth (I/O throttling)
→device whitelisting
control groups (cgroups)
→grouping processes
→allocating resources to particular groups
→memory
→network
→CPU
→storage bandwidth (I/O throttling)
→device whitelisting
control groups (cgroups)
→grouping processes
→allocating resources to particular groups
→memory
→network
→CPU
→storage bandwidth (I/O throttling)
→device whitelisting
control groups (cgroups)
→grouping processes
→allocating resources to particular groups
→memory
→network
→CPU
→storage bandwidth (I/O throttling)
→device whitelisting
control groups (cgroups)
→grouping processes
→allocating resources to particular groups
→memory
→network
→CPU
→storage bandwidth (I/O throttling)
→device whitelisting
control groups (cgroups)
little demo?
control groups (cgroups)
Providing a unique views of the system for processes.
→ PID – PIDs isolation
→ NET – network isolation (via virt-ifaces; demo)
→ IPC – won't use this
→ MNT – chroot like; deals w/mountpoints
→ UTS – deals w/hostname
Kernel Namespaces
Providing a unique views of the system for processes.
→ PID – PIDs isolation
→ NET – network isolation (via virt-ifaces; demo)
→ IPC – won't use this
→ MNT – chroot like; deals w/mountpoints
→ UTS – deals w/hostname
Kernel Namespaces
Providing a unique views of the system for processes.
→ PID – PIDs isolation
→ NET – network isolation (via virt-ifaces; demo)
→ IPC – won't use this
→ MNT – chroot like; deals w/mountpoints
→ UTS – deals w/hostname
Kernel Namespaces
Providing a unique views of the system for processes.
→ PID – PIDs isolation
→ NET – network isolation (via virt-ifaces; demo)
→ IPC – won't use this
→ MNT – chroot like; deals w/mountpoints
→ UTS – deals w/hostname
Kernel Namespaces
Providing a unique views of the system for processes.
→ PID – PIDs isolation
→ NET – network isolation (via virt-ifaces; demo)
→ IPC – won't use this
→ MNT – chroot like; deals w/mountpoints
→ UTS – deals w/hostname
Kernel Namespaces
Providing a unique views of the system for processes.
→ PID – PIDs isolation
→ NET – network isolation (via virt-ifaces; demo)
→ IPC – won't use this
→ MNT – chroot like; deals w/mountpoints
→ UTS – deals w/hostname
Kernel Namespaces
Providing a unique views of the system for processes.
→ PID – PIDs isolation
→ NET – network isolation (via virt-ifaces; demo)
→ IPC – won't use this
→ MNT – chroot like; deals w/mountpoints
→ UTS – deals w/hostname
Kernel Namespaces
little demo?
Kernel Namespaces
→ hell fast (you'll see)
→ page cache sharing
→ finally in upstream kernel (in rhel from 7.2)
→ finally supported by docker (-s overlay)
→ SELinux not there yet (but will be)
OverlayFS
→ hell fast (you'll see)
→ page cache sharing
→ finally in upstream kernel (in rhel from 7.2)
→ finally supported by docker (-s overlay)
→ SELinux not there yet (but will be)
OverlayFS
→ hell fast (you'll see)
→ page cache sharing
→ finally in upstream kernel (in rhel from 7.2)
→ finally supported by docker (-s overlay)
→ SELinux not there yet (but will be)
OverlayFS
→ hell fast (you'll see)
→ page cache sharing
→ finally in upstream kernel (in rhel from 7.2)
→ finally supported by docker (-s overlay)
→ SELinux not there yet (but will be)
OverlayFS
→ hell fast (you'll see)
→ page cache sharing
→ finally in upstream kernel (in rhel from 7.2)
→ finally supported by docker (-s overlay)
→ SELinux not there yet (but will be)
OverlayFS
http://developerblog.redhat.com/2014/09/30/overview-storage-scalability-docker/
OverlayFS
http://developerblog.redhat.com/2014/09/30/overview-storage-scalability-docker/
OverlayFS
http://developerblog.redhat.com/2014/09/30/overview-storage-scalability-docker/
OverlayFS
Developers' envs?
→ use containers!
→ configure cgroups
→ use LXC / LXC Web Panel
→ use Ansible for spinning up anything!
Developers' envs?
→ use containers!
→ configure cgroups
→ use LXC / LXC Web Panel
→ use Ansible for spinning up anything!
Developers' envs?
→ use containers!
→ configure cgroups
→ use LXC / LXC Web Panel
→ use Ansible for spinning up anything!
Developers' envs?
→ use containers!
→ configure cgroups
→ use LXC / LXC Web Panel
→ use Ansible for spinning up anything!
Under the Dome (of failure driven pipeline)
Containers embraces granularity → microservices!
Containers embraces granularity → microservices!
Watch out for microservices architecture, or...
Containers embraces granularity → microservices!
Watch out for microservices architecture, or...
Under the Dome (of failure driven pipeline)
Who knows FHS?
Who knows FHS?
→ 'temp' – what it consist?
Who knows FHS?
→ 'temp' – what it consist?
→ actually: “This Entity Must Persist” ;)
Who knows FHS?
→ 'temp' – what it consist?
→ actually: “This Entity Must Persist” ;)
→ Define your FHS!
Mikado Method for the win!
→ set a goal
→ experiment
→ visualize
→ rollback
Mikado Method for the win!
→ set a goal
→ experiment
→ visualize
→ rollback
Mikado Method for the win!
→ set a goal
→ experiment
→ visualize
→ rollback
Mikado Method for the win!
→ set a goal
→ experiment
→ visualize
→ rollback
Mikado Method for the win!
→ before any work and rollbacks..
→ remember: monitoring & tests are your friends!
→ think about testing strategy – think heatmaps!
Ansible & infra layers
Layer 1: bare metal,
Layer 2: VM
Layer 3: container
Networking
Hypervisor + VM provisioning
Storage
Networking
Container's engine & provisioning
Application build
Application env
Network interfaces
Storage mounts
Resources allocation
repo1
repo2
repo3
Much simpler w/one, flat network (for small envs)!
Ansible & infra layers
Layer 1: bare metal,
Layer 2: VM
Layer 3: container
Networking
Hypervisor + VM provisioning
Storage
Networking
Container's engine & provisioning
Application build
Application env
Network interfaces
Storage mounts
Resources allocation
repo1
repo2
repo3
Much simpler w/one, flat network (for small envs)!
repo2
Layer 2: VM
Networking
Container's engine & provisioning
repo2
Ansible & infra layers
Layer 1: bare metal,
Layer 2: VM
Layer 3: container
Networking
Hypervisor + VM provisioning
Storage
Networking
Container's engine & provisioning
Application build
Application env
Network interfaces
Storage mounts
Resources allocation
repo1
repo2
repo3
Much simpler w/one, flat network (for small envs)!
repo2
Layer 2: VM
Networking
Container's engine & provisioning
repo2
Network interfaces
Storage mounts
repo2
Ansible & infra layers
Layer 1: bare metal,
Layer 2: VM
Layer 3: container
Networking
Hypervisor + VM provisioning
Storage
Networking
Container's engine & provisioning
Application build
Application env
Network interfaces
Storage mounts
Resources allocation
repo1
repo2
repo3
Much simpler w/one, flat network (for small envs)!
repo2
Layer 2: VM
Networking
Container's engine & provisioning
repo2
Network interfaces
Storage mounts
repo2
Layer 3: container
Application build
Application env
repo3
Ansible & infra layers
Layer 1: bare metal,
Layer 2: VM
Layer 3: container
Networking
Hypervisor + VM provisioning
Storage
Networking
Container's engine & provisioning
Application build
Application env
Network interfaces
Storage mounts
Resources allocation
repo1
repo2
repo3
Much simpler w/one, flat network (for small envs)!
repo2
Layer 2: VM
Networking
Container's engine & provisioning
repo2
Network interfaces
Storage mounts
repo2
Layer 3: container
Application build
Application env
repo3
Resources allocation
repo3
Ansible & infra layers
Layer 1: bare metal,
Layer 2: VM
Layer 3: container
Networking
Hypervisor + VM provisioning
Storage
Networking
Container's engine & provisioning
Application build
Application env
Network interfaces
Storage mounts
Resources allocation
repo1
repo2
repo3
Much simpler w/one, flat network (for small envs)!
→ automated service discovery and registration framework
→ ideal for SOA architectures
→ ideal for continuous integration & delivery
→ solves “works on my machine” problem
SmartStack
→ automated service discovery and registration framework
→ ideal for SOA architectures
→ ideal for continuous integration & delivery
→ solves “works on my machine” problem
SmartStack
haproxy + nerve + synapse + zookeper = smartstack
Synapse
→ discovery service (via zookeeper or etcd)
→ installed on every node
→ writes haproxy configuration
→ application doesn't have to be aware of this
→ works same on bare / VM / docker
→ https://github.com/airbnb/nerve
SmartStack
SmartStack
Nerve
→ health checks (pluggable)
→ register service info to zookeper (or etcd)
→ https://github.com/airbnb/synapse
SmartStack
SmartStack
SmartStack
Smartstack + Docker = <3
Smartstack + Docker = <3
but also remember about Consul
(come to #dockerkrk 2 meetup!)
questions?
Archaeological workshop
Archaeological workshop
→ nmap, tcpdump, lsof, strace, sysdig, sar
→ cgroups throttling on-the-fly
Do we have time for demo?
Hardware: disks?
→ RAID5 vs RAID10
→ Howto RAID over 1 disk ;)
→ Cheap SSD drives?
Hardware: disks?
→ RAID5 vs RAID10
→ Howto RAID over 1 disk ;)
→ Cheap SSD drives?
Hardware: disks?
→ RAID5 vs RAID10
→ Howto RAID over 1 disk ;)
→ Cheap SSD drives?
http://techreport.com/review/27909/the-ssd-endurance-experiment-theyre-all-dead
Why use LVM?
→ indexation (capacity, inodes check)
→ capacity planning / iops per mount
Under the dome
(of failure driven pipeline)
Maciej Lasyk
4developers – Warsaw
2015-04-20

More Related Content

What's hot

DevOops & How I hacked you DevopsDays DC June 2015
DevOops & How I hacked you DevopsDays DC June 2015DevOops & How I hacked you DevopsDays DC June 2015
DevOops & How I hacked you DevopsDays DC June 2015
Chris Gates
 
Boycott Docker
Boycott DockerBoycott Docker
Boycott Docker
Paolo Tonin
 
Docker: automation for the rest of us
Docker: automation for the rest of usDocker: automation for the rest of us
Docker: automation for the rest of us
Jérôme Petazzoni
 
DevOOPS: Attacks and Defenses for DevOps Toolchains
DevOOPS: Attacks and Defenses for DevOps ToolchainsDevOOPS: Attacks and Defenses for DevOps Toolchains
DevOOPS: Attacks and Defenses for DevOps Toolchains
Chris Gates
 
Introducing Docker
Introducing DockerIntroducing Docker
Introducing Docker
Francesco Pantano
 
Devoops: DoJ Annual Cybersecurity Training Symposium Edition 2015
Devoops: DoJ Annual Cybersecurity Training Symposium Edition 2015Devoops: DoJ Annual Cybersecurity Training Symposium Edition 2015
Devoops: DoJ Annual Cybersecurity Training Symposium Edition 2015
Chris Gates
 
Windows Attacks AT is the new black
Windows Attacks   AT is the new blackWindows Attacks   AT is the new black
Windows Attacks AT is the new black
Rob Fuller
 
Getting root with benign app store apps vsecurityfest
Getting root with benign app store apps vsecurityfestGetting root with benign app store apps vsecurityfest
Getting root with benign app store apps vsecurityfest
Csaba Fitzl
 
Shared Object images in Docker: What you need is what you want.
Shared Object images in Docker: What you need is what you want.Shared Object images in Docker: What you need is what you want.
Shared Object images in Docker: What you need is what you want.
Workhorse Computing
 
The Dirty Little Secrets They Didn’t Teach You In Pentesting Class
The Dirty Little Secrets They Didn’t Teach You In Pentesting ClassThe Dirty Little Secrets They Didn’t Teach You In Pentesting Class
The Dirty Little Secrets They Didn’t Teach You In Pentesting Class
Rob Fuller
 
Docker and Puppet — Puppet Camp L.A. — SCALE12X
Docker and Puppet — Puppet Camp L.A. — SCALE12XDocker and Puppet — Puppet Camp L.A. — SCALE12X
Docker and Puppet — Puppet Camp L.A. — SCALE12X
Jérôme Petazzoni
 
Django로 만든 웹 애플리케이션 도커라이징하기 + 도커 컴포즈로 개발 환경 구축하기
Django로 만든 웹 애플리케이션 도커라이징하기 + 도커 컴포즈로 개발 환경 구축하기Django로 만든 웹 애플리케이션 도커라이징하기 + 도커 컴포즈로 개발 환경 구축하기
Django로 만든 웹 애플리케이션 도커라이징하기 + 도커 컴포즈로 개발 환경 구축하기
raccoony
 
Running Docker in Development & Production (#ndcoslo 2015)
Running Docker in Development & Production (#ndcoslo 2015)Running Docker in Development & Production (#ndcoslo 2015)
Running Docker in Development & Production (#ndcoslo 2015)
Ben Hall
 
Vagrant for real codemotion (moar tips! ;-))
Vagrant for real codemotion (moar tips! ;-))Vagrant for real codemotion (moar tips! ;-))
Vagrant for real codemotion (moar tips! ;-))
Michele Orselli
 
Lessons from running potentially malicious code inside Docker containers
Lessons from running potentially malicious code inside Docker containersLessons from running potentially malicious code inside Docker containers
Lessons from running potentially malicious code inside Docker containers
Ben Hall
 
Jump into Squeak - Integrate Squeak projects with Docker & Github
Jump into Squeak - Integrate Squeak projects with Docker & GithubJump into Squeak - Integrate Squeak projects with Docker & Github
Jump into Squeak - Integrate Squeak projects with Docker & Github
hubx
 
Scaling Django
Scaling DjangoScaling Django
Scaling Django
Mike Malone
 
Getting root with benign app store apps
Getting root with benign app store appsGetting root with benign app store apps
Getting root with benign app store apps
Csaba Fitzl
 
Regex Considered Harmful: Use Rosie Pattern Language Instead
Regex Considered Harmful: Use Rosie Pattern Language InsteadRegex Considered Harmful: Use Rosie Pattern Language Instead
Regex Considered Harmful: Use Rosie Pattern Language Instead
All Things Open
 
Appsec DC - wXf -2010
Appsec DC - wXf  -2010Appsec DC - wXf  -2010
Appsec DC - wXf -2010
Chris Gates
 

What's hot (20)

DevOops & How I hacked you DevopsDays DC June 2015
DevOops & How I hacked you DevopsDays DC June 2015DevOops & How I hacked you DevopsDays DC June 2015
DevOops & How I hacked you DevopsDays DC June 2015
 
Boycott Docker
Boycott DockerBoycott Docker
Boycott Docker
 
Docker: automation for the rest of us
Docker: automation for the rest of usDocker: automation for the rest of us
Docker: automation for the rest of us
 
DevOOPS: Attacks and Defenses for DevOps Toolchains
DevOOPS: Attacks and Defenses for DevOps ToolchainsDevOOPS: Attacks and Defenses for DevOps Toolchains
DevOOPS: Attacks and Defenses for DevOps Toolchains
 
Introducing Docker
Introducing DockerIntroducing Docker
Introducing Docker
 
Devoops: DoJ Annual Cybersecurity Training Symposium Edition 2015
Devoops: DoJ Annual Cybersecurity Training Symposium Edition 2015Devoops: DoJ Annual Cybersecurity Training Symposium Edition 2015
Devoops: DoJ Annual Cybersecurity Training Symposium Edition 2015
 
Windows Attacks AT is the new black
Windows Attacks   AT is the new blackWindows Attacks   AT is the new black
Windows Attacks AT is the new black
 
Getting root with benign app store apps vsecurityfest
Getting root with benign app store apps vsecurityfestGetting root with benign app store apps vsecurityfest
Getting root with benign app store apps vsecurityfest
 
Shared Object images in Docker: What you need is what you want.
Shared Object images in Docker: What you need is what you want.Shared Object images in Docker: What you need is what you want.
Shared Object images in Docker: What you need is what you want.
 
The Dirty Little Secrets They Didn’t Teach You In Pentesting Class
The Dirty Little Secrets They Didn’t Teach You In Pentesting ClassThe Dirty Little Secrets They Didn’t Teach You In Pentesting Class
The Dirty Little Secrets They Didn’t Teach You In Pentesting Class
 
Docker and Puppet — Puppet Camp L.A. — SCALE12X
Docker and Puppet — Puppet Camp L.A. — SCALE12XDocker and Puppet — Puppet Camp L.A. — SCALE12X
Docker and Puppet — Puppet Camp L.A. — SCALE12X
 
Django로 만든 웹 애플리케이션 도커라이징하기 + 도커 컴포즈로 개발 환경 구축하기
Django로 만든 웹 애플리케이션 도커라이징하기 + 도커 컴포즈로 개발 환경 구축하기Django로 만든 웹 애플리케이션 도커라이징하기 + 도커 컴포즈로 개발 환경 구축하기
Django로 만든 웹 애플리케이션 도커라이징하기 + 도커 컴포즈로 개발 환경 구축하기
 
Running Docker in Development & Production (#ndcoslo 2015)
Running Docker in Development & Production (#ndcoslo 2015)Running Docker in Development & Production (#ndcoslo 2015)
Running Docker in Development & Production (#ndcoslo 2015)
 
Vagrant for real codemotion (moar tips! ;-))
Vagrant for real codemotion (moar tips! ;-))Vagrant for real codemotion (moar tips! ;-))
Vagrant for real codemotion (moar tips! ;-))
 
Lessons from running potentially malicious code inside Docker containers
Lessons from running potentially malicious code inside Docker containersLessons from running potentially malicious code inside Docker containers
Lessons from running potentially malicious code inside Docker containers
 
Jump into Squeak - Integrate Squeak projects with Docker & Github
Jump into Squeak - Integrate Squeak projects with Docker & GithubJump into Squeak - Integrate Squeak projects with Docker & Github
Jump into Squeak - Integrate Squeak projects with Docker & Github
 
Scaling Django
Scaling DjangoScaling Django
Scaling Django
 
Getting root with benign app store apps
Getting root with benign app store appsGetting root with benign app store apps
Getting root with benign app store apps
 
Regex Considered Harmful: Use Rosie Pattern Language Instead
Regex Considered Harmful: Use Rosie Pattern Language InsteadRegex Considered Harmful: Use Rosie Pattern Language Instead
Regex Considered Harmful: Use Rosie Pattern Language Instead
 
Appsec DC - wXf -2010
Appsec DC - wXf  -2010Appsec DC - wXf  -2010
Appsec DC - wXf -2010
 

Viewers also liked

Reply by First Appellate Authority CIC New Delhi against Non-Implementation o...
Reply by First Appellate Authority CIC New Delhi against Non-Implementation o...Reply by First Appellate Authority CIC New Delhi against Non-Implementation o...
Reply by First Appellate Authority CIC New Delhi against Non-Implementation o...
Om Prakash Poddar
 
Second Appeal against CIC New Delhi for Non-Implementation of Section 7(1) of...
Second Appeal against CIC New Delhi for Non-Implementation of Section 7(1) of...Second Appeal against CIC New Delhi for Non-Implementation of Section 7(1) of...
Second Appeal against CIC New Delhi for Non-Implementation of Section 7(1) of...
Om Prakash Poddar
 
Communication with CIC New Delhi and PMO to make applicable Section 7 (1) of ...
Communication with CIC New Delhi and PMO to make applicable Section 7 (1) of ...Communication with CIC New Delhi and PMO to make applicable Section 7 (1) of ...
Communication with CIC New Delhi and PMO to make applicable Section 7 (1) of ...
Om Prakash Poddar
 
RTI against Lodgment of W.P.(criminal) D.NO 2188 of 2017 by Registrar Supreme...
RTI against Lodgment of W.P.(criminal) D.NO 2188 of 2017 by Registrar Supreme...RTI against Lodgment of W.P.(criminal) D.NO 2188 of 2017 by Registrar Supreme...
RTI against Lodgment of W.P.(criminal) D.NO 2188 of 2017 by Registrar Supreme...
Om Prakash Poddar
 
Programowanie AWSa z CLI, boto, Ansiblem i libcloudem
Programowanie AWSa z CLI, boto, Ansiblem i libcloudemProgramowanie AWSa z CLI, boto, Ansiblem i libcloudem
Programowanie AWSa z CLI, boto, Ansiblem i libcloudem
Maciej Lasyk
 
Git Submodules
Git SubmodulesGit Submodules
Git Submodules
Maciej Lasyk
 
About cultural change w/Devops
About cultural change w/DevopsAbout cultural change w/Devops
About cultural change w/Devops
Maciej Lasyk
 
Lodgement Order dated 28.01.2017 of Registrar Supreme Court of India
Lodgement Order dated 28.01.2017 of  Registrar Supreme Court of IndiaLodgement Order dated 28.01.2017 of  Registrar Supreme Court of India
Lodgement Order dated 28.01.2017 of Registrar Supreme Court of India
Om Prakash Poddar
 
Writ Petition Criminal NO.......of 2017 vide D.NO.3913 against Registrar Supr...
Writ Petition Criminal NO.......of 2017 vide D.NO.3913 against Registrar Supr...Writ Petition Criminal NO.......of 2017 vide D.NO.3913 against Registrar Supr...
Writ Petition Criminal NO.......of 2017 vide D.NO.3913 against Registrar Supr...
Om Prakash Poddar
 
Pregunta Pleno Febrero 2017 - Sareb, vivienda emergencia social
Pregunta Pleno Febrero 2017 - Sareb, vivienda emergencia socialPregunta Pleno Febrero 2017 - Sareb, vivienda emergencia social
Pregunta Pleno Febrero 2017 - Sareb, vivienda emergencia social
Unión Progreso y Democracia Alcobendas
 
High Availability (HA) Explained - second edition
High Availability (HA) Explained - second editionHigh Availability (HA) Explained - second edition
High Availability (HA) Explained - second edition
Maciej Lasyk
 
Stop disabling SELinux!
Stop disabling SELinux!Stop disabling SELinux!
Stop disabling SELinux!
Maciej Lasyk
 
Scaling and securing node.js apps
Scaling and securing node.js appsScaling and securing node.js apps
Scaling and securing node.js apps
Maciej Lasyk
 
Tema 2. al andalus y sus principales etapas
Tema 2. al andalus y sus principales etapasTema 2. al andalus y sus principales etapas
Tema 2. al andalus y sus principales etapas
copybird
 
Tema 1. el planeta tierra
Tema 1. el planeta tierraTema 1. el planeta tierra
Tema 1. el planeta tierra
copybird
 
Origen del castellano
Origen del castellanoOrigen del castellano
Origen del castellano
copybird
 

Viewers also liked (16)

Reply by First Appellate Authority CIC New Delhi against Non-Implementation o...
Reply by First Appellate Authority CIC New Delhi against Non-Implementation o...Reply by First Appellate Authority CIC New Delhi against Non-Implementation o...
Reply by First Appellate Authority CIC New Delhi against Non-Implementation o...
 
Second Appeal against CIC New Delhi for Non-Implementation of Section 7(1) of...
Second Appeal against CIC New Delhi for Non-Implementation of Section 7(1) of...Second Appeal against CIC New Delhi for Non-Implementation of Section 7(1) of...
Second Appeal against CIC New Delhi for Non-Implementation of Section 7(1) of...
 
Communication with CIC New Delhi and PMO to make applicable Section 7 (1) of ...
Communication with CIC New Delhi and PMO to make applicable Section 7 (1) of ...Communication with CIC New Delhi and PMO to make applicable Section 7 (1) of ...
Communication with CIC New Delhi and PMO to make applicable Section 7 (1) of ...
 
RTI against Lodgment of W.P.(criminal) D.NO 2188 of 2017 by Registrar Supreme...
RTI against Lodgment of W.P.(criminal) D.NO 2188 of 2017 by Registrar Supreme...RTI against Lodgment of W.P.(criminal) D.NO 2188 of 2017 by Registrar Supreme...
RTI against Lodgment of W.P.(criminal) D.NO 2188 of 2017 by Registrar Supreme...
 
Programowanie AWSa z CLI, boto, Ansiblem i libcloudem
Programowanie AWSa z CLI, boto, Ansiblem i libcloudemProgramowanie AWSa z CLI, boto, Ansiblem i libcloudem
Programowanie AWSa z CLI, boto, Ansiblem i libcloudem
 
Git Submodules
Git SubmodulesGit Submodules
Git Submodules
 
About cultural change w/Devops
About cultural change w/DevopsAbout cultural change w/Devops
About cultural change w/Devops
 
Lodgement Order dated 28.01.2017 of Registrar Supreme Court of India
Lodgement Order dated 28.01.2017 of  Registrar Supreme Court of IndiaLodgement Order dated 28.01.2017 of  Registrar Supreme Court of India
Lodgement Order dated 28.01.2017 of Registrar Supreme Court of India
 
Writ Petition Criminal NO.......of 2017 vide D.NO.3913 against Registrar Supr...
Writ Petition Criminal NO.......of 2017 vide D.NO.3913 against Registrar Supr...Writ Petition Criminal NO.......of 2017 vide D.NO.3913 against Registrar Supr...
Writ Petition Criminal NO.......of 2017 vide D.NO.3913 against Registrar Supr...
 
Pregunta Pleno Febrero 2017 - Sareb, vivienda emergencia social
Pregunta Pleno Febrero 2017 - Sareb, vivienda emergencia socialPregunta Pleno Febrero 2017 - Sareb, vivienda emergencia social
Pregunta Pleno Febrero 2017 - Sareb, vivienda emergencia social
 
High Availability (HA) Explained - second edition
High Availability (HA) Explained - second editionHigh Availability (HA) Explained - second edition
High Availability (HA) Explained - second edition
 
Stop disabling SELinux!
Stop disabling SELinux!Stop disabling SELinux!
Stop disabling SELinux!
 
Scaling and securing node.js apps
Scaling and securing node.js appsScaling and securing node.js apps
Scaling and securing node.js apps
 
Tema 2. al andalus y sus principales etapas
Tema 2. al andalus y sus principales etapasTema 2. al andalus y sus principales etapas
Tema 2. al andalus y sus principales etapas
 
Tema 1. el planeta tierra
Tema 1. el planeta tierraTema 1. el planeta tierra
Tema 1. el planeta tierra
 
Origen del castellano
Origen del castellanoOrigen del castellano
Origen del castellano
 

Similar to Under the Dome (of failure driven pipeline)

Porting your favourite cmdline tool to Android
Porting your favourite cmdline tool to AndroidPorting your favourite cmdline tool to Android
Porting your favourite cmdline tool to Android
Vlatko Kosturjak
 
Hybrid Apps with Ionic Framework
Hybrid Apps with Ionic FrameworkHybrid Apps with Ionic Framework
Hybrid Apps with Ionic Framework
Bramus Van Damme
 
DevOpSec_DockerNPodMan-20230220.pdf
DevOpSec_DockerNPodMan-20230220.pdfDevOpSec_DockerNPodMan-20230220.pdf
DevOpSec_DockerNPodMan-20230220.pdf
kanedafromparis
 
Shell scripting
Shell scriptingShell scripting
Shell scripting
Geeks Anonymes
 
Ant Build Tool
Ant Build ToolAnt Build Tool
Ant Build Tool
Rafael Mumme
 
Automate Yo' Self
Automate Yo' SelfAutomate Yo' Self
Automate Yo' Self
John Anderson
 
Demystifying Maven
Demystifying MavenDemystifying Maven
Demystifying Maven
Mike Desjardins
 
Long life to vagrant… Vagrant is dead
Long life to vagrant… Vagrant is deadLong life to vagrant… Vagrant is dead
Long life to vagrant… Vagrant is dead
Federico Panini
 
[MeetUp][2nd] 컭on턺
[MeetUp][2nd] 컭on턺[MeetUp][2nd] 컭on턺
[MeetUp][2nd] 컭on턺
InfraEngineer
 
Makefiles in 2020 — Why they still matter
Makefiles in 2020 — Why they still matterMakefiles in 2020 — Why they still matter
Makefiles in 2020 — Why they still matter
Simon Brüggen
 
TinyOS installation Guide And Manual
TinyOS installation Guide And ManualTinyOS installation Guide And Manual
TinyOS installation Guide And Manual
Ankit Singh
 
Joxean Koret - Database Security Paradise [Rooted CON 2011]
Joxean Koret - Database Security Paradise [Rooted CON 2011]Joxean Koret - Database Security Paradise [Rooted CON 2011]
Joxean Koret - Database Security Paradise [Rooted CON 2011]
RootedCON
 
JsDay - It's not you, It's me (or how to avoid being coupled with a Javascrip...
JsDay - It's not you, It's me (or how to avoid being coupled with a Javascrip...JsDay - It's not you, It's me (or how to avoid being coupled with a Javascrip...
JsDay - It's not you, It's me (or how to avoid being coupled with a Javascrip...
Marco Cedaro
 
Linux advanced privilege escalation
Linux advanced privilege escalationLinux advanced privilege escalation
Linux advanced privilege escalation
Jameel Nabbo
 
Mothra - A FreeBSD send-pr tool for bugzilla system
Mothra - A FreeBSD send-pr tool for bugzilla systemMothra - A FreeBSD send-pr tool for bugzilla system
Mothra - A FreeBSD send-pr tool for bugzilla system
Daniel Lin
 
Open event (show&tell april 2016)
Open event (show&tell april 2016)Open event (show&tell april 2016)
Open event (show&tell april 2016)
Jorge López-Lago
 
Trusting files (and their formats)
Trusting files (and their formats)Trusting files (and their formats)
Trusting files (and their formats)
Ange Albertini
 
Riding on rails3 with full stack of gems
Riding on rails3 with full stack of gemsRiding on rails3 with full stack of gems
Riding on rails3 with full stack of gems
Andy Wang
 
Design and Evolution of cyber-dojo
Design and Evolution of cyber-dojoDesign and Evolution of cyber-dojo
Design and Evolution of cyber-dojo
Jon Jagger
 
Docker app armor_usecase
Docker app armor_usecaseDocker app armor_usecase
Docker app armor_usecase
Kazuki Omo
 

Similar to Under the Dome (of failure driven pipeline) (20)

Porting your favourite cmdline tool to Android
Porting your favourite cmdline tool to AndroidPorting your favourite cmdline tool to Android
Porting your favourite cmdline tool to Android
 
Hybrid Apps with Ionic Framework
Hybrid Apps with Ionic FrameworkHybrid Apps with Ionic Framework
Hybrid Apps with Ionic Framework
 
DevOpSec_DockerNPodMan-20230220.pdf
DevOpSec_DockerNPodMan-20230220.pdfDevOpSec_DockerNPodMan-20230220.pdf
DevOpSec_DockerNPodMan-20230220.pdf
 
Shell scripting
Shell scriptingShell scripting
Shell scripting
 
Ant Build Tool
Ant Build ToolAnt Build Tool
Ant Build Tool
 
Automate Yo' Self
Automate Yo' SelfAutomate Yo' Self
Automate Yo' Self
 
Demystifying Maven
Demystifying MavenDemystifying Maven
Demystifying Maven
 
Long life to vagrant… Vagrant is dead
Long life to vagrant… Vagrant is deadLong life to vagrant… Vagrant is dead
Long life to vagrant… Vagrant is dead
 
[MeetUp][2nd] 컭on턺
[MeetUp][2nd] 컭on턺[MeetUp][2nd] 컭on턺
[MeetUp][2nd] 컭on턺
 
Makefiles in 2020 — Why they still matter
Makefiles in 2020 — Why they still matterMakefiles in 2020 — Why they still matter
Makefiles in 2020 — Why they still matter
 
TinyOS installation Guide And Manual
TinyOS installation Guide And ManualTinyOS installation Guide And Manual
TinyOS installation Guide And Manual
 
Joxean Koret - Database Security Paradise [Rooted CON 2011]
Joxean Koret - Database Security Paradise [Rooted CON 2011]Joxean Koret - Database Security Paradise [Rooted CON 2011]
Joxean Koret - Database Security Paradise [Rooted CON 2011]
 
JsDay - It's not you, It's me (or how to avoid being coupled with a Javascrip...
JsDay - It's not you, It's me (or how to avoid being coupled with a Javascrip...JsDay - It's not you, It's me (or how to avoid being coupled with a Javascrip...
JsDay - It's not you, It's me (or how to avoid being coupled with a Javascrip...
 
Linux advanced privilege escalation
Linux advanced privilege escalationLinux advanced privilege escalation
Linux advanced privilege escalation
 
Mothra - A FreeBSD send-pr tool for bugzilla system
Mothra - A FreeBSD send-pr tool for bugzilla systemMothra - A FreeBSD send-pr tool for bugzilla system
Mothra - A FreeBSD send-pr tool for bugzilla system
 
Open event (show&tell april 2016)
Open event (show&tell april 2016)Open event (show&tell april 2016)
Open event (show&tell april 2016)
 
Trusting files (and their formats)
Trusting files (and their formats)Trusting files (and their formats)
Trusting files (and their formats)
 
Riding on rails3 with full stack of gems
Riding on rails3 with full stack of gemsRiding on rails3 with full stack of gems
Riding on rails3 with full stack of gems
 
Design and Evolution of cyber-dojo
Design and Evolution of cyber-dojoDesign and Evolution of cyber-dojo
Design and Evolution of cyber-dojo
 
Docker app armor_usecase
Docker app armor_usecaseDocker app armor_usecase
Docker app armor_usecase
 

More from Maciej Lasyk

Rundeck & Ansible
Rundeck & AnsibleRundeck & Ansible
Rundeck & Ansible
Maciej Lasyk
 
Docker 1.11
Docker 1.11Docker 1.11
Docker 1.11
Maciej Lasyk
 
Co powinieneś wiedzieć na temat devops?f
Co powinieneś wiedzieć na temat devops?f Co powinieneś wiedzieć na temat devops?f
Co powinieneś wiedzieć na temat devops?f
Maciej Lasyk
 
"Containers do not contain"
"Containers do not contain""Containers do not contain"
"Containers do not contain"
Maciej Lasyk
 
Continuous Security in DevOps
Continuous Security in DevOpsContinuous Security in DevOps
Continuous Security in DevOps
Maciej Lasyk
 
Orchestrating docker containers at scale (#DockerKRK edition)
Orchestrating docker containers at scale (#DockerKRK edition)Orchestrating docker containers at scale (#DockerKRK edition)
Orchestrating docker containers at scale (#DockerKRK edition)
Maciej Lasyk
 
Orchestrating docker containers at scale (PJUG edition)
Orchestrating docker containers at scale (PJUG edition)Orchestrating docker containers at scale (PJUG edition)
Orchestrating docker containers at scale (PJUG edition)
Maciej Lasyk
 
Orchestrating Docker containers at scale
Orchestrating Docker containers at scaleOrchestrating Docker containers at scale
Orchestrating Docker containers at scale
Maciej Lasyk
 
Ghost in the shell
Ghost in the shellGhost in the shell
Ghost in the shell
Maciej Lasyk
 
Node.js security
Node.js securityNode.js security
Node.js security
Maciej Lasyk
 
Monitoring with Nagios and Ganglia
Monitoring with Nagios and GangliaMonitoring with Nagios and Ganglia
Monitoring with Nagios and Ganglia
Maciej Lasyk
 
RHEL/Fedora + Docker (and SELinux)
RHEL/Fedora + Docker (and SELinux)RHEL/Fedora + Docker (and SELinux)
RHEL/Fedora + Docker (and SELinux)
Maciej Lasyk
 
High Availability (HA) Explained
High Availability (HA) ExplainedHigh Availability (HA) Explained
High Availability (HA) Explained
Maciej Lasyk
 
Shall we play a game? PL version
Shall we play a game? PL versionShall we play a game? PL version
Shall we play a game? PL version
Maciej Lasyk
 
Shall we play a game?
Shall we play a game?Shall we play a game?
Shall we play a game?
Maciej Lasyk
 

More from Maciej Lasyk (15)

Rundeck & Ansible
Rundeck & AnsibleRundeck & Ansible
Rundeck & Ansible
 
Docker 1.11
Docker 1.11Docker 1.11
Docker 1.11
 
Co powinieneś wiedzieć na temat devops?f
Co powinieneś wiedzieć na temat devops?f Co powinieneś wiedzieć na temat devops?f
Co powinieneś wiedzieć na temat devops?f
 
"Containers do not contain"
"Containers do not contain""Containers do not contain"
"Containers do not contain"
 
Continuous Security in DevOps
Continuous Security in DevOpsContinuous Security in DevOps
Continuous Security in DevOps
 
Orchestrating docker containers at scale (#DockerKRK edition)
Orchestrating docker containers at scale (#DockerKRK edition)Orchestrating docker containers at scale (#DockerKRK edition)
Orchestrating docker containers at scale (#DockerKRK edition)
 
Orchestrating docker containers at scale (PJUG edition)
Orchestrating docker containers at scale (PJUG edition)Orchestrating docker containers at scale (PJUG edition)
Orchestrating docker containers at scale (PJUG edition)
 
Orchestrating Docker containers at scale
Orchestrating Docker containers at scaleOrchestrating Docker containers at scale
Orchestrating Docker containers at scale
 
Ghost in the shell
Ghost in the shellGhost in the shell
Ghost in the shell
 
Node.js security
Node.js securityNode.js security
Node.js security
 
Monitoring with Nagios and Ganglia
Monitoring with Nagios and GangliaMonitoring with Nagios and Ganglia
Monitoring with Nagios and Ganglia
 
RHEL/Fedora + Docker (and SELinux)
RHEL/Fedora + Docker (and SELinux)RHEL/Fedora + Docker (and SELinux)
RHEL/Fedora + Docker (and SELinux)
 
High Availability (HA) Explained
High Availability (HA) ExplainedHigh Availability (HA) Explained
High Availability (HA) Explained
 
Shall we play a game? PL version
Shall we play a game? PL versionShall we play a game? PL version
Shall we play a game? PL version
 
Shall we play a game?
Shall we play a game?Shall we play a game?
Shall we play a game?
 

Recently uploaded

Figma AI Design Generator_ In-Depth Review.pdf
Figma AI Design Generator_ In-Depth Review.pdfFigma AI Design Generator_ In-Depth Review.pdf
Figma AI Design Generator_ In-Depth Review.pdf
Management Institute of Skills Development
 
Feature sql server terbaru performance.pptx
Feature sql server terbaru performance.pptxFeature sql server terbaru performance.pptx
Feature sql server terbaru performance.pptx
ssuser1915fe1
 
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
Priyanka Aash
 
CiscoIconsLibrary cours de réseau VLAN.ppt
CiscoIconsLibrary cours de réseau VLAN.pptCiscoIconsLibrary cours de réseau VLAN.ppt
CiscoIconsLibrary cours de réseau VLAN.ppt
moinahousna
 
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-InTrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
TrustArc
 
Data Integration Basics: Merging & Joining Data
Data Integration Basics: Merging & Joining DataData Integration Basics: Merging & Joining Data
Data Integration Basics: Merging & Joining Data
Safe Software
 
The Role of IoT in Australian Mobile App Development - PDF Guide
The Role of IoT in Australian Mobile App Development - PDF GuideThe Role of IoT in Australian Mobile App Development - PDF Guide
The Role of IoT in Australian Mobile App Development - PDF Guide
Shiv Technolabs
 
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
sunilverma7884
 
July Patch Tuesday
July Patch TuesdayJuly Patch Tuesday
July Patch Tuesday
Ivanti
 
IPLOOK Remote-Sensing Satellite Solution
IPLOOK Remote-Sensing Satellite SolutionIPLOOK Remote-Sensing Satellite Solution
IPLOOK Remote-Sensing Satellite Solution
IPLOOK Networks
 
find out more about the role of autonomous vehicles in facing global challenges
find out more about the role of autonomous vehicles in facing global challengesfind out more about the role of autonomous vehicles in facing global challenges
find out more about the role of autonomous vehicles in facing global challenges
huseindihon
 
Introduction-to-the-IAM-Platform-Implementation-Plan.pptx
Introduction-to-the-IAM-Platform-Implementation-Plan.pptxIntroduction-to-the-IAM-Platform-Implementation-Plan.pptx
Introduction-to-the-IAM-Platform-Implementation-Plan.pptx
313mohammedarshad
 
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
alexjohnson7307
 
How Social Media Hackers Help You to See Your Wife's Message.pdf
How Social Media Hackers Help You to See Your Wife's Message.pdfHow Social Media Hackers Help You to See Your Wife's Message.pdf
How Social Media Hackers Help You to See Your Wife's Message.pdf
HackersList
 
Tirana Tech Meetup - Agentic RAG with Milvus, Llama3 and Ollama
Tirana Tech Meetup - Agentic RAG with Milvus, Llama3 and OllamaTirana Tech Meetup - Agentic RAG with Milvus, Llama3 and Ollama
Tirana Tech Meetup - Agentic RAG with Milvus, Llama3 and Ollama
Zilliz
 
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
bhumivarma35300
 
Uncharted Together- Navigating AI's New Frontiers in Libraries
Uncharted Together- Navigating AI's New Frontiers in LibrariesUncharted Together- Navigating AI's New Frontiers in Libraries
Uncharted Together- Navigating AI's New Frontiers in Libraries
Brian Pichman
 
Evolution of iPaaS - simplify IT workloads to provide a unified view of data...
Evolution of iPaaS - simplify IT workloads to provide a unified view of  data...Evolution of iPaaS - simplify IT workloads to provide a unified view of  data...
Evolution of iPaaS - simplify IT workloads to provide a unified view of data...
Torry Harris
 
The Rise of AI in Cybersecurity How Machine Learning Will Shape Threat Detect...
The Rise of AI in Cybersecurity How Machine Learning Will Shape Threat Detect...The Rise of AI in Cybersecurity How Machine Learning Will Shape Threat Detect...
The Rise of AI in Cybersecurity How Machine Learning Will Shape Threat Detect...
digitalxplive
 
Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)
Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)
Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)
Muhammad Ali
 

Recently uploaded (20)

Figma AI Design Generator_ In-Depth Review.pdf
Figma AI Design Generator_ In-Depth Review.pdfFigma AI Design Generator_ In-Depth Review.pdf
Figma AI Design Generator_ In-Depth Review.pdf
 
Feature sql server terbaru performance.pptx
Feature sql server terbaru performance.pptxFeature sql server terbaru performance.pptx
Feature sql server terbaru performance.pptx
 
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
 
CiscoIconsLibrary cours de réseau VLAN.ppt
CiscoIconsLibrary cours de réseau VLAN.pptCiscoIconsLibrary cours de réseau VLAN.ppt
CiscoIconsLibrary cours de réseau VLAN.ppt
 
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-InTrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
 
Data Integration Basics: Merging & Joining Data
Data Integration Basics: Merging & Joining DataData Integration Basics: Merging & Joining Data
Data Integration Basics: Merging & Joining Data
 
The Role of IoT in Australian Mobile App Development - PDF Guide
The Role of IoT in Australian Mobile App Development - PDF GuideThe Role of IoT in Australian Mobile App Development - PDF Guide
The Role of IoT in Australian Mobile App Development - PDF Guide
 
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
 
July Patch Tuesday
July Patch TuesdayJuly Patch Tuesday
July Patch Tuesday
 
IPLOOK Remote-Sensing Satellite Solution
IPLOOK Remote-Sensing Satellite SolutionIPLOOK Remote-Sensing Satellite Solution
IPLOOK Remote-Sensing Satellite Solution
 
find out more about the role of autonomous vehicles in facing global challenges
find out more about the role of autonomous vehicles in facing global challengesfind out more about the role of autonomous vehicles in facing global challenges
find out more about the role of autonomous vehicles in facing global challenges
 
Introduction-to-the-IAM-Platform-Implementation-Plan.pptx
Introduction-to-the-IAM-Platform-Implementation-Plan.pptxIntroduction-to-the-IAM-Platform-Implementation-Plan.pptx
Introduction-to-the-IAM-Platform-Implementation-Plan.pptx
 
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
 
How Social Media Hackers Help You to See Your Wife's Message.pdf
How Social Media Hackers Help You to See Your Wife's Message.pdfHow Social Media Hackers Help You to See Your Wife's Message.pdf
How Social Media Hackers Help You to See Your Wife's Message.pdf
 
Tirana Tech Meetup - Agentic RAG with Milvus, Llama3 and Ollama
Tirana Tech Meetup - Agentic RAG with Milvus, Llama3 and OllamaTirana Tech Meetup - Agentic RAG with Milvus, Llama3 and Ollama
Tirana Tech Meetup - Agentic RAG with Milvus, Llama3 and Ollama
 
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
 
Uncharted Together- Navigating AI's New Frontiers in Libraries
Uncharted Together- Navigating AI's New Frontiers in LibrariesUncharted Together- Navigating AI's New Frontiers in Libraries
Uncharted Together- Navigating AI's New Frontiers in Libraries
 
Evolution of iPaaS - simplify IT workloads to provide a unified view of data...
Evolution of iPaaS - simplify IT workloads to provide a unified view of  data...Evolution of iPaaS - simplify IT workloads to provide a unified view of  data...
Evolution of iPaaS - simplify IT workloads to provide a unified view of data...
 
The Rise of AI in Cybersecurity How Machine Learning Will Shape Threat Detect...
The Rise of AI in Cybersecurity How Machine Learning Will Shape Threat Detect...The Rise of AI in Cybersecurity How Machine Learning Will Shape Threat Detect...
The Rise of AI in Cybersecurity How Machine Learning Will Shape Threat Detect...
 
Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)
Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)
Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)
 

Under the Dome (of failure driven pipeline)

  • 1. Under the dome (of failure driven pipeline) Maciej Lasyk 4developers – Warsaw 2015-04-20
  • 2. Join Fedora Infrastructure! - learn Ansible - learn Docker with Fedora Dockerfiles http://fedoraproject.org/en/join-fedora
  • 4. […] Situations like this only reinforce my deep suspicion of developers: They're often carelessly breaking things and then disappearing, leaving Operations to clean up the Mess. […] “The Phoenix Project” by Gene Kim, Kevin Behr and George Spafford
  • 8. Conway's law (1968) organizations which design systems ... are constrained to produce designs which are copies of the communication structures of these organizations http://en.wikipedia.org/wiki/Conway%27s_law
  • 9. Ruth Malan (2008) if the architecture of the system and the architecture of the organization are at odds, the architecture of the organization wins. The organizational divides are going to drive the true seams in the system. http://traceinthesand.com/blog/2008/02/13/conways-law/
  • 10. Yup, you're gut is telling truth...
  • 11. Yup, you're gut is telling truth... This will be another devops indoctrination
  • 12. Yup, you're gut is telling truth... This will be another devops indoctrination What did you expect? ;)
  • 13. Yup, you're gut is telling truth... This will be another devops indoctrination What did you expect? ;) This presentation includes gentle product placement
  • 14. Yup, you're gut is telling truth... This will be another devops indoctrination What did you expect? ;) This presentation includes gentle product placement
  • 15. DevOps Anti-Types & patterns This is a copy/paste from http://blog.matthewskelton.net/ w/my comments included Great job Matthew! Thanks!
  • 25. DevOPS ?== CAMS (culture, automation, measurement, sharing)
  • 26. DevOPS !== CAMS DevOPS === people!
  • 28. C for Culture A for Automation M for Monitoring S for Sharing
  • 31. Is there a need for change? “agile” and “cloud”: → focus on delivery → close collaboration → lightweight environment and components
  • 32. cultural change modification of a society through innovation, invention, discovery, or contact with other societies
  • 33. Dead sea effect → most talented evaporates → the residue → maintenance experts & bus factor == 1 http://brucefwebster.com/2008/04/11/the-wetware-crisis-the-dead-sea-effect/
  • 34. → talk. often. and get along → take responsibility - from beginning to the end → continuous improvement. seriously → be brave. don't be silent → it's better to be unpolite l/German than polite l/Englishman
  • 36. GTD? (getting things done) JFDI? (just fuckin' do it)
  • 37. GTD? (getting things done) JFDI? (just fuckin' do it) MFBT? (move fast, break things)
  • 38. GTD + JFDI + MFBT = FCH
  • 39. GTD + JFDI + MFBT = FCH (Fuckin' Customer Happy)
  • 41. C for Culture A for Automation M for Monitoring S for Sharing
  • 42. Automation is big for most sysadmins. We’re inherently lazy, so the idea of pushing a button and making programs work for us? Appealing. Standalone Sysadmin http://www.standalone-sysadmin.com/blog/2011/04/view-from-the-other-side/
  • 43. → it has to be simple → don't reinvent the wheel. don't fabric → automate from very beginning
  • 44. → repeatable tasks leads to automation
  • 45. → repeatable tasks leads to automation → automation leads to consistency
  • 46. → repeatable tasks leads to automation → automation leads to consistency → consistency reduces errors
  • 47. → repeatable tasks leads to automation → automation leads to consistency → consistency reduces errors → reducing errors leads to stable environment
  • 48. → repeatable tasks leads to automation → automation leads to consistency → consistency reduces errors → reducing errors leads to stable environment → stable environment leads to less unplanned work
  • 49. → repeatable tasks leads to automation → automation leads to consistency → consistency reduces errors → reducing errors leads to stable environment → stable environment leads to less unplanned work → less unplanned work leads to focus on delivery
  • 51. Short story of Anti-Type C “we don't need ops” # it's madness with paths for different users and such option as: # sudo su # sudo -i # su - # su # that is why we add variables to two places ENVIRONMENT_FILE = '/etc/environment' PROFILE_FILE = '/etc/profile' INITIAL_PATH = '/usr/local/bin:/usr/bin:/bin' # due to sudo issues (resetting PATH by /etc/sudoers) # we have to add PATH to /root/.profile as well
  • 52. Short story of Anti-Type C “we don't need ops” # it's madness with paths for different users and such option as: # sudo su # sudo -i # su - # su # that is why we add variables to two places ENVIRONMENT_FILE = '/etc/environment' PROFILE_FILE = '/etc/profile' INITIAL_PATH = '/usr/local/bin:/usr/bin:/bin' # due to sudo issues (resetting PATH by /etc/sudoers) # we have to add PATH to /root/.profile as well
  • 53. Short story of Anti-Type C “we don't need ops” Shells: → login → non-login → interactive → non – interactive
  • 54. Short story of Anti-Type C “we don't need ops” Shells: → login → non-login → interactive → non – interactive → su → sudo su: interactive, non-login, .bashrc → sudo su -: interactive, login, /etc/profile;/root/.profile;/root/.bashrc → sudo -i: interactive, login, /root/.profile;/root/.bashrc;/root/.login → sudo /bin/bash: interactive, non-login, ~/.bashrc → sudo -s: reads $SHELL and executes it
  • 55. Short story of Anti-Type C “we don't need ops” Shells: → login → non-login → interactive → non – interactive → su → sudo su: interactive, non-login, .bashrc → sudo su -: interactive, login, /etc/profile;/root/.profile;/root/.bashrc → sudo -i: interactive, login, /root/.profile;/root/.bashrc;/root/.login → sudo /bin/bash: interactive, non-login, ~/.bashrc → sudo -s: reads $SHELL and executes it
  • 56. Short story of Anti-Type C “we don't need ops” Shells: → login → non-login → interactive → non – interactive → su → sudo su: interactive, non-login, .bashrc → sudo su -: interactive, login, /etc/profile;/root/.profile;/root/.bashrc → sudo -i: interactive, login, /root/.profile;/root/.bashrc;/root/.login → sudo /bin/bash: interactive, non-login, ~/.bashrc → sudo -s: reads $SHELL and executes it
  • 57. Short story of Anti-Type C “we don't need ops” Shells: → login → non-login → interactive → non – interactive → su → sudo su: interactive, non-login, .bashrc → sudo su -: interactive, login, /etc/profile;/root/.profile;/root/.bashrc → sudo -i: interactive, login, /root/.profile;/root/.bashrc;/root/.login → sudo /bin/bash: interactive, non-login, ~/.bashrc → sudo -s: reads $SHELL and executes it
  • 58. Short story of Anti-Type C “we don't need ops” Shells: → login → non-login → interactive → non – interactive → su → sudo su: interactive, non-login, .bashrc → sudo su -: interactive, login, /etc/profile;/root/.profile;/root/.bashrc → sudo -i: interactive, login, /root/.profile;/root/.bashrc;/root/.login → sudo /bin/bash: interactive, non-login, ~/.bashrc → sudo -s: reads $SHELL and executes it
  • 59. Short story of Anti-Type C “we don't need ops” Shells: → login → non-login → interactive → non – interactive → su → sudo su: interactive, non-login, .bashrc → sudo su -: interactive, login, /etc/profile;/root/.profile;/root/.bashrc → sudo -i: interactive, login, /root/.profile;/root/.bashrc;/root/.login → sudo /bin/bash: interactive, non-login, ~/.bashrc → sudo -s: reads $SHELL and executes it
  • 60. def is_ubuntu(): return run("uname -a | grep Ubuntu | wc -l") == "1" def install_apache_fix(): if is_ubuntu(): if exists("/lib/x86_64-linux-gnu/libssl.so.0.9.8"): print "libssl.so.0.9.8 already installed - SKIPPING" else: sudo("apt-get -y install libssl0.9.8") else: #Debian if exists("/usr/lib/libssl.so.0.9.8"): print "libssl.so.0.9.8 already installed - SKIPPING" else: #download if necessary url = "http://.../libssl0.9.8_0.9.8o-squeeze14_amd64.deb" if download.sync_opt_download(_download_libssl_lock, url, store_file_path): sudo('chmod ug+x %s' % store_file_path) sudo("dpkg -i %s" % store_file_path)
  • 61. def is_ubuntu(): return run("uname -a | grep Ubuntu | wc -l") == "1" /etc/issue maybe? def install_apache_fix(): if is_ubuntu(): if exists("/lib/x86_64-linux-gnu/libssl.so.0.9.8"): print "libssl.so.0.9.8 already installed - SKIPPING" else: sudo("apt-get -y install libssl0.9.8") else: #Debian if exists("/usr/lib/libssl.so.0.9.8"): print "libssl.so.0.9.8 already installed - SKIPPING" else: #download if necessary url = "http://.../libssl0.9.8_0.9.8o-squeeze14_amd64.deb" if download.sync_opt_download(_download_libssl_lock, url, store_file_path): sudo('chmod ug+x %s' % store_file_path) sudo("dpkg -i %s" % store_file_path)
  • 62. def is_ubuntu(): return run("uname -a | grep Ubuntu | wc -l") == "1" def install_apache_fix(): if is_ubuntu(): if exists("/lib/x86_64-linux-gnu/libssl.so.0.9.8"): print "libssl.so.0.9.8 already installed - SKIPPING" else: sudo("apt-get -y install libssl0.9.8") else: #Debian if exists("/usr/lib/libssl.so.0.9.8"): print "libssl.so.0.9.8 already installed - SKIPPING" else: #download if necessary url = "http://.../libssl0.9.8_0.9.8o-squeeze14_amd64.deb" if download.sync_opt_download(_download_libssl_lock, url, store_file_path): sudo('chmod ug+x %s' % store_file_path) sudo("dpkg -i %s" % store_file_path)
  • 63. def is_ubuntu(): return run("uname -a | grep Ubuntu | wc -l") == "1" def install_apache_fix(): if is_ubuntu(): if exists("/lib/x86_64-linux-gnu/libssl.so.0.9.8"): ldconfig maybe? print "libssl.so.0.9.8 already installed - SKIPPING" else: sudo("apt-get -y install libssl0.9.8") else: #Debian if exists("/usr/lib/libssl.so.0.9.8"): print "libssl.so.0.9.8 already installed - SKIPPING" else: #download if necessary url = "http://.../libssl0.9.8_0.9.8o-squeeze14_amd64.deb" if download.sync_opt_download(_download_libssl_lock, url, store_file_path): sudo('chmod ug+x %s' % store_file_path) sudo("dpkg -i %s" % store_file_path)
  • 64. def is_ubuntu(): return run("uname -a | grep Ubuntu | wc -l") == "1" def install_apache_fix(): if is_ubuntu(): if exists("/lib/x86_64-linux-gnu/libssl.so.0.9.8"): print "libssl.so.0.9.8 already installed - SKIPPING" else: sudo("apt-get -y install libssl0.9.8") else: #Debian if exists("/usr/lib/libssl.so.0.9.8"): print "libssl.so.0.9.8 already installed - SKIPPING" else: #download if necessary url = "http://.../libssl0.9.8_0.9.8o-squeeze14_amd64.deb" if download.sync_opt_download(_download_libssl_lock, url, store_file_path): sudo('chmod ug+x %s' % store_file_path) sudo("dpkg -i %s" % store_file_path)
  • 65. def is_ubuntu(): return run("uname -a | grep Ubuntu | wc -l") == "1" def install_apache_fix(): if is_ubuntu(): if exists("/lib/x86_64-linux-gnu/libssl.so.0.9.8"): print "libssl.so.0.9.8 already installed - SKIPPING" else: sudo("apt-get -y install libssl0.9.8") else: #Debian What about RHEL, Fedora, Slackware, Gentoo? if exists("/usr/lib/libssl.so.0.9.8"): print "libssl.so.0.9.8 already installed - SKIPPING" else: #downl. if necessary So whole this is for particular distro version? url = "http://.../libssl0.9.8_0.9.8o-squeeze14_amd64.deb" if download.sync_opt_download(_download_libssl_lock, url, store_file_path): sudo('chmod ug+x %s' % store_file_path) sudo("dpkg -i %s" % store_file_path)
  • 66. def is_ubuntu(): return run("uname -a | grep Ubuntu | wc -l") == "1" def install_apache_fix(): if is_ubuntu(): if exists("/lib/x86_64-linux-gnu/libssl.so.0.9.8"): print "libssl.so.0.9.8 already installed - SKIPPING" else: sudo("apt-get -y install libssl0.9.8") else: #Debian if exists("/usr/lib/libssl.so.0.9.8"): print "libssl.so.0.9.8 already installed - SKIPPING" else: #downl. if necessary url = "http://libssl0.9.8_0.9.8o-squeeze14_amd64.deb" if download.sync_opt_download(_download_libssl_lock, url, store_file_path): sudo('chmod ug+x %s' % store_file_path) sudo("dpkg -i %s" % store_file_path)
  • 67. def is_ubuntu(): return run("uname -a | grep Ubuntu | wc -l") == "1" def install_apache_fix(): if is_ubuntu(): if exists("/lib/x86_64-linux-gnu/libssl.so.0.9.8"): print "libssl.so.0.9.8 already installed - SKIPPING" else: sudo("apt-get -y install libssl0.9.8") else: #Debian if exists("/usr/lib/libssl.so.0.9.8"): print "libssl.so.0.9.8 already installed - SKIPPING" else: #downl. if necessary url = "http://libssl0.9.8_0.9.8o-squeeze14_amd64.deb" if download.sync_opt_download(_download_libssl_lock, url, store_file_path): sudo('chmod ug+x %s' % store_file_path) # declarative madness sudo("dpkg -i %s" % store_file_path)
  • 69. Imperativeness vs declarativeness def configure(dst_dir, config_properties, installer_file): _copy_conf_file(dst_dir, properties) def _copy_conf_file(dst_dir, properties): sudo("cp %s %s" % (srcConfigPath, targetConfigPath)) change_directory_owner(targetConfigPath) sudo('chmod ug+x %s' % store_file_path) - name: configure this hosts: all tasks: - name: copy conf file file: > src={{ some_source }} dest={{ some_destination }} perms=0750
  • 70. Imperativeness vs declarativeness def configure(dst_dir, config_properties, installer_file): _copy_conf_file(dst_dir, properties) def _copy_conf_file(dst_dir, properties): sudo("cp %s %s" % (srcConfigPath, targetConfigPath)) change_directory_owner(targetConfigPath) sudo('chmod ug+x %s' % store_file_path) - name: configure this hosts: all tasks: - name: copy conf file file: > src={{ some_source }} dest={{ some_destination }} perms=0750
  • 73. → flat learning curve → doesn't required additional resources
  • 74. → flat learning curve → doesn't required additional resources → fit for maintenance jobs / procedures
  • 75. → flat learning curve → doesn't required additional resources → fit for maintenance jobs / procedures → great for any containers as non-daemon
  • 76. → flat learning curve → doesn't required additional resources → fit for maintenance jobs / procedures → great for any containers as non-daemon → deals with “deployment specs”
  • 77. → flat learning curve → doesn't required additional resources → fit for maintenance jobs / procedures → great for any containers as non-daemon → deals with “deployment specs” → might be easily adopted as universal language
  • 80. →selinux enforcing i -rw-r--r--. stash stash unconfined_u:object_r:mysqld_db_t:s0 authorized_keys →/etc/ssh/sshd_config && /etc/network/interfaces → iptables-save nope? → broken _netfs ?
  • 84. What if... → ./configure && make && make install → .zip → Dev & Ops have 2 different build & installation methods? Plz.. → pkg repos (or Nexus) → use fpm for creating pkgs if needed (demo)
  • 85. C for Culture A for Automation M for Monitoring S for Sharing
  • 87. → make developers create monitoring → find yourself between RRD and InfluxDB → will product team be able to query your monitoring DB? → Etsy case (Ganglia / Graphite)
  • 88. → make developers create monitoring → find yourself between RRD and InfluxDB → will product team be able to query your monitoring DB? → Etsy case (Ganglia / Graphite)
  • 89. → make developers create monitoring → find yourself between RRD and InfluxDB → will product team be able to query your monitoring DB? → Etsy case (Ganglia / Graphite)
  • 90. → make developers create monitoring → find yourself between RRD and InfluxDB → will product team be able to query your monitoring DB? → Etsy case (Ganglia / Graphite)
  • 91. C for Culture A for Automation M for Monitoring S for Sharing
  • 92. → learn on OPS mistakes → Major Incident Reports – source of improvement → Learn developers about change management → Make CM an easy process. Use simple tools.
  • 93. → learn on OPS mistakes → Major Incident Reports – source of improvement → Learn developers about change management → Make CM an easy process. Use simple tools.
  • 94. → learn on OPS mistakes → Major Incident Reports – source of improvement → Learn developers about change management → Make CM an easy process. Use simple tools.
  • 95. → learn on OPS mistakes → Major Incident Reports – source of improvement → Learn developers about change management → Make CM an easy process. Use simple tools.
  • 96. Let's arch the infrastructure
  • 97. Addressing the space → VLSM → DHCP & DDNS → KISS: flat networks! → stop /24!
  • 98. Addressing the space → VLSM → DHCP & DDNS → KISS: flat networks! → stop /24!
  • 99. Addressing the space → VLSM → DHCP & DDNS → KISS: flat networks! → stop /24!
  • 100. Addressing the space → VLSM → DHCP & DDNS → KISS: flat networks! → stop /24!
  • 102. What about DNS? → BIND roxx (views etc) → KISS: maybe decentralized w/Ansible?
  • 103. view "internal-view" { match-clients { internal; }; recursion yes; zone "lasyk.info" IN { type master; file "internal.lasyk.info.conf"; allow-transfer { any; } }; view "external-view" { match-clients { any; }; recursion no; zone "lasyk.info" IN { type master; file "external.lasyk.info.conf"; allow-transfer { none; }; };
  • 104. view "internal-view" { match-clients { internal; }; recursion yes; zone "lasyk.info" IN { type master; file "internal.lasyk.info.conf"; allow-transfer { any; } }; view "external-view" { match-clients { any; }; recursion no; zone "lasyk.info" IN { type master; file "external.lasyk.info.conf"; allow-transfer { none; }; };
  • 106. Linux Containers = namespaces + cgroups + storage Linux containers equation
  • 107. Control Groups provide a mechanism for aggregating/partitioning sets of tasks, and all their future children, into hierarchical groups with specialized behavior control groups (cgroups)
  • 108. →grouping processes →allocating resources to particular groups →memory →network →CPU →storage bandwidth (I/O throttling) →device whitelisting control groups (cgroups)
  • 109. →grouping processes →allocating resources to particular groups →memory →network →CPU →storage bandwidth (I/O throttling) →device whitelisting control groups (cgroups)
  • 110. →grouping processes →allocating resources to particular groups →memory →network →CPU →storage bandwidth (I/O throttling) →device whitelisting control groups (cgroups)
  • 111. →grouping processes →allocating resources to particular groups →memory →network →CPU →storage bandwidth (I/O throttling) →device whitelisting control groups (cgroups)
  • 112. →grouping processes →allocating resources to particular groups →memory →network →CPU →storage bandwidth (I/O throttling) →device whitelisting control groups (cgroups)
  • 113. →grouping processes →allocating resources to particular groups →memory →network →CPU →storage bandwidth (I/O throttling) →device whitelisting control groups (cgroups)
  • 114. →grouping processes →allocating resources to particular groups →memory →network →CPU →storage bandwidth (I/O throttling) →device whitelisting control groups (cgroups)
  • 116. Providing a unique views of the system for processes. → PID – PIDs isolation → NET – network isolation (via virt-ifaces; demo) → IPC – won't use this → MNT – chroot like; deals w/mountpoints → UTS – deals w/hostname Kernel Namespaces
  • 117. Providing a unique views of the system for processes. → PID – PIDs isolation → NET – network isolation (via virt-ifaces; demo) → IPC – won't use this → MNT – chroot like; deals w/mountpoints → UTS – deals w/hostname Kernel Namespaces
  • 118. Providing a unique views of the system for processes. → PID – PIDs isolation → NET – network isolation (via virt-ifaces; demo) → IPC – won't use this → MNT – chroot like; deals w/mountpoints → UTS – deals w/hostname Kernel Namespaces
  • 119. Providing a unique views of the system for processes. → PID – PIDs isolation → NET – network isolation (via virt-ifaces; demo) → IPC – won't use this → MNT – chroot like; deals w/mountpoints → UTS – deals w/hostname Kernel Namespaces
  • 120. Providing a unique views of the system for processes. → PID – PIDs isolation → NET – network isolation (via virt-ifaces; demo) → IPC – won't use this → MNT – chroot like; deals w/mountpoints → UTS – deals w/hostname Kernel Namespaces
  • 121. Providing a unique views of the system for processes. → PID – PIDs isolation → NET – network isolation (via virt-ifaces; demo) → IPC – won't use this → MNT – chroot like; deals w/mountpoints → UTS – deals w/hostname Kernel Namespaces
  • 122. Providing a unique views of the system for processes. → PID – PIDs isolation → NET – network isolation (via virt-ifaces; demo) → IPC – won't use this → MNT – chroot like; deals w/mountpoints → UTS – deals w/hostname Kernel Namespaces
  • 124. → hell fast (you'll see) → page cache sharing → finally in upstream kernel (in rhel from 7.2) → finally supported by docker (-s overlay) → SELinux not there yet (but will be) OverlayFS
  • 125. → hell fast (you'll see) → page cache sharing → finally in upstream kernel (in rhel from 7.2) → finally supported by docker (-s overlay) → SELinux not there yet (but will be) OverlayFS
  • 126. → hell fast (you'll see) → page cache sharing → finally in upstream kernel (in rhel from 7.2) → finally supported by docker (-s overlay) → SELinux not there yet (but will be) OverlayFS
  • 127. → hell fast (you'll see) → page cache sharing → finally in upstream kernel (in rhel from 7.2) → finally supported by docker (-s overlay) → SELinux not there yet (but will be) OverlayFS
  • 128. → hell fast (you'll see) → page cache sharing → finally in upstream kernel (in rhel from 7.2) → finally supported by docker (-s overlay) → SELinux not there yet (but will be) OverlayFS
  • 132. Developers' envs? → use containers! → configure cgroups → use LXC / LXC Web Panel → use Ansible for spinning up anything!
  • 133. Developers' envs? → use containers! → configure cgroups → use LXC / LXC Web Panel → use Ansible for spinning up anything!
  • 134. Developers' envs? → use containers! → configure cgroups → use LXC / LXC Web Panel → use Ansible for spinning up anything!
  • 135. Developers' envs? → use containers! → configure cgroups → use LXC / LXC Web Panel → use Ansible for spinning up anything!
  • 137. Containers embraces granularity → microservices!
  • 138. Containers embraces granularity → microservices! Watch out for microservices architecture, or...
  • 139. Containers embraces granularity → microservices! Watch out for microservices architecture, or...
  • 142. Who knows FHS? → 'temp' – what it consist?
  • 143. Who knows FHS? → 'temp' – what it consist? → actually: “This Entity Must Persist” ;)
  • 144. Who knows FHS? → 'temp' – what it consist? → actually: “This Entity Must Persist” ;) → Define your FHS!
  • 145. Mikado Method for the win! → set a goal → experiment → visualize → rollback
  • 146. Mikado Method for the win! → set a goal → experiment → visualize → rollback
  • 147. Mikado Method for the win! → set a goal → experiment → visualize → rollback
  • 148. Mikado Method for the win! → set a goal → experiment → visualize → rollback
  • 149. Mikado Method for the win! → before any work and rollbacks.. → remember: monitoring & tests are your friends! → think about testing strategy – think heatmaps!
  • 150. Ansible & infra layers Layer 1: bare metal, Layer 2: VM Layer 3: container Networking Hypervisor + VM provisioning Storage Networking Container's engine & provisioning Application build Application env Network interfaces Storage mounts Resources allocation repo1 repo2 repo3 Much simpler w/one, flat network (for small envs)!
  • 151. Ansible & infra layers Layer 1: bare metal, Layer 2: VM Layer 3: container Networking Hypervisor + VM provisioning Storage Networking Container's engine & provisioning Application build Application env Network interfaces Storage mounts Resources allocation repo1 repo2 repo3 Much simpler w/one, flat network (for small envs)! repo2 Layer 2: VM Networking Container's engine & provisioning repo2
  • 152. Ansible & infra layers Layer 1: bare metal, Layer 2: VM Layer 3: container Networking Hypervisor + VM provisioning Storage Networking Container's engine & provisioning Application build Application env Network interfaces Storage mounts Resources allocation repo1 repo2 repo3 Much simpler w/one, flat network (for small envs)! repo2 Layer 2: VM Networking Container's engine & provisioning repo2 Network interfaces Storage mounts repo2
  • 153. Ansible & infra layers Layer 1: bare metal, Layer 2: VM Layer 3: container Networking Hypervisor + VM provisioning Storage Networking Container's engine & provisioning Application build Application env Network interfaces Storage mounts Resources allocation repo1 repo2 repo3 Much simpler w/one, flat network (for small envs)! repo2 Layer 2: VM Networking Container's engine & provisioning repo2 Network interfaces Storage mounts repo2 Layer 3: container Application build Application env repo3
  • 154. Ansible & infra layers Layer 1: bare metal, Layer 2: VM Layer 3: container Networking Hypervisor + VM provisioning Storage Networking Container's engine & provisioning Application build Application env Network interfaces Storage mounts Resources allocation repo1 repo2 repo3 Much simpler w/one, flat network (for small envs)! repo2 Layer 2: VM Networking Container's engine & provisioning repo2 Network interfaces Storage mounts repo2 Layer 3: container Application build Application env repo3 Resources allocation repo3
  • 155. Ansible & infra layers Layer 1: bare metal, Layer 2: VM Layer 3: container Networking Hypervisor + VM provisioning Storage Networking Container's engine & provisioning Application build Application env Network interfaces Storage mounts Resources allocation repo1 repo2 repo3 Much simpler w/one, flat network (for small envs)!
  • 156. → automated service discovery and registration framework → ideal for SOA architectures → ideal for continuous integration & delivery → solves “works on my machine” problem SmartStack
  • 157. → automated service discovery and registration framework → ideal for SOA architectures → ideal for continuous integration & delivery → solves “works on my machine” problem SmartStack haproxy + nerve + synapse + zookeper = smartstack
  • 158. Synapse → discovery service (via zookeeper or etcd) → installed on every node → writes haproxy configuration → application doesn't have to be aware of this → works same on bare / VM / docker → https://github.com/airbnb/nerve SmartStack
  • 160. Nerve → health checks (pluggable) → register service info to zookeper (or etcd) → https://github.com/airbnb/synapse SmartStack
  • 164. Smartstack + Docker = <3 but also remember about Consul (come to #dockerkrk 2 meetup!)
  • 167. Archaeological workshop → nmap, tcpdump, lsof, strace, sysdig, sar → cgroups throttling on-the-fly Do we have time for demo?
  • 168. Hardware: disks? → RAID5 vs RAID10 → Howto RAID over 1 disk ;) → Cheap SSD drives?
  • 169. Hardware: disks? → RAID5 vs RAID10 → Howto RAID over 1 disk ;) → Cheap SSD drives?
  • 170. Hardware: disks? → RAID5 vs RAID10 → Howto RAID over 1 disk ;) → Cheap SSD drives?
  • 172. Why use LVM? → indexation (capacity, inodes check) → capacity planning / iops per mount
  • 173. Under the dome (of failure driven pipeline) Maciej Lasyk 4developers – Warsaw 2015-04-20