SlideShare a Scribd company logo
1 of 86
Download to read offline
Advanced technic
for OS upgrading in 3 minutes
Deployment strategy for next generation
self.introduce
=>
{
name: “SHIBATA Hiroshi”,
nickname: “hsbt”,
title: “Chief engineer at GMO Pepabo, Inc.”,
commit_bits: [“ruby”, “rake”, “rubygems”, “rdoc”, “tdiary”,
“hiki”, “railsgirls”, “railsgirls-jp”, “jenkins”],
sites: [“ruby-lang.org”, “rubyci.com”, “railsgirls.com”,
“railsgirls.jp”],
}
I’m from Asakusa.rb
Asakusa.rb is one of the most active meet-ups in Tokyo, Japan.
@a_matsuda (Ruby/Rails committer, RubyKaigi organizer)
@kakutani (RubyKaigi organizer)
@ko1 (Ruby committer)
@takkanm (Ruby/Rails programmer)
@gunjisatoshi (Rubyist Magazine editor)
@hsbt (Me!)
もっと
おもしろく
できる
2014/11/xx
2014/11/xx …
“しばたさん… ちょっといいですか”
“100倍の話しか聞きませんよ”
CTO: antipop
2014/11/xx …
“ある意味 100 倍すね…3ヶ月後に某サービスで
CM打つことになりまして…放映までにサービス
をバーン!!1としてもらいたいんですよ”
“(うわー)”
CTO: antipop
Our service status at 2014/11
• Simply Rails Service with IaaS
• 6 application servers
• To use capistrano 2 for deployment
• Mixed worker and application role
• Unknown role server like handled only POST request server
Our service issue
Do scale-out
Do scale-out with automation!
Do scale-out with rapid automation!!!
Do scale-out with extremely rapid automation!!!1
Team member
hsbt: Director, fullstack Programmer
udzura: fullstack Programmer
yano3: fullstack Programmer
Do scale-out
Web operation is manual instructions
• We have been created OS Image called “Golden Image” from
running server
• Web operations such as os configuration and instances launch
are manual instruction.
• Working time is about 4-6 hours
• We say it “Tanpopo works…”
• It’s blocker for scale-out largely.
puppet
Fixed all of puppet manifests
• It based on Scientific Linux 6.x
• Some manifest is broken…
• Service developers didn’t use puppet for production
At first, We fixed all of manifests and enabled to deploy to
production environments.
% ls **/*.pp | xargs wc -l | tail -1
5546 total
Setting up puppetmasterd
• We choice master/agent model
• It’s large scaled architecture because we didn’t need to deploy
puppet manifests each servers.
• We already have puppetmasterd manifests written by puppet
using passenger named rails application server.
https://docs.puppetlabs.com/guides/passenger.html
Use provision tool for scale-out
• Launch instance from raw linux image that it’s not customized
with our service.
• Deploy rails application with basic instructions.
• Test with single instance
• Attach instance to load balancer
It’s puppet
work, not
tanpopo work
Check Point 0
We need to understand our server configuration via “CODE”
Use provision tool like puppet/chef/ansible etc etc…
Bootstrap time = 4-6 hours
Do scale-out
with automation
Concerns of bootstrap instructions
Typical scenario of server set-up for scale out.
• OS boot
• OS Configuration
• Provisioning with puppet/chef
• Setting up to capistrano
• Deploy rails application
• Added load balancer (= Service in)
No ssh
We added “No SSH” into our rule of Web operation
Background of “No SSH”
In large scale service, 1 instance is like a “1 process” in Unix
environments.
We didn’t attach process using gdb usually.
• We don’t access instance via ssh
We didn’t modify program variables in memory usually.
• We don’t modify configuration on instance
We can handle instance/process status using signal/api only.
We have awesome operation tools
• clout-init
• packer
• consul
• IaaS api/cli
cloud-init
What’s cloud-init
“Cloud-init is the defacto multi-distribution package that handles
early initialization of a cloud instance.”
https://cloudinit.readthedocs.org/en/latest/
• We(and you) already used cloud-init for customizing to OS
configuration at initialization process on IaaS
• It has few documents for our use-case…
Tuning tools(cloud-init)
We only use OS configuration. Do not use “run_cmd”
#cloud-config
repo_update: true
repo_upgrade: none
packages:
- git
- curl
- unzip
users:
- default
locale: ja_JP.UTF-8
timezone: Asia/Tokyo
Do not use hostname/ip dependency
We discarded dependencies of hostname and ip address.
Use API of IaaS for our use-case.
config.ru:
10: defaults = `hostname`.start_with?('job') ?
config/database.yml:
37: if `hostname`.start_with?(‘solr')
config/unicorn.conf:
6: if `hostname`.start_with?('job')
Image creation with itself
We use IaaS API for image creation with cloud-init userdata.
We can create OS Image using cloud-init and provisioned puppet
when boot time of instance.
puppet agent -t
rm -rf /var/lib/cloud/sem /var/lib/cloud/instances/*
aws ec2 create-image --instance-id `cat /var/lib/cloud/data/instance-id` --name
www_base_`date +%Y%m%d%H%M`
Rails
Upgrading Rails 4
• I am very good at “Rails Upgrading”
• Deploying in Production was performed with @amacou
% g show c1d698e
commit c1d698ec444df1c137a301e01f59e659593ecf76
Author: amacou <amacou.abf@gmail.com>
Date: Mon Dec 15 18:22:34 2014 +0900
Revert "Revert "Revert "Revert "[WIP] Rails 4.1.X へのアップグレード""""
Check point 1
• DO NOT change main architecture
• Write real-world instructions
• Pick instruction for automation
• DO automation
Bootstrap time = 1hours
Do scale-out
with rapid
automation
capistrano3
What’s new for capistrano3
“A remote server automation and deployment tool written in
Ruby.”
http://capistranorb.com/
Example of Capfile:
We rewrite own capstrano2 tasks to capistrano3 convention
require 'capistrano/bundler'
require 'capistrano/rails/assets'
require 'capistrano3/unicorn'
require 'capistrano/banner'
require 'capistrano/npm'
require 'slackistrano'
Rails bundle
Bundled package of Rails application
Prepared to standalone Rails application with rubygems and
precompiled assets
Part of capistrano tasks:
$ bundle exec cap production archive_project ROLES=build
desc "Create a tarball that is set up for deploy"
task :archive_project =>
[:ensure_directories, :checkout_local, :bundle, :npm_install, :bower_install,
:asset_precompile, :create_tarball, :upload_tarball, :cleanup_dirs]
Distributed rails package
build server
rails bundle
object
storage
(s3)
application
server
application
server
application
server
application
server
# 最新のアプリケーションの取得
RELEASE=`date +%Y%m%d%H%M`
ARCHIVE_ROOT=‘s3://rails-application-bundle/production/'
ARCHIVE_FILE=$(
aws s3 ls $ARCHIVE_ROOT | grep -E 'application-.*.tgz' | awk '{print $4}' | sort -r | head -n1
)
aws s3 cp "${ARCHIVE_ROOT}${ARCHIVE_FILE}" /tmp/rails-application.tar.gz
# cap setup 相当を実行
(snip)
# chown を実行
(snip)
We extracted rails bundle when instance creates self image with
clout-init.
Integration of image creation
consul
Nagios
We used nagios for monitoring to service and instance status.
But we have following issue:
• nagios don’t support dynamic scaled architecture
• Complex syntax and configuration
We decided to use nagios for service monitoring like http status
with load balancer only.
consul + consul-alert
We use consul and consul-alerts for
process monitoring.
https://github.com/hashicorp/consul
https://github.com/AcalephStorage/
consul-alerts
It provided to discover to new
instances automatically and alert
mechanism with slack integration.
mackerel
munin
We used munin for resource monitoring
But munin doesn’t support dynamic scaled architecture. We
decided to use mackerel.io instead of munin.
Mackerel
“A Revolutionary New Kind ofApplication Performance
Management. Realize the potential in Cloud Computingby
managing cloud servers through “roles””
https://mackerel.io
Auto join and leave with mackrel
You can added instance to role(server group) on mackerel with
mackerel.con
You can remove instance from mackerel when instance shutdown.
We added following script to initscripts
※ It’s official support now http://blog-ja.mackerel.io/entry/2015/07/31/105300
[user@www ~]$ cat /etc/mackerel-agent/mackerel-agent.conf
apikey = “your_api_key”
role = [ "service:web" ]
curl -s -X POST -H 'Content-type: application/json' -H ‘X-Api-Key:api_key' 
https://mackerel.io/api/v0/hosts/`cat /var/lib/mackerel-agent/id`/retire
fluentd
access_log aggregator with td-agent
We need to collect
access log of all
servers with scale-out.
<match nginx.**>
type forward
send_timeout 60s
recover_wait 10s
heartbeat_interval 1s
phi_threshold 16
hard_timeout 60s
<server>
name aggregate.server
host aggregate.server
weight 100
</server>
<server>
name aggregate2.server
host aggregate2.server
weight 100
standby
</server>
</match>
<match nginx.access.*>
type copy
<store>
type file
(snip)
</store>
<store>
type tdlog
apikey api_key
auto_create_table true
database database
table access
use_ssl true
flush_interval 120
buffer_path /data/tmp/td-agent-td/access
</store>
</match>
thor
What’s thor
“Thor is a toolkit for building powerful command-line interfaces.
It is used in Bundler, Vagrant, Rails and others.”
http://whatisthor.com/
module AwesomeTool
class Cli < Thor
class_option :verbose, type: :boolean, default: false
desc 'instances [COMMAND]', ‘Desc’
subcommand('instances', Instances)
end
end
module AwesomeTool
class Instances < Thor
desc 'launch', ‘Desc'
method_option :count, type: :numeric, aliases: "-c", default: 1
def launch
(snip)
end
end
end
We can scale out with one command via our cli tool
All of web operations should be implement by command line tools
Scale out with cli command
$ some_cli_tool instances launch -c …
$ some_cli_tool mackerel fixrole
$ some_cli_tool scale up
$ some_cli_tool deploy blue-green
Check point 2
• Use cloud-oriented architecture
• Adopt next generation architecture aggressively
• Web operations should be provided from programs
Bootstrap time = 20-30min
CM
Do scale-out
with extremely
rapid automation
Concerns of bootstrap time
Typical scenario of server set-up for scale out.
• OS boot
• OS Configuration
• Provisioning with puppet/chef
• Setting up to capistrano
• Deploy rails application
• Added load balancer (= Service in)
We need to enhance to bootstrap time extremely.
Concerns of bootstrap time
Slow operation
• OS boot
• Provisioning with puppet/chef
• Deploy rails application
Fast operation
• OS Configuration
• Setting up to capistrano
• Added load balancer (=
Service in)
Check point of Image creation
Slow operation
• OS boot
• Provisioning with puppet/chef
• Deploy rails application
Fast operation
• OS Configuration
• Setting up to capistrano
• Added load balancer (=
Service in)
Step1
Step2
2 phase strategy
• Official OS image
• Provided from platform like AWS, Azure, GCP, OpenStack…
• Minimal image(phase 1)
• Network, User, Package configuration
• Installed puppet/chef and platform cli-tools.
• Role specified(phase 2)
• Only boot OS and Rails application
Packer
After packer age
I couldn’t understand use-case of packer. Is it Provision tool?
Deployment tool?
I think “Learning”
inside image creation with Packer
• Packer configuration
• JSON format
• select instance size, block volume,
• cloud-init
• Basic configuration of OS
• only default module of cloud-init
• provisioner
• shell script :)
minimal image
cloud-init provisioner
#cloud-config
repo_update: true
repo_upgrade: none
packages:
- git
- curl
- unzip
users:
- default
locale: ja_JP.UTF-8
timezone: Asia/Tokyo
rpm -ivh http://yum.puppetlabs.com/
puppetlabs-release-el-7.noarch.rpm
yum -y update
yum -y install puppet
yum -y install python-pip
pip install awscli
sed -i 's/name: centos/name: cloud-user/' /etc/
cloud/cloud.cfg
echo 'preserve_hostname: true' >> /etc/cloud/
cloud.cfg
www image
cloud-init provisioner
#cloud-config
preserve_hostname: false
puppet agent -t
set -e
monit stop unicorn
/usr/local/bin/globefish -w
rm -rf /var/www/deploys/minne/releases/*
rm -f /var/www/deploys/minne/current
# tar xf するだけで動くRails アプリケーションを取得
(snip)
# mackerel のホスト設定が packer 実行時のものとかぶらないように初期化
rm /var/lib/mackerel-agent/id
# cloud-init をもう一度動かすようにする準備
rm -rf /var/lib/cloud/sem /var/lib/cloud/instances/*
Integration tests with Packer
We can tests results of Packer running. (Impl by @udzura)
"provisioners": [
(snip)
{
"type": "shell",
"script": "{{user `project_root`}}packer/minimal/provisioners/run-serverspec.sh",
"execute_command": "{{ .Vars }} sudo -E sh '{{ .Path }}'"
}
]
yum -y -q install rubygem-bundler
cd /tmp/serverspec
bundle install --path vendor/bundle
bundle exec rake spec
packer configuration
run-serverspec.sh
We created cli tool with thor
We can run packer over thor code with advanced options.
$ some_cli_tool ami build-minimal
$ some_cli_tool ami build-www
$ some_cli_tool ami build-www —init
$ some_cli_tool ami build-www -a ami-id
module SomeCliTool
class Ami < Thor
method_option :ami_id, type: :string, aliases: "-a"
method_option :init, type: :boolean
desc 'build-www', 'wwwの最新イメージをビルドします'
def build_www
…
end
end
end
Infra CI
What's Infra CI
We test server status such as lists of installed packages, running
processes and configuration details continuously.
Puppet + Drone CI(with Docker) + Serverspec = WIN
We can refactoring puppet manifests aggressively.
Drone CI
“CONTINUOUS INTEGRATION FOR GITHUB AND BITBUCKET THAT
MONITORS YOUR CODE FOR BUGS”
https://drone.io/
We use Drone CI on our Openstack platform named “nyah”
Serverspec
“RSpec tests for your servers configured
by CFEngine, Puppet, Ansible, Itamae or anything else.”
http://serverspec.org/
% rake -T
rake mtest # Run mruby-mtest
rake spec # Run serverspec code for all
rake spec:base # Run serverspec code for base.minne.pbdev
rake spec:batch # Run serverspec code for batch.minne.pbdev
rake spec:db:master # Run serverspec code for master db
rake spec:db:slave # Run serverspec code for slave db
rake spec:gateway # Run serverspec code for gateway.minne.pbdev
(snip)
Refactoring puppet manifets
We replaced “puppetserver”
written by Clojure.
We enabled future-parser. We
fixed all of warnings and
syntax error.
We added and removed
manifests everyday.
CentOS 7
Switch Scientific Linux 6 to CentOS 7
We can refactoring to puppet manifests with infra CI.
We added case-condition for SL6 and Centos7
if $::operatingsystemmajrelease >= 6 {
$curl_devel = 'libcurl-devel'
} else {
$curl_devel = 'curl-devel'
}
How to test instance behavior
We need to guarantee http
status from instance response.
We removed package version
control from our concerns.
Check point 3
• Packer is best tool of Image creation
• Infra CI is over evaluation phase
• You can refactor provision manifests now
Bootstrap time = 3-5min
Blue-Green
Deployment
What’s Blue-Green Deployment
http://martinfowler.com/bliki/BlueGreenDeployment.html
Instructions of Blue-Green deployment
Basic concept is following instructions.
1. Launch instances using OS imaged created from Packer
2. Wait to change “InService” status
3. Terminate old instances
That’s all!!1
Dynamic upstream with load balancer
ELB
• Provided by AWS, It’s best choice for B-G deployment
• Can handle only AWS instances
nginx + consul-template
• Change upstream directive used consul and consul-template
ngx_mruby
• Change upstream directive used mruby
Slack integration of consul-template
Example code of thor
old_instances = running_instances(load_balancer_name)
invoke Instances, [:launch], options.merge(:count => old_instances.count)
catch(:in_service) do
sleep_time = 60
loop do
instances = running_instances(load_balancer_name)
throw(:in_service) if (instances.count == old_instances.count * 2) &&
instances.all?{|i| i.status == 'InService'}
sleep sleep_time
sleep_time = [sleep_time - 10, 10].max
end
end
old_instances.each do |oi|
oi.delete
end
Check point 4
We can upgrade OS version
in 3 minutes
\ http://pepabo.com/recruit/ /
Next step of our stage
• Automated all of test with image creation and launching
• Flexible architecture includes mutable roles
• Sync deployment with image creation cycle
• Use Docker
http://euphrates.jp/1859898
Enjoy
Pythagoraswich
Infrastracture

More Related Content

What's hot

Webconf nodejs-production-architecture
Webconf nodejs-production-architectureWebconf nodejs-production-architecture
Webconf nodejs-production-architecture
Ben Lin
 
Deploying Plack Web Applications: OSCON 2011
Deploying Plack Web Applications: OSCON 2011Deploying Plack Web Applications: OSCON 2011
Deploying Plack Web Applications: OSCON 2011
Tatsuhiko Miyagawa
 

What's hot (20)

Webconf nodejs-production-architecture
Webconf nodejs-production-architectureWebconf nodejs-production-architecture
Webconf nodejs-production-architecture
 
Running JavaScript Efficiently in a Java World
Running JavaScript Efficiently in a Java WorldRunning JavaScript Efficiently in a Java World
Running JavaScript Efficiently in a Java World
 
Introduction to node js - From "hello world" to deploying on azure
Introduction to node js - From "hello world" to deploying on azureIntroduction to node js - From "hello world" to deploying on azure
Introduction to node js - From "hello world" to deploying on azure
 
Ansible+docker (highload++2015)
Ansible+docker (highload++2015)Ansible+docker (highload++2015)
Ansible+docker (highload++2015)
 
Managing Your Cisco Datacenter Network with Ansible
Managing Your Cisco Datacenter Network with AnsibleManaging Your Cisco Datacenter Network with Ansible
Managing Your Cisco Datacenter Network with Ansible
 
JavaScript Event Loop
JavaScript Event LoopJavaScript Event Loop
JavaScript Event Loop
 
Node js
Node jsNode js
Node js
 
Openstack study-nova-02
Openstack study-nova-02Openstack study-nova-02
Openstack study-nova-02
 
Understanding the Single Thread Event Loop
Understanding the Single Thread Event LoopUnderstanding the Single Thread Event Loop
Understanding the Single Thread Event Loop
 
10 Million hits a day with WordPress using a $15 VPS
10 Million hits a day  with WordPress using a $15 VPS10 Million hits a day  with WordPress using a $15 VPS
10 Million hits a day with WordPress using a $15 VPS
 
Plack basics for Perl websites - YAPC::EU 2011
Plack basics for Perl websites - YAPC::EU 2011Plack basics for Perl websites - YAPC::EU 2011
Plack basics for Perl websites - YAPC::EU 2011
 
Getting started with Ansible
Getting started with AnsibleGetting started with Ansible
Getting started with Ansible
 
Deploying Plack Web Applications: OSCON 2011
Deploying Plack Web Applications: OSCON 2011Deploying Plack Web Applications: OSCON 2011
Deploying Plack Web Applications: OSCON 2011
 
now
nownow
now
 
Deploying VMware vCloud Hybrid Service with Puppet - PuppetConf 2013
Deploying VMware vCloud Hybrid Service with Puppet - PuppetConf 2013Deploying VMware vCloud Hybrid Service with Puppet - PuppetConf 2013
Deploying VMware vCloud Hybrid Service with Puppet - PuppetConf 2013
 
20141210 rakuten techtalk
20141210 rakuten techtalk20141210 rakuten techtalk
20141210 rakuten techtalk
 
Nashorn
NashornNashorn
Nashorn
 
SaltConf 2014: Safety with powertools
SaltConf 2014: Safety with powertoolsSaltConf 2014: Safety with powertools
SaltConf 2014: Safety with powertools
 
Serverless 프레임워크로 Nuxt 앱 배포하기
Serverless 프레임워크로 Nuxt 앱 배포하기Serverless 프레임워크로 Nuxt 앱 배포하기
Serverless 프레임워크로 Nuxt 앱 배포하기
 
Introducing in-house PaaS in SmartNews
Introducing in-house PaaS in SmartNewsIntroducing in-house PaaS in SmartNews
Introducing in-house PaaS in SmartNews
 

Viewers also liked

Viewers also liked (20)

Prometheus触ってみた
Prometheus触ってみたPrometheus触ってみた
Prometheus触ってみた
 
Cephを用いたwordpressの構築[LT版]
Cephを用いたwordpressの構築[LT版] Cephを用いたwordpressの構築[LT版]
Cephを用いたwordpressの構築[LT版]
 
MySQLやSSDとかの話 後編
MySQLやSSDとかの話 後編MySQLやSSDとかの話 後編
MySQLやSSDとかの話 後編
 
性能測定道 実践編
性能測定道 実践編性能測定道 実践編
性能測定道 実践編
 
MySQLやSSDとかの話 前編
MySQLやSSDとかの話 前編MySQLやSSDとかの話 前編
MySQLやSSDとかの話 前編
 
Chunked encoding を使った高速化の考察
Chunked encoding を使った高速化の考察Chunked encoding を使った高速化の考察
Chunked encoding を使った高速化の考察
 
性能測定道 事始め編
性能測定道 事始め編性能測定道 事始め編
性能測定道 事始め編
 
Web現場Meetup #2 圧倒的成長環境の作り方
Web現場Meetup #2 圧倒的成長環境の作り方Web現場Meetup #2 圧倒的成長環境の作り方
Web現場Meetup #2 圧倒的成長環境の作り方
 
スクラムを導入してみて一回挫折したけど再起させた話
スクラムを導入してみて一回挫折したけど再起させた話スクラムを導入してみて一回挫折したけど再起させた話
スクラムを導入してみて一回挫折したけど再起させた話
 
情報科学における18のメタテクニック
情報科学における18のメタテクニック情報科学における18のメタテクニック
情報科学における18のメタテクニック
 
サービスのスケール化のための検索システム改善
サービスのスケール化のための検索システム改善サービスのスケール化のための検索システム改善
サービスのスケール化のための検索システム改善
 
ウェブエンジニアのための色の話
ウェブエンジニアのための色の話ウェブエンジニアのための色の話
ウェブエンジニアのための色の話
 
プロダクトに 1 から Vue.js を導入した話
プロダクトに 1 から Vue.js を導入した話プロダクトに 1 から Vue.js を導入した話
プロダクトに 1 から Vue.js を導入した話
 
エフェクト用 Shader 機能紹介
エフェクト用 Shader 機能紹介エフェクト用 Shader 機能紹介
エフェクト用 Shader 機能紹介
 
5年しかもたない最高のシステムとの向き合い方
5年しかもたない最高のシステムとの向き合い方5年しかもたない最高のシステムとの向き合い方
5年しかもたない最高のシステムとの向き合い方
 
Node Foundation Membership Overview 20160907
Node Foundation Membership Overview 20160907Node Foundation Membership Overview 20160907
Node Foundation Membership Overview 20160907
 
エンジニアがデザインやってみた @ Aimning MeetUp 2017/10
エンジニアがデザインやってみた @ Aimning MeetUp 2017/10エンジニアがデザインやってみた @ Aimning MeetUp 2017/10
エンジニアがデザインやってみた @ Aimning MeetUp 2017/10
 
エフェクトにしっかり色を付ける方法
エフェクトにしっかり色を付ける方法エフェクトにしっかり色を付ける方法
エフェクトにしっかり色を付ける方法
 
Node.js API 서버 성능 개선기
Node.js API 서버 성능 개선기Node.js API 서버 성능 개선기
Node.js API 서버 성능 개선기
 
スマホゲームのUI仕様書
スマホゲームのUI仕様書スマホゲームのUI仕様書
スマホゲームのUI仕様書
 

Similar to Advanced technic for OS upgrading in 3 minutes

Apache Druid Auto Scale-out/in for Streaming Data Ingestion on Kubernetes
Apache Druid Auto Scale-out/in for Streaming Data Ingestion on KubernetesApache Druid Auto Scale-out/in for Streaming Data Ingestion on Kubernetes
Apache Druid Auto Scale-out/in for Streaming Data Ingestion on Kubernetes
DataWorks Summit
 
Cannibalising The Google App Engine
Cannibalising The  Google  App  EngineCannibalising The  Google  App  Engine
Cannibalising The Google App Engine
catherinewall
 

Similar to Advanced technic for OS upgrading in 3 minutes (20)

Large-scaled Deploy Over 100 Servers in 3 Minutes
Large-scaled Deploy Over 100 Servers in 3 MinutesLarge-scaled Deploy Over 100 Servers in 3 Minutes
Large-scaled Deploy Over 100 Servers in 3 Minutes
 
Toolbox of a Ruby Team
Toolbox of a Ruby TeamToolbox of a Ruby Team
Toolbox of a Ruby Team
 
Devops continuousintegration and deployment onaws puttingmoneybackintoyourmis...
Devops continuousintegration and deployment onaws puttingmoneybackintoyourmis...Devops continuousintegration and deployment onaws puttingmoneybackintoyourmis...
Devops continuousintegration and deployment onaws puttingmoneybackintoyourmis...
 
DevOps, Continuous Integration and Deployment on AWS: Putting Money Back into...
DevOps, Continuous Integration and Deployment on AWS: Putting Money Back into...DevOps, Continuous Integration and Deployment on AWS: Putting Money Back into...
DevOps, Continuous Integration and Deployment on AWS: Putting Money Back into...
 
Serverless in Production, an experience report (AWS UG South Wales)
Serverless in Production, an experience report (AWS UG South Wales)Serverless in Production, an experience report (AWS UG South Wales)
Serverless in Production, an experience report (AWS UG South Wales)
 
Usecase examples of Packer
Usecase examples of Packer Usecase examples of Packer
Usecase examples of Packer
 
Serverless in production, an experience report (FullStack 2018)
Serverless in production, an experience report (FullStack 2018)Serverless in production, an experience report (FullStack 2018)
Serverless in production, an experience report (FullStack 2018)
 
Wider than rails
Wider than railsWider than rails
Wider than rails
 
AWS Lambda from the trenches
AWS Lambda from the trenchesAWS Lambda from the trenches
AWS Lambda from the trenches
 
ApacheCon NA - Apache Camel K: connect your Knative serverless applications w...
ApacheCon NA - Apache Camel K: connect your Knative serverless applications w...ApacheCon NA - Apache Camel K: connect your Knative serverless applications w...
ApacheCon NA - Apache Camel K: connect your Knative serverless applications w...
 
Hosting Ruby Web Apps
Hosting Ruby Web AppsHosting Ruby Web Apps
Hosting Ruby Web Apps
 
Serverless in production, an experience report (CoDe-Conf)
Serverless in production, an experience report (CoDe-Conf)Serverless in production, an experience report (CoDe-Conf)
Serverless in production, an experience report (CoDe-Conf)
 
Ratpack Web Framework
Ratpack Web FrameworkRatpack Web Framework
Ratpack Web Framework
 
Software as a Service workshop / Unlocked: the Hybrid Cloud 12th May 2014
Software as a Service workshop / Unlocked: the Hybrid Cloud 12th May 2014Software as a Service workshop / Unlocked: the Hybrid Cloud 12th May 2014
Software as a Service workshop / Unlocked: the Hybrid Cloud 12th May 2014
 
Easy Cloud Native Transformation using HashiCorp Nomad
Easy Cloud Native Transformation using HashiCorp NomadEasy Cloud Native Transformation using HashiCorp Nomad
Easy Cloud Native Transformation using HashiCorp Nomad
 
Function as a Service
Function as a ServiceFunction as a Service
Function as a Service
 
Apache Druid Auto Scale-out/in for Streaming Data Ingestion on Kubernetes
Apache Druid Auto Scale-out/in for Streaming Data Ingestion on KubernetesApache Druid Auto Scale-out/in for Streaming Data Ingestion on Kubernetes
Apache Druid Auto Scale-out/in for Streaming Data Ingestion on Kubernetes
 
Why scala is not my ideal language and what I can do with this
Why scala is not my ideal language and what I can do with thisWhy scala is not my ideal language and what I can do with this
Why scala is not my ideal language and what I can do with this
 
Cannibalising The Google App Engine
Cannibalising The  Google  App  EngineCannibalising The  Google  App  Engine
Cannibalising The Google App Engine
 
Performance Tuning Your Puppet Infrastructure - PuppetConf 2014
Performance Tuning Your Puppet Infrastructure - PuppetConf 2014Performance Tuning Your Puppet Infrastructure - PuppetConf 2014
Performance Tuning Your Puppet Infrastructure - PuppetConf 2014
 

More from Hiroshi SHIBATA

More from Hiroshi SHIBATA (20)

Long journey of Ruby Standard library at RubyKaigi 2024
Long journey of Ruby Standard library at RubyKaigi 2024Long journey of Ruby Standard library at RubyKaigi 2024
Long journey of Ruby Standard library at RubyKaigi 2024
 
Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024
 
Deep dive into Ruby's require - RubyConf Taiwan 2023
Deep dive into Ruby's require - RubyConf Taiwan 2023Deep dive into Ruby's require - RubyConf Taiwan 2023
Deep dive into Ruby's require - RubyConf Taiwan 2023
 
How resolve Gem dependencies in your code?
How resolve Gem dependencies in your code?How resolve Gem dependencies in your code?
How resolve Gem dependencies in your code?
 
How resolve Gem dependencies in your code?
How resolve Gem dependencies in your code?How resolve Gem dependencies in your code?
How resolve Gem dependencies in your code?
 
Ruby コミッターと歩む Ruby を用いたプロダクト開発
Ruby コミッターと歩む Ruby を用いたプロダクト開発Ruby コミッターと歩む Ruby を用いたプロダクト開発
Ruby コミッターと歩む Ruby を用いたプロダクト開発
 
Why ANDPAD commit Ruby and RubyKaigi?
Why ANDPAD commit Ruby and RubyKaigi?Why ANDPAD commit Ruby and RubyKaigi?
Why ANDPAD commit Ruby and RubyKaigi?
 
RailsGirls から始める エンジニアリングはじめの一歩
RailsGirls から始める エンジニアリングはじめの一歩RailsGirls から始める エンジニアリングはじめの一歩
RailsGirls から始める エンジニアリングはじめの一歩
 
How to develop the Standard Libraries of Ruby?
How to develop the Standard Libraries of Ruby?How to develop the Standard Libraries of Ruby?
How to develop the Standard Libraries of Ruby?
 
The details of CI/CD environment for Ruby
The details of CI/CD environment for RubyThe details of CI/CD environment for Ruby
The details of CI/CD environment for Ruby
 
Dependency Resolution with Standard Libraries
Dependency Resolution with Standard LibrariesDependency Resolution with Standard Libraries
Dependency Resolution with Standard Libraries
 
Roadmap for RubyGems 4 and Bundler 3
Roadmap for RubyGems 4 and Bundler 3Roadmap for RubyGems 4 and Bundler 3
Roadmap for RubyGems 4 and Bundler 3
 
The Future of library dependency management of Ruby
 The Future of library dependency management of Ruby The Future of library dependency management of Ruby
The Future of library dependency management of Ruby
 
Ruby Security the Hard Way
Ruby Security the Hard WayRuby Security the Hard Way
Ruby Security the Hard Way
 
OSS Security the hard way
OSS Security the hard wayOSS Security the hard way
OSS Security the hard way
 
The Future of library dependency manageement of Ruby
The Future of library dependency manageement of RubyThe Future of library dependency manageement of Ruby
The Future of library dependency manageement of Ruby
 
The Future of Dependency Management for Ruby
The Future of Dependency Management for RubyThe Future of Dependency Management for Ruby
The Future of Dependency Management for Ruby
 
The Future of Bundled Bundler
The Future of Bundled BundlerThe Future of Bundled Bundler
The Future of Bundled Bundler
 
What's new in RubyGems3
What's new in RubyGems3What's new in RubyGems3
What's new in RubyGems3
 
Productive Organization with Ruby
Productive Organization with RubyProductive Organization with Ruby
Productive Organization with Ruby
 

Recently uploaded

Structuring Teams and Portfolios for Success
Structuring Teams and Portfolios for SuccessStructuring Teams and Portfolios for Success
Structuring Teams and Portfolios for Success
UXDXConf
 

Recently uploaded (20)

How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdfHow Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
 
AI revolution and Salesforce, Jiří Karpíšek
AI revolution and Salesforce, Jiří KarpíšekAI revolution and Salesforce, Jiří Karpíšek
AI revolution and Salesforce, Jiří Karpíšek
 
Integrating Telephony Systems with Salesforce: Insights and Considerations, B...
Integrating Telephony Systems with Salesforce: Insights and Considerations, B...Integrating Telephony Systems with Salesforce: Insights and Considerations, B...
Integrating Telephony Systems with Salesforce: Insights and Considerations, B...
 
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
 
Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf
Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdfSimplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf
Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf
 
WSO2CONMay2024OpenSourceConferenceDebrief.pptx
WSO2CONMay2024OpenSourceConferenceDebrief.pptxWSO2CONMay2024OpenSourceConferenceDebrief.pptx
WSO2CONMay2024OpenSourceConferenceDebrief.pptx
 
Custom Approval Process: A New Perspective, Pavel Hrbacek & Anindya Halder
Custom Approval Process: A New Perspective, Pavel Hrbacek & Anindya HalderCustom Approval Process: A New Perspective, Pavel Hrbacek & Anindya Halder
Custom Approval Process: A New Perspective, Pavel Hrbacek & Anindya Halder
 
Salesforce Adoption – Metrics, Methods, and Motivation, Antone Kom
Salesforce Adoption – Metrics, Methods, and Motivation, Antone KomSalesforce Adoption – Metrics, Methods, and Motivation, Antone Kom
Salesforce Adoption – Metrics, Methods, and Motivation, Antone Kom
 
ECS 2024 Teams Premium - Pretty Secure
ECS 2024   Teams Premium - Pretty SecureECS 2024   Teams Premium - Pretty Secure
ECS 2024 Teams Premium - Pretty Secure
 
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
 
Buy Epson EcoTank L3210 Colour Printer Online.pptx
Buy Epson EcoTank L3210 Colour Printer Online.pptxBuy Epson EcoTank L3210 Colour Printer Online.pptx
Buy Epson EcoTank L3210 Colour Printer Online.pptx
 
ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...
ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...
ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...
 
IoT Analytics Company Presentation May 2024
IoT Analytics Company Presentation May 2024IoT Analytics Company Presentation May 2024
IoT Analytics Company Presentation May 2024
 
Structuring Teams and Portfolios for Success
Structuring Teams and Portfolios for SuccessStructuring Teams and Portfolios for Success
Structuring Teams and Portfolios for Success
 
Powerful Start- the Key to Project Success, Barbara Laskowska
Powerful Start- the Key to Project Success, Barbara LaskowskaPowerful Start- the Key to Project Success, Barbara Laskowska
Powerful Start- the Key to Project Success, Barbara Laskowska
 
Linux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdf
Linux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdfLinux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdf
Linux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdf
 
Optimizing NoSQL Performance Through Observability
Optimizing NoSQL Performance Through ObservabilityOptimizing NoSQL Performance Through Observability
Optimizing NoSQL Performance Through Observability
 
The Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdf
The Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdfThe Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdf
The Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdf
 
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...
 
Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptx
Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptxUnpacking Value Delivery - Agile Oxford Meetup - May 2024.pptx
Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptx
 

Advanced technic for OS upgrading in 3 minutes

  • 1. Advanced technic for OS upgrading in 3 minutes Deployment strategy for next generation
  • 2. self.introduce => { name: “SHIBATA Hiroshi”, nickname: “hsbt”, title: “Chief engineer at GMO Pepabo, Inc.”, commit_bits: [“ruby”, “rake”, “rubygems”, “rdoc”, “tdiary”, “hiki”, “railsgirls”, “railsgirls-jp”, “jenkins”], sites: [“ruby-lang.org”, “rubyci.com”, “railsgirls.com”, “railsgirls.jp”], }
  • 3. I’m from Asakusa.rb Asakusa.rb is one of the most active meet-ups in Tokyo, Japan. @a_matsuda (Ruby/Rails committer, RubyKaigi organizer) @kakutani (RubyKaigi organizer) @ko1 (Ruby committer) @takkanm (Ruby/Rails programmer) @gunjisatoshi (Rubyist Magazine editor) @hsbt (Me!)
  • 4.
  • 6.
  • 7.
  • 10. 2014/11/xx … “ある意味 100 倍すね…3ヶ月後に某サービスで CM打つことになりまして…放映までにサービス をバーン!!1としてもらいたいんですよ” “(うわー)” CTO: antipop
  • 11. Our service status at 2014/11 • Simply Rails Service with IaaS • 6 application servers • To use capistrano 2 for deployment • Mixed worker and application role • Unknown role server like handled only POST request server
  • 12. Our service issue Do scale-out Do scale-out with automation! Do scale-out with rapid automation!!! Do scale-out with extremely rapid automation!!!1
  • 13. Team member hsbt: Director, fullstack Programmer udzura: fullstack Programmer yano3: fullstack Programmer
  • 15. Web operation is manual instructions • We have been created OS Image called “Golden Image” from running server • Web operations such as os configuration and instances launch are manual instruction. • Working time is about 4-6 hours • We say it “Tanpopo works…” • It’s blocker for scale-out largely.
  • 17. Fixed all of puppet manifests • It based on Scientific Linux 6.x • Some manifest is broken… • Service developers didn’t use puppet for production At first, We fixed all of manifests and enabled to deploy to production environments. % ls **/*.pp | xargs wc -l | tail -1 5546 total
  • 18. Setting up puppetmasterd • We choice master/agent model • It’s large scaled architecture because we didn’t need to deploy puppet manifests each servers. • We already have puppetmasterd manifests written by puppet using passenger named rails application server. https://docs.puppetlabs.com/guides/passenger.html
  • 19. Use provision tool for scale-out • Launch instance from raw linux image that it’s not customized with our service. • Deploy rails application with basic instructions. • Test with single instance • Attach instance to load balancer It’s puppet work, not tanpopo work
  • 20. Check Point 0 We need to understand our server configuration via “CODE” Use provision tool like puppet/chef/ansible etc etc… Bootstrap time = 4-6 hours
  • 22. Concerns of bootstrap instructions Typical scenario of server set-up for scale out. • OS boot • OS Configuration • Provisioning with puppet/chef • Setting up to capistrano • Deploy rails application • Added load balancer (= Service in)
  • 23. No ssh We added “No SSH” into our rule of Web operation
  • 24. Background of “No SSH” In large scale service, 1 instance is like a “1 process” in Unix environments. We didn’t attach process using gdb usually. • We don’t access instance via ssh We didn’t modify program variables in memory usually. • We don’t modify configuration on instance We can handle instance/process status using signal/api only.
  • 25. We have awesome operation tools • clout-init • packer • consul • IaaS api/cli
  • 27. What’s cloud-init “Cloud-init is the defacto multi-distribution package that handles early initialization of a cloud instance.” https://cloudinit.readthedocs.org/en/latest/ • We(and you) already used cloud-init for customizing to OS configuration at initialization process on IaaS • It has few documents for our use-case…
  • 28. Tuning tools(cloud-init) We only use OS configuration. Do not use “run_cmd” #cloud-config repo_update: true repo_upgrade: none packages: - git - curl - unzip users: - default locale: ja_JP.UTF-8 timezone: Asia/Tokyo
  • 29. Do not use hostname/ip dependency We discarded dependencies of hostname and ip address. Use API of IaaS for our use-case. config.ru: 10: defaults = `hostname`.start_with?('job') ? config/database.yml: 37: if `hostname`.start_with?(‘solr') config/unicorn.conf: 6: if `hostname`.start_with?('job')
  • 30. Image creation with itself We use IaaS API for image creation with cloud-init userdata. We can create OS Image using cloud-init and provisioned puppet when boot time of instance. puppet agent -t rm -rf /var/lib/cloud/sem /var/lib/cloud/instances/* aws ec2 create-image --instance-id `cat /var/lib/cloud/data/instance-id` --name www_base_`date +%Y%m%d%H%M`
  • 31. Rails
  • 32. Upgrading Rails 4 • I am very good at “Rails Upgrading” • Deploying in Production was performed with @amacou % g show c1d698e commit c1d698ec444df1c137a301e01f59e659593ecf76 Author: amacou <amacou.abf@gmail.com> Date: Mon Dec 15 18:22:34 2014 +0900 Revert "Revert "Revert "Revert "[WIP] Rails 4.1.X へのアップグレード""""
  • 33. Check point 1 • DO NOT change main architecture • Write real-world instructions • Pick instruction for automation • DO automation Bootstrap time = 1hours
  • 36. What’s new for capistrano3 “A remote server automation and deployment tool written in Ruby.” http://capistranorb.com/ Example of Capfile: We rewrite own capstrano2 tasks to capistrano3 convention require 'capistrano/bundler' require 'capistrano/rails/assets' require 'capistrano3/unicorn' require 'capistrano/banner' require 'capistrano/npm' require 'slackistrano'
  • 38. Bundled package of Rails application Prepared to standalone Rails application with rubygems and precompiled assets Part of capistrano tasks: $ bundle exec cap production archive_project ROLES=build desc "Create a tarball that is set up for deploy" task :archive_project => [:ensure_directories, :checkout_local, :bundle, :npm_install, :bower_install, :asset_precompile, :create_tarball, :upload_tarball, :cleanup_dirs]
  • 39. Distributed rails package build server rails bundle object storage (s3) application server application server application server application server
  • 40. # 最新のアプリケーションの取得 RELEASE=`date +%Y%m%d%H%M` ARCHIVE_ROOT=‘s3://rails-application-bundle/production/' ARCHIVE_FILE=$( aws s3 ls $ARCHIVE_ROOT | grep -E 'application-.*.tgz' | awk '{print $4}' | sort -r | head -n1 ) aws s3 cp "${ARCHIVE_ROOT}${ARCHIVE_FILE}" /tmp/rails-application.tar.gz # cap setup 相当を実行 (snip) # chown を実行 (snip) We extracted rails bundle when instance creates self image with clout-init. Integration of image creation
  • 42. Nagios We used nagios for monitoring to service and instance status. But we have following issue: • nagios don’t support dynamic scaled architecture • Complex syntax and configuration We decided to use nagios for service monitoring like http status with load balancer only.
  • 43. consul + consul-alert We use consul and consul-alerts for process monitoring. https://github.com/hashicorp/consul https://github.com/AcalephStorage/ consul-alerts It provided to discover to new instances automatically and alert mechanism with slack integration.
  • 45. munin We used munin for resource monitoring But munin doesn’t support dynamic scaled architecture. We decided to use mackerel.io instead of munin.
  • 46. Mackerel “A Revolutionary New Kind ofApplication Performance Management. Realize the potential in Cloud Computingby managing cloud servers through “roles”” https://mackerel.io
  • 47. Auto join and leave with mackrel You can added instance to role(server group) on mackerel with mackerel.con You can remove instance from mackerel when instance shutdown. We added following script to initscripts ※ It’s official support now http://blog-ja.mackerel.io/entry/2015/07/31/105300 [user@www ~]$ cat /etc/mackerel-agent/mackerel-agent.conf apikey = “your_api_key” role = [ "service:web" ] curl -s -X POST -H 'Content-type: application/json' -H ‘X-Api-Key:api_key' https://mackerel.io/api/v0/hosts/`cat /var/lib/mackerel-agent/id`/retire
  • 49. access_log aggregator with td-agent We need to collect access log of all servers with scale-out. <match nginx.**> type forward send_timeout 60s recover_wait 10s heartbeat_interval 1s phi_threshold 16 hard_timeout 60s <server> name aggregate.server host aggregate.server weight 100 </server> <server> name aggregate2.server host aggregate2.server weight 100 standby </server> </match> <match nginx.access.*> type copy <store> type file (snip) </store> <store> type tdlog apikey api_key auto_create_table true database database table access use_ssl true flush_interval 120 buffer_path /data/tmp/td-agent-td/access </store> </match>
  • 50. thor
  • 51. What’s thor “Thor is a toolkit for building powerful command-line interfaces. It is used in Bundler, Vagrant, Rails and others.” http://whatisthor.com/ module AwesomeTool class Cli < Thor class_option :verbose, type: :boolean, default: false desc 'instances [COMMAND]', ‘Desc’ subcommand('instances', Instances) end end module AwesomeTool class Instances < Thor desc 'launch', ‘Desc' method_option :count, type: :numeric, aliases: "-c", default: 1 def launch (snip) end end end
  • 52. We can scale out with one command via our cli tool All of web operations should be implement by command line tools Scale out with cli command $ some_cli_tool instances launch -c … $ some_cli_tool mackerel fixrole $ some_cli_tool scale up $ some_cli_tool deploy blue-green
  • 53. Check point 2 • Use cloud-oriented architecture • Adopt next generation architecture aggressively • Web operations should be provided from programs Bootstrap time = 20-30min
  • 54. CM
  • 56. Concerns of bootstrap time Typical scenario of server set-up for scale out. • OS boot • OS Configuration • Provisioning with puppet/chef • Setting up to capistrano • Deploy rails application • Added load balancer (= Service in) We need to enhance to bootstrap time extremely.
  • 57. Concerns of bootstrap time Slow operation • OS boot • Provisioning with puppet/chef • Deploy rails application Fast operation • OS Configuration • Setting up to capistrano • Added load balancer (= Service in)
  • 58. Check point of Image creation Slow operation • OS boot • Provisioning with puppet/chef • Deploy rails application Fast operation • OS Configuration • Setting up to capistrano • Added load balancer (= Service in) Step1 Step2
  • 59. 2 phase strategy • Official OS image • Provided from platform like AWS, Azure, GCP, OpenStack… • Minimal image(phase 1) • Network, User, Package configuration • Installed puppet/chef and platform cli-tools. • Role specified(phase 2) • Only boot OS and Rails application
  • 61. After packer age I couldn’t understand use-case of packer. Is it Provision tool? Deployment tool?
  • 63. inside image creation with Packer • Packer configuration • JSON format • select instance size, block volume, • cloud-init • Basic configuration of OS • only default module of cloud-init • provisioner • shell script :)
  • 64. minimal image cloud-init provisioner #cloud-config repo_update: true repo_upgrade: none packages: - git - curl - unzip users: - default locale: ja_JP.UTF-8 timezone: Asia/Tokyo rpm -ivh http://yum.puppetlabs.com/ puppetlabs-release-el-7.noarch.rpm yum -y update yum -y install puppet yum -y install python-pip pip install awscli sed -i 's/name: centos/name: cloud-user/' /etc/ cloud/cloud.cfg echo 'preserve_hostname: true' >> /etc/cloud/ cloud.cfg
  • 65. www image cloud-init provisioner #cloud-config preserve_hostname: false puppet agent -t set -e monit stop unicorn /usr/local/bin/globefish -w rm -rf /var/www/deploys/minne/releases/* rm -f /var/www/deploys/minne/current # tar xf するだけで動くRails アプリケーションを取得 (snip) # mackerel のホスト設定が packer 実行時のものとかぶらないように初期化 rm /var/lib/mackerel-agent/id # cloud-init をもう一度動かすようにする準備 rm -rf /var/lib/cloud/sem /var/lib/cloud/instances/*
  • 66. Integration tests with Packer We can tests results of Packer running. (Impl by @udzura) "provisioners": [ (snip) { "type": "shell", "script": "{{user `project_root`}}packer/minimal/provisioners/run-serverspec.sh", "execute_command": "{{ .Vars }} sudo -E sh '{{ .Path }}'" } ] yum -y -q install rubygem-bundler cd /tmp/serverspec bundle install --path vendor/bundle bundle exec rake spec packer configuration run-serverspec.sh
  • 67. We created cli tool with thor We can run packer over thor code with advanced options. $ some_cli_tool ami build-minimal $ some_cli_tool ami build-www $ some_cli_tool ami build-www —init $ some_cli_tool ami build-www -a ami-id module SomeCliTool class Ami < Thor method_option :ami_id, type: :string, aliases: "-a" method_option :init, type: :boolean desc 'build-www', 'wwwの最新イメージをビルドします' def build_www … end end end
  • 69. What's Infra CI We test server status such as lists of installed packages, running processes and configuration details continuously. Puppet + Drone CI(with Docker) + Serverspec = WIN We can refactoring puppet manifests aggressively.
  • 70. Drone CI “CONTINUOUS INTEGRATION FOR GITHUB AND BITBUCKET THAT MONITORS YOUR CODE FOR BUGS” https://drone.io/ We use Drone CI on our Openstack platform named “nyah”
  • 71. Serverspec “RSpec tests for your servers configured by CFEngine, Puppet, Ansible, Itamae or anything else.” http://serverspec.org/ % rake -T rake mtest # Run mruby-mtest rake spec # Run serverspec code for all rake spec:base # Run serverspec code for base.minne.pbdev rake spec:batch # Run serverspec code for batch.minne.pbdev rake spec:db:master # Run serverspec code for master db rake spec:db:slave # Run serverspec code for slave db rake spec:gateway # Run serverspec code for gateway.minne.pbdev (snip)
  • 72. Refactoring puppet manifets We replaced “puppetserver” written by Clojure. We enabled future-parser. We fixed all of warnings and syntax error. We added and removed manifests everyday.
  • 74. Switch Scientific Linux 6 to CentOS 7 We can refactoring to puppet manifests with infra CI. We added case-condition for SL6 and Centos7 if $::operatingsystemmajrelease >= 6 { $curl_devel = 'libcurl-devel' } else { $curl_devel = 'curl-devel' }
  • 75. How to test instance behavior We need to guarantee http status from instance response. We removed package version control from our concerns.
  • 76. Check point 3 • Packer is best tool of Image creation • Infra CI is over evaluation phase • You can refactor provision manifests now Bootstrap time = 3-5min
  • 79. Instructions of Blue-Green deployment Basic concept is following instructions. 1. Launch instances using OS imaged created from Packer 2. Wait to change “InService” status 3. Terminate old instances That’s all!!1
  • 80. Dynamic upstream with load balancer ELB • Provided by AWS, It’s best choice for B-G deployment • Can handle only AWS instances nginx + consul-template • Change upstream directive used consul and consul-template ngx_mruby • Change upstream directive used mruby
  • 81. Slack integration of consul-template
  • 82. Example code of thor old_instances = running_instances(load_balancer_name) invoke Instances, [:launch], options.merge(:count => old_instances.count) catch(:in_service) do sleep_time = 60 loop do instances = running_instances(load_balancer_name) throw(:in_service) if (instances.count == old_instances.count * 2) && instances.all?{|i| i.status == 'InService'} sleep sleep_time sleep_time = [sleep_time - 10, 10].max end end old_instances.each do |oi| oi.delete end
  • 83. Check point 4 We can upgrade OS version in 3 minutes
  • 85. Next step of our stage • Automated all of test with image creation and launching • Flexible architecture includes mutable roles • Sync deployment with image creation cycle • Use Docker