Viele Server - Wenig Arbeit: Betriebsautomation bei ImmobilienScout24

Viele Server – wenig Arbeit
Berlin | 07.06.2013 | Schlomo Schapiro | @SchlomoSchapiro
Systemarchitekt, Open Source Evangelist
License: http://creativecommons.org/licenses/by-nc-nd/3.0/
Betriebsautomation bei ImmobilienScout24

| Betriebsautomation bei ImmobilienScout24 | Schlomo Schapiro
www.ImmobilienScout24.de
>2 billion PI per month
2 data center with ~1400 VM
total of ~600 employees
 ~30 crossfunctional IT teams
 ~160 in IT
15 years in business
part of Deutsche Telekom

Papa,was
machst Du
eigentlich
auf Arbeit?

Lego City

Orchestrierung & Automation

Spielen die
wirklich
nur mit
LEGO rum?

Business
Decision
To go
Live
All
Humans
are on the
Same Side
Build
Config
Deploy
Test
Systems-
Management
Automation
Scale Out
Data Centers
through
packages

PROInfrastructure
Build
Application
Build
Configuration
Build
QA
YUM
Repos
Interface
DEV
github.com/ImmobilienScout24/yum-repo-server

“Any relevant file should be either
deployed via a package
or
completely managed by an
application that is thus deployed.”

„Every package must be verifyable –
stay away from package scripts.“
“Reducing the config package
reduces the deployment risk.”

Config
SVN
post-commit
→ YUM repos
Infrastructure
as
Code

TSTWEB05
Location &
Environment
Function Group
Instance
Configuration over Convention

Big Picture – Static Structure
Config
SVN
YUM
Repository
IS24 software
Automated RPM
creation
config
├── host
│ └── tstweb02
│
├── loc
│ └── tst
│ ├── VARIABLES
│ │ ├── RPM_REQUIRES
│ │ ├── RPM_PROVIDES
│ │ ├── DB_HOST
│ │ └── DB_USER
│ └── etc
│ └── is24
│ ├── web.properties
│ └── db.properties
├── typ
│ └── web
│
├── loctyp
│ └── tstweb
│ └── etc
│ └── is24
│ └── web.properties
└── all
├── VARIABLES
│ └── SYSLOG_HOST
└── etc
└── is24
└── system.properties
is24-config-tstweb01-1.0-$rev.rpm:
/etc/is24/system.properties
/etc/is24/db.properties
/etc/is24/web.properties
[root@tstweb01 ~]# yum update
svn ci
db.host=@@@DB_HOST@@@
db.user=@@@DB_USER@@@
db.port=3306
Post-commit hook creates
is24-config-$hostname RPM
●
svn export
●
patch VARIABLES
●
fill in metadata
svn co
loghost=log.domain.com

Static Structure – Pros and Cons
Variables follow same
structure as config data
Only one tool (SVN) needed
GUI talks only to SVN
Versioning and change
tracking for variables
Variables and config files
change together (atomic)
easy rollback→
Is this simple overlaying structure sufficient?
 Simplify the world so that it is good enough!
Config
SVN
YUM
Repository
IS24 software
Automated RPM
creation
config
├── host
│ └── tstweb02
│
├── loc
│ └── tst
│ ├── VARIABLES
│ │ ├── RPM_REQUIRES
│ │ ├── RPM_PROVIDES
│ │ ├── DB_HOST
│ │ └── DB_USER
│ └── etc
│ └── is24
│ ├── web.properties
│ └── db.properties
├── typ
│ └── web
│
├── loctyp
│ └── tstweb
│ └── etc
│ └── is24
│ └── web.properties
└── all
├── VARIABLES
│ └── SYSLOG_HOST
└── etc
└── is24
└── system.properties
is24-config-tstweb01-1.0-$rev.rpm:
/etc/is24/system.properties
/etc/is24/db.properties
/etc/is24/web.properties
[root@tstweb01 ~]# yum update
svn ci
db.host=@@@DB_HOST@@@
db.user=@@@DB_USER@@@
db.port=3306
Post-commit hook creates
is24-config-$hostname RPM
●
svn export
●
patch VARIABLES
●
fill in metadata
svn co
loghost=log.domain.com
yadt-config-rpm-maker

File Management with RPM & YUM
PROD
DEV
Infrastructure
Build
Application
Build
Configuration
Build
QA
YUM
Repos

Example: is24-tomcat Packages
is24-tomcat-welcome = 6.0.35
is24-tomcat-standalone is24-tomcat-behind-httpd
is24-tomcat-base
is24-tomcat = 6.0.35
XOR
Req
Req Req
Req
is24-tomcat-welcome = 7.0.40
is24-tomcat = 7.0.40

Example: Apache HTTPD
Design Goals:
Use and extend upstream httpd RPM
Configure MPM and service user per application
IS24 standard configuration everywhere
httpd RPM
/etc/httpd/conf/httpd.conf
/etc/sysconfig/httpd

Example: is24-httpd RPM
if ! echo '# IS24 HTTPD conf framework. Read IS24_README!
# This file is managed by %{name}%{version}.%{release}
# Put your stuff in /etc/conf/*/is24*.conf files!
ServerRoot "/etc/httpd"
Include conf/basic/is24*.conf
Include conf/main/is24*.conf
Include conf/other/is24*.conf
' >/etc/httpd/conf/httpd.conf ; then
logger p user.err s t %name "ERROR: …"
fi
Requires: httpd
Requires(Pre): httpd
%post

Example: is24-httpd RPM
if ! echo '# HTTPD options can be configured in
# additional /etc/httpd/conf/env/*.sh files
# This file is managed by %{name}%{version}.%{release}
# Please add extra options to the OPTIONS Bash Array
shopt s nullglob
HTTPD=/usr/sbin/httpd.worker
for f in /etc/httpd/conf/env/*.sh ; do
source $f
done
OPTIONS="${OPTIONS[*]}" # flatten array
' >/etc/sysconfig/httpd ; then
logger p user.err s t %name "ERROR: ..."
fi
...
%post (continued)

Systems Management
yadtshell

YADT – Unified Dependency Tree
Web Server
HTTPD
Tomcat
App Server
Load Balancer

HTTPD
Tomcat
web-
app
httpd
config
tomcat
YADT – Unified Dependency Tree
init
script
init
script
/etc/yadt.services
- httpd:
is_frontservice: true
needs_services:
- tomcat
- tomcat:

YADT – Large Scale System Management
Web Server
HTTPD
Tomcat
web-
app
httpd
config
tomcat
App Server
Java App
app
config

LB: www.immobilienscout24.de
Web Server
HTTPD
Tomcat
web-
app
httpd
config
tomcat
App Server
Java App
app
config
Web Server
HTTPD
Tomcat
web-
app
httpd
config
tomcat
App Server
Java App
app
config

Web Server
HTTPD
Tomcat
web-
app
httpd
config
tomcat
App Server
Java App
app
config
Web Server
HTTPD
Tomcat
web-
app
httpd
config
tomcat
App Server
Java App
app
config
1st
chunk

Web Server
HTTPD
Tomcat
web-
app
httpd
config
tomcat
App Server
Java App
app
config
Web Server
HTTPD
Tomcat
web-
app
httpd
config
tomcat
App Server
Java App
app
config
1st
chunk
2nd
chunk

yadtshell Wave Deployment

Systems Management with YADT
Reduce Systemic Complexity
Keep It Simple

Continuous Delivery Platform
Business
Decision
To Go
Live
All Humans
are on the
Same Side
Build
Config
Deploy
Test
Systems-
Management
Automation
Scale Out
Data Centers

DevOps

Abstraktion
Starke Vereinfachung
Standardbausteine
Lösungen wiederverwenden
Kleine Komponenten zu Großem verbinden
Alles ist machbar

www.yadt-project.org
go.schapiro.org/slides

bit.ly/is24techjobs

Kontakt:
Immobilien Scout GmbH
Andreasstraße 10
10243 Berlin
Fon: +49 30 243 01-1229
Email: schlomo.schapiro@immobilienscout24.de
URL: www.immobilienscout24.de
Thank you very much!
Please contact me for further
questions and discussions.

Photo URLs:
http://skepchick.org/wp-content/uploads/lego-1.jpg
http://smithduck.home.comcast.net/~smithduck/bmw/kwiring/Early%20K75%20K100/K100_Early_Wiring_Diagram.jpeg
http://brickbrowse.com/creator/small-cottage-31009/
http://static.desktopnexus.com/owallpaper/15044-1024x768-4097LinuxSkeleton1024768.jpg (John Schriener)
http://images.brickbrowse.com/technic/motorized-excavator-8043/technic-motorized-excavator-8043-3.jpg
http://images.brickbrowse.com/technic/container-truck-8052/technic-container-truck-8052-3.jpg
http://www.flickr.com/photos/dhardacre/5754596620/
http://images.brickbrowse.com/architecture/brandenburg-gate-21011/architecture-brandenburg-gate-21011-1.jpg
http://www.gtaberlin.de/models/brandenburger_tor_1.jpg
All remaining images are custom-made or from http://openclipart.org

Viele Server - Wenig Arbeit: Betriebsautomation bei ImmobilienScout24

Recommended

Recommended

More Related Content

Similar to Viele Server - Wenig Arbeit: Betriebsautomation bei ImmobilienScout24

Similar to Viele Server - Wenig Arbeit: Betriebsautomation bei ImmobilienScout24 (20)

More from Schlomo Schapiro

More from Schlomo Schapiro (20)

Viele Server - Wenig Arbeit: Betriebsautomation bei ImmobilienScout24