VSHN manages servers for customers using automated tools. Aarno from VSHN discussed their implementation which includes a web interface for customers, infrastructure as code using YAML, and automated monitoring and updates. Some lessons learned included using tools to separate customers while sharing code, monitoring and backups by default, and good processes for frequent updates.

1. VSHN - The DevOps Company
Automated Server
Administration for
DevSecOps
Aarno Aukia, CTO @ VSHN - The DevOps Company
DevSecOps Forum 12.3.2019

2. VSHN - The DevOps Company
● Who is Aarno and VSHN - The DevOps Company ?
● Managing 1200 servers
○ Challenges
○ Our implementation & user interface
○ Demo
○ Lessons learned
22
Agenda

3. VSHN - The DevOps Company
@aarnoaukia http://about.me/aarno aarno.aukia@vshn.ch
ETH → Google → Atrila → VSHN
VSHN - The DevOps Company
Since 2014, currently 35 VSHNeers in Zürich, Switzerland
Helping Developers run applications on any infrastructure making both visitors
happy with stability and developers happy with agility
33
About Aarno & VSHN.ch

4. VSHN - The DevOps Company 5
Operations = Firefighting-as-a-Service ?
5

5. VSHN - The DevOps Company
Capability Maturity Model Integration (CMMI)
66
Operations
2014
How to get to
this level?

6. VSHN - The DevOps Company
DevOps: CMMI Level 5:
People, Processes & Tools
77

7. VSHN - The DevOps Company
● Interface Dev <-> Ops
○ DevOps, CI/CD, Containers, etc
● Interface Ops <-> Infra
○ Hypervisors, APIs, Metrics, etc
● Manage VMs on any infrastructure
● Service Monitoring, Alerting, Metrics, Logs
● Updates, Backups, Configuration Management
88
VSHN Managed Service

8. VSHN - The DevOps Company
● you don’t like proper (software) release management
○ or else you’ll be in dependency hell with 200 modules in 200 environments
● you practice “don’t touch a running system”
○ you’ll be updating all servers all the time - both good and bad
● standardizing/optimizing
○ no more handcrafted snowflake pets, lots and lots of cattle
● you don’t actively develop monitoring/checks
○ or else you’ll get alert fatigue from noisy checks
99
Don’t do automatic server management if...

9. VSHN - The DevOps Company
● Web-based self-service for customers
● CRUD server definitions -> CMDB
1010
VSHN Control Panel

10. VSHN - The DevOps Company 11
Demo
11

11. VSHN - The DevOps Company 1212
Infrastructure as
Code
Single source of truth:
Configuration YAML in GIT
---
classes:
- profile_certificates
- profile_webhosting
packages:
- nagios-plugins-wordpress
- nagios-plugins-wordpress-config
profile_php::php_settings:
date.timezone: 'Europe/Zurich'
acmetool::use_production_endpoint: true
profile_php::php_version: '7.0'
profile_php::extensions:
- mysql
- gd
- mbstring
- curl
profile_icinga2::hiera_services:
check_wordpress_dev:
check_command: 'wordpress_updates'
display_name: 'Wordpress Updates test.vshn.net'
vars:
wordpress_instance: 'vshn_dev'
wordpress_db_name: 'vshn_dev'
wordpress_db_user: 'vshn_dev'
wordpress_db_pass: 'replaced'
check_wordpress_prod:
check_command: 'wordpress_updates'
display_name: 'Wordpress Updates vshn.net'
vars:
wordpress_instance: 'vshn_prod'
wordpress_db_name: 'vshn_prod'
wordpress_db_user: 'vshn_prod'
wordpress_db_pass: 'replaced'
profile_mariadb::db_root_pw: 'replaced'
profile_webhosting::sites:
vshn_dev:
stack_type: 'php'
db_type: 'mariadb'
db_password: 'replaced'
php_admin_values:
memory_limit: '300M'
post_max_size: '20M'
upload_max_filesize: '20M'
manage_tls: letsencrypt

12. VSHN - The DevOps Company
● who
● when
● what
● tests successful?
● why
1313
GIT: changelog

13. VSHN - The DevOps Company 1414
Logic
# https check
if $manage_tls and $manage_tls != 'false' {
::profile_icinga2::resources::check { "${name}_nginx-${main_domain}-https":
check_command => 'http',
display_name => "nginx https ${name}",
check_zone => $monitoring_check_zone,
notes => 'https check on port 443, supports authentication and URI',
notes_url => 'http',
production_level => $monitoring_production_level,
vars => merge({
'http_address' => $::fqdn,
'http_vhost' => $main_domain,
'http_ssl' => true,
'http_uri' => $monitor_path,
'http_sni' => true,
}, $_auth_var, $monitor_extra_check_vars),
}
}
# https certificate check
if ($manage_tls and $manage_tls != 'false' and $manage_tls != 'snakeoil') {
if $manage_tls == 'trusted' {
$_days = 30
} else {
$_days = 20
}
::profile_icinga2::resources::check { "${name}_nginx-${main_domain}-certificate":
check_command => 'http',
display_name => "nginx certificate ${main_domain}",
notes => "https certificate check, checks days (${_days}) left until
expire",
notes_url => 'https_certificate',
production_level => $monitoring_production_level,
check_interval => '2h',
vars => {
'http_address' => $::fqdn,
'http_vhost' => $main_domain,
'http_tls' => true,
'http_certificate' => $_days,
'http_sni' => true,
},
}
}
}
e.g. automatically monitor all
HTTPS sites & certificates

14. VSHN - The DevOps Company
● Trade-off between multi-tenancy (separation of customers/environments)
and common code base development -> Tools &
Software-Release-Processes (e.g. https://github.com/vshn/crmngr)
● Monitoring & Backup by default: value > cost
● Hierarchical CMDB: configuration for policy groups
● Staged/canary rollouts (e.g. for intrusive things like firewall changes)
● Local tests (pacco: puppet-in-docker-in-vagrant:
https://github.com/vshn/puppet-in-docker)
● Good and robust process for regular & frequent updates
1515
Lessons learned

15. Come visit us for a coffee!
VSHN AG - Neugasse 10 - CH-8005 Zürich - +41 44 545 53 00 - https://vshn.ch/ - info@vshn.ch
https://vshn.ch/kontakt/
Follow us on Twitter!
@vshn_ch
16