Webinar slides: DevOps Tutorial: how to automate your database infrastructure

Copyright 2017 Severalnines ABCopyright 2017 Severalnines AB
I'm Jean-Jérôme from the Severalnines Team and
I'm your host for today's webinar!
Feel free to ask any questions in the Questions
section of this application or via the Chat box.
You can also contact me directly via the chat box or
via email: jj@severalnines.com during or after
the webinar.
Your host & some logistics

About Severalnines and ClusterControl

What we do
Manage Scale
MonitorDeploy

● Provisioning
○ Deploy a cluster in minutes
○ On-premises or in the cloud (AWS)
● Monitoring
○ Systems view
○ 1sec resolution
○ DB / OS stats & performance
advisors
○ Configurable dashboards
○ Query Analyzer
○ Real-time / historical
ClusterControl Automation & Management
● Management
○ Multi cluster/data-center
○ Automate repair/recovery
○ Database upgrades
○ Backups
○ Configuration management
○ Cloning
○ One-click scaling

Supported Databases

Customers

Copyright 2017 Severalnines AB
DevOps Tutorial
July, 18th 2017
‘How to Automate Your Database Infrastructure’
Riaan Nolan, Devops Lead @Mukuru.com
Presenter
riaan@mukuru.com
Art van Scheppingen, Senior Support Engineer @Severalnines
Presenter
art@severalnines.com

● DevOps and databases - what are the challenges
● Managing databases in a DevOps environment
● Data distribution
● Cloud Automation on AWS
● Demo of DevOps features in ClusterControl
Agenda

Devops and Databases - What are the Challenges?

● Collaboration and communication between Dev and Ops
○ Change in culture and environment
○ Work closely together
● More (frequent) development cycles
A different approach to Development and Ops

● RDBMS databases are mostly built around securing the integrity of the data
○ Schema changes are slow and will create locks
○ Frequent changes will inevitably cause production issues
● NoSQL databases have schemaless design
○ Trade off of flexibility over consistency
How do databases fit in DevOps?

● DevOps roles are less clearly defined
○ Developer is now involved in operations
○ Ops is now actively involved in development release cycles
○ The whole team shares responsibility over the application
● A DBA is a too specialized role
○ Collaborative role between developers and ops
○ Developer will drive initiative, changes and performance
○ Ops will handle consistency and security
Where does the DBA fit in DevOps?

● Deployment automation for frequent release cycles
○ Continuous deployment needs quick environment deployments
○ Full environment also requires database copies
● Incompatibility between databases and microservices
○ Microservices have a shared nothing architecture
○ Databases generally have a relational schema
● Collaboration
○ Communication is key
○ Every team member needs to be up to date at all times
○ Collaboration tools can help
What challenges are there for Databases and DevOps?

● Get all credentials in code, whether you manage the resource or not.
- Start by getting access to the resources your are to manage.
- Store these details encrypted.
- Share/Transfer/Make available these credentials securely
● Weird code being deployed, causing outages or undesired performance degradation.
- Developers can push code that cause outages and performance degradation, I use Newrelic,
Cloudwatch, Kibana for high level performance dashboards.
● Once you do have this resource under DevOps control, how do you present it to the app?
- Capistrano, Puppet and Composer for example can write config files, I use a combination of
Capistrano executing Composer that use DotEnv https://github.com/vlucas/phpdotenv
● Alerting, who’s responsible (and able) to get up at 3am and solve a complex outage, caused by
code, you have no knowledge of, pending on application versions you have no authority to
deploy or commit to? Feedback loops?
- Slack, Collectd, Kibana, New Relic, Cloudwatch
What are the challenges

Managing Databases in a DevOps environment

● Microservices are very popular among DevOps
○ Easy to deploy
○ Strong cohesion
○ Loose coupling
● Shared-nothing databases
○ Simple schema designs
○ Interchangeable with any other storage system
● How to join data between microservices?
The impact of microservice architectures

● Direct querying creates a dependency between
them
○ Any change to the schema could make the
depending microservice to fail
○ Any infrastructural change needs to be
propagated to the depending mircoservice
○ Duplicated configurations can get out of sync
How to join data between microservices?

● The only proper way would be by making a request to
the API
○ In case of a slow response or error the
microservice should be resilient
○ Generic error handling can be used for multiple
(depending) microservices
○ This creates data layer abstraction
● Microservices can now be independently changed
○ Non-standard datastores can be used
○ Migrations are much easier
How to join data between microservices?

● Full environment deployments
○ Release development/testing
○ Disaster recovery (testing)
● Automatic database deployments
○ Restore/copy (production) data sets
○ Install necessary packages and helper software
○ Create necessary (application users)
○ Setup load balancers
● Automation frameworks
○ Puppet
○ Chef
○ Ansible
Automated deployments

● Make sure insights are available for anyone in the team
● Create dashboards around application / services
● Incorporate existing application / service metrics with (detailed) database metrics
○ Correlation of these metrics can help in tracing problems
○ Pre-emptive troubleshooting when (application) latency increases
● Application performance monitoring systems lack database performance metrics
○ No insight in what happens inside the database
○ Only insight from the application point of view
Performance monitoring

● Rapid and continuous development may lead to many schema changes
● Adding new columns and tables is no problem
○ Altering existing columns will create structural changes
● ORMs are known to generate frequent schema changes
● Not all ORMs can properly handle schema changes in production
○ Requires create / alter rights
○ Applies changes directly to any connection
● Automate using available tools
○ Percona Online Schema Change
○ Gh-ost
Schema changes

● New versions contain (important) bug fixes and resolve vulnerabilities
○ Upgrade regularly
● Minor version vs major versions
○ Minor versions are in-place replacements
○ Major versions (may) require dumping/loading of data
● Upgrades are routine for sysadmins and DBAs
○ DevOps may not be that experienced
○ Automate as much as possible
● Retain high availability
○ Upgrade slaves first and then promote an upgraded slave to become master
○ Use a proxy to divert traffic from nodes that get upgraded
● Test scenarios through
Version upgrades

● Microservices should be made resilient against failure
● Topology changes should not result in failure
● Master node fails
○ Slave promotion necessary
○ This may take up to 20 seconds to happen
● Failover handling
○ Configfile: topology discovery and failover has to be handled by the application
○ Resource manager: provides topology to the application, (may) handle failover
○ Load Balancer / Proxy: provides connections to the application and handles failover
Automated failover

● One of the most important aspects of DevOps is the communication between members
● Improve communication
○ Put people physically in a team?
○ Distributed teams?
○ Enrich communication channels
● ChatOps
○ Integrate (vital) information as metrics, alerts and status
○ Add database commands
○ Pre-deployment backups
○ Schema changes
Chatops integrations

Data distribution

● Not all data may be in the same location
○ Multi data center topologies
○ Disaster recovery
○ Uneven (storage) usage
● Sharding will separate data
○ Functional sharding by service separation
○ Horizontal sharding by every growing data
Data locality

● Microservices abstract (database)
complexity away from the application
● Microservices increases complexity on
infrastructure level
○ Data locality
○ Sharding
○ Schema changes
● Load balancer / proxy will abstract the
infrastructure complexity away
○ Help with maintenance
○ Automatic scaling
Managing data flows

Cloud Automation on AWS

Atom is a great IDE for DevOps, developed by https://github.com
https://atom.io
https://atom.io/packages/hiera-eyaml
Using Hiera EYAML, you can encrypt credentials, and store this under version control
Mailvelope is an easy to use PGP client for popular browsers
and email provider to send secure emails
https://www.mailvelope.com/en/
Store and Share credentials Securely

DotEnv to Autoload credentials, developed by https://github.com/vlucas/phpdotenv
Store and Share credentials Securely continued..

https://gist.github.com/star3am/d5c84ba4a6a00f6abdb57d1d0a50fac3
Managing DBs in a DevOps environment

# Hiera hash
profile_mysql::servers:
company-stack-env-role:
endpoint: 'company-stack-env-role.private.vpc'
user: 'root'
password: ENC[PKCS7,MIIBi..==]
# Hiera Eyaml - https://puppet.com/blog/encrypt-your-data-using-hiera-eyaml
databases:
database_env: 'present'
test_database_env: 'absent'
# Set .my.cnf in root with above hira details so we can manage the resources
$servers = hiera_hash('profile_mysql::servers', {})
file { '/root/.my.cnf':
content => template("profile_mysql/root/.my.cnf"),
}

# Call the defined type to create the DBs
create_resources('profile_mysql::databases', $servers)
# Puppet ERB template profile_mysql/root/.my.cnf
<% @servers.each_pair do |key, value_hash| %>
# use on the command line: mysql --defaults-group-suffix=<%= key %>
[client<%= key %>]
user=<%= value_hash['user'] %>
host=<%= value_hash['endpoint'] %>
password=<%= value_hash['password'] %>
connect_timeout=2
<% end %>

define profile_mysql::databases (
$profile = $name,
$endpoint,
$user,
$password,
$databases,
) {
$databases.each |$database_data| {
$database = $database_data[0]
$ensure = $database_data[1]
if $ensure == 'present' {
exec { "${ensure}-${profile}-${database}":
unless => "/usr/bin/mysql --defaults-file=/root/.my.cnf --defaults-group-suffix=${profile} -e "show databases;" | grep -q
"${database}" 2>/dev/null",
command => "/usr/bin/mysql --defaults-file=/root/.my.cnf --defaults-group-suffix=${profile} -e "create database
${database};"",
require => Class['mysql::client'],
}
}
Managing DBs in a DevOps environment..

Demo of DevOps features in ClusterControl

Q & A

● Riaan Nolan’s blog on DevOps
http://www.riaannolan.com
● DevOps whitepaper
https://severalnines.com/resources/whitepapers#download_whitepaper/5073
● DevOps @ Severalnines blog
https://severalnines.com/blog?categories=306
● ClusterControl Command Line Interface (CLI)
https://severalnines.com/product/clustercontrol/command-line-client-cli
Additional resources

Webinar slides: DevOps Tutorial: how to automate your database infrastructure

Recommended

Recommended

More Related Content

Similar to Webinar slides: DevOps Tutorial: how to automate your database infrastructure

Similar to Webinar slides: DevOps Tutorial: how to automate your database infrastructure (20)

More from Severalnines

More from Severalnines (20)

Recently uploaded

Recently uploaded (20)

Webinar slides: DevOps Tutorial: how to automate your database infrastructure