Successfully reported this slideshow.
Your SlideShare is downloading. ×

Webinar slides: DevOps Tutorial: how to automate your database infrastructure

Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Upcoming SlideShare
IPv17 teaser
IPv17 teaser
Loading in …3
×

Check these out next

1 of 38 Ad

Webinar slides: DevOps Tutorial: how to automate your database infrastructure

Download to read offline

Join our guest speaker Riaan Nolan of mukuru.com, the First Puppet Labs Certified Professional in South Africa, as he walks us through the facets of DevOps integrations and the mission-critical advantages that database automation can bring to your database infrastructure.

Infrastructure automation isn’t easy, but it’s not rocket science either. Done right, it is a worthwhile investment, but deciding on which tools to invest in can be a confusing and overwhelming process. Riaan will share some of his secrets on how to proceed with this and he knows what he’s talking about: he saves the companies he works for substantial amounts on their monthly IT bills, typically around 50%.

Don’t miss out on this opportunity to understand how you can find efficiencies for your database infrastructure and do watch this webinar to understand the key pain points, which indicate that it’s time to invest in database automation.


AGENDA

DevOps and databases - what are the challenges
Managing databases in a DevOps environment
- Requirements from microservice environments
- Automated deployments
- Performance monitoring
- Backups
- Schema changes
- Version upgrades
- Automated failover
- Integration with ChatOps and other tools
Data distribution
- Database hosting in cloud environments
- Managing data flows
Cloud Automation on AWS



SPEAKERS

Riaan Nolan was the First Puppet Labs Certified Professional in South Africa. Riaan uses Amazon EC2, VPC and Autoscale with Cloudformation to spin up complete stacks with Autoscaling Fleets. He saves companies substantial amounts on their monthly IT bills, typically around 50% - yes, at one company that meant $500k+ per year. And he’s participated in a number of community tech related forums. He uses next generation technologies such as AWS, Cloudformation, Autoscale, Puppet, GlusterFS, NGINX, Magento and PHP to power huge eCommerce stores. His specialties are Puppet Automation, Cloud Deployments, eCommerce, eMarketing, Specialized Linux Services, Windows, Process making, Budgets, Asset Tracking, Procurement.
- Devops Lead, Mukuru
- Expert Live Systems Administrator, foodpanda | Hellofood
- Senior Systems Administrator / Infrastructure Lead, Rocket Internet GmbH
- Senior Technology Manager, Africa Internet Accelerator

Art van Scheppingen is a Senior Support Engineer at Severalnines. He’s a pragmatic MySQL and Database expert with over 15 years experience in web development. He previously worked at Spil Games as Head of Database Engineering, where he kept a broad vision upon the whole database environment: from MySQL to Couchbase, Vertica to Hadoop and from Sphinx Search to SOLR. He regularly presents his work and projects at various conferences (Percona Live, FOSDEM) and related meetups.

Join our guest speaker Riaan Nolan of mukuru.com, the First Puppet Labs Certified Professional in South Africa, as he walks us through the facets of DevOps integrations and the mission-critical advantages that database automation can bring to your database infrastructure.

Infrastructure automation isn’t easy, but it’s not rocket science either. Done right, it is a worthwhile investment, but deciding on which tools to invest in can be a confusing and overwhelming process. Riaan will share some of his secrets on how to proceed with this and he knows what he’s talking about: he saves the companies he works for substantial amounts on their monthly IT bills, typically around 50%.

Don’t miss out on this opportunity to understand how you can find efficiencies for your database infrastructure and do watch this webinar to understand the key pain points, which indicate that it’s time to invest in database automation.


AGENDA

DevOps and databases - what are the challenges
Managing databases in a DevOps environment
- Requirements from microservice environments
- Automated deployments
- Performance monitoring
- Backups
- Schema changes
- Version upgrades
- Automated failover
- Integration with ChatOps and other tools
Data distribution
- Database hosting in cloud environments
- Managing data flows
Cloud Automation on AWS



SPEAKERS

Riaan Nolan was the First Puppet Labs Certified Professional in South Africa. Riaan uses Amazon EC2, VPC and Autoscale with Cloudformation to spin up complete stacks with Autoscaling Fleets. He saves companies substantial amounts on their monthly IT bills, typically around 50% - yes, at one company that meant $500k+ per year. And he’s participated in a number of community tech related forums. He uses next generation technologies such as AWS, Cloudformation, Autoscale, Puppet, GlusterFS, NGINX, Magento and PHP to power huge eCommerce stores. His specialties are Puppet Automation, Cloud Deployments, eCommerce, eMarketing, Specialized Linux Services, Windows, Process making, Budgets, Asset Tracking, Procurement.
- Devops Lead, Mukuru
- Expert Live Systems Administrator, foodpanda | Hellofood
- Senior Systems Administrator / Infrastructure Lead, Rocket Internet GmbH
- Senior Technology Manager, Africa Internet Accelerator

Art van Scheppingen is a Senior Support Engineer at Severalnines. He’s a pragmatic MySQL and Database expert with over 15 years experience in web development. He previously worked at Spil Games as Head of Database Engineering, where he kept a broad vision upon the whole database environment: from MySQL to Couchbase, Vertica to Hadoop and from Sphinx Search to SOLR. He regularly presents his work and projects at various conferences (Percona Live, FOSDEM) and related meetups.

Advertisement
Advertisement

More Related Content

Similar to Webinar slides: DevOps Tutorial: how to automate your database infrastructure (20)

More from Severalnines (20)

Advertisement

Webinar slides: DevOps Tutorial: how to automate your database infrastructure

  1. 1. Copyright 2017 Severalnines ABCopyright 2017 Severalnines AB I'm Jean-Jérôme from the Severalnines Team and I'm your host for today's webinar! Feel free to ask any questions in the Questions section of this application or via the Chat box. You can also contact me directly via the chat box or via email: jj@severalnines.com during or after the webinar. Your host & some logistics
  2. 2. Copyright 2017 Severalnines ABCopyright 2017 Severalnines AB About Severalnines and ClusterControl
  3. 3. Copyright 2017 Severalnines ABCopyright 2017 Severalnines AB What we do Manage Scale MonitorDeploy
  4. 4. Copyright 2017 Severalnines ABCopyright 2017 Severalnines AB ● Provisioning ○ Deploy a cluster in minutes ○ On-premises or in the cloud (AWS) ● Monitoring ○ Systems view ○ 1sec resolution ○ DB / OS stats & performance advisors ○ Configurable dashboards ○ Query Analyzer ○ Real-time / historical ClusterControl Automation & Management ● Management ○ Multi cluster/data-center ○ Automate repair/recovery ○ Database upgrades ○ Backups ○ Configuration management ○ Cloning ○ One-click scaling
  5. 5. Copyright 2017 Severalnines ABCopyright 2017 Severalnines AB Supported Databases
  6. 6. Copyright 2017 Severalnines ABCopyright 2017 Severalnines AB Customers
  7. 7. Copyright 2017 Severalnines AB DevOps Tutorial July, 18th 2017 ‘How to Automate Your Database Infrastructure’ Riaan Nolan, Devops Lead @Mukuru.com Presenter riaan@mukuru.com Art van Scheppingen, Senior Support Engineer @Severalnines Presenter art@severalnines.com
  8. 8. Copyright 2017 Severalnines ABCopyright 2017 Severalnines AB ● DevOps and databases - what are the challenges ● Managing databases in a DevOps environment ● Data distribution ● Cloud Automation on AWS ● Demo of DevOps features in ClusterControl Agenda
  9. 9. Copyright 2017 Severalnines ABCopyright 2017 Severalnines AB Devops and Databases - What are the Challenges?
  10. 10. Copyright 2017 Severalnines ABCopyright 2017 Severalnines AB ● Collaboration and communication between Dev and Ops ○ Change in culture and environment ○ Work closely together ● More (frequent) development cycles A different approach to Development and Ops
  11. 11. Copyright 2017 Severalnines ABCopyright 2017 Severalnines AB ● RDBMS databases are mostly built around securing the integrity of the data ○ Schema changes are slow and will create locks ○ Frequent changes will inevitably cause production issues ● NoSQL databases have schemaless design ○ Trade off of flexibility over consistency How do databases fit in DevOps?
  12. 12. Copyright 2017 Severalnines ABCopyright 2017 Severalnines AB ● DevOps roles are less clearly defined ○ Developer is now involved in operations ○ Ops is now actively involved in development release cycles ○ The whole team shares responsibility over the application ● A DBA is a too specialized role ○ Collaborative role between developers and ops ○ Developer will drive initiative, changes and performance ○ Ops will handle consistency and security Where does the DBA fit in DevOps?
  13. 13. Copyright 2017 Severalnines ABCopyright 2017 Severalnines AB ● Deployment automation for frequent release cycles ○ Continuous deployment needs quick environment deployments ○ Full environment also requires database copies ● Incompatibility between databases and microservices ○ Microservices have a shared nothing architecture ○ Databases generally have a relational schema ● Collaboration ○ Communication is key ○ Every team member needs to be up to date at all times ○ Collaboration tools can help What challenges are there for Databases and DevOps?
  14. 14. Copyright 2017 Severalnines ABCopyright 2017 Severalnines AB ● Get all credentials in code, whether you manage the resource or not. - Start by getting access to the resources your are to manage. - Store these details encrypted. - Share/Transfer/Make available these credentials securely ● Weird code being deployed, causing outages or undesired performance degradation. - Developers can push code that cause outages and performance degradation, I use Newrelic, Cloudwatch, Kibana for high level performance dashboards. ● Once you do have this resource under DevOps control, how do you present it to the app? - Capistrano, Puppet and Composer for example can write config files, I use a combination of Capistrano executing Composer that use DotEnv https://github.com/vlucas/phpdotenv ● Alerting, who’s responsible (and able) to get up at 3am and solve a complex outage, caused by code, you have no knowledge of, pending on application versions you have no authority to deploy or commit to? Feedback loops? - Slack, Collectd, Kibana, New Relic, Cloudwatch What are the challenges
  15. 15. Copyright 2017 Severalnines ABCopyright 2017 Severalnines AB Managing Databases in a DevOps environment
  16. 16. Copyright 2017 Severalnines ABCopyright 2017 Severalnines AB ● Microservices are very popular among DevOps ○ Easy to deploy ○ Strong cohesion ○ Loose coupling ● Shared-nothing databases ○ Simple schema designs ○ Interchangeable with any other storage system ● How to join data between microservices? The impact of microservice architectures
  17. 17. Copyright 2017 Severalnines ABCopyright 2017 Severalnines AB ● Direct querying creates a dependency between them ○ Any change to the schema could make the depending microservice to fail ○ Any infrastructural change needs to be propagated to the depending mircoservice ○ Duplicated configurations can get out of sync How to join data between microservices?
  18. 18. Copyright 2017 Severalnines ABCopyright 2017 Severalnines AB ● The only proper way would be by making a request to the API ○ In case of a slow response or error the microservice should be resilient ○ Generic error handling can be used for multiple (depending) microservices ○ This creates data layer abstraction ● Microservices can now be independently changed ○ Non-standard datastores can be used ○ Migrations are much easier How to join data between microservices?
  19. 19. Copyright 2017 Severalnines ABCopyright 2017 Severalnines AB ● Full environment deployments ○ Release development/testing ○ Disaster recovery (testing) ● Automatic database deployments ○ Restore/copy (production) data sets ○ Install necessary packages and helper software ○ Create necessary (application users) ○ Setup load balancers ● Automation frameworks ○ Puppet ○ Chef ○ Ansible Automated deployments
  20. 20. Copyright 2017 Severalnines ABCopyright 2017 Severalnines AB ● Make sure insights are available for anyone in the team ● Create dashboards around application / services ● Incorporate existing application / service metrics with (detailed) database metrics ○ Correlation of these metrics can help in tracing problems ○ Pre-emptive troubleshooting when (application) latency increases ● Application performance monitoring systems lack database performance metrics ○ No insight in what happens inside the database ○ Only insight from the application point of view Performance monitoring
  21. 21. Copyright 2017 Severalnines ABCopyright 2017 Severalnines AB ● Rapid and continuous development may lead to many schema changes ● Adding new columns and tables is no problem ○ Altering existing columns will create structural changes ● ORMs are known to generate frequent schema changes ● Not all ORMs can properly handle schema changes in production ○ Requires create / alter rights ○ Applies changes directly to any connection ● Automate using available tools ○ Percona Online Schema Change ○ Gh-ost Schema changes
  22. 22. Copyright 2017 Severalnines ABCopyright 2017 Severalnines AB ● New versions contain (important) bug fixes and resolve vulnerabilities ○ Upgrade regularly ● Minor version vs major versions ○ Minor versions are in-place replacements ○ Major versions (may) require dumping/loading of data ● Upgrades are routine for sysadmins and DBAs ○ DevOps may not be that experienced ○ Automate as much as possible ● Retain high availability ○ Upgrade slaves first and then promote an upgraded slave to become master ○ Use a proxy to divert traffic from nodes that get upgraded ● Test scenarios through Version upgrades
  23. 23. Copyright 2017 Severalnines ABCopyright 2017 Severalnines AB ● Microservices should be made resilient against failure ● Topology changes should not result in failure ● Master node fails ○ Slave promotion necessary ○ This may take up to 20 seconds to happen ● Failover handling ○ Configfile: topology discovery and failover has to be handled by the application ○ Resource manager: provides topology to the application, (may) handle failover ○ Load Balancer / Proxy: provides connections to the application and handles failover Automated failover
  24. 24. Copyright 2017 Severalnines ABCopyright 2017 Severalnines AB ● One of the most important aspects of DevOps is the communication between members ● Improve communication ○ Put people physically in a team? ○ Distributed teams? ○ Enrich communication channels ● ChatOps ○ Integrate (vital) information as metrics, alerts and status ○ Add database commands ○ Pre-deployment backups ○ Schema changes Chatops integrations
  25. 25. Copyright 2017 Severalnines ABCopyright 2017 Severalnines AB Data distribution
  26. 26. Copyright 2017 Severalnines ABCopyright 2017 Severalnines AB ● Not all data may be in the same location ○ Multi data center topologies ○ Disaster recovery ○ Uneven (storage) usage ● Sharding will separate data ○ Functional sharding by service separation ○ Horizontal sharding by every growing data Data locality
  27. 27. Copyright 2017 Severalnines ABCopyright 2017 Severalnines AB ● Microservices abstract (database) complexity away from the application ● Microservices increases complexity on infrastructure level ○ Data locality ○ Sharding ○ Schema changes ● Load balancer / proxy will abstract the infrastructure complexity away ○ Help with maintenance ○ Automatic scaling Managing data flows
  28. 28. Copyright 2017 Severalnines ABCopyright 2017 Severalnines AB Cloud Automation on AWS
  29. 29. Copyright 2017 Severalnines ABCopyright 2017 Severalnines AB Atom is a great IDE for DevOps, developed by https://github.com https://atom.io https://atom.io/packages/hiera-eyaml Using Hiera EYAML, you can encrypt credentials, and store this under version control Mailvelope is an easy to use PGP client for popular browsers and email provider to send secure emails https://www.mailvelope.com/en/ Store and Share credentials Securely
  30. 30. Copyright 2017 Severalnines ABCopyright 2017 Severalnines AB DotEnv to Autoload credentials, developed by https://github.com/vlucas/phpdotenv Store and Share credentials Securely continued..
  31. 31. Copyright 2017 Severalnines ABCopyright 2017 Severalnines AB https://gist.github.com/star3am/d5c84ba4a6a00f6abdb57d1d0a50fac3 Managing DBs in a DevOps environment
  32. 32. Copyright 2017 Severalnines ABCopyright 2017 Severalnines AB # Hiera hash profile_mysql::servers: company-stack-env-role: endpoint: 'company-stack-env-role.private.vpc' user: 'root' password: ENC[PKCS7,MIIBi..==] # Hiera Eyaml - https://puppet.com/blog/encrypt-your-data-using-hiera-eyaml databases: database_env: 'present' test_database_env: 'absent' # Set .my.cnf in root with above hira details so we can manage the resources $servers = hiera_hash('profile_mysql::servers', {}) file { '/root/.my.cnf': content => template("profile_mysql/root/.my.cnf"), } Managing DBs in a DevOps environment
  33. 33. Copyright 2017 Severalnines ABCopyright 2017 Severalnines AB # Call the defined type to create the DBs create_resources('profile_mysql::databases', $servers) # Puppet ERB template profile_mysql/root/.my.cnf <% @servers.each_pair do |key, value_hash| %> # use on the command line: mysql --defaults-group-suffix=<%= key %> [client<%= key %>] user=<%= value_hash['user'] %> host=<%= value_hash['endpoint'] %> password=<%= value_hash['password'] %> connect_timeout=2 <% end %> Managing DBs in a DevOps environment
  34. 34. Copyright 2017 Severalnines ABCopyright 2017 Severalnines AB define profile_mysql::databases ( $profile = $name, $endpoint, $user, $password, $databases, ) { $databases.each |$database_data| { $database = $database_data[0] $ensure = $database_data[1] if $ensure == 'present' { exec { "${ensure}-${profile}-${database}": unless => "/usr/bin/mysql --defaults-file=/root/.my.cnf --defaults-group-suffix=${profile} -e "show databases;" | grep -q "${database}" 2>/dev/null", command => "/usr/bin/mysql --defaults-file=/root/.my.cnf --defaults-group-suffix=${profile} -e "create database ${database};"", require => Class['mysql::client'], } } Managing DBs in a DevOps environment..
  35. 35. Copyright 2017 Severalnines ABCopyright 2017 Severalnines AB Demo of DevOps features in ClusterControl
  36. 36. Copyright 2017 Severalnines ABCopyright 2017 Severalnines AB Q & A
  37. 37. Copyright 2017 Severalnines ABCopyright 2017 Severalnines AB ● Riaan Nolan’s blog on DevOps http://www.riaannolan.com ● DevOps whitepaper https://severalnines.com/resources/whitepapers#download_whitepaper/5073 ● DevOps @ Severalnines blog https://severalnines.com/blog?categories=306 ● ClusterControl Command Line Interface (CLI) https://severalnines.com/product/clustercontrol/command-line-client-cli Additional resources
  38. 38. Copyright 2017 Severalnines AB Thank You!

×