Master Chef class: learn how to quickly cook delightful CQ/AEM infrastructures

1. © 2014 Adobe Systems Incorporated. All Rights Reserved. Master Chef class https://github.com/francoisledroff/connectcon-chef-repo http://www.slideshare.net/francoisledroff/master-chef-class-learn-how-to-quickly-cook-delightful-cqaem-infrastructures Francois Le Droff – Nicolas Peltier Let’s cook delightful AEM infrastructures /master-chef-class-learn-how-to-quickly-cook-delightful-cqaem-infrastructures

7. © 2014 Adobe Systems Incorporated. All Rights Reserved. Infrastructure ? noun ˈin-frə-ˌstrək-chər [1] A Collection of –!Resources: •! Network nodes •! File systems, files, folders, symbolic links, disk mounts •! Users, groups •! Packages, software •! Configurations –!Acting in concert –!To oﬀer a service –! [1] Introduction to Chef

8. © 2014 Adobe Systems Incorporated. All Rights Reserved. Infrastructure as Code? •! Does not only replace your shell scripts •! Does allow: –! to build your infra from a set of bare metal servers (called Nodes) Even on heterogeneous OS and Architecture –! to abstract the convergence of your infrastructure (your Nodes) through code –! to make this (Nodes) convergence idempotent State A --> State B (ex: install) Chef Client finished, 147/153 resources updated State B --> State B (ex: check) Chef Client finished, 0/153 resources updated State C --> State B (ex: back to normal) Chef Client finished, 48/153 resources updated

9. © 2014 Adobe Systems Incorporated. All Rights Reserved. •! Created in 2009, Edited by Opscode/Chef •! Apache License •! On-top-of/in Ruby •! Very active community •! http://community.opscode.com •! #learnChef •! https://learnchef.opscode.com/ Chef ? p://community.opscode.com

11. © 2014 Adobe Systems Incorporated. All Rights Reserved. Recipes Fundamental configuration element •! Authored in Ruby •! Queries , defines attributes •! Manipulates Resources •! Manipulates LW Resources •! Leverages Libraries & Templates •! Stored in Cookbooks –! Default Attributes –! Recipes –! LW Resource Providers –! Libraries –! Templates recipes cookbooks

12. © 2014 Adobe Systems Incorporated. All Rights Reserved. •! aem-cookbook/recipes/default.rb case node['platform_family'] when 'rhel' log 'this platform family is supported' do level :info end else log 'this platform family is not supported' do level :warn end end •! aem-cookbook/attributes/node.rb default['aem']['mode'] = 'publish' default['aem']['port'] = '4503' default['aem']['url'] = 'http://localhost:'+ node['aem'] ['port'] default['aem']['mode'] = 'publish' default['aem']['port'] = '4503' default['aem']['url'] = 'http://localhost:'+ node['aemaem'] ['port'] aem-cookbook/recipes/default.rb case node['platform_family'] when 'rhel' log 'this platform family is supported' do level :info end else log 'this platform family is not supported' do level :warn end end Attributes attributes

13. © 2014 Adobe Systems Incorporated. All Rights Reserved. Recipes : Resources & LW Resources •! aem-cookbook/recipes/default.rb remote_file node['aem']['quickstart_jar'] do source node['aem']['repo_url'] owner node['aem']['user'] group node['aem']['group'] checksum '3043859473' action :create_if_missing notifies :restart, 'service[aem]', :delayed end or artifact_file node['aem']['quickstart_jar'] do location 'com.day.cq:cq5:jar:5.5.0.20120220' nexus_configuration nexus_configuration_object owner node['aem']['user'] group node['aem']['group'] notifies :restart, 'service[aem]', :delayed end artifact_file node['aem']['quickstart_jar'] do location 'com.day.cq:cq5:jar:5.5.0.20120220' nexus_configuration nexus_configuration_object owner node['aem']['user'] group node['aem']['group'] notifies :restart, 'service[aem]', :delayed end remote_file node['aem']['quickstart_jar'] do source node['aem']['repo_url'] owner node['aem']['user'] group node['aem']['group'] checksum '3043859473' action :create_if_missing notifies :restart, 'service[aem]', :delayed end recipes

14. © 2014 Adobe Systems Incorporated. All Rights Reserved. Recipes : AEM LW Resources •! tacit-aem-cookbook/recipes/author.rb node[:aem][:author][:deploy_pkgs].each do |pkg| aem_package pkg[:name] do version pkg[:version] aem_instance ‘author’ package_url pkg[:url] update pkg[:update] user node[:aem][:author][:admin_user] password node[:aem][:author][:admin_password] port node[:aem][:author][:port] group_id pkg[:group_id] recursive pkg[:recursive] properties_file pkg[:properties_file] version_pattern pkg[:version_pattern] action pkg[:action] end end node[:aem][:author][:deploy_pkgs].each do |pkg| aem_package pkg[:name] do version pkg[:version] aem_instance ‘author’ package_url pkg[:url] update pkg[:update] user node[:aem][:author][:admin_user] password node[:aem][:author][:admin_password] port node[:aem][:author][:port] group_id pkg[:group_id] recursive pkg[:recursive] properties_file pkg[:properties_file] version_pattern pkg[:version_pattern] action pkg[:action] end end recipes

15. © 2014 Adobe Systems Incorporated. All Rights Reserved. –! aem-cookbook/recipes/start.rb ruby_block 'block_until_cq_operational' do block do Chef::Log.info 'Waiting until CQ is listening on port '+node['aem']['port'] until CQHelper.service_listening?(node['aem']['port']) sleep 1 Chef::Log.info('.') end Chef::Log.info 'Waiting until the CQ default page is responding' test_url = URI.parse(node['aem']['url']) until CQHelper.endpoint_responding?(test_url) sleep 1 Chef::Log.info('.') end end action :nothing end Recipes: Libraries recipes © 2014 Adobe Systems Incorporated.© 2014 Adobe Systems Incorporated. ruby_block 'block_until_cq_operational' do block do Chef::Log.info 'Waiting until CQ is listening on port '+node['aem']['port'] until CQHelper.service_listening?(node['aem']['port']) sleep 1 Chef::Log.info('.') end Chef::Log.info 'Waiting until the CQ default page is responding' test_url = URI.parse(node['aem']['url']) until CQHelper.endpoint_responding?(test_url) sleep 1 Chef::Log.info('.') end end action :nothing end

16. © 2014 Adobe Systems Incorporated. All Rights Reserved. Cookbook patterns [1] •! Library cookbooks: –! https://github.com/RiotGames/artifact-cookbook –! https://github.com/francoisledroﬀ/chef-vault-util •! Application cookbooks –! https://github.com/tacitknowledge/aem-cookbook –! https://github.com/francoisledroﬀ/aem-cookbook –! https://github.com/onehealth-cookbooks/apache2 –! https://github.com/socrata-cookbooks/java –! https://github.com/hw-cookbooks/haproxy –! https://github.com/stevendanna/logrotate –! https://github.com/opscode-cookbooks/chef-splunk •! Organization specific –! Wrapper cookbooks –! Base cookbooks –! Environment cookbooks [1] http://blog.vialstudios.com/the-environment-cookbook-pattern/ Library cookbooks: ps://github.com/RiotGames/artifact-cookbook ps://github.com/francoisledro cookbooks

22. © 2014 Adobe Systems Incorporated. All Rights Reserved. workstation Git Chef-server nodes RSA key Auth ssh knife env. roles attributes recipes Chef-DK data bags cookbooks RSA Keys Search API cookbooksorg data bags attributes env. recipes node objectrun-list rolesWeb UI versions

23. © 2014 Adobe Systems Incorporated. All Rights Reserved. workstation Git Chef-server nodes RSA key Auth ssh RSA key Auth knife env. roles attributes recipes Chef-DK data bags cookbooks RSA Keys Search API cookbooksorg data bags attributes env. recipes node objectrun-list rolesWeb UI versions chef-clients RSA Key

24. © 2014 Adobe Systems Incorporated. All Rights Reserved. Search APIknife workstation env. roles attributes cookbooks recipes Git org Chef-DK github Nexus opscode rubygem data bags attributes env. data bags cookbooks recipes Chef-server nodes RSA key Auth ssh https yymaven redhat RSA Keys node objectrun-list rolesWeb UI versions RSA key Auth chef-clients RSA Key

27. © 2014 Adobe Systems Incorporated. All Rights Reserved. Use case 0 : Chef Automation –! Install the jdk –! Download the jar –! Install it as a service –! A few aem cookbooks on github •! https://github.com/francoisledroﬀ/aem-cookbook •! https://github.com/tacitknowledge/aem-cookbook •! https://github.com/manosriglis/chef-aem •! https://github.com/QVCItalia/chef-aem

28. © 2014 Adobe Systems Incorporated. All Rights Reserved. OSS from opscode and elsewhere Chef, Ruby, rvm, bundler knife workstation Chef-DK opscode $ curl –L https://www.opscode.com/chef/install.sh | sudo bash $ curl -sSL https://get.rvm.io | bash -s stable –- ruby=1.9.3 $ gem install bundler http://www.getchef.com/downloads/chef-dk

29. © 2014 Adobe Systems Incorporated. All Rights Reserved. workstation Git ssh A Chef repo in Git Every Chef automated infra needs a Chef Repository github $ git clone https://github.com/opscode/chef-repo.git Made available for you at https://github.com/francoisledroﬀ/connectcon-chef-repo

30. © 2014 Adobe Systems Incorporated. All Rights Reserved. node Get Few Machines/Nodes to deploy your code/infra to workstation ssh ~/workspace/github/connectcon-chef-repo on ! master! $ vagrant plugin install vagrant-berkshelf --plugin-version 2.0.1 $ cat Vagrantfile Vagrant.configure("2") do |config| config.vm.box = "CentOS-6.4-x86_64" config.vm.hostname = "connectcon-francois" config.vm.box_url = "http://developer.nrel.gov/downloads/vagrant-boxes/ CentOS-6.4-x86_64-v20130427.box" config.berkshelf.enabled = true End $ vagrant up $ vagrant ssh Welcome to your Vagrant-built virtual machine. [vagrant@connectcon-francois ~]$

31. © 2014 Adobe Systems Incorporated. All Rights Reserved. A Chef Server comes in 3 flavors •! On premise OS Chef Server •! On premise Enterprise Chef •! Hosted Enterprise Chef server •! Local Alternatives: •! Chef-zero •! Chef-solo Chef-server

32. © 2014 Adobe Systems Incorporated. All Rights Reserved. A Chef Org top-level entity for role-based access org Chef-server https (ldap) Auth Web UI ps (ldap) Authps (ldap) Auth https://chef.corp.adobe.com/organizations

33. © 2014 Adobe Systems Incorporated. All Rights Reserved. workstation Git ssh A few private keys to associate your new chef repo with your chef server user and org ~/workspace/github/connectcon-chef-repo on ! master! $ ll .chef total 24 connectcon-validator.pem knife.rb ledroff.pem org Chef-server Web UI https(ldap)Auth RSA Keys

34. © 2014 Adobe Systems Incorporated. All Rights Reserved. Bootstrap your nodes knife workstation Chef-server RSAkeyAuth node org RSA Keys chef-clients RSA Key RSAkeyAuth ~/workspace/github/connectcon-chef-repo on ! master! $ knife bootstrap <your-node-fqdn> --sudo -x <your-sudoer>

36. © 2014 Adobe Systems Incorporated. All Rights Reserved. Start Coding: Manage your Ruby dependencies •! Use Bundler for RubyGem ~/workspace/github/connectcon-chef-repo on ! master! $ cat Gemfile source 'https://rubygems.org’ gem 'chef', '~> 11.10.0’ gem 'berkshelf', '~> 3.1.3‘ $ bundle install

37. © 2014 Adobe Systems Incorporated. All Rights Reserved. Start Coding: Manage your Chef dependencies •! A few aem cookbooks on github •! https://github.com/francoisledroﬀ/aem-cookbook •! https://github.com/tacitknowledge/aem-cookbook •! Use Berkshelf $ cat Berksfile source "https://api.berkshelf.com” cookbook 'ntp', '~> 1.5.4' cookbook 'chef-client', '~> 3.2.0’ cookbook 'artifact', git: 'https://github.com/francoisledroff/artifact- cookbook.git', tag: '1.11.4’ cookbook 'aem', git: 'https://github.com/francoisledroff/aem-cookbook

38. © 2014 Adobe Systems Incorporated. All Rights Reserved. Upload your cookbooks chef-client and ntp declared in your chef server org knife workstation org Chef-DK Chef-server httpsRSAkeyAuth RSA Keyscookbooks ~/workspace/github/connectcon-chef-repo on ! master! $ berks install $ berks upload cookbooks

40. © 2014 Adobe Systems Incorporated. All Rights Reserved. Role Base name ”base” Description ”connecton base server role” run_list "recipe[chef-client::default]", "recipe[chef-client::delete_validation]", "recipe[ntp]", "recipe[openssl::upgrade]"

41. © 2014 Adobe Systems Incorporated. All Rights Reserved. Typical Roles connectcon-chef-repo/roles/ name ”publish” Description ”connecton aem publish server role” run_list ”role[base]”,”recipe[aem::publish]” Mongo Author Publish Dispatcher HA/LB name "lb_dispatcher" description "connectcon roles for lb" run_list ”role[base]”,”recipe[haproxy::app_lb]” override_attributes('haproxy' => {'app_server_role' => ’dispatcher’}) Base name ”base” Description ”connecton base server role” run_list "recipe[chef-client::default]", "recipe[chef-client::delete_validation]", "recipe[ntp]", "recipe[openssl::upgrade]"

42. © 2014 Adobe Systems Incorporated. All Rights Reserved. Apply Run-List to Nodesattributes https://chef.corp.adobe.com/organizations ~/workspace/github/connectcon-chef-repo on ! master! $ knife node list $ knife node edit one.node.fqdn

43. © 2014 Adobe Systems Incorporated. All Rights Reserved. Chef-client run [root@ot1slu010 ~]# sudo chef-client Starting Chef Client, version 11.12.8 resolving cookbooks for run list: ["chef-client::delete_validation", "ntp", “aem::author", “aem::start”] * remote_file[/apps/publish/aem-author-4502.jar] action create_if_missing (up to date) * file[/apps/publish/license.properties] action create (up to date) * cookbook_file[/apps/author/serverctl] action create (up to date) * execute[java] action run (skipped due to not_if) * service[aem] action enable (up to date) Recipe: aaem::start * ruby_block[block_until_cq_operational] action nothing (skipped due to action :nothing) * log[ensure_cq_is_running] action write Recipe: aaem::default * service[aem] action start - start service service[aem] Recipe: aaem::start * ruby_block[block_until_cq_operational] action create - execute the ruby block block_until_cq_operational Running handlers: Running handlers complete Chef Client finished, 2/27 resources updated in 9.770664 seconds

44. © 2014 Adobe Systems Incorporated. All Rights Reserved. So we have an Author : Use Case 0 AuthorAuthor { "name": ”<author-connectcon-fqdn>", "chef_environment": "dev”, "run_list": [ "role[author]" ] } $ knife node edit author-connectcon-fqdn

45. © 2014 Adobe Systems Incorporated. All Rights Reserved. Let’s add a publish : Use Case 1 { "name": ”<publish-connectcon-fqdn>", "chef_environment": "dev”, "run_list": [ "role[publish]" ] } $ knife node edit publish-connectcon-fqdn Publish Author

46. © 2014 Adobe Systems Incorporated. All Rights Reserved. And there is magic: Chef search API •! tacit-aem-cookbook/providers/replicator.rb https://github.com/tacitknowledge/aem-cookbook hosts = [] search(:node, %Q(role:"#{role}" AND aem_cluster_name:"#{cluster_name}")) do |n| log "Found host: #{n[:fqdn]}" hosts << { :ipaddress => n[:ipaddress], … :name => n[:fqdn] } end … hosts.each do |h| … end

48. © 2014 Adobe Systems Incorporated. All Rights Reserved. Aem-Replicator API aem_replicator "replicate_to_publish_servers" do local_user node[:aem][:author][:admin_user] local_password node[:aem][:author][:admin_password] local_port node[:aem][:author][:port] remote_hosts node[:aem][:author][:replication_hosts] dynamic_cluster node[:aem][:author][:find_replication_hosts_dynamically] cluster_name node[:aem][:cluster_name] cluster_role node[:aem][:publish][:cluster_role] type :publish action :add end •! tacit-aem-cookbook/recipes/author.rb https://github.com/tacitknowledge/aem-cookbook

49. © 2014 Adobe Systems Incorporated. All Rights Reserved. What about Secret Management ? aem_replicator "replicate_to_publish_servers" do local_user node[:aem][:author][:admin_user] local_password node[:aem][:author][:admin_password] local_port node[:aem][:author][:port] remote_hosts node[:aem][:author][:replication_hosts] dynamic_cluster node[:aem][:author][:find_replication_hosts_dynamically] cluster_name node[:aem][:cluster_name] cluster_role node[:aem][:publish][:cluster_role] type :publish action :add end •! tacit-aem-cookbook/recipes/author.rb https://github.com/tacitknowledge/aem-cookbook

50. © 2014 Adobe Systems Incorporated. All Rights Reserved. Git UX/Dev/QA/Ops dev dev-stable prod ps Chef-server https RSA private key Auth chef-client chef-client chef-client chef-client https RSA private key Auth •! Chef encrypted data bags •! Encrypted for •! admin users •! whitelisted nodes •! Managed by chef-vault ruby gem •! Git Back up •! Encrypted obviously Encrypted Data Bags with Chef-vault

51. © 2014 Adobe Systems Incorporated. All Rights Reserved. Chef-Vault in Action: include_recipe 'chef-vault-util::default' item = chef_vault_item(node['aem']['vault_accounts'], node['aem'] ['vault_accounts_cq_admin_item']) cq_admin_username = item[node['aem']['vault_accounts_username_property']] cq_admin_password = item[node['aem']['vault_accounts_password_property']] log 'the secret accounts credential was fetched from a chef-vault enabled encrypted data_bag for username '+ cq_admin_username do level :info end •! francois-aem-cookbook/recipes/secure.rb https://github.com/francoisledroﬀ/aem-cookbook https://github.com/francoisledroﬀ/chef-vault-util $ knife vault update accounts prod_cq_admin -J accounts/prod_cq_admin.json -A "ledroff," --mode client -S "role:aem_server environment:prod"

52. © 2014 Adobe Systems Incorporated. All Rights Reserved. Let’s add a Dispatcher : Use Case 2 Dispatcher Author Publish DispatcherDispatcher { "name": ”<dispatcher-connectcon-fqdn>", "chef_environment": "dev”, "run_list": [ "role[dispatcher]" ] } $ knife node edit dispatcher-connectcon-fqdn Another Search Publish Another Search

53. © 2014 Adobe Systems Incorporated. All Rights Reserved. Let’s Add a Load Balancer : Use Case 3 HA/LB Dispatcher { "name": ”<lb-connectcon-fqdn>", "chef_environment": "dev”, "run_list": [ "role[lb_dispatcher]" ] } $ knife node edit lb-connectcon-fqdn Author Publish Author Publish Publish Publish Dispatcher Dispatcher Dispatcher Another Search } DispatcherDispatcher DispatcherDispatcher Another Search

54. © 2014 Adobe Systems Incorporated. All Rights Reserved. •! myapp-cookbook/recipes/log.rb logrotate_app 'aem' do cookbook 'logrotate' path node['myapp']['log_dir'] frequency node['myapp']['logrotate']['frequency'] rotate node['myapp']['logrotate']['rotate'] end include_recipe 'it-splunkforwarder::default’ •! myapp-cookbook/attributes/log.rb default.splunkforwarder.inputs = [ {"input_path" => "#{node['aem']['log_dir']}/error.log", … ] default.splunkforwarder.inputs = [ {"input_path" => "#{node['aem']['log_dir']}/error.log", … ] Let’s add log monitoring: Use Case 4 myapp /recipes/log.rb logrotate_app 'aem' do cookbook 'logrotate' path node['myapp']['log_dir'] frequency node['myapp']['logrotate']['frequency'] rotate node['myapp']['logrotate']['rotate'] end include_recipe 'it-splunkforwarder::default’ Dispatcher Author Publish Author Publish Publish Publish Dispatcher Dispatcher DispatcherDispatcherDispatcher DispatcherDispatcher DispatcherDispatcherDispatcherDispatcher Publish DispatcherDispatcherDispatcherDispatcherDispatcherDispatcher Publish Publish DispatcherDispatcher DispatcherDispatcher Publish DispatcherDispatcherDispatcherDispatcher Add the above in the base role Add the above in the base role

58. © 2014 Adobe Systems Incorporated. All Rights Reserved. Search APIknife workstation env. roles attributes cookbooks recipes Git org Chef-DK github Nexus opscode rubygem data bags attributes env. data bags cookbooks recipes Chef-server nodes RSA key Auth ssh https yymaven redhat RSA Keys node objectrun-list rolesWeb UI versions RSA key Auth chef-clients RSA Key

60. © 2014 Adobe Systems Incorporated. All Rights Reserved. What Stuff ? •! Git Stuff •! Ruby stuff •! VM / Container stuff •! Cloud stuff •! Network stuff •! *nix tools and stuff •! OS Stuff –! package management –! services •! Chef Community Stuff •! Build and packaging stuff •! Continuous Delivery stuff •! Monitoring stuff •! Analytics stuff •! Messaging stuff •! Security Stuff

65. © 2014 Adobe Systems Incorporated. All Rights Reserved. Why ? “Accelerate, Simplify, Scale” •! Breaking down the wall of confusion •! As infra is code, it becomes: –! Testable –! Versionable –! Disposable –! Reproducible •! “If it’s not in the source control system, it does not exist” @bdelacretaz

Master Chef class: learn how to quickly cook delightful CQ/AEM infrastructures

François Le Droff

Master Chef class: learn how to quickly cook delightful CQ/AEM infrastructures