Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Autoscale a self-healing cluster in OpenStack with Heat

122 views

Published on

Autoscale a self-healing cluster in OpenStack with heat
Example Template for this: https://github.com/ricolin/heat-templates/tree/master/automatic

Published in: Technology
  • Be the first to comment

Autoscale a self-healing cluster in OpenStack with Heat

  1. 1. AUTO-SCALE A SELF-HEALING CLUSTER IN OPENSTACK 2018 Việt Nam OpenInfraDay Rico Lin, irc: ricolin <rico.lin@easystack.cn> @ EasyStack Xin chào các bạn, Mình tên là Rico Lin, đến từ Đài Loan, lần đầu tiên sang Việt Nam, cảm thấy rất thích và vui. Hôm nay Mình sẽ chia sẽ cho các bạn, chủ đề là AUTO-SCALE A SELF-HEALING CLUSTER IN OPENSTACK October 2018
  2. 2. _____________ A _______________ _________ IN OPENSTACK
  3. 3. _____________ A _______________ CLUSTER IN OPENSTACK
  4. 4. A Unit in Application cluster Pool Network Subnet Loadbalancer Floating IP Heal monitor Pool Member Nova Nginx
  5. 5. Unit with Heat Software Deploy Nova Server What you can install with ● heat-config-ansible ● heat-config-apply-config ● heat-config-cfn-init ● heat-config-chef ● heat-config-docker-cmd ● heat-config-docker-compose ● heat-config-hiera ● heat-config-json-file ● heat-config-kubelet ● heat-config-puppet ● heat-config-salt ● heat-config-script And you can customize your own hook os-collect-config os-refresh-config os-apply-config kubelet-hook$ kubelet Webserver done config-notify Signal ● CCFN_SIGNAL ● TEMP_URL_SIGNAL ● NO_SIGNAL ● HEAT_SIGNAL ● ZAQAR_SIGNAL Software Config Pool Network Subnet Loadbalancer Floating IP Heal monitor Pool Member Nginx
  6. 6. Heat Container Agent
  7. 7. Heat container agents [sample in repo] Software Deploy Nova Server What you can install with ● heat-config-ansible ● heat-config-apply-config ● heat-config-cfn-init ● heat-config-chef ● heat-config-docker-cmd ● heat-config-docker-compose ● heat-config-hiera ● heat-config-json-file ● heat-config-kubelet ● heat-config-puppet ● heat-config-salt ● heat-config-script And you can customize your own hook os-collect-config os-refresh-config os-apply-config kubelet-hook$ kubelet Webserver done config-notify Signal ● CCFN_SIGNAL ● TEMP_URL_SIGNAL ● NO_SIGNAL ● HEAT_SIGNAL ● ZAQAR_SIGNAL Dockers Software Config Pool Network Subnet Loadbalancer Floating IP Heal monitor Pool Member
  8. 8. Heat container agents [sample in repo] config: type: OS::Heat::SoftwareConfig properties: group: script outputs: - name: result config: { get_file: example-script.sh } deployment: type: OS::Heat::SoftwareDeployment properties: config: { get_resource: config } server: { get_resource: server } start_container_agent: type: OS::Heat::SoftwareConfig properties: group: ungrouped config: {get_file: ./start-container-agent.sh} server: type: OS::Nova::Server properties: image: {get_param: image} flavor: {get_param: flavor} key_name: {get_param: key_name} networks: - network: {get_param: private_net} security_groups: - {get_resource: the_sg} user_data_format: SOFTWARE_CONFIG user_data: {get_attr: [start_container_agent, config]} #!/bin/bash set -ux # heat-docker-agent service cat <<EOF > /etc/systemd/system/heat-container-agent.service [Unit] Description=Heat Container Agent After=docker.service Requires=docker.service [Service] TimeoutSec=5min RestartSec=5min User=root Restart=on-failure ExecStartPre=-/usr/bin/docker rm -f heat-container-agent ExecStartPre=-/usr/bin/docker pull docker.io/rico/heat-container-agent ExecStart=/usr/bin/docker run --name heat-container-agent --privileged --net=host -v /run/systemd:/run/systemd -v /etc/sysconfig:/etc/sysconfig -v /etc/systemd/system:/etc/systemd/system -v /var/lib/heat-cfntools:/var/lib/heat-cfntools -v /var/lib/cloud:/var/lib/cloud -v /tmp:/tmp -v /etc/hosts:/etc/hosts docker.io/rico/heat-container-agent ExecStop=/usr/bin/docker stop heat-container-agent [Install] WantedBy=multi-user.target EOF # enable and start heat-container-agent chmod 0640 /etc/systemd/system/heat-container-agent.service /usr/bin/systemctl enable heat-container-agent.service /usr/bin/systemctl start --no-block heat-container-agent.service
  9. 9. Demo
  10. 10. _____________ A SELF-HEALING CLUSTER IN OPENSTACK
  11. 11. Self Healing XXX::Server XXX::Signal XXX::Alarm XXX::Workflow Signal Meter Trigger XXX::AutoScaling Fix
  12. 12. Self Healing XXX::Server XXX::Signal XXX::Alarm XXX::Workflow Signal Meter Trigger XXX::AutoScaling How you metering? How you handle signal? How you trigger a fix job What's meter to you? Fix
  13. 13. Self Healing server: type: OS::Nova::Server properties: ... alarm_queue: type: OS::Zaqar::Queue error_event_alarm: type: OS::Aodh::EventAlarm properties: event_type: compute.instance.update query: - field: traits.instance_id value: {get_resource: server} op: eq - field: traits.state value: error op: eq alarm_queues: - {get_resource: alarm_queue} alarm_subscription: type: OS::Zaqar::MistralTrigger properties: queue_name: {get_resource: alarm_queue} workflow_id: {get_resource: autoheal} input: stack_id: {get_param: "OS::stack_id"} root_stack_id: if: - is_standalone - {get_param: "OS::stack_id"} - {get_param: "root_stack_id"} autoheal: type: OS::Mistral::Workflow properties: description: > Mark a server as unhealthy and commence a stack update to replace it. input: stack_id: root_stack_id: type: direct tasks: - name: resources_mark_unhealthy action: list_join: - ' ' - - heat.resources_mark_unhealthy - stack_id=<% $.stack_id %> - resource_name=<% env().notification.body.reason_data.event.traits.where($[0] = 'instance_id').select($[2]).first() %> - mark_unhealthy=true - resource_status_reason='Marked by alarm' on_success: - stacks_update - name: stacks_update action: heat.stacks_update stack_id=<% $.root_stack_id %> existing=true
  14. 14. OpenStack Self Healing SIG[link]
  15. 15. Demo
  16. 16. AUTO-SCALE A SELF-HEALING CLUSTER IN OPENSTACK
  17. 17. Auto Scaling AutoScalingGroup ScalingPolicy XXX::Alarm Signal Meter Trigger Scale
  18. 18. Auto Scaling AutoScalingGroup ScalingPolicy XXX::Alarm Signal Meter Trigger What to Alarm Scale What to scale
  19. 19. Auto Scaling https://github.com/openstack/heat-templates/tree/master/hot/autoscaling.yaml resources: asg: type: OS::Heat::AutoScalingGroup properties: min_size: 1 max_size: 3 resource: type: lb_server.yaml properties: flavor: {get_param: flavor} image: {get_param: image} web_server_scaleup_policy: type: OS::Heat::ScalingPolicy properties: adjustment_type: change_in_capacity auto_scaling_group_id: {get_resource: asg} cooldown: 60 scaling_adjustment: 1 # min_adjustment_step: web_server_scaledown_policy: type: OS::Heat::ScalingPolicy properties: adjustment_type: change_in_capacity auto_scaling_group_id: {get_resource: asg} cooldown: 60 scaling_adjustment: -1 cpu_alarm_high: type: OS::Aodh::GnocchiAggregationByResourcesAlarm properties: description: Scale up if CPU > 80% metric: cpu_util aggregation_method: mean granularity: 300 evaluation_periods: 1 threshold: 80 resource_type: instance comparison_operator: gt alarm_actions: - str_replace: template: trust+url params: url: {get_attr: [web_server_scaleup_policy, signal_url]} query: list_join: - '' - - {'=': {server_group: {get_param: "OS::stack_id"}}} cpu_alarm_low: type: OS::Aodh::GnocchiAggregationByResourcesAlarm
  20. 20. resources: asg: type: OS::Heat::AutoScalingGroup properties: min_size: 1 max_size: 3 resource: type: lb_server.yaml properties: flavor: {get_param: flavor} image: {get_param: image} web_server_scaleup_policy: type: OS::Heat::ScalingPolicy properties: adjustment_type: change_in_capacity auto_scaling_group_id: {get_resource: asg} cooldown: 60 scaling_adjustment: 1 monitoring: type: monitor.yaml properties: url: get_attr: [web_server_scaleup_policy, signal_url] ScalingPolicy Stack Monitor service AutoScalingGroup Instance 1 1.Metering 2 N 2.Alarm 3.Scale Choose your own structure
  21. 21. resources: asg: type: OS::Heat::AutoScalingGroup properties: min_size: 1 max_size: 3 resource: type: lb_server.yaml properties: flavor: {get_param: flavor} image: {get_param: image} web_server_scaleup_policy: type: OS::Heat::ScalingPolicy properties: adjustment_type: change_in_capacity auto_scaling_group_id: {get_resource: asg} cooldown: 60 scaling_adjustment: 1 outputs: signal_url: value: {get_attr: [web_server_scaleup_policy, signal_url]} ScalingPolicy Stack Monitor service AutoScalingGroup Instance 1 1.Metering 2 N 2.Alarm 3.Scale Choose your own structure
  22. 22. resources: asg: type: OS::Heat::AutoScalingGroup properties: min_size: 1 max_size: 3 resource: type: lb_server.yaml properties: flavor: {get_param: flavor} image: {get_param: image} web_server_scaleup_policy: type: OS::Heat::ScalingPolicy properties: adjustment_type: change_in_capacity auto_scaling_group_id: {get_resource: asg} cooldown: 60 scaling_adjustment: 1 outputs: signal_url: value: {get_attr: [web_server_scaleup_policy, signal_url]} ScalingPolicy Stack Monitor service AutoScalingGroup Instance 1 1.Metering 2 N 2.Alarm 3.Scale Choose your own structure curl -i -H "X-Auth-Token: $TOKEN" -X POST $Signal_url curl -i -H "Content-Type: application/json" -d '{ "auth": { "identity": { "methods": ["password"], "password": { "user": { "name": "admin", "domain": { "id": "default" }, "password": "password" } } }, "scope": { "project": { "name": "admin", "domain": { "id": "default" } } } }}' http://$KEYSTONE/identity/v3/auth/tokens ; echo
  23. 23. Look into options for auto-scaling OS::Heat::AutoScalingGroup ● Properties ○ resource: ■ type: web_server.yaml ■ properties ○ min_size: 10 ○ max_size: 100 ○ cooldown: 30 ○ desired_capacity: 30 ○ rolling_updates ■ min_in_service: 5 ■ max_batch_size: 10 ■ pause_time: 15 ● Attributes ○ outputs ○ outputs_list ○ current_size ○ refs [IDs] ○ refs_map {[names: IDs]}
  24. 24. Look into options for auto-scaling OS::Heat::ScalingPolicy ● Properties ○ adjustment_type: change_in_capacity ■ exact_capacity ■ change_in_capacity ■ percent_change_in_capacity ○ auto_scaling_group_id: asg_id ○ cooldown: 60 ○ scaling_adjustment: 5 ○ # min_adjustment_step: ● Attributes ○ alarm_url ○ signal_url
  25. 25. Demo
  26. 26. • Review https://goo.gl/4KL1gN • StoryBoard (Bugs/BP) https://storyboard.openstack.org/#!/project_group/82 • StoryBoard guide https://etherpad.openstack.org/p/Heat-StoryBoard-Migration-Info • Documents https://docs.openstack.org/heat/latest/ • Release Notes https://docs.openstack.org/releasenotes/heat/ • Feedback or Provide ideas = irc: #heat • Feedback your Use cases https://etherpad.openstack.org/p/heat-usecases • Team meeting time Wednesday 14:00 UTC #heat (meeting wiki and archive) Join Heat ➔ Boston Summit ◆ Heat project update [ slide & video ] ◆ Heat Onboarding [ slide & video ] ➔ Sydney Summit ◆ Heat project update [ slide & video ] ◆ Heat Onboarding [ slide & video ] ➔ Vancouver Summit ◆ Heat project update [ slide & video ] ◆ Heat Onboarding [ slide & video ] ➔ Heat templates ➔ PTG Etherpad
  27. 27. Q & A Links: demo video If you wondering what your product or you can interact with Open Source Cloud community: Embrace community! Embrace Life!

×