Autoscaling Distributed System with BOSH (Cloud Foundry Summit 2014)

5,468 views

Published on

Technical Track presented by Yudai Iwasaki, Lead Engineer, Software Innovation Center, NTT.

Yudai Iwasaki is a research engineer at NTT Software Innovation Center. He is a core member of the development team of Cloudn PaaS, which is a public PaaS solution provided by NTT Communications, and is leading the development of their deployment system for Cloud Foundry. He is also the leader of community relationships at NTT and is a member of the Japan Cloud Foundry Group, in which capacity he gives lectures on the structure of Cloud Foundry for Japanese Cloud Foundry developers. He is a developer of Nise BOSH, which is a light weight BOSH emulator, BOSH CloudStack CPI, and BOSH AutoScaler.

Published in: Technology, News & Politics
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
5,468
On SlideShare
0
From Embeds
0
Number of Embeds
3,245
Actions
Shares
0
Downloads
70
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

Autoscaling Distributed System with BOSH (Cloud Foundry Summit 2014)

  1. 1. © 2014 Nippon Telegraph and Telephone Corporation AUTOSCALING CLOUD FOUNDRY WITH BOSH Yudai Iwasaki NTT Service Innovation Laboratory Group CF Summit 2014
  2. 2. © 2014 Nippon Telegraph and Telephone Corporation •  Core developer of Cloudn PaaS •  Working on Cloud Foundry since 2012 –  Nise BOSH, BOSH CloudStack CPI and BOSH AutoScaler •  Twitter: @i_yudai
  3. 3. © 2014 Nippon Telegraph and Telephone Corporation Agenda •  Problem – System load is not fixed •  Solution – BOSH AutoScaler •  Getting Started
  4. 4. © 2014 Nippon Telegraph and Telephone Corporation Problem: System Load is Not Fixed
  5. 5. © 2014 Nippon Telegraph and Telephone Corporation Daily, Weekly, Monthly, and Yearly Peaks
  6. 6. © 2014 Nippon Telegraph and Telephone Corporation Daily, Weekly, Monthly, and Yearly Peaks
  7. 7. © 2014 Nippon Telegraph and Telephone Corporation Manually Scaling? Always monitor load?
  8. 8. © 2014 Nippon Telegraph and Telephone Corporation Impossible No one wants to do such a boring job
  9. 9. © 2014 Nippon Telegraph and Telephone Corporation Solution: BOSH AutoScaler
  10. 10. © 2014 Nippon Telegraph and Telephone Corporation BOSH AutoScaler •  Extension for BOSH •  Flexible scaling policies in deployment manifest files •  Special support for Cloud Foundry
  11. 11. © 2014 Nippon Telegraph and Telephone Corporation Architecture Overview
  12. 12. © 2014 Nippon Telegraph and Telephone Corporation Director BOSH BOSH Client BOSH Scaler (Listener) NATS BOSH NATS Collector AutoScaler (with CF Plugin) Heartbeat Heartbeat (subscribe) process (BOSH::Monitor::Event::Heartbeat) Event Processor process (BOSH::Monitor::Event::Base) Deploy (PUT /deployments) matching rules against log periodically Logging metrics(@buffers)run() Agent Agent Agent Agent Agent CF Varz Collector process (Scaler::CfVarzMetric) CF CF Component CF Component CF Component Collector TSDB Varz Load policies (GET /deployments)
  13. 13. © 2014 Nippon Telegraph and Telephone Corporation Director BOSH BOSH Client BOSH Scaler (Listener) NATS BOSH NATS Collector AutoScaler (with CF Plugin) Heartbeat Heartbeat (subscribe) process (BOSH::Monitor::Event::Heartbeat) Event Processor process (BOSH::Monitor::Event::Base) Deploy (PUT /deployments) matching rules against log periodically Logging metrics(@buffers)run() Agent Agent Agent Agent Agent CF Varz Collector process (Scaler::CfVarzMetric) CF CF Component CF Component CF Component Collector TSDB Varz Load policies (GET /deployments)
  14. 14. © 2014 Nippon Telegraph and Telephone Corporation Director BOSH Client BOSH Scaler (Listener) NATS BOSH NATS Collector AutoScaler (with CF Plugin) Heartbeat (subscribe) process (BOSH::Monitor::Event::Heartbeat) Event Processor process (BOSH::Monitor::Event::Base) Deploy (PUT /deployments) matching rules against log periodically Logging metrics(@buffers)run() Agent CF Varz Collector process (Scaler::CfVarzMetric) Collector TSDBLoad policies (GET /deployments)
  15. 15. © 2014 Nippon Telegraph and Telephone Corporation Director BOSH Client BOSH Scaler (Listener) NATS BOSH NATS Collector AutoScaler (with CF Plugin) Heartbeat (subscribe) process (BOSH::Monitor::Event::Heartbeat) Event Processor process (BOSH::Monitor::Event::Base) Deploy (PUT /deployments) matching rules against log periodically Logging metrics(@buffers)run() Agent CF Varz Collector process (Scaler::CfVarzMetric) Collector TSDBLoad policies (GET /deployments)
  16. 16. © 2014 Nippon Telegraph and Telephone Corporation Policy Definitions
  17. 17. © 2014 Nippon Telegraph and Telephone Corporation Defining policies •  Adding “scale” section •  Defining conditions for each job •  Various condition classes to define flexible policies -­‐-­‐-­‐   name:  cf     ....     scale:      jobs:          -­‐  name:  router              cooldown:  300              out:                  limit:  10                  unit:  2                  conditions:                      -­‐  class:  CpuAverage                          larger_than:  80                          duration:  300                      -­‐  class:  MemoryAverage                          larger_than:  90                          duration:  300              in:                  limit:  3                  conditions:                      -­‐  class:  CpuAverage                          smaller_than:  10                          duration:  300                      -­‐  class:  MemoryAverage                          smaller_than:  20                          duration:  300  
  18. 18. © 2014 Nippon Telegraph and Telephone Corporation Conditions: BOSH Heartbeat •  CpuAverage –  Average CPU percentage for duration •  MemoryAverage –  Average memory percentage for duration •  LoadAverage1 –  Latest Load Average in 1 minute •  LoadAverage5 –  Latest Load Average in 5 minutes •  LoadAverage15 –  Latest Load Average in 15 minutes
  19. 19. © 2014 Nippon Telegraph and Telephone Corporation Conditions: CF Plugin •  CFVarzAverage –  Average Varz value for duration DEA available_memory_ratio available_disk_ratio Router total_routes latency.1m HM9000 NumberOfRunningInstances NumberOfCrashedInstances etcd SendingRequestRate ReceivingRequestRate Loggregator Server receivedMessageCount numberOfWebsocketSinks Cloud Controller connection_count threadqueue.num_waiting
  20. 20. © 2014 Nippon Telegraph and Telephone Corporation Web UI
  21. 21. © 2014 Nippon Telegraph and Telephone Corporation
  22. 22. © 2014 Nippon Telegraph and Telephone Corporation
  23. 23. © 2014 Nippon Telegraph and Telephone Corporation
  24. 24. © 2014 Nippon Telegraph and Telephone Corporation Getting Started It’s easy to plug in
  25. 25. © 2014 Nippon Telegraph and Telephone Corporation https://github.com/nttlabs/bosh-scaler
  26. 26. © 2014 Nippon Telegraph and Telephone Corporation -­‐-­‐-­‐ #  BOSH  NATS  settings   nats:  &nats      uri:  mbus://192.168.50.4:21084      user:  nats      pass:  nats     #  BOSH  Director  REST  API  settings   rest:  &rest      endpoint_uri:  https://192.168.50.4:25555      user:  scaler      password:  scaler      disable_verify_certification:  true   collectors:      -­‐  class:  BoshNatsCollector          bosh_nats:  *nats          bosh_rest:  *rest      -­‐  class:  CfVarzTsdbCollector          port:  4567     listeners:      -­‐  class:  BoshScaler          bosh_rest:  *rest          interval:  60  #  seconds          buffer_size:  1000          ui:              enable:  true              port:  8888   Configuration
  27. 27. © 2014 Nippon Telegraph and Telephone Corporation ...   properties:      collector:          use_tsdb:  true          deployment_name:  cf      opentsdb:          #  your  AutoScaler  address          address:  192.168.15.139          port:  4567                 Cloud Foundry Manifest
  28. 28. © 2014 Nippon Telegraph and Telephone Corporation Optimize Running Costs with BOSH AutoScaler

×