SlideShare a Scribd company logo
© 2015 NTT Software Innovation Center
仮想マシンの高可用性と
異常系試験フレームワーク
Sampath Priyankara
SIC I三P I方D
2Copyright©2015 NTT corp. All Rights Reserv
仮想マシンの高可用性
Masakari
3Copyright©2015 NTT corp. All Rights Reserv
What is Masakari?
Service : Instances High Availability Service
Mission:
Provide instances high availability service
for OpenStack clouds by automatically
recovering the instances from failures.
https://governance.openstack.org/tc/refer
ence/projects/masakari.html
4Copyright©2015 NTT corp. All Rights Reserv
Masakri in action
Virtual Machine High Availability
with Platform9 OpenStack
• https://platform9.com/blog/virtual-machine-high-
availability-platform9-openstack/
• https://www.openstack.org/summit/boston-
2017/summit-schedule/events/17502/zero-touch-
high-availability-with-masakari-and-consul
5Copyright©2015 NTT corp. All Rights Reserv
Project History
[2015 Jul] Started on Github
https://github.com/ntt-sic/masakari [no longer maintained]
[2016 Jun] Moved into OpenStack name space and rebuild
[2017 Oct] Became official OpenStack project
Related work:
[openstack-ha team]
Define spcs for converged upstream solution for VMHA
Wiki: https://wiki.openstack.org/wiki/Meetings/HATeamMeeting
spec: https://review.openstack.org/#/q/project:openstack/openstack-resource-agents-
specs
[OpenStack Product Working Group]
Development Proposal for VMHA
Wiki: https://wiki.openstack.org/wiki/ProductTeam
Feature: http://featuretracker.openstack.org/projectDetail/0003
6Copyright©2015 NTT corp. All Rights Reserv
Project Structure
Masakari
• Masakari
• Masakari Engine and APIs
• Repositories: openstack/masakari
• masakari-monitors
• Monitors to detect failures and send notifications to masakari API
• Repositories: openstack/masakari-monitors
• python-masakariclient
• Python wrapper for masakari APIs
• Repositories: openstack/python-masakariclient
• Masakari Specifications
• masakari-specs
• Specifications for all masakari projects
• Repositories: openstack/masakari-specs
7Copyright©2015 NTT corp. All Rights Reserv
Masakari Architecture
masakari
masakari-monitors
python-masakariclient
8Copyright©2015 NTT corp. All Rights Reserv
Masakri Features
 Supported failure models
• Host failures, Process failures, Instance failures
Full set of monitors to detect failures
Client Library for easy operation
Customizable host recovery work flow
• Auto(use nova scheduler), Reserved host, Hybrid
 Masakari APIs:
Failover Segments (segments):
Define boundaries and behaviors for failover
Hosts (hosts):
Define compute nodes and Reserved compute nodes
Notifications (notifications):
Receive failure notifications from monitors
9Copyright©2015 NTT corp. All Rights Reserv
Current contributors
Mitaka to Queens In Queens
http://stackalytics.com/?module=masakari&release=all http://stackalytics.com/?module=masakari&release=queens
1
0Copyright©2015 NTT corp. All Rights Reserv
Work in progress
• Recovery Method customization
• [spec] https://review.openstack.org/#/c/458023/
• Intrusive Instance Monitoring
• [spec] https://review.openstack.org/#/c/469070/
• Add event notification feature to masakari
• [spec] https://review.openstack.org/#/c/473057/
• Masakari Horizon plugin
• In progress
1
1Copyright©2015 NTT corp. All Rights Reserv
Future work items
• Ironic Bare Metal Instance HA
• Volume boot Bare Metal Instance
• External monitoring mechanism
• Fencing mechanism
• Instance recovery work flow
• Force STONITH and Fencing
• Node health prediction
• Resource migration
• Forcefully bring down the node
• New masakri-monitors
• Something other than pacemaker. (ex: consul)
1
2Copyright©2015 NTT corp. All Rights Reserv
Masakari Community
Where to find us
IRC: #openstack-masakari
ML: openstack-dev with [masakari]
Weekly IRC meeting
Agenda:
https://wiki.openstack.org/wiki/Meetings/Masakari
On Tuesday at 0400 UTC
http://eavesdrop.openstack.org/#Masakari_Team_Meeti
ng
1
3Copyright©2015 NTT corp. All Rights Reserv
異常系試験フレームワーク
Eris
1
4Copyright©2015 NTT corp. All Rights Reserv
What is Eris?
Destructive test framework for OpenStack
• Test robustness/resiliency of OpenStack at the
CI/CD gates.
• Test different architectures on 3rd party CIs.
• Common test cases developed by community.
• Automated evaluation of KPIs and report
generation.
Current Team
• NTT, AT&T, Intel (LCOO WG members)
1
5Copyright©2015 NTT corp. All Rights Reserv
About Eris
Eris Wiki:
https://openstack-lcoo.atlassian.net/wiki/spaces/LCOO/pages/13393034/Eris+-
+Extreme+Testing+Framework+for+OpenStack
Summit Forum:
• https://etherpad.openstack.org/p/SYD-extreme-testing
• https://etherpad.openstack.org/p/LCOO-Extreme_Testing-QA-ERIS
Demo:
https://openstack-
lcoo.atlassian.net/wiki/spaces/LCOO/pages/22872097/Extreme+Testing+Demo
1
6Copyright©2015 NTT corp. All Rights Reserv
Form Discussion
Similar Efforts
 OPNFV Yardstick[1]
 enos: Experimental eNvironment for OpenStack
New requirements
 scale/performance testing for nova
 Integrate with OPNFV testing tools
Discussion with QA team
 PoC demo
 Discuss future road map
「1」https://wiki.opnfv.org/display/yardstick/Yardstick
1
7Copyright©2015 NTT corp. All Rights Reserv
Work Items
• Create SIG
• Discuss with other community
such as OPNFV, nova, Rally..etc.
• Update QA Spec with details in
Eris wiki
• Probably divide in to several spes
• Enhance Eris PoC and integrate
into Masakari CI
1
8Copyright©2015 NTT corp. All Rights Reserv
Self-healing SIG
1
9Copyright©2015 NTT corp. All Rights Reserv
Self-healing SIG
Manage OpenStack infrastructure in a policy-driven
fashion, reacting to failures and other events by
automatically healing or optimize services.
2
0Copyright©2015 NTT corp. All Rights Reserv
Self-healing SIG Members
HA of individual services
Monasca: monitoring
Aodh: alarming
Congress: policy-based governance
Mistral: workflow
Senlin: clustering service
Vitrage: root cause analysis
Watcher: optimization
Masakari: compute plane HA
Freezer-dr: compute plane HA
Heat: orchestration (normally used for cloud applications, but can also deploy
cloud infrastructure via TripleO)
Doctor: fault management and maintenance for NFV
Fault Genes Working Group: Fault classification & Recovery Strategy
Craton: Fleet management
Kolla: Containerized OpenStack deployment tool
kolla-k8s: same as above but in kubernetes cluster

More Related Content

What's hot

OpenStack Israel 2015 keynote
OpenStack Israel 2015 keynoteOpenStack Israel 2015 keynote
OpenStack Israel 2015 keynote
Nicolas (Nick) Barcet
 
So Your OpenStack Cloud is Built... Now What's Next - Walter Bentley - OpenSt...
So Your OpenStack Cloud is Built... Now What's Next - Walter Bentley - OpenSt...So Your OpenStack Cloud is Built... Now What's Next - Walter Bentley - OpenSt...
So Your OpenStack Cloud is Built... Now What's Next - Walter Bentley - OpenSt...
Cloud Native Day Tel Aviv
 
Automating OpenStack Deployment with Fuel
Automating OpenStack Deployment with FuelAutomating OpenStack Deployment with Fuel
Automating OpenStack Deployment with Fuel
Tomasz Zen Napierala
 
Shaker
ShakerShaker
Shaker
Ilya Shakhat
 
Bringing New Experience with Openstack and Fuel (Ihor Dvoretskyi, Oleksandr M...
Bringing New Experience with Openstack and Fuel (Ihor Dvoretskyi, Oleksandr M...Bringing New Experience with Openstack and Fuel (Ihor Dvoretskyi, Oleksandr M...
Bringing New Experience with Openstack and Fuel (Ihor Dvoretskyi, Oleksandr M...
IT Arena
 
OpenStack Dashboard Folsom Roadmap
OpenStack Dashboard Folsom RoadmapOpenStack Dashboard Folsom Roadmap
OpenStack Dashboard Folsom Roadmap
devcamcar
 
OpenStack reliability metrics
OpenStack reliability metricsOpenStack reliability metrics
OpenStack reliability metrics
Ilya Shakhat
 
Scalable Python with Docker, Kubernetes, OpenShift
Scalable Python with Docker, Kubernetes, OpenShiftScalable Python with Docker, Kubernetes, OpenShift
Scalable Python with Docker, Kubernetes, OpenShift
Aarno Aukia
 
Spinnaker Summit 2019: Where are we heading? The Future of Continuous Delivery
Spinnaker Summit 2019: Where are we heading? The Future of Continuous DeliverySpinnaker Summit 2019: Where are we heading? The Future of Continuous Delivery
Spinnaker Summit 2019: Where are we heading? The Future of Continuous Delivery
Andrew Phillips
 
CI and CD with Spinnaker
CI and CD with SpinnakerCI and CD with Spinnaker
CI and CD with Spinnaker
VMware Tanzu
 
Deployment Automation & Self-Healing with Dynatrace & Ansible
Deployment Automation & Self-Healing with Dynatrace & AnsibleDeployment Automation & Self-Healing with Dynatrace & Ansible
Deployment Automation & Self-Healing with Dynatrace & Ansible
Jürgen Etzlstorfer
 
Modern DevOps with Spinnaker - Olga Kundzich
Modern DevOps with Spinnaker - Olga KundzichModern DevOps with Spinnaker - Olga Kundzich
Modern DevOps with Spinnaker - Olga Kundzich
VMware Tanzu
 
A Tale of Two OpenStack Contributors: A Newbie Developer and a Frustrated Ope...
A Tale of Two OpenStack Contributors: A Newbie Developer and a Frustrated Ope...A Tale of Two OpenStack Contributors: A Newbie Developer and a Frustrated Ope...
A Tale of Two OpenStack Contributors: A Newbie Developer and a Frustrated Ope...
Cloud Native Day Tel Aviv
 
Dear IT...I'd Like A Kubernetes Cluster
Dear IT...I'd Like A Kubernetes ClusterDear IT...I'd Like A Kubernetes Cluster
Dear IT...I'd Like A Kubernetes Cluster
Shannon Williams
 
CNCF App-Delivery SIG Presentation - Litmus Chaos Engineering
CNCF App-Delivery SIG Presentation - Litmus Chaos EngineeringCNCF App-Delivery SIG Presentation - Litmus Chaos Engineering
CNCF App-Delivery SIG Presentation - Litmus Chaos Engineering
Umasankar Mukkara
 
Continuous Deployment to the Cloud using Spinnaker
Continuous Deployment to the Cloud using SpinnakerContinuous Deployment to the Cloud using Spinnaker
Continuous Deployment to the Cloud using Spinnaker
Tim Ysewyn
 
OpenStack and DevOps - DevOps Meetup
OpenStack and DevOps - DevOps MeetupOpenStack and DevOps - DevOps Meetup
OpenStack and DevOps - DevOps Meetup
Opsta
 
Why observability matters - now and in the future (w/guest Grafana)
Why observability matters - now and in the future (w/guest Grafana)Why observability matters - now and in the future (w/guest Grafana)
Why observability matters - now and in the future (w/guest Grafana)
Weaveworks
 
OSMC 2017 | Current State of Icinga by Erk Bernd
OSMC 2017 | Current State of Icinga by Erk BerndOSMC 2017 | Current State of Icinga by Erk Bernd
OSMC 2017 | Current State of Icinga by Erk Bernd
NETWAYS
 
OpenStack Summit Storlets Project Update Queens
OpenStack Summit Storlets Project Update QueensOpenStack Summit Storlets Project Update Queens
OpenStack Summit Storlets Project Update Queens
Kota Tsuyuzaki
 

What's hot (20)

OpenStack Israel 2015 keynote
OpenStack Israel 2015 keynoteOpenStack Israel 2015 keynote
OpenStack Israel 2015 keynote
 
So Your OpenStack Cloud is Built... Now What's Next - Walter Bentley - OpenSt...
So Your OpenStack Cloud is Built... Now What's Next - Walter Bentley - OpenSt...So Your OpenStack Cloud is Built... Now What's Next - Walter Bentley - OpenSt...
So Your OpenStack Cloud is Built... Now What's Next - Walter Bentley - OpenSt...
 
Automating OpenStack Deployment with Fuel
Automating OpenStack Deployment with FuelAutomating OpenStack Deployment with Fuel
Automating OpenStack Deployment with Fuel
 
Shaker
ShakerShaker
Shaker
 
Bringing New Experience with Openstack and Fuel (Ihor Dvoretskyi, Oleksandr M...
Bringing New Experience with Openstack and Fuel (Ihor Dvoretskyi, Oleksandr M...Bringing New Experience with Openstack and Fuel (Ihor Dvoretskyi, Oleksandr M...
Bringing New Experience with Openstack and Fuel (Ihor Dvoretskyi, Oleksandr M...
 
OpenStack Dashboard Folsom Roadmap
OpenStack Dashboard Folsom RoadmapOpenStack Dashboard Folsom Roadmap
OpenStack Dashboard Folsom Roadmap
 
OpenStack reliability metrics
OpenStack reliability metricsOpenStack reliability metrics
OpenStack reliability metrics
 
Scalable Python with Docker, Kubernetes, OpenShift
Scalable Python with Docker, Kubernetes, OpenShiftScalable Python with Docker, Kubernetes, OpenShift
Scalable Python with Docker, Kubernetes, OpenShift
 
Spinnaker Summit 2019: Where are we heading? The Future of Continuous Delivery
Spinnaker Summit 2019: Where are we heading? The Future of Continuous DeliverySpinnaker Summit 2019: Where are we heading? The Future of Continuous Delivery
Spinnaker Summit 2019: Where are we heading? The Future of Continuous Delivery
 
CI and CD with Spinnaker
CI and CD with SpinnakerCI and CD with Spinnaker
CI and CD with Spinnaker
 
Deployment Automation & Self-Healing with Dynatrace & Ansible
Deployment Automation & Self-Healing with Dynatrace & AnsibleDeployment Automation & Self-Healing with Dynatrace & Ansible
Deployment Automation & Self-Healing with Dynatrace & Ansible
 
Modern DevOps with Spinnaker - Olga Kundzich
Modern DevOps with Spinnaker - Olga KundzichModern DevOps with Spinnaker - Olga Kundzich
Modern DevOps with Spinnaker - Olga Kundzich
 
A Tale of Two OpenStack Contributors: A Newbie Developer and a Frustrated Ope...
A Tale of Two OpenStack Contributors: A Newbie Developer and a Frustrated Ope...A Tale of Two OpenStack Contributors: A Newbie Developer and a Frustrated Ope...
A Tale of Two OpenStack Contributors: A Newbie Developer and a Frustrated Ope...
 
Dear IT...I'd Like A Kubernetes Cluster
Dear IT...I'd Like A Kubernetes ClusterDear IT...I'd Like A Kubernetes Cluster
Dear IT...I'd Like A Kubernetes Cluster
 
CNCF App-Delivery SIG Presentation - Litmus Chaos Engineering
CNCF App-Delivery SIG Presentation - Litmus Chaos EngineeringCNCF App-Delivery SIG Presentation - Litmus Chaos Engineering
CNCF App-Delivery SIG Presentation - Litmus Chaos Engineering
 
Continuous Deployment to the Cloud using Spinnaker
Continuous Deployment to the Cloud using SpinnakerContinuous Deployment to the Cloud using Spinnaker
Continuous Deployment to the Cloud using Spinnaker
 
OpenStack and DevOps - DevOps Meetup
OpenStack and DevOps - DevOps MeetupOpenStack and DevOps - DevOps Meetup
OpenStack and DevOps - DevOps Meetup
 
Why observability matters - now and in the future (w/guest Grafana)
Why observability matters - now and in the future (w/guest Grafana)Why observability matters - now and in the future (w/guest Grafana)
Why observability matters - now and in the future (w/guest Grafana)
 
OSMC 2017 | Current State of Icinga by Erk Bernd
OSMC 2017 | Current State of Icinga by Erk BerndOSMC 2017 | Current State of Icinga by Erk Bernd
OSMC 2017 | Current State of Icinga by Erk Bernd
 
OpenStack Summit Storlets Project Update Queens
OpenStack Summit Storlets Project Update QueensOpenStack Summit Storlets Project Update Queens
OpenStack Summit Storlets Project Update Queens
 

Similar to OpenStack Sydney summit - OpenStack HA and Testing

Tempest scenariotests 20140512
Tempest scenariotests 20140512Tempest scenariotests 20140512
Tempest scenariotests 20140512
Masayuki Igawa
 
Webinar: OpenStack Best Practices for Production
Webinar: OpenStack Best Practices for ProductionWebinar: OpenStack Best Practices for Production
Webinar: OpenStack Best Practices for Production
Platform9
 
2307 - DevBCN - Otel 101_compressed.pdf
2307 - DevBCN - Otel 101_compressed.pdf2307 - DevBCN - Otel 101_compressed.pdf
2307 - DevBCN - Otel 101_compressed.pdf
DimitrisFinas1
 
OpenStack & MySQL
OpenStack & MySQLOpenStack & MySQL
OpenStack & MySQL
MySQL Brasil
 
Automated Deployment & Benchmarking with Chef, Cobbler and Rally for OpenStack
Automated Deployment & Benchmarking with Chef, Cobbler and Rally for OpenStackAutomated Deployment & Benchmarking with Chef, Cobbler and Rally for OpenStack
Automated Deployment & Benchmarking with Chef, Cobbler and Rally for OpenStack
NTT Communications Technology Development
 
Enterprise Ready OpenStack, Wiekus Beukes, Oracle
Enterprise Ready OpenStack,  Wiekus Beukes, OracleEnterprise Ready OpenStack,  Wiekus Beukes, Oracle
Enterprise Ready OpenStack, Wiekus Beukes, Oracle
Sriram Subramanian
 
OpenStack: Everything You Need to Know To Get Started
OpenStack: Everything You Need to Know To Get StartedOpenStack: Everything You Need to Know To Get Started
OpenStack: Everything You Need to Know To Get Started
All Things Open
 
OpenStack: Everything You Need To Know to Get Started (ATO2014)
OpenStack: Everything You Need To Know to Get Started (ATO2014)OpenStack: Everything You Need To Know to Get Started (ATO2014)
OpenStack: Everything You Need To Know to Get Started (ATO2014)
Mark Voelker
 
Using OpenStack to Accelerate New Product Development: Rik Harris, Telstra
Using OpenStack to Accelerate New Product Development: Rik Harris, TelstraUsing OpenStack to Accelerate New Product Development: Rik Harris, Telstra
Using OpenStack to Accelerate New Product Development: Rik Harris, Telstra
OpenStack
 
Toronto MuleSoft_Meetup_Run Time Fabric - Self Managed Kubernetes.pptx
Toronto MuleSoft_Meetup_Run Time Fabric - Self Managed Kubernetes.pptxToronto MuleSoft_Meetup_Run Time Fabric - Self Managed Kubernetes.pptx
Toronto MuleSoft_Meetup_Run Time Fabric - Self Managed Kubernetes.pptx
Anurag Dwivedi
 
OpenStack Murano
OpenStack MuranoOpenStack Murano
OpenStack Murano
openstackstl
 
Learn OpenStack from trystack.cn
Learn OpenStack from trystack.cnLearn OpenStack from trystack.cn
Learn OpenStack from trystack.cn
OpenCity Community
 
Creating the open source building blocks for IoT
Creating the open source building blocks for IoT Creating the open source building blocks for IoT
Creating the open source building blocks for IoT
Ian Skerrett
 
OpenStackユーザ会資料 - Masakari
OpenStackユーザ会資料 - MasakariOpenStackユーザ会資料 - Masakari
OpenStackユーザ会資料 - Masakari
masahito12
 
Make the Shift from Manual to Automation with Open Source
Make the Shift from Manual to Automation with Open SourceMake the Shift from Manual to Automation with Open Source
Make the Shift from Manual to Automation with Open Source
Perfecto by Perforce
 
Spring boot microservice metrics monitoring
Spring boot   microservice metrics monitoringSpring boot   microservice metrics monitoring
Spring boot microservice metrics monitoring
Oracle Korea
 
Spring Boot - Microservice Metrics Monitoring
Spring Boot - Microservice Metrics MonitoringSpring Boot - Microservice Metrics Monitoring
Spring Boot - Microservice Metrics Monitoring
DonghuKIM2
 
Distributed application usecase on docker
Distributed application usecase on dockerDistributed application usecase on docker
Distributed application usecase on docker
Hiroshi Miura
 
Open Source IoT at Eclipse
Open Source IoT at EclipseOpen Source IoT at Eclipse
Open Source IoT at Eclipse
Ian Skerrett
 
Security & Resiliency of Cloud Native Apps with Weave GitOps & Tetrate Servic...
Security & Resiliency of Cloud Native Apps with Weave GitOps & Tetrate Servic...Security & Resiliency of Cloud Native Apps with Weave GitOps & Tetrate Servic...
Security & Resiliency of Cloud Native Apps with Weave GitOps & Tetrate Servic...
Weaveworks
 

Similar to OpenStack Sydney summit - OpenStack HA and Testing (20)

Tempest scenariotests 20140512
Tempest scenariotests 20140512Tempest scenariotests 20140512
Tempest scenariotests 20140512
 
Webinar: OpenStack Best Practices for Production
Webinar: OpenStack Best Practices for ProductionWebinar: OpenStack Best Practices for Production
Webinar: OpenStack Best Practices for Production
 
2307 - DevBCN - Otel 101_compressed.pdf
2307 - DevBCN - Otel 101_compressed.pdf2307 - DevBCN - Otel 101_compressed.pdf
2307 - DevBCN - Otel 101_compressed.pdf
 
OpenStack & MySQL
OpenStack & MySQLOpenStack & MySQL
OpenStack & MySQL
 
Automated Deployment & Benchmarking with Chef, Cobbler and Rally for OpenStack
Automated Deployment & Benchmarking with Chef, Cobbler and Rally for OpenStackAutomated Deployment & Benchmarking with Chef, Cobbler and Rally for OpenStack
Automated Deployment & Benchmarking with Chef, Cobbler and Rally for OpenStack
 
Enterprise Ready OpenStack, Wiekus Beukes, Oracle
Enterprise Ready OpenStack,  Wiekus Beukes, OracleEnterprise Ready OpenStack,  Wiekus Beukes, Oracle
Enterprise Ready OpenStack, Wiekus Beukes, Oracle
 
OpenStack: Everything You Need to Know To Get Started
OpenStack: Everything You Need to Know To Get StartedOpenStack: Everything You Need to Know To Get Started
OpenStack: Everything You Need to Know To Get Started
 
OpenStack: Everything You Need To Know to Get Started (ATO2014)
OpenStack: Everything You Need To Know to Get Started (ATO2014)OpenStack: Everything You Need To Know to Get Started (ATO2014)
OpenStack: Everything You Need To Know to Get Started (ATO2014)
 
Using OpenStack to Accelerate New Product Development: Rik Harris, Telstra
Using OpenStack to Accelerate New Product Development: Rik Harris, TelstraUsing OpenStack to Accelerate New Product Development: Rik Harris, Telstra
Using OpenStack to Accelerate New Product Development: Rik Harris, Telstra
 
Toronto MuleSoft_Meetup_Run Time Fabric - Self Managed Kubernetes.pptx
Toronto MuleSoft_Meetup_Run Time Fabric - Self Managed Kubernetes.pptxToronto MuleSoft_Meetup_Run Time Fabric - Self Managed Kubernetes.pptx
Toronto MuleSoft_Meetup_Run Time Fabric - Self Managed Kubernetes.pptx
 
OpenStack Murano
OpenStack MuranoOpenStack Murano
OpenStack Murano
 
Learn OpenStack from trystack.cn
Learn OpenStack from trystack.cnLearn OpenStack from trystack.cn
Learn OpenStack from trystack.cn
 
Creating the open source building blocks for IoT
Creating the open source building blocks for IoT Creating the open source building blocks for IoT
Creating the open source building blocks for IoT
 
OpenStackユーザ会資料 - Masakari
OpenStackユーザ会資料 - MasakariOpenStackユーザ会資料 - Masakari
OpenStackユーザ会資料 - Masakari
 
Make the Shift from Manual to Automation with Open Source
Make the Shift from Manual to Automation with Open SourceMake the Shift from Manual to Automation with Open Source
Make the Shift from Manual to Automation with Open Source
 
Spring boot microservice metrics monitoring
Spring boot   microservice metrics monitoringSpring boot   microservice metrics monitoring
Spring boot microservice metrics monitoring
 
Spring Boot - Microservice Metrics Monitoring
Spring Boot - Microservice Metrics MonitoringSpring Boot - Microservice Metrics Monitoring
Spring Boot - Microservice Metrics Monitoring
 
Distributed application usecase on docker
Distributed application usecase on dockerDistributed application usecase on docker
Distributed application usecase on docker
 
Open Source IoT at Eclipse
Open Source IoT at EclipseOpen Source IoT at Eclipse
Open Source IoT at Eclipse
 
Security & Resiliency of Cloud Native Apps with Weave GitOps & Tetrate Servic...
Security & Resiliency of Cloud Native Apps with Weave GitOps & Tetrate Servic...Security & Resiliency of Cloud Native Apps with Weave GitOps & Tetrate Servic...
Security & Resiliency of Cloud Native Apps with Weave GitOps & Tetrate Servic...
 

Recently uploaded

Blood finder application project report (1).pdf
Blood finder application project report (1).pdfBlood finder application project report (1).pdf
Blood finder application project report (1).pdf
Kamal Acharya
 
1FIDIC-CONSTRUCTION-CONTRACT-2ND-ED-2017-RED-BOOK.pdf
1FIDIC-CONSTRUCTION-CONTRACT-2ND-ED-2017-RED-BOOK.pdf1FIDIC-CONSTRUCTION-CONTRACT-2ND-ED-2017-RED-BOOK.pdf
1FIDIC-CONSTRUCTION-CONTRACT-2ND-ED-2017-RED-BOOK.pdf
MadhavJungKarki
 
Levelised Cost of Hydrogen (LCOH) Calculator Manual
Levelised Cost of Hydrogen  (LCOH) Calculator ManualLevelised Cost of Hydrogen  (LCOH) Calculator Manual
Levelised Cost of Hydrogen (LCOH) Calculator Manual
Massimo Talia
 
Digital Twins Computer Networking Paper Presentation.pptx
Digital Twins Computer Networking Paper Presentation.pptxDigital Twins Computer Networking Paper Presentation.pptx
Digital Twins Computer Networking Paper Presentation.pptx
aryanpankaj78
 
Ericsson LTE Throughput Troubleshooting Techniques.ppt
Ericsson LTE Throughput Troubleshooting Techniques.pptEricsson LTE Throughput Troubleshooting Techniques.ppt
Ericsson LTE Throughput Troubleshooting Techniques.ppt
wafawafa52
 
This study Examines the Effectiveness of Talent Procurement through the Imple...
This study Examines the Effectiveness of Talent Procurement through the Imple...This study Examines the Effectiveness of Talent Procurement through the Imple...
This study Examines the Effectiveness of Talent Procurement through the Imple...
DharmaBanothu
 
Introduction to Computer Networks & OSI MODEL.ppt
Introduction to Computer Networks & OSI MODEL.pptIntroduction to Computer Networks & OSI MODEL.ppt
Introduction to Computer Networks & OSI MODEL.ppt
Dwarkadas J Sanghvi College of Engineering
 
Mechanical Engineering on AAI Summer Training Report-003.pdf
Mechanical Engineering on AAI Summer Training Report-003.pdfMechanical Engineering on AAI Summer Training Report-003.pdf
Mechanical Engineering on AAI Summer Training Report-003.pdf
21UME003TUSHARDEB
 
一比一原版(uofo毕业证书)美国俄勒冈大学毕业证如何办理
一比一原版(uofo毕业证书)美国俄勒冈大学毕业证如何办理一比一原版(uofo毕业证书)美国俄勒冈大学毕业证如何办理
一比一原版(uofo毕业证书)美国俄勒冈大学毕业证如何办理
upoux
 
ITSM Integration with MuleSoft.pptx
ITSM  Integration with MuleSoft.pptxITSM  Integration with MuleSoft.pptx
ITSM Integration with MuleSoft.pptx
VANDANAMOHANGOUDA
 
Unit -II Spectroscopy - EC I B.Tech.pdf
Unit -II Spectroscopy - EC  I B.Tech.pdfUnit -II Spectroscopy - EC  I B.Tech.pdf
Unit -II Spectroscopy - EC I B.Tech.pdf
TeluguBadi
 
FULL STACK PROGRAMMING - Both Front End and Back End
FULL STACK PROGRAMMING - Both Front End and Back EndFULL STACK PROGRAMMING - Both Front End and Back End
FULL STACK PROGRAMMING - Both Front End and Back End
PreethaV16
 
Applications of artificial Intelligence in Mechanical Engineering.pdf
Applications of artificial Intelligence in Mechanical Engineering.pdfApplications of artificial Intelligence in Mechanical Engineering.pdf
Applications of artificial Intelligence in Mechanical Engineering.pdf
Atif Razi
 
Zener Diode and its V-I Characteristics and Applications
Zener Diode and its V-I Characteristics and ApplicationsZener Diode and its V-I Characteristics and Applications
Zener Diode and its V-I Characteristics and Applications
Shiny Christobel
 
Transformers design and coooling methods
Transformers design and coooling methodsTransformers design and coooling methods
Transformers design and coooling methods
Roger Rozario
 
Impartiality as per ISO /IEC 17025:2017 Standard
Impartiality as per ISO /IEC 17025:2017 StandardImpartiality as per ISO /IEC 17025:2017 Standard
Impartiality as per ISO /IEC 17025:2017 Standard
MuhammadJazib15
 
Accident detection system project report.pdf
Accident detection system project report.pdfAccident detection system project report.pdf
Accident detection system project report.pdf
Kamal Acharya
 
UNIT 4 LINEAR INTEGRATED CIRCUITS-DIGITAL ICS
UNIT 4 LINEAR INTEGRATED CIRCUITS-DIGITAL ICSUNIT 4 LINEAR INTEGRATED CIRCUITS-DIGITAL ICS
UNIT 4 LINEAR INTEGRATED CIRCUITS-DIGITAL ICS
vmspraneeth
 
SENTIMENT ANALYSIS ON PPT AND Project template_.pptx
SENTIMENT ANALYSIS ON PPT AND Project template_.pptxSENTIMENT ANALYSIS ON PPT AND Project template_.pptx
SENTIMENT ANALYSIS ON PPT AND Project template_.pptx
b0754201
 
4. Mosca vol I -Fisica-Tipler-5ta-Edicion-Vol-1.pdf
4. Mosca vol I -Fisica-Tipler-5ta-Edicion-Vol-1.pdf4. Mosca vol I -Fisica-Tipler-5ta-Edicion-Vol-1.pdf
4. Mosca vol I -Fisica-Tipler-5ta-Edicion-Vol-1.pdf
Gino153088
 

Recently uploaded (20)

Blood finder application project report (1).pdf
Blood finder application project report (1).pdfBlood finder application project report (1).pdf
Blood finder application project report (1).pdf
 
1FIDIC-CONSTRUCTION-CONTRACT-2ND-ED-2017-RED-BOOK.pdf
1FIDIC-CONSTRUCTION-CONTRACT-2ND-ED-2017-RED-BOOK.pdf1FIDIC-CONSTRUCTION-CONTRACT-2ND-ED-2017-RED-BOOK.pdf
1FIDIC-CONSTRUCTION-CONTRACT-2ND-ED-2017-RED-BOOK.pdf
 
Levelised Cost of Hydrogen (LCOH) Calculator Manual
Levelised Cost of Hydrogen  (LCOH) Calculator ManualLevelised Cost of Hydrogen  (LCOH) Calculator Manual
Levelised Cost of Hydrogen (LCOH) Calculator Manual
 
Digital Twins Computer Networking Paper Presentation.pptx
Digital Twins Computer Networking Paper Presentation.pptxDigital Twins Computer Networking Paper Presentation.pptx
Digital Twins Computer Networking Paper Presentation.pptx
 
Ericsson LTE Throughput Troubleshooting Techniques.ppt
Ericsson LTE Throughput Troubleshooting Techniques.pptEricsson LTE Throughput Troubleshooting Techniques.ppt
Ericsson LTE Throughput Troubleshooting Techniques.ppt
 
This study Examines the Effectiveness of Talent Procurement through the Imple...
This study Examines the Effectiveness of Talent Procurement through the Imple...This study Examines the Effectiveness of Talent Procurement through the Imple...
This study Examines the Effectiveness of Talent Procurement through the Imple...
 
Introduction to Computer Networks & OSI MODEL.ppt
Introduction to Computer Networks & OSI MODEL.pptIntroduction to Computer Networks & OSI MODEL.ppt
Introduction to Computer Networks & OSI MODEL.ppt
 
Mechanical Engineering on AAI Summer Training Report-003.pdf
Mechanical Engineering on AAI Summer Training Report-003.pdfMechanical Engineering on AAI Summer Training Report-003.pdf
Mechanical Engineering on AAI Summer Training Report-003.pdf
 
一比一原版(uofo毕业证书)美国俄勒冈大学毕业证如何办理
一比一原版(uofo毕业证书)美国俄勒冈大学毕业证如何办理一比一原版(uofo毕业证书)美国俄勒冈大学毕业证如何办理
一比一原版(uofo毕业证书)美国俄勒冈大学毕业证如何办理
 
ITSM Integration with MuleSoft.pptx
ITSM  Integration with MuleSoft.pptxITSM  Integration with MuleSoft.pptx
ITSM Integration with MuleSoft.pptx
 
Unit -II Spectroscopy - EC I B.Tech.pdf
Unit -II Spectroscopy - EC  I B.Tech.pdfUnit -II Spectroscopy - EC  I B.Tech.pdf
Unit -II Spectroscopy - EC I B.Tech.pdf
 
FULL STACK PROGRAMMING - Both Front End and Back End
FULL STACK PROGRAMMING - Both Front End and Back EndFULL STACK PROGRAMMING - Both Front End and Back End
FULL STACK PROGRAMMING - Both Front End and Back End
 
Applications of artificial Intelligence in Mechanical Engineering.pdf
Applications of artificial Intelligence in Mechanical Engineering.pdfApplications of artificial Intelligence in Mechanical Engineering.pdf
Applications of artificial Intelligence in Mechanical Engineering.pdf
 
Zener Diode and its V-I Characteristics and Applications
Zener Diode and its V-I Characteristics and ApplicationsZener Diode and its V-I Characteristics and Applications
Zener Diode and its V-I Characteristics and Applications
 
Transformers design and coooling methods
Transformers design and coooling methodsTransformers design and coooling methods
Transformers design and coooling methods
 
Impartiality as per ISO /IEC 17025:2017 Standard
Impartiality as per ISO /IEC 17025:2017 StandardImpartiality as per ISO /IEC 17025:2017 Standard
Impartiality as per ISO /IEC 17025:2017 Standard
 
Accident detection system project report.pdf
Accident detection system project report.pdfAccident detection system project report.pdf
Accident detection system project report.pdf
 
UNIT 4 LINEAR INTEGRATED CIRCUITS-DIGITAL ICS
UNIT 4 LINEAR INTEGRATED CIRCUITS-DIGITAL ICSUNIT 4 LINEAR INTEGRATED CIRCUITS-DIGITAL ICS
UNIT 4 LINEAR INTEGRATED CIRCUITS-DIGITAL ICS
 
SENTIMENT ANALYSIS ON PPT AND Project template_.pptx
SENTIMENT ANALYSIS ON PPT AND Project template_.pptxSENTIMENT ANALYSIS ON PPT AND Project template_.pptx
SENTIMENT ANALYSIS ON PPT AND Project template_.pptx
 
4. Mosca vol I -Fisica-Tipler-5ta-Edicion-Vol-1.pdf
4. Mosca vol I -Fisica-Tipler-5ta-Edicion-Vol-1.pdf4. Mosca vol I -Fisica-Tipler-5ta-Edicion-Vol-1.pdf
4. Mosca vol I -Fisica-Tipler-5ta-Edicion-Vol-1.pdf
 

OpenStack Sydney summit - OpenStack HA and Testing

  • 1. © 2015 NTT Software Innovation Center 仮想マシンの高可用性と 異常系試験フレームワーク Sampath Priyankara SIC I三P I方D
  • 2. 2Copyright©2015 NTT corp. All Rights Reserv 仮想マシンの高可用性 Masakari
  • 3. 3Copyright©2015 NTT corp. All Rights Reserv What is Masakari? Service : Instances High Availability Service Mission: Provide instances high availability service for OpenStack clouds by automatically recovering the instances from failures. https://governance.openstack.org/tc/refer ence/projects/masakari.html
  • 4. 4Copyright©2015 NTT corp. All Rights Reserv Masakri in action Virtual Machine High Availability with Platform9 OpenStack • https://platform9.com/blog/virtual-machine-high- availability-platform9-openstack/ • https://www.openstack.org/summit/boston- 2017/summit-schedule/events/17502/zero-touch- high-availability-with-masakari-and-consul
  • 5. 5Copyright©2015 NTT corp. All Rights Reserv Project History [2015 Jul] Started on Github https://github.com/ntt-sic/masakari [no longer maintained] [2016 Jun] Moved into OpenStack name space and rebuild [2017 Oct] Became official OpenStack project Related work: [openstack-ha team] Define spcs for converged upstream solution for VMHA Wiki: https://wiki.openstack.org/wiki/Meetings/HATeamMeeting spec: https://review.openstack.org/#/q/project:openstack/openstack-resource-agents- specs [OpenStack Product Working Group] Development Proposal for VMHA Wiki: https://wiki.openstack.org/wiki/ProductTeam Feature: http://featuretracker.openstack.org/projectDetail/0003
  • 6. 6Copyright©2015 NTT corp. All Rights Reserv Project Structure Masakari • Masakari • Masakari Engine and APIs • Repositories: openstack/masakari • masakari-monitors • Monitors to detect failures and send notifications to masakari API • Repositories: openstack/masakari-monitors • python-masakariclient • Python wrapper for masakari APIs • Repositories: openstack/python-masakariclient • Masakari Specifications • masakari-specs • Specifications for all masakari projects • Repositories: openstack/masakari-specs
  • 7. 7Copyright©2015 NTT corp. All Rights Reserv Masakari Architecture masakari masakari-monitors python-masakariclient
  • 8. 8Copyright©2015 NTT corp. All Rights Reserv Masakri Features  Supported failure models • Host failures, Process failures, Instance failures Full set of monitors to detect failures Client Library for easy operation Customizable host recovery work flow • Auto(use nova scheduler), Reserved host, Hybrid  Masakari APIs: Failover Segments (segments): Define boundaries and behaviors for failover Hosts (hosts): Define compute nodes and Reserved compute nodes Notifications (notifications): Receive failure notifications from monitors
  • 9. 9Copyright©2015 NTT corp. All Rights Reserv Current contributors Mitaka to Queens In Queens http://stackalytics.com/?module=masakari&release=all http://stackalytics.com/?module=masakari&release=queens
  • 10. 1 0Copyright©2015 NTT corp. All Rights Reserv Work in progress • Recovery Method customization • [spec] https://review.openstack.org/#/c/458023/ • Intrusive Instance Monitoring • [spec] https://review.openstack.org/#/c/469070/ • Add event notification feature to masakari • [spec] https://review.openstack.org/#/c/473057/ • Masakari Horizon plugin • In progress
  • 11. 1 1Copyright©2015 NTT corp. All Rights Reserv Future work items • Ironic Bare Metal Instance HA • Volume boot Bare Metal Instance • External monitoring mechanism • Fencing mechanism • Instance recovery work flow • Force STONITH and Fencing • Node health prediction • Resource migration • Forcefully bring down the node • New masakri-monitors • Something other than pacemaker. (ex: consul)
  • 12. 1 2Copyright©2015 NTT corp. All Rights Reserv Masakari Community Where to find us IRC: #openstack-masakari ML: openstack-dev with [masakari] Weekly IRC meeting Agenda: https://wiki.openstack.org/wiki/Meetings/Masakari On Tuesday at 0400 UTC http://eavesdrop.openstack.org/#Masakari_Team_Meeti ng
  • 13. 1 3Copyright©2015 NTT corp. All Rights Reserv 異常系試験フレームワーク Eris
  • 14. 1 4Copyright©2015 NTT corp. All Rights Reserv What is Eris? Destructive test framework for OpenStack • Test robustness/resiliency of OpenStack at the CI/CD gates. • Test different architectures on 3rd party CIs. • Common test cases developed by community. • Automated evaluation of KPIs and report generation. Current Team • NTT, AT&T, Intel (LCOO WG members)
  • 15. 1 5Copyright©2015 NTT corp. All Rights Reserv About Eris Eris Wiki: https://openstack-lcoo.atlassian.net/wiki/spaces/LCOO/pages/13393034/Eris+- +Extreme+Testing+Framework+for+OpenStack Summit Forum: • https://etherpad.openstack.org/p/SYD-extreme-testing • https://etherpad.openstack.org/p/LCOO-Extreme_Testing-QA-ERIS Demo: https://openstack- lcoo.atlassian.net/wiki/spaces/LCOO/pages/22872097/Extreme+Testing+Demo
  • 16. 1 6Copyright©2015 NTT corp. All Rights Reserv Form Discussion Similar Efforts  OPNFV Yardstick[1]  enos: Experimental eNvironment for OpenStack New requirements  scale/performance testing for nova  Integrate with OPNFV testing tools Discussion with QA team  PoC demo  Discuss future road map 「1」https://wiki.opnfv.org/display/yardstick/Yardstick
  • 17. 1 7Copyright©2015 NTT corp. All Rights Reserv Work Items • Create SIG • Discuss with other community such as OPNFV, nova, Rally..etc. • Update QA Spec with details in Eris wiki • Probably divide in to several spes • Enhance Eris PoC and integrate into Masakari CI
  • 18. 1 8Copyright©2015 NTT corp. All Rights Reserv Self-healing SIG
  • 19. 1 9Copyright©2015 NTT corp. All Rights Reserv Self-healing SIG Manage OpenStack infrastructure in a policy-driven fashion, reacting to failures and other events by automatically healing or optimize services.
  • 20. 2 0Copyright©2015 NTT corp. All Rights Reserv Self-healing SIG Members HA of individual services Monasca: monitoring Aodh: alarming Congress: policy-based governance Mistral: workflow Senlin: clustering service Vitrage: root cause analysis Watcher: optimization Masakari: compute plane HA Freezer-dr: compute plane HA Heat: orchestration (normally used for cloud applications, but can also deploy cloud infrastructure via TripleO) Doctor: fault management and maintenance for NFV Fault Genes Working Group: Fault classification & Recovery Strategy Craton: Fleet management Kolla: Containerized OpenStack deployment tool kolla-k8s: same as above but in kubernetes cluster