DOST 2016 Cloud Without Failures

OpenStack: Eine Cloud ohne Fehler?
Höhere Zuverlässigkeit durch gezieltes Provozieren von Fehlern!
Using Fault Injection to Increase Cloud Reliability!
Götz Brasche / Jorge Cardoso
CTO IT PL RnD and Director CSI / Lead Architect Cloud Operations and Analytics
21.06. - 22.06.2016
Cologne, Germany

1
• 47% aller Probleme in Datenzentren
resultieren aus Ausfällen von
Cloudiensten
• Dauer der Ausfälle reicht von
40 Minuten bis zu fünf Tagen
• Kosten pro Ausfall durchschnittlich
690.204 USD
• Kosten pro Minute knapp 7.000 USD

2
Unplanned downtime
is caused by*
software bugs … 27%
hardware … 23%
human error … 18%
network failures … 17%
natural disasters … 8%
* Marcus, E., and Stern, H. Blueprints for High Availability: Designing Resilient Distributed Systems. John Wiley & Sons, Inc., 2003.

3
Google's 2007 found annualized failure
rates (AFRs) for drives
1 year old 1.7%
3 year old >8.6%
Eduardo Pinheiro, Wolf-Dietrich Weber, and Luiz André Barroso. 2007. Failure trends in a large disk drive population. In Proc.
of the 5th USENIX conference on File and Storage Technologies (FAST '07). USENIX Association, Berkeley, CA, USA, 2-2.

4
OpenStack is Growing
Apr 6, 2016
March 2016

5
OpenStack User Survey: A snapshot of OpenStack users’ attitudes and deployments.
April 2016. (https://www.openstack.org/assets/survey/April-2016-User-Survey-Report.pdf). Fig. 2.1, Page 9.
Key User Interests

6
What Companies Think About…
Cloud Reliability

7
Gartner
• Member of the Magic
Quadrant for x86 Server
Virtualization
Infrastructure
• Member of the Magic
Quadrant for Integrated
Systems
Cloud
Infrastructure
Product
Innovation Award
Frost & Sullivan 2013
10,000
Huawei employees working
in cloud computing and
dedicated to meeting every
IT requirement
No. 1
Industry-leading performance
according to the SPECvirt
server virtualization
performance benchmark
Best of Show
Award
Nomination
Interop 2013
DCD Blueprint
Award
China's first in the data
center industry
100,000
Desktops in the world's
largest-scale deployment
Huawei Cloud Computing Investment & Rewards

9
Fusion Sphere Architecture
VMware vSphere
Third-party Virtualization
Architecture
FusionCompute FusionStorage FusionNetwork
Huawei Virtualization Architecture
Server Storage Network & Security
Physical Infrastructure
Cloud Storage APIHuawei Open API
eBackup
UltraVR
Backup & DR
SNMP/REST NBI
FusionSphere SOI
FusionManager
Portal
RBAC
Alarm
Log
Open API
Resource
Management
Configuration API Adapter
Cloud Storage
Management
VDC/VPC

10
Market Recognitions Grows…
2 times higher scalability
Industry-leading performance in the
SPECvirt test
For the first time in 3 years, Gartner has
introduced a new company, Huawei
(FusionSphere), into the Magic Quadrant
for x86 Server Virtualization
Infrastructure.
Hypervisor
SPECvirt
Score
Ranking
FusionSphere 5.0 632 1
Linux 6.4 (KVM) 625 2
ESXi 5.1 472 3
http://www.spec.org/virt_sc2013/results/specvirt_sc2013_perf.html
vSphere
5.1
vSphere
6.0
FusionSphere
5.0
32
64
128
Nodes per cluster
• High performance
− < 5% CPU performance overheads
− Support for database, email, and
ERP and CRM services
• High reliability
− Proactive event detection
− Active/Standby management nodes
− Upgrade without service interruption
− Multi-level disaster recovery plans
Support for Critical Applications
VMware
Microsoft
Oracle
Parallel
CitrixHUAWEI
LEADERSCHALLENGERS
NICHE PLAYERS VISIONARIES
As of July 2014COMPLETENESS OF VISION
Source: Gartner (July 2014)
ABILITYTOEXECUTE
Red Hat
Member of the Gartner Magic Quadrant
for x86 Server Virtualization
Infrastructure

11
FusionSphere 5.1 Key Performance Indicators
Physical Server/VM Performance Indicator Value
Max. number of vCPUs (virtual SMP) per VM 128
Max. memory size per VM 4 TB
Max. virtual disk capacity per VM 64 TB
Max. number of virtual disks per VM 60
Max. number of virtual NICs per VM 12
Max. number of logical CPU cores per physical server 480
Max. memory size per physical server 12 TB
Max. number of powered-on VMs per physical server 1024

12
FusionSphere 5.1 Key Performance Indicators
Management Indicator Value
Max. number of physical servers per logical cluster 128
Max. number of VMs per logical cluster 3000
Max. number of logical clusters supported by a VRM node 32
Max. number of hosts supported by a VRM node 1024
Max. number of VMs supported by a VRM node 10,000
Max. number of VRM nodes that can be cascaded 16
Max. number of physical servers supported by cascaded VRM nodes 4096
Max. number of VMs supported by cascaded VRM nodes 80,000 (best practice in the industry)

15
FAILURES ARE INEVITABLE!
THE BEST WE CAN DO IS BE
PREPARED FOR THEM AND LEARN
FROM THEM
TEST, REPAIR, LEARN & PREDICT !
Kripa Krishnan, Technical Program Director from Google

16
One reason [Netflix]: It’s the lack of control over the underlying
hardware, the inability to configure it to ensure 100% uptime
Why does using a cloud infrastructure requires
advanced approaches for resiliency?

17
A program designed to increase resilience by purposely injecting
major failures
Discover flaws and subtle dependencies
Amazon AWS: GameDay
“That seems totally bizarre on the face of it, but as you dig down, you end up finding
some dependency no one knew about previously […] We’ve had situations where we
brought down a network in, say, São Paulo, only to find that in doing so we broke our
links in Mexico.”

18
 Google DIRT (Disaster Recovery Test)
 Annual disaster recovery & testing exercise
 8 years since inception
 Multi-day exercise triggering (controlled) failures in systems and process
 Premise
 30-day incapacitation of headquarters following a disaster
 Other offices and facilities may be affected
 When
 “Big disaster”: Annually for 3-5 days
 Continuous testing: Year-round
 Who
 100s of engineers (Site Reliability, Network, Hardware, Software, Security, Facilities)
 Business units (Human Resources, Finance, Safety, Crisis response etc.)
Google: DiRT
Source http://flowcon.org/dl/flowcon-sanfran-2014/slides/KripaKrishnan_LearningContinuouslyFromFailures.pdf

19
Netflix: Chaos Monkey
Fewer alerts for
ops team
Amazon EC2 and Amazon RDS Service
Disruption in the US East Region
April 29, 2011
September 20th, 2015: Amazon’s DynamoDB service experienced an availability issue in their US-EAST-1
Transfer traffic
to east region

20
Huawei: Butterfly Effect
-- Butterfly Effect System --
Enables to Automatically Test and Repair OpenStack and Cloud
Applications
CLOUD APPLICATION
HUAWEI FusionSphere
The system works by intentionally injecting different failures, test the ability to
survive them, and learn how to predict and repair failures preemptively
Failure
Repair
Test

21
Approach to Cloud Reliability
1
2 3
4
7
5
6
Design &
Deploy
Test
Infrastructure
Monitoring
Facilities
Design & Execute
Fault-Injection Plan
Identify Damages
Predict
Future Errors
Automatic
Repair
Repair & Learn
OpenStack
Inject Fault F51
Fault F51
(mysql unavailable)
OpenStack

22
Design & Execute Fault-Injection Plan
Best way to avoid failure: Fail constantly
Kill cinder database
(Simulate update failure)
Introduce delay in messages
(Full-scale traffic shows where
the real bottlenecks are)
Operation Error
OPENSTACK_KEYSTONE_URL = "http://%s:5000/v2.0" % OPENSTACK_HOST
Operation Error
/etc/nova/nova.conf
Delete: auth_strategy=keystone
Remove driver to HD
Remove access to NFS
(Simulate hardware failure)
The main testing framework of OpenStack is called Tempest, an opensource project with more than 2000 tests: only black-box testing (test only access the public interfaces)
2

OpenStack
image list
server create
security group create
Learning

OpenStack
Inject Fault F51
Learning

OpenStack
image list
Tuesday
?

OpenStack
image list
Tuesday
Fault F51
(mysql unavailable)

OpenStack
Inject Fault F51
Faults Behavior
Fault F51
(mysql unavailable)
Root Cause

2016-04-28 08:05:30.549 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] NeedRegenerationException _enter /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:94 <<keystone-main.service>>2016-04-28 08:05:30.549 23842 DEBU
dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Released creation lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154 <<keystone-main.service>>2016-04-28 08:05:30.549 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9ea
d070f93f2181 - - - - -] no value, waiting for create lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:127 <<keystone-main.service>>2016-04-28 08:05:30.550 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Calling creation functio
_enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:148 <<keystone-main.service>>2016-04-28 08:05:30.550 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] value creation lock <dogpile.cache.region._LockWrapper object a
0x7f8f004ac810> acquired _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:131 <<keystone-main.service>>2016-04-28 08:05:30.552 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] NeedRegenerationException _ente
/usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:94 <<keystone-main.service>>2016-04-28 08:05:30.552 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Released creation lock _enter_create /usr/lib/python2.7/site
packages/dogpile/core/dogpile.py:154 <<keystone-main.service>>2016-04-28 08:05:30.553 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Calling creation function _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:148 <<keystone
main.service>>2016-04-28 08:05:30.553 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] no value, waiting for create lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:127 <<keystone-main.service>>2016-04-28 08:05:30.553 2384
DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] value creation lock <dogpile.cache.region._LockWrapper object at 0x7f8f004acd50> acquired _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:131 <<keystone-main.service>>2016-04-2
08:05:30.555 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Released creation lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154 <<keystone-main.service>>2016-04-28 08:05:30.562 23842 DEBUG dogpile.core.dogpile [ii
e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Calling creation function _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:148 <<keystone-main.service>>2016-04-28 08:05:30.562 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - -
NeedRegenerationException _enter /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:94 <<keystone-main.service>>2016-04-28 08:05:30.562 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] no value, waiting for create lock _enter_creat
/usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:127 <<keystone-main.service>>2016-04-28 08:05:30.562 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] value creation lock <dogpile.cache.region._LockWrapper object at 0x7f8f004ac810> acquire
_enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:131 <<keystone-main.service>>2016-04-28 08:05:30.565 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] NeedRegenerationException _enter /usr/lib/python2.7/site
packages/dogpile/core/dogpile.py:94 <<keystone-main.service>>2016-04-28 08:05:30.565 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Released creation lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154 <<keystone
DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] value creation lock <dogpile.cache.region._LockWrapper object at 0x7f8f004ac810> acquired _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:131 <<keystone-main.service>>2016-04-2
08:05:30.566 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Calling creation function _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:148 <<keystone-main.service>>2016-04-28 08:05:30.567 23842 DEBUG dogpile.core.dogpile [ii
e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Released creation lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154 <<keystone-main.service>>2016-04-28 08:05:30.573 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - -
NeedRegenerationException _enter /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:94 <<keystone-main.service>>2016-04-28 08:05:30.574 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Calling creation function _enter_creat
/usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:148 <<keystone-main.service>>2016-04-28 08:05:30.574 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] no value, waiting for create lock _enter_create /usr/lib/python2.7/site
packages/dogpile/core/dogpile.py:127 <<keystone-main.service>>2016-04-28 08:05:30.574 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] value creation lock <dogpile.cache.region._LockWrapper object at 0x7f8f0049b090> acquired _enter_creat
packages/dogpile/core/dogpile.py:154 <<keystone-main.service>>2016-04-28 08:05:30.581 23842 DEBUG oslo.middleware.request_id [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] <BE_ID> (backward<<<) new x-openstack-request-id: <BE_ID><iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181
__call__ /usr/lib/python2.7/site-packages/oslo_middleware/request_id.py:95 <<keystone-main.service>>2016-04-28 08:05:30.585 11693 DEBUG eventlet.wsgi.server [-] (11693) accepted ('192.168.4.231', 51774) server /usr/lib/python2.7/site-packages/eventlet/wsgi.py:826 <<glance
api.service>>2016-04-28 08:05:30.588 11693 DEBUG oslo.middleware.request_id [iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1 REQxxUSER REQxTENAN - - REQ_DOMA] <BE_ID> (forward>>>) openstack.request_id: Correlation <BE_ID><NONE><iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1
__call__ /usr/lib/python2.7/site-packages/oslo_middleware/request_id.py:88 <<glance-api.service>>2016-04-28 08:05:30.589 11693 DEBUG oslo.middleware.request_id [iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1 REQxxUSER REQxTENAN - - REQ_DOMA] <BE_ID> (forward>>>) x-openstac
request-id: Correlation <BE_ID><NONE><iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1> __call__ /usr/lib/python2.7/site-packages/oslo_middleware/request_id.py:89 <<glance-api.service>>2016-04-28 08:05:30.590 11693 DEBUG glance.api.middleware.version_negotiation [iii-9a7ba455-6a15
422d-bd1a-a41adbb0afc1 REQxxUSER REQxTENAN - - REQ_DOMA] Determining version of request: GET /v1/images/detail Accept: */* process_request /usr/lib/python2.7/site-packages/glance/api/middleware/version_negotiation.py:47 <<glance-api.service>>2016-04-28 08:05:30.590 1169
DEBUG glance.api.middleware.version_negotiation [iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1 REQxxUSER REQxTENAN - - REQ_DOMA] Matched version: v1 process_request /usr/lib/python2.7/site-packages/glance/api/middleware/version_negotiation.py:72 <<glance-api.service>>2016-04-2
08:05:30.590 11693 DEBUG glance.api.middleware.version_negotiation [iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1 REQxxUSER REQxTENAN - - REQ_DOMA] Using url versioning process_request /usr/lib/python2.7/site-packages/glance/api/middleware/version_negotiation.py:60 <<glance
api.service>>2016-04-28 08:05:30.591 11693 DEBUG glance.api.middleware.version_negotiation [iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1 REQxxUSER REQxTENAN - - REQ_DOMA] new path /v1/images/detail process_request /usr/lib/python2.7/site
packages/glance/api/middleware/version_negotiation.py:73 <<glance-api.service>>2016-04-28 08:05:30.592 11693 DEBUG keystoneclient.auth.identity.v2 [iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1 REQxxUSER REQxTENAN - - REQ_DOMA] Making authentication request t
http://192.168.4.231:35357/v2.0/tokens get_auth_ref /usr/lib/python2.7/site-packages/keystoneclient/auth/identity/v2.py:86 <<glance-api.service>>2016-04-28 08:05:30.592 11693 DEBUG keystoneclient.session [iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1 REQxxUSER REQxTENAN -
REQ_DOMA] <BE_ID> (bridge>>>) Call from keystoneclient/session.py to Keystone <BE_ID> from request /usr/lib/python2.7/site-packages/keystoneclient/session.py:373 <<glance-api.service>>2016-04-28 08:05:30.596 23841 DEBUG oslo.middleware.request_id [iii-83c5225b-1d19-44d2-a338
24c8ee994bd2 REQxxUSER REQxTENAN - - REQ_DOMA] <BE_ID> (forward>>>) openstack.request_id: Correlation <BE_ID><NONE><iii-83c5225b-1d19-44d2-a338-24c8ee994bd2> __call__ /usr/lib/python2.7/site-packages/oslo_middleware/request_id.py:88 <<keystone-admin.service>>2016-04
28 08:05:30.596 23841 DEBUG oslo.middleware.request_id [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 REQxxUSER REQxTENAN - - REQ_DOMA] <BE_ID> (forward>>>) x-openstack-request-id: Correlation <BE_ID><NONE><iii-83c5225b-1d19-44d2-a338-24c8ee994bd2> __call_
/usr/lib/python2.7/site-packages/oslo_middleware/request_id.py:89 <<keystone-admin.service>>2016-04-28 08:05:30.597 23841 DEBUG keystone.middleware.core [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] 08:05:30.597 23841 DEBUG keystone.middleware.core [iii-83c5225b-1d19
44d2-a338-24c8ee994bd2 - - - - -] There is either no auth token in the request or the certificate issuer is not trusted. No auth context will be set. process_request /usr/lib/python2.7/site-packages/keystone/middleware/core.py:310 <<keystone-admin.service>>2016-04-28 08:05:30.598 23841 INF
keystone.common.wsgi [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] POST http://192.168.4.231:35357/v2.0/tokens <<keystone-admin.service>>2016-04-28 08:05:30.599 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] NeedRegenerationExceptio
_enter /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:94 <<keystone-admin.service>>2016-04-28 08:05:30.599 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] no value, waiting for create lock _enter_create /usr/lib/python2.7/site
packages/dogpile/core/dogpile.py:127 <<keystone-admin.service>>2016-04-28 08:05:30.599 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] value creation lock <dogpile.cache.region._LockWrapper object at 0x7f8f0740f610> acquired _enter_creat
/usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:131 <<keystone-admin.service>>2016-04-28 08:05:30.600 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] Calling creation function _enter_create /usr/lib/python2.7/site
packages/dogpile/core/dogpile.py:148 <<keystone-admin.service>>2016-04-28 08:05:30.603 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] Released creation lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154 <<keystone
admin.service>>2016-04-28 08:05:30.618 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] NeedRegenerationException _enter /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:94 <<keystone-admin.service>>2016-04-28 08:05:30.619 2384
DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] Calling creation function _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:148 <<keystone-admin.service>>2016-04-28 08:05:30.619 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19
44d2-a338-24c8ee994bd2 - - - - -] no value, waiting for create lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:127 <<keystone-admin.service>>2016-04-28 08:05:30.619 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] valu
creation lock <dogpile.cache.region._LockWrapper object at 0x7f8f0043b6d0> acquired _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:131 <<keystone-admin.service>>2016-04-28 08:05:30.622 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338
24c8ee994bd2 - - - - -] NeedRegenerationException _enter /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:94 <<keystone-admin.service>>2016-04-28 08:05:30.622 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] Released creation loc
_enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154 <<keystone-admin.service>>2016-04-28 08:05:30.622 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] no value, waiting for create lock _enter_create /usr/lib/python2.7/site
packages/dogpile/core/dogpile.py:127 <<keystone-admin.service>>2016-04-28 08:05:30.623 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] Calling creation function _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:14
<<keystone-admin.service>>2016-04-28 08:05:30.623 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] value creation lock <dogpile.cache.region._LockWrapper object at 0x7f8f0043b6d0> acquired _enter_create /usr/lib/python2.7/site
packages/dogpile/core/dogpile.py:131 <<keystone-admin.service>>2016-04-28 08:05:30.625 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] NeedRegenerationException _enter /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:94 <<keystone
admin.service>>2016-04-28 08:05:30.625 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] Released creation lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154 <<keystone-admin.service>>2016-04-28 08:05:30.625 2384
DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] no value, waiting for create lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:127 <<keystone-admin.service>>2016-04-28 08:05:30.626 23841 DEBUG dogpile.core.dogpile [iii-83c5225b
1d19-44d2-a338-24c8ee994bd2 - - - - -] Calling creation function _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:148 <<keystone-admin.service>>2016-04-28 08:05:30.626 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] valu
Root Cause Localization

$ egrep ‘ERROR | TRACE’ /var/log/nova/nova-api.log | wc
$ grep ‘req-’ /var/log/glance/glance-api.log

Typically, the first thing to look in the logs when you hit a
problem is for the error message or the stack trace that has
the details on which part of the python module the problem
is generated, for that particular component.
This should explain the problem in detail, as well as its origin.
You should also look for the exception that gets logged for
any failure, which will also help identify the root cause of the
failure. If there are no exception/stack trace messages getting
dumped to the logs, it means that the problem might not be of
severe impact and the user can continue with their operation.
However, in such cases, at least a warning message will get
logged to make sure that we don't miss out on anything that's
happening in the FusionSphere environment.
The root cause of most of the issues seen in FusionSphere can
be triaged with the default logging levels. If the support team
requires more detailed logging, you can enable the debug
logs and perform the action again to capture all the details.
To have a stable environment, you want to detect failure
promptly and determine causes efficiently. With a distributed
system, it's even more important to track the right items to
meet a service-level target. By knowing where the logs are and
how to manage them, you can analyze most issues you
encounter, allowing you to keep your environment running
smoothly.
Limitations of Troubleshooting Approaches
Although today’s programs are orders of magnitude more complex than those of
30 years ago, many people still use printf to log to console or local disk, and use
some combination of manual inspection and regular expressions to locate specific
messages or patterns.
FEBRUARY 2012, VOL. 55, NO. 2, COMMUNICATIONS OF THE ACM
Manual, complex, error-prone, and expensive
http://www.slideshare.net/tomoya/openstack-at-ntt-resonant-lessons-learned-in-web-infrastructure
One (very) simple command: glance.images.list()
Size: 316K
# messages:1499 # DEBUG: 1068
# INFO: 23 # Others: 408
DOCOMO has shown
(100GB and 80M lines)/day for
100 nodes

42
Activity Monitoring

43
Activity Monitoring

44
Activity Monitoring

Activity Monitoring
2016-04-28 08:05:30.549 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] NeedRegenerationException _enter /usr/lib/python2.7/site-
packages/dogpile/core/dogpile.py:94 <<keystone-main.service>>2016-04-28 08:05:30.549 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Released
creation lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154 <<keystone-main.service>>2016-04-28 08:05:30.549 23842 DEBUG dogpile.core.dogpile [iii-
e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] no value, waiting for create lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:127 <<keystone-
main.service>>2016-04-28 08:05:30.550 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Calling creation function _enter_create
/usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:148 <<keystone-main.service>>2016-04-28 08:05:30.550 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-
d070f93f2181 - - - - -] value creation lock <dogpile.cache.region._LockWrapper object at 0x7f8f004ac810> acquired _enter_create /usr/lib/python2.7/site-
packages/dogpile/core/dogpile.py:131 <<keystone-main.service>>2016-04-28 08:05:30.552 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -]
NeedRegenerationException _enter /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:94 <<keystone-main.service>>2016-04-28 08:05:30.552 23842 DEBUG dogpile.core.dogpile
[iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Released creation lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154 <<keystone-main.service>>2016-04-
28 08:05:30.553 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Calling creation function _enter_create /usr/lib/python2.7/site-
packages/dogpile/core/dogpile.py:148 <<keystone-main.service>>2016-04-28 08:05:30.553 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] no value,
waiting for create lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:127 <<keystone-main.service>>2016-04-28 08:05:30.553 23842 DEBUG dogpile.core.dogpile
[iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] value creation lock <dogpile.cache.region._LockWrapper object at 0x7f8f004acd50> acquired _enter_create /usr/lib/python2.7/site-
packages/dogpile/core/dogpile.py:131 <<keystone-main.service>>2016-04-28 08:05:30.555 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Released
creation lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154 <<keystone-main.service>>2016-04-28 08:05:30.562 23842 DEBUG dogpile.core.dogpile [iii-
e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Calling creation function _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:148 <<keystone-main.service>>2016-04-
28 08:05:30.562 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] NeedRegenerationException _enter /usr/lib/python2.7/site-
packages/dogpile/core/dogpile.py:94 <<keystone-main.service>>2016-04-28 08:05:30.562 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] no value,
waiting for create lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:127 <<keystone-main.service>>2016-04-28 08:05:30.562 23842 DEBUG dogpile.core.dogpile
[iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] value creation lock <dogpile.cache.region._LockWrapper object at 0x7f8f004ac810> acquired _enter_create /usr/lib/python2.7/site-
packages/dogpile/core/dogpile.py:131 <<keystone-main.service>>2016-04-28 08:05:30.565 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -]
NeedRegenerationException _enter /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:94 <<keystone-main.service>>2016-04-28 08:05:30.565 23842 DEBUG dogpile.core.dogpile
[iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Released creation lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154 <<keystone-main.service>>2016-04-
28 08:05:30.565 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] no value, waiting for create lock _enter_create /usr/lib/python2.7/site-
packages/dogpile/core/dogpile.py:127 <<keystone-main.service>>2016-04-28 08:05:30.565 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] value
creation lock <dogpile.cache.region._LockWrapper object at 0x7f8f004ac810> acquired _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:131 <<keystone-
main.service>>2016-04-28 08:05:30.566 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Calling creation function _enter_create
d070f93f2181 - - - - -] Released creation lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154 <<keystone-main.service>>2016-04-28 08:05:30.573 23842
DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] NeedRegenerationException _enter /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:94
<<keystone-main.service>>2016-04-28 08:05:30.574 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Calling creation function _enter_create
d070f93f2181 - - - - -] no value, waiting for create lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:127 <<keystone-main.service>>2016-04-28 08:05:30.574
23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] value creation lock <dogpile.cache.region._LockWrapper object at 0x7f8f0049b090> acquired
_enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:131 <<keystone-main.service>>2016-04-28 08:05:30.576 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-
4ee2-9eaf-d070f93f2181 - - - - -] Released creation lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154 <<keystone-main.service>>2016-04-28 08:05:30.581
23842 DEBUG oslo.middleware.request_id [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] <BE_ID> (backward<<<) new x-openstack-request-id: <BE_ID><iii-e3cf60c9-ba7f-4ee2-9eaf-
d070f93f2181> __call__ /usr/lib/python2.7/site-packages/oslo_middleware/request_id.py:95 <<keystone-main.service>>2016-04-28 08:05:30.585 11693 DEBUG eventlet.wsgi.server [-]
(11693) accepted ('192.168.4.231', 51774) server /usr/lib/python2.7/site-packages/eventlet/wsgi.py:826 <<glance-api.service>>2016-04-28 08:05:30.588 11693 DEBUG
oslo.middleware.request_id [iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1 REQxxUSER REQxTENAN - - REQ_DOMA] <BE_ID> (forward>>>) openstack.request_id: Correlation
<BE_ID><NONE><iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1> __call__ /usr/lib/python2.7/site-packages/oslo_middleware/request_id.py:88 <<glance-api.service>>2016-04-28
08:05:30.589 11693 DEBUG oslo.middleware.request_id [iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1 REQxxUSER REQxTENAN - - REQ_DOMA] <BE_ID> (forward>>>) x-openstack-request-
id: Correlation <BE_ID><NONE><iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1> __call__ /usr/lib/python2.7/site-packages/oslo_middleware/request_id.py:89 <<glance-api.service>>2016-
04-28 08:05:30.590 11693 DEBUG glance.api.middleware.version_negotiation [iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1 REQxxUSER REQxTENAN - - REQ_DOMA] Determining version of
request: GET /v1/images/detail Accept: */* process_request /usr/lib/python2.7/site-packages/glance/api/middleware/version_negotiation.py:47 <<glance-api.service>>2016-04-28
08:05:30.590 11693 DEBUG glance.api.middleware.version_negotiation [iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1 REQxxUSER REQxTENAN - - REQ_DOMA] Matched version: v1
process_request /usr/lib/python2.7/site-packages/glance/api/middleware/version_negotiation.py:72 <<glance-api.service>>2016-04-28 08:05:30.590 11693 DEBUG
glance.api.middleware.version_negotiation [iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1 REQxxUSER REQxTENAN - - REQ_DOMA] Using url versioning process_request
/usr/lib/python2.7/site-packages/glance/api/middleware/version_negotiation.py:60 <<glance-api.service>>2016-04-28 08:05:30.591 11693 DEBUG
glance.api.middleware.version_negotiation [iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1 REQxxUSER REQxTENAN - - REQ_DOMA] new path /v1/images/detail process_request
/usr/lib/python2.7/site-packages/glance/api/middleware/version_negotiation.py:73 <<glance-api.service>>2016-04-28 08:05:30.592 11693 DEBUG keystoneclient.auth.identity.v2 [iii-
9a7ba455-6a15-422d-bd1a-a41adbb0afc1 REQxxUSER REQxTENAN - - REQ_DOMA] Making authentication request to http://192.168.4.231:35357/v2.0/tokens get_auth_ref
/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/v2.py:86 <<glance-api.service>>2016-04-28 08:05:30.592 11693 DEBUG keystoneclient.session [iii-9a7ba455-6a15-422d-
bd1a-a41adbb0afc1 REQxxUSER REQxTENAN - - REQ_DOMA] <BE_ID> (bridge>>>) Call from keystoneclient/session.py to Keystone <BE_ID> from request /usr/lib/python2.7/site-
packages/keystoneclient/session.py:373 <<glance-api.service>>2016-04-28 08:05:30.596 23841 DEBUG oslo.middleware.request_id [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2
Logicaltime
0
10
20
30
40
50
60
30%
70%
Variability
Warning Y
Warning A
Warning B
Warning C
Error Z
Distributed Execution Graph

 Google's Tracing System (see also X-Trace and Magpie)
 Originally created to understand the system behavior from a search request
 Today Google's production clusters generate >1 TB/day of sampled trace data
Dapper: a Large-Scale Distributed Systems Tracing Infrastructure
Benjamin H. Sigelman, Luiz André Barroso, Mike Burrows, Pat Stephenson, Manoj Plakal, Donald Beaver, Saul Jaspan, Chandan Shanbhag, Dapper, a Large-Scale Distributed Systems Tracing Infrastructure, Google, Inc. (2010).
// Java:
Tracer t = Tracer.getCurrentTracer();
String request = ...;
if (hitCache())
t.record("cache hit for " + request);
else
t.record("cache miss for " + request);
 Annotation
 41 Java and 68 C++ applications have
custom annotations to better understand
intra-span activity
 Overall approach
 When a thread handles a traced control path, Dapper attaches a trace
context to thread-local storage.
 When the control flow library (threading, control flow, RPC) is used to
schedule callbacks, Dapper attached a trace context.
 Performance
 Basic instrumentation as small as possible and record only a fraction
of all traces using sampling (1/1000).
 Daemon uses < 0.3% CPU during collection. Small memory footprint

47
Dapper
Benjamin H. Sigelman, Luiz André Barroso, Mike Burrows, Pat Stephenson, Manoj Plakal, Donald Beaver, Saul Jaspan, Chandan Shanbhag, Dapper, a Large-Scale
Distributed Systems Tracing Infrastructure, Google, Inc. (2010).

FingerprintLogicaltime
0
10
20
30
40
50
60
30%
70%
Variability
Warning Y
Warning A
Warning B
Warning C
Error Z
Distributed Execution Graph

http://acumendatasystems.com/how-fingerprint-biometric-time-clocks-work/
failureOK

Fingerprint Analytics
Can you identify your failures?
Key Challenges Analysis  New matching algorithms  Real-time fingerprint detection  Fingerprint prediction  Alignments and nested FP
M8
M1
Service a Service kService b
$ openstack image list Fault 15aOK

51
Application Fields
Failure Mode and Effect Analysis (FMEA)
Security and Intrusion detection Performance Analysis
Bottleneck
Cloud Accountability

52
 OpenStack Engineers positions
 Rapid prototyping of cool ideas: propose it today, code it, and show it running in
3 months…
 Internship positions for MSc students
 Fault injection, fault models, fault libraries, fault plans, brake and rebuild systems
all day long, …
 Innovative PoCs
 Solving difficult challenges of real problems using quick and dirty prototyping
Join the Cause!

53
 Industry-Academia Workshop on
Cloud Reliability and Resilience
 This workshop intends to bring
together industry and academia
to identify the most relevant
requirements in the field of cloud
reliability and resilience, on one
hand, and existing state-of-the-art
solutions, on the other.
 We invite engineers, scientists,
regulators, and experts to discuss
and contribute to the creation of a
new generation of highly reliable
cloud platforms.
November Event
BERLIN 7-8 NOVEMBER 2016
6, Ernst-Reuter-Platz 7
10587 Berlin
Germany

Copyright©2016 Huawei Technologies Co., Ltd. All Rights Reserved.
The information in this document may contain predictive statements including, without limitation, statements regarding the future financial and operating results, future product
portfolio, new technology, etc. There are a number of factors that could cause actual results and developments to differ materially from those expressed or implied in the predictive
statements. Therefore, such information is provided for reference purpose only and constitutes neither an offer nor an acceptance. Huawei may change the information at any time
without notice.
HUAWEI ENTERPRISE ICT SOLUTIONS A BETTER WAY
Thank you!

DOST 2016 Cloud Without Failures

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to DOST 2016 Cloud Without Failures

Similar to DOST 2016 Cloud Without Failures (20)

More from Jorge Cardoso

More from Jorge Cardoso (16)

Recently uploaded

Recently uploaded (16)

DOST 2016 Cloud Without Failures