SlideShare a Scribd company logo
1 of 18
Download to read offline
Debugging OOM Failures
An ONAP Webinar
© 2018 Aarna Networks, Inc.
© 2017 Aarna Networks, Inc.
Some Introductions
Vivekanandan Muthukrishnan (Vivek)
● Over 17 years of experience in the field of OpenStack,
Storage, Network and Server Virtualization
Amar Kapadia
● Author of “Understanding OPNFV”
● NFV specialist with background in OpenStack, Ceph,
dataplane acceleration technologies
© 2017 Aarna Networks, Inc.
Goals
To create a lab environment for
A) ONAP training
B) Development purposes
© 2017 Aarna Networks, Inc.
Requirements/Solution
● Requirements:
○ Light footprint
○ Self-service or instructor provided infrastructure for online and in-
class users
● Solution
○ Laptop was eliminated because ONAP is quite heavy
○ So we went with a Google Cloud Platform (GCP) VM
○ GCP offers both options — self-service or instructor provided (ssh)
○ Considered but set aside public OpenStack clouds it is more
complicated and OpenStack clouds are regional in nature (easy to
add on though for specific needs)
○ Finally decided to create an ONAP+OPNFV-all-in-one VM
© 2017 Aarna Networks, Inc.
GCP VM Architecture
192.168.x.x
SOCKS5
© 2017 Aarna Networks, Inc.
Our Approach
● Used OOM to get a lighter footprint
○ Amsterdam release
○ Rancher to deploy Kubernetes
○ Scripts to allow users to deploy a subset of
ONAP projects vs. all projects
● Used OPNFV for OpenStack
○ Euphrates release
○ Apex installer
○ Basic scenario: no ha, no sdn, no features
© 2017 Aarna Networks, Inc.
OOM Issues Summary
Issues
● AAI containers failed to transition to
Running state
● SDC UI is not getting loaded
● SDC Service Distribution Error
● VID Service Deployment Error
● VID ADD VNF Error
● SDNC User creation failed
● Robot init_robot failed with missing
attributes
Tools Used
● Healthchecks
● Robot logs
● Container logs
Outcomes
● Single-command workaround
scripts
© 2017 Aarna Networks, Inc.
OOM Issues & Workaround
● AAI containers failed to transition to Running state
○ NAMESPACE NAME READY STATUS RESTARTS AGE
○ onap aai-service-5698ddc455-s25dd 0/1 Init:0/1 33 6h
● Workaround: bounce AAI containers
○ cd oom/kubernetes/oneclick/
○ source ./setenv.bash
○ ./deleteAll.bash -n onap -a aai
○ ./createAll.bash -n onap -a aai
© 2017 Aarna Networks, Inc.
OOM Issues & Workaround
● SDC UI is not getting loaded
○ SDC UI Failed to load
○ Status code : 503 Service Unavailable
● Workaround
○ Bounce SDC containers in specific order: SDC → CS → BE → FE
○ Wait for one container to come up before bouncing the next dependent container
© 2017 Aarna Networks, Inc.
OOM Issues & Workaround
● SDC Service Distribution Error
○ Error Code : POL5000
○ Status Code : 500
● Workaround
○ Bounce global-kafka container
○ Bounce dmaap & dmaap-listener container
○ Bounce SDC containers in order
○ Bounce the vnc-portal & portalwidgets containers
© 2017 Aarna Networks, Inc.
OOM Issues & Workaround
● VID Service Deployment Error
○ "requestType": "createInstance"
○ "requestStatus": "SVC3001Resource not found...
● Workaround
○ Bounce model-loader-service
○ Run ROBOT scripts init_robot & init
© 2017 Aarna Networks, Inc.
OOM Issues & Workaround
● VID ADD VNF Error
○ Error : SVC0002 400 Bad request Error
● Workaround
○ Bounce Model-loader-service
○ Bounce mso container
○ Bounce SDC containers in order
○ Execute ROBOT init_robot & init
○ Redistribute the service
© 2017 Aarna Networks, Inc.
OOM Issues & Workaround
● SDNC User creation failed
○ Database connection issue
● Workaround
○ Bounce sdnc container
○ Bounce sdnc-portal container
© 2017 Aarna Networks, Inc.
OOM Issues & Workaround
● Robot init_robot failed with missing attributes
○ Missing attribute GLOBAL_INJECTED_UBUNTU_1604_IMAGE
○ Missing attribute GLOBAL_INJECTED_UBUNTU_1404_IMAGE
● Workaround
○ Update file /dockerdata-nfs/onap/robot/eteshare/config/vm_properties.py
○ Include lines after line (GLOBAL_INJECTED_SDNC_PORTAL_IP_ADDR)
■ GLOBAL_INJECTED_UBUNTU_1604_IMAGE = "xenial"
■ GLOBAL_INJECTED_UBUNTU_1404_IMAGE = "trusty
○ JSON attributes under GLOBAL_INJECTED_PROPERTIES
■ "GLOBAL_INJECTED_UBUNTU_1604_IMAGE" : "xenial"
■ "GLOBAL_INJECTED_UBUNTU_1404_IMAGE" : "trusty"
© 2017 Aarna Networks, Inc.
OOM Debugging Approaches
● Check SDC BE & FE health status
○ curl -s -X GET "http://<ONAP SERVER IP>:30205/sdc2/rest/healthCheck"
| python -m json.tool
○ curl -s -X GET http://<ONAP SERVER IP>:30206/sdc1/rest/healthCheck |
python -m json.tool
● Check ROBOT container logs
○ Robot portal URL = http://<ONAP SERVER IP>:30209/logs/demo/
○ username/password = test/test
● Connect & check the target container logs (folder /var/log/onap)
○ kubectl exec -it -n {NAME SPACE} {CONTAINER ID} -- /bin/bash
○ kubectl describe pod -n {NAME SPACE} {CONTAINER ID}
● Bouncing a failed container
○ kubectl delete pod -n {NAME SPACE} {CONTAINER ID}
© 2017 Aarna Networks, Inc.
Debugging Outcome
● Created easy to use scripts that address each issue
● Training participants are given instructions on which workaround to run in a
particular error situation
© 2017 Aarna Networks, Inc.
About Aarna
ONAP related products, training, and
professional services
Founded in 2017 by
serial entrepreneurs
(NFVI/VIM experts)
Located in San
Jose, CA &
Bangalore, India
60 participants
trained in 8
sessions
2 development
distros & 4
packaged services
Thank You!
www.aarnanetworks.com
vmuthukrishnan@aarnanetworks.com
akapadia@aarnanetworks.com

More Related Content

What's hot

ONAP Overview Webinar - Aarna Networks & Cloudify
ONAP Overview Webinar - Aarna Networks & CloudifyONAP Overview Webinar - Aarna Networks & Cloudify
ONAP Overview Webinar - Aarna Networks & CloudifyCloudify Community
 
ONAP SDC - Model driven design
ONAP SDC - Model driven designONAP SDC - Model driven design
ONAP SDC - Model driven designEden Rozin
 
Introducing the ADVA MicroMux™
Introducing the ADVA MicroMux™Introducing the ADVA MicroMux™
Introducing the ADVA MicroMux™ADVA
 
From Waterfall to Agile - Managing Disruptive Change Without Disrupting the B...
From Waterfall to Agile - Managing Disruptive Change Without Disrupting the B...From Waterfall to Agile - Managing Disruptive Change Without Disrupting the B...
From Waterfall to Agile - Managing Disruptive Change Without Disrupting the B...ADVA
 
Packet Optical SDN Field Trial for Multi-Layer Network Optimization
Packet Optical SDN Field Trial for Multi-Layer Network OptimizationPacket Optical SDN Field Trial for Multi-Layer Network Optimization
Packet Optical SDN Field Trial for Multi-Layer Network OptimizationADVA
 
Barriers to content production & distribution in Africa
Barriers to content production & distribution in AfricaBarriers to content production & distribution in Africa
Barriers to content production & distribution in AfricaInternet Society
 
Drawing Customers North - Highlighting the Benefits of Nordic Data Centers
Drawing Customers North - Highlighting the Benefits of Nordic Data CentersDrawing Customers North - Highlighting the Benefits of Nordic Data Centers
Drawing Customers North - Highlighting the Benefits of Nordic Data CentersADVA
 
Pure-Play Virtualization for Rural Broadband
Pure-Play Virtualization for Rural BroadbandPure-Play Virtualization for Rural Broadband
Pure-Play Virtualization for Rural BroadbandADVA
 
Leveraging NFV Infrastructure to Drive Revenue
Leveraging NFV Infrastructure to Drive RevenueLeveraging NFV Infrastructure to Drive Revenue
Leveraging NFV Infrastructure to Drive RevenueADVA
 
Polymorphic Attacks on Data-in-Motion Require a New Security Approach From Bo...
Polymorphic Attacks on Data-in-Motion Require a New Security Approach From Bo...Polymorphic Attacks on Data-in-Motion Require a New Security Approach From Bo...
Polymorphic Attacks on Data-in-Motion Require a New Security Approach From Bo...ADVA
 
NEC Platforms' Activity For IPv6 Deployment
NEC Platforms' Activity For IPv6 DeploymentNEC Platforms' Activity For IPv6 Deployment
NEC Platforms' Activity For IPv6 DeploymentAPNIC
 
V cpe deployment-best-practices-presentation
V cpe deployment-best-practices-presentationV cpe deployment-best-practices-presentation
V cpe deployment-best-practices-presentationNir Cohen
 
Scalable and Secure Connectivity for Seamless Cloud Evolution
Scalable and Secure Connectivity for Seamless Cloud EvolutionScalable and Secure Connectivity for Seamless Cloud Evolution
Scalable and Secure Connectivity for Seamless Cloud EvolutionADVA
 
ICN Akamai's Backbone
ICN Akamai's BackboneICN Akamai's Backbone
ICN Akamai's BackboneAPNIC
 
ONAP TOSCA Orchestration with Cloudify
ONAP TOSCA Orchestration with CloudifyONAP TOSCA Orchestration with Cloudify
ONAP TOSCA Orchestration with CloudifyCloudify Community
 
Transforming Packet Networks With Open Optical Transport
Transforming Packet Networks With Open Optical TransportTransforming Packet Networks With Open Optical Transport
Transforming Packet Networks With Open Optical TransportADVA
 
Sckipio broadband vision G.fast Presentation oct 2014 by mileend gadkari
Sckipio broadband vision G.fast Presentation oct 2014 by mileend gadkariSckipio broadband vision G.fast Presentation oct 2014 by mileend gadkari
Sckipio broadband vision G.fast Presentation oct 2014 by mileend gadkariSckipio
 
Fostering the Evolution of Network Based Cloud Service Providers.
Fostering the Evolution of Network Based Cloud Service Providers.Fostering the Evolution of Network Based Cloud Service Providers.
Fostering the Evolution of Network Based Cloud Service Providers.Cisco Service Provider
 
SDN and Photonics for Dynamic Cloud Connectivity
SDN and Photonics for Dynamic Cloud Connectivity SDN and Photonics for Dynamic Cloud Connectivity
SDN and Photonics for Dynamic Cloud Connectivity ADVA
 

What's hot (20)

ONAP Overview Webinar - Aarna Networks & Cloudify
ONAP Overview Webinar - Aarna Networks & CloudifyONAP Overview Webinar - Aarna Networks & Cloudify
ONAP Overview Webinar - Aarna Networks & Cloudify
 
ONAP SDC - Model driven design
ONAP SDC - Model driven designONAP SDC - Model driven design
ONAP SDC - Model driven design
 
Introducing the ADVA MicroMux™
Introducing the ADVA MicroMux™Introducing the ADVA MicroMux™
Introducing the ADVA MicroMux™
 
From Waterfall to Agile - Managing Disruptive Change Without Disrupting the B...
From Waterfall to Agile - Managing Disruptive Change Without Disrupting the B...From Waterfall to Agile - Managing Disruptive Change Without Disrupting the B...
From Waterfall to Agile - Managing Disruptive Change Without Disrupting the B...
 
Packet Optical SDN Field Trial for Multi-Layer Network Optimization
Packet Optical SDN Field Trial for Multi-Layer Network OptimizationPacket Optical SDN Field Trial for Multi-Layer Network Optimization
Packet Optical SDN Field Trial for Multi-Layer Network Optimization
 
Barriers to content production & distribution in Africa
Barriers to content production & distribution in AfricaBarriers to content production & distribution in Africa
Barriers to content production & distribution in Africa
 
Drawing Customers North - Highlighting the Benefits of Nordic Data Centers
Drawing Customers North - Highlighting the Benefits of Nordic Data CentersDrawing Customers North - Highlighting the Benefits of Nordic Data Centers
Drawing Customers North - Highlighting the Benefits of Nordic Data Centers
 
Pure-Play Virtualization for Rural Broadband
Pure-Play Virtualization for Rural BroadbandPure-Play Virtualization for Rural Broadband
Pure-Play Virtualization for Rural Broadband
 
Leveraging NFV Infrastructure to Drive Revenue
Leveraging NFV Infrastructure to Drive RevenueLeveraging NFV Infrastructure to Drive Revenue
Leveraging NFV Infrastructure to Drive Revenue
 
Polymorphic Attacks on Data-in-Motion Require a New Security Approach From Bo...
Polymorphic Attacks on Data-in-Motion Require a New Security Approach From Bo...Polymorphic Attacks on Data-in-Motion Require a New Security Approach From Bo...
Polymorphic Attacks on Data-in-Motion Require a New Security Approach From Bo...
 
NEC Platforms' Activity For IPv6 Deployment
NEC Platforms' Activity For IPv6 DeploymentNEC Platforms' Activity For IPv6 Deployment
NEC Platforms' Activity For IPv6 Deployment
 
V cpe deployment-best-practices-presentation
V cpe deployment-best-practices-presentationV cpe deployment-best-practices-presentation
V cpe deployment-best-practices-presentation
 
Scalable and Secure Connectivity for Seamless Cloud Evolution
Scalable and Secure Connectivity for Seamless Cloud EvolutionScalable and Secure Connectivity for Seamless Cloud Evolution
Scalable and Secure Connectivity for Seamless Cloud Evolution
 
ICN Akamai's Backbone
ICN Akamai's BackboneICN Akamai's Backbone
ICN Akamai's Backbone
 
ONAP TOSCA Orchestration with Cloudify
ONAP TOSCA Orchestration with CloudifyONAP TOSCA Orchestration with Cloudify
ONAP TOSCA Orchestration with Cloudify
 
Q logic xmonth_jan 2011
Q logic xmonth_jan 2011Q logic xmonth_jan 2011
Q logic xmonth_jan 2011
 
Transforming Packet Networks With Open Optical Transport
Transforming Packet Networks With Open Optical TransportTransforming Packet Networks With Open Optical Transport
Transforming Packet Networks With Open Optical Transport
 
Sckipio broadband vision G.fast Presentation oct 2014 by mileend gadkari
Sckipio broadband vision G.fast Presentation oct 2014 by mileend gadkariSckipio broadband vision G.fast Presentation oct 2014 by mileend gadkari
Sckipio broadband vision G.fast Presentation oct 2014 by mileend gadkari
 
Fostering the Evolution of Network Based Cloud Service Providers.
Fostering the Evolution of Network Based Cloud Service Providers.Fostering the Evolution of Network Based Cloud Service Providers.
Fostering the Evolution of Network Based Cloud Service Providers.
 
SDN and Photonics for Dynamic Cloud Connectivity
SDN and Photonics for Dynamic Cloud Connectivity SDN and Photonics for Dynamic Cloud Connectivity
SDN and Photonics for Dynamic Cloud Connectivity
 

Similar to Aarna networks debugging oom failures webinar

LINE's Private Cloud - Meet Cloud Native World
LINE's Private Cloud - Meet Cloud Native WorldLINE's Private Cloud - Meet Cloud Native World
LINE's Private Cloud - Meet Cloud Native WorldLINE Corporation
 
Automation of Hadoop cluster operations in Arm Treasure Data
Automation of Hadoop cluster operations in Arm Treasure DataAutomation of Hadoop cluster operations in Arm Treasure Data
Automation of Hadoop cluster operations in Arm Treasure DataYan Wang
 
DevOpsDays Taipei 2017 從打鐵到雲端
DevOpsDays Taipei 2017 從打鐵到雲端DevOpsDays Taipei 2017 從打鐵到雲端
DevOpsDays Taipei 2017 從打鐵到雲端Hung-Yen Chen
 
OSDC 2017 - Dr. Udo Seidel - VMwares (open source) Way of Container
OSDC 2017 - Dr. Udo Seidel - VMwares (open source) Way of ContainerOSDC 2017 - Dr. Udo Seidel - VMwares (open source) Way of Container
OSDC 2017 - Dr. Udo Seidel - VMwares (open source) Way of ContainerNETWAYS
 
OSDC 2017 | VMware's (Open Source) way of Container by Dr. Udo Seidel
OSDC 2017 | VMware's (Open Source) way of Container by Dr. Udo SeidelOSDC 2017 | VMware's (Open Source) way of Container by Dr. Udo Seidel
OSDC 2017 | VMware's (Open Source) way of Container by Dr. Udo SeidelNETWAYS
 
Second Skin: Real-Time Retheming a Legacy Web Application with Diazo in the C...
Second Skin: Real-Time Retheming a Legacy Web Application with Diazo in the C...Second Skin: Real-Time Retheming a Legacy Web Application with Diazo in the C...
Second Skin: Real-Time Retheming a Legacy Web Application with Diazo in the C...Chris Shenton
 
To Russia with Love: Deploying Kubernetes in Exotic Locations On Prem
To Russia with Love: Deploying Kubernetes in Exotic Locations On PremTo Russia with Love: Deploying Kubernetes in Exotic Locations On Prem
To Russia with Love: Deploying Kubernetes in Exotic Locations On PremCloudOps2005
 
Automating using Ansible
Automating using AnsibleAutomating using Ansible
Automating using AnsibleAlok Patra
 
Self service cloud resources
Self service cloud resourcesSelf service cloud resources
Self service cloud resourcesAppvia
 
Rise of the machines: Continuous Delivery at SEEK - YOW! Night Summary Slides
Rise of the machines: Continuous Delivery at SEEK - YOW! Night Summary SlidesRise of the machines: Continuous Delivery at SEEK - YOW! Night Summary Slides
Rise of the machines: Continuous Delivery at SEEK - YOW! Night Summary SlidesDiUS
 
Not Your Father’s Web App: The Cloud-Native Architecture of images.nasa.gov
Not Your Father’s Web App: The Cloud-Native Architecture of images.nasa.govNot Your Father’s Web App: The Cloud-Native Architecture of images.nasa.gov
Not Your Father’s Web App: The Cloud-Native Architecture of images.nasa.govChris Shenton
 
[db tech showcase Tokyo 2017] C24:Taking off to the clouds. How to use DMS in...
[db tech showcase Tokyo 2017] C24:Taking off to the clouds. How to use DMS in...[db tech showcase Tokyo 2017] C24:Taking off to the clouds. How to use DMS in...
[db tech showcase Tokyo 2017] C24:Taking off to the clouds. How to use DMS in...Insight Technology, Inc.
 
Run Apache Spark on Kubernetes in Large Scale_ Challenges and Solutions-2.pdf
Run Apache Spark on Kubernetes in Large Scale_ Challenges and Solutions-2.pdfRun Apache Spark on Kubernetes in Large Scale_ Challenges and Solutions-2.pdf
Run Apache Spark on Kubernetes in Large Scale_ Challenges and Solutions-2.pdfAnya Bida
 
Glauber Costa on OSv as NoSQL platform
Glauber Costa on OSv as NoSQL platformGlauber Costa on OSv as NoSQL platform
Glauber Costa on OSv as NoSQL platformDon Marti
 
EDB Postgres with Containers
EDB Postgres with ContainersEDB Postgres with Containers
EDB Postgres with ContainersEDB
 
Web componenet using angular element
Web componenet using angular elementWeb componenet using angular element
Web componenet using angular elementHimanshu Tamrakar
 
Netflix Container Scheduling and Execution - QCon New York 2016
Netflix Container Scheduling and Execution - QCon New York 2016Netflix Container Scheduling and Execution - QCon New York 2016
Netflix Container Scheduling and Execution - QCon New York 2016aspyker
 
Scheduling a fuller house - Talk at QCon NY 2016
Scheduling a fuller house - Talk at QCon NY 2016Scheduling a fuller house - Talk at QCon NY 2016
Scheduling a fuller house - Talk at QCon NY 2016Sharma Podila
 

Similar to Aarna networks debugging oom failures webinar (20)

LINE's Private Cloud - Meet Cloud Native World
LINE's Private Cloud - Meet Cloud Native WorldLINE's Private Cloud - Meet Cloud Native World
LINE's Private Cloud - Meet Cloud Native World
 
Automation of Hadoop cluster operations in Arm Treasure Data
Automation of Hadoop cluster operations in Arm Treasure DataAutomation of Hadoop cluster operations in Arm Treasure Data
Automation of Hadoop cluster operations in Arm Treasure Data
 
DevOpsDays Taipei 2017 從打鐵到雲端
DevOpsDays Taipei 2017 從打鐵到雲端DevOpsDays Taipei 2017 從打鐵到雲端
DevOpsDays Taipei 2017 從打鐵到雲端
 
OSDC 2017 - Dr. Udo Seidel - VMwares (open source) Way of Container
OSDC 2017 - Dr. Udo Seidel - VMwares (open source) Way of ContainerOSDC 2017 - Dr. Udo Seidel - VMwares (open source) Way of Container
OSDC 2017 - Dr. Udo Seidel - VMwares (open source) Way of Container
 
OSDC 2017 | VMware's (Open Source) way of Container by Dr. Udo Seidel
OSDC 2017 | VMware's (Open Source) way of Container by Dr. Udo SeidelOSDC 2017 | VMware's (Open Source) way of Container by Dr. Udo Seidel
OSDC 2017 | VMware's (Open Source) way of Container by Dr. Udo Seidel
 
Second Skin: Real-Time Retheming a Legacy Web Application with Diazo in the C...
Second Skin: Real-Time Retheming a Legacy Web Application with Diazo in the C...Second Skin: Real-Time Retheming a Legacy Web Application with Diazo in the C...
Second Skin: Real-Time Retheming a Legacy Web Application with Diazo in the C...
 
To Russia with Love: Deploying Kubernetes in Exotic Locations On Prem
To Russia with Love: Deploying Kubernetes in Exotic Locations On PremTo Russia with Love: Deploying Kubernetes in Exotic Locations On Prem
To Russia with Love: Deploying Kubernetes in Exotic Locations On Prem
 
Automating using Ansible
Automating using AnsibleAutomating using Ansible
Automating using Ansible
 
Self service cloud resources
Self service cloud resourcesSelf service cloud resources
Self service cloud resources
 
Rise of the machines: Continuous Delivery at SEEK - YOW! Night Summary Slides
Rise of the machines: Continuous Delivery at SEEK - YOW! Night Summary SlidesRise of the machines: Continuous Delivery at SEEK - YOW! Night Summary Slides
Rise of the machines: Continuous Delivery at SEEK - YOW! Night Summary Slides
 
Not Your Father’s Web App: The Cloud-Native Architecture of images.nasa.gov
Not Your Father’s Web App: The Cloud-Native Architecture of images.nasa.govNot Your Father’s Web App: The Cloud-Native Architecture of images.nasa.gov
Not Your Father’s Web App: The Cloud-Native Architecture of images.nasa.gov
 
[db tech showcase Tokyo 2017] C24:Taking off to the clouds. How to use DMS in...
[db tech showcase Tokyo 2017] C24:Taking off to the clouds. How to use DMS in...[db tech showcase Tokyo 2017] C24:Taking off to the clouds. How to use DMS in...
[db tech showcase Tokyo 2017] C24:Taking off to the clouds. How to use DMS in...
 
Run Apache Spark on Kubernetes in Large Scale_ Challenges and Solutions-2.pdf
Run Apache Spark on Kubernetes in Large Scale_ Challenges and Solutions-2.pdfRun Apache Spark on Kubernetes in Large Scale_ Challenges and Solutions-2.pdf
Run Apache Spark on Kubernetes in Large Scale_ Challenges and Solutions-2.pdf
 
Glauber Costa on OSv as NoSQL platform
Glauber Costa on OSv as NoSQL platformGlauber Costa on OSv as NoSQL platform
Glauber Costa on OSv as NoSQL platform
 
EDB Postgres with Containers
EDB Postgres with ContainersEDB Postgres with Containers
EDB Postgres with Containers
 
Web componenet using angular element
Web componenet using angular elementWeb componenet using angular element
Web componenet using angular element
 
Varnish - PLNOG 4
Varnish - PLNOG 4Varnish - PLNOG 4
Varnish - PLNOG 4
 
Netflix Container Scheduling and Execution - QCon New York 2016
Netflix Container Scheduling and Execution - QCon New York 2016Netflix Container Scheduling and Execution - QCon New York 2016
Netflix Container Scheduling and Execution - QCon New York 2016
 
Scheduling a fuller house - Talk at QCon NY 2016
Scheduling a fuller house - Talk at QCon NY 2016Scheduling a fuller house - Talk at QCon NY 2016
Scheduling a fuller house - Talk at QCon NY 2016
 
Hot deploy
Hot deployHot deploy
Hot deploy
 

Recently uploaded

UiPath Studio Web workshop series - Day 1
UiPath Studio Web workshop series  - Day 1UiPath Studio Web workshop series  - Day 1
UiPath Studio Web workshop series - Day 1DianaGray10
 
The New Cloud World Order Is FinOps (Slideshow)
The New Cloud World Order Is FinOps (Slideshow)The New Cloud World Order Is FinOps (Slideshow)
The New Cloud World Order Is FinOps (Slideshow)codyslingerland1
 
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptx
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptxEmil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptx
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptxNeo4j
 
.NET 8 ChatBot with Azure OpenAI Services.pptx
.NET 8 ChatBot with Azure OpenAI Services.pptx.NET 8 ChatBot with Azure OpenAI Services.pptx
.NET 8 ChatBot with Azure OpenAI Services.pptxHansamali Gamage
 
How to become a GDSC Lead GDSC MI AOE.pptx
How to become a GDSC Lead GDSC MI AOE.pptxHow to become a GDSC Lead GDSC MI AOE.pptx
How to become a GDSC Lead GDSC MI AOE.pptxKaustubhBhavsar6
 
Key Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdfKey Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdfCheryl Hung
 
TrustArc Webinar - How to Live in a Post Third-Party Cookie World
TrustArc Webinar - How to Live in a Post Third-Party Cookie WorldTrustArc Webinar - How to Live in a Post Third-Party Cookie World
TrustArc Webinar - How to Live in a Post Third-Party Cookie WorldTrustArc
 
Introduction - IPLOOK NETWORKS CO., LTD.
Introduction - IPLOOK NETWORKS CO., LTD.Introduction - IPLOOK NETWORKS CO., LTD.
Introduction - IPLOOK NETWORKS CO., LTD.IPLOOK Networks
 
IT Service Management (ITSM) Best Practices for Advanced Computing
IT Service Management (ITSM) Best Practices for Advanced ComputingIT Service Management (ITSM) Best Practices for Advanced Computing
IT Service Management (ITSM) Best Practices for Advanced ComputingMAGNIntelligence
 
UiPath Studio Web workshop series - Day 4
UiPath Studio Web workshop series - Day 4UiPath Studio Web workshop series - Day 4
UiPath Studio Web workshop series - Day 4DianaGray10
 
Webinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - Tech
Webinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - TechWebinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - Tech
Webinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - TechProduct School
 
LF Energy Webinar - Unveiling OpenEEMeter 4.0
LF Energy Webinar - Unveiling OpenEEMeter 4.0LF Energy Webinar - Unveiling OpenEEMeter 4.0
LF Energy Webinar - Unveiling OpenEEMeter 4.0DanBrown980551
 
Introduction to RAG (Retrieval Augmented Generation) and its application
Introduction to RAG (Retrieval Augmented Generation) and its applicationIntroduction to RAG (Retrieval Augmented Generation) and its application
Introduction to RAG (Retrieval Augmented Generation) and its applicationKnoldus Inc.
 
CyberSecurity - Computers In Libraries 2024
CyberSecurity - Computers In Libraries 2024CyberSecurity - Computers In Libraries 2024
CyberSecurity - Computers In Libraries 2024Brian Pichman
 
Flow Control | Block Size | ST Min | First Frame
Flow Control | Block Size | ST Min | First FrameFlow Control | Block Size | ST Min | First Frame
Flow Control | Block Size | ST Min | First FrameKapil Thakar
 
From the origin to the future of Open Source model and business
From the origin to the future of  Open Source model and businessFrom the origin to the future of  Open Source model and business
From the origin to the future of Open Source model and businessFrancesco Corti
 
Extra-120324-Visite-Entreprise-icare.pdf
Extra-120324-Visite-Entreprise-icare.pdfExtra-120324-Visite-Entreprise-icare.pdf
Extra-120324-Visite-Entreprise-icare.pdfInfopole1
 
Keep Your Finger on the Pulse of Your Building's Performance with IES Live
Keep Your Finger on the Pulse of Your Building's Performance with IES LiveKeep Your Finger on the Pulse of Your Building's Performance with IES Live
Keep Your Finger on the Pulse of Your Building's Performance with IES LiveIES VE
 
UiPath Studio Web workshop Series - Day 3
UiPath Studio Web workshop Series - Day 3UiPath Studio Web workshop Series - Day 3
UiPath Studio Web workshop Series - Day 3DianaGray10
 

Recently uploaded (20)

UiPath Studio Web workshop series - Day 1
UiPath Studio Web workshop series  - Day 1UiPath Studio Web workshop series  - Day 1
UiPath Studio Web workshop series - Day 1
 
The New Cloud World Order Is FinOps (Slideshow)
The New Cloud World Order Is FinOps (Slideshow)The New Cloud World Order Is FinOps (Slideshow)
The New Cloud World Order Is FinOps (Slideshow)
 
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptx
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptxEmil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptx
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptx
 
.NET 8 ChatBot with Azure OpenAI Services.pptx
.NET 8 ChatBot with Azure OpenAI Services.pptx.NET 8 ChatBot with Azure OpenAI Services.pptx
.NET 8 ChatBot with Azure OpenAI Services.pptx
 
SheDev 2024
SheDev 2024SheDev 2024
SheDev 2024
 
How to become a GDSC Lead GDSC MI AOE.pptx
How to become a GDSC Lead GDSC MI AOE.pptxHow to become a GDSC Lead GDSC MI AOE.pptx
How to become a GDSC Lead GDSC MI AOE.pptx
 
Key Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdfKey Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdf
 
TrustArc Webinar - How to Live in a Post Third-Party Cookie World
TrustArc Webinar - How to Live in a Post Third-Party Cookie WorldTrustArc Webinar - How to Live in a Post Third-Party Cookie World
TrustArc Webinar - How to Live in a Post Third-Party Cookie World
 
Introduction - IPLOOK NETWORKS CO., LTD.
Introduction - IPLOOK NETWORKS CO., LTD.Introduction - IPLOOK NETWORKS CO., LTD.
Introduction - IPLOOK NETWORKS CO., LTD.
 
IT Service Management (ITSM) Best Practices for Advanced Computing
IT Service Management (ITSM) Best Practices for Advanced ComputingIT Service Management (ITSM) Best Practices for Advanced Computing
IT Service Management (ITSM) Best Practices for Advanced Computing
 
UiPath Studio Web workshop series - Day 4
UiPath Studio Web workshop series - Day 4UiPath Studio Web workshop series - Day 4
UiPath Studio Web workshop series - Day 4
 
Webinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - Tech
Webinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - TechWebinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - Tech
Webinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - Tech
 
LF Energy Webinar - Unveiling OpenEEMeter 4.0
LF Energy Webinar - Unveiling OpenEEMeter 4.0LF Energy Webinar - Unveiling OpenEEMeter 4.0
LF Energy Webinar - Unveiling OpenEEMeter 4.0
 
Introduction to RAG (Retrieval Augmented Generation) and its application
Introduction to RAG (Retrieval Augmented Generation) and its applicationIntroduction to RAG (Retrieval Augmented Generation) and its application
Introduction to RAG (Retrieval Augmented Generation) and its application
 
CyberSecurity - Computers In Libraries 2024
CyberSecurity - Computers In Libraries 2024CyberSecurity - Computers In Libraries 2024
CyberSecurity - Computers In Libraries 2024
 
Flow Control | Block Size | ST Min | First Frame
Flow Control | Block Size | ST Min | First FrameFlow Control | Block Size | ST Min | First Frame
Flow Control | Block Size | ST Min | First Frame
 
From the origin to the future of Open Source model and business
From the origin to the future of  Open Source model and businessFrom the origin to the future of  Open Source model and business
From the origin to the future of Open Source model and business
 
Extra-120324-Visite-Entreprise-icare.pdf
Extra-120324-Visite-Entreprise-icare.pdfExtra-120324-Visite-Entreprise-icare.pdf
Extra-120324-Visite-Entreprise-icare.pdf
 
Keep Your Finger on the Pulse of Your Building's Performance with IES Live
Keep Your Finger on the Pulse of Your Building's Performance with IES LiveKeep Your Finger on the Pulse of Your Building's Performance with IES Live
Keep Your Finger on the Pulse of Your Building's Performance with IES Live
 
UiPath Studio Web workshop Series - Day 3
UiPath Studio Web workshop Series - Day 3UiPath Studio Web workshop Series - Day 3
UiPath Studio Web workshop Series - Day 3
 

Aarna networks debugging oom failures webinar

  • 1. Debugging OOM Failures An ONAP Webinar © 2018 Aarna Networks, Inc.
  • 2. © 2017 Aarna Networks, Inc. Some Introductions Vivekanandan Muthukrishnan (Vivek) ● Over 17 years of experience in the field of OpenStack, Storage, Network and Server Virtualization Amar Kapadia ● Author of “Understanding OPNFV” ● NFV specialist with background in OpenStack, Ceph, dataplane acceleration technologies
  • 3. © 2017 Aarna Networks, Inc. Goals To create a lab environment for A) ONAP training B) Development purposes
  • 4. © 2017 Aarna Networks, Inc. Requirements/Solution ● Requirements: ○ Light footprint ○ Self-service or instructor provided infrastructure for online and in- class users ● Solution ○ Laptop was eliminated because ONAP is quite heavy ○ So we went with a Google Cloud Platform (GCP) VM ○ GCP offers both options — self-service or instructor provided (ssh) ○ Considered but set aside public OpenStack clouds it is more complicated and OpenStack clouds are regional in nature (easy to add on though for specific needs) ○ Finally decided to create an ONAP+OPNFV-all-in-one VM
  • 5. © 2017 Aarna Networks, Inc. GCP VM Architecture 192.168.x.x SOCKS5
  • 6. © 2017 Aarna Networks, Inc. Our Approach ● Used OOM to get a lighter footprint ○ Amsterdam release ○ Rancher to deploy Kubernetes ○ Scripts to allow users to deploy a subset of ONAP projects vs. all projects ● Used OPNFV for OpenStack ○ Euphrates release ○ Apex installer ○ Basic scenario: no ha, no sdn, no features
  • 7. © 2017 Aarna Networks, Inc. OOM Issues Summary Issues ● AAI containers failed to transition to Running state ● SDC UI is not getting loaded ● SDC Service Distribution Error ● VID Service Deployment Error ● VID ADD VNF Error ● SDNC User creation failed ● Robot init_robot failed with missing attributes Tools Used ● Healthchecks ● Robot logs ● Container logs Outcomes ● Single-command workaround scripts
  • 8. © 2017 Aarna Networks, Inc. OOM Issues & Workaround ● AAI containers failed to transition to Running state ○ NAMESPACE NAME READY STATUS RESTARTS AGE ○ onap aai-service-5698ddc455-s25dd 0/1 Init:0/1 33 6h ● Workaround: bounce AAI containers ○ cd oom/kubernetes/oneclick/ ○ source ./setenv.bash ○ ./deleteAll.bash -n onap -a aai ○ ./createAll.bash -n onap -a aai
  • 9. © 2017 Aarna Networks, Inc. OOM Issues & Workaround ● SDC UI is not getting loaded ○ SDC UI Failed to load ○ Status code : 503 Service Unavailable ● Workaround ○ Bounce SDC containers in specific order: SDC → CS → BE → FE ○ Wait for one container to come up before bouncing the next dependent container
  • 10. © 2017 Aarna Networks, Inc. OOM Issues & Workaround ● SDC Service Distribution Error ○ Error Code : POL5000 ○ Status Code : 500 ● Workaround ○ Bounce global-kafka container ○ Bounce dmaap & dmaap-listener container ○ Bounce SDC containers in order ○ Bounce the vnc-portal & portalwidgets containers
  • 11. © 2017 Aarna Networks, Inc. OOM Issues & Workaround ● VID Service Deployment Error ○ "requestType": "createInstance" ○ "requestStatus": "SVC3001Resource not found... ● Workaround ○ Bounce model-loader-service ○ Run ROBOT scripts init_robot & init
  • 12. © 2017 Aarna Networks, Inc. OOM Issues & Workaround ● VID ADD VNF Error ○ Error : SVC0002 400 Bad request Error ● Workaround ○ Bounce Model-loader-service ○ Bounce mso container ○ Bounce SDC containers in order ○ Execute ROBOT init_robot & init ○ Redistribute the service
  • 13. © 2017 Aarna Networks, Inc. OOM Issues & Workaround ● SDNC User creation failed ○ Database connection issue ● Workaround ○ Bounce sdnc container ○ Bounce sdnc-portal container
  • 14. © 2017 Aarna Networks, Inc. OOM Issues & Workaround ● Robot init_robot failed with missing attributes ○ Missing attribute GLOBAL_INJECTED_UBUNTU_1604_IMAGE ○ Missing attribute GLOBAL_INJECTED_UBUNTU_1404_IMAGE ● Workaround ○ Update file /dockerdata-nfs/onap/robot/eteshare/config/vm_properties.py ○ Include lines after line (GLOBAL_INJECTED_SDNC_PORTAL_IP_ADDR) ■ GLOBAL_INJECTED_UBUNTU_1604_IMAGE = "xenial" ■ GLOBAL_INJECTED_UBUNTU_1404_IMAGE = "trusty ○ JSON attributes under GLOBAL_INJECTED_PROPERTIES ■ "GLOBAL_INJECTED_UBUNTU_1604_IMAGE" : "xenial" ■ "GLOBAL_INJECTED_UBUNTU_1404_IMAGE" : "trusty"
  • 15. © 2017 Aarna Networks, Inc. OOM Debugging Approaches ● Check SDC BE & FE health status ○ curl -s -X GET "http://<ONAP SERVER IP>:30205/sdc2/rest/healthCheck" | python -m json.tool ○ curl -s -X GET http://<ONAP SERVER IP>:30206/sdc1/rest/healthCheck | python -m json.tool ● Check ROBOT container logs ○ Robot portal URL = http://<ONAP SERVER IP>:30209/logs/demo/ ○ username/password = test/test ● Connect & check the target container logs (folder /var/log/onap) ○ kubectl exec -it -n {NAME SPACE} {CONTAINER ID} -- /bin/bash ○ kubectl describe pod -n {NAME SPACE} {CONTAINER ID} ● Bouncing a failed container ○ kubectl delete pod -n {NAME SPACE} {CONTAINER ID}
  • 16. © 2017 Aarna Networks, Inc. Debugging Outcome ● Created easy to use scripts that address each issue ● Training participants are given instructions on which workaround to run in a particular error situation
  • 17. © 2017 Aarna Networks, Inc. About Aarna ONAP related products, training, and professional services Founded in 2017 by serial entrepreneurs (NFVI/VIM experts) Located in San Jose, CA & Bangalore, India 60 participants trained in 8 sessions 2 development distros & 4 packaged services