Contributing to OpenStack

1. Tutorial on Contribu-ng to OpenStack Devda8a Kulkarni PTL, OpenStack Solum

2. Agenda •  Part I –  Introduc-ons –  Introduc-on to OpenStack •  Part II –  Deep dive of a OpenStack project -‐ Solum •  Part III –  Contribu-ng to OpenStack –  OpenStack CI system –  Development best prac-ces –  Hands-‐on exercise •  Ques-ons and Answers

3. Agenda •  Part I –  Introduc,ons –  Introduc,on to OpenStack •  Part II –  Deep dive of a OpenStack project -‐ Solum •  Part III –  Contribu-ng to OpenStack –  OpenStack CI system –  Development best prac-ces –  Hands-‐on exercise •  Ques-ons and Answers

4. Introduc-on •  Me – Devda8a Kulkarni – First contributor on Solum – Currently Project Team Lead (PTL) of Solum for Mitaka cycle – PhD, University of Minnesota Minneapolis •  You

5. Introduc-on to OpenStack

6. What is OpenStack? A family of “open” projects that together enable diﬀerent aspects of cloud compu-ng Openness of OpenStack h8ps://wiki.openstack.org/wiki/Open •  Open source •  Open design •  Open development •  Open community

7. OpenStack family •  Family of projects enabling diﬀerent aspects of cloud compu-ng –  Authen-ca-on and authoriza-on layer •  Keystone –  Infrastructure-‐as-‐a-‐service layer •  Nova, Glance –  Storage layer •  Swi[, Cinder –  Networking layer •  Neutron –  Plaorm-‐as-‐a-‐service layer •  Solum, Heat, Murano, Mistral, Trove •  Project types –  OpenStack big tent vs. OpenStack related projects

8. OpenStack by numbers •  49 projects in OpenStack big tent •  Several hundred organiza-ons involved •  Several thousand developers par-cipa-ng

9. Agenda •  Part I –  Introduc-ons –  Introduc-on to OpenStack •  Part II –  Deep dive of a OpenStack project -‐ Solum •  Part III –  Contribu-ng to OpenStack –  OpenStack CI system –  Development best prac-ces –  Hands-‐on exercise •  Ques-ons and Answers

10. What is Solum? OpenStack project that provides easy mechanism for applica-on developers to deploy and run their applica-ons on OpenStack star-ng from applica-on’s source code Why should you care? (the value proposi-on) –  For operators, Solum provides ability to make your OpenStack cloud more useful for your clouds’ applica-on developers by simplifying the process of deploying applica-ons to it –  For developers, Solum provides an easy-‐to-‐use plaorm for building, tes-ng, and deploying applica-ons on OpenStack clouds

11. Project goals Applica-on Stack Flexibility Ability to support applica-ons wri8en in different languages and using different applica-on frameworks Developer Produc-vity Ability to perform CI/CD, integra-on with github Add-‐On Services Extensibility Ability to support different add-‐on services Applica-on Portability Ability to deploy applica-ons across different OpenStack clouds (use na-ve OpenStack services)

12. Project details •  Wiki: h8ps://wiki.openstack.org/wiki/Solum •  Mee-ngs: Tuesdays 1700 UTC in #openstack-‐mee-ng-‐3 •  IRC: Channel: #solum on freenode •  Code: h8ps://github.com/openstack/solum h8ps://github.com/openstack/python-‐solumclient

13. Solum abstrac-ons •  App •  Languagepack (LP) •  Deployment Unit (DU) •  Workﬂow •  Add-‐on

14. Solum abstrac-ons -‐ App Declara-ve applica-on deﬁni-on •  source code repository •  run command •  port •  languagepack version: 1 name: cherrypy description: python web app languagepack: python source: repository: https://github.com/ rackspace-solum-samples/solum- python-sample-app.git revision: master workflow_config: run_cmd: python app.py ports: - 80

15. Solum abstrac-ons -‐ Languagepack (LP) •  Docker image which contains applica-on specific build and run-me libraries •  A LP needs to implement Solum’s languagepack contract A well-‐known file available at a known loca-on on the languagepack image /solum/bin/build.sh •  How to build a LP? Dockerfile •  Operator-‐defined or user-‐defined

16. Solum abstrac-ons -‐ Languagepack Example: Python FROM ubuntu:precise MAINTAINER Murali Allada <murali.allada@rackspace.com> RUN apt-get -yqq update RUN apt-get -yqq install python-pip RUN apt-get -yqq install python-dev COPY build.sh /solum/bin/ https://github.com/rackspace-solum-samples/solum-languagepack-python #!/bin/bash # Check if pip is installed pip help [[ $? != 0 ]] && echo python-pip is not installed. && exit 1 # Install app dependencies cd /app pip install -r requirements.txt build.shDockerfile

17. Solum abstrac-ons -‐ Deployment Unit (DU) (1/2) •  Docker image that is formed from the languagepack image with applica-on’s source code added to it DU = LP + applica-on source code •  Solum guarantees the DU contract Applica-on source code available at a known loca-on on the DU image /app •  How to build a DU? –  Construct Dockerﬁle with languagepack as the base image, applica-on source code injected, run command as the entry point –  Build the DU image from this Dockerﬁle

18. Solum abstrac-ons -‐ Deployment Unit (DU) (2/2) •  Building a DU –  Start from the specified languagepack LP –  Specify execu-on of languagepack’s ‘build.sh’ –  Inject applica-on source code –  Use the run command specified in app defini-on as the default Entrypoint to run the DU •  LP and DU storage Glance, Swi[, Docker registry 18

19. Solum abstrac-ons -‐ Workflow •  Abstrac-on to represent execu-on of applica-on deployment consis-ng of one or more workflow stages •  Supported workflow stages –  Build DU (and store it for future use) –  Run unit tests, build DU –  Run unit tests, build DU, deploy DU (if unit tests pass) –  Build DU, deploy DU –  Deploy a previously built DU (not yet available) •  A workflow can be triggered from github webhooks

20. Solum abstrac-ons -‐ Add-‐ons •  Services needed by an applica-on E.g.: rela-onal database such as Trove •  DU parameters Solum supports ability to pass service’s connec-on parameters to applica-on DU

21. Architecture and system details

22. High-‐level architecture Solum API Solum Worker Solum Deployer Queue Queue Queue Solum Conductor Heat Solum Database Glance, Swi[, Docker Registry Builds LP and DU Deploys DU -‐  Decoupled architecture -‐  Asynchronous communica-on

23. Demo h8ps://vimeo.com/143425822

24. Agenda •  Part I –  Introduc-ons –  Introduc-on to OpenStack •  Part II –  Deep dive of a OpenStack project -‐ Solum •  Part III –  Contribu,ng to OpenStack –  OpenStack CI system –  Development best prac,ces –  Hands-‐on exercise •  Ques-ons and Answers

25. Contribu-ng to OpenStack

26. Tools/Systems •  Gerrit: Code review system –  h8ps://review.openstack.org/ •  Launchpad: Work (bugs/blueprints) tracking system –  h8ps://launchpad.net/ •  Github: Code repository –  h8ps://github.com/openstack •  Git: Code contribu-on •  Mailing list: openstack-‐dev –  h8p://lists.openstack.org/cgi-‐bin/mailman/lis-nfo/openstack-‐dev •  IRC: chat.freenode.net

27. Setup steps (1/2) •  h8p://docs.openstack.org/infra/manual/developers.html •  Create Launchpad account •  Join OpenStack founda-on as founda-on member •  Sign individual contributor’s license agreement •  Install git-‐review •  Install git •  Run git review -‐s

28. Setup steps: Problems and troubleshoo-ng •  Problem: We don't know where your gerrit is. Please manually create a remote •  Solu-on: – Create a username in gerrit if not already created – Manually set gerrit remote. E.g.: git remote add gerrit "ssh:// djkulkarni@review.openstack.org:29418/openstack/ solum.git”

29. Setup steps: Problems and troubleshoo-ng •  Problem: Problem running 'git remote update gerrit’ Fetching gerrit Permission denied (publickey). fatal: Could not read from remote repository. •  Solu-on: –  Generate sshkey and add it to gerrit –  ssh-‐keygen

30. Steps to submit a patch •  Create a Launchpad bug or assign an exis-ng bug to yourself •  Fixing the bug –  Itera-ve process •  Understand the issue (be in touch with project developers on project’s IRC channel or on openstack-‐dev mailing list) •  Design solu-on •  Write code •  Write tests •  Commit code •  Provide good commit message •  Submit patch for review

31. Steps to merge a patch •  Patch needs to pass conﬁgured automated tests •  Patch needs to get two +2 votes •  Patch needs to be approved for merging by project’s core reviewer

32. OpenStack CI System

33. Con-nuous Integra-on of patches •  Systems involved: – Zuul – Gerrit – Jenkins – Github •  Pre-‐merge checking – Patch is merged to master only if all the conﬁgured tests pass

34. OpenStack’s CI setup Gerrit Zuul Jenkins Github 2 3 4 5 1 Patch submi8ed for review Patch picked up for tes-ng Tests ran on the patch Jenkins gives its vote Code merged

35. Patch review page details

36. Gerrit: Patch that fails gate tests

37. Development best prac-ces

38. Best prac-ces, tricks, -ps •  Devstack •  Vagrant and Virtualbox •  Python •  Screen •  Debugging •  Git

39. Devstack •  OpenStack environment consis-ng of all the openstack-‐services – Keystone, Nova, Heat, Solum, etc. •  Can be run within a virtual machine or independently •  Useful for development

40. Vagrant and VirtualBox •  Vagrant –  Conﬁgura-on management tool for conﬁguring virtual machines •  VirtualBox –  Virtualiza-on system •  Spinning up a development environment using Vagrant + VirtualBox –  h8ps://github.com/rackerlabs/vagrant-‐solum-‐dev –  git clone <solum-‐vagrant-‐repo> –  vagrant up -‐-‐provision

41. Sexng up Solum development env •  Develop on Host, debug on the VM •  Install Vagrant and Virtualbox •  Clone solum and python-‐solumclient repositories on the Host •  Set following environment variables: –  SOLUM=<path on Host system to solum repo> –  SOLUMCLIENT=<path on Host system to python-‐solumclient repo> •  vagrant up -‐-‐provision •  vagrant ssh

42. Python •  PyCharm IDE •  pdb – Put breakpoint in the code where you want to break •  import pdb; pdb.set_trace() – pdb commands •  s: Execute the current line, stop at ﬁrst possible occasion •  c: Con-nue execu-on, only stop when a breakpoint is encountered •  h8ps://docs.python.org/2/library/pdb.html

43. Tes-ng •  Run tests locally before submixng patch for review – From within solum directory •  tox –epep8 •  tox –epy27 •  Tox – Command line tool for running tests within virtualenv

44. Code paths on Devstack •  solum /opt/stack/solum •  python-‐solumclient /opt/stack/python-‐solumclient •  Solum logs –  /var/log/solum/deployer –  /var/log/solum/worker •  Devstack logs (of various screen sessions) –  /opt/stack/logs/

45. Screen •  Program that allows sharing physical terminal between diﬀerent processes •  Entering Solum’s screen session in Devstack –  screen –x stack •  Screen commands –  Move to next window: Ctrl+a n –  Move to previous window: Ctrl+a p –  See all windows: Ctrl+a “ –  Enter scrollback/copy mode: Ctrl+a [ –  Leave scrollback/copy mode: Ctrl+a ] –  h8p://www.pixelbeat.org/lkdb/screen.html

46. Debugging •  Debug code/feature – Put breakpoints in the code – Invoke code-‐under-‐test from CLI – Use screen to enter breakpoint and then debug using Pdb •  Debug failing func-onal test – h8ps://review.openstack.org/#/c/254344/ •  Debug failing unit test

47. Git •  Crea-ng dependent commits •  Squashing commits – git rebase -‐i HEAD~<number-‐of-‐commits> – pick/squash/edit •  Upda-ng dependent commit and submixng for review •  h8ps://ask.openstack.org/en/ques-on/31633/ gerrit-‐best-‐way-‐to-‐make-‐a-‐series-‐of-‐dependent-‐ commits/

48. Hands-‐on exercise •  Submit ﬁrst patch to OpenStack (45 minutes) •  h8ps://etherpad.openstack.org/p/solum-‐low-‐ hanging-‐fruit-‐bugs

49. Ques-ons

Contributing to OpenStack

devkulkarni

Contributing to OpenStack