Overview of OpenStack, tools required for contributing to OpenStack, high-level introduction of Project Solum

1. Tutorial
on
Contribu-ng
to
OpenStack
Devda8a
Kulkarni
PTL,
OpenStack
Solum

2. Agenda
• Part
I
– Introduc-ons
– Introduc-on
to
OpenStack
• Part
II
– Deep
dive
of
a
OpenStack
project
-­‐
Solum
• Part
III
– Contribu-ng
to
OpenStack
– OpenStack
CI
system
– Development
best
prac-ces
– Hands-­‐on
exercise
• Ques-ons
and
Answers

3. Agenda
• Part
I
– Introduc,ons
– Introduc,on
to
OpenStack
• Part
II
– Deep
dive
of
a
OpenStack
project
-­‐
Solum
• Part
III
– Contribu-ng
to
OpenStack
– OpenStack
CI
system
– Development
best
prac-ces
– Hands-­‐on
exercise
• Ques-ons
and
Answers

4. Introduc-on
• Me
– Devda8a
Kulkarni
– First
contributor
on
Solum
– Currently
Project
Team
Lead
(PTL)
of
Solum
for
Mitaka
cycle
– PhD,
University
of
Minnesota
Minneapolis
• You

5. Introduc-on
to
OpenStack

6. What
is
OpenStack?
A
family
of
“open”
projects
that
together
enable
different
aspects
of
cloud
compu-ng
Openness
of
OpenStack
h8ps://wiki.openstack.org/wiki/Open
• Open
source
• Open
design
• Open
development
• Open
community

7. OpenStack
family
• Family
of
projects
enabling
different
aspects
of
cloud
compu-ng
– Authen-ca-on
and
authoriza-on
layer
• Keystone
– Infrastructure-­‐as-­‐a-­‐service
layer
• Nova,
Glance
– Storage
layer
• Swi[,
Cinder
– Networking
layer
• Neutron
– Plaorm-­‐as-­‐a-­‐service
layer
• Solum,
Heat,
Murano,
Mistral,
Trove
• Project
types
– OpenStack
big
tent
vs.
OpenStack
related
projects

8. OpenStack
by
numbers
• 49
projects
in
OpenStack
big
tent
• Several
hundred
organiza-ons
involved
• Several
thousand
developers
par-cipa-ng

9. Agenda
• Part
I
– Introduc-ons
– Introduc-on
to
OpenStack
• Part
II
– Deep
dive
of
a
OpenStack
project
-­‐
Solum
• Part
III
– Contribu-ng
to
OpenStack
– OpenStack
CI
system
– Development
best
prac-ces
– Hands-­‐on
exercise
• Ques-ons
and
Answers

10.
What
is
Solum?
OpenStack
project
that
provides
easy
mechanism
for
applica-on
developers
to
deploy
and
run
their
applica-ons
on
OpenStack
star-ng
from
applica-on’s
source
code
Why
should
you
care?
(the
value
proposi-on)
– For
operators,
Solum
provides
ability
to
make
your
OpenStack
cloud
more
useful
for
your
clouds’
applica-on
developers
by
simplifying
the
process
of
deploying
applica-ons
to
it
– For
developers,
Solum
provides
an
easy-­‐to-­‐use
plaorm
for
building,
tes-ng,
and
deploying
applica-ons
on
OpenStack
clouds

11.
Project
goals
Applica-on
Stack
Flexibility
Ability
to
support
applica-ons
wri8en
in
different
languages
and
using
different
applica-on
frameworks
Developer
Produc-vity
Ability
to
perform
CI/CD,
integra-on
with
github
Add-­‐On
Services
Extensibility
Ability
to
support
different
add-­‐on
services
Applica-on
Portability
Ability
to
deploy
applica-ons
across
different
OpenStack
clouds
(use
na-ve
OpenStack
services)

12.
Project
details
• Wiki:
h8ps://wiki.openstack.org/wiki/Solum
• Mee-ngs:
Tuesdays
1700
UTC
in
#openstack-­‐mee-ng-­‐3
• IRC:
Channel:
#solum
on
freenode
• Code:
h8ps://github.com/openstack/solum
h8ps://github.com/openstack/python-­‐solumclient

13.
Solum
abstrac-ons
• App
• Languagepack
(LP)
• Deployment
Unit
(DU)
• Workflow
• Add-­‐on

14.
Solum
abstrac-ons
-­‐
App
Declara-ve
applica-on
defini-on
• source
code
repository
• run
command
• port
• languagepack
version: 1
name: cherrypy
description: python web app
languagepack: python
source:
repository: https://github.com/
rackspace-solum-samples/solum-
python-sample-app.git
revision: master
workflow_config:
run_cmd: python app.py
ports:
- 80

15.
Solum
abstrac-ons
-­‐
Languagepack
(LP)
• Docker
image
which
contains
applica-on
specific
build
and
run-me
libraries
• A
LP
needs
to
implement
Solum’s
languagepack
contract
A
well-­‐known
file
available
at
a
known
loca-on
on
the
languagepack
image
/solum/bin/build.sh
• How
to
build
a
LP?
Dockerfile
• Operator-­‐defined
or
user-­‐defined

16.
Solum
abstrac-ons
-­‐
Languagepack
Example:
Python
FROM ubuntu:precise
MAINTAINER Murali Allada
<murali.allada@rackspace.com>
RUN apt-get -yqq update
RUN apt-get -yqq install python-pip
RUN apt-get -yqq install python-dev
COPY build.sh /solum/bin/
https://github.com/rackspace-solum-samples/solum-languagepack-python
#!/bin/bash
# Check if pip is installed
pip help
[[ $? != 0 ]] && echo python-pip is
not installed. && exit 1
# Install app dependencies
cd /app
pip install -r requirements.txt
build.shDockerfile

17.
Solum
abstrac-ons
-­‐
Deployment
Unit
(DU)
(1/2)
• Docker
image
that
is
formed
from
the
languagepack
image
with
applica-on’s
source
code
added
to
it
DU
=
LP
+
applica-on
source
code
• Solum
guarantees
the
DU
contract
Applica-on
source
code
available
at
a
known
loca-on
on
the
DU
image
/app
• How
to
build
a
DU?
– Construct
Dockerfile
with
languagepack
as
the
base
image,
applica-on
source
code
injected,
run
command
as
the
entry
point
– Build
the
DU
image
from
this
Dockerfile

18.
Solum
abstrac-ons
-­‐
Deployment
Unit
(DU)
(2/2)
• Building
a
DU
– Start
from
the
specified
languagepack
LP
– Specify
execu-on
of
languagepack’s
‘build.sh’
– Inject
applica-on
source
code
– Use
the
run
command
specified
in
app
defini-on
as
the
default
Entrypoint
to
run
the
DU
• LP
and
DU
storage
Glance,
Swi[,
Docker
registry
18

19. Solum
abstrac-ons
-­‐
Workflow
• Abstrac-on
to
represent
execu-on
of
applica-on
deployment
consis-ng
of
one
or
more
workflow
stages
• Supported
workflow
stages
– Build
DU
(and
store
it
for
future
use)
– Run
unit
tests,
build
DU
– Run
unit
tests,
build
DU,
deploy
DU
(if
unit
tests
pass)
– Build
DU,
deploy
DU
– Deploy
a
previously
built
DU
(not
yet
available)
• A
workflow
can
be
triggered
from
github
webhooks

20. Solum
abstrac-ons
-­‐
Add-­‐ons
• Services
needed
by
an
applica-on
E.g.:
rela-onal
database
such
as
Trove
• DU
parameters
Solum
supports
ability
to
pass
service’s
connec-on
parameters
to
applica-on
DU

21. Architecture
and
system
details

22.
High-­‐level
architecture
Solum
API
Solum
Worker
Solum
Deployer
Queue
Queue
Queue
Solum
Conductor
Heat
Solum
Database
Glance,
Swi[,
Docker
Registry
Builds
LP
and
DU
Deploys
DU
-­‐ Decoupled
architecture
-­‐ Asynchronous
communica-on

23. Demo
h8ps://vimeo.com/143425822

24. Agenda
• Part
I
– Introduc-ons
– Introduc-on
to
OpenStack
• Part
II
– Deep
dive
of
a
OpenStack
project
-­‐
Solum
• Part
III
– Contribu,ng
to
OpenStack
– OpenStack
CI
system
– Development
best
prac,ces
– Hands-­‐on
exercise
• Ques-ons
and
Answers

25. Contribu-ng
to
OpenStack

26. Tools/Systems
• Gerrit:
Code
review
system
– h8ps://review.openstack.org/
• Launchpad:
Work
(bugs/blueprints)
tracking
system
– h8ps://launchpad.net/
• Github:
Code
repository
– h8ps://github.com/openstack
• Git:
Code
contribu-on
• Mailing
list:
openstack-­‐dev
– h8p://lists.openstack.org/cgi-­‐bin/mailman/lis-nfo/openstack-­‐dev
• IRC:
chat.freenode.net

27. Setup
steps
(1/2)
• h8p://docs.openstack.org/infra/manual/developers.html
• Create
Launchpad
account
• Join
OpenStack
founda-on
as
founda-on
member
• Sign
individual
contributor’s
license
agreement
• Install
git-­‐review
• Install
git
• Run
git
review
-­‐s

28. Setup
steps:
Problems
and
troubleshoo-ng
• Problem:
We
don't
know
where
your
gerrit
is.
Please
manually
create
a
remote
• Solu-on:
– Create
a
username
in
gerrit
if
not
already
created
– Manually
set
gerrit
remote.
E.g.:
git
remote
add
gerrit
"ssh://
djkulkarni@review.openstack.org:29418/openstack/
solum.git”

29. Setup
steps:
Problems
and
troubleshoo-ng
• Problem:
Problem
running
'git
remote
update
gerrit’
Fetching
gerrit
Permission
denied
(publickey).
fatal:
Could
not
read
from
remote
repository.
• Solu-on:
– Generate
sshkey
and
add
it
to
gerrit
– ssh-­‐keygen

30. Steps
to
submit
a
patch
• Create
a
Launchpad
bug
or
assign
an
exis-ng
bug
to
yourself
• Fixing
the
bug
– Itera-ve
process
• Understand
the
issue
(be
in
touch
with
project
developers
on
project’s
IRC
channel
or
on
openstack-­‐dev
mailing
list)
• Design
solu-on
• Write
code
• Write
tests
• Commit
code
• Provide
good
commit
message
• Submit
patch
for
review

31. Steps
to
merge
a
patch
• Patch
needs
to
pass
configured
automated
tests
• Patch
needs
to
get
two
+2
votes
• Patch
needs
to
be
approved
for
merging
by
project’s
core
reviewer

32. OpenStack
CI
System

33. Con-nuous
Integra-on
of
patches
• Systems
involved:
– Zuul
– Gerrit
– Jenkins
– Github
• Pre-­‐merge
checking
– Patch
is
merged
to
master
only
if
all
the
configured
tests
pass

34. OpenStack’s
CI
setup
Gerrit
Zuul
Jenkins
Github
2
3
4
5
1
Patch
submi8ed
for
review
Patch
picked
up
for
tes-ng
Tests
ran
on
the
patch
Jenkins
gives
its
vote
Code
merged

35. Patch
review
page
details

36. Gerrit:
Patch
that
fails
gate
tests

37. Development
best
prac-ces

38. Best
prac-ces,
tricks,
-ps
• Devstack
• Vagrant
and
Virtualbox
• Python
• Screen
• Debugging
• Git

39. Devstack
• OpenStack
environment
consis-ng
of
all
the
openstack-­‐services
– Keystone,
Nova,
Heat,
Solum,
etc.
• Can
be
run
within
a
virtual
machine
or
independently
• Useful
for
development

40. Vagrant
and
VirtualBox
• Vagrant
– Configura-on
management
tool
for
configuring
virtual
machines
• VirtualBox
– Virtualiza-on
system
• Spinning
up
a
development
environment
using
Vagrant
+
VirtualBox
– h8ps://github.com/rackerlabs/vagrant-­‐solum-­‐dev
– git
clone
<solum-­‐vagrant-­‐repo>
– vagrant
up
-­‐-­‐provision

41. Sexng
up
Solum
development
env
• Develop
on
Host,
debug
on
the
VM
• Install
Vagrant
and
Virtualbox
• Clone
solum
and
python-­‐solumclient
repositories
on
the
Host
• Set
following
environment
variables:
– SOLUM=<path
on
Host
system
to
solum
repo>
– SOLUMCLIENT=<path
on
Host
system
to
python-­‐solumclient
repo>
• vagrant
up
-­‐-­‐provision
• vagrant
ssh

42. Python
• PyCharm
IDE
• pdb
– Put
breakpoint
in
the
code
where
you
want
to
break
• import
pdb;
pdb.set_trace()
– pdb
commands
• s:
Execute
the
current
line,
stop
at
first
possible
occasion
• c:
Con-nue
execu-on,
only
stop
when
a
breakpoint
is
encountered
• h8ps://docs.python.org/2/library/pdb.html

43. Tes-ng
• Run
tests
locally
before
submixng
patch
for
review
– From
within
solum
directory
• tox
–epep8
• tox
–epy27
• Tox
– Command
line
tool
for
running
tests
within
virtualenv

44. Code
paths
on
Devstack
• solum
/opt/stack/solum
• python-­‐solumclient
/opt/stack/python-­‐solumclient
• Solum
logs
– /var/log/solum/deployer
– /var/log/solum/worker
• Devstack
logs
(of
various
screen
sessions)
– /opt/stack/logs/

45. Screen
• Program
that
allows
sharing
physical
terminal
between
different
processes
• Entering
Solum’s
screen
session
in
Devstack
– screen
–x
stack
• Screen
commands
– Move
to
next
window:
Ctrl+a
n
– Move
to
previous
window:
Ctrl+a
p
– See
all
windows:
Ctrl+a
“
– Enter
scrollback/copy
mode:
Ctrl+a
[
– Leave
scrollback/copy
mode:
Ctrl+a
]
– h8p://www.pixelbeat.org/lkdb/screen.html

46. Debugging
• Debug
code/feature
– Put
breakpoints
in
the
code
– Invoke
code-­‐under-­‐test
from
CLI
– Use
screen
to
enter
breakpoint
and
then
debug
using
Pdb
• Debug
failing
func-onal
test
– h8ps://review.openstack.org/#/c/254344/
• Debug
failing
unit
test

47. Git
• Crea-ng
dependent
commits
• Squashing
commits
– git
rebase
-­‐i
HEAD~<number-­‐of-­‐commits>
– pick/squash/edit
• Upda-ng
dependent
commit
and
submixng
for
review
• h8ps://ask.openstack.org/en/ques-on/31633/
gerrit-­‐best-­‐way-­‐to-­‐make-­‐a-­‐series-­‐of-­‐dependent-­‐
commits/

48. Hands-­‐on
exercise
• Submit
first
patch
to
OpenStack
(45
minutes)
• h8ps://etherpad.openstack.org/p/solum-­‐low-­‐
hanging-­‐fruit-­‐bugs

49. Ques-ons