The document discusses using Jenkins as a Python execution platform. It covers running Python code in Jenkins jobs, performing input/output operations, and persisting state across jobs using artifacts. The key points covered include running Python via shell scripts or plugins, passing data between jobs using environment variables, files, and serialized objects stored as artifacts, and sharing a JVM between Jenkins and Python using Jython. Examples of applications like the oVirt CI system are also mentioned.

1. The Snake and theThe Snake and the
ButlerButler
Jenkins as a Python execution
platform

2. Who am I?Who am I?
Barak Korren
bkorren@redhat.com
http://ifireball.wordpress.com
https://twitter.com/BKorren

3. AgendaAgenda
● What is Jenkins / Why do it
● Running Python code
● Doing I/O
● Persisting state

4. Jenkins?Jenkins?
“CI/CD Automation tool"

5. Jenkins?Jenkins?

6. Jenkins?Jenkins?

7. Jenkins?Jenkins?

8. Why?Why?

9. Jenkins JobsJenkins Jobs
“Free Style" Pipeline
String code
String status_file = 'calculator_status.dat'
stage('Load code') {
dir('python-on-jenkins') {
git(
poll: false,
url: repo_url
)
code = readFile code_file
}
}
stage('Load data') {
step([
$class: 'CopyArtifact',
filter: status_file,
fingerprintArtifacts: true,
projectName: env.JOB_NAME,
optional: true,
])
}

10. RunningRunning
Python codePython code

11. Python cmd linePython cmd line
def main():
args = parse_args()
print("Hello {}".format(args.who))
def parse_args():
"""Parse command line arguments
"""
parser = argparse.ArgumentParser(
Description=
'Tool that says hello'
)
parser.add_argument(
'who',
default='world',
nargs='?'
)
return parser.parse_args()

12. Python pluginsPython plugins
● "python"
● "ShiningPanda"
● "jython"

13. Python for “free”Python for “free”
node {
stage('Say hello') {
sh '''
#!/usr/bin/env python
from __future__ import
print_function
print('Hello world!')
'''.stripIndent()
}
}

14. Doing I/ODoing I/O

15. InputInput
from os import environ
name = environ.get('YOUR_NAME', 'World')
print('Hello {0}!'.format(name))

16. OutputOutput
with open('job_params.properties', 'w') as f:
f.write('YOUR_NAME={0}n'.format(name))

17. Move ObjectsMove Objects
def object_to_param_str(obj):
return b64encode(compress(
cPickle.dumps(obj)
)).decode('utf8')
def param_str_to_object(param_str):
return cPickle.loads(decompress(
b64decode(param_str.encode('utf8'))
))

18. Output to GroovyOutput to Groovy
build_spec = dict(
job='hello-with-params',
parameters=[{
'$class':
'StringParameterValue',
'name': 'YOUR_NAME',
'value': name
}]
)
with open('bld_spec.json', 'w') as f:
json.dump(build_spec, f)

19. Output to GroovyOutput to Groovy
build_spec = dict(
job='hello-with-params',
parameters=[{
'$class':
'StringParameterValue',
'name': 'YOUR_NAME',
'value': name
}]
)
with open('bld_spec.json', 'w') as f:
json.dump(build_spec, f)
build readJSON(
file: 'bld_spec.json'
)

20. Persisting statePersisting state

21. Persisting statePersisting state
Build #8Build #8 Build #9Build #9
Artifact StorageArtifact Storage
Archive Load

22. Persisting statePersisting state
def object_from_artifact(artifact_file, fallback_cls):
try:
with open(artifact_file) as fd:
return cPickle.load(fd)
except IOError as e:
# errno 2 is 'No such file or directory'
if e.errno == 2:
return fallback_cls()
raise
def object_to_artifact(obj, artifact_file):
with open(artifact_file, 'w') as fd:
cPickle.dump(obj, fd)

23. Persisting statePersisting state
@contextmanager
def persist_in_artifacts(artifact_file, fallback_cls):
obj = object_from_artifact(artifact_file,
fallback_cls)
yield obj
object_to_artifact(obj, artifact_file)

24. Persisting statePersisting state
@contextmanager
def persist_in_artifacts(artifact_file, fallback_cls):
obj = object_from_artifact(artifact_file,
fallback_cls)
yield obj
object_to_artifact(obj, artifact_file)
class CalculatorStatus(object):
def __init__(self):
self.current_value = 0

25. Persisting statePersisting state
@contextmanager
def persist_in_artifacts(artifact_file, fallback_cls):
obj = object_from_artifact(artifact_file,
fallback_cls)
yield obj
object_to_artifact(obj, artifact_file)
class CalculatorStatus(object):
def __init__(self):
self.current_value = 0
with persist_in_artifacts(status_file,
CalculatorStatus) as status:
status.current_value = ...

26. Persisting statePersisting state
stage('Load data') {
dir('state') { deleteDir(); touch(file: '_dummy_') }
step([
$class: 'CopyArtifact',
filter: status_file,
projectName: env.JOB_NAME,
optional: true,
])
}
stage('Run python') {
// ...
}
stage('Save data') {
archive status_file
}

27. RecapRecap
●
Running code via simple "shell" build
steps
● Input via env vars
●
Output via files (properties, json,
Pickle)
●
Object persistence via archived Pickle
files

28. Sharing the JVMSharing the JVM
● Jython:
– http://www.jython.org/
● Java classes in Jython
– https://github.com/jythontools/clamp/
● Python (Jython) Plugins
– https://github.com/jenkinsci/jenkins.py

29. Jython from GroovyJython from Groovy
// load interpreter from Maven
@Grab(group='org.python', module='jython-standalone',
version='2.7.0')
import org.python.util.PythonInterpreter
import org.python.core.PyString

30. Jython from GroovyJython from Groovy
@NonCPS
def loadPython() {
// create interpreter object
PythonInterpreter interpreter = new PythonInterpreter()
// compile and run python
interpreter.exec """
def some_python_func(some_arg):
# python code here...
""".stripIndent()
// return some functions
return interpreter.get('some_python_func')
}

31. Jython from GroovyJython from Groovy
pyfunc = loadPython()
# Call Python function
pyfunc.__call__(new PyString("some str..."))

32. ApplicationsApplications
//
DemoDemo

33. Thank You!Thank You!
● Slides:
– http://wp.me/p7png-Ct
● Source code:
– https://github.com/ifireball/python-on-jenkins
● Contact me:
– bkorren@redhat.com

34. Jenkins Job BuilderJenkins Job Builder
https://docs.openstack.org/infra/jenkins-job-builder/
- job-template:
name: some_job_a
parameters:
- name-params
builders:
- shell: !include-raw-escape: code.py
publishers:
- trigger-parameterized-builds:
- project: some_job_b
property-file: job_params.properties
condition: SUCCESS

35. ApplicationsApplications
● oVirt "Standard-CI":
– oVirt has many
different sub-projects
– The CI system is
project-agnostic
– Each project tell the CI
what to do
– We have local tool to
emulate CI system

36. ApplicationsApplications
●
oVirt “change-gating" flow:
– Runs “heavy” tests on batches of changes
– Runs bisection on failure to find root cause
Project 1Project 1
Project 2Project 2
Project NProject N
Change
queue
Change
queue
TestsTests