Download to read offline
![node (‘docker’) {
dir (“ws${env.BRANCH_NAME}”) {
withEnv ([PATH+M=${tool ‘M3’]) {
sh ‘mvn test-compile’
}
archive ‘**/license/*.*’
}
}
11
The Jenkinsfile
sh
timestamps
node
parallel
checkout
readFile
marathon
Pipelines as code](https://image.slidesharecdn.com/jamlimitaciones-dificultades-170426135638/85/Madrid-JAM-limitaciones-dificultades-11-320.jpg)
![def S1 = ‘FAI’
def S2 = “${-> S1}TH”
echo “${S2}”
===>
org.jenkinsci.plugins.scriptsecurity.sandbox
.RejectedAccessException: unclassified new
org.codehaus.groovy.runtime.GStringImpl
java.lang.String java.lang.String[]
17
Double Lazy GString
What if I need to lazily
evaluate a variable?
Oops](https://image.slidesharecdn.com/jamlimitaciones-dificultades-170426135638/85/Madrid-JAM-limitaciones-dificultades-17-320.jpg)
![def L1 = [] //constructed as List
synchronized(L1) {
L1.add (1)
}
echo L1.inspect()
===>
java.lang.UnsupportedOperationException
Collections.synchronizedList([]) seems
to work (now)
20
.synchronizedList
Parallel step is powerful
But it is parallel and
synchronization should
be needed
Oops](https://image.slidesharecdn.com/jamlimitaciones-dificultades-170426135638/85/Madrid-JAM-limitaciones-dificultades-20-320.jpg)
More Related Content
![[COSCUP 2021] A trip about how I contribute to LLVM](https://cdn.slidesharecdn.com/ss_thumbnails/20210731coscup2021atripabouthowicontributetollvm-210731072923-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to Madrid JAM limitaciones - dificultades
Madrid JAM limitaciones - dificultades
- 1.
- 2. QA @ StratioBig Data Inc. @witokondoria https://github.com/witokondoria/ id 2 Javier Delgado Garrido
- 3.
- 4. 1 BP Status beforepipelines
- 5. Chained! Hell of jobs 5 FETCH UNITTESTS INTEGRATION TESTS CODE QUALITY
- 6. Chained! Hell of jobs 6 FETCH UNITTESTS INTEGRATION TESTS CODE QUALITY
- 7. Chained! Hell of jobs 7 FETCH UNITTESTS INTEGRATION TESTS CODE QUALITY
- 8. Chained! Hell of jobs 8 FETCH UNITTESTS INTEGRATION TESTS CODE QUALITY
- 9. Chained! Hell of jobs 9 FETCH UNITTESTS INTEGRATION TESTS CODE QUALITY
- 10. Chained! Hell of jobs 10 THISSMELLS FISHY
- 11. node (‘docker’) { dir(“ws${env.BRANCH_NAME}”) { withEnv ([PATH+M=${tool ‘M3’]) { sh ‘mvn test-compile’ } archive ‘**/license/*.*’ } } 11 The Jenkinsfile sh timestamps node parallel checkout readFile marathon Pipelines as code
- 12. Pipelines aren’t foranybody Limits
- 13. • Script securityplugin protects us • Blocking some useful methods • Able to whitelist those calls 13 Sandboxed env Chains and locks
- 14. • Looking afterserialization (imposed) • Own implementation over groovy classes • Runs on the master agent 14 CPS (Continuation-passing style) Chains and locks
- 15. • Really nicevisualization • Limited under stages and parallel blocks calls 15 Blueocean, stages and parallel Chains and locks
- 16.
- 17. def S1 =‘FAI’ def S2 = “${-> S1}TH” echo “${S2}” ===> org.jenkinsci.plugins.scriptsecurity.sandbox .RejectedAccessException: unclassified new org.codehaus.groovy.runtime.GStringImpl java.lang.String java.lang.String[] 17 Double Lazy GString What if I need to lazily evaluate a variable? Oops
- 18. • Rewrite thetoString method for GStringImpl objects • Iterate over GString content: values and strings • Blend them appropriately • Shamefully badly coded (expects a Gstring starting with a non closure element) 18 Double Lazy GString - Solution Oops
- 19. • CPS forcesserialization at the master node • @NonCPS helps • Mind the usage of: 1) JsonSlurper (since Groovy 2) 2) Most of Jenkins API interesting classes (Job, Run, Cause) 3) Iterators, from Groovy sugar (.each, .grep, .findAll, .collect, … ) • CPS global libraries must implement Serializable 19 NonSerializableException Oops
- 20. def L1 =[] //constructed as List synchronized(L1) { L1.add (1) } echo L1.inspect() ===> java.lang.UnsupportedOperationException Collections.synchronizedList([]) seems to work (now) 20 .synchronizedList Parallel step is powerful But it is parallel and synchronization should be needed Oops
- 21. if (e instanceof java.io.NotSerializableException|| e instanceof groovy.lang.MissingPropertyException || e instanceof org.jenkinsci.plugins.scriptsecurity.sandbox .RejectedAccessException) { throw e } 21 Script approval Cool feature Works upon exception capture So, rethrow the captured exception sh timestamps node parallel checkout readFile Oops
- 22. • Several accounts •Nicer API usage (trust related) 22 API rate limits Oops
- 23. package com.stratio.qa.pipelines import org.junit.Test classsimpleTest extends GroovyTestCase { void echo(object) {println "${object}" } @Test void testGstringUsageWorkaround() { def str = '33' def input = "${-> str}" assert gstringUsageWorkaround(input) == '33' } } 23 (Unit) testing CPS global library mavenized gmaven-plus-plugin surefire-plugin jacoco-plugin Oops
- 24. • Cloudbees operationcenter handles them • Yet another docker plugin will do so (pending PR) • JNLP slaves allow the agents to work over dis/reconnections 24 Agent disconnections Oops
- 25. • Markup’ed internallyhandled • But without versioning , step reference for the current version • Welcome Groovidocs: http://stratiodocs.s3.amazonaws.com/pipelines/0.4.0- SNAPSHOT/index.html 25 Documentation Oops
- 26.
- 27. Treat well yourself 27 CPSlibraries Pipelines from a shared library Mind the not-such singleton objects Reusability over all the things
- 28. Treat well yourself 28 Configurationby default Let developers to declare just what they need Organization elements by default, being configurable URLs (artifact repository, docker registry) Pipeline timeouts Email/slack contact
- 29. The overhead forsetting up a durable task can be minimized Lower the steps, the less failure points 29 Merge your sh steps //Don’t node (‘master‘) { sh “echo 1” sh “echo 2” } //But node (‘master‘) { sh “echo 1 && echo 2” } Treat well yourself
- 30. Treat well yourself 30 Stageyour pipeline Better failure control Blueocean!
- 31. Treat well yourself 31 Mindthe parallel blocks Parallelize as much as you can splitTests aided Multithreaded builds Cloud-based agents
- 32. Treat well yourself 32 Jenkinsis a FOSS project Contribute!
- 33.
- 34. Flexible & reliable? Ourfinal stack 34
- 35.
- 36. Openstack (*) EC2 (*) VSphere(*) Marathon (*) Monitoring Datadog (*) Lockable resources Pipeline full stack (inc. milestone) Blue ocean 36 Jenkins <3 plugins Ansicolor Timestamper Github branch source Bitbucket branck source jUnit / xUnit Parallel test executor Slack notification Mailer Yet another docker (*) Structure
- 37. THANK YOU contact@stratio.com www.stratio.com UNITED STATES Tel:(+1) 408 5998830 EUROPE Tel: (+34) 91 828 64 73
Editor's Notes
- #3 I was contracted with a SQA role, but life rolls on and I’m nearer to a DevOps role
- #4 We will pass by 4 main points
- #5 Life at year one before pipelines: helluva amount of screens and manual work
- #6 Minimal freestyle job, performing a maven installation and local artifact archive
- #7 Minimal freestyle job, performing a maven installation and local artifact archive (2)
- #8 Minimal freestyle job, performing a maven installation and local artifact archive (3)
- #9 Several freestyle jobs were chained for emulating a pipeline
- #10 A minimal a rough parallelization was available
- #11 But it was a dull work
- #12 Enters the pipeline plugin (previously nown as workflow), with a really lean usage
- #14 The pipeline plugin lives in a complex and restrictive ecosystem
- #15 Full of tricks, limiting its scope
- #16 But with Blue Ocean, visualization is godly
- #17 As every other software, pipeline plugins have bugs
- #18 The double Lazy GString (TM) was a hard one to solve. In words of someone near me, we unlocked pipelines development around the world… Talk about overselling.
- #19 Easy peasy: rewrite the proper Groovy methods so the bug doesn’t block you! The reality was weeks for guessing a workaround
- #20 CPS and hellish serialization everywhere. Watch for no such objects and get rid of them
- #21 CPS plugin doesn’t handle groovy idioms
- #22 Approving non-whitelisted methods is a cool feature, but depends on an exception being thrown. Do so
- #23 The github API rate limit is pretty narrow under some circumstances. It has been heavily worked on by the Jenkins side
- #24 We could even test parts of our pipeline library. 34 awesome tests are being run with each change.
- #25 Transient agents are a pain. CPS serialization is supposed to help, but it still doesnt. WIP: https://github.com/KostyaSha/yet-another-docker-plugin/pull/130
- #26 Groovydocs own over internal docs, as the latter doesn’t handle versioning
- #28 Yes to CPS, from a global and shared library (DRY)
- #29 The simplest the Jenkinsfiles, the better. Thus, configuration by default from a trusted source are nice ideas
- #30 The more amount and complexity of actions, the more failure prone a pipeline is. Merge as much steps as able
- #31 Stages empower Blue Ocean, so squeeze it to the max
- #32 Also, parallelizing lowers total runtime so go for it
- #33 And become a contributor for such an awesome project: code, test, docs… there are always tasks for everybody
- #35 Our current stack
- #36 And how relations arise among elements
- #37 Our plugin installations. Asterisks mean plugins contributed to, some still WIP