Я поделюсь с вами опытом разработки конвейерных скриптов Jenkins для организации процессов непрерывной интеграции и развертывания микросервисов. Акцент будет сделан на применении средств Jenkins для разделяемых библиотек. Я продемонстрирую подходы к созданию модульных, тестируемых и повторно используемых компонентов для сборки и развертывания. Доклад будет полезен каждому, кто так или иначе связан с автоматизацией непрерывной интеграции и развертывания ПО, будь то разработчик или же DevOps

1. Jenkin’s shared libraries in action
by Ed Yablonsky

2. ● Simple examples
● Issues we meet
● Solutions we made
● Code
You will see

3. Monolith to Microservices
Continuous Integration
Continuous Deployment
We do

4. AWS, Kubernetes, Helm
Linkerd, Prometheus, Grafana
.Net Core, C#, Python, JavaScript, NodeJS
MySQL, Amazon Aurora, Redis
GitHub, Jenkins, Nexus
We use

5. ● Script publishes service’s image ~300 lines
● Script deploys service ~250 lines
● Each service main script ~100 lines
● Inlined shell complex scripts
● Many parameters ~30
● Change script for each service
Problems

6. ● Is an automated expression of your process for
getting software from version control to your
customers
● Declarative
● Scripted
Jenkins Pipeline

7. ● Is a text file that contains the definition of a
Jenkins Pipeline and is checked into source
control
● Self-documented, versioned
● Single definition of build steps
● Make pipeline extendable
Jenkinsfile

8. pipeline {
agent any
stages {
stage('run') {
steps {
sh 'env'
}
}
}
}
Simple declarative pipeline

9. podTemplate {
node {
stage('run') {
sh 'env'
}
}
}
Simple scripted pipeline uses k8s plug-in

10. Demo
simple pipeline

11. Create new pipeline 1 - 2 - 3

12. Create new pipeline 1 - 2 - 3

13. Create new pipeline 1 - 2 - 3

14. Run pipeline

15. Use Jenkinsfile

16. Demo
dotnet pipeline

17. podTemplate {
node {
stage('checkout') {
git url: 'https://github.com/eDyablo/demo-dotnet.git'
}
stage('test') {
sh 'dotnet test Demo.Test'
}
}
}
Nice Jenkinsfile

18. def podLabel = "worker-${UUID.randomUUID()}"
podTemplate(
label: podLabel,
containers: [
containerTemplate(
name: 'dotnet',
image: 'microsoft/dotnet',
ttyEnabled: true,
command: 'cat')
]) {
node(podLabel) {
stage('checkout') {
git url: 'https://github.com/eDyablo/demo-dotnet.git'
}
stage('test') {
container('dotnet') {
sh 'dotnet test Demo.Test'
}
}
}
}
Jenkinsfile

19. 1. Create new Git repository
2. Create file vars/dotnetTemplate.groovy
Do following

20. def call(String podLabel, code) {
podTemplate(
cloud: 'kubernetes-dev',
namespace: 'jenkins',
label: podLabel,
containers: [
containerTemplate(
name: 'dotnet',
image: 'microsoft/dotnet',
ttyEnabled: true,
command: 'cat')
]) {
code()
}
}
vars / dotnetTemplate.groovy

21. 1. Create new Git repository
2. Create file vars/dotnetTemplate.groovy
3. Add repository reference as pipeline library
a. Set it “Load implicitly”
4. Modify pipeline script
Do following

22. Refer shared library

23. def podLabel = "worker-${UUID.randomUUID()}"
dotnetTemplate(podLabel) {
node(podLabel) {
stage('checkout') {
git url: 'https://github.com/eDyablo/demo-dotnet.git'
}
stage('test') {
container('dotnet') {
sh 'dotnet test Demo.Test'
}
}
}
}
Jenkinsfile

24. (root)
+- vars
| +- dotnetTemplate.groovy
| +- dotnetTemplate.txt
|
+- src
|
+- resources
Shared library directory structure

25. • Global
• Folder-level
• Dynamically loaded
Shared libraries can be

26. @Library('demo-shared@branch') _
def podLabel = "worker-${UUID.randomUUID()}"
dotnetTemplate(podLabel) {
node(podLabel) {
stage('checkout') {
git url: 'https://github.com/eDyablo/demo-dotnet.git'
}
stage('test') {
container('dotnet') {
sh 'dotnet test Demo.Test'
}
}
}
}
Jenkinsfile

27. Demo
parameterized pipeline
complex shell script
library resources

28. properties([
parameters([
string(name: 'REPOSITORY', defaultValue: 'edyablo/demo-dotnet', descrip
string(name: 'TEST_PROJECT', defaultValue: 'Demo.Test', description: 'N
choice(name: 'CONFIGURATION', defaultValue: 'Debug', choices: ['Debug',
])
])
dotnetTemplate(podLabel) {
node(podLabel) {
stage('checkout') {
git url: "https://github.com/${params.REPOSITORY}.git"
}
stage('test') {
container('dotnet') {
sh "dotnet test ./${params.TEST_PROJECT} --configuration ${params.C
}
}
}
}
Parameterized pipeline

29. Parameterized pipeline

30. properties([
parameters([
string(name: 'REPOSITORY', defaultValue: 'edyablo/demo-dotnet', description: 'Name
choice(name: 'CONFIGURATION', defaultValue: 'Debug', choices: ['Debug', 'Release']
])
])
dotnetTemplate(podLabel) {
node(podLabel) {
stage('checkout') {
git url: "https://github.com/${params.REPOSITORY}.git"
}
stage('test') {
container('dotnet') {
sh """
projects=`find . -regex '.*.Test.csproj'`
for project in $projects
do
dotnet test $project --configuration ${params.CONFIGURATION}
done
"""
}
More complex shell script

31. projects=`find . -regex '.*.Test.csproj'`
for project in $projects
do
dotnet test $project --configuration $CONFIGURATION
done
resources / dotnet-test.sh

32. properties([
parameters([
string(name: 'REPOSITORY', defaultValue: 'edyablo/demo-dotnet', descri
choice(name: 'CONFIGURATION', defaultValue: 'Debug', choices: ['Debug'
])
])
dotnetTemplate(podLabel) {
node(podLabel) {
stage('checkout') {
git url: "https://github.com/${params.REPOSITORY}.git"
}
stage('test') {
container('dotnet') {
sh(libraryResource('dotnet-test.sh'))
}
}
}
}
More complex shell script

33. properties([
parameters([
string(name: 'REPOSITORY', defaultValue: 'edyablo/demo-dotnet', descri
choice(name: 'BUILD_CONFIGURATION', defaultValue: 'Debug', choices: ['
])
])
dotnetTemplate(podLabel) {
node(podLabel) {
stage('checkout') {
git url: "https://github.com/${params.REPOSITORY}.git"
}
stage('test') {
container('dotnet') {
withEnv(["CONFIGURATION=$params.BUILD_CONFIGURATION"]) {
sh(libraryResource('dotnet-test.sh'))
}
}
}
}
More complex shell script

34. ● Script’s overgrowth
● Complex shell scripts
● Escape character
● Code duplication
● Parameters
Issues

35. ● Script’s overgrowth Use vars
● Complex shell scripts Use resources
● Escape character Use resources
● Code duplication Use vars,
resources, classes
● Parameters Use classes
Solution Use
shared library

36. Components
pipeline
kubectl
docker
dotnet
nexus
mysql

37. Tools
kubectl docker dotnet nexus
tool

38. Tool
DotnetConfig
defineParameters
loadParameters
defineEnvVars
DotnetClient
setup
buid
test
build.sh
test.sh

39. Demo
real code

40. package com.lohika.dotnet
import com.lohika.build.SecretReference
class DotnetConfig {
def nugetConfigRef
def defineParameters(pipeline) {
[ pipeline.string(name: 'DOTNET_NUGET_CONFIG_REF',
defaultValue: "$nugetConfigRef",
description: 'Reference to secret contains configuration for nuget package man
}
def loadParameters(params) {
nugetConfigRef = SecretReference.fromText(
params.DOTNET_NUGET_CONFIG_REF)
}
def defineEnvVars(pipeline) {
[ pipeline.secretEnvVar(key: 'NUGET_CONFIG',
secretName: nugetConfigRef.name, secretKey: nugetConfigRef.key) ]
}
src / com / lohika / dotnet / DotnetConfig.groovy

41. package com.lohika.dotnet
import com.lohika.shell.ShellClient
class DotnetClient extends ShellClient {
DotnetConfig config
def setup(pipeline) {
pipeline.sh('echo $NUGET_CONFIG > nuget.config')
}
def build(Map kwargs) {
def include = kwargs.include ?: ''
def exclude = kwargs.exclude ?: ''
def configuration = kwargs.configuration ?: 'Debug'
pipeline.withEnv(["BASE_DIR=$kwargs.baseDir",
"INCLUDE_PATTERN=${encodeForShell(include)}",
"EXCLUDE_PATTERN=${encodeForShell(exclude)}",
"BUILD_CONFIGURATION=$configuration"]) {
execute(pipeline.libraryResource(
'com/lohika/dotnet/dotnet-client-build.sh'))
}
}
src / com / lohika / dotnet / DotnetClient.groovy

42. baseDir=`realpath $BASE_DIR`
projects=`find "$baseDir" -regex "$INCLUDE_PATTERN"
-and -not -regex "$EXCLUDE_PATTERN"`
for project in $projects
do
projectDir=`dirname $project`
dotnet restore "$projectDir" --configfile "$baseDir/nuget.config"
dotnet build "$project" --no-restore
${BUILD_CONFIGURATION:+ --configuration "$BUILD_CONFIGURATION"}
done
resources/com/lohika/dotnet/dotnet-client-build.sh

43. Use DotnetClient
dotnetClient.build(
baseDir: '.',
include: /.*.csproj/,
configuration: 'Release')
dotnetClient.test(
baseDir: '.',
includeProjects: /.*.UnitTest.csproj/,
testFilter: 'Name~Ignore')

44. Job
PipelineJob
defineParameters
loadParameters
defineEnvVars
run
Tool
Tool
ToolConfig
Tool
Tool
ToolClient
Tool
scripts
runPipelineJob

45. import com.lohika.job.*
runPipelineJob(new PublishDotnetServiceJob(this)) {
dotnetConfig.nugetConfigRef = 'secrets : nuget.config'
gitConfig.baseUrl = 'https://github.com/demo'
gitConfig.branch = 'develop'
gitConfig.credsId = 'github-ci'
pipConfig.configRef = 'secrets : pip.config'
k8sConfig.configRef = 'secrets : kube.config'
service = 'demo-service'
}
Jenkinsfile

46. autoDeployServices {
services = [
'first : svc-first',
'second : svc-second : second-service',
'third'
]
nexus.baseUrl = 'artifacts.demo.com'
nexus.credsId = 'demo.nexus.ci'
k8s.configRef = 'secrets : kube.config : dev'
git.baseUrl = 'https://github.com/demo
git.branch = 'develop'
notify {
success {
slack url: '#demo.slack'
}
failure {
slack url: '#demo.slack'
}
}
}
Jenkinsfile

47. +- src
| +- com
| +- lohika
| +- dotnet
| +- DotnetConfig.groovy
| +- DotnetClient.groovy
+- resources
| +- com
| +- lohika
| +- dotnet
| +- dotnet-client-build.sh
| +- dotnet-client-test.sh
Directory structure
+- src
+- com
+- lohika
+- docker
+- dotnet
+- git
+- k8s
+- mysql
+- nexus
+- job

48. ● Modularization
● Single responsibility
● Re-use
● Testability
Jenkins shared library

49. ● Setup unit testing framework
● Use Groovy annotations
Next improvements

50. Thank You!