Download as PDF, PPTX
Uploaded byJorge Hidalgo
Multilanguage Pipelines with Jenkins, Docker and Kubernetes (Commit Conf 2018)
The document discusses modern engineering practices using Jenkins, Docker, and Kubernetes to enhance software delivery and development efficiency. It emphasizes the importance of Agile and DevOps principles for automating pipelines, improving application deployment speed, and ensuring high-quality code through continuous testing and integration. Additionally, it provides an overview of the typical steps involved in a CI/CD pipeline, encouraging collaboration and innovation within cross-functional teams.
More Related Content
Similar to Multilanguage Pipelines with Jenkins, Docker and Kubernetes (Commit Conf 2018)
Multilanguage Pipelines with Jenkins, Docker and Kubernetes (Commit Conf 2018)
- 1. SPAIN ATCINFINITE POSSIBILITIES MULTILANGUAGE PIPELINES WITH JENKINS, DOCKER ANDKUBERNETES Jorge Hidalgo @_deors @deors Toni de la Prieta @tldpa @antonio-de-la-prieta-antoli COMMIT CONF NOVEMBER 2018
- 2. WHO WE ARE Tonide la Prieta @tldpa @antonio-de-la-prieta-antoli Director Accenture Technology Spain Spain Custom Software Engineering Lead Jorge Hidalgo @_deors @deors Senior Technology Architect Accenture Technology Spain Spain ATC Custom & Architecture Lead Honorable mentions: Vicente González, Julio Palma and the whole Architecture & DevOps Squad at Spain ATC
- 3. TL;DR All code andpipeline examples can be found online in these repositories: https://github.com/deors/deors-demos-petclinic https://github.com/deors/deors-demos-python-pipeline Don’t hesitate to ask, log issues or send your pull requests – HACKERS WELCOMED!
- 4. AT THE BEGINNING... ...ONLYPAIN AND BREAKING BUILDS
- 5. ARE THESE FAMILIARTO YOU? Code change paralysis Production environments hard to observe Lack of automation Inadequate code analysis Obsolete processes (waste) Brittle, tightly coupled applications Limited test environments Stop-the-world deployments
- 6. 50 DEPLOYMENTS PERDAY IS POSSIBLE Netflix proactively identify and resolve issues before they impact customers Amazon deploys new features to production every 11 seconds Uber deploys more than 500 times per day
- 7. WHAT ‘AWESOME’ LOOKSLIKE (THE VALUE PROPOSITION OF MODERN ENGINEERING) Provision environments dynamically Spawn a new environment in under 5 minutes Have “eagle eyes” to observe everything Precise, real-time metrics on KPIs of interest No manual effort in repetitive tasks All common tasks are automated Eliminate regression and security defects Achieving 0 regression defects is feasible Deliver at the pace of business From weeks to hours or minutes Build resilient, self-healing apps No downtime, no unexpected errors
- 8. MODERN ENGINEERING IS AKEY ENABLER FOR BUSINESS AGILITY COST OF INEFFICIENCY COST OF DELAY “In the new world, it is not the big fish which eats the small fish, it's the fast fish which eats the slow fish.” -- Klaus Schwab
- 9. AGILE DEVOPS CLOUD-NATIVE ARCHITECTURE VALUE PRIORITISED DELIVER–REFLECT–IMPROVE DECOUPLED &DISTRIBUTED RESILIENT BY DESIGN AUTOMATED PIPELINES CROSS-FUNCTIONAL TEAMS PRODUCT AND CUSTOMER CENTRIC BUILT-IN QUALITY DECENTRALISED EXECUTION ELASTIC SUPPLY OF RESOURCES M O D E R N E N G I N E E R I N G
- 10. AGILE & DEVOPS– BRIEF DEFINITIONS Agile & DevOps are sets of principles for iterative development, promoting lean processes, communication and collaboration, adaptation to changes, and continuous improvement, to speed up the TIME TO MARKET and the CONTINUOUS DELIVERY of business value Agile & DevOps both focus on nurturing SELF-ORGANISING, CROSS-FUNCTIONAL teams, with all the skills and capabilities needed to achieve the PRODUCT VISION IDEATION VALUE Adapted image from Wikimedia – https://commons.wikimedia.org/wiki/File:Devops-toolchain.svg Agile focuses on the interactions between the CUSTOMER (business) and DEVELOPMENT DevOps focuses on the interactions between DEVELOPMENT and OPERATIONS
- 11. AGILE & DEVOPSDRIVE A NEW WAY OF WORKING TOGETHER AS TEAMS
- 12. BY DOING… OWNERSHIP &COMMITMENT BREAKING DOWN SILOS COLLABORATING LEARNING TOGETHER LOOKING FOR SOLUTIONS SHARING RESPONSIBILITIES CRAFTMANSHIP
- 13. CREATIVE THINKING PROBLEM SOLVING PRODUCTVISION CUSTOMER EXPERIENCE DESIGN & BUILD CONTINUOUS INTEGRATION CONTINUOUS TESTING CONTINUOUS DEPLOYMENT CONTINUOUS MONITORING HUMAN + MACHINE
- 14.
- 15.
- 16.
- 17.
- 18.
- 19.
- 20.
- 21.
- 22. K8S – HAVEIT YOUR WAY
- 23.
- 24.
- 25.
- 26.
- 27. AND LET THE PLATFORM CAREOF ANYTHING ELSE
- 28. INCLUDING MANAGEMENT OF BUILDER NODES (NO NEEDTO CARE ABOUT SLAVES/SWARMS ANYMORE)
- 29. STANDARDIZATION TAKES FORM OF ANAGREED PIPELINE SCHEMA “Behavior-over-tool”
- 30. WHAT’S IN ACI PIPELINE ENVIRONMENT PREPARATION COMPILE UNIT TESTS MUTATION TESTS PACKAGE BUILD DOCKER IMAGE RUN DOCKER IMAGE INTEGRATION TESTS PERFORMANCE TESTS DEPENDENCY VULNERABILITY TESTS CODE INSPECTION PUSH DOCKER IMAGE “THE BRIEF HISTORY OF A COMMIT”
- 31. • Prepare environmentfor pipeline execution • Builder image with basic stuff: JDK, Maven, npm, Node.js, Python… • Install all needed dependencies • Python: pip install • Node.js: npm install • Java: Maven, either explicit or along the pipeline ENVIRONMENT PREPARATION ENVIRONMENT PREPARATION COMPILE UNIT TESTS MUTATION TESTS PACKAGE BUILD DOCKER IMAGE RUN DOCKER IMAGE INTEGRATION TESTS PERFORMANCE TESTS DEPENDENCY VULNERABILITY TESTS CODE INSPECTION PUSH DOCKER IMAGE
- 32. • Transform sourcecode to binaries • In some cases compilation is replaced by other activities like uglyfication, minification, transpilation… • For interpreted languages, when viable, check for syntax errors COMPILE ENVIRONMENT PREPARATION COMPILE UNIT TESTS MUTATION TESTS PACKAGE BUILD DOCKER IMAGE RUN DOCKER IMAGE INTEGRATION TESTS PERFORMANCE TESTS DEPENDENCY VULNERABILITY TESTS CODE INSPECTION PUSH DOCKER IMAGE
- 33. • Run projectunit tests • And never forget: gather code coverage metrics • Coverage will highlight which code is not being exercized • “EXERCIZED” does not imply “TESTED” UNIT TESTS ENVIRONMENT PREPARATION COMPILE UNIT TESTS MUTATION TESTS PACKAGE BUILD DOCKER IMAGE RUN DOCKER IMAGE INTEGRATION TESTS PERFORMANCE TESTS DEPENDENCY VULNERABILITY TESTS CODE INSPECTION PUSH DOCKER IMAGE
- 34. • In anutshell: validates whether your tests are actually testing anything • Mutation tests will uncover: • Untested use cases • Test cases without assertions • Also, test cases with insufficient or wrong assertions MUTATION TESTS ENVIRONMENT PREPARATION COMPILE UNIT TESTS MUTATION TESTS PACKAGE BUILD DOCKER IMAGE RUN DOCKER IMAGE INTEGRATION TESTS PERFORMANCE TESTS DEPENDENCY VULNERABILITY TESTS CODE INSPECTION PUSH DOCKER IMAGE
- 35. • Package allapplication resources: • Binaries • Static resources • Application configuration which does not depend on the environment, e.g. literals, error codes… • Environment specific configuration must be externalized (package once run everywhere) PACKAGE ENVIRONMENT PREPARATION COMPILE UNIT TESTS MUTATION TESTS PACKAGE BUILD DOCKER IMAGE RUN DOCKER IMAGE INTEGRATION TESTS PERFORMANCE TESTS DEPENDENCY VULNERABILITY TESTS CODE INSPECTION PUSH DOCKER IMAGE
- 36. • Put togetherall pieces which constitute our application • Base image • Build artifacts • Dependencies • Base images have their own pipeline • Typically maintained by someone else (vendor, community, other team…) BUILD DOCKER IMAGE ENVIRONMENT PREPARATION COMPILE UNIT TESTS MUTATION TESTS PACKAGE BUILD DOCKER IMAGE RUN DOCKER IMAGE INTEGRATION TESTS PERFORMANCE TESTS DEPENDENCY VULNERABILITY TESTS CODE INSPECTION PUSH DOCKER IMAGE
- 37. • The purposeof this step is to ensure that our application runs and to make it available for tests • If application depends on other applications (database, message queue, other services…) they will be executed all together (docker compose, application.yml…) • The test environment is not lasting — will be available only to tests during the life of the pipelines • Should allow for concurrent execution (fast feedback on a continuous integration build-per-commit approach) RUN DOCKER IMAGE ENVIRONMENT PREPARATION COMPILE UNIT TESTS MUTATION TESTS PACKAGE BUILD DOCKER IMAGE RUN DOCKER IMAGE INTEGRATION TESTS PERFORMANCE TESTS DEPENDENCY VULNERABILITY TESTS CODE INSPECTION PUSH DOCKER IMAGE
- 38. • Run testsrequiring the application running and its dependencies • Integration tests may include: UI tests, API tests… • Should be user-oriented whenever possible • Test data preparation is performed at this step too • For web applications use vulnerability scan agent/proxy • And remember: gather code coverage metrics INTEGRATION TESTS ENVIRONMENT PREPARATION COMPILE UNIT TESTS MUTATION TESTS PACKAGE BUILD DOCKER IMAGE RUN DOCKER IMAGE INTEGRATION TESTS PERFORMANCE TESTS DEPENDENCY VULNERABILITY TESTS CODE INSPECTION PUSH DOCKER IMAGE
- 39. • Exercise theapplication under load • Response times are important but will also uncover errors due to concurrency usage • Quality gate should be asserting both metrics • Should be user-oriented too — reuse integration tests whenever possible • Use of dynamic performance monitor agents is also advised PERFORMANCE TESTS ENVIRONMENT PREPARATION COMPILE UNIT TESTS MUTATION TESTS PACKAGE BUILD DOCKER IMAGE RUN DOCKER IMAGE INTEGRATION TESTS PERFORMANCE TESTS DEPENDENCY VULNERABILITY TESTS CODE INSPECTION PUSH DOCKER IMAGE
- 40. • Assessing applicationsecurity is not negotiable • But usually we forget that most vulnerabilities, if not all, come from external dependencies • Some tools for this step include: • OWASP Dependency Check • Snyk • Black Duck • Nexus Lifecycle • PyUp Safety DEPENDENCY VULNERABILITY TESTS ENVIRONMENT PREPARATION COMPILE UNIT TESTS MUTATION TESTS PACKAGE BUILD DOCKER IMAGE RUN DOCKER IMAGE INTEGRATION TESTS PERFORMANCE TESTS DEPENDENCY VULNERABILITY TESTS CODE INSPECTION PUSH DOCKER IMAGE
- 41. • Static codeanalysis: • Adherence to coding standards • Possible bugs • Code smells and best practices • Security vulnerabilities • Documentation and complexity metrics • Code duplication • This step usually happens at the same time that the overall quality gate (SonarQube) • Ideally this should happen earlier in the pipeline — however we assume that this is happening before code is pushed and/or merged CODE INSPECTION ENVIRONMENT PREPARATION COMPILE UNIT TESTS MUTATION TESTS PACKAGE BUILD DOCKER IMAGE RUN DOCKER IMAGE INTEGRATION TESTS PERFORMANCE TESTS DEPENDENCY VULNERABILITY TESTS CODE INSPECTION PUSH DOCKER IMAGE
- 42. • After thefinal quality gate, if all pass, the image is pushed to a shared registry • Once pushed it is available to be used by: • Other applications that depends on this image • To be promoted to a stage environment • All the way up to production PUSH DOCKER IMAGE ENVIRONMENT PREPARATION COMPILE UNIT TESTS MUTATION TESTS PACKAGE BUILD DOCKER IMAGE RUN DOCKER IMAGE INTEGRATION TESTS PERFORMANCE TESTS DEPENDENCY VULNERABILITY TESTS CODE INSPECTION PUSH DOCKER IMAGE
- 43.