Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Collibra wrojug-ontrack-20100424


Published on

Hundreds of projects, thousands of jobs, each generating data about the status, commit, change logs and quality reports of your projects. How can you restore order in this amount of data and use it to gather useful information about your projects?
> In this talk, I will explain how I collect the data over thousands of jobs, commit after commit, build after build, how he stored it and used it in turn to extract useful information and to inject this data as feedback into our pipelines.

Published in: Software
  • Be the first to comment

Collibra wrojug-ontrack-20100424

  1. 1. ©2019 Collibra Inc Keeping track of your CI/CD mess @DamienCoraboeuf (Damien Coraboeuf / CICD lead at Collibra) Wrocław Java User Group - April, 25th, 2019
  2. 2. A few words about myself…
  3. 3. WHO AM I? ▸ Damien Coraboeuf (@DamienCoraboeuf) ▸ Family, hiking, good food, good beer, role playing, … ▸ Born to Java in 1996, developer, architect ▸ Working in Continuous Delivery since 2008 ▸ Automating pipelines at scale
  4. 4. What are we talking about?
  5. 5. CI/CD IN A NUTSHELL Ideas, features, fixes Happy customer Fast feedback Security Quality at all times Automation Traceability / Accountability Ready to release at all times I work (mostly) here
  6. 6. CI/CD ECOSYSTEM It starts with some code… Linked with some issues… We build it…. We package and publish… We run some tests… We perform quality scans…. We need some infra…
 Depen d en cies Triggering other pipelinesRuns tests
 Collects results Scans co d e
 M easures perform ances Tags / com m its
 Tickets Deploying
  9. 9. ALL DAY LONG…
  10. 10. PUTTING SOME ORDER INTO THIS DATA ▸ Projects ▸ Branches ▸ Builds ▸ Validations ▸ Promotions Metadata!!
  11. 11. SEEING IS BELIEVING ▸ One tool to rule them all ▸ Builds Promotions Validations Validation status System of records for anything related to CI/CD
  12. 12. SEEING IS BELIEVING ▸ Filtering Promoted … despite anomalies in deployments ▸ Status of your pipeline in one glance
  15. 15. SEEING IS BELIEVING ▸ Overview of all your branches
  16. 16. SEEING IS BELIEVING ▸ Change logs between builds, between versions ▸ Permalink for automated release notes ▸ Supports GitHub, JIRA, Subversion, BitBucket…
  17. 17. SEEING IS BELIEVING ▸ Using cross referenced information for searches: ▸ Where is my ticket? ▸ Where is my commit? Example: where is my #600 ticket? Here you go!
  18. 18. SEARCH ON BUILDS Change log
  19. 19. MANY MORE FEATURES… Auto promotions… Filtered & custom views… … and many more… + + + =
  20. 20. Integration is everything
  21. 21. SINGLE POINT OF KNOWLEDGE Ontrack Artifactory GitLab InfluxDB Jenkins Vault JIRA BitBucket Subversion GitHub Git
  22. 22. Ontrack INTEGRATION WITH THE REST OF THE ECOSYSTEM Jenkins plug-inGUI REST API GraphQL InfluxDB metrics Groovy DSL
  23. 23. JENKINS PLUG-IN Post build actions and steps
  24. 24. JENKINS PLUG-IN Integration with the Ontrack DSL
  25. 25. ONTRACK DSL ▸ Groovy based DSL Automation of Change log
  27. 27. FEEDING INFORMATION INTO ONTRACK ‣ Ontrack has a rich API ‣ … but better use the Ontrack Jenkins plug-in ‣ 100% integration with the Jenkins pipeline DSL ‣ … but @ Collibra, we provide integrated wrappers Typical workflow:
  28. 28. GETTING INFORMATION FROM ONTRACK ‣ Ontrack DSL can be used ‣ … wrappers provided @ Collibra
  29. 29. AUTOMATED FEEDBACK ▸ How to get the best out of your information
  30. 30. AUTOMATED FEEDBACK ▸ The collected information can be used in the ecosystem ▸ Based on Ontrack DSL
 or GraphQL ▸ Example: promotions used
 as inputs for on-demand deployments jobs
  31. 31. META INFORMATION AT ALL LEVELS ▸ Attaching built-in or custom properties to all levels - projects, branches, builds, etc. ▸ Used for reporting or to drive processes ▸ Can be entered manually or automatically
  32. 32. And this is NEVER enough…
  33. 33. COLLECT YOUR OWN CUSTOM INFORMATION Ontrack ▸ Extension system ▸ New connectors ▸ New services, screens and interfaces
  34. 34. CUSTOMISING THE SOLUTION Ontrack Core Extension Ontrack Java API Extension
  35. 35. CUSTOMISING THE SOLUTION ▸ TV wallboards for development teams ▸ Version auto upgrade with PR on promotion ▸ Dependency graph ▸ … ▸ Imagination is the limit
  36. 36. CUSTOMISING THE SOLUTION ▸ TV wallboards for development teams GraphQL / REST
  37. 37. PR CREATION ON PROMOTION FOR AUTO VERSION UPGRADES A B depends on Manual update of version of A in B Little automation Loose coupling Little control of quality
  38. 38. PR CREATION ON PROMOTION FOR AUTO VERSION UPGRADES A B depends on Automated PR for update of version of A in B Control of quality Strong coupling Automation
  39. 39. PR CREATION ON PROMOTION FOR AUTO VERSION UPGRADES A B depends on Promotion B A dependency
 configuration A Subscription Queue —> Listener Creates a PR
 for update of A
 in B PR build for
 control Automated merge
 after controls OK Listens to Control of quality Loose coupling Automation
  40. 40. DEPENDENCY GRAPHS Ontrack has built-in dependencies… … which can be used to build a graph (in Neo4j) … and execute complex queries … to detect dependencies anomalies
  41. 41. Getting Ontrack on track…
  42. 42. ONTRACK MANAGEMENT ▸ Jobs ▸ Metrics ▸ Deployment ▸ Security model
  43. 43. SYSTEM JOBS ▸ ~ 1000 jobs running in the background ▸ Critical system ▸ Collecting information all day long ▸ System administration interface
  44. 44. METRICS ▸ Ontrack can export lot of information to InfluxDB ▸ Additional metrics can be created through extensions
  45. 45. SECURITY ▸ Authentication ▸ Built-in users ▸ LDAP integration ▸ Extensible ▸ Authorisations ▸ Built-in ACL (roles & functions) ▸ Extensible ▸ Strong isolation between projects ▸ … but cross-projects roles
  46. 46. Getting started with Ontrack…
  47. 47. ONTRACK ▸ ▸ Use it, fill tickets, extend, contribute
  48. 48. THANK YOU! Thank you: Contact / Info: ▸ @DamienCoraboeuf ▸ Marcin Stachniuk