Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Devoxx 2016 Using Jenkins, Gerrit and Spark for Continuous Delivery Analytics

838 views

Published on

Our journey and experience in dealing with the collection/analysis of Continuous Delivery log events using Gerrit Code Review, Jenkins with Apache Flume, ElasticSearch, Kibana and Spark

Published in: Technology
  • Be the first to comment

Devoxx 2016 Using Jenkins, Gerrit and Spark for Continuous Delivery Analytics

  1. 1. @gitenterprise#DevoxxPL Platinum Sponsors: Speed-up your Continuous Delivery with BigData Luca Milanesio GerritForge
  2. 2. @gitenterprise#DevoxxPL • Founded in the UK • HQ in London, Offices in the UK and Italy • Committed to OpenSource About GerritForge
  3. 3. @gitenterprise#DevoxxPL • Co-founder and Director of GerritForge • over 20 years in Agile Development and ALM • Contributor to many OpenSource projects • Mind and hands of @gitenterprise and @gerritreview About Luca Milanesio
  4. 4. @gitenterprise#DevoxxPL NOT a lecture on Continuous Delivery NOT a lecture on Code Review NOT a lecture on BigData What NOT to expect?
  5. 5. @gitenterprise#DevoxxPL SO WHAT? #!@$!
  6. 6. @gitenterprise#DevoxxPL SHOW a scenario of Continuous Delivery
  7. 7. @gitenterprise#DevoxxPL LEARN together about the problems
  8. 8. @gitenterprise#DevoxxPL EXPERIMENT my approach leveraging BigData
  9. 9. @gitenterprise#DevoxxPL DISCUSS and evolve the approach together
  10. 10. @gitenterprise#DevoxxPL Continuous Delivery Pipeline Push Build Unit Tests Push Build Unit Tests Automated Tests Push Build Unit Tests Automated Tests Beta Users Tests User's feedback Publish to all Users Failed Failed
  11. 11. @gitenterprise#DevoxxPL Code Review & CD Validation A1 refs/for/master master C2 C2* A1 C2 (local) master C2* push Verified: -1 (build failed) push Verified: +1 (build OK) Reviewed: +2 CI Build: OK C2* CI Build: OK amend
  12. 12. @gitenterprise#DevoxxPL Git/Gerrit (Code Review) Jenkins (Continuous Integration & Delivery) Docker (Release and Service Packaging) Mesos & Marathon (Delivery and Resource Manangement) Components
  13. 13. @gitenterprise#DevoxxPL DEMO See the pipeline in action
  14. 14. @gitenterprise#DevoxxPL We use micro-services • 100+ Components • 200+ Repositories • 300+ People and 100,000+ of review comments • Cross-components [implicit] dependencies • Lots of build chains • 10,000+ of builds Now feel the pain :-(
  15. 15. @gitenterprise#DevoxxPL "It was working until 5 mins ago … then someone changed something"  Rings any bell? Problem #1 : Ripple effect
  16. 16. @gitenterprise#DevoxxPL In order to deliver Feature ABC I NEED Ver. N of Service-A with Ver. M of Service-B and Ver. K of Service C Problem #2 : Dependencies
  17. 17. @gitenterprise#DevoxxPL "Where can I find an environment to run ALL my Micro-Services integration tests?" Problem #3 : Resources
  18. 18. @gitenterprise#DevoxxPL Release small changes is GOOD Release often every change is GOOD BUT How long it takes the "Big Picture" and End-to-End? Where is my bottleneck? Problem #4 : Cycle time
  19. 19. @gitenterprise#DevoxxPL Micro-Services are distributed Builds are distributed Deployments are distributed  Tackling THIS have you ever felt like drowning in your logs? Problem #5 : Lots of logs
  20. 20. @gitenterprise#DevoxxPL Problems #1 + #2 + #3 + #4 + #5 = lead to your daily Micro-Services mental meltdown Problem #6 : all together
  21. 21. @gitenterprise#DevoxxPL • Collect all review events • Collect all logs • Channel them to a central store • Crunch and Crunch continuously • Never delete • Process, inspect and learn BigData to the rescue
  22. 22. @gitenterprise#DevoxxPL DEMO Continuous Delivery Analytics
  23. 23. @gitenterprise#DevoxxPL •Problem #1 – Ripple effect? •Problem #2 – Dependencies? •Problem #3 – Resources? •Problem #4 – Cycle Time? •Problem #5 – Logs aggregation? •Problem #6 – Avoid mental meltdown? Did we solve the problems?
  24. 24. @gitenterprise#DevoxxPL • Collect more data (System, JVM, Review scores) • Real-time statistics • Events correlation • Feedback loop to Code-Review • Production data harvest • More? Future directions
  25. 25. @gitenterprise#DevoxxPL • My blog www.gitenterprise.me • Gerrit Code Review www.gerritcodereview.co m • Jenkins www.jenkins.io • RabbitMQ rabbitmq.com • Apache Flume flume.apache.org • Hadoop HDFS hadoop.apache.org • Apache Spark spark.apache.org • Kibana github.com/elastic/kibana Resources

×