There is hidden value in the data produced by your Continous Delivery Pipeline that could help you achieve more efficient processing. The main objective of this workshop is to show you how to extract that value and benefit from it. We will introduce you to how to configure and improve your Continuous Delivery Pipeline using data. After an overview of a Continuous Delivery Pipeline setup using Gerrit Code Review, Jenkins, and Docker, we will go through the steps on how to extract and analyze data across all the pipeline stages of the delivery chain.

1. Data-Driven Pipeline
with Gerrit Code Review and Jenkins
Luca Milanesio - GerritForge
luca@gerritforge.com

2. About GerritForge
• Born 8 years ago in the UK
• HQ in London, Offices in the USA, Italy, Germany
• Committed to OpenSource

3. Agenda
• 9:00 - Jenkins Pipeline, Gerrit and Docker 101
• 9:30 - Logs and Events flowing through the pipeline
• 9:45 - Event-sourced data extraction
• 10:30 <break>
• 10:30 - Data crunching and correlation
• 11:15 - Rich data-analysis dashboards with Kibana
• 11:45 - Data-analysis on the Cloud
• 12:15 – Q&A - AOB

4. Why?
AIM OF DEVOPS?

5. Why?
WHAT IS CYCLE TIME?

6. Why?
WHERE IS THE BOTTLENECK?

7. Why?
… OR *WHO* IS THE BOTTLENECK?

8. Anatomy of a Continuous Delivery Pipeline
WHERE DO WE
START?

9. Anatomy of a Continuous Delivery Pipeline
REALLY?

10. Anatomy of a Continuous Delivery Pipeline
<< LET'S REWIND

11. Anatomy of a Continuous Delivery Pipeline
DEVS IN THE LOOP

12. Anatomy of a Continuous Delivery Pipeline
TIME TO BUILD

13. Anatomy of a Continuous Delivery Pipeline
CYCLE TIME

14. Git and Gerrit 101

15. Add Gerrit to /etc/hosts
127.0.0.1 review-pipeline

16. Git and Gerrit Code Review
git clone https://github.com/gerritforge/jw2017-pipeline

17. Run Gerrit Code Review
git checkout v1

18. Run Gerrit Code Review
docker-compose up -d

19. Run Gerrit Code Review
http://review-pipeline

20. Git and Gerrit Code Review

21. Git and Gerrit Code Review
git commit -am "My first change in Gerrit"

22. Git and Gerrit Code Review
git push origin HEAD:refs/for/master

23. Git and Gerrit Code Review

24. Jenkins Pipeline 101

25. Run Gerrit Code Review
git checkout v2

26. Add Gerrit to /etc/hosts
127.0.0.1 review-pipeline

27. Run Jenkins
docker-compose up -d jenkins

28. Jenkins Pipeline

29. Docker and Jenkins Pipeline
/var/run/docker.sock
/var/run/docker.sock

30. Collecting logs through the pipeline
SHOW ME YOUR LOGS !

31. Collecting logs through the pipeline
gerrit.event.queue jenkins.jobs.queue

32. Code Review Events
GERRIT STREAM EVENTS
• patchset-created
• ref-updated
• comment-added

33. Extracting Jenkins builds logs
Jenkins Logstash Wrapper

34. Pipeline events and logs in RabbitMQ

35. Collecting events together: ELK stack

36. Run Logstash Collectors
git checkout v3

37. Run Logstash Collectors
git up -d logstash-gerrit
git up -d logstash-jenkins

38. Gerrit Stream Events into RabbitMQ
https://gerrit.googlesource.com/plugins/rabbitmq

39. Jenkins logstash collector
https://github.com/jenkinsci/logstash-plugin

40. Look at Gerrit + Jenkins logs in logstash
docker-compose exec logstash-jenkins find /var/output

41. Data Crunching and Lambda Architecture
Batch Layer
Speed Layer
ETL Jobs
Streaming

42. Run Spark
git checkout v4

43. Run Logstash Collectors
git up -d spark

44. Apache Spark Architecture

45. Apache Spark 101
docker-compose exec spark spark-shell

46. Hello Spark
val sc = spark.sparkContext
val rdd = sc.parallelize(Seq("hello spark"))
rdd.count()
val rddWords = rdd.flatMap(_.split(" "))
rddWords.count()
rddWords.collect()

47. Lambda Architecture: aggregation
Jenkins
raw JSON
Gerrit
raw JSON
Job Log
CaseClass
Gerrit
Events
CaseClass
Flow
Event
Filter
Interesting
Events
Aggregated
Statistics

48. Correlation Lab: commit, review, build, run
docker-compose exec spark bash

49. Raw JSON Readers into Case Classes
git clone https://github.com/GerritForge/jw2017-spark.git

50. Raw JSON Readers into Case Classes
git checkout v1

51. Raw JSON Readers into Case Classes
sbt test

52. Merge into generic Flow Events
git checkout v2

53. Merge into generic Flow Events
sbt test

54. Filter transform and compute data
git checkout v3

55. Filter transform and compute data
sbt test

56. Put it all together: Spark Job
git checkout v4

57. Put it all together: Spark Job
sbt test

58. Running with Spark
spark-submit

59. The last mile: ElasticSearch + Kibana
Batch Push
Streaming
Query / Drill Down
Aggregation

60. Data analysis in the Cloud
On-Premises
Private Clouds

61. Data analysis in the Cloud
Hosted
Private Clouds

62. Security implications
Personal Information (Names, e-mails)
Confidentiality (Job logs, Review messages)
Project secrecy (Project and Repository Names)

63. How GerritForge can help
Gerrit Code Review Enterprise Support
Pipeline Integration and Optimisation
DevOps Analytics "As-a-Service"

64. AOB / Q&A
Come to visit us at
Booth 204