Scaling software builds with Jenkins

3,224 views

Published on

This talk revolves around the scalability strategies for Jenkins, the most used Continuous Integration server.

In particular, the talk focuses on how to scale Jenkins on the cloud (Amazon EC2) and create a build matrix so that the code can be built on different OSes.

Furthermore, we will also look at the Cloudbees offering.

Published in: Technology, Business

Scaling software builds with Jenkins

  1. 1. Scaling Software Builds with Jenkins 01
  2. 2. About me • • • • • Name: Andrey Adamovich Bio: Developer, coach, speaker, author Company: Aestas/IT E-mail: andrey@aestasit.com Linkedin: http://www.linkedin.com/in/andreyadamovich 02
  3. 3. Software development process 03
  4. 4. Why scale? 04
  5. 5. Feedback loop 05
  6. 6. Reason 1 06
  7. 7. Monolithic build 07
  8. 8. Monolithic build • • • Time-consuming Resource-consuming Frustration-accumulating 08
  9. 9. Before you can scale... 09
  10. 10. SPLIT!!! 10
  11. 11. Reason 2 11
  12. 12. 12 Too many things to build
  13. 13. Too many things to build • • • • Too many components Too many branches Too many unit/integration tests Too many validation rules 13
  14. 14. Reason 3 14
  15. 15. Too many things to test 15
  16. 16. Too many test variations • • • • Different operating systems Different target platforms Different markets Different customers 16
  17. 17. Reason 4 17
  18. 18. Lack of resources 18
  19. 19. Reason 5 19
  20. 20. Lack of time! 20
  21. 21. Solutions? 21
  22. 22. Add more hardware! 22
  23. 23. or... 23
  24. 24. Go to the Cloud! 24
  25. 25. Who is Jenkins? 25
  26. 26. Architecture 26
  27. 27. UI 27
  28. 28. Master/Slave 28
  29. 29. Labelling 29
  30. 30. Slaves vs. Executors 30
  31. 31. Executor contention 31
  32. 32. Other types of contention • • CPU contention I/O contention 32
  33. 33. Cloud slaves 33
  34. 34. Executor scaling 34
  35. 35. Amazon Web Services 35
  36. 36. Elastic Compute Cloud • • • instances storage images 36
  37. 37. EC2 Console 37
  38. 38. ALERT!!! 38
  39. 39. It's not free! 39
  40. 40. But... 40
  41. 41. There is a Free Tier! 41
  42. 42. Cost calculations Name Compute Units Memory Hourly Cost t1.micro 2 0.60 GB $0.02 m1.small 1 1.70 GB $0.06 c1.medium 5 1.70 GB $0.14 m1.medium 2 3.75 GB $0.12 m1.large 4 7.50 GB $0.24 c1.xlarge 20 7.00 GB $0.58 m3.xlarge 42 13 15.00 GB $0.50
  43. 43. Labor cost 43
  44. 44. Public access 44
  45. 45. Public Clouds • • • • • • VMWare vCloud Ubuntu Enterprise Cloud Microsoft Azur Google Cloud Digital Ocean Cloudbees Cloud 45
  46. 46. Private Clouds • • • OpenStack VMWare vSphere Amazon VPC 46
  47. 47. Jenkins plugins 47
  48. 48. “ Jenkins EC2 plugin Allow Jenkins to start slaves on EC2 or Ubuntu Enterprise Cloud (Eucalyptus) on demand, and kill them as they get unused. With this plugin, if Jenkins notices that your build cluster is overloaded, it'll start instances using the EC2 API and automatically connect them as Jenkins slaves. 48
  49. 49. Preparation 49
  50. 50. 1. Configure access/secret key 50
  51. 51. 2. Configure key pair 51
  52. 52. 3. Configure security group 52
  53. 53. 4. Select machine image • • • Amazon base images AWS market place Create your own 53
  54. 54. 5.1. Configure EC2 plugin 54
  55. 55. 5.2. Configure EC2 plugin 55
  56. 56. 5.3. Configure EC2 plugin 56
  57. 57. Demo 57
  58. 58. Cloudbees • • • • Company behind Jenkins Kohsuke Kawaguchi (author of Hudson/Jenkins) works there Jenkins Enterprise Edition Jenkins Cloud = DEV@Cloud 58
  59. 59. DEV@Cloud • • • API & SDK Pricing based on size (# masters/executors/slaves) Slave OS variety • • • • iOS coming soon Windows Phones/Tablets coming soon Manual or auto-scaling Minute-based pricing 59
  60. 60. Free-tier • • • 300 free build minutes a month 1 build executor FOSS programme for open-source projects 60
  61. 61. Services Partner 61
  62. 62. Questions? 62
  63. 63. GitHub projects Jenkins setup: https://github.com/aestasit/talks2013-devconfu-scaling-builds-setup Slides: https://github.com/aestasit/talks2013-devconfu-scaling-builds-slides 63
  64. 64. Scaling Software Builds with Jenkins • • • • Author: Andrey Adamovich Company: Aestas/IT E-mail: andrey@aestasit.com Linkedin: http://www.linkedin.com/in/andreyadamovich 64
  65. 65. Feedback 65

×