Mykola Marzhan - Jenkins on aws spot instance

1. JENKINS ON SPOT
Mykola Marzhan
Version: 08.10.18

2. Mykola Marzhan
Release Engineer, Percona
AWS Certified Solutions
Architect - Professional
I have been working as Release
Engineer for 10 years.

3. NOW MASTER,
WORKERS LATER

4. JENKINS MASTER
ON LIGHTSAIL INSTANCE

5. LIGHTSAIL LIMITATIONS
➤ Cannot run in VPC, Subnet
➤ Cannot assign IAM Role,
Tags, Route 53
➤ Cannot manage with
CloudFormation
➤ Terraform cannot manage
firewall and additional
storage

6. ON-DEMAND VS
SPOT INSTANCES

9. JENKINS MASTER
ON SPOT INSTANCE

10. MURDERED DUE TO
PRICE, CAPACITY,
CONSTRAINTS }

11. FREQUENCY OF
INTERRUPTION
JENKINS MASTER

13. AMAZON
ELASTIC FILE SYSTEM
JENKINS MASTER

14. EFS PERFORMANCE
MiB/s Operations/s AVG latency (ms)
Sequential read 122,96 7869,10 0,126
Random read 21,13 1352,28 0,737
Sequential write 1,95 124,72 8,026
Sequential rewrite 1,99 127,33 7,869
Random write 1,47 94,14 10,616
Random rewrite 1,55 98,90 4,044

15. ENHANCED NETWORKING
FOR EC2 INSTANCES
JENKINS MASTER

16. EFS PERFORMANCE
MiB/s Operations/s AVG latency (ms)
Sequential read 122,96 +30% 7869,10 +30% 0,126
Random read 21,13 +30% 1352,28 +30% 0,737
Sequential write 1,95 +30% 124,72 +30% 8,026
Sequential rewrite 1,99 +30% 127,33 +30% 7,869
Random write 1,47 +30% 94,14 +30% 10,616
Random rewrite 1,55 +30% 98,90 +30% 4,044

17. EFS LIMITATIONS
➤ EFS not suitable for many
small files and one-thread
applications
➤ All artifacts to S3
➤ Tens of jobs/pipelines
➤ Few builds to keep
➤ Don’t “stash” files
(extremely slow)

18. EFS LIMITATIONS
➤ EFS not suitable for many
small files and one-thread
applications
➤ All artifacts to S3
➤ Tens of jobs/pipelines
➤ Few builds to keep
➤ Don’t “stash” files
(extremely slow)

19. EFS LIMITATIONS
➤ EFS not suitable for many
small files and one-thread
applications
➤ All artifacts to S3
➤ Tens of jobs/pipelines
➤ Few builds to keep
➤ Don’t “stash” files
(extremely slow)

20. EFS LIMITATIONS
➤ EFS not suitable for many
small files and one-thread
applications
➤ All artifacts to S3
➤ Tens of jobs/pipelines
➤ Few builds to keep
➤ Don’t “stash” files
(extremely slow)

21. EFS LIMITATIONS
➤ EFS not suitable for many
small files and one-thread
applications
➤ All artifacts to S3
➤ Tens of jobs/pipelines
➤ Few builds to keep
➤ Don’t “stash” files
(extremely slow)

22. AMAZON
ELASTIC BLOCK STORE
JENKINS MASTER

23. 2-MINUTE INSTANCE
TERMINATION NOTICE
JENKINS MASTER

24. EBS LIMITATIONS
➤ EBS limits to use one AZ

25. AWS SPOT FLEET
JENKINS MASTER

26. PIPELINES IN
MAX_SURVIVABILITY
JENKINS MASTER

27. NO EXECUTORS ON
MASTER NODE
JENKINS MASTER

28. NO WEB-HOOKS
JENKINS MASTER
use pollCSM trigger

29. ELASTIC IP
JENKINS MASTER

30. JENKINS WORKERS
ON SPOT INSTANCES
Artwork author: Ksenia Nenasheva

31. EC2 FLEET JENKINS PLUGIN
➤ Official AWS plugin
➤ Configuration on AWS side
➤ Spot Fleet can start instances in different AZs
➤ Starts two times more instances than needed (PR#29)
➤ On scale-down AWS terminates instances under load
(issues#36)
➤ New Spot Fleet on changes

32. EC2 FLEET JENKINS PLUGIN
➤ Official AWS plugin
➤ Configuration on AWS side
➤ Spot Fleet can start instances in different AZs
➤ Starts two times more instances than needed (PR#29)
➤ On scale-down AWS terminates instances under load
(issues#36)
➤ New Spot Fleet on changes

33. EC2 FLEET JENKINS PLUGIN
➤ Official AWS plugin
➤ Configuration on AWS side
➤ Spot Fleet can start instances in different AZs
➤ Starts two times more instances than needed (PR#29)
➤ On scale-down AWS terminates instances under load
(issues#36)
➤ New Spot Fleet on changes

34. EC2 FLEET JENKINS PLUGIN
➤ Official AWS plugin
➤ Configuration on AWS side
➤ Spot Fleet can start instances in different AZs
➤ Starts two times more instances than needed (PR#29)
➤ On scale-down AWS terminates instances under load
(issues#36)
➤ New Spot Fleet on changes

35. EC2 FLEET JENKINS PLUGIN
➤ Official AWS plugin
➤ Configuration on AWS side
➤ Spot Fleet can start instances in different AZs
➤ Starts two times more instances than needed (PR#29)
➤ On scale-down AWS terminates instances under load
(issues#36)
➤ New Spot Fleet on changes

36. EC2 FLEET JENKINS PLUGIN
➤ Official AWS plugin
➤ Configuration on AWS side
➤ Spot Fleet can start instances in different AZs
➤ Starts two times more instances than needed (PR#29)
➤ On scale-down AWS terminates instances under load
(issues#36)
➤ New Spot Fleet on changes

37. AMAZON EC2 PLUGIN
➤ Most popular plugin
➤ Configuration on Jenkins side
➤ Possible to use external ssh process as transport
➤ Has init script which executed before agent start
➤ Used to starts instances one by one (fixed in 1.40)
➤ Instances cannot be started in different AZs
➤ Instance cap is tied to AMI (PR#274)
➤ Doesn’t tag instances correctly since 2013
(JENKINS-19058, JENKINS-39702)

38. AMAZON EC2 PLUGIN
➤ Most popular plugin
➤ Configuration on Jenkins side
➤ Possible to use external ssh process as transport
➤ Has init script which executed before agent start
➤ Used to starts instances one by one (fixed in 1.40)
➤ Instances cannot be started in different AZs
➤ Instance cap is tied to AMI (PR#274)
➤ Doesn’t tag instances correctly since 2013
(JENKINS-19058, JENKINS-39702)

39. AMAZON EC2 PLUGIN
➤ Most popular plugin
➤ Configuration on Jenkins side
➤ Possible to use external ssh process as transport
➤ Has init script which executed before agent start
➤ Used to starts instances one by one (fixed in 1.40)
➤ Instances cannot be started in different AZs
➤ Instance cap is tied to AMI (PR#274)
➤ Doesn’t tag instances correctly since 2013
(JENKINS-19058, JENKINS-39702)

40. AMAZON EC2 PLUGIN
➤ Most popular plugin
➤ Configuration on Jenkins side
➤ Possible to use external ssh process as transport
➤ Has init script which executed before agent start
➤ Used to starts instances one by one (fixed in 1.40)
➤ Instances cannot be started in different AZs
➤ Instance cap is tied to AMI (PR#274)
➤ Doesn’t tag instances correctly since 2013
(JENKINS-19058, JENKINS-39702)

41. AMAZON EC2 PLUGIN
➤ Most popular plugin
➤ Configuration on Jenkins side
➤ Possible to use external ssh process as transport
➤ Has init script which executed before agent start
➤ Used to starts instances one by one (fixed in 1.40)
➤ Instances cannot be started in different AZs
➤ Instance cap is tied to AMI (PR#274)
➤ Doesn’t tag instances correctly since 2013
(JENKINS-19058, JENKINS-39702)

42. AMAZON EC2 PLUGIN
➤ Most popular plugin
➤ Configuration on Jenkins side
➤ Possible to use external ssh process as transport
➤ Has init script which executed before agent start
➤ Used to starts instances one by one (fixed in 1.40)
➤ Instances cannot be started in different AZs
➤ Instance cap is tied to AMI (PR#274)
➤ Doesn’t tag instances correctly since 2013
(JENKINS-19058, JENKINS-39702)

43. AMAZON EC2 PLUGIN
➤ Most popular plugin
➤ Configuration on Jenkins side
➤ Possible to use external ssh process as transport
➤ Has init script which executed before agent start
➤ Used to starts instances one by one (fixed in 1.40)
➤ Instances cannot be started in different AZs
➤ Instance cap is tied to AMI (PR#274)
➤ Doesn’t tag instances correctly since 2013
(JENKINS-19058, JENKINS-39702)

44. AMAZON EC2 PLUGIN
➤ Most popular plugin
➤ Configuration on Jenkins side
➤ Possible to use external ssh process as transport
➤ Has init script which executed before agent start
➤ Used to starts instances one by one (fixed in 1.40)
➤ Instances cannot be started in different AZs
➤ Instance cap is tied to AMI (PR#274)
➤ Doesn’t tag instances correctly since 2013
(JENKINS-19058, JENKINS-39702)

45. JENKINS WORKER
RETRY }

46. EXPENSES
Master
USD/month
Worker
USD/month
PMM 47.59 11.07
PSMDB 33.12 45.03
XtraBackup 33.60 30.21
Percona Server 65.98 399.57

47. CLOUDFORMATION CONFIG
https://git.io/fxOZY

48. TERRAFORM CONFIG
https://git.io/fxO4Y

49. Mykola Marzhan
delgod@delgod.com