By Denis Chernilevskiy, Yandex Attendees will learn about the concept of using several nodes as a single Jenkins slave. At Yandex, we’re deploying software environments consisting of many nodes from the cloud and we can leverage existing Jenkins slave management mechanisms - such as scheduling, automated provisioning and load-balancing - via this single-slave abstraction. This greatly decreases the time needed to support lots of separate slave nodes. At Yandex, we have created a plugin for these purposes and it will be presented in this session.

1. Multi-node environment as
a Jenkins slave
aka “compound-slave”
Denis Chernilevskiy
Head of Media Advertising Department

2. The image cannot be displayed. Your computer may not have
enough memory to open the image, or the image may have
been corrupted. Restart your computer, and then open the file
again. If the red x still appears, you may have to delete the
image and then insert it again.
Common use-case
the “prehistory”
Multi-node environment as a Jenkins slave

3. Common use-case
4
He’s famous The “slave” The “job”

4. │ Test this please!

5. The quest
6
Nodes Cloud Deploy Get resultsTest

6. The solution
▌ Cloud – Amazon/OpenStack/Docker/…
▌ Deploy – JClouds/DockerCloud + Chef/Salt/Ansible/…
▌ Test – PyTest/TestNG/…
▌ Results – Allure Framework/xUnit/…
7

7. How it works
▌ The common job concept
› Take 1 slave
› Deploy the system under test to the slave
› Deploy and run tests on the same slave
› Clean the slave
› Return the slave to the pool
8

8. │ Perfectly fits with 1 slave

9. │ But what if…

10. 11

11. The image cannot be displayed. Your computer may not have
enough memory to open the image, or the image may have
been corrupted. Restart your computer, and then open the file
again. If the red x still appears, you may have to delete the
image and then insert it again.
The “real life” of QA/DevOps
the problem
Multi-node environment as a Jenkins slave

12. │ Yandex MediaAd platform

13. │ Test this please!

14. Process requirements
▌ Auto build, deploy, test
▌ Continuous Integration
▌ “Green trunk”
15

15. System components
▌ Front-end UI and API
▌ Ad.Management
▌ Ad.Selection
▌ Ad.Storage
▌ SocDem prediction & machine learning
16

16. │ None fits into 1 slave…

17. The obvious “not a solution” J
▌ Use Jenkins
▌ Get a slave for the job
▌ Get some more cloud nodes (somehow)
▌ Use the slave as a launchpad (run deploy, tests remotely)
18

18. │ Not for us J

19. Why not?
▌ Laziness. The less new code, the better!
▌ Separate nodes management
▌ JClouds not reused
▌ Lack of visibility
20

20. │ Several slaves per job?

21. │ Need to refactor core…

22. │ Several nodes as a slave!

23. The image cannot be displayed. Your computer may not have
enough memory to open the image, or the image may have
been corrupted. Restart your computer, and then open the file
again. If the red x still appears, you may have to delete the
image and then insert it again.
“Compound-slaves” plugin
the solution
Multi-node environment as a Jenkins slave

24. Features
▌ Group several nodes as a compound-slave
› Existing nodes grouping
› Get & group from the cloud
25

25. Features
▌ Cloud provisioning
› Plugins with a “Cloud” interface
› Using labels from the cloud plugin
› Number of each label to provision
26

26. Features
▌ Roles and labelling
› Distinct nodes
› Default labels
27

27. The image cannot be displayed. Your computer may not have
enough memory to open the image, or the image may have
been corrupted. Restart your computer, and then open the file
again. If the red x still appears, you may have to delete the
image and then insert it again.
Here comes the DEMO! J

28. Features
▌ General
› Built-in slave management
› Built-in job-slave provisioning
▌ Visibility
› Built-in dashboards
› Built-in job steps configuration
29

29. │ Profit! J

30. Links
▌ wiki.jenkins-ci.org/display/JENKINS/Compound+Slaves
▌ github.com/jenkinsci/compound-slaves-plugin
31

31. Contacts
dchr@yandex-team.ru
+79250351311 @dchernilevskiy
denis.chernilevskiy
Denis Chernilevskiy
Head of Yandex MediaAd Department