More Related Content
What's hot
What's hot (20)
Similar to Continuous integration is not a solved problem
Similar to Continuous integration is not a solved problem (20)
Recently uploaded
Recently uploaded (20)
Continuous integration is not a solved problem
- 1. https://cyclid.io @cyclidci github.com/Cyclid/ Continuous Integration is not a solved problem Config Management Camp, Gent 2017
- 2. https://cyclid.io @cyclidci github.com/Cyclid/ Kristian van der Vliet ● DevOpsing hard for the previous 7+ years Puppet 0.24 Chef 0.9 ● Gainfully employed as an SRE for a CDN company ● You may remember me from such projects as Syllable ● Don’t be fooled by the name, mijn Nederlands is niet goed!
- 3. https://cyclid.io @cyclidci github.com/Cyclid/ Why isn’t CI a solved problem? ● I’ve seen many CI pipelines for things like Chef; I’ve never met one I liked ● Continuous Integration tools make assumptions ○ You’re “building” software ○ It will result in an “artifact” ○ Your software is one of a known set of languages ○ The build process is well defined ● None of these are true for “non-traditional” software ○ Configuration Management/Infrastructure as Code ○ Network configuration ○ Database schemas
- 5. https://cyclid.io @cyclidci github.com/Cyclid/ “How hard can it be?”
- 6. https://cyclid.io @cyclidci github.com/Cyclid/ My ideal Continuous Integration ● Unopinionated Not tied to any workflow or toolset. ● Agnostic Tools change constantly. ● Flexible ...so do functional requirements. ● Operable I’m an Operations person! ● Open Source I’m also an Open Source person.
- 8. https://cyclid.io @cyclidci github.com/Cyclid/ Unopinionated? Agnostic? Flexible? Operable? Open Source? Jenkins Y Y Y N Y Travis N N N N Note 1 CircleCI N N N - N Codeship Y N N - N GoCD Y N Y Note 2 Y Semaphore Y N N - N Concourse Y N Note 3 Note 4 Y Drone Note 5 N N Y Y 1. Mostly Open Source but key components are not. 2. YUM/APT packages, but UI or giant XML configuration file of Doom configuration. 3. Concourse supports plugins but the plugin interface is basic. 4. Basic setup is simple (Go binary) but it gets complex, fast E.g. BOSH clusters. Lots of components. 5. Drone jobs are single scripts, so yes in the sense that you can anything you can do in a script.
- 10. https://cyclid.io @cyclidci github.com/Cyclid/ ● Written in Ruby, REST API ○ Boring technologies, because boring means well understood ○ I wanted plugins! ● “Everything is a plugin” philosophy ○ Even where something isn’t a plugin, it’s highly decoupled ● Keep the component & dependency footprint as small as possible ○ Cyclid, Nginx or Apache, Sidekiq, Redis, “a SQL database”
- 11. https://cyclid.io @cyclidci github.com/Cyclid/ Everything is a plugin ● Builder Create build instances to run jobs ● Transport Communicate with build instances ● Provisioner Configures build instances ● Action Do things when running a job ● Source Get the source(s) to build ● API Extend the core REST API ● Dispatcher Run a job somewhere
- 12. https://cyclid.io @cyclidci github.com/Cyclid/ Builders ● Google ● Digitalocean ● LXD Provisioners ● Ubuntu ● Debian Transports ● SSH ● LXD Actions ● Command ● Script ● Log ● Slack ● Email Sources ● Git API ● Github Dispatchers ● “Local” (Sidekiq)
- 13. https://cyclid.io @cyclidci github.com/Cyclid/ { "name" : "minimal-example", "environment": { }, "stages" : [ { "name" : "hello-world", "steps" : [ { "action" : "log", "message" : "hello from %{name}" } ] } ], "sequence" : [ { "stage" : "hello-world" } ] } ● Jobs can be JSON or YAML ● Single file, with the source ● Defines the Environment, Stages that make up the Sequence and the Sequence itself (pipeline)
- 14. https://cyclid.io @cyclidci github.com/Cyclid/ --- name: minimal-example environment: {} stages: - name: hello-world steps: - action: log message: Hello from %{name} sequence: - stage: hello-world
- 15. https://cyclid.io @cyclidci github.com/Cyclid/ 2017-01-07 12:49:58 +0000 : Obtaining build host... 2017-01-07 12:51:38 +0000 : Preparing build host... =============================================================================== =============================================================================== 2017-01-07 12:51:38 +0000 : Job started. Context: {"job_id"=>88, "job_name"=>"minimal-example", "job_version"=>"1.0.0", "organization"=>"test", "name"=>"buildhost-a2713b62c93f7fd828c35261b1535614", "host"=>"104.197.96.230", "username"=>"build", "workspace"=>"/home/build", "password"=>nil, "key"=>"/etc/mist/id_rsa_build", "server"=>"prod-euw-build01", "distro"=>"ubuntu", "release"=>"trusty"} ------------------------------------------------------------------------------- 2017-01-07 12:51:39 +0000 : Running stage hello-world v1.0.0 Hello from buildhost-a2713b62c93f7fd828c35261b1535614
- 16. https://cyclid.io @cyclidci github.com/Cyclid/ - name: bundle-install steps: - action: command cmd: sudo gem install bundler --no-ri --no-doc - action: command cmd: bundle install --path vendor/bundle path: '%{workspace}/Cyclid' sequence: - stage: bundle-install on_success: lint on_failure: failure - stage: lint on_success: rspec on_failure: failure ...
- 17. https://cyclid.io @cyclidci github.com/Cyclid/ Even more jobs ● Environments ○ Operating System/Distro & version ○ Additional package repositories ○ Additional packages ● Sources ● Secrets ○ Tokens, passwords, keys etc. ● Stages can also be defined on the server
- 18. https://cyclid.io @cyclidci github.com/Cyclid/ $ cyclid org show Name: test Owner Email: operations@cyclid.io Public Key: -----BEGIN PUBLIC KEY----- ... -----END PUBLIC KEY----- Members: vanders $ cyclid job submit job.yaml Job: 89 $ cyclid job status 89 Status: Waiting $ cyclid job log 89 2017-01-07 13:38:04 +0000 : Obtaining build host... ...
- 21. https://cyclid.io @cyclidci github.com/Cyclid/ What we’ve learned from Config Mgmt 1. Flexibility is good 2. It’s fine to have more than one tool Competition is good 3. Different tools and workflows suit different users 4. Composability is good 5. Data driven is good DRY
- 22. https://cyclid.io @cyclidci github.com/Cyclid/ Composable CI ● Every CI job is a snowflake ● Cyclid is built with Composability in mind ○ Every Job & Stage is versioned ○ Stages can be defined on the server and used across different jobs ● Next: shareable Stages ○ We’ve already done it with Config Management (Forge, Supermarket, Modules etc.) ○ Just another server with an API ○ ...but will require dependency management
- 23. https://cyclid.io @cyclidci github.com/Cyclid/ Data driven ● Data Driven is a natural requirement of Composability ● What would it even look for for Continuous Integration? ○ What data sources would be useful? ○ How do we organize that data? ○ How do we expose that data to both jobs & users? ● Jenkins is the closest with Parameterized Builds ○ Limited in scope ● I genuinely don’t know what this looks like ○ Let’s start the conversation!
- 24. https://cyclid.io @cyclidci github.com/Cyclid/ Logic ● Let’s stop pretending CI jobs don’t require logic ○ Stop externalizing the logic into shell scripts ● only_if/not_if ○ Same as conditionals on resources in Configuration Management tools sequence: - stage: release only_if: %{branch} eq ‘master’
- 25. https://cyclid.io @cyclidci github.com/Cyclid/ Event Framework ● Cyclid jobs are run by a simple state machine ● Originally I tried to make “Notifiers” Action plugins ○ Whoops! ● Events will be emitted during the job run, and consumed by plugins etc. ○ Fan-out & Fan-in (Pipelines!) ○ Proper Notifications, and Notification chains ○ Deep job introspection & profiling ○ Monitoring & Statistics
- 26. https://cyclid.io @cyclidci github.com/Cyclid/ Lots more ● More Plugins ○ AWS Builder ○ Redhat-ish Provisioner(s) ○ Not-Linux ○ Deployment ● Docker? ● Vagrant? ● Qemu?
- 28. https://cyclid.io @cyclidci github.com/Cyclid/ Thank you! Website cyclid.io Twitter @cyclidci Github github.com/cyclid Email contact@cyclid.io kristian@cyclid.io