In a world where infrastructure is code, CI is important. Yet most Continuous Integration platforms are targeted at software developers, making them a poor fit for "non-traditional" workflows like Configuration Management. Many people struggle to fit these workflows into existing CI systems. In this talk I'll explain why I believe CI is not a solved problem, and how Config Management Camp inspired to me to finally do something about it. I'll introduce Cyclid, a Configuration Management system that tries to take some of lessons we've learned from the past 5 years of Configuration Management and apply them to Continuous Integration.
Given at Config Management Camp 2017
2. https://cyclid.io @cyclidci github.com/Cyclid/
Kristian van der Vliet
● DevOpsing hard for the previous 7+ years
Puppet 0.24
Chef 0.9
● Gainfully employed as an SRE for a CDN company
● You may remember me from such projects as Syllable
● Don’t be fooled by the name, mijn Nederlands is niet
goed!
3. https://cyclid.io @cyclidci github.com/Cyclid/
Why isn’t CI a solved problem?
● I’ve seen many CI pipelines for things like Chef; I’ve never met one I liked
● Continuous Integration tools make assumptions
○ You’re “building” software
○ It will result in an “artifact”
○ Your software is one of a known set of languages
○ The build process is well defined
● None of these are true for “non-traditional” software
○ Configuration Management/Infrastructure as Code
○ Network configuration
○ Database schemas
6. https://cyclid.io @cyclidci github.com/Cyclid/
My ideal Continuous Integration
● Unopinionated Not tied to any workflow or toolset.
● Agnostic Tools change constantly.
● Flexible ...so do functional requirements.
● Operable I’m an Operations person!
● Open Source I’m also an Open Source person.
8. https://cyclid.io @cyclidci github.com/Cyclid/
Unopinionated? Agnostic? Flexible? Operable? Open Source?
Jenkins Y Y Y N Y
Travis N N N N Note 1
CircleCI N N N - N
Codeship Y N N - N
GoCD Y N Y Note 2 Y
Semaphore Y N N - N
Concourse Y N Note 3 Note 4 Y
Drone Note 5 N N Y Y
1. Mostly Open Source but key components are not.
2. YUM/APT packages, but UI or giant XML configuration file of Doom configuration.
3. Concourse supports plugins but the plugin interface is basic.
4. Basic setup is simple (Go binary) but it gets complex, fast E.g. BOSH clusters. Lots of components.
5. Drone jobs are single scripts, so yes in the sense that you can anything you can do in a script.
10. https://cyclid.io @cyclidci github.com/Cyclid/
● Written in Ruby, REST API
○ Boring technologies, because boring means well understood
○ I wanted plugins!
● “Everything is a plugin” philosophy
○ Even where something isn’t a plugin, it’s highly decoupled
● Keep the component & dependency footprint as small as possible
○ Cyclid, Nginx or Apache, Sidekiq, Redis, “a SQL database”
11. https://cyclid.io @cyclidci github.com/Cyclid/
Everything is a plugin
● Builder Create build instances to run jobs
● Transport Communicate with build instances
● Provisioner Configures build instances
● Action Do things when running a job
● Source Get the source(s) to build
● API Extend the core REST API
● Dispatcher Run a job somewhere
13. https://cyclid.io @cyclidci github.com/Cyclid/
{
"name" : "minimal-example",
"environment": {
},
"stages" : [
{
"name" : "hello-world",
"steps" : [
{
"action" : "log",
"message" : "hello from %{name}"
}
]
}
],
"sequence" : [
{
"stage" : "hello-world"
}
]
}
● Jobs can be JSON or YAML
● Single file, with the source
● Defines the Environment,
Stages that make up the
Sequence and the Sequence
itself (pipeline)
17. https://cyclid.io @cyclidci github.com/Cyclid/
Even more jobs
● Environments
○ Operating System/Distro & version
○ Additional package repositories
○ Additional packages
● Sources
● Secrets
○ Tokens, passwords, keys etc.
● Stages can also be defined on the server
18. https://cyclid.io @cyclidci github.com/Cyclid/
$ cyclid org show
Name: test
Owner Email: operations@cyclid.io
Public Key: -----BEGIN PUBLIC KEY-----
...
-----END PUBLIC KEY-----
Members:
vanders
$ cyclid job submit job.yaml
Job: 89
$ cyclid job status 89
Status: Waiting
$ cyclid job log 89
2017-01-07 13:38:04 +0000 : Obtaining build host...
...
21. https://cyclid.io @cyclidci github.com/Cyclid/
What we’ve learned from Config Mgmt
1. Flexibility is good
2. It’s fine to have more than one tool
Competition is good
3. Different tools and workflows suit different users
4. Composability is good
5. Data driven is good
DRY
22. https://cyclid.io @cyclidci github.com/Cyclid/
Composable CI
● Every CI job is a snowflake
● Cyclid is built with Composability in mind
○ Every Job & Stage is versioned
○ Stages can be defined on the server and used across different jobs
● Next: shareable Stages
○ We’ve already done it with Config Management (Forge, Supermarket, Modules
etc.)
○ Just another server with an API
○ ...but will require dependency management
23. https://cyclid.io @cyclidci github.com/Cyclid/
Data driven
● Data Driven is a natural requirement of Composability
● What would it even look for for Continuous Integration?
○ What data sources would be useful?
○ How do we organize that data?
○ How do we expose that data to both jobs & users?
● Jenkins is the closest with Parameterized Builds
○ Limited in scope
● I genuinely don’t know what this looks like
○ Let’s start the conversation!
24. https://cyclid.io @cyclidci github.com/Cyclid/
Logic
● Let’s stop pretending CI jobs don’t require logic
○ Stop externalizing the logic into shell scripts
● only_if/not_if
○ Same as conditionals on resources in Configuration Management tools
sequence:
- stage: release
only_if: %{branch} eq ‘master’
25. https://cyclid.io @cyclidci github.com/Cyclid/
Event Framework
● Cyclid jobs are run by a simple state machine
● Originally I tried to make “Notifiers” Action plugins
○ Whoops!
● Events will be emitted during the job run, and consumed by
plugins etc.
○ Fan-out & Fan-in (Pipelines!)
○ Proper Notifications, and Notification chains
○ Deep job introspection & profiling
○ Monitoring & Statistics