2. This presentation is NOT …
A detailed description of how the MailOnline uses cfengine
Comparing cfengine to puppet/chef/salt/ansible etc
Tutoring you on configuration management - that’s easy and it’s well understood
Selling you any tool or product mentioned herein
Providing you the answers to all your questions
Telling you what tools you should/shouldn’t be using
Regurgitating the last five years of rigmarole
Ruby
3. This presentation is about …
FINDING a definition for an autonomic cloud platform
EXAMINING our definition of systems engineering
EVOLVING our understanding of operations engineering
LEARNING cfengine and how to use and abuse it
EXPLORING the promise of autonomic machine operation
4. What does an autonomic cloud platform look like?
Maximises the performance available from the underlying hardware to increase ROI.
Scales robustly and easily both vertically and horizontally, programmatically
Provides durable, secure, fast and reliable storage.
Allows deep, full stack introspection easily and without restriction.
Protects your instances from the effects or attacks of others on the same platform.
Provides a flexible service management toolset.
Defines an open set of API’s which allow access to the full range of capabilities and data.
5. Recommended Technology Stack
SmartOS operating system based on Illumos kernel with ZFS, DTrace, SMF and Zones.
Based off the Illumos OpenSolaris fork.
Joyent Public Cloud for a real alternative to Amazon Web Services.
More horsepower for your dollar/pound/euro than AWS in my experience. YMMV.
Engineering excellence - Bryan Cantrill, Brendan Gregg, Ben Rockwood to name a couple
of greats who work there.
6. The Halcyon Dream
User defined software managed by
release manager.
cf-engine allows the machine to
manage itself under our constraint
models.
DTrace provides diagnostic and
performance feedback across the
stack.
JoyentSDC APIs provide easy
functionality for all platform
operations.
7. Systems Engineering is the application
of technical expertise, diligence,
reflection, communication,
collaboration, patience and innovation
between multi-disciplinary teams to
create something of use.
8.
9. What’s wrong with DevOps?
DevOps (a portmanteau of development and operations) is a software
development method that stresses communication, collaboration and
integration between software developers and information technology
(IT) professionals. DevOps is a response to the interdependence of
software development and IT operations. It aims to help an
organisation rapidly produce software products and services.
10. What we do to our machines at the moment …
Imperative approach to machine operation
Obligatory behavioural model where we force or
coerce our machines into actions
Unsympathetic to real-time environmental events
Scales inefficiently introducing risk
You begin to doubt your machines
You being to doubt yourself
13. What we should be getting our machines to do …
“I’m sorry Dave, I can’t do that…. right now. It would be unwise. I
am aware of events in realtime that would affect the outcome of
your request adversely that you are not aware of.
Don’t worry, I promise to do it as soon as the probability of
success has improved and will orchestrate the dependant
activities so you reach your goals.
There is a lot else you could be doing right now - you don’t have
to wait for me Dave.
I’ve got you Dave. We are a team.
Did you see the game last night?”
14. What is Operations Engineering anyway …
Enable better machine/human relationships through
evolving theory of mind, communication and trust
15. Theory of Mind (ToM)
The ability to attribute mental states to oneself and others and to
understand that others also have mental states that may differ
from one’s own. The basis of empathy some might say.
16. Communication
The activity of conveying information through an exchange of
thoughts, messages or information as by speech, visual signals,
writing or by behaviour. It is the meaningful exchange of
information between two or more entities.
17. Meaningful Information
Knowledge communicated or received concerning a
particular fact or circumstance, expressed with clarity,
within context, concisely, in a timely manner,
significant and with purpose.
18. Trust
Socially we require trust when operating on, and often, beyond the edge of what is known
through practical experience and that which may arise from new possibilities.
!
Psychologically human trust is believing that an entity that is trusted will do what is expected of
it by you.
!
We must trust our machines to operate not only within known boundaries but those that we
cannot yet envisage.
19. The Machine
Electro-mechanical computation engine with mechanisms for input and output.
Capable of sustained activity without distraction nor deviation.
Fast, accurate, reliable and repeatable task handling.
No capacity for independent imagination.
Able to generate and analyse vast quantities of information.
Inefficient and communicating with humans.
20. I, Human
Biological system with facilities for input and output.
Incapable of sustained activity without distraction and deviation.
Slow, prone to error, unreliable and fragile task handling.
Excessive capacity for independent imagination.
Unable to generate and analyse vast quantities of information.
Inefficient and communicating with anyone or anything.
21. Why we fail our Machines
Human process is based on human understanding of events.
Human process is designed for human implementation.
Human process maps dangerously to machine computation and understanding of events.
Humans have hidden sanity checking which machines are unable to deduce or reproduce.
Human imperative command structures do not suit machines which are convergent by nature.
Humans are unable to maintain focus.
22. VIEW YOUR GOALS FROM THE MACHINES PERSPECTIVE
!
YOUR MACHINES CARRY YOUR LOGIC INTO ACTION
23. Promises
Promises are the foundations of trust between two entities - the machine and the human.
The promiser requires certain promises from the promisee.
Our need for trust in our machines is fulfilled by the machine promising to actuate the
promises we request from it.
We ask that the machine promises these actuations in a timely manner in the machines
context - not our own.
We declare goals and ask the machine to converge on that goal when it’s able to do so.
24. Promise Theory
Proposed by Mark Burgess in 2004.
Autonomy - we do not make assumptions about others behaviour. We only
document that behaviour on which we can speak authoritatively. This forces us to
more completely define what behaviour we are able to promise and this in turn
leads us to a more complete understanding of what are desired state entails.
Emergent Behaviour - when we behave in a model of voluntary cooperation as
independent and autonomous agents, certain behavioural patterns must naturally
emerge. The atomicity of promises enables us to better understand what we are
promising and thus find those contradictions which might else have been missed.
25. The Forgotten Orchestration
An operating system is a complex collection of software that orchestrates computer hardware
resources and provides common services for user defined software.
All user defined software sits within this orchestration framework and is already being orchestrated.
It is dangerous therefore to try to further orchestrate user defined software in an obligatory
manner.
Obligatory behaviour is where you force or coerce behaviour from your software or the operating
system with little or no regard to the underlying orchestration.
It’s a bit like the Universe really…
26. Universal Orchestration
QUARK - packets of energy with mass-like properties which exist in pairs or triplets (we think).
PROTON/NEUTRON - three or more quarks interacting and oh you know, orchestrating!
ATOM - one or more protons, zero or more neutrons and some electrons whizzing about orchestrating the hell out of a snazzy
number!
MOLECULE - made of atoms all orchestrating to their own private adagio!
ORGANELLE - molecules that orchestrate their way to things like a cell nucleus or ribosomes!
CELL - organelles just orchestrating away the Sunday afternoon with things like cytoplasm to make the little things, like the building
blocks of life.
TISSUE - a set of cells orchestrating their way out of boredom to make things like muscle tissue or heart tissue.
ORGAN - a set of at least two types of tissues orchestrating their way through a chorus to something like a heart or a pair of lungs.
28. Human Orchestration
Micro-management incurs great operational cost and entails an increased risk from unknowable events.
You neither force nor coerce your engineers into behavioural patterns which are alien to them.
You trust your engineers to adapt and use their own preferred behaviour in order to reach set goals.
Your goals might be decided for you by other people or organisations.
These people and organisations trust you to meet the goals and keep your promises.
Why then do you not apply the same trust to your machines?
31. What is it?
A systems engineering framework that enables autonomous behaviour of agents.
Created by Mark Burgess in the early 1990’s - the original DevOps tool.
What the hell is DevOps anyway - it’s called Engineering, stop making up words!
Written in C and runs on most unices and even Windows.
Small footprint, very fast execution.
Best of all …
33. Autonomic Operation
Machines are best placed to make decisions based on environmental conditions.
Machines do not require obligatory behaviour imposed upon them by humans.
Machines require logic and freedom to enable them to achieve the goals we set them.
This area needs more research and experimentation.
34. Components of CFengine
cf-execd - scheduling daemon which runs cf-agent, gathers output and
send reports.
cf-agent - evaluates policies and actuates changes to the machine.
cf-monitord - samples probes defined in policies and attempts to learn
normal system state.
cf-server - daemon which allows authorised access to policy files and
allow authorised access to cf-runagent.
cf-runagent - connects to a list of cf-server instances and is able to ask
for policy evaluation on these instances foregoing the normal cf-execd
scheduling on these instances
https://cfengine.com/docs/3.5/manuals-components.html
35. Components of CFengine
cf-hub - collects data about hosts managed by cfengine.
cf-promises - policy validation tool to aid development. Parses policies for
syntax errors. Validates policies composed on multiple files. Validates semantic
correctness of policies. Partially evaluates policies to expose any errors.
Makes NO CHANGES to the system.
cf-key - generates key pairs for remote authentication.
36. COMMUNITY TOOLS
The following from http://www.cfengineers.net/downloads/cfengine-tools-and-utilities/
§ cf-keycrypt - encrypt/decrypt arbitrary files using cfengine crypto keys for extra security.
§ cf-profile - parses verbose cf-agent execution and records timings and execution trees.
§ cf-runwrapper - cf-runagent wrapper for extra control over cf-runagent behaviour
!
The following from https://github.com/lpefferkorn/cfe-profiler
§ cfe-profiler - measures policy execution times to find top consumers inline with run.
!
The following from https://github.com/cfengine/design-center/tree/master/tools/hcgrep
§ hcgrep - make Hard Classes easier to view and search for
37. Promise Recap
Make a promise about something and cfengine will attempt to keep it.
Each promise is actuated three times to allow convergence to occur.
Everything is a promise and some have commitments:
§ A file exists.
§ It commits to being owned by root
§ A user is present on the system.
§ It commits to having a home directory at /home/khushil
§ The CPU load is below a certain value.
The policies in cfengine are comprised of promises
Convergence is about making the promises to get to the state we want to be in.
What do you need? What must you avoid? Define these promises.
Don’t get distracted by how you get there!
38. Language concepts
https://cfengine.com/docs/3.5/manuals-language-concepts.html
One grammatical form for all statements in the cfengine DSL.
It is a DOMAIN SPECIFIC LANGUAGE.
Everything in cfengine is made thus.
Promises
Bundles
Bodies
Classes / Classifiers
Variables
Datatypes
bundle bundle_type name
{
promise_type:
classes::
“promiser" -‐> { "promisee1", "promisee2", ... }
attribute_1 => value_1,
attribute_2 => value_2,
…
attribute_n => value_n;
}
39. How MailOnline thinks about promises
Discovery - examine the system and raise policy defined global classes to
augment hard classes discovered by cfengine.
Contract - select from a set of pre-defined bundles of promises which are
relevant to our desired state for the machine within it’s function context.
Actuation - selected contracts will ensure that our desired state is converged
upon as quickly and safely as possible.
40. An example desired state to converge upon
I want to deploy our snazzy new web application.
nginx is required on the machine.
An application specific nginx configuration is required for nginx.
Our application code must be on the machine
It must be accessible my the nginx service
It must be secure
nginx must be running to serve traffic
If any of these promises are not met we will not reach our goal.
41. Simple language rules
https://cfengine.com/docs/3.5/reference-syntax.html
Keywords, variable names, bundles, bodies and classes must be composed of (a-zA-Z-9_)
Literal data must be quoted
Promise bundles are declared thus:
bundle agent-type identifier { … }
Promise bodies are declared thus:
body constraint_type template_identifier { … }
Body attributes are declared thus:
LHS (cfengine word) => RHS (user defined data)
42. Classes / Classifiers
https://cfengine.com/docs/3.5/manuals-language-concepts-classes.html
HARD classes are discovered by cfengine upon cf-agent execution before any other policies are
converged.
27_0_0_1 64_bit 8_cpus Afternoon
SOFT classes are user defined and used to implement classification and logic.
Evaluated when bundles are evaluated.
Classes are LOCAL to the bundle they are defined in and are NOT accessible outside the bundle.
Classes defined in common bundles ARE accessible as they are GLOBAL classes within the
namespace.
Classes can be raised by promises upon promise outcomes to further aid classification and logic.
44. Scalar variables hold single values:
List variables hold several values:
!
!
vars:
Vars:
“little”
string
=>
“little”;
“strings”
slist
=>
{“this”,”is”,”a”,”list”};
“boy”
int
=>
“4”;
“ints”
ilist
=>
{“1”,”2”,”3”};
“blue”
rela
=>
“3.147”;
“reals”
rlist
=>
{“1.1”,”2.2”,”3.3”};
!
!
$(little) or $(bundle_name.little)
${little} or ${bundle_name.little}
@(strings) refers to the whole list
$(strings) will loop through each element
46. Promises
https://cfengine.com/docs/3.5/manuals-language-concepts-promises.html
Everything is a promise
Promises have types
files, commands, methods, reports, packages, processes, storage, services, databases, guest_environments, outputs
Promises have bodies
Promises are grouped into bundles
Bundles have types
agent, common, edit_line, server, knowledge, monitor
Bundles live in namespaces
Namespaces have access to global classes
Promises can call modules which are external scripts
Modules can be sent classes or variables on calling
Modules can send classes or variables on exit
47. Everything is a promise
Promises can be made about different
subjects such as command execution,
service control, ACL’s.
Promises have types depending on the
bundle you’re working in.
Promisers promise and can be any
object such as a file or network or even
a port.
Promises have attributes which affect the
behaviour of the promise.
Implicit promises such as reports or
commands have implicit behaviour.
https://cfengine.com/docs/3.5/reference-promise-types.html
49. Looping in CFengine
https://cfengine.com/docs/3.5/manuals-language-concepts-loops.html
No explicit loops anywhere to be found!
cfengine uses lists
Referencing a list as a scalar will cause implicit
looping through the list
Powerful behaviour
Demands a shift in thinking
bundle agent implicit_looping
{
vars:
“my_list”
slist
=>
{“a”,”b”,”c”,”d”};
!
}
reports:
“$(my_list)”;
50. Multiple list looping in CFengine
2013-‐11-‐12T15:34:11+0000
notice:
R:
Simple
list
element
is
a
2013-‐11-‐12T15:34:11+0000
notice:
R:
Simple
list
element
is
b
2013-‐11-‐12T15:34:11+0000
notice:
R:
Simple
list
element
is
c
bundle
common
control
{
bundlesequence
=>
{"simple_implicit_looping",
"stats_iteration"};
}
!
bundle
agent
simple_implicit_looping
{
vars:
"simple_list"
slist
=>
reports:
"Simple
list
element
is
";
}
2013-‐11-‐12T15:34:11+0000
notice:
R:
Simple
list
element
is
d
2013-‐11-‐12T15:34:11+0000
notice:
R:
mon.value_rootprocs
is
230.00
2013-‐11-‐12T15:34:11+0000
notice:
R:
mon.av_rootprocs
is
209.30
2013-‐11-‐12T15:34:11+0000
notice:
R:
mon.dev_rootprocs
is
150.77
{
"a","b","c","d"
};
!
bundle
agent
stats_iteration
{
vars:
"stats"
slist
=>
{
"value",
"av",
"dev"
};
"monvars"
slist
=>
{
"rootprocs",
"otherprocs",
"diskfree",
"loadavg"
};
reports:
"mon.$(stats)_$(monvars)
is
$(mon.$(stats)_$(monvars))";
}
2013-‐11-‐12T15:34:11+0000
notice:
R:
mon.value_otherprocs
is
17.00
2013-‐11-‐12T15:34:11+0000
notice:
R:
mon.av_otherprocs
is
15.47
2013-‐11-‐12T15:34:11+0000
notice:
R:
mon.dev_otherprocs
is
11.16
2013-‐11-‐12T15:34:11+0000
notice:
R:
mon.value_diskfree
is
93.00
2013-‐11-‐12T15:34:11+0000
notice:
R:
mon.av_diskfree
is
84.63
2013-‐11-‐12T15:34:11+0000
notice:
R:
mon.dev_diskfree
is
61.07
2013-‐11-‐12T15:34:11+0000
notice:
R:
mon.value_loadavg
is
1.36
2013-‐11-‐12T15:34:11+0000
notice:
R:
mon.av_loadavg
is
1.44
2013-‐11-‐12T15:34:11+0000
notice:
R:
mon.dev_loadavg
is
3.10
51. It’s not rocket science
bundle
agent
ensure_nginx_configuration_file()
{
!
#
The
files
section
deals
with
promising
things
about,
well,
files!
files:
!
#
We
only
want
to
do
the
following
if
we’re
on
a
Linux
box
indicated
by
the
‘linux’
hard
class
linux::
#
This
is
the
file
we
want
to
promise
will
be
there
“/opt/local/etc/nginx.conf”
!
#
These
are
the
properties
about
the
promise
create
=>
“true;
source
=>
local_dcp(“/var/cfengine/inputs/templates/linux-‐nginx.conf”);
perms
=>
mog(“755”,”root”,”root”);
handle
=>
“ensure_linux_nginx_conf_maintained”;
comment
=>
“Maintain
the
nginx.conf
file
for
Linux
machines”;
classes
=>
if_repaired(“nginx_file_changed”);
!
#
The
reports
section
allows
us
to
output
messages
to
the
user
reports:
!
#
We
only
want
to
say
something
if
something
has
changed
nginx_file_changed::
“WARNING:
NGINX
configuration
file
has
been
changed!
Restart
required!”;
!
}
52. Further Reading & Doing
www.cfengine.com - CFengine AS company site. Documentation and a good starting point.
www.daemondreams.co.uk - an updated blog site where I keep articles and notes of interest on cfengine.
www.cfengineers.net - a community focused site lead by a group of consultants in cfengineering.
www.watson-wilson.ca - a highly recommended cfengineering consultant.
www.normation.com - a commercial cfengineering company with a project called Rudder which is cool.
www.loicp.eu/blog - a cfengine centric blog exploring some newer functions in 3.5.
evolvethinking.com/evolve-thinkings-free-cfengine-library/ - good cfengine library to help you get started.
