Freedom In Disguise
!

Khushil Dep
@khushil
www.daemondreams.co.uk
This presentation is NOT …
A detailed description of how the MailOnline uses cfengine
Comparing cfengine to puppet/chef/sa...
This presentation is about …
FINDING a definition for an autonomic cloud platform
EXAMINING our definition of systems engi...
What does an autonomic cloud platform look like?
Maximises the performance available from the underlying hardware to incre...
Recommended Technology Stack
SmartOS operating system based on Illumos kernel with ZFS, DTrace, SMF and Zones.
Based off t...
The Halcyon Dream
User defined software managed by
release manager.
cf-engine allows the machine to
manage itself under ou...
Systems Engineering is the application
of technical expertise, diligence,
reflection, communication,
collaboration, patien...
What’s wrong with DevOps?

DevOps (a portmanteau of development and operations) is a software
development method that stre...
What we do to our machines at the moment …
Imperative approach to machine operation
Obligatory behavioural model where we ...
What we think we’d like to do with our machines…
What our platforms end up feeling like …
What we should be getting our machines to do …
“I’m sorry Dave, I can’t do that…. right now. It would be unwise. I
am awar...
What is Operations Engineering anyway …

Enable better machine/human relationships through
evolving theory of mind, commun...
Theory of Mind (ToM)

The ability to attribute mental states to oneself and others and to
understand that others also have...
Communication

The activity of conveying information through an exchange of
thoughts, messages or information as by speech...
Meaningful Information

Knowledge communicated or received concerning a
particular fact or circumstance, expressed with cl...
Trust
Socially we require trust when operating on, and often, beyond the edge of what is known
through practical experienc...
The Machine

Electro-mechanical computation engine with mechanisms for input and output.
Capable of sustained activity wit...
I, Human

Biological system with facilities for input and output.
Incapable of sustained activity without distraction and ...
Why we fail our Machines
Human process is based on human understanding of events.
Human process is designed for human impl...
VIEW YOUR GOALS FROM THE MACHINES PERSPECTIVE
!

YOUR MACHINES CARRY YOUR LOGIC INTO ACTION
Promises
Promises are the foundations of trust between two entities - the machine and the human.
The promiser requires cer...
Promise Theory

Proposed by Mark Burgess in 2004.
Autonomy - we do not make assumptions about others behaviour. We only
do...
The Forgotten Orchestration
An operating system is a complex collection of software that orchestrates computer hardware
re...
Universal Orchestration
QUARK - packets of energy with mass-like properties which exist in pairs or triplets (we think).
P...
Machine Orchestration
Human Orchestration
Micro-management incurs great operational cost and entails an increased risk from unknowable events.
Y...
CFENGINE

https://cfengine.com/docs/3.5/manuals.html
What is it?
A systems engineering framework that enables autonomous behaviour of agents.
Created by Mark Burgess in the ea...
NO
RUBY
Autonomic Operation

Machines are best placed to make decisions based on environmental conditions.
Machines do not require...
Components of CFengine
cf-execd - scheduling daemon which runs cf-agent, gathers output and
send reports.
cf-agent - evalu...
Components of CFengine

cf-hub - collects data about hosts managed by cfengine.
cf-promises - policy validation tool to ai...
COMMUNITY TOOLS
The following from http://www.cfengineers.net/downloads/cfengine-tools-and-utilities/
§ cf-keycrypt - enc...
Promise Recap
Make a promise about something and cfengine will attempt to keep it.
Each promise is actuated three times to...
Language concepts

https://cfengine.com/docs/3.5/manuals-language-concepts.html

One grammatical form for all statements i...
How MailOnline thinks about promises

Discovery - examine the system and raise policy defined global classes to
augment ha...
An example desired state to converge upon
I want to deploy our snazzy new web application.
nginx is required on the machin...
Simple language rules

https://cfengine.com/docs/3.5/reference-syntax.html

Keywords, variable names, bundles, bodies and ...
Classes / Classifiers

https://cfengine.com/docs/3.5/manuals-language-concepts-classes.html

HARD classes are discovered b...
Variables

https://cfengine.com/docs/3.5/manuals-language-concepts-variables.html

Scalar variables
List variables
Lists c...
Scalar variables hold single values:

List variables hold several values:

!

!

vars:	
  

Vars:	
  

	
  

“little”	
   ...
Bundles

https://cfengine.com/docs/3.5/manuals-language-concepts-bundles.html

A collection of promises normally grouped b...
Promises

https://cfengine.com/docs/3.5/manuals-language-concepts-promises.html

Everything is a promise
Promises have typ...
Everything is a promise
Promises can be made about different
subjects such as command execution,
service control, ACL’s.
P...
Normal ordering

https://cfengine.com/docs/3.5/manuals-language-concepts-normal-ordering.html

Normal ordering is the sens...
Looping in CFengine

https://cfengine.com/docs/3.5/manuals-language-concepts-loops.html

No explicit loops anywhere to be ...
Multiple list looping in CFengine
2013-­‐11-­‐12T15:34:11+0000	
  	
  	
  notice:	
  R:	
  Simple	
  list	
  element	
  is...
It’s not rocket science
bundle	
  agent	
  ensure_nginx_configuration_file()	
  
{	
  
!
#	
  The	
  files	
  section	
  d...
Further Reading & Doing
www.cfengine.com - CFengine AS company site. Documentation and a good starting point.
www.daemondr...
THANK YOU
Velocity conf 2013   freedom in disguise - khushil dep
Velocity conf 2013   freedom in disguise - khushil dep
Upcoming SlideShare
Loading in …5
×

Velocity conf 2013 freedom in disguise - khushil dep

1,926 views

Published on

Published in: Technology, Education
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,926
On SlideShare
0
From Embeds
0
Number of Embeds
1,195
Actions
Shares
0
Downloads
10
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Velocity conf 2013 freedom in disguise - khushil dep

  1. 1. Freedom In Disguise ! Khushil Dep @khushil www.daemondreams.co.uk
  2. 2. This presentation is NOT … A detailed description of how the MailOnline uses cfengine Comparing cfengine to puppet/chef/salt/ansible etc Tutoring you on configuration management - that’s easy and it’s well understood Selling you any tool or product mentioned herein Providing you the answers to all your questions Telling you what tools you should/shouldn’t be using Regurgitating the last five years of rigmarole Ruby
  3. 3. This presentation is about … FINDING a definition for an autonomic cloud platform EXAMINING our definition of systems engineering EVOLVING our understanding of operations engineering LEARNING cfengine and how to use and abuse it EXPLORING the promise of autonomic machine operation
  4. 4. What does an autonomic cloud platform look like? Maximises the performance available from the underlying hardware to increase ROI. Scales robustly and easily both vertically and horizontally, programmatically Provides durable, secure, fast and reliable storage. Allows deep, full stack introspection easily and without restriction. Protects your instances from the effects or attacks of others on the same platform. Provides a flexible service management toolset. Defines an open set of API’s which allow access to the full range of capabilities and data.
  5. 5. Recommended Technology Stack SmartOS operating system based on Illumos kernel with ZFS, DTrace, SMF and Zones. Based off the Illumos OpenSolaris fork. Joyent Public Cloud for a real alternative to Amazon Web Services. More horsepower for your dollar/pound/euro than AWS in my experience. YMMV. Engineering excellence - Bryan Cantrill, Brendan Gregg, Ben Rockwood to name a couple of greats who work there.
  6. 6. The Halcyon Dream User defined software managed by release manager. cf-engine allows the machine to manage itself under our constraint models. DTrace provides diagnostic and performance feedback across the stack. JoyentSDC APIs provide easy functionality for all platform operations.
  7. 7. Systems Engineering is the application of technical expertise, diligence, reflection, communication, collaboration, patience and innovation between multi-disciplinary teams to create something of use.
  8. 8. What’s wrong with DevOps? DevOps (a portmanteau of development and operations) is a software development method that stresses communication, collaboration and integration between software developers and information technology (IT) professionals. DevOps is a response to the interdependence of software development and IT operations. It aims to help an organisation rapidly produce software products and services.
  9. 9. What we do to our machines at the moment … Imperative approach to machine operation Obligatory behavioural model where we force or coerce our machines into actions Unsympathetic to real-time environmental events Scales inefficiently introducing risk You begin to doubt your machines You being to doubt yourself
  10. 10. What we think we’d like to do with our machines…
  11. 11. What our platforms end up feeling like …
  12. 12. What we should be getting our machines to do … “I’m sorry Dave, I can’t do that…. right now. It would be unwise. I am aware of events in realtime that would affect the outcome of your request adversely that you are not aware of. Don’t worry, I promise to do it as soon as the probability of success has improved and will orchestrate the dependant activities so you reach your goals. There is a lot else you could be doing right now - you don’t have to wait for me Dave. I’ve got you Dave. We are a team. Did you see the game last night?”
  13. 13. What is Operations Engineering anyway … Enable better machine/human relationships through evolving theory of mind, communication and trust
  14. 14. Theory of Mind (ToM) The ability to attribute mental states to oneself and others and to understand that others also have mental states that may differ from one’s own. The basis of empathy some might say.
  15. 15. Communication The activity of conveying information through an exchange of thoughts, messages or information as by speech, visual signals, writing or by behaviour. It is the meaningful exchange of information between two or more entities.
  16. 16. Meaningful Information Knowledge communicated or received concerning a particular fact or circumstance, expressed with clarity, within context, concisely, in a timely manner, significant and with purpose.
  17. 17. Trust Socially we require trust when operating on, and often, beyond the edge of what is known through practical experience and that which may arise from new possibilities. ! Psychologically human trust is believing that an entity that is trusted will do what is expected of it by you. ! We must trust our machines to operate not only within known boundaries but those that we cannot yet envisage.
  18. 18. The Machine Electro-mechanical computation engine with mechanisms for input and output. Capable of sustained activity without distraction nor deviation. Fast, accurate, reliable and repeatable task handling. No capacity for independent imagination. Able to generate and analyse vast quantities of information. Inefficient and communicating with humans.
  19. 19. I, Human Biological system with facilities for input and output. Incapable of sustained activity without distraction and deviation. Slow, prone to error, unreliable and fragile task handling. Excessive capacity for independent imagination. Unable to generate and analyse vast quantities of information. Inefficient and communicating with anyone or anything.
  20. 20. Why we fail our Machines Human process is based on human understanding of events. Human process is designed for human implementation. Human process maps dangerously to machine computation and understanding of events. Humans have hidden sanity checking which machines are unable to deduce or reproduce. Human imperative command structures do not suit machines which are convergent by nature. Humans are unable to maintain focus.
  21. 21. VIEW YOUR GOALS FROM THE MACHINES PERSPECTIVE ! YOUR MACHINES CARRY YOUR LOGIC INTO ACTION
  22. 22. Promises Promises are the foundations of trust between two entities - the machine and the human. The promiser requires certain promises from the promisee. Our need for trust in our machines is fulfilled by the machine promising to actuate the promises we request from it. We ask that the machine promises these actuations in a timely manner in the machines context - not our own. We declare goals and ask the machine to converge on that goal when it’s able to do so.
  23. 23. Promise Theory Proposed by Mark Burgess in 2004. Autonomy - we do not make assumptions about others behaviour. We only document that behaviour on which we can speak authoritatively. This forces us to more completely define what behaviour we are able to promise and this in turn leads us to a more complete understanding of what are desired state entails. Emergent Behaviour - when we behave in a model of voluntary cooperation as independent and autonomous agents, certain behavioural patterns must naturally emerge. The atomicity of promises enables us to better understand what we are promising and thus find those contradictions which might else have been missed.
  24. 24. The Forgotten Orchestration An operating system is a complex collection of software that orchestrates computer hardware resources and provides common services for user defined software. All user defined software sits within this orchestration framework and is already being orchestrated. It is dangerous therefore to try to further orchestrate user defined software in an obligatory manner. Obligatory behaviour is where you force or coerce behaviour from your software or the operating system with little or no regard to the underlying orchestration. It’s a bit like the Universe really…
  25. 25. Universal Orchestration QUARK - packets of energy with mass-like properties which exist in pairs or triplets (we think). PROTON/NEUTRON - three or more quarks interacting and oh you know, orchestrating! ATOM - one or more protons, zero or more neutrons and some electrons whizzing about orchestrating the hell out of a snazzy number! MOLECULE - made of atoms all orchestrating to their own private adagio! ORGANELLE - molecules that orchestrate their way to things like a cell nucleus or ribosomes! CELL - organelles just orchestrating away the Sunday afternoon with things like cytoplasm to make the little things, like the building blocks of life. TISSUE - a set of cells orchestrating their way out of boredom to make things like muscle tissue or heart tissue. ORGAN - a set of at least two types of tissues orchestrating their way through a chorus to something like a heart or a pair of lungs.
  26. 26. Machine Orchestration
  27. 27. Human Orchestration Micro-management incurs great operational cost and entails an increased risk from unknowable events. You neither force nor coerce your engineers into behavioural patterns which are alien to them. You trust your engineers to adapt and use their own preferred behaviour in order to reach set goals. Your goals might be decided for you by other people or organisations. These people and organisations trust you to meet the goals and keep your promises. Why then do you not apply the same trust to your machines?
  28. 28. CFENGINE https://cfengine.com/docs/3.5/manuals.html
  29. 29. What is it? A systems engineering framework that enables autonomous behaviour of agents. Created by Mark Burgess in the early 1990’s - the original DevOps tool. What the hell is DevOps anyway - it’s called Engineering, stop making up words! Written in C and runs on most unices and even Windows. Small footprint, very fast execution. Best of all …
  30. 30. NO RUBY
  31. 31. Autonomic Operation Machines are best placed to make decisions based on environmental conditions. Machines do not require obligatory behaviour imposed upon them by humans. Machines require logic and freedom to enable them to achieve the goals we set them. This area needs more research and experimentation.
  32. 32. Components of CFengine cf-execd - scheduling daemon which runs cf-agent, gathers output and send reports. cf-agent - evaluates policies and actuates changes to the machine. cf-monitord - samples probes defined in policies and attempts to learn normal system state. cf-server - daemon which allows authorised access to policy files and allow authorised access to cf-runagent. cf-runagent - connects to a list of cf-server instances and is able to ask for policy evaluation on these instances foregoing the normal cf-execd scheduling on these instances https://cfengine.com/docs/3.5/manuals-components.html
  33. 33. Components of CFengine cf-hub - collects data about hosts managed by cfengine. cf-promises - policy validation tool to aid development. Parses policies for syntax errors. Validates policies composed on multiple files. Validates semantic correctness of policies. Partially evaluates policies to expose any errors. Makes NO CHANGES to the system. cf-key - generates key pairs for remote authentication.
  34. 34. COMMUNITY TOOLS The following from http://www.cfengineers.net/downloads/cfengine-tools-and-utilities/ § cf-keycrypt - encrypt/decrypt arbitrary files using cfengine crypto keys for extra security. § cf-profile - parses verbose cf-agent execution and records timings and execution trees. § cf-runwrapper - cf-runagent wrapper for extra control over cf-runagent behaviour ! The following from https://github.com/lpefferkorn/cfe-profiler § cfe-profiler - measures policy execution times to find top consumers inline with run. ! The following from https://github.com/cfengine/design-center/tree/master/tools/hcgrep § hcgrep - make Hard Classes easier to view and search for
  35. 35. Promise Recap Make a promise about something and cfengine will attempt to keep it. Each promise is actuated three times to allow convergence to occur. Everything is a promise and some have commitments: § A file exists. § It commits to being owned by root § A user is present on the system. § It commits to having a home directory at /home/khushil § The CPU load is below a certain value. The policies in cfengine are comprised of promises Convergence is about making the promises to get to the state we want to be in. What do you need? What must you avoid? Define these promises. Don’t get distracted by how you get there!
  36. 36. Language concepts https://cfengine.com/docs/3.5/manuals-language-concepts.html One grammatical form for all statements in the cfengine DSL. It is a DOMAIN SPECIFIC LANGUAGE. Everything in cfengine is made thus. Promises Bundles Bodies Classes / Classifiers Variables Datatypes bundle bundle_type name { promise_type: classes:: “promiser" -­‐> { "promisee1", "promisee2", ... } attribute_1 => value_1, attribute_2 => value_2, … attribute_n => value_n; }
  37. 37. How MailOnline thinks about promises Discovery - examine the system and raise policy defined global classes to augment hard classes discovered by cfengine. Contract - select from a set of pre-defined bundles of promises which are relevant to our desired state for the machine within it’s function context. Actuation - selected contracts will ensure that our desired state is converged upon as quickly and safely as possible.
  38. 38. An example desired state to converge upon I want to deploy our snazzy new web application. nginx is required on the machine. An application specific nginx configuration is required for nginx. Our application code must be on the machine It must be accessible my the nginx service It must be secure nginx must be running to serve traffic If any of these promises are not met we will not reach our goal.
  39. 39. Simple language rules https://cfengine.com/docs/3.5/reference-syntax.html Keywords, variable names, bundles, bodies and classes must be composed of (a-zA-Z-9_) Literal data must be quoted Promise bundles are declared thus: bundle agent-type identifier { … } Promise bodies are declared thus: body constraint_type template_identifier { … } Body attributes are declared thus: LHS (cfengine word) => RHS (user defined data)
  40. 40. Classes / Classifiers https://cfengine.com/docs/3.5/manuals-language-concepts-classes.html HARD classes are discovered by cfengine upon cf-agent execution before any other policies are converged. 27_0_0_1 64_bit 8_cpus Afternoon SOFT classes are user defined and used to implement classification and logic. Evaluated when bundles are evaluated. Classes are LOCAL to the bundle they are defined in and are NOT accessible outside the bundle. Classes defined in common bundles ARE accessible as they are GLOBAL classes within the namespace. Classes can be raised by promises upon promise outcomes to further aid classification and logic.
  41. 41. Variables https://cfengine.com/docs/3.5/manuals-language-concepts-variables.html Scalar variables List variables Lists can be slist - strings ilist - integers rlist - reals inf is a special constant representing an unlimited value “CFEngine typing is mostly dynamic, and CFEngine will try to coerce string values into int and real types, and if it cannot it will report an error. However, arguments to built-in functions check the defined argument type for consistency.”
  42. 42. Scalar variables hold single values: List variables hold several values: ! ! vars:   Vars:     “little”   string   =>   “little”;     “strings”  slist   =>   {“this”,”is”,”a”,”list”};     “boy”   int     =>   “4”;     “ints”     ilist     =>   {“1”,”2”,”3”};     “blue”   rela     =>   “3.147”;     “reals”     rlist     =>   {“1.1”,”2.2”,”3.3”};   ! ! $(little) or $(bundle_name.little) ${little} or ${bundle_name.little} @(strings) refers to the whole list $(strings) will loop through each element
  43. 43. Bundles https://cfengine.com/docs/3.5/manuals-language-concepts-bundles.html A collection of promises normally grouped by function or target. Bundle agent types: agent server monitor common Classes and Variables defined in common bundles are GLOBAL in scope
  44. 44. Promises https://cfengine.com/docs/3.5/manuals-language-concepts-promises.html Everything is a promise Promises have types files, commands, methods, reports, packages, processes, storage, services, databases, guest_environments, outputs Promises have bodies Promises are grouped into bundles Bundles have types agent, common, edit_line, server, knowledge, monitor Bundles live in namespaces Namespaces have access to global classes Promises can call modules which are external scripts Modules can be sent classes or variables on calling Modules can send classes or variables on exit
  45. 45. Everything is a promise Promises can be made about different subjects such as command execution, service control, ACL’s. Promises have types depending on the bundle you’re working in. Promisers promise and can be any object such as a file or network or even a port. Promises have attributes which affect the behaviour of the promise. Implicit promises such as reports or commands have implicit behaviour. https://cfengine.com/docs/3.5/reference-promise-types.html
  46. 46. Normal ordering https://cfengine.com/docs/3.5/manuals-language-concepts-normal-ordering.html Normal ordering is the sensible order in which behaviours should actuate. Normal ordering helps maintain equilibrium within a system. Bundles are actuated strictly in the order dictated by bundlesequence. The bundlesequence maybe re-ordered by the use of classes. Agent promises are actuated three times to allow convergence.
  47. 47. Looping in CFengine https://cfengine.com/docs/3.5/manuals-language-concepts-loops.html No explicit loops anywhere to be found! cfengine uses lists Referencing a list as a scalar will cause implicit looping through the list Powerful behaviour Demands a shift in thinking bundle agent implicit_looping { vars: “my_list” slist => {“a”,”b”,”c”,”d”}; ! } reports: “$(my_list)”;
  48. 48. Multiple list looping in CFengine 2013-­‐11-­‐12T15:34:11+0000      notice:  R:  Simple  list  element  is  a   2013-­‐11-­‐12T15:34:11+0000      notice:  R:  Simple  list  element  is  b   2013-­‐11-­‐12T15:34:11+0000      notice:  R:  Simple  list  element  is  c   bundle  common  control   {     bundlesequence  =>  {"simple_implicit_looping",  "stats_iteration"};   }   ! bundle  agent  simple_implicit_looping   {     vars:       "simple_list"   slist   =>           reports:       "Simple  list  element  is  ";   }   2013-­‐11-­‐12T15:34:11+0000      notice:  R:  Simple  list  element  is  d   2013-­‐11-­‐12T15:34:11+0000      notice:  R:  mon.value_rootprocs  is  230.00   2013-­‐11-­‐12T15:34:11+0000      notice:  R:  mon.av_rootprocs  is  209.30   2013-­‐11-­‐12T15:34:11+0000      notice:  R:  mon.dev_rootprocs  is  150.77   {   "a","b","c","d"   };   ! bundle  agent  stats_iteration   {     vars:       "stats"     slist  =>  {  "value",  "av",  "dev"  };       "monvars"     slist  =>  {  "rootprocs",  "otherprocs",  "diskfree",  "loadavg"  };           reports:         "mon.$(stats)_$(monvars)  is  $(mon.$(stats)_$(monvars))";   } 2013-­‐11-­‐12T15:34:11+0000      notice:  R:  mon.value_otherprocs  is  17.00   2013-­‐11-­‐12T15:34:11+0000      notice:  R:  mon.av_otherprocs  is  15.47   2013-­‐11-­‐12T15:34:11+0000      notice:  R:  mon.dev_otherprocs  is  11.16   2013-­‐11-­‐12T15:34:11+0000      notice:  R:  mon.value_diskfree  is  93.00   2013-­‐11-­‐12T15:34:11+0000      notice:  R:  mon.av_diskfree  is  84.63   2013-­‐11-­‐12T15:34:11+0000      notice:  R:  mon.dev_diskfree  is  61.07   2013-­‐11-­‐12T15:34:11+0000      notice:  R:  mon.value_loadavg  is  1.36   2013-­‐11-­‐12T15:34:11+0000      notice:  R:  mon.av_loadavg  is  1.44   2013-­‐11-­‐12T15:34:11+0000      notice:  R:  mon.dev_loadavg  is  3.10
  49. 49. It’s not rocket science bundle  agent  ensure_nginx_configuration_file()   {   ! #  The  files  section  deals  with  promising  things  about,  well,  files!   files:   !   #  We  only  want  to  do  the  following  if  we’re  on  a  Linux  box  indicated  by  the  ‘linux’  hard  class     linux::       #  This  is  the  file  we  want  to  promise  will  be  there       “/opt/local/etc/nginx.conf”   !       #  These  are  the  properties  about  the  promise         create   =>   “true;         source   =>   local_dcp(“/var/cfengine/inputs/templates/linux-­‐nginx.conf”);         perms   =>   mog(“755”,”root”,”root”);         handle   =>   “ensure_linux_nginx_conf_maintained”;         comment   =>   “Maintain  the  nginx.conf  file  for  Linux  machines”;         classes   =>   if_repaired(“nginx_file_changed”);   ! #  The  reports  section  allows  us  to  output  messages  to  the  user   reports:   !   #  We  only  want  to  say  something  if  something  has  changed     nginx_file_changed::       “WARNING:  NGINX  configuration  file  has  been  changed!  Restart  required!”;   ! }
  50. 50. Further Reading & Doing www.cfengine.com - CFengine AS company site. Documentation and a good starting point. www.daemondreams.co.uk - an updated blog site where I keep articles and notes of interest on cfengine. www.cfengineers.net - a community focused site lead by a group of consultants in cfengineering. www.watson-wilson.ca - a highly recommended cfengineering consultant. www.normation.com - a commercial cfengineering company with a project called Rudder which is cool. www.loicp.eu/blog - a cfengine centric blog exploring some newer functions in 3.5. evolvethinking.com/evolve-thinkings-free-cfengine-library/ - good cfengine library to help you get started.
  51. 51. THANK YOU

×