Beyond the Node: Arkestration with Noah

Beyond the Node Advanced 'Arkestration' with Noah PuppetConf 2011

.plan
What's the problem?
Orchestration
Overview of Noah
Use Cases
Q&A

A Brief History of Time (with apologies to Hawking)

Configuration Management
Manual

Manual
Scripted

Manual
Scripted
Intelligent

CFEngine

Puppet

Manual
Scripted
Intelligent

So we've solved this, right?

“ I think..and I don't know if anyone would agree, that configuration management is a solved problem at this point, right?”

“ I think..and I don't know if anyone would agree, that configuration management is a solved problem at this point, right?” WTF?

"The point I want to make..Configuration management is not a solved problem...and it's dangerous to make the mistake to think that the way we do things now is the best way to do them..." - Andrew Clay Shafer

“ what I was attempting to say ... is that the current crop of configuration management tools have reached a usable point where they do enough (for now). What we’re seeing as questions now are 'How do I think beyond the single node where this tool is running?'”

“ what I was attempting to say (epic fail, I might add) is that the current crop of configuration management tools have reached a usable point where they do enough (for now). What we’re seeing as questions now are ' How do I think beyond the single node where this tool is running? '”

Orchestration and Noah Noah attempts to do two things:
Bridge an impedance mismatch between operational, developmental and application configuration
Provide mechanisms for coordination between applications, nodes, services, configuration management and other infrastructure aspects

Example Use Case 1) Capacity Reached. Tell Noah 2) Noah tells provisioning system 3) Capacity allocated. Tell Noah 4) Noah triggers Puppet on LB 5) Noah triggers Puppet on Nagios Noah Nagios Puppet (app) Load Balancer

“ Inspired” by

"ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services. All of these kinds of services are used in some form or another by distributed applications."

Features
Hierarchical name space of data registers similar to a Unix filesystem ( znodes )
Leader election, locking, configuration, sequences
Distributed and highly available
Watches
Downsides
Persistent connections required
Official bindings are C and Java
Somewhat "complex" to setup and run

Noah and ZooKeeper Noah is essentially a re-imagining of ZK, taking the things I liked:
Znode concept
Watches
and making it work over HTTP using JSON

Noah Basics

The Primitives
Hosts
Has status ( up , down , pending )
Can have Services
Services
Has status ( up , down , pending )
Must belong to a Host
Applications
Can have Configurations
Configurations
Can belong to many Applications

The Not So Primitives
Ephemerals
Identified by a “path”
Tags
Can be applied to any Primitive or Ephemeral
Links
Can be applied to any Primitive, Ephemeral or Tag
Watches
Can be applied at any hierarchy or specific object

URL Patterns
Hosts
/ hosts / app01 / services / httpd
Services
/ services / httpd / app02
Applications
/ applications / my_app / configurations / log_level
Configurations
/ configurations / log_level

GET ting Data

{ "foo": { "created_at" : "2011-04-20 04:50:41 UTC", "updated_at" : "2011-04-20 04:50:41 UTC", "tags" : [ ], "id" : "4af53f8d-36d2-2158-c4a1-514ff3578048", "links" : [ ], "services": { }, "status": "up" }, "bar": { } }

GET ting Data
Hosts

{ "foo" : { "created_at": "2011-04-20 04:50:41 UTC", "updated_at": "2011-04-20 04:50:41 UTC", "tags": [ ], "id": "4af53f8d-36d2-2158-c4a1-514ff3578048", "links": [ ], "services": { “ mysql”:”pending”, “ httpd”:”up” }, "status": "up" }, "bar": { } } Host Name

{ "foo": { "created_at": "2011-04-20 04:50:41 UTC", "updated_at": "2011-04-20 04:50:41 UTC", "tags": [ ], "id": "4af53f8d-36d2-2158-c4a1-514ff3578048", "links": [ ], "services" : { “ mysql”:”pending”, “ httpd”:”up” }, "status": "up" }, "bar": { } } Services

GET ting Data
Hosts
Services

{ "foo_svc" : { "foo" : { "created_at": "2011-04-20 05:09:44 UTC", "updated_at": "2011-04-20 05:09:44 UTC", "tags": [ ], "id": "53504526-a5d7-a604-5fe8-1f8fd2e1aef0", "links": [ ], "status": "down" } }, "both": { } } Service name

{ "foo_svc" : { "foo" : { "created_at": "2011-04-20 05:09:44 UTC", "updated_at": "2011-04-20 05:09:44 UTC", "tags": [ ], "id": "53504526-a5d7-a604-5fe8-1f8fd2e1aef0", "links": [ ], "status": "down" } }, "both": { } } Host Name

GET ting Data
Hosts
Services
Applications

{ "myrailsapp1" : { "created_at": "2011-04-21 06:57:42 UTC", "updated_at": "2011-04-21 06:57:42 UTC", "tags": [ "sample_data", "production" ], "id": "8774fead-e63a-8e5c-0453-1b5330bbd04c", "links": [ "/my_sample_organization" ], "configurations": { "database.yml": { "format": "yaml", "body": "development:n database: development_databasen adapter: mysqln username: dev_usern password: dev_passwordn" } } }, "myrestapp1": { } } App Name

{ "myrailsapp1": { "created_at": "2011-04-21 06:57:42 UTC", "updated_at": "2011-04-21 06:57:42 UTC", "tags": [ "sample_data", "production" ], "id": "8774fead-e63a-8e5c-0453-1b5330bbd04c", "links": [ "/my_sample_organization" ], "configurations" : { "database.yml": { "format": "yaml", "body": "development:n database: development_databasen adapter: mysqln username: dev_usern password: dev_passwordn" } } }, "myrestapp1": { } } Configs

GET ting Data
Hosts
Services
Applications
Configurations *

{ "barconf1" : { "created_at": "2011-04-27 01:20:37 UTC", "format": "string", "body": "barbody1", "updated_at": "2011-04-27 01:20:37 UTC", "tags": [ ], "id": "aa183dce-30ec-2d7e-c6f9-f3385161a7e6", "links": [ ] }, "barconf2": { } }

GET ting Data
Hosts
Services
Applications
Configurations *
Ephemerals *

GET ting Data
Hosts
Services
Applications
Configurations *
Ephemerals *
/ephemerals/my/config/setting returns the data stored there.

PUT ting Data
/ hosts /host_name
/ services /service_name/host_name
/ applications /application_name
/ configurations /configuration_name
/ ephemerals /arbitrary/path/to/whatever

PUT ting Data
/ hosts / host_name
/ services / service_name / host_name
/ applications / application_name
/ configurations / configuration_name
/ ephemerals / arbitrary / path / to / whatever

PUT ting Data
Host {“status”:”up”}
Service {“status”:”up”,”host_status”:”up”}
Application {“name”:”app_name”}
Configuration {“format”:”format_type”,”body”:”data”}
Ephemeral raw data

DELETE ing Data
/hosts/ host_name
/services/ service_name / host_name
/applications/ application_name
/configurations/ configuration_name
/ephemerals/ arbitrary / path / to / whatever

Tagging
Append “tag” to any PUT url with PUT method
e.g. ' /applications/some_application/tag'
{“tags”:”sometag”} or {“tags”:[“tag1”,”tag2”]}
Call DELETE instead of PUT to remove tags
GET ' /tags ' for all tags or ' /tag/tagname ' for specific tag

Linking
Append “link” to any PUT url with PUT method
e.g. ' /applications/some_application/link'
{“link_name”:”my_organization”}
Call DELETE instead of PUT to remove links *
GET ' /link_name ' for all linked objects *
Designed to allow custom namespaces

Watches

What are Watches?
A bit different than ZK watches
Async
Persistent *
Pluggable (URI pattern based)
Hierarchical (superset and subset)
Append ' /watch ' to almost any url path
{“endpoint”:”http://some_host:port/ping_me”}
' /watch ' root endpoint as well
{
“ endpoint”:”http://some_host/ping”,
” pattern”:”//noah/applications/app_name”
}

Create a Watch $ curl -XPUT -d '{“pattern”:”//noah/applications”,"endpoint":" http://host/ping "}' http://noahserver/watches

Perform some operation $ curl -XPUT -d '{"name":"fooapp2"}' http://noahserver/applications/fooapp2

Message sent to Endpoint { "id":"be9f6a45-0d49-3f00-59a8-c6ad61da24e1", "tags":[], "links":[], "name":"fooapp2" , "created_at":"2011-09-15 03:07:39 UTC", "updated_at":"2011-09-15 03:07:39 UTC", "configurations":{}, "action":"create" , "pubcategory":"//noah/applications/fooapp2" }

Perform another operation $ curl -XPUT -d '{"tags":"footag2"}' http://noahserver/applications/fooapp2/tag

Message sent to Endpoint { "id":"be9f6a45-0d49-3f00-59a8-c6ad61da24e1", "tags":["footag"] , "links":[], "name":"fooapp2" , "created_at":"2011-09-15 03:07:39 UTC", "updated_at":"2011-09-15 03:15:03 UTC", "configurations":{}, "action":"update" , "pubcategory":"//noah/applications/fooapp2" }

Watches are “plugin” based You can write your own plugins with Ruby.
URI Pattern *
amqp://user:pass@host:port/exchange
rundeck: //token@host :port/job_id
puppet://hostname/
statsd://host:port/metric.path.name.action
Currently based on Event Machine
Future based on Actors and worker pools

Watch Notes
Creator of the watch is unrelated to receiver
Don't have to use the “official” watch daemon
You can watch the watches (PubSub)

Basic Puppet Integration
http://github.com/jamtur01/puppet-noah
ENC, Functions and Facts
Hosts -> Nodes
Host's Services -> Classes
Configurations -> Facts (string types only)

So where does it fit? Noah Puppet Apps Scripts Monitoring Deploy CI

Noah + Puppet + Applications Noah Puppet Applications

Common P+A Use Case
Puppet knows Nodes and Classes
App has configuration file
Configuration file controlled by Puppet
Puppet client run required to update file

Common P+A Issues
Puppet repo is not app repo
Devs must learn ERB
Devs might even have to learn Puppet
Introducing new settings and communication
Daemon-mode is exception rather than rule

N+P+A Version
Noah knows Nodes and Classes
Move volatile settings into Noah
Configuration file controlled by Puppet
App gets volatile settings from Noah *
Future Puppet runs update Noah
Noah tells App *
Truth vs. Reality
My DB master is foo
My DB master right NOW is bar

Volatile Settings?
Not all configuration is created equal
“ Elasticity ” - Pool of worker node EC2 ips
Ad-hoc configuration – Logging levels
Environment aware – Nodes of class “foo”

N+P+A Benefits
Devs don't HAVE to learn ERB
Devs don't HAVE to learn Puppet
Ops and Dev friendly bridge point
No unintended changes released

Integration with CI Noah Puppet CI Deploy

Integration with CI
Jenkins asks Noah for nodes of class “foo”
Jenkins triggers Puppet (or deploy) on appropriate nodes
Jenkins is always “current”

Integration with Monitoring Noah Puppet Monitoring App

Integration with Monitoring
Puppet manages Nagios configs
Nagios updates Noah status (event handler)
Noah optionally fires watches based on status change
Endpoint is application. App “reconfigures”

Integration with Scripts Noah Scripts (node 1) Scripts (node 2) Scripts (node 3)

Integration with Scripts
Provide “locking” and “coordination” of cron jobs across hosts
Stop dealing with lock files on host
Use ephemerals and curl

Gotchas
Noah itself is NOT distributed *
Data redundancy based on Redis
No ACLs *
Async

Other options
ZooKeeper
Doozer
NoSQL stores (Redis + Webdis, Riak) *
Nesoi *
Write your own!

Questions?

Noah
Github: http://github.com/lusis /Noah
IRC: #noah
Blog(s): http://blog.lusis.org/blog/categories/noah
http://lusislog.blogspot.com/search/label/noah

John E. Vincent
Twitter: @lusis
Github: http://github.com/lusis
IRC: lusis
LinkedIn: http://linkedin.com/in/lusis
Blog(s): http://blog.lusis.org
http://lusislog.blogspot.com

Thank You!

http://soa.dzone.com	140
http://lanyrd.com	26
http://paper.li	9
http://java.dzone.com	6
http://a0.twimg.com	3
http://blog.fasoulas.com	2
http://us-w1.rockmelt.com	1
https://twitter.com	1
http://twitter.com	1

Beyond the Node: Arkestration with Noah

by lusis on Sep 22, 2011

Accessibility

Categories

Tags

Upload Details

Usage Rights

9 Embeds 189

Statistics

Beyond the Node: Arkestration with Noah — Presentation Transcript