Skynet project: Monitor, analyze, scale, and maintain a system in the Cloud

Skynet ProjectMonitor, analyze, scale, and maintain an infrastructure in the Cloud Or everything humans should not do @SylvainKalache

Whatabout me?•Operations Engineer @SlideShare since 2011•Fan of system automation•@SylvainKalache on Twitter

The CloudThe cloud is now everywhere, its elastic, dynamic and far from the rigid infrastructures what we had before.Most of the modern architecture are using "The Cloud" because give so much possibilities that no one can ignore it.

What about our tools?But what about our tools? Are they designed to deal with the cloud?Nagios and Ganglia, for example, are obviously not designed for. What are the alternative? Not much heh?Of course we could have use Cloudwatch because we are using EC2, but you the possibilities are still limited in term ofgranularity, personalization and you hit a limit at some point.

When these same tools does their job, something break, Nagios complains, Ganglia show the problem in its graph...And then what? Nothing, we need human intervention. Why?

Automation?Why humans, still do repetitive things, have to react to events that system could handle?Why we arent we automating the management of systems?

What brought Skynet to life?Slideshare has an infrastructure hosted in EC2 for the document conversion.We have a lot of "factories" which are converting different types of documents.This infrastructure scale up and down on demand.

Old scaling scriptOur all scaling process was divided in 3 parts: 1. A Bash script to launch and conﬁgure instances 2. Monit and Ruby code to maintain thecode running.3. Another Ruby script to shutdown the server.These 3 pieces of code were not communicating between each other and were not failure safe.

Zombie instances •Idle •Immortal •Wasting moneyMeaning that we ended up with instances not running code.That would not shutdown themselves and were doing nothing but wasting money.

Mummy Ops •No metrics - blind •Wasting time ﬁxingFor Ops we had issue with no visibility about what was going on with our infrastructure (we were only monitoring the SQS queue).We ended up wasting so much time investigating and then eventually ﬁxing problem.There was also a lack of feedback for the developers working on the conversion code, are we converting faster? Better?

With Skynet ControllerThe idea was to create a Controller that would manage the whole instance life time but not only.It would scale inteligently based on the current state of the system but also based on trend that we can generate by usinghistorical data.

Skynet architectureThe idea of Skynet was to make something ﬂexible so that we could use it elsewhere.It should not be architecture speciﬁcally for our problem but for any system.Why not open sourcing it at some point?

Collectors: Ruby daemons for collecting system metrics and a gem for application logs.Log collection: FluentdDatastore: MongoDBQuery API: Ruby + SinatraController: Ruby + MCollective + Puppet + FogDashboard: Investigation and monitoring tools

Collectors Collect system metrics Collect applications logsCollect system metrics via a Ruby daemon running on each machine, we can collect any metrics via plugins.Collect application logs via a gem.Those 2 components send data to Fluentd locally.

Fluentd Why? What? •Light •Local forwarder •Written in Ruby •Aggregation & routing •Handle failure •Stream processing •PluginsFluentd is in charge of collecting the log, routing and carrying them to their endpoint.It handle failure properly (failover to over node + will backup log if next hope is down)Written in Ruby, super light.The advantage is also the infrastructure, any output/input is managed via plugin that you create/customize without have to mess with the Fluentd core.

Why? What? •Schema less •Store system metrics & application logs •Key/value match log format •Jobs metadataMongoDB is schema less, which give us the possibility to make our data schema evolve very easily.MongoDB is ﬁting-ish the bill for the format 1 log entry one Mongo document.We store system metrics and informations (CPU, memory, top output, disk, Nginx active connection...) but also applicationlogs.

API Abstraction of the datastore Easy REST interface Keep control over requests processed Post-request computing via pluginThe abstraction layer/REST API give use the possibility to use any datastore in the backend, we are now using MongoDB but we could use Elastic search in front of it later.Also depending of the type data, we store them in different MongoDB cluster, but this is totally transparent to the data consumer.We keep control over whatʼs possible to do so that a single data consumer does not crash the whole datastore.We offer the possibility to have computing plugin to process the data right on the API server before returning them to the requester. Better to process data as near as by thesource to avoid having to transport big chunk of data over network.

Dashboard Monitor all the things Granular & global view Investigation toolWe can monitor all kind of stuff, possibilities are inﬁnite with the combination of the Rubym daemon and the gem.We can use all these data to have an global overview of our infrastructure but also have granularity and see that at thisparticular second there was a CPU spike for a job on this machine converting this document.Its also a useful debugging tool.

Controller Automate Scale Fix AlertCan use all the data collected to scale, based on current state but also trends.Can take action if a system is in a abnormal state try to fix it via possibility trees.And finally Skynet Controller can also alert, in case the system reach its limit and finally need a human, again...!

Thank you! Let Skynet take over the world to make it a better place. @SylvainKalacheIf you are interested in Skynet please donʼt hesitate to reach me!

http://www.hkmurakami.com	78
http://www.thebackoftheenvelope.com	21
https://twitter.com	15
http://www.onlydoo.com	4
http://www.linkedin.com	1

Skynet project: Monitor, analyze, scale, and maintain a system in the Cloud

by Sylvain Kalache on Mar 08, 2013

Accessibility

Categories

Upload Details

Usage Rights

5 Embeds 119

Statistics

Skynet project: Monitor, analyze, scale, and maintain a system in the Cloud Presentation Transcript