SlideShare a Scribd company logo
Skynet Project
Monitor, analyze, scale, and maintain an infrastructure in the Cloud
Or everything humans should not do
@SylvainKalache
What
about me?
•Operations Engineer @SlideShare since 2011
•Fan of system automation
•@SylvainKalache on Twitter
The Cloud
The cloud is now everywhere, it's elastic, dynamic and far from the rigid infrastructures what we had before.
Most of the modern architecture are using "The Cloud" because give so much possibilities that no one can ignore it.
What about our tools?
But what about our tools? Are they designed to deal with the cloud?
Nagios and Ganglia, for example, are obviously not designed for. What are the alternative? Not much heh?
Of course we could have use Cloudwatch because we are using EC2, but you the possibilities are still limited in term of
granularity, personalization and you hit a limit at some point.
When these same tools does their job, something break, Nagios complains, Ganglia show the problem in its graph...
And then what? Nothing, we need human intervention. Why?
Automation?
Why humans, still do repetitive things, have to react to events that system could handle?
Why we aren't we automating the management of systems?
What brought Skynet to life?
Slideshare has an infrastructure hosted in EC2 for the document conversion.
We have a lot of "factories" which are converting different types of documents.
This infrastructure scale up and down on demand.
Old scaling script
Our all scaling process was divided in 3 parts: 1. A Bash script to launch and configure instances 2. Monit and Ruby code to maintain the
code running.
3. Another Ruby script to shutdown the server.
These 3 pieces of code were not communicating between each other and were not failure safe.
•Idle
•Immortal
•Wasting money
Zombie instances
Meaning that we ended up with instances not running code.
That would not shutdown themselves and were doing nothing but wasting money.
•No metrics - blind
•Wasting time fixing
Mummy Ops
For Ops we had issue with no visibility about what was going on with our infrastructure (we were only monitoring the SQS queue).
We ended up wasting so much time investigating and then eventually fixing problem.
There was also a lack of feedback for the developers working on the conversion code, are we converting faster? Better?
With Skynet Controller
The idea was to create a Controller that would manage the whole instance life time but not only.
It would scale inteligently based on the current state of the system but also based on trend that we can generate by using
historical data.
Skynet architecture
The idea of Skynet was to make something flexible so that we could use it elsewhere.
It should not be architecture specifically for our problem but for any system.
Why not open sourcing it at some point?
Collectors: Ruby daemons for collecting system metrics and a gem for application logs.
Log collection: Fluentd
Datastore: MongoDB
Query API: Ruby + Sinatra
Controller: Ruby + MCollective + Puppet + Fog
Dashboard: Investigation and monitoring tools
Collect system metrics
Collect applications logs
Collectors
Collect system metrics via a Ruby daemon running on each machine, we can collect any metrics via plugins.
Collect application logs via a gem.
Those 2 components send data to Fluentd locally.
Fluentd
•Light
•Written in Ruby
•Handle failure
•Plugins
•Local forwarder
•Aggregation & routing
•Stream processing
Why? What?
Fluentd is in charge of collecting the log, routing and carrying them to their endpoint.
It handle failure properly (failover to over node + will backup log if next hope is down)
Written in Ruby, super light.
The advantage is also the infrastructure, any output/input is managed via plugin that you create/customize without have to mess with the Fluentd core.
•Schema less
•Key/value match log
format
•Store system metrics &
application logs
•Jobs metadata
Why? What?
MongoDB is schema less, which give us the possibility to make our data schema evolve very easily.
MongoDB is fiting-ish the bill for the format 1 log entry one Mongo document.
We store system metrics and informations (CPU, memory, top output, disk, Nginx active connection...) but also application
logs.
Abstraction of the datastore
Easy REST interface
Keep control over requests processed
Post-request computing via plugin
API
The abstraction layer/REST API give use the possibility to use any datastore in the backend, we are now using MongoDB but we could use Elastic search in front of it later.
Also depending of the type data, we store them in different MongoDB cluster, but this is totally transparent to the data consumer.
We keep control over whatʼs possible to do so that a single data consumer does not crash the whole datastore.
We offer the possibility to have computing plugin to process the data right on the API server before returning them to the requester. Better to process data as near as by the
source to avoid having to transport big chunk of data over network.
Monitor all the things
Granular & global view
Investigation tool
Dashboard
We can monitor all kind of stuff, possibilities are infinite with the combination of the Rubym daemon and the gem.
We can use all these data to have an global overview of our infrastructure but also have granularity and see that at this
particular second there was a CPU spike for a job on this machine converting this document.
It's also a useful debugging tool.
Automate
Scale
Fix
Alert
Controller
Can use all the data collected to scale, based on current state but also trends.
Can take action if a system is in a abnormal state try to fix it via possibility trees.
And finally Skynet Controller can also alert, in case the system reach its limit and finally need a human, again...!
Want to read more about it?
!
Check out the blog post!
!
http://engineering.slideshare.net/2014/04/skynet-project-monitor-scale-and-auto-heal-a-system-in-the-cloud/
Thank you!
Let Skynet take over the world to make it a better place.
@SylvainKalache
If you are interested in Skynet please donʼt hesitate to reach me!

More Related Content

What's hot

HTTP/2 standard for video streaming
HTTP/2 standard for video streamingHTTP/2 standard for video streaming
HTTP/2 standard for video streaming
Hung Thai Le
 
Share File easily between computers using sftp
Share File easily between computers using sftpShare File easily between computers using sftp
Share File easily between computers using sftp
Tushar B Kute
 
Cương ôn thi lại tiếng anh lớp 7
Cương ôn thi lại tiếng anh lớp 7Cương ôn thi lại tiếng anh lớp 7
Cương ôn thi lại tiếng anh lớp 7
Học Tập Long An
 
Composer
ComposerComposer
New Approaches to Faster Oracle Forms System Performance
New Approaches to Faster Oracle Forms System PerformanceNew Approaches to Faster Oracle Forms System Performance
New Approaches to Faster Oracle Forms System Performance
Correlsense
 
Tibco ems admin commands
Tibco ems admin commandsTibco ems admin commands
Tibco ems admin commands
Cblsolutions.com
 
Skkn day tu vung tieng anh
Skkn day tu vung tieng anhSkkn day tu vung tieng anh
Skkn day tu vung tieng anh
mrwindy_3282
 
Giáo trình minna no nihongo I - Bản tiếng Việt
Giáo trình minna no nihongo I - Bản tiếng ViệtGiáo trình minna no nihongo I - Bản tiếng Việt
Giáo trình minna no nihongo I - Bản tiếng Việt
Japanese Club
 
100 PHP question and answer
100 PHP  question and answer100 PHP  question and answer
100 PHP question and answer
Sandip Murari
 
Graduation Report Improving Foreign Language Ability For Students.docx
Graduation Report Improving Foreign Language Ability For Students.docxGraduation Report Improving Foreign Language Ability For Students.docx
Graduation Report Improving Foreign Language Ability For Students.docx
Nhận Viết Đề Tài Trọn Gói ZALO 0932091562
 

What's hot (10)

HTTP/2 standard for video streaming
HTTP/2 standard for video streamingHTTP/2 standard for video streaming
HTTP/2 standard for video streaming
 
Share File easily between computers using sftp
Share File easily between computers using sftpShare File easily between computers using sftp
Share File easily between computers using sftp
 
Cương ôn thi lại tiếng anh lớp 7
Cương ôn thi lại tiếng anh lớp 7Cương ôn thi lại tiếng anh lớp 7
Cương ôn thi lại tiếng anh lớp 7
 
Composer
ComposerComposer
Composer
 
New Approaches to Faster Oracle Forms System Performance
New Approaches to Faster Oracle Forms System PerformanceNew Approaches to Faster Oracle Forms System Performance
New Approaches to Faster Oracle Forms System Performance
 
Tibco ems admin commands
Tibco ems admin commandsTibco ems admin commands
Tibco ems admin commands
 
Skkn day tu vung tieng anh
Skkn day tu vung tieng anhSkkn day tu vung tieng anh
Skkn day tu vung tieng anh
 
Giáo trình minna no nihongo I - Bản tiếng Việt
Giáo trình minna no nihongo I - Bản tiếng ViệtGiáo trình minna no nihongo I - Bản tiếng Việt
Giáo trình minna no nihongo I - Bản tiếng Việt
 
100 PHP question and answer
100 PHP  question and answer100 PHP  question and answer
100 PHP question and answer
 
Graduation Report Improving Foreign Language Ability For Students.docx
Graduation Report Improving Foreign Language Ability For Students.docxGraduation Report Improving Foreign Language Ability For Students.docx
Graduation Report Improving Foreign Language Ability For Students.docx
 

Viewers also liked

Fluentd at SlideShare
Fluentd at SlideShareFluentd at SlideShare
Fluentd at SlideShare
Sylvain Kalache
 
CloudCamp Chicago - Big Data & Cloud May 2015 - All Slides
CloudCamp Chicago - Big Data & Cloud May 2015 - All SlidesCloudCamp Chicago - Big Data & Cloud May 2015 - All Slides
CloudCamp Chicago - Big Data & Cloud May 2015 - All Slides
CloudCamp Chicago
 
CloudCamp Chicago lightning talk "Connecting Vehicles on Google Cloud Platfor...
CloudCamp Chicago lightning talk "Connecting Vehicles on Google Cloud Platfor...CloudCamp Chicago lightning talk "Connecting Vehicles on Google Cloud Platfor...
CloudCamp Chicago lightning talk "Connecting Vehicles on Google Cloud Platfor...
CloudCamp Chicago
 
Cashing in on logging and exception data
Cashing in on logging and exception dataCashing in on logging and exception data
Cashing in on logging and exception data
Stackify
 
Native container monitoring
Native container monitoringNative container monitoring
Native container monitoring
Rohit Jnagal
 
Monitorama - Please, no more Minutes, Milliseconds, Monoliths or Monitoring T...
Monitorama - Please, no more Minutes, Milliseconds, Monoliths or Monitoring T...Monitorama - Please, no more Minutes, Milliseconds, Monoliths or Monitoring T...
Monitorama - Please, no more Minutes, Milliseconds, Monoliths or Monitoring T...
Adrian Cockcroft
 
Monitoring kubernetes across data center and cloud
Monitoring kubernetes across data center and cloudMonitoring kubernetes across data center and cloud
Monitoring kubernetes across data center and cloud
Datadog
 
Lifting the Blinds: Monitoring Windows Server 2012
Lifting the Blinds: Monitoring Windows Server 2012Lifting the Blinds: Monitoring Windows Server 2012
Lifting the Blinds: Monitoring Windows Server 2012
Datadog
 
Potentio lab report
Potentio lab reportPotentio lab report
Potentio lab report
Janine Samelo
 
Data Logging and Telemetry
Data Logging and TelemetryData Logging and Telemetry
Data Logging and Telemetry
Francesco Meschia
 
Deep-Dive to Application Insights
Deep-Dive to Application Insights Deep-Dive to Application Insights
Deep-Dive to Application Insights
Gunnar Peipman
 
Intro to open source telemetry linux con 2016
Intro to open source telemetry   linux con 2016Intro to open source telemetry   linux con 2016
Intro to open source telemetry linux con 2016
Matthew Broberg
 
Sysdig Monitorama Slides
Sysdig Monitorama SlidesSysdig Monitorama Slides
Sysdig Monitorama Slides
Loris Degioanni
 
RMG203 Cloud Infrastructure and Application Monitoring with Amazon CloudWatch...
RMG203 Cloud Infrastructure and Application Monitoring with Amazon CloudWatch...RMG203 Cloud Infrastructure and Application Monitoring with Amazon CloudWatch...
RMG203 Cloud Infrastructure and Application Monitoring with Amazon CloudWatch...
Amazon Web Services
 
An Introduction to Prometheus (GrafanaCon 2016)
An Introduction to Prometheus (GrafanaCon 2016)An Introduction to Prometheus (GrafanaCon 2016)
An Introduction to Prometheus (GrafanaCon 2016)
Brian Brazil
 
Intel SoC as a Platform to Connect Sensor Data to AWS
Intel SoC as a Platform to Connect Sensor Data to AWSIntel SoC as a Platform to Connect Sensor Data to AWS
Intel SoC as a Platform to Connect Sensor Data to AWS
Amazon Web Services
 
Skynet Week 9 H4D Stanford 2016
Skynet Week 9 H4D Stanford 2016Skynet Week 9 H4D Stanford 2016
Skynet Week 9 H4D Stanford 2016
Stanford University
 
Volta: Logging, Metrics, and Monitoring as a Service
Volta: Logging, Metrics, and Monitoring as a ServiceVolta: Logging, Metrics, and Monitoring as a Service
Volta: Logging, Metrics, and Monitoring as a Service
LN Renganarayana
 
Netflix: From Clouds to Roots
Netflix: From Clouds to RootsNetflix: From Clouds to Roots
Netflix: From Clouds to Roots
Brendan Gregg
 
AWS Re:Invent - High Availability Architecture at Netflix
AWS Re:Invent - High Availability Architecture at NetflixAWS Re:Invent - High Availability Architecture at Netflix
AWS Re:Invent - High Availability Architecture at Netflix
Adrian Cockcroft
 

Viewers also liked (20)

Fluentd at SlideShare
Fluentd at SlideShareFluentd at SlideShare
Fluentd at SlideShare
 
CloudCamp Chicago - Big Data & Cloud May 2015 - All Slides
CloudCamp Chicago - Big Data & Cloud May 2015 - All SlidesCloudCamp Chicago - Big Data & Cloud May 2015 - All Slides
CloudCamp Chicago - Big Data & Cloud May 2015 - All Slides
 
CloudCamp Chicago lightning talk "Connecting Vehicles on Google Cloud Platfor...
CloudCamp Chicago lightning talk "Connecting Vehicles on Google Cloud Platfor...CloudCamp Chicago lightning talk "Connecting Vehicles on Google Cloud Platfor...
CloudCamp Chicago lightning talk "Connecting Vehicles on Google Cloud Platfor...
 
Cashing in on logging and exception data
Cashing in on logging and exception dataCashing in on logging and exception data
Cashing in on logging and exception data
 
Native container monitoring
Native container monitoringNative container monitoring
Native container monitoring
 
Monitorama - Please, no more Minutes, Milliseconds, Monoliths or Monitoring T...
Monitorama - Please, no more Minutes, Milliseconds, Monoliths or Monitoring T...Monitorama - Please, no more Minutes, Milliseconds, Monoliths or Monitoring T...
Monitorama - Please, no more Minutes, Milliseconds, Monoliths or Monitoring T...
 
Monitoring kubernetes across data center and cloud
Monitoring kubernetes across data center and cloudMonitoring kubernetes across data center and cloud
Monitoring kubernetes across data center and cloud
 
Lifting the Blinds: Monitoring Windows Server 2012
Lifting the Blinds: Monitoring Windows Server 2012Lifting the Blinds: Monitoring Windows Server 2012
Lifting the Blinds: Monitoring Windows Server 2012
 
Potentio lab report
Potentio lab reportPotentio lab report
Potentio lab report
 
Data Logging and Telemetry
Data Logging and TelemetryData Logging and Telemetry
Data Logging and Telemetry
 
Deep-Dive to Application Insights
Deep-Dive to Application Insights Deep-Dive to Application Insights
Deep-Dive to Application Insights
 
Intro to open source telemetry linux con 2016
Intro to open source telemetry   linux con 2016Intro to open source telemetry   linux con 2016
Intro to open source telemetry linux con 2016
 
Sysdig Monitorama Slides
Sysdig Monitorama SlidesSysdig Monitorama Slides
Sysdig Monitorama Slides
 
RMG203 Cloud Infrastructure and Application Monitoring with Amazon CloudWatch...
RMG203 Cloud Infrastructure and Application Monitoring with Amazon CloudWatch...RMG203 Cloud Infrastructure and Application Monitoring with Amazon CloudWatch...
RMG203 Cloud Infrastructure and Application Monitoring with Amazon CloudWatch...
 
An Introduction to Prometheus (GrafanaCon 2016)
An Introduction to Prometheus (GrafanaCon 2016)An Introduction to Prometheus (GrafanaCon 2016)
An Introduction to Prometheus (GrafanaCon 2016)
 
Intel SoC as a Platform to Connect Sensor Data to AWS
Intel SoC as a Platform to Connect Sensor Data to AWSIntel SoC as a Platform to Connect Sensor Data to AWS
Intel SoC as a Platform to Connect Sensor Data to AWS
 
Skynet Week 9 H4D Stanford 2016
Skynet Week 9 H4D Stanford 2016Skynet Week 9 H4D Stanford 2016
Skynet Week 9 H4D Stanford 2016
 
Volta: Logging, Metrics, and Monitoring as a Service
Volta: Logging, Metrics, and Monitoring as a ServiceVolta: Logging, Metrics, and Monitoring as a Service
Volta: Logging, Metrics, and Monitoring as a Service
 
Netflix: From Clouds to Roots
Netflix: From Clouds to RootsNetflix: From Clouds to Roots
Netflix: From Clouds to Roots
 
AWS Re:Invent - High Availability Architecture at Netflix
AWS Re:Invent - High Availability Architecture at NetflixAWS Re:Invent - High Availability Architecture at Netflix
AWS Re:Invent - High Availability Architecture at Netflix
 

Similar to Skynet project: Monitor, analyze, scale, and maintain a system in the Cloud

Evolution of Monitoring and Prometheus (Dublin 2018)
Evolution of Monitoring and Prometheus (Dublin 2018)Evolution of Monitoring and Prometheus (Dublin 2018)
Evolution of Monitoring and Prometheus (Dublin 2018)
Brian Brazil
 
From Duke of DevOps to Queen of Chaos - Api days 2018
From Duke of DevOps to Queen of Chaos - Api days 2018From Duke of DevOps to Queen of Chaos - Api days 2018
From Duke of DevOps to Queen of Chaos - Api days 2018
Christophe Rochefolle
 
Your data is in Prometheus, now what? (CurrencyFair Engineering Meetup, 2016)
Your data is in Prometheus, now what? (CurrencyFair Engineering Meetup, 2016)Your data is in Prometheus, now what? (CurrencyFair Engineering Meetup, 2016)
Your data is in Prometheus, now what? (CurrencyFair Engineering Meetup, 2016)
Brian Brazil
 
Monitoring What Matters: The Prometheus Approach to Whitebox Monitoring (Berl...
Monitoring What Matters: The Prometheus Approach to Whitebox Monitoring (Berl...Monitoring What Matters: The Prometheus Approach to Whitebox Monitoring (Berl...
Monitoring What Matters: The Prometheus Approach to Whitebox Monitoring (Berl...
Brian Brazil
 
stackconf 2024 | Netdata: Open Source, Distributed Observability Pipeline – J...
stackconf 2024 | Netdata: Open Source, Distributed Observability Pipeline – J...stackconf 2024 | Netdata: Open Source, Distributed Observability Pipeline – J...
stackconf 2024 | Netdata: Open Source, Distributed Observability Pipeline – J...
NETWAYS
 
Evolving your api architecture with the strangler pattern
Evolving your api architecture with the strangler patternEvolving your api architecture with the strangler pattern
Evolving your api architecture with the strangler pattern
dwcarter74
 
Streaming analytics on Google Cloud Platform, by Javier Ramirez, teowaki
Streaming analytics on Google Cloud Platform, by Javier Ramirez, teowakiStreaming analytics on Google Cloud Platform, by Javier Ramirez, teowaki
Streaming analytics on Google Cloud Platform, by Javier Ramirez, teowaki
javier ramirez
 
Microservices and Prometheus (Microservices NYC 2016)
Microservices and Prometheus (Microservices NYC 2016)Microservices and Prometheus (Microservices NYC 2016)
Microservices and Prometheus (Microservices NYC 2016)
Brian Brazil
 
Distributesd Tracing in Serverless Systems - Shannon Hogue, Epsagon - Cloud N...
Distributesd Tracing in Serverless Systems - Shannon Hogue, Epsagon - Cloud N...Distributesd Tracing in Serverless Systems - Shannon Hogue, Epsagon - Cloud N...
Distributesd Tracing in Serverless Systems - Shannon Hogue, Epsagon - Cloud N...
Cloud Native Day Tel Aviv
 
From 🤦 to 🐿️
From 🤦 to 🐿️From 🤦 to 🐿️
From 🤦 to 🐿️
Ori Pekelman
 
Security for AWS : Journey to Least Privilege (update)
Security for AWS : Journey to Least Privilege (update)Security for AWS : Journey to Least Privilege (update)
Security for AWS : Journey to Least Privilege (update)
dhubbard858
 
Security for AWS: Journey to Least Privilege
Security for AWS: Journey to Least PrivilegeSecurity for AWS: Journey to Least Privilege
Security for AWS: Journey to Least Privilege
Lacework
 
A Practical Guide to Anomaly Detection for DevOps
A Practical Guide to Anomaly Detection for DevOpsA Practical Guide to Anomaly Detection for DevOps
A Practical Guide to Anomaly Detection for DevOps
BigPanda
 
Prometheus - Open Source Forum Japan
Prometheus  - Open Source Forum JapanPrometheus  - Open Source Forum Japan
Prometheus - Open Source Forum Japan
Brian Brazil
 
Go Observability (in practice)
Go Observability (in practice)Go Observability (in practice)
Go Observability (in practice)
Eran Levy
 
Has serverless adoption hit a roadblock?
Has serverless adoption hit a roadblock?Has serverless adoption hit a roadblock?
Has serverless adoption hit a roadblock?
Veselin Pizurica
 
Dev Ops without the Ops
Dev Ops without the OpsDev Ops without the Ops
Dev Ops without the Ops
Konstantin Gredeskoul
 
Dori Exterman, Considerations for choosing the parallel computing strategy th...
Dori Exterman, Considerations for choosing the parallel computing strategy th...Dori Exterman, Considerations for choosing the parallel computing strategy th...
Dori Exterman, Considerations for choosing the parallel computing strategy th...
Sergey Platonov
 
Infrastructure as Code, Theory Crash Course
Infrastructure as Code, Theory Crash CourseInfrastructure as Code, Theory Crash Course
Infrastructure as Code, Theory Crash Course
Dr. Sven Balnojan
 
Serverless microservices in the wild
Serverless microservices in the wildServerless microservices in the wild
Serverless microservices in the wild
Rotem Tamir
 

Similar to Skynet project: Monitor, analyze, scale, and maintain a system in the Cloud (20)

Evolution of Monitoring and Prometheus (Dublin 2018)
Evolution of Monitoring and Prometheus (Dublin 2018)Evolution of Monitoring and Prometheus (Dublin 2018)
Evolution of Monitoring and Prometheus (Dublin 2018)
 
From Duke of DevOps to Queen of Chaos - Api days 2018
From Duke of DevOps to Queen of Chaos - Api days 2018From Duke of DevOps to Queen of Chaos - Api days 2018
From Duke of DevOps to Queen of Chaos - Api days 2018
 
Your data is in Prometheus, now what? (CurrencyFair Engineering Meetup, 2016)
Your data is in Prometheus, now what? (CurrencyFair Engineering Meetup, 2016)Your data is in Prometheus, now what? (CurrencyFair Engineering Meetup, 2016)
Your data is in Prometheus, now what? (CurrencyFair Engineering Meetup, 2016)
 
Monitoring What Matters: The Prometheus Approach to Whitebox Monitoring (Berl...
Monitoring What Matters: The Prometheus Approach to Whitebox Monitoring (Berl...Monitoring What Matters: The Prometheus Approach to Whitebox Monitoring (Berl...
Monitoring What Matters: The Prometheus Approach to Whitebox Monitoring (Berl...
 
stackconf 2024 | Netdata: Open Source, Distributed Observability Pipeline – J...
stackconf 2024 | Netdata: Open Source, Distributed Observability Pipeline – J...stackconf 2024 | Netdata: Open Source, Distributed Observability Pipeline – J...
stackconf 2024 | Netdata: Open Source, Distributed Observability Pipeline – J...
 
Evolving your api architecture with the strangler pattern
Evolving your api architecture with the strangler patternEvolving your api architecture with the strangler pattern
Evolving your api architecture with the strangler pattern
 
Streaming analytics on Google Cloud Platform, by Javier Ramirez, teowaki
Streaming analytics on Google Cloud Platform, by Javier Ramirez, teowakiStreaming analytics on Google Cloud Platform, by Javier Ramirez, teowaki
Streaming analytics on Google Cloud Platform, by Javier Ramirez, teowaki
 
Microservices and Prometheus (Microservices NYC 2016)
Microservices and Prometheus (Microservices NYC 2016)Microservices and Prometheus (Microservices NYC 2016)
Microservices and Prometheus (Microservices NYC 2016)
 
Distributesd Tracing in Serverless Systems - Shannon Hogue, Epsagon - Cloud N...
Distributesd Tracing in Serverless Systems - Shannon Hogue, Epsagon - Cloud N...Distributesd Tracing in Serverless Systems - Shannon Hogue, Epsagon - Cloud N...
Distributesd Tracing in Serverless Systems - Shannon Hogue, Epsagon - Cloud N...
 
From 🤦 to 🐿️
From 🤦 to 🐿️From 🤦 to 🐿️
From 🤦 to 🐿️
 
Security for AWS : Journey to Least Privilege (update)
Security for AWS : Journey to Least Privilege (update)Security for AWS : Journey to Least Privilege (update)
Security for AWS : Journey to Least Privilege (update)
 
Security for AWS: Journey to Least Privilege
Security for AWS: Journey to Least PrivilegeSecurity for AWS: Journey to Least Privilege
Security for AWS: Journey to Least Privilege
 
A Practical Guide to Anomaly Detection for DevOps
A Practical Guide to Anomaly Detection for DevOpsA Practical Guide to Anomaly Detection for DevOps
A Practical Guide to Anomaly Detection for DevOps
 
Prometheus - Open Source Forum Japan
Prometheus  - Open Source Forum JapanPrometheus  - Open Source Forum Japan
Prometheus - Open Source Forum Japan
 
Go Observability (in practice)
Go Observability (in practice)Go Observability (in practice)
Go Observability (in practice)
 
Has serverless adoption hit a roadblock?
Has serverless adoption hit a roadblock?Has serverless adoption hit a roadblock?
Has serverless adoption hit a roadblock?
 
Dev Ops without the Ops
Dev Ops without the OpsDev Ops without the Ops
Dev Ops without the Ops
 
Dori Exterman, Considerations for choosing the parallel computing strategy th...
Dori Exterman, Considerations for choosing the parallel computing strategy th...Dori Exterman, Considerations for choosing the parallel computing strategy th...
Dori Exterman, Considerations for choosing the parallel computing strategy th...
 
Infrastructure as Code, Theory Crash Course
Infrastructure as Code, Theory Crash CourseInfrastructure as Code, Theory Crash Course
Infrastructure as Code, Theory Crash Course
 
Serverless microservices in the wild
Serverless microservices in the wildServerless microservices in the wild
Serverless microservices in the wild
 

More from Sylvain Kalache

Will 2018 be the year a robot takes your job? It's not clear whether AI and a...
Will 2018 be the year a robot takes your job? It's not clear whether AI and a...Will 2018 be the year a robot takes your job? It's not clear whether AI and a...
Will 2018 be the year a robot takes your job? It's not clear whether AI and a...
Sylvain Kalache
 
Why more tech companies are looking for workers equipped with 'soft skills'
Why more tech companies are looking for workers equipped with 'soft skills'Why more tech companies are looking for workers equipped with 'soft skills'
Why more tech companies are looking for workers equipped with 'soft skills'
Sylvain Kalache
 
Speakers for the Pathways to Youth Employment conference
Speakers for the Pathways to Youth Employment conferenceSpeakers for the Pathways to Youth Employment conference
Speakers for the Pathways to Youth Employment conference
Sylvain Kalache
 
Are you ready for the 4th industrial revolution?
Are you ready for the 4th industrial revolution?Are you ready for the 4th industrial revolution?
Are you ready for the 4th industrial revolution?
Sylvain Kalache
 
Company culture difference between France & USA
Company culture difference between France & USACompany culture difference between France & USA
Company culture difference between France & USA
Sylvain Kalache
 
while42 Saint-Émilion - La Fleur Picon
while42 Saint-Émilion - La Fleur Piconwhile42 Saint-Émilion - La Fleur Picon
while42 Saint-Émilion - La Fleur Picon
Sylvain Kalache
 
while42 the untold story
while42 the untold storywhile42 the untold story
while42 the untold story
Sylvain Kalache
 
Guillaume & Maurice invitation
Guillaume & Maurice invitationGuillaume & Maurice invitation
Guillaume & Maurice invitation
Sylvain Kalache
 
Sylvain-Kalache-resume
Sylvain-Kalache-resumeSylvain-Kalache-resume
Sylvain-Kalache-resume
Sylvain Kalache
 
SlideShare culture
SlideShare cultureSlideShare culture
SlideShare culture
Sylvain Kalache
 
SlideShare turns 5
SlideShare turns 5SlideShare turns 5
SlideShare turns 5
Sylvain Kalache
 
Tartiflette
TartifletteTartiflette
Tartiflette
Sylvain Kalache
 

More from Sylvain Kalache (12)

Will 2018 be the year a robot takes your job? It's not clear whether AI and a...
Will 2018 be the year a robot takes your job? It's not clear whether AI and a...Will 2018 be the year a robot takes your job? It's not clear whether AI and a...
Will 2018 be the year a robot takes your job? It's not clear whether AI and a...
 
Why more tech companies are looking for workers equipped with 'soft skills'
Why more tech companies are looking for workers equipped with 'soft skills'Why more tech companies are looking for workers equipped with 'soft skills'
Why more tech companies are looking for workers equipped with 'soft skills'
 
Speakers for the Pathways to Youth Employment conference
Speakers for the Pathways to Youth Employment conferenceSpeakers for the Pathways to Youth Employment conference
Speakers for the Pathways to Youth Employment conference
 
Are you ready for the 4th industrial revolution?
Are you ready for the 4th industrial revolution?Are you ready for the 4th industrial revolution?
Are you ready for the 4th industrial revolution?
 
Company culture difference between France & USA
Company culture difference between France & USACompany culture difference between France & USA
Company culture difference between France & USA
 
while42 Saint-Émilion - La Fleur Picon
while42 Saint-Émilion - La Fleur Piconwhile42 Saint-Émilion - La Fleur Picon
while42 Saint-Émilion - La Fleur Picon
 
while42 the untold story
while42 the untold storywhile42 the untold story
while42 the untold story
 
Guillaume & Maurice invitation
Guillaume & Maurice invitationGuillaume & Maurice invitation
Guillaume & Maurice invitation
 
Sylvain-Kalache-resume
Sylvain-Kalache-resumeSylvain-Kalache-resume
Sylvain-Kalache-resume
 
SlideShare culture
SlideShare cultureSlideShare culture
SlideShare culture
 
SlideShare turns 5
SlideShare turns 5SlideShare turns 5
SlideShare turns 5
 
Tartiflette
TartifletteTartiflette
Tartiflette
 

Recently uploaded

Types of Weaving loom machine & it's technology
Types of Weaving loom machine & it's technologyTypes of Weaving loom machine & it's technology
Types of Weaving loom machine & it's technology
ldtexsolbl
 
Pigging Unit Lubricant Oil Blending Plant
Pigging Unit Lubricant Oil Blending PlantPigging Unit Lubricant Oil Blending Plant
Pigging Unit Lubricant Oil Blending Plant
LINUS PROJECTS (INDIA)
 
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
sunilverma7884
 
Best Practices for Effectively Running dbt in Airflow.pdf
Best Practices for Effectively Running dbt in Airflow.pdfBest Practices for Effectively Running dbt in Airflow.pdf
Best Practices for Effectively Running dbt in Airflow.pdf
Tatiana Al-Chueyr
 
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
Priyanka Aash
 
IPLOOK Remote-Sensing Satellite Solution
IPLOOK Remote-Sensing Satellite SolutionIPLOOK Remote-Sensing Satellite Solution
IPLOOK Remote-Sensing Satellite Solution
IPLOOK Networks
 
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
bhumivarma35300
 
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptxRPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
SynapseIndia
 
Salesforce AI & Einstein Copilot Workshop
Salesforce AI & Einstein Copilot WorkshopSalesforce AI & Einstein Copilot Workshop
Salesforce AI & Einstein Copilot Workshop
CEPTES Software Inc
 
"Mastering Graphic Design: Essential Tips and Tricks for Beginners and Profes...
"Mastering Graphic Design: Essential Tips and Tricks for Beginners and Profes..."Mastering Graphic Design: Essential Tips and Tricks for Beginners and Profes...
"Mastering Graphic Design: Essential Tips and Tricks for Beginners and Profes...
Anant Gupta
 
Vulnerability Management: A Comprehensive Overview
Vulnerability Management: A Comprehensive OverviewVulnerability Management: A Comprehensive Overview
Vulnerability Management: A Comprehensive Overview
Steven Carlson
 
Amul milk launches in US: Key details of its new products ...
Amul milk launches in US: Key details of its new products ...Amul milk launches in US: Key details of its new products ...
Amul milk launches in US: Key details of its new products ...
chetankumar9855
 
High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...
High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...
High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...
aslasdfmkhan4750
 
Feature sql server terbaru performance.pptx
Feature sql server terbaru performance.pptxFeature sql server terbaru performance.pptx
Feature sql server terbaru performance.pptx
ssuser1915fe1
 
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
maigasapphire
 
Recent Advancements in the NIST-JARVIS Infrastructure
Recent Advancements in the NIST-JARVIS InfrastructureRecent Advancements in the NIST-JARVIS Infrastructure
Recent Advancements in the NIST-JARVIS Infrastructure
KAMAL CHOUDHARY
 
WhatsApp Spy Online Trackers and Monitoring Apps
WhatsApp Spy Online Trackers and Monitoring AppsWhatsApp Spy Online Trackers and Monitoring Apps
WhatsApp Spy Online Trackers and Monitoring Apps
HackersList
 
The importance of Quality Assurance for ICT Standardization
The importance of Quality Assurance for ICT StandardizationThe importance of Quality Assurance for ICT Standardization
The importance of Quality Assurance for ICT Standardization
Axel Rennoch
 
How to Build a Profitable IoT Product.pptx
How to Build a Profitable IoT Product.pptxHow to Build a Profitable IoT Product.pptx
How to Build a Profitable IoT Product.pptx
Adam Dunkels
 
The Rise of AI in Cybersecurity How Machine Learning Will Shape Threat Detect...
The Rise of AI in Cybersecurity How Machine Learning Will Shape Threat Detect...The Rise of AI in Cybersecurity How Machine Learning Will Shape Threat Detect...
The Rise of AI in Cybersecurity How Machine Learning Will Shape Threat Detect...
digitalxplive
 

Recently uploaded (20)

Types of Weaving loom machine & it's technology
Types of Weaving loom machine & it's technologyTypes of Weaving loom machine & it's technology
Types of Weaving loom machine & it's technology
 
Pigging Unit Lubricant Oil Blending Plant
Pigging Unit Lubricant Oil Blending PlantPigging Unit Lubricant Oil Blending Plant
Pigging Unit Lubricant Oil Blending Plant
 
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
 
Best Practices for Effectively Running dbt in Airflow.pdf
Best Practices for Effectively Running dbt in Airflow.pdfBest Practices for Effectively Running dbt in Airflow.pdf
Best Practices for Effectively Running dbt in Airflow.pdf
 
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
 
IPLOOK Remote-Sensing Satellite Solution
IPLOOK Remote-Sensing Satellite SolutionIPLOOK Remote-Sensing Satellite Solution
IPLOOK Remote-Sensing Satellite Solution
 
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
 
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptxRPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
 
Salesforce AI & Einstein Copilot Workshop
Salesforce AI & Einstein Copilot WorkshopSalesforce AI & Einstein Copilot Workshop
Salesforce AI & Einstein Copilot Workshop
 
"Mastering Graphic Design: Essential Tips and Tricks for Beginners and Profes...
"Mastering Graphic Design: Essential Tips and Tricks for Beginners and Profes..."Mastering Graphic Design: Essential Tips and Tricks for Beginners and Profes...
"Mastering Graphic Design: Essential Tips and Tricks for Beginners and Profes...
 
Vulnerability Management: A Comprehensive Overview
Vulnerability Management: A Comprehensive OverviewVulnerability Management: A Comprehensive Overview
Vulnerability Management: A Comprehensive Overview
 
Amul milk launches in US: Key details of its new products ...
Amul milk launches in US: Key details of its new products ...Amul milk launches in US: Key details of its new products ...
Amul milk launches in US: Key details of its new products ...
 
High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...
High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...
High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...
 
Feature sql server terbaru performance.pptx
Feature sql server terbaru performance.pptxFeature sql server terbaru performance.pptx
Feature sql server terbaru performance.pptx
 
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
 
Recent Advancements in the NIST-JARVIS Infrastructure
Recent Advancements in the NIST-JARVIS InfrastructureRecent Advancements in the NIST-JARVIS Infrastructure
Recent Advancements in the NIST-JARVIS Infrastructure
 
WhatsApp Spy Online Trackers and Monitoring Apps
WhatsApp Spy Online Trackers and Monitoring AppsWhatsApp Spy Online Trackers and Monitoring Apps
WhatsApp Spy Online Trackers and Monitoring Apps
 
The importance of Quality Assurance for ICT Standardization
The importance of Quality Assurance for ICT StandardizationThe importance of Quality Assurance for ICT Standardization
The importance of Quality Assurance for ICT Standardization
 
How to Build a Profitable IoT Product.pptx
How to Build a Profitable IoT Product.pptxHow to Build a Profitable IoT Product.pptx
How to Build a Profitable IoT Product.pptx
 
The Rise of AI in Cybersecurity How Machine Learning Will Shape Threat Detect...
The Rise of AI in Cybersecurity How Machine Learning Will Shape Threat Detect...The Rise of AI in Cybersecurity How Machine Learning Will Shape Threat Detect...
The Rise of AI in Cybersecurity How Machine Learning Will Shape Threat Detect...
 

Skynet project: Monitor, analyze, scale, and maintain a system in the Cloud

  • 1. Skynet Project Monitor, analyze, scale, and maintain an infrastructure in the Cloud Or everything humans should not do @SylvainKalache
  • 2. What about me? •Operations Engineer @SlideShare since 2011 •Fan of system automation •@SylvainKalache on Twitter
  • 3. The Cloud The cloud is now everywhere, it's elastic, dynamic and far from the rigid infrastructures what we had before. Most of the modern architecture are using "The Cloud" because give so much possibilities that no one can ignore it.
  • 4. What about our tools? But what about our tools? Are they designed to deal with the cloud? Nagios and Ganglia, for example, are obviously not designed for. What are the alternative? Not much heh? Of course we could have use Cloudwatch because we are using EC2, but you the possibilities are still limited in term of granularity, personalization and you hit a limit at some point.
  • 5. When these same tools does their job, something break, Nagios complains, Ganglia show the problem in its graph... And then what? Nothing, we need human intervention. Why?
  • 6. Automation? Why humans, still do repetitive things, have to react to events that system could handle? Why we aren't we automating the management of systems?
  • 7. What brought Skynet to life? Slideshare has an infrastructure hosted in EC2 for the document conversion. We have a lot of "factories" which are converting different types of documents. This infrastructure scale up and down on demand.
  • 8. Old scaling script Our all scaling process was divided in 3 parts: 1. A Bash script to launch and configure instances 2. Monit and Ruby code to maintain the code running. 3. Another Ruby script to shutdown the server. These 3 pieces of code were not communicating between each other and were not failure safe.
  • 9. •Idle •Immortal •Wasting money Zombie instances Meaning that we ended up with instances not running code. That would not shutdown themselves and were doing nothing but wasting money.
  • 10. •No metrics - blind •Wasting time fixing Mummy Ops For Ops we had issue with no visibility about what was going on with our infrastructure (we were only monitoring the SQS queue). We ended up wasting so much time investigating and then eventually fixing problem. There was also a lack of feedback for the developers working on the conversion code, are we converting faster? Better?
  • 11. With Skynet Controller The idea was to create a Controller that would manage the whole instance life time but not only. It would scale inteligently based on the current state of the system but also based on trend that we can generate by using historical data.
  • 12. Skynet architecture The idea of Skynet was to make something flexible so that we could use it elsewhere. It should not be architecture specifically for our problem but for any system. Why not open sourcing it at some point?
  • 13. Collectors: Ruby daemons for collecting system metrics and a gem for application logs. Log collection: Fluentd Datastore: MongoDB Query API: Ruby + Sinatra Controller: Ruby + MCollective + Puppet + Fog Dashboard: Investigation and monitoring tools
  • 14. Collect system metrics Collect applications logs Collectors Collect system metrics via a Ruby daemon running on each machine, we can collect any metrics via plugins. Collect application logs via a gem. Those 2 components send data to Fluentd locally.
  • 15. Fluentd •Light •Written in Ruby •Handle failure •Plugins •Local forwarder •Aggregation & routing •Stream processing Why? What? Fluentd is in charge of collecting the log, routing and carrying them to their endpoint. It handle failure properly (failover to over node + will backup log if next hope is down) Written in Ruby, super light. The advantage is also the infrastructure, any output/input is managed via plugin that you create/customize without have to mess with the Fluentd core.
  • 16. •Schema less •Key/value match log format •Store system metrics & application logs •Jobs metadata Why? What? MongoDB is schema less, which give us the possibility to make our data schema evolve very easily. MongoDB is fiting-ish the bill for the format 1 log entry one Mongo document. We store system metrics and informations (CPU, memory, top output, disk, Nginx active connection...) but also application logs.
  • 17. Abstraction of the datastore Easy REST interface Keep control over requests processed Post-request computing via plugin API The abstraction layer/REST API give use the possibility to use any datastore in the backend, we are now using MongoDB but we could use Elastic search in front of it later. Also depending of the type data, we store them in different MongoDB cluster, but this is totally transparent to the data consumer. We keep control over whatʼs possible to do so that a single data consumer does not crash the whole datastore. We offer the possibility to have computing plugin to process the data right on the API server before returning them to the requester. Better to process data as near as by the source to avoid having to transport big chunk of data over network.
  • 18. Monitor all the things Granular & global view Investigation tool Dashboard We can monitor all kind of stuff, possibilities are infinite with the combination of the Rubym daemon and the gem. We can use all these data to have an global overview of our infrastructure but also have granularity and see that at this particular second there was a CPU spike for a job on this machine converting this document. It's also a useful debugging tool.
  • 19. Automate Scale Fix Alert Controller Can use all the data collected to scale, based on current state but also trends. Can take action if a system is in a abnormal state try to fix it via possibility trees. And finally Skynet Controller can also alert, in case the system reach its limit and finally need a human, again...!
  • 20. Want to read more about it? ! Check out the blog post! ! http://engineering.slideshare.net/2014/04/skynet-project-monitor-scale-and-auto-heal-a-system-in-the-cloud/
  • 21. Thank you! Let Skynet take over the world to make it a better place. @SylvainKalache If you are interested in Skynet please donʼt hesitate to reach me!