SlideShare a Scribd company logo
1 of 42
DIY: A simple orchestrator built step by step
TIAD PARIS 2015
ChallengesHi, I’m Ahmet Demir
Also known as @ahmet2mir
Opensource Enthusiast @D2SI
Why Orchestration
All IT want
But we’ve got
When we try to
assemble them
http://gothicpug.deviantart.com/art/Wolverine-s-lego-cy-wtf-140295823
Choose the orchestrator
What we orchestrate ?
IT Process: disaster recovery, restart process
Infrastructure: Build a VM, upgrade it
Application Deployment: Deploy an application on x nodes
Developpements: Continuous integration, Continuous Delivery
State of the art
Eliminate "server management deployment"
FusionDirectory - http://www.fusiondirectory.org
m23 - http://m23.sourceforge.net/PostNuke-0.750/html/index.php
Fabric - http://www.fabfile.org/
Archipel - http://archipelproject.org/
AppScale - https://github.com/AppScale/appscale
OpenBuild - http://openbuildservice.org/
CompatibleOne - http://compatibleone.org/
Opsi - http://www.opsi.org
Eliminate config manager
Puppet - https://puppetlabs.com/
Chef - https://chef.io
SaltStack - http://saltstack.com/
Ansible - http://www.ansible.com/home
CFengine - http://cfengine.com/
Eliminate "server task oriented"
Rundeck - http://rundeck.org/
Eliminate Only Cloud oriented
Eucalyptus - https://github.com/eucalyptus/eucalyptus/wiki
Openstack Heat - https://github.com/openstack/heat
Finally 1/3
Terraform - https://terraform.io/
Terraform provides a flexible abstraction of resources and providers. This
model allows for representing everything from physical hardware, virtual
machines, and containers, to email and DNS providers
#Go
Finally 2/3
Salt Cloud - https://salt-cloud.readthedocs.org
Salt cloud is a public cloud provisioning tool. Salt cloud is made to integrate
Salt into cloud providers in a clean way so that minions on public cloud
systems can be quickly and easily modeled and provisioned.
#Python
Finally 3/3
OpenScore - http://openscore.github.io/
Flow-based orchestration engine to manage deployed applications
Allows you to rapidly automate your DevOps use cases using ready-made
workflows. Based on Java and HP OO engine.
#Java
OpenStack Heat based:
Resource based, CRUD, Graph dependency, describe in Yaml
Terraform:
Execution Plan, Diff on infra and all heat features, describe in “json like” format
Salt Cloud:
Easy modules integration but no plan, no graph
OpenScore:
Service in Yaml, no plan but runbook featured
Features
API access ?
How to scale ? If I had more than 100 requests in same time ?
Where can I store all my services ?
How can I track ?
Is modules integration easy ?
What about lifecycle ?
Versioning the service ?
Multi-user ?
Organization > 100 peoples ?
High Availability ?
But
Try to solve or ...
Do It Yourself
Simple to maintain
Think tomorrow
Infrastructure agnostic
Easy development
Don't forget Windows users :)
Fundamentals
NetworkStorage Compute
Orchestrator
Referential Config Manager
Catalog
HP MP, OpenCart...
Monitoring
Nagios...
CI / CD
Jenkins, Bamboo ...
Service Designer
Big Picture
Engine
API
Resource runner
Resource 1 Resource 2 Resource 3 Resource 4
Run 1 & 2 in parallel Then run 3 Then run 4
Service Designer & Orchestrator
Components
If you think to something, you will find your answer in Python
Both of Devs and Ops can use Python and use it quickly
No compilation and not "magic"
Write less and do more
Limits the number of times your brain has to do a "context switch."
Helps productivity
The language
YAML is simple to read and write
Many projects use it: Heat, Fig etc...
Service Description: YAML
Many tasks management
Easy to scale
Dedicate queue to some tasks (DMZ, Windows...)
Asynchronous task queue/job
Real-time operation, but supports scheduling
Tasks are executed concurrently
Used in production systems to process millions of tasks a day
Tasks elery
Messaging broker
Common platform to send and receive messages
Safe place to live until received
Can hits one million messages per second
Message Delivery
rapid and agile web development
No migrations
Not a relational database
Schemaless
Database
Let’s Get Started
Step 0: Build workers and broker
Rabbit MQ
Celery 1 Celery 2
Flower GUI
orion ~ $ git clone https://github.com/ahmet2mir/tiad-demo.git demo
orion ~ $ cd demo
orion demo $ cd 00_celery_rabbit
# start celery
orion 00_celery_rabbit $ fig -f celery.yml -p demo up -d
# start rabbitmq
orion 00_celery_rabbit $ fig -f rabbitmq.yml -p demo up -d
# create rabbitmq user
orion 00_celery_rabbit $ docker exec -t demo_broker_1 rabbitmqctl add_user myuser mypassword
orion 00_celery_rabbit $ docker exec -t demo_broker_1 rabbitmqctl add_vhost myvhost
orion 00_celery_rabbit $ docker exec -t demo_broker_1 rabbitmqctl set_permissions -p myvhost
myuser ".*" ".*" ".*"
# gui for flower and rabbitmq
orion 00_celery_rabbit $ fig -f flower.yml -p demo up -d
# Visit http://localhost:5555
Step 0: Demo
Step 1: Execute a simple resource
Rabbit MQ
Celery 1 Celery 2
AWS Instance
Resource
add in queue
Flower GUI
Celery 1
Wrapper
AWS
Wrapper
Docker
Docker
CRUD
AWS CRUD
Pure Python Package deployed
with pip
Wrapper to simplify execution
orion demo $ cd 01_aws_resource
# virtual python environment
orion 01_aws_resource $ virtualenv .
orion 01_aws_resource $ source bin/activate
orion 01_aws_resource $ pip install -r requirements.txt
# add credentials in client_provider.py
# edit client_provider with some values (optionnal)
…
# run the resource
orion 01_aws_resource $ python client.py
Step 1: Demo
Step 2: Store outside
Rabbit MQ
Celery 1 Celery 2
Flower GUI
MongoDBApplication
orion demo $ cd 02_db_provider
# virtual python environment
orion 02_db_provider $ virtualenv .
orion 02_db_provider $ source bin/activate
orion 02_db_provider $ pip install -r requirements.txt
# start mongodb
orion 02_db_provider $ fig -f mongodb.yml -p demo up -d
# add credentials in add_*.py
orion 02_db_provider $ python add_aws.py
orion 02_db_provider $ python add_docker.py
# visit http://localhost:8080
Step 2: Demo
Application
Step 3: Service level
Rabbit MQ
Celery 1 Celery 2
Service
AWS Instance
Resource
Flower GUI
Docker Resource
1 2
MongoDB
orion demo $ cd 03_app
# virtual python environment
orion 03_app $ virtualenv .
orion 03_app $ source bin/activate
orion 03_app $ pip install -r requirements.txt
# run the service
orion 03_app $ python teecli.py create sample01.yml
Step 3: Demo
Commit API
Commit Admin GUI
Commit Referential Management
Commit all modules and wrappers
Build a service catalog
Role based access to the app
http://teerex.io (soon)
https://github.com/ahmet2mir/teerex (soon)
Next steps
TIAD - DYI: A simple orchestrator built step by step

More Related Content

What's hot

DevOps in a Regulated World - aka 'Ansible, AWS, and Jenkins'
DevOps in a Regulated World - aka 'Ansible, AWS, and Jenkins'DevOps in a Regulated World - aka 'Ansible, AWS, and Jenkins'
DevOps in a Regulated World - aka 'Ansible, AWS, and Jenkins'rmcleay
 
Using Docker with Puppet - PuppetConf 2014
Using Docker with Puppet - PuppetConf 2014Using Docker with Puppet - PuppetConf 2014
Using Docker with Puppet - PuppetConf 2014Puppet
 
Scaling Your App With Docker Swarm using Terraform, Packer on Openstack
Scaling Your App With Docker Swarm using Terraform, Packer on OpenstackScaling Your App With Docker Swarm using Terraform, Packer on Openstack
Scaling Your App With Docker Swarm using Terraform, Packer on OpenstackBobby DeVeaux, DevOps Consultant
 
Ansible new paradigms for orchestration
Ansible new paradigms for orchestrationAnsible new paradigms for orchestration
Ansible new paradigms for orchestrationPaolo Tonin
 
Ansible Introduction
Ansible Introduction Ansible Introduction
Ansible Introduction Robert Reiz
 
Reusable, composable, battle-tested Terraform modules
Reusable, composable, battle-tested Terraform modulesReusable, composable, battle-tested Terraform modules
Reusable, composable, battle-tested Terraform modulesYevgeniy Brikman
 
How to test infrastructure code: automated testing for Terraform, Kubernetes,...
How to test infrastructure code: automated testing for Terraform, Kubernetes,...How to test infrastructure code: automated testing for Terraform, Kubernetes,...
How to test infrastructure code: automated testing for Terraform, Kubernetes,...Yevgeniy Brikman
 
Achieving Continuous Delivery: An Automation Story
Achieving Continuous Delivery: An Automation StoryAchieving Continuous Delivery: An Automation Story
Achieving Continuous Delivery: An Automation Storyjimi-c
 
Play Framework + Docker + CircleCI + AWS + EC2 Container Service
Play Framework + Docker + CircleCI + AWS + EC2 Container ServicePlay Framework + Docker + CircleCI + AWS + EC2 Container Service
Play Framework + Docker + CircleCI + AWS + EC2 Container ServiceJosh Padnick
 
Puppet and Apache CloudStack
Puppet and Apache CloudStackPuppet and Apache CloudStack
Puppet and Apache CloudStackPuppet
 
Go Faster with Ansible (PHP meetup)
Go Faster with Ansible (PHP meetup)Go Faster with Ansible (PHP meetup)
Go Faster with Ansible (PHP meetup)Richard Donkin
 
Automating aws infrastructure and code deployments using Ansible @WebEngage
Automating aws infrastructure and code deployments using Ansible @WebEngageAutomating aws infrastructure and code deployments using Ansible @WebEngage
Automating aws infrastructure and code deployments using Ansible @WebEngageVishal Uderani
 
Docker at Shopify: From This-Looks-Fun to Production by Simon Eskildsen (Shop...
Docker at Shopify: From This-Looks-Fun to Production by Simon Eskildsen (Shop...Docker at Shopify: From This-Looks-Fun to Production by Simon Eskildsen (Shop...
Docker at Shopify: From This-Looks-Fun to Production by Simon Eskildsen (Shop...Docker, Inc.
 
Introduction to Amazon EC2 Container Service and setting up build pipeline wi...
Introduction to Amazon EC2 Container Service and setting up build pipeline wi...Introduction to Amazon EC2 Container Service and setting up build pipeline wi...
Introduction to Amazon EC2 Container Service and setting up build pipeline wi...Swapnil Dahiphale
 
Infrastructure as code
Infrastructure as codeInfrastructure as code
Infrastructure as codeAxel Quack
 
Running your Java EE 6 applications in the Cloud (FISL 12)
Running your Java EE 6 applications in the Cloud (FISL 12)Running your Java EE 6 applications in the Cloud (FISL 12)
Running your Java EE 6 applications in the Cloud (FISL 12)Arun Gupta
 
Kube-AWS
Kube-AWSKube-AWS
Kube-AWSCoreOS
 

What's hot (20)

DevOps in a Regulated World - aka 'Ansible, AWS, and Jenkins'
DevOps in a Regulated World - aka 'Ansible, AWS, and Jenkins'DevOps in a Regulated World - aka 'Ansible, AWS, and Jenkins'
DevOps in a Regulated World - aka 'Ansible, AWS, and Jenkins'
 
Using Docker with Puppet - PuppetConf 2014
Using Docker with Puppet - PuppetConf 2014Using Docker with Puppet - PuppetConf 2014
Using Docker with Puppet - PuppetConf 2014
 
Scaling Your App With Docker Swarm using Terraform, Packer on Openstack
Scaling Your App With Docker Swarm using Terraform, Packer on OpenstackScaling Your App With Docker Swarm using Terraform, Packer on Openstack
Scaling Your App With Docker Swarm using Terraform, Packer on Openstack
 
Ansible new paradigms for orchestration
Ansible new paradigms for orchestrationAnsible new paradigms for orchestration
Ansible new paradigms for orchestration
 
Ansible and AWS
Ansible and AWSAnsible and AWS
Ansible and AWS
 
Ansible Introduction
Ansible Introduction Ansible Introduction
Ansible Introduction
 
Ansible Case Studies
Ansible Case StudiesAnsible Case Studies
Ansible Case Studies
 
Reusable, composable, battle-tested Terraform modules
Reusable, composable, battle-tested Terraform modulesReusable, composable, battle-tested Terraform modules
Reusable, composable, battle-tested Terraform modules
 
How to test infrastructure code: automated testing for Terraform, Kubernetes,...
How to test infrastructure code: automated testing for Terraform, Kubernetes,...How to test infrastructure code: automated testing for Terraform, Kubernetes,...
How to test infrastructure code: automated testing for Terraform, Kubernetes,...
 
Achieving Continuous Delivery: An Automation Story
Achieving Continuous Delivery: An Automation StoryAchieving Continuous Delivery: An Automation Story
Achieving Continuous Delivery: An Automation Story
 
Play Framework + Docker + CircleCI + AWS + EC2 Container Service
Play Framework + Docker + CircleCI + AWS + EC2 Container ServicePlay Framework + Docker + CircleCI + AWS + EC2 Container Service
Play Framework + Docker + CircleCI + AWS + EC2 Container Service
 
Puppet and Apache CloudStack
Puppet and Apache CloudStackPuppet and Apache CloudStack
Puppet and Apache CloudStack
 
Go Faster with Ansible (PHP meetup)
Go Faster with Ansible (PHP meetup)Go Faster with Ansible (PHP meetup)
Go Faster with Ansible (PHP meetup)
 
Automating aws infrastructure and code deployments using Ansible @WebEngage
Automating aws infrastructure and code deployments using Ansible @WebEngageAutomating aws infrastructure and code deployments using Ansible @WebEngage
Automating aws infrastructure and code deployments using Ansible @WebEngage
 
Docker at Shopify: From This-Looks-Fun to Production by Simon Eskildsen (Shop...
Docker at Shopify: From This-Looks-Fun to Production by Simon Eskildsen (Shop...Docker at Shopify: From This-Looks-Fun to Production by Simon Eskildsen (Shop...
Docker at Shopify: From This-Looks-Fun to Production by Simon Eskildsen (Shop...
 
Introduction to Amazon EC2 Container Service and setting up build pipeline wi...
Introduction to Amazon EC2 Container Service and setting up build pipeline wi...Introduction to Amazon EC2 Container Service and setting up build pipeline wi...
Introduction to Amazon EC2 Container Service and setting up build pipeline wi...
 
Cyansible
CyansibleCyansible
Cyansible
 
Infrastructure as code
Infrastructure as codeInfrastructure as code
Infrastructure as code
 
Running your Java EE 6 applications in the Cloud (FISL 12)
Running your Java EE 6 applications in the Cloud (FISL 12)Running your Java EE 6 applications in the Cloud (FISL 12)
Running your Java EE 6 applications in the Cloud (FISL 12)
 
Kube-AWS
Kube-AWSKube-AWS
Kube-AWS
 

Viewers also liked

Rundeck: The missing tool
Rundeck: The missing toolRundeck: The missing tool
Rundeck: The missing toolArtur Martins
 
RabbitMQ 101 : job scheduling, micro service communication, event based data...
 RabbitMQ 101 : job scheduling, micro service communication, event based data... RabbitMQ 101 : job scheduling, micro service communication, event based data...
RabbitMQ 101 : job scheduling, micro service communication, event based data...Quentin Adam
 
Rundeck Open Source Workflow Automation
Rundeck Open Source Workflow AutomationRundeck Open Source Workflow Automation
Rundeck Open Source Workflow Automationinovex GmbH
 
One Click Deploys using Rundeck
One Click Deploys using RundeckOne Click Deploys using Rundeck
One Click Deploys using RundeckSai Kothapalle
 
Rundeck's History and Future
Rundeck's History and FutureRundeck's History and Future
Rundeck's History and Futuredev2ops
 
How do I do DevOps when all I have is Ops?
How do I do DevOps when all I have is Ops?How do I do DevOps when all I have is Ops?
How do I do DevOps when all I have is Ops?Chris Swan
 
Robert Kubis - gRPC - boilerplate to high-performance scalable APIs - code.t...
 Robert Kubis - gRPC - boilerplate to high-performance scalable APIs - code.t... Robert Kubis - gRPC - boilerplate to high-performance scalable APIs - code.t...
Robert Kubis - gRPC - boilerplate to high-performance scalable APIs - code.t...AboutYouGmbH
 

Viewers also liked (11)

Rundeck: The missing tool
Rundeck: The missing toolRundeck: The missing tool
Rundeck: The missing tool
 
CSS Reset
CSS ResetCSS Reset
CSS Reset
 
Techzone 2014 presentation rundeck
Techzone 2014 presentation rundeckTechzone 2014 presentation rundeck
Techzone 2014 presentation rundeck
 
RabbitMQ 101 : job scheduling, micro service communication, event based data...
 RabbitMQ 101 : job scheduling, micro service communication, event based data... RabbitMQ 101 : job scheduling, micro service communication, event based data...
RabbitMQ 101 : job scheduling, micro service communication, event based data...
 
Rundeck Open Source Workflow Automation
Rundeck Open Source Workflow AutomationRundeck Open Source Workflow Automation
Rundeck Open Source Workflow Automation
 
One Click Deploys using Rundeck
One Click Deploys using RundeckOne Click Deploys using Rundeck
One Click Deploys using Rundeck
 
RunDeck
RunDeckRunDeck
RunDeck
 
Rundeck's History and Future
Rundeck's History and FutureRundeck's History and Future
Rundeck's History and Future
 
How do I do DevOps when all I have is Ops?
How do I do DevOps when all I have is Ops?How do I do DevOps when all I have is Ops?
How do I do DevOps when all I have is Ops?
 
Robert Kubis - gRPC - boilerplate to high-performance scalable APIs - code.t...
 Robert Kubis - gRPC - boilerplate to high-performance scalable APIs - code.t... Robert Kubis - gRPC - boilerplate to high-performance scalable APIs - code.t...
Robert Kubis - gRPC - boilerplate to high-performance scalable APIs - code.t...
 
Rundeck & Ansible
Rundeck & AnsibleRundeck & Ansible
Rundeck & Ansible
 

Similar to TIAD - DYI: A simple orchestrator built step by step

Developing and Deploying PHP with Docker
Developing and Deploying PHP with DockerDeveloping and Deploying PHP with Docker
Developing and Deploying PHP with DockerPatrick Mizer
 
Docker Security workshop slides
Docker Security workshop slidesDocker Security workshop slides
Docker Security workshop slidesDocker, Inc.
 
Weave User Group Talk - DockerCon 2017 Recap
Weave User Group Talk - DockerCon 2017 RecapWeave User Group Talk - DockerCon 2017 Recap
Weave User Group Talk - DockerCon 2017 RecapPatrick Chanezon
 
Better Operations into the Cloud
Better Operations  into the CloudBetter Operations  into the Cloud
Better Operations into the CloudFabio Ferrari
 
Adopt DevOps philosophy on your Symfony projects (Symfony Live 2011)
Adopt DevOps philosophy on your Symfony projects (Symfony Live 2011)Adopt DevOps philosophy on your Symfony projects (Symfony Live 2011)
Adopt DevOps philosophy on your Symfony projects (Symfony Live 2011)Fabrice Bernhard
 
Docker - Der Wal in der Kiste
Docker - Der Wal in der KisteDocker - Der Wal in der Kiste
Docker - Der Wal in der KisteUlrich Krause
 
Codetainer: a Docker-based browser code 'sandbox'
Codetainer: a Docker-based browser code 'sandbox'Codetainer: a Docker-based browser code 'sandbox'
Codetainer: a Docker-based browser code 'sandbox'Jen Andre
 
Docker Platform and Ecosystem
Docker Platform and EcosystemDocker Platform and Ecosystem
Docker Platform and EcosystemPatrick Chanezon
 
Docker Azure Friday OSS March 2017 - Developing and deploying Java & Linux on...
Docker Azure Friday OSS March 2017 - Developing and deploying Java & Linux on...Docker Azure Friday OSS March 2017 - Developing and deploying Java & Linux on...
Docker Azure Friday OSS March 2017 - Developing and deploying Java & Linux on...Patrick Chanezon
 
Azure Bootcamp 2016 - Docker Orchestration on Azure with Rancher
Azure Bootcamp 2016 - Docker Orchestration on Azure with RancherAzure Bootcamp 2016 - Docker Orchestration on Azure with Rancher
Azure Bootcamp 2016 - Docker Orchestration on Azure with RancherKarim Vaes
 
Docker - Demo on PHP Application deployment
Docker - Demo on PHP Application deployment Docker - Demo on PHP Application deployment
Docker - Demo on PHP Application deployment Arun prasath
 
Stups.io - an Open Source Cloud Framework for AWS
Stups.io - an Open Source Cloud Framework for AWSStups.io - an Open Source Cloud Framework for AWS
Stups.io - an Open Source Cloud Framework for AWSJan Löffler
 
Infrastructureascode slideshare-160331143725
Infrastructureascode slideshare-160331143725Infrastructureascode slideshare-160331143725
Infrastructureascode slideshare-160331143725miguel dominguez
 
Infrastructureascode slideshare-160331143725
Infrastructureascode slideshare-160331143725Infrastructureascode slideshare-160331143725
Infrastructureascode slideshare-160331143725MortazaJohari
 
Infrastructure as code: running microservices on AWS using Docker, Terraform,...
Infrastructure as code: running microservices on AWS using Docker, Terraform,...Infrastructure as code: running microservices on AWS using Docker, Terraform,...
Infrastructure as code: running microservices on AWS using Docker, Terraform,...Yevgeniy Brikman
 
Containers, Docker, and Microservices: the Terrific Trio
Containers, Docker, and Microservices: the Terrific TrioContainers, Docker, and Microservices: the Terrific Trio
Containers, Docker, and Microservices: the Terrific TrioJérôme Petazzoni
 
Docker presentasjon java bin
Docker presentasjon java binDocker presentasjon java bin
Docker presentasjon java binOlve Hansen
 
DCSF 19 Building Your Development Pipeline
DCSF 19 Building Your Development Pipeline  DCSF 19 Building Your Development Pipeline
DCSF 19 Building Your Development Pipeline Docker, Inc.
 

Similar to TIAD - DYI: A simple orchestrator built step by step (20)

Developing and Deploying PHP with Docker
Developing and Deploying PHP with DockerDeveloping and Deploying PHP with Docker
Developing and Deploying PHP with Docker
 
Docker Security workshop slides
Docker Security workshop slidesDocker Security workshop slides
Docker Security workshop slides
 
Weave User Group Talk - DockerCon 2017 Recap
Weave User Group Talk - DockerCon 2017 RecapWeave User Group Talk - DockerCon 2017 Recap
Weave User Group Talk - DockerCon 2017 Recap
 
Better Operations into the Cloud
Better Operations  into the CloudBetter Operations  into the Cloud
Better Operations into the Cloud
 
Adopt DevOps philosophy on your Symfony projects (Symfony Live 2011)
Adopt DevOps philosophy on your Symfony projects (Symfony Live 2011)Adopt DevOps philosophy on your Symfony projects (Symfony Live 2011)
Adopt DevOps philosophy on your Symfony projects (Symfony Live 2011)
 
Docker - Der Wal in der Kiste
Docker - Der Wal in der KisteDocker - Der Wal in der Kiste
Docker - Der Wal in der Kiste
 
Codetainer: a Docker-based browser code 'sandbox'
Codetainer: a Docker-based browser code 'sandbox'Codetainer: a Docker-based browser code 'sandbox'
Codetainer: a Docker-based browser code 'sandbox'
 
Docker Platform and Ecosystem
Docker Platform and EcosystemDocker Platform and Ecosystem
Docker Platform and Ecosystem
 
Docker Azure Friday OSS March 2017 - Developing and deploying Java & Linux on...
Docker Azure Friday OSS March 2017 - Developing and deploying Java & Linux on...Docker Azure Friday OSS March 2017 - Developing and deploying Java & Linux on...
Docker Azure Friday OSS March 2017 - Developing and deploying Java & Linux on...
 
One-Man Ops
One-Man OpsOne-Man Ops
One-Man Ops
 
Azure Bootcamp 2016 - Docker Orchestration on Azure with Rancher
Azure Bootcamp 2016 - Docker Orchestration on Azure with RancherAzure Bootcamp 2016 - Docker Orchestration on Azure with Rancher
Azure Bootcamp 2016 - Docker Orchestration on Azure with Rancher
 
Docker - Demo on PHP Application deployment
Docker - Demo on PHP Application deployment Docker - Demo on PHP Application deployment
Docker - Demo on PHP Application deployment
 
Stups.io - an Open Source Cloud Framework for AWS
Stups.io - an Open Source Cloud Framework for AWSStups.io - an Open Source Cloud Framework for AWS
Stups.io - an Open Source Cloud Framework for AWS
 
Infrastructureascode slideshare-160331143725
Infrastructureascode slideshare-160331143725Infrastructureascode slideshare-160331143725
Infrastructureascode slideshare-160331143725
 
Infrastructureascode slideshare-160331143725
Infrastructureascode slideshare-160331143725Infrastructureascode slideshare-160331143725
Infrastructureascode slideshare-160331143725
 
Infrastructure as code: running microservices on AWS using Docker, Terraform,...
Infrastructure as code: running microservices on AWS using Docker, Terraform,...Infrastructure as code: running microservices on AWS using Docker, Terraform,...
Infrastructure as code: running microservices on AWS using Docker, Terraform,...
 
Containers, Docker, and Microservices: the Terrific Trio
Containers, Docker, and Microservices: the Terrific TrioContainers, Docker, and Microservices: the Terrific Trio
Containers, Docker, and Microservices: the Terrific Trio
 
Docker presentasjon java bin
Docker presentasjon java binDocker presentasjon java bin
Docker presentasjon java bin
 
Us 17-krug-hacking-severless-runtimes
Us 17-krug-hacking-severless-runtimesUs 17-krug-hacking-severless-runtimes
Us 17-krug-hacking-severless-runtimes
 
DCSF 19 Building Your Development Pipeline
DCSF 19 Building Your Development Pipeline  DCSF 19 Building Your Development Pipeline
DCSF 19 Building Your Development Pipeline
 

More from The Incredible Automation Day

A smooth migration to Docker focusing on build pipelines - TIAD Camp Docker
A smooth migration to Docker focusing on build pipelines - TIAD Camp DockerA smooth migration to Docker focusing on build pipelines - TIAD Camp Docker
A smooth migration to Docker focusing on build pipelines - TIAD Camp DockerThe Incredible Automation Day
 
Docker in real life and in the Cloud - TIAD Camp Docker
Docker in real life and in the Cloud - TIAD Camp DockerDocker in real life and in the Cloud - TIAD Camp Docker
Docker in real life and in the Cloud - TIAD Camp DockerThe Incredible Automation Day
 
Orchestrating Docker in production - TIAD Camp Docker
Orchestrating Docker in production - TIAD Camp DockerOrchestrating Docker in production - TIAD Camp Docker
Orchestrating Docker in production - TIAD Camp DockerThe Incredible Automation Day
 
Strategy, planning and governance for enterprise deployments of containers - ...
Strategy, planning and governance for enterprise deployments of containers - ...Strategy, planning and governance for enterprise deployments of containers - ...
Strategy, planning and governance for enterprise deployments of containers - ...The Incredible Automation Day
 
Opening Keynote - TIAD Camp Microsoft Cloud Readiness
Opening Keynote - TIAD Camp Microsoft Cloud ReadinessOpening Keynote - TIAD Camp Microsoft Cloud Readiness
Opening Keynote - TIAD Camp Microsoft Cloud ReadinessThe Incredible Automation Day
 
GitLab CI Packer - TIAD Camp Microsoft Cloud Readiness
GitLab CI Packer - TIAD Camp Microsoft Cloud ReadinessGitLab CI Packer - TIAD Camp Microsoft Cloud Readiness
GitLab CI Packer - TIAD Camp Microsoft Cloud ReadinessThe Incredible Automation Day
 
Active Directory - TIAD Camp Microsoft Cloud Readiness
Active Directory - TIAD Camp Microsoft Cloud ReadinessActive Directory - TIAD Camp Microsoft Cloud Readiness
Active Directory - TIAD Camp Microsoft Cloud ReadinessThe Incredible Automation Day
 
Application Stack - TIAD Camp Microsoft Cloud Readiness
Application Stack - TIAD Camp Microsoft Cloud ReadinessApplication Stack - TIAD Camp Microsoft Cloud Readiness
Application Stack - TIAD Camp Microsoft Cloud ReadinessThe Incredible Automation Day
 
Serverless low cost analytics by Adways y Audric Guigon
Serverless low cost analytics by Adways y Audric GuigonServerless low cost analytics by Adways y Audric Guigon
Serverless low cost analytics by Adways y Audric GuigonThe Incredible Automation Day
 
Operationnal challenges behind Serverless architectures by Laurent Bernaille
Operationnal challenges behind Serverless architectures by Laurent BernailleOperationnal challenges behind Serverless architectures by Laurent Bernaille
Operationnal challenges behind Serverless architectures by Laurent BernailleThe Incredible Automation Day
 
Build chatbots with api.ai and Google cloud functions
Build chatbots with api.ai and Google cloud functionsBuild chatbots with api.ai and Google cloud functions
Build chatbots with api.ai and Google cloud functionsThe Incredible Automation Day
 

More from The Incredible Automation Day (20)

A smooth migration to Docker focusing on build pipelines - TIAD Camp Docker
A smooth migration to Docker focusing on build pipelines - TIAD Camp DockerA smooth migration to Docker focusing on build pipelines - TIAD Camp Docker
A smooth migration to Docker focusing on build pipelines - TIAD Camp Docker
 
Docker in real life and in the Cloud - TIAD Camp Docker
Docker in real life and in the Cloud - TIAD Camp DockerDocker in real life and in the Cloud - TIAD Camp Docker
Docker in real life and in the Cloud - TIAD Camp Docker
 
Orchestrating Docker in production - TIAD Camp Docker
Orchestrating Docker in production - TIAD Camp DockerOrchestrating Docker in production - TIAD Camp Docker
Orchestrating Docker in production - TIAD Camp Docker
 
Monitoring in 2017 - TIAD Camp Docker
Monitoring in 2017 - TIAD Camp DockerMonitoring in 2017 - TIAD Camp Docker
Monitoring in 2017 - TIAD Camp Docker
 
Strategy, planning and governance for enterprise deployments of containers - ...
Strategy, planning and governance for enterprise deployments of containers - ...Strategy, planning and governance for enterprise deployments of containers - ...
Strategy, planning and governance for enterprise deployments of containers - ...
 
Cluster SQL - TIAD Camp Microsoft Cloud Readiness
Cluster SQL - TIAD Camp Microsoft Cloud ReadinessCluster SQL - TIAD Camp Microsoft Cloud Readiness
Cluster SQL - TIAD Camp Microsoft Cloud Readiness
 
Build the VPC - TIAD Camp Microsoft Cloud Readiness
Build the VPC - TIAD Camp Microsoft Cloud ReadinessBuild the VPC - TIAD Camp Microsoft Cloud Readiness
Build the VPC - TIAD Camp Microsoft Cloud Readiness
 
Opening Keynote - TIAD Camp Microsoft Cloud Readiness
Opening Keynote - TIAD Camp Microsoft Cloud ReadinessOpening Keynote - TIAD Camp Microsoft Cloud Readiness
Opening Keynote - TIAD Camp Microsoft Cloud Readiness
 
Replatforming - TIAD Camp Microsoft Cloud Readiness
Replatforming - TIAD Camp Microsoft Cloud ReadinessReplatforming - TIAD Camp Microsoft Cloud Readiness
Replatforming - TIAD Camp Microsoft Cloud Readiness
 
GitLab CI Packer - TIAD Camp Microsoft Cloud Readiness
GitLab CI Packer - TIAD Camp Microsoft Cloud ReadinessGitLab CI Packer - TIAD Camp Microsoft Cloud Readiness
GitLab CI Packer - TIAD Camp Microsoft Cloud Readiness
 
Active Directory - TIAD Camp Microsoft Cloud Readiness
Active Directory - TIAD Camp Microsoft Cloud ReadinessActive Directory - TIAD Camp Microsoft Cloud Readiness
Active Directory - TIAD Camp Microsoft Cloud Readiness
 
Application Stack - TIAD Camp Microsoft Cloud Readiness
Application Stack - TIAD Camp Microsoft Cloud ReadinessApplication Stack - TIAD Camp Microsoft Cloud Readiness
Application Stack - TIAD Camp Microsoft Cloud Readiness
 
Keynote TIAD Camp Serverless
Keynote TIAD Camp ServerlessKeynote TIAD Camp Serverless
Keynote TIAD Camp Serverless
 
From AIX to Zero-ops by Pierre Baillet
From AIX to Zero-ops by Pierre BailletFrom AIX to Zero-ops by Pierre Baillet
From AIX to Zero-ops by Pierre Baillet
 
Serverless low cost analytics by Adways y Audric Guigon
Serverless low cost analytics by Adways y Audric GuigonServerless low cost analytics by Adways y Audric Guigon
Serverless low cost analytics by Adways y Audric Guigon
 
Operationnal challenges behind Serverless architectures by Laurent Bernaille
Operationnal challenges behind Serverless architectures by Laurent BernailleOperationnal challenges behind Serverless architectures by Laurent Bernaille
Operationnal challenges behind Serverless architectures by Laurent Bernaille
 
Build chatbots with api.ai and Google cloud functions
Build chatbots with api.ai and Google cloud functionsBuild chatbots with api.ai and Google cloud functions
Build chatbots with api.ai and Google cloud functions
 
Real time serverless data pipelines on AWS
Real time serverless data pipelines on AWSReal time serverless data pipelines on AWS
Real time serverless data pipelines on AWS
 
Azure functions
Azure functionsAzure functions
Azure functions
 
TIAD 2016 - Beyond windowsautomation
TIAD 2016 - Beyond windowsautomation TIAD 2016 - Beyond windowsautomation
TIAD 2016 - Beyond windowsautomation
 

Recently uploaded

ACM CHT Best Inspection Practices Kinben Innovation MIC Slideshare.pdf
ACM CHT Best Inspection Practices Kinben Innovation MIC Slideshare.pdfACM CHT Best Inspection Practices Kinben Innovation MIC Slideshare.pdf
ACM CHT Best Inspection Practices Kinben Innovation MIC Slideshare.pdfKinben Innovation Private Limited
 
Databricks Machine Learning Associate Exam Dumps 2024.pdf
Databricks Machine Learning Associate Exam Dumps 2024.pdfDatabricks Machine Learning Associate Exam Dumps 2024.pdf
Databricks Machine Learning Associate Exam Dumps 2024.pdfSkillCertProExams
 
Understanding Poverty: A Community Questionnaire
Understanding Poverty: A Community QuestionnaireUnderstanding Poverty: A Community Questionnaire
Understanding Poverty: A Community Questionnairebazilnaeem7
 
2024-05-15-Surat Meetup-Hyperautomation.pptx
2024-05-15-Surat Meetup-Hyperautomation.pptx2024-05-15-Surat Meetup-Hyperautomation.pptx
2024-05-15-Surat Meetup-Hyperautomation.pptxnitishjain2015
 
The Influence and Evolution of Mogul Press in Contemporary Public Relations.docx
The Influence and Evolution of Mogul Press in Contemporary Public Relations.docxThe Influence and Evolution of Mogul Press in Contemporary Public Relations.docx
The Influence and Evolution of Mogul Press in Contemporary Public Relations.docxMogul Press
 
Deciding The Topic of our Magazine.pptx.
Deciding The Topic of our Magazine.pptx.Deciding The Topic of our Magazine.pptx.
Deciding The Topic of our Magazine.pptx.bazilnaeem7
 
ServiceNow CIS-Discovery Exam Dumps 2024
ServiceNow CIS-Discovery Exam Dumps 2024ServiceNow CIS-Discovery Exam Dumps 2024
ServiceNow CIS-Discovery Exam Dumps 2024SkillCertProExams
 
Microsoft Fabric Analytics Engineer (DP-600) Exam Dumps 2024.pdf
Microsoft Fabric Analytics Engineer (DP-600) Exam Dumps 2024.pdfMicrosoft Fabric Analytics Engineer (DP-600) Exam Dumps 2024.pdf
Microsoft Fabric Analytics Engineer (DP-600) Exam Dumps 2024.pdfSkillCertProExams
 
Breathing in New Life_ Part 3 05 22 2024.pptx
Breathing in New Life_ Part 3 05 22 2024.pptxBreathing in New Life_ Part 3 05 22 2024.pptx
Breathing in New Life_ Part 3 05 22 2024.pptxFamilyWorshipCenterD
 
DAY 0 8 A Revelation 05-19-2024 PPT.pptx
DAY 0 8 A Revelation 05-19-2024 PPT.pptxDAY 0 8 A Revelation 05-19-2024 PPT.pptx
DAY 0 8 A Revelation 05-19-2024 PPT.pptxFamilyWorshipCenterD
 

Recently uploaded (10)

ACM CHT Best Inspection Practices Kinben Innovation MIC Slideshare.pdf
ACM CHT Best Inspection Practices Kinben Innovation MIC Slideshare.pdfACM CHT Best Inspection Practices Kinben Innovation MIC Slideshare.pdf
ACM CHT Best Inspection Practices Kinben Innovation MIC Slideshare.pdf
 
Databricks Machine Learning Associate Exam Dumps 2024.pdf
Databricks Machine Learning Associate Exam Dumps 2024.pdfDatabricks Machine Learning Associate Exam Dumps 2024.pdf
Databricks Machine Learning Associate Exam Dumps 2024.pdf
 
Understanding Poverty: A Community Questionnaire
Understanding Poverty: A Community QuestionnaireUnderstanding Poverty: A Community Questionnaire
Understanding Poverty: A Community Questionnaire
 
2024-05-15-Surat Meetup-Hyperautomation.pptx
2024-05-15-Surat Meetup-Hyperautomation.pptx2024-05-15-Surat Meetup-Hyperautomation.pptx
2024-05-15-Surat Meetup-Hyperautomation.pptx
 
The Influence and Evolution of Mogul Press in Contemporary Public Relations.docx
The Influence and Evolution of Mogul Press in Contemporary Public Relations.docxThe Influence and Evolution of Mogul Press in Contemporary Public Relations.docx
The Influence and Evolution of Mogul Press in Contemporary Public Relations.docx
 
Deciding The Topic of our Magazine.pptx.
Deciding The Topic of our Magazine.pptx.Deciding The Topic of our Magazine.pptx.
Deciding The Topic of our Magazine.pptx.
 
ServiceNow CIS-Discovery Exam Dumps 2024
ServiceNow CIS-Discovery Exam Dumps 2024ServiceNow CIS-Discovery Exam Dumps 2024
ServiceNow CIS-Discovery Exam Dumps 2024
 
Microsoft Fabric Analytics Engineer (DP-600) Exam Dumps 2024.pdf
Microsoft Fabric Analytics Engineer (DP-600) Exam Dumps 2024.pdfMicrosoft Fabric Analytics Engineer (DP-600) Exam Dumps 2024.pdf
Microsoft Fabric Analytics Engineer (DP-600) Exam Dumps 2024.pdf
 
Breathing in New Life_ Part 3 05 22 2024.pptx
Breathing in New Life_ Part 3 05 22 2024.pptxBreathing in New Life_ Part 3 05 22 2024.pptx
Breathing in New Life_ Part 3 05 22 2024.pptx
 
DAY 0 8 A Revelation 05-19-2024 PPT.pptx
DAY 0 8 A Revelation 05-19-2024 PPT.pptxDAY 0 8 A Revelation 05-19-2024 PPT.pptx
DAY 0 8 A Revelation 05-19-2024 PPT.pptx
 

TIAD - DYI: A simple orchestrator built step by step

  • 1. DIY: A simple orchestrator built step by step TIAD PARIS 2015
  • 2. ChallengesHi, I’m Ahmet Demir Also known as @ahmet2mir Opensource Enthusiast @D2SI
  • 6. When we try to assemble them http://gothicpug.deviantart.com/art/Wolverine-s-lego-cy-wtf-140295823
  • 8. What we orchestrate ? IT Process: disaster recovery, restart process Infrastructure: Build a VM, upgrade it Application Deployment: Deploy an application on x nodes Developpements: Continuous integration, Continuous Delivery
  • 10. Eliminate "server management deployment" FusionDirectory - http://www.fusiondirectory.org m23 - http://m23.sourceforge.net/PostNuke-0.750/html/index.php Fabric - http://www.fabfile.org/ Archipel - http://archipelproject.org/ AppScale - https://github.com/AppScale/appscale OpenBuild - http://openbuildservice.org/ CompatibleOne - http://compatibleone.org/ Opsi - http://www.opsi.org
  • 11. Eliminate config manager Puppet - https://puppetlabs.com/ Chef - https://chef.io SaltStack - http://saltstack.com/ Ansible - http://www.ansible.com/home CFengine - http://cfengine.com/
  • 12. Eliminate "server task oriented" Rundeck - http://rundeck.org/
  • 13. Eliminate Only Cloud oriented Eucalyptus - https://github.com/eucalyptus/eucalyptus/wiki Openstack Heat - https://github.com/openstack/heat
  • 14. Finally 1/3 Terraform - https://terraform.io/ Terraform provides a flexible abstraction of resources and providers. This model allows for representing everything from physical hardware, virtual machines, and containers, to email and DNS providers #Go
  • 15. Finally 2/3 Salt Cloud - https://salt-cloud.readthedocs.org Salt cloud is a public cloud provisioning tool. Salt cloud is made to integrate Salt into cloud providers in a clean way so that minions on public cloud systems can be quickly and easily modeled and provisioned. #Python
  • 16. Finally 3/3 OpenScore - http://openscore.github.io/ Flow-based orchestration engine to manage deployed applications Allows you to rapidly automate your DevOps use cases using ready-made workflows. Based on Java and HP OO engine. #Java
  • 17. OpenStack Heat based: Resource based, CRUD, Graph dependency, describe in Yaml Terraform: Execution Plan, Diff on infra and all heat features, describe in “json like” format Salt Cloud: Easy modules integration but no plan, no graph OpenScore: Service in Yaml, no plan but runbook featured Features
  • 18. API access ? How to scale ? If I had more than 100 requests in same time ? Where can I store all my services ? How can I track ? Is modules integration easy ? What about lifecycle ? Versioning the service ? Multi-user ? Organization > 100 peoples ? High Availability ? But Try to solve or ...
  • 20.
  • 21. Simple to maintain Think tomorrow Infrastructure agnostic Easy development Don't forget Windows users :) Fundamentals
  • 22. NetworkStorage Compute Orchestrator Referential Config Manager Catalog HP MP, OpenCart... Monitoring Nagios... CI / CD Jenkins, Bamboo ... Service Designer Big Picture
  • 23. Engine API Resource runner Resource 1 Resource 2 Resource 3 Resource 4 Run 1 & 2 in parallel Then run 3 Then run 4 Service Designer & Orchestrator
  • 25. If you think to something, you will find your answer in Python Both of Devs and Ops can use Python and use it quickly No compilation and not "magic" Write less and do more Limits the number of times your brain has to do a "context switch." Helps productivity The language
  • 26. YAML is simple to read and write Many projects use it: Heat, Fig etc... Service Description: YAML
  • 27. Many tasks management Easy to scale Dedicate queue to some tasks (DMZ, Windows...) Asynchronous task queue/job Real-time operation, but supports scheduling Tasks are executed concurrently Used in production systems to process millions of tasks a day Tasks elery
  • 28. Messaging broker Common platform to send and receive messages Safe place to live until received Can hits one million messages per second Message Delivery
  • 29. rapid and agile web development No migrations Not a relational database Schemaless Database
  • 31.
  • 32. Step 0: Build workers and broker Rabbit MQ Celery 1 Celery 2 Flower GUI
  • 33. orion ~ $ git clone https://github.com/ahmet2mir/tiad-demo.git demo orion ~ $ cd demo orion demo $ cd 00_celery_rabbit # start celery orion 00_celery_rabbit $ fig -f celery.yml -p demo up -d # start rabbitmq orion 00_celery_rabbit $ fig -f rabbitmq.yml -p demo up -d # create rabbitmq user orion 00_celery_rabbit $ docker exec -t demo_broker_1 rabbitmqctl add_user myuser mypassword orion 00_celery_rabbit $ docker exec -t demo_broker_1 rabbitmqctl add_vhost myvhost orion 00_celery_rabbit $ docker exec -t demo_broker_1 rabbitmqctl set_permissions -p myvhost myuser ".*" ".*" ".*" # gui for flower and rabbitmq orion 00_celery_rabbit $ fig -f flower.yml -p demo up -d # Visit http://localhost:5555 Step 0: Demo
  • 34. Step 1: Execute a simple resource Rabbit MQ Celery 1 Celery 2 AWS Instance Resource add in queue Flower GUI
  • 35. Celery 1 Wrapper AWS Wrapper Docker Docker CRUD AWS CRUD Pure Python Package deployed with pip Wrapper to simplify execution
  • 36. orion demo $ cd 01_aws_resource # virtual python environment orion 01_aws_resource $ virtualenv . orion 01_aws_resource $ source bin/activate orion 01_aws_resource $ pip install -r requirements.txt # add credentials in client_provider.py # edit client_provider with some values (optionnal) … # run the resource orion 01_aws_resource $ python client.py Step 1: Demo
  • 37. Step 2: Store outside Rabbit MQ Celery 1 Celery 2 Flower GUI MongoDBApplication
  • 38. orion demo $ cd 02_db_provider # virtual python environment orion 02_db_provider $ virtualenv . orion 02_db_provider $ source bin/activate orion 02_db_provider $ pip install -r requirements.txt # start mongodb orion 02_db_provider $ fig -f mongodb.yml -p demo up -d # add credentials in add_*.py orion 02_db_provider $ python add_aws.py orion 02_db_provider $ python add_docker.py # visit http://localhost:8080 Step 2: Demo
  • 39. Application Step 3: Service level Rabbit MQ Celery 1 Celery 2 Service AWS Instance Resource Flower GUI Docker Resource 1 2 MongoDB
  • 40. orion demo $ cd 03_app # virtual python environment orion 03_app $ virtualenv . orion 03_app $ source bin/activate orion 03_app $ pip install -r requirements.txt # run the service orion 03_app $ python teecli.py create sample01.yml Step 3: Demo
  • 41. Commit API Commit Admin GUI Commit Referential Management Commit all modules and wrappers Build a service catalog Role based access to the app http://teerex.io (soon) https://github.com/ahmet2mir/teerex (soon) Next steps