DOCKER @ FLUX7 LABS
WHO AM I?
Aater Suleman
Part-time UT Professor
Geek, Architect, Developer, Ops, DevOps …
Co-founder & CEO Flux7 Labs
DevOp...
RESOURCES
Blog: http://flux7.com/blogs/category/docker/

Slideshare: http://www.slideshare.net/AaterSuleman

Twitter: @fut...
WHY WE USE DOCKER?
Better than running processes

Low CPU/memory overhead

The ability to diff containers
- Easier to shar...
HOW DO WE FEEL?

Enough said!

Read the blog post here

The aforementioned reasons are real -- very real
Four N projects:

Dev Flow projects: VyScale, flux7.com, Clients
Docker's impact on performance (whitepaper WIP)
Multi-ten...
APPLICATION: SINGLE SERVICE
PROVIDER
Internet of Things -- Solar Panel Monitoring

Receive
Sensor Data

XML Data
over TCP
...
Gateway3

sensors
Gateway1

Gateway2

Location1

Location3

Location2

Provider Span
A provider has Mifi routers installed...
COMPONENTS

1. Cassandra for data persistence which we later use for generating reports for
each gateway.

2. A Twisted TC...
Customer 1

G

Customer 2 …

Customer N

G

G

G

Each customer can have
multiple gateways
commissioned to them.

*G - Gat...
SINGLE PROVIDER LAUNCH
For launching the single provider version, the following was done:

1. nohup python tcp_server.py &...
APPLICATION: MULTIPLE SERVICE
PROVIDERS

…
Provider 2 sends data to port
6002 and accesses flask app at
port 8082

Provider 1 sends
data to port 6001
and accesses fl...
KNEE-JERK APPROACH
Sprinkle Tenant ID everywhere in the code and DB

Time consuming

Expensive

Poor isolation

Security

...
MULTIPLE HOSTS/VMS

An alternate solution is to use Virtual Machine (VM) or Multiple Hosts

Downside:

Both VMs and Multip...
AND THE SOLUTION
WHY DOCKER?

Docker containers provide isolation that is
Fast

Inexpensive

How: Isolated environments for running multipl...
PLAN

Create a docker container for the new version of the app

Setup environments/dependencies correctly

Start a Cassand...
AUTOMATION
An automation was the next foreseeable step, and for that we found Docker-py
extremely useful. We used somethin...
OUR SOLUTION- EXPLAINED

For now, a locally running container serving at PORT 9160 using the command
similar to this:
dock...
OUR SOLUTION- EXPLAINED
Create a keyspace ‘provider1’ using pycassaShell.
We fired up our two code bases on two separate c...
DOCKER ISSUES DISCOVERED

Docker does not support multiple instances of Cassandra running on the
same machine.

Hosting mu...
OUR SOLUTION
❑ Followed the traditional solution to make an application multi-tenant
Use of KEYSPACE as the namespace for ...
LESSONS WE LEARNT
Docker is an extremely fast and elegant isolation framework: easy to port,
cheap to run, easy to orchest...
DOCKER IN DEVELOPER
ENVIRONMENT
4 Reasons
Performance Overhead

Fast Boot

Container Size is small

Ability to put on top ...
DOCKER AUSTIN

MARCH 13TH, 6PM @ RACKSPACE
DOCKER AND DEV FLOW OPTIMIZATION
Upcoming SlideShare
Loading in...5
×

Docker @ Flux7 Labs

3,574

Published on

Link to Blog Post: http://flux7.com/blogs/docker/docker-saves-the-day-at-flux7/

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
3,574
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
15
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Docker @ Flux7 Labs

  1. 1. DOCKER @ FLUX7 LABS
  2. 2. WHO AM I? Aater Suleman Part-time UT Professor Geek, Architect, Developer, Ops, DevOps … Co-founder & CEO Flux7 Labs DevOps Solutions ■ Deployments ■ Cost/performance optimized large scale website (Ruby on rails, node.js, Django) and Hadoop deployments
  3. 3. RESOURCES Blog: http://flux7.com/blogs/category/docker/ Slideshare: http://www.slideshare.net/AaterSuleman Twitter: @futurechips Email: aater@flux7.com
  4. 4. WHY WE USE DOCKER? Better than running processes Low CPU/memory overhead The ability to diff containers - Easier to share containers than VMs Fast boot/shutdown Cross cloud/infrastructure: The ability to put on top of cloud
  5. 5. HOW DO WE FEEL? Enough said! Read the blog post here The aforementioned reasons are real -- very real
  6. 6. Four N projects: Dev Flow projects: VyScale, flux7.com, Clients Docker's impact on performance (whitepaper WIP) Multi-tenancy Live process migration using CRIU (criu.org) Wordpress hosting
  7. 7. APPLICATION: SINGLE SERVICE PROVIDER Internet of Things -- Solar Panel Monitoring Receive Sensor Data XML Data over TCP Report Generation based on data Big Data Analytics Report sent to End User
  8. 8. Gateway3 sensors Gateway1 Gateway2 Location1 Location3 Location2 Provider Span A provider has Mifi routers installed at multiple Locations which collect data from sensors and sends it to a remote TCP server via the internet. Browser Uses the Flask app at port 80 Internet Flask App Port 80 TCP server Port 6000 Cassandra port 9160 Single Provider System
  9. 9. COMPONENTS 1. Cassandra for data persistence which we later use for generating reports for each gateway. 2. A Twisted TCP server listening at PORT 6000, for data ingestion from multiple gateways owned by the provider. 3. A Flask app serving at PORT 80 as the admin panel for setting customizations and viewing reports.
  10. 10. Customer 1 G Customer 2 … Customer N G G G Each customer can have multiple gateways commissioned to them. *G - Gateway Remote Twister TCP Server (Non–Blocking I/O) Web App Power consumption status on website and mails Mailer Cassandra NoSQL data store (High Volume High Velocity Write which scales Linearly across the cluster )
  11. 11. SINGLE PROVIDER LAUNCH For launching the single provider version, the following was done: 1. nohup python tcp_server.py & # For firing up the TCP server. 2. nohup python flask_app.py & # For firing up the admin panel Both these code bases houses hard-coded Cassandra KEYSPACE
  12. 12. APPLICATION: MULTIPLE SERVICE PROVIDERS …
  13. 13. Provider 2 sends data to port 6002 and accesses flask app at port 8082 Provider 1 sends data to port 6001 and accesses flask app at port 8081 Internet TCP server containerruns at port 6000. Exposes port 6000 and published it to port 6001 for provider 1 Flask container-runs flask app at port 80. Exposes port 80 and published it to port 8081 for provider 1 TCP server container-runs at port 6000. Exposes port 6000 and published it to port 6002 for provider 2 Flask container-runs flask app at port 80. Exposes port 80 and published it to port 8082 for provider 2 Cassandra
  14. 14. KNEE-JERK APPROACH Sprinkle Tenant ID everywhere in the code and DB Time consuming Expensive Poor isolation Security Maintenance Rigidity
  15. 15. MULTIPLE HOSTS/VMS An alternate solution is to use Virtual Machine (VM) or Multiple Hosts Downside: Both VMs and Multiple Hosts cost a lot of money
  16. 16. AND THE SOLUTION
  17. 17. WHY DOCKER? Docker containers provide isolation that is Fast Inexpensive How: Isolated environments for running multiple instances of the app
  18. 18. PLAN Create a docker container for the new version of the app Setup environments/dependencies correctly Start a Cassandra container.
  19. 19. AUTOMATION An automation was the next foreseeable step, and for that we found Docker-py extremely useful. We used something like: # Yes. We love Python! def start_provider(provider_id, gateway_port, admin_port ): docker_client = docker.Client(base_url='unix://var/run/docker.sock', version='1.6', timeout=100) # start a docker container for consuming gateway data at gateway_port start_command = 'python software/remote_server.py ' + provider_id remote_server = docker_client.create_container('flux7/labs', # docker image command=start_command, # start command contains the keyspace parameter, keyspace is the provider_id name='remote_server_' + provider_id, # name the container, name is provider_id ports=[(6000, 'tcp'),]) # open port for binding, remote_server.py listens at 6000 docker_client.start(remote_server, port_bindings={6000: ('0.0.0.0', gateway_port)}, links={'db': 'cassandra'}) # start a docker container for serving admin panel at admin_port start_command = 'python software/flask_app.py ' + provider_id remote_server = docker_client.create_container('flux7/labs', # docker image command=start_command, # start command contains the keyspace parameter, keyspace is the provider_id name='admin_panel_' + provider_id, # name the container, name is provider_id ports=[(80, 'tcp'),]) # open port for binding, remote_server.py listens at 6000 docker_client.start(remote_server, port_bindings={80: ('0.0.0.0',admin_port)}, links= {'db': 'cassandra'})
  20. 20. OUR SOLUTION- EXPLAINED For now, a locally running container serving at PORT 9160 using the command similar to this: docker run -d -p 9160:9160 -name db flux7/cassandra
  21. 21. OUR SOLUTION- EXPLAINED Create a keyspace ‘provider1’ using pycassaShell. We fired up our two code bases on two separate containers like this: docker run -name remote_server_1 -link db:cassandra p 6001:6000 flux7/labs python software/remote_server. py provider1 docker run -name flask_app_1 -link db:cassandra -p 8081:80 flux7/labs python software/flask_app.py provider1
  22. 22. DOCKER ISSUES DISCOVERED Docker does not support multiple instances of Cassandra running on the same machine. Hosting multiple database instances on a single machine can quickly cause resource shortages
  23. 23. OUR SOLUTION ❑ Followed the traditional solution to make an application multi-tenant Use of KEYSPACE as the namespace for each provider in the data store Code Changes Cassandra KEYSPACE / provider ID •• To data ingestion server and web server by adding the keyspace parameter to the DB accesses. •• Passed to each instance of the app on the command line. ❑ Each provider in the data store gets a separate namespace without making any changes to the column family schema.
  24. 24. LESSONS WE LEARNT Docker is an extremely fast and elegant isolation framework: easy to port, cheap to run, easy to orchestrate Multi-tenancy != changing the app to support multiple tenants Docker orchestration frameworks are not at par with Docker today. What we have written is yet another one but for multi-tenancy. Dockerfiles still need work -- we used shell scripts in some places We can run multiple commands/container
  25. 25. DOCKER IN DEVELOPER ENVIRONMENT 4 Reasons Performance Overhead Fast Boot Container Size is small Ability to put on top of cloud (and even cross cloud)
  26. 26. DOCKER AUSTIN MARCH 13TH, 6PM @ RACKSPACE DOCKER AND DEV FLOW OPTIMIZATION
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×