Your SlideShare is downloading. ×
Docker at flux7 - Docker meetup kickoff
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Saving this for later?

Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime - even offline.

Text the download link to your phone

Standard text messaging rates apply

Docker at flux7 - Docker meetup kickoff

221
views

Published on

Published in: Technology

0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
221
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
17
Comments
0
Likes
3
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. DOCKER @ FLUX7 LABS
  • 2. WHO AM I? Aater Suleman Part-time UT Professor Geek, Architect, Developer, Ops, DevOps … Co-founder & CEO Flux7 Labs ○ DevOps Solutions ■ Deployments ■ Cost/performance optimized large scale website (Ruby on rails, node.js, Django) and Hadoop deployments
  • 3. VyScale Dev Flow Docker's impact on performance (whitepaper WIP) Multi-tenancy Live process migration using CRIU (criu.org) Four projects:
  • 4. APPLICATION: SINGLE SERVICE PROVIDER Receive Sensor Data Report Generation based on data Report sent to End User Internet of Things -- Solar Panel Monitoring XML Data over TCP Big Data Analytics
  • 5. Single Provider System Provider Span Location1 Location2 Location3 Gateway1 Gateway2 Gateway3sensors A provider has Mifi routers installed at multiple Locations which collect data from sensors and sends it to a remote TCP server via the internet. TCP server Port 6000 Cassandra port 9160 Flask App Port 80 Browser Uses the Flask app at port 80Internet
  • 6. COMPONENTS 1. Cassandra for data persistence which we later use for generating reports for each gateway. 2. A Twisted TCP server listening at PORT 6000, for data ingestion from multiple gateways owned by the provider. 3. A Flask app serving at PORT 80 as the admin panel for setting customizations and viewing reports.
  • 7. G G G G Customer NCustomer 2 …Customer 1 Each customer can have multiple gateways commissioned to them. Remote Twister TCP Server (Non–Blocking I/O) Cassandra NoSQL data store (High Volume High Velocity Write which scales Linearly across the cluster ) Power consumption status on website and mails *G - Gateway Web App Mailer
  • 8. SINGLE PROVIDER LAUNCH For launching the single provider version, the following was done: 1. nohup python tcp_server.py & # For firing up the TCP server. 2. nohup python flask_app.py & # For firing up the admin panel Both these code bases houses hard-coded Cassandra KEYSPACE
  • 9. APPLICATION: MULTIPLE SERVICE PROVIDERS …
  • 10. Provider 1 sends data to port 6001 and accesses flask app at port 8081 Provider 2 sends data to port 6002 and accesses flask app at port 8082 Flask container-runs flask app at port 80. Exposes port 80 and published it to port 8081 for provider 1 Flask container-runs flask app at port 80. Exposes port 80 and published it to port 8082 for provider 2 TCP server container- runs at port 6000. Exposes port 6000 and published it to port 6001 for provider 1 TCP server container-runs at port 6000. Exposes port 6000 and published it to port 6002 for provider 2 Cassandra Internet
  • 11. KNEE-JERK APPROACH Sprinkle Tenant ID everywhere in the code and DB Time consuming Expensive Security Maintenance Rigidity Poor isolation
  • 12. An alternate solution is to use Virtual Machine (VM) or Multiple Hosts Downside: MULTIPLE HOSTS/VMS Both VMs and Multiple Hosts cost a lot of money
  • 13. AND THE SOLUTION
  • 14. How: Isolated environments for running multiple instances of the app WHY DOCKER? Docker containers provide isolation that is Fast Inexpensive
  • 15. Create a docker container for the new version of the app PLAN Setup environments/dependencies correctly Start a Cassandra container.
  • 16. # start a docker container for consuming gateway data at gateway_port start_command = 'python software/remote_server.py ' + provider_id remote_server = docker_client.create_container('flux7/labs', # docker image command=start_command, # start command contains the keyspace parameter, keyspace is the provider_id name='remote_server_' + provider_id, # name the container, name is provider_id ports=[(6000, 'tcp'),]) # open port for binding, remote_server.py listens at 6000 docker_client.start(remote_server, port_bindings={6000: ('0.0.0.0', gateway_port)}, links={'db': 'cassandra'}) AUTOMATION # start a docker container for serving admin panel at admin_port start_command = 'python software/flask_app.py ' + provider_id remote_server = docker_client.create_container('flux7/labs', # docker image command=start_command, # start command contains the keyspace parameter, keyspace is the provider_id name='admin_panel_' + provider_id, # name the container, name is provider_id ports=[(80, 'tcp'),]) # open port for binding, remote_server.py listens at 6000 docker_client.start(remote_server, port_bindings={80: ('0.0.0.0',admin_port)}, links= {'db': 'cassandra'}) An automation was the next foreseeable step, and for that we found Docker-py extremely useful. We used something like: # Yes. We love Python! def start_provider(provider_id, gateway_port, admin_port ): docker_client = docker.Client(base_url='unix://var/run/docker.sock', version='1.6', timeout=100)
  • 17. For now, a locally running container serving at PORT 9160 using the command similar to this: docker run -d -p 9160:9160 -name db flux7/cassandra OUR SOLUTION- EXPLAINED
  • 18. Create a keyspace ‘provider1’ using pycassaShell. We fired up our two code bases on two separate containers like this: OUR SOLUTION- EXPLAINED docker run -name remote_server_1 -link db:cassandra - p 6001:6000 flux7/labs python software/remote_server. py provider1 docker run -name flask_app_1 -link db:cassandra -p 8081:80 flux7/labs python software/flask_app.py provider1
  • 19. DOCKER ISSUES DISCOVERED Docker does not support multiple instances of Cassandra running on the same machine. Hosting multiple database instances on a single machine can quickly cause resource shortages
  • 20. ❑ Followed the traditional solution to make an application multi-tenant OUR SOLUTION Code Changes •• To data ingestion server and web server by adding the keyspace parameter to the DB accesses. Cassandra KEYSPACE / provider ID •• Passed to each instance of the app on the command line. ❑ Each provider in the data store gets a separate namespace without making any changes to the column family schema. Use of KEYSPACE as the namespace for each provider in the data store
  • 21. LESSONS WE LEARNT Docker is an extremely fast and elegant isolation framework: easy to port, cheap to run, easy to orchestrate Multi-tenancy != changing the app to support multiple tenants Docker orchestration frameworks are not at par with Docker today. What we have written is yet another one but for multi-tenancy. Dockerfiles still need work -- we used shell scripts in some places We can run multiple commands/container
  • 22. DOCKER IN DEVELOPER ENVIRONMENT 4 Reasons Performance Overhead Fast Boot Container Size is small Ability to put on top of cloud (and even cross cloud)
  • 23. DOCKER AUSTIN FOR MORE DISCUSSIONS ON DOCKER!