Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Project COLA: Use Case to
create a scalable application in
the cloud based on the MICADO
concept
Peter Kacsuk
SZTAKI
kacsu...
Introduction to the application
The application is Data Avenue:
• This application can
• transfer data from one data stora...
Data Avenue services
3
FS1 FSn
Data Avenue Blacktop service
Openstack Amazon
Data Avenue
@ SZTAKI
Java or
other appli
code...
Introduction to the application
Detailed objectives:
• We want to create a scalable service out of this application. It
me...
The Data Avenue service
How to use Data Avenue?
How to use Data Avenue?
Use-case1: DataAvenue@SZTAKI
> Select storage type (Protocol)
> Set storage host address (URL)
> Press Go
Use-case1: DataAvenue@SZTAKI
> Select authentication method
> Give credentials
> Press Go
The Data Avenue Concept
Copy between different storages
Progress bar
Directory contents: files,
directories
Problems with the current service
• It uses a single Data Avenue application and it becomes a
bottleneck in the following ...
Solution
• The Data Avenue application can run in several instances in parallel in
the cloud
• If a single Data Avenue (DA...
Scale up
Scale down
Automatic scalable architecture for Data Avenue
More details for monitoring
Alert
Manager
Executor
Available as tutorial at the Occopus web page: http://occopus.lpds.szta...
CPU Usage
Number of VMs
15/14
Demonstrating scalability
0
1
2
3
4
5
6
7
8
9
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37
...
Limitations
• It uses VMs for every service and hence the deployment time
of the whole infrastructure is 8-10 minutes in C...
Docker container based architecture
• In order to reduce deployment time we placed every service in
a separate docker cont...
General architecture adaptable for
other applications -> MICADO v0
• If the DA description is replaced by another service-...
Towards a more generic MICADO
architecture -> MICADO v1
• Goal:
• create a MICADO architecture in which the application ca...
2-layer MICADO v1/a architecture
Swarm
docker
cluster
Assessment of MICADO v1/a
Advantage:
• Adaptable for different applications without modifying the cloud-init file
• Scales...
3-layer MICADO v1/b architecture
Assessment of MICADO v1/b
Advantage:
• Adaptable for different applications without modifying the cloud-init file
• Scales...
Acknowledgement
• The development of MICADO is in strong collaboration
between the SZTAKI and UoW teams.
• SZTAKI team: Pr...
For more information please visit
www.project-cola.eu
twitter.com/projectCOLA
facebook.com/projectCOLA
Thank you!
Upcoming SlideShare
Loading in …5
×

Project COLA: Use Case to create a scalable application in the cloud based on the MICADO concept

244 views

Published on

Project COLA member Peter Kacsuk presents a use case on how to create a scalable application on the MiCADO platform

Published in: Internet
  • Be the first to comment

  • Be the first to like this

Project COLA: Use Case to create a scalable application in the cloud based on the MICADO concept

  1. 1. Project COLA: Use Case to create a scalable application in the cloud based on the MICADO concept Peter Kacsuk SZTAKI kacsuk@sztaki.hu
  2. 2. Introduction to the application The application is Data Avenue: • This application can • transfer data from one data storage to another one • Upload data from your computer to a data storage • Download data from a data storage to your computer • The data storage can be any of the following type: • HTTP, HTTPS, SFTP, GSIFTP, SRM, iRODS and S3
  3. 3. Data Avenue services 3 FS1 FSn Data Avenue Blacktop service Openstack Amazon Data Avenue @ SZTAKI Java or other appli code (JAVA API or WS API) Data Avenue Portlet gateways FS2OpenNebula storage-related protocols web service interface
  4. 4. Introduction to the application Detailed objectives: • We want to create a scalable service out of this application. It means that users from all over the world can access this service in order to transfer their data from one data storage to another one provided that they have got access right to the used data storages. • The current service can be accessed at: • https://data-avenue.eu/en_GB/
  5. 5. The Data Avenue service
  6. 6. How to use Data Avenue?
  7. 7. How to use Data Avenue?
  8. 8. Use-case1: DataAvenue@SZTAKI > Select storage type (Protocol) > Set storage host address (URL) > Press Go
  9. 9. Use-case1: DataAvenue@SZTAKI > Select authentication method > Give credentials > Press Go
  10. 10. The Data Avenue Concept Copy between different storages Progress bar Directory contents: files, directories
  11. 11. Problems with the current service • It uses a single Data Avenue application and it becomes a bottleneck in the following cases: • One user initiates many transfers in parallel • Many users initiate single transfers in parallel • Many users initiate many transfers in parallel
  12. 12. Solution • The Data Avenue application can run in several instances in parallel in the cloud • If a single Data Avenue (DA) becomes bottleneck new DA is instantiated in the cloud and some of the required data transfers are automatically re- directed to the new DA instance • In order to organize all the services as a set of coordinated services we need an information service => Consul service • In order to observe if a data instance becomes overloaded we need a monitoring tool => Prometheus service • In order to deploy and manage all the services as a set of coordinated services we need a cloud orchestrator service => Occopus service • In order to evenly exploit the DA instances we need a load balancer service that directs the users’ DA service calls evenly to the available DA instances => haproxy service
  13. 13. Scale up Scale down Automatic scalable architecture for Data Avenue
  14. 14. More details for monitoring Alert Manager Executor Available as tutorial at the Occopus web page: http://occopus.lpds.sztaki.hu/tutorials Soon will be available as tutorial at the COLA web page
  15. 15. CPU Usage Number of VMs 15/14 Demonstrating scalability
  16. 16. 0 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 numberofvirtualmachines Measure every 40 minute, starting at 8:00 am. One-day long test bench 16/14
  17. 17. Limitations • It uses VMs for every service and hence the deployment time of the whole infrastructure is 8-10 minutes in CloudSigma • Good scalability performance results can be achieved only if the used storage has a scalable access mechanism. For example, S3 cloud storage enables scalability but https does not.
  18. 18. Docker container based architecture • In order to reduce deployment time we placed every service in a separate docker container. • The modification required: • Dockerized version of all the services (HAProxy, Prometheus, Consul) • Dockerized version of the data node application (DA) • Modification of the cloud-init file that contains the description of cloud- related parameters (cloud-init file became much simpler)
  19. 19. General architecture adaptable for other applications -> MICADO v0 • If the DA description is replaced by another service-oriented application in the cloud-init file, then it can be used for other applications, too. • Therefore this architecture can be considered as the first version of the MICADO architecture -> MICADO v0
  20. 20. Towards a more generic MICADO architecture -> MICADO v1 • Goal: • create a MICADO architecture in which the application can be accommodated without modifying cloud-init • To achieve this we introduce a docker cluster and the applications will run within in docker containers within this docker cluster • Therefore this architecture can be considered as the improved version of the MICADO v0 architecture -> MICADO v1 • MICADO v1 could have to alternatives: • A. 2-layered MICADO architecture without load balancer layer • B. 3-layered MICADO architecture with load balancer layer
  21. 21. 2-layer MICADO v1/a architecture Swarm docker cluster
  22. 22. Assessment of MICADO v1/a Advantage: • Adaptable for different applications without modifying the cloud-init file • Scales up and down the number of data nodes according to the actual load of the data nodes • Guarantees the balanced usage of data nodes (responsibility of the Docker Master) Disadvantage: • No guarantee that the data nodes are used in a balanced way to serve user requests • All the data nodes should run the same applications
  23. 23. 3-layer MICADO v1/b architecture
  24. 24. Assessment of MICADO v1/b Advantage: • Adaptable for different applications without modifying the cloud-init file • Scales up and down the number of data nodes according to the actual load of the data nodes • Guarantees the balanced usage of data nodes (responsibility of the Docker Master) • Guarantees that the data nodes are used in a balanced way to serve user requests (responsibility of the HAProxy services) Disadvantage: • All the data nodes should run the same applications Future work: • MICADO v2 to enable the balanced and scalable usage of different applications
  25. 25. Acknowledgement • The development of MICADO is in strong collaboration between the SZTAKI and UoW teams. • SZTAKI team: Prof. Peter Kacsuk • Dr. Jozsef Kovacs • Enikő Nagy • Attila Farkas • Dr. Robert Lovas • Dr. Zoltan Farkas • UoW team: Dr. Tamas Kiss • Prof. Gabor Terstyanszky • Botond Rákoczi • Gregoire Gesmier • Gabriele Pierantoni
  26. 26. For more information please visit www.project-cola.eu twitter.com/projectCOLA facebook.com/projectCOLA Thank you!

×