Successfully reported this slideshow.
Your SlideShare is downloading. ×

Project COLA: Use Case to create a scalable application in the cloud based on the MICADO concept

Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Loading in …3
×

Check these out next

1 of 26 Ad
Advertisement

More Related Content

Slideshows for you (20)

Similar to Project COLA: Use Case to create a scalable application in the cloud based on the MICADO concept (20)

Advertisement

Recently uploaded (20)

Advertisement

Project COLA: Use Case to create a scalable application in the cloud based on the MICADO concept

  1. 1. Project COLA: Use Case to create a scalable application in the cloud based on the MICADO concept Peter Kacsuk SZTAKI kacsuk@sztaki.hu
  2. 2. Introduction to the application The application is Data Avenue: • This application can • transfer data from one data storage to another one • Upload data from your computer to a data storage • Download data from a data storage to your computer • The data storage can be any of the following type: • HTTP, HTTPS, SFTP, GSIFTP, SRM, iRODS and S3
  3. 3. Data Avenue services 3 FS1 FSn Data Avenue Blacktop service Openstack Amazon Data Avenue @ SZTAKI Java or other appli code (JAVA API or WS API) Data Avenue Portlet gateways FS2OpenNebula storage-related protocols web service interface
  4. 4. Introduction to the application Detailed objectives: • We want to create a scalable service out of this application. It means that users from all over the world can access this service in order to transfer their data from one data storage to another one provided that they have got access right to the used data storages. • The current service can be accessed at: • https://data-avenue.eu/en_GB/
  5. 5. The Data Avenue service
  6. 6. How to use Data Avenue?
  7. 7. How to use Data Avenue?
  8. 8. Use-case1: DataAvenue@SZTAKI > Select storage type (Protocol) > Set storage host address (URL) > Press Go
  9. 9. Use-case1: DataAvenue@SZTAKI > Select authentication method > Give credentials > Press Go
  10. 10. The Data Avenue Concept Copy between different storages Progress bar Directory contents: files, directories
  11. 11. Problems with the current service • It uses a single Data Avenue application and it becomes a bottleneck in the following cases: • One user initiates many transfers in parallel • Many users initiate single transfers in parallel • Many users initiate many transfers in parallel
  12. 12. Solution • The Data Avenue application can run in several instances in parallel in the cloud • If a single Data Avenue (DA) becomes bottleneck new DA is instantiated in the cloud and some of the required data transfers are automatically re- directed to the new DA instance • In order to organize all the services as a set of coordinated services we need an information service => Consul service • In order to observe if a data instance becomes overloaded we need a monitoring tool => Prometheus service • In order to deploy and manage all the services as a set of coordinated services we need a cloud orchestrator service => Occopus service • In order to evenly exploit the DA instances we need a load balancer service that directs the users’ DA service calls evenly to the available DA instances => haproxy service
  13. 13. Scale up Scale down Automatic scalable architecture for Data Avenue
  14. 14. More details for monitoring Alert Manager Executor Available as tutorial at the Occopus web page: http://occopus.lpds.sztaki.hu/tutorials Soon will be available as tutorial at the COLA web page
  15. 15. CPU Usage Number of VMs 15/14 Demonstrating scalability
  16. 16. 0 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 numberofvirtualmachines Measure every 40 minute, starting at 8:00 am. One-day long test bench 16/14
  17. 17. Limitations • It uses VMs for every service and hence the deployment time of the whole infrastructure is 8-10 minutes in CloudSigma • Good scalability performance results can be achieved only if the used storage has a scalable access mechanism. For example, S3 cloud storage enables scalability but https does not.
  18. 18. Docker container based architecture • In order to reduce deployment time we placed every service in a separate docker container. • The modification required: • Dockerized version of all the services (HAProxy, Prometheus, Consul) • Dockerized version of the data node application (DA) • Modification of the cloud-init file that contains the description of cloud- related parameters (cloud-init file became much simpler)
  19. 19. General architecture adaptable for other applications -> MICADO v0 • If the DA description is replaced by another service-oriented application in the cloud-init file, then it can be used for other applications, too. • Therefore this architecture can be considered as the first version of the MICADO architecture -> MICADO v0
  20. 20. Towards a more generic MICADO architecture -> MICADO v1 • Goal: • create a MICADO architecture in which the application can be accommodated without modifying cloud-init • To achieve this we introduce a docker cluster and the applications will run within in docker containers within this docker cluster • Therefore this architecture can be considered as the improved version of the MICADO v0 architecture -> MICADO v1 • MICADO v1 could have to alternatives: • A. 2-layered MICADO architecture without load balancer layer • B. 3-layered MICADO architecture with load balancer layer
  21. 21. 2-layer MICADO v1/a architecture Swarm docker cluster
  22. 22. Assessment of MICADO v1/a Advantage: • Adaptable for different applications without modifying the cloud-init file • Scales up and down the number of data nodes according to the actual load of the data nodes • Guarantees the balanced usage of data nodes (responsibility of the Docker Master) Disadvantage: • No guarantee that the data nodes are used in a balanced way to serve user requests • All the data nodes should run the same applications
  23. 23. 3-layer MICADO v1/b architecture
  24. 24. Assessment of MICADO v1/b Advantage: • Adaptable for different applications without modifying the cloud-init file • Scales up and down the number of data nodes according to the actual load of the data nodes • Guarantees the balanced usage of data nodes (responsibility of the Docker Master) • Guarantees that the data nodes are used in a balanced way to serve user requests (responsibility of the HAProxy services) Disadvantage: • All the data nodes should run the same applications Future work: • MICADO v2 to enable the balanced and scalable usage of different applications
  25. 25. Acknowledgement • The development of MICADO is in strong collaboration between the SZTAKI and UoW teams. • SZTAKI team: Prof. Peter Kacsuk • Dr. Jozsef Kovacs • Enikő Nagy • Attila Farkas • Dr. Robert Lovas • Dr. Zoltan Farkas • UoW team: Dr. Tamas Kiss • Prof. Gabor Terstyanszky • Botond Rákoczi • Gregoire Gesmier • Gabriele Pierantoni
  26. 26. For more information please visit www.project-cola.eu twitter.com/projectCOLA facebook.com/projectCOLA Thank you!

×