Opening the Path to Technical Excellence
Upcoming SlideShare
Loading in...5
×

Like this? Share it with your network

Share
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
1,928
On Slideshare
555
From Embeds
1,373
Number of Embeds
2

Actions

Shares
Downloads
10
Comments
0
Likes
0

Embeds 1,373

http://opennebulaconf.com 1,372
http://translate.googleusercontent.com 1

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide
  • Explain briefly what EO Products, Re-processing and CalVal mean.Amazon  market leaderData  Large data and expensive processingSystem  Amazon defaults to a ceiling of 50 VMs but it was very easily extendedUsage  Very satisfied users as they could have their results very early and identify processor anomalies which could be resolved in a “relatively” fast engineering process
  • Technical pre-requisites for deploymentApplication developed in line with specific development environment and/or compliant with established interfaces.Embedded software licensed to run on the cloud platform

Transcript

  • 1. Page 1ONE Conference 2013 - 26th September 2013 J.Farres – European Space Agency R&D for Earth Observation Ground Segment OpenNebula Conference, Berlin 26/9/2013 Cloud Computing in Space: Opening the path to technical excellence
  • 2. Page 2ONE Conference 2013 - 26th September 2013 Agenda 1.Background and Objectives 2.ESA Experiences 1. EO Re-processing on Amazon 2. Dissemination and Processing on Hetzner 3. SuperSites Exploitation platform with Helix Nebula 4. A sandbox service for Science 3.Summary of lessons learnt 4.Future prospects
  • 3. Page 3 Objectives (1) ONE Conference 2013 - 26th September 2013 3- Processing bursting 2- Dissemination peaks 1- ICT Costs savings 4- Collaboration platform Cloud Computing IaaS SaaS Hosting (VPS, Rental) CDN PaaS A model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction [NIST]
  • 4. Page 4 Background: ESA Mission Statement ESA's purpose shall be to provide for, and to promote, for exclusively peaceful purposes, cooperation among European States in space research and technology and their space applications, with a view to their being used for scientific purposes and for operational space applications systems: • by elaborating and implementing a long-term European space policy, by recommending space objectives to the Member States, and by concerting the policies of the Member States with respect to other national and international organisations and institutions; • by elaborating and implementing activities and programmes in the space field; • by coordinating the European space programme and national programmes, and by integrating the latter progressively and as completely as possible into the European space programme, in particular as regards the development of applications satellites; • by elaborating and implementing the industrial policy appropriate to its programme and by recommending a coherent industrial policy to the Member States.ONE Conference 2013 - 26th September 2013
  • 5. Page 5 Objectives (1) ONE Conference 2013 - 26th September 2013 3- Processing bursting 2- Dissemination peaks 1- ICT Costs savings 4- Collaboration platform Cloud Computing IaaS SaaS Hosting (VPS, Rental) CDN PaaS A model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction [NIST] 5- Lead effective use of modern computing infrastructures by European industry
  • 6. Page 6ONE Conference 2013 - 26th September 2013 Agenda 1.Background and Objectives 2.ESA Experiences 1. Mass processing on Amazon 2. Dissemination and Processing on Hetzner 3. SuperSites Exploitation platform with Helix Nebula 4. A sandbox service for Science 3.Summary of lessons learnt 4.Future prospects
  • 7. Page 7 Case 1: Mass processing on Amazon(1) Purpose •Fast re-processing of large EO products collections for CalVal purposes. Project / Service •Timeframe: 2009 and 2011 •Provider: Amazon, EC2, S3 •Data: ERS SAR Wave, MIPAS (30,000 products) •System: 200 Virtual Servers configured as Working Nodes to an ESA grid. •Usage: 11 CPU years of processing in 5 weeks ONE Conference 2013 - 26th September 2013
  • 8. Page 8ONE Conference 2013 - 26th September 2013 https://earth.esa.int/web/guest/missions/esa-operational-eo-missions/envisat/instruments/mipas
  • 9. Page 9 Pros Cons ONE Conference 2013 - 26th September 2013 Case 1: Mass processing on Amazon(2) Pros 1. Excellent processing scalability 2. Efficient bulk-in/out data service (via HD) 3. “The faster the cheaper” as it reduces storing costs 4. Good application portability for gridified applications Cons 1. Complex and changing pricing, e.g. periods with cheaper hosts and free data upload. 2. In-bound / Out-bound costs 3. Ad-hoc scripts to command provisioning 4. AMI format portability
  • 10. Page 10ONE Conference 2013 - 26th September 2013 Agenda 1.Background and Objectives 2.ESA Experiences 1. Mass processing on Amazon 2. Dissemination and Processing on Hetzner 3. SuperSites Exploitation platform with Helix Nebula 4. A sandbox service for Science 3.Summary of lessons learnt 4.Future prospects
  • 11. Page 11 Case 2: Dissemination and Processing on Hetzner (1) Purpose •Couple large processing and dissemination capabilities for low cost. Project / Service •Timeframe: 2011 •Provider: Hetzner •Data: 60TB •System: 1 Head: Catalogue, Processor Register n Nodes: Data Dissemination, Hadoop Processing Cluster Packaged as a back-end for web portal services •Usage: GeoHazards SuperSites (38,000 SAR images and 3,000 users) ONE Conference 2013 - 26th September 2013
  • 12. Page 12 InSAR Processing ~150 Satellite images: 1.5TB Time Processing: 150h CNR
  • 13. Page 13 Pros Cons ONE Conference 2013 - 26th September 2013 Case 2: Dissemination and Processing on Hetzner (2) Pros 1. Good archive scalability (chunks of 8TB) 2. Synergy of processing - dissemination: processing peaks followed by dissemination peaks 3. Much cheaper than Amazon services. No in-boud/out-boud costs 4. Physical dedicated servers enabled easier security Cons 1. System scales storage, dissemination and processing capabilities simultaneously. 2. Lower service levels than Level 3 or Amazon 3. Virtualization layer had to be deployed (KVM and ONE)
  • 14. Page 14ONE Conference 2013 - 26th September 2013 Agenda 1.Background and Objectives 2.ESA Experiences 1. Mass processing on Amazon 2. Dissemination and Processing on Hetzner 3. SuperSites Exploitation platform with Helix Nebula 4. A sandbox service for Science 3.Summary of lessons learnt 4.Future prospects
  • 15. Page 15 Case 3: Exploitation platform with Helix Nebula (1) Purpose •Pilot a collaborative platform for EO exploitation using multisourced cloud provisioning. Project / Service •Timeframe: 2012-2013 •Providers: ATOS, CloudSigma, Interoute, T-Systems, EGI •Cloud Brokers: SlipStream, Entratius •Data: GeoHazards data (ESA, CNES, DLR, …) •Processors: ESA, CNR, Gamma, … •System: 15 TB of raw data, 4 processing services, dedicated VM for selected users, >200 users ONE Conference 2013 - 26th September 2013
  • 16. Page 16 Model EO Data Software ICT Exploitation Platform Science Services Services ESA UNCLASSIFIED – For Internal Use – Version 2013/04/29
  • 17. Page 17 SSEP SuperSites Exploitation Platform ESA UNCLASSIED - For Offical Use 29/04/2013
  • 18. Page 1810/9/2013 18 ESA PoC on EGI Federated Cloud The ESA Proof of Concepts on the EGI Federated Cloud focuses on demonstrating the possibility to provide Processing Services to ESA scientists using the EGI Federated Cloud resources • Participants: – User community • ESA Research and Service Support section: Configuration and execution of the tests – Technology providers • Sixsq: Provided Open Source SlipStream software and OCCI connector – Resource providers • CESNET (OpenNebula): Performance tests and multi-site tests • GRNET (synnefo): Multi-site tests • CESGA (OpenNebula): Hosting of the SlipStream server
  • 19. Page 19 Pros Cons ONE Conference 2013 - 26th September 2013 Case 3: Exploitation platform with Helix Nebula (2) Pros 1. High performance storage and dissemination from cloud 2. Scalable processing co-located with data (same VDC) 3. Multi-sourced via cloud brokering services 4. Direct provision to virtual hosts to users in the cloud, via ESA 5. Easy application deployment via grid controller Cons 1. Frequent platform upgrades 2. Need to distribute processing resources near distributed data 3. No Cloud federation. Limitations of brokers 4. COTS licensing 5. Grid on Cloud approach. Workaround to high effort in application “cloudification”
  • 20. Page 20ONE Conference 2013 - 26th September 2013 Agenda 1.Background and Objectives 2.ESA Experiences 1. Mass processing on Amazon 2. Dissemination and Processing on Hetzner 3. SuperSites Exploitation platform with Helix Nebula 4. A sandbox service for Science 3.Summary of lessons learnt 4.Future prospects
  • 21. Page 21 Case 4: A sandbox service for Science (1) Purpose • Provide researchers and service providers a development environment for cloudifying and exploiting their algorithms/services. Project / Service •Timeframe: 2013-2014 •Providers: Private cloud + Helix Nebula •Data: Multiple reference data sets from ESA archives •Processors: Those developed by the users ONE Conference 2013 - 26th September 2013
  • 22. Page 22 Case 4: A sandbox service for Science (2) 10/9/2013 22 Many Service Providers
  • 23. Page 23 Pros Cons ONE Conference 2013 - 26th September 2013 Case 4: A sandbox service for Science (3) Pros 1. Hybrid cloud model in support of development (private) -> deployment (public) 2. Deployment model via PaaS and SaaS 3. Simplified “cloudification” via Cloudera + supporting tools (for SPMD paradigm) Cons 1. Need for new CSP drivers for Helix Nebula: T-Systems (Zimory), Interoute (Jclouds) 2. Limited CSP support to PaaS and SaaS services. 3. Slow adoption of cloud-reduce paradigm among application developers in Remote Sensing
  • 24. Page 24 Role of OpenNebula on the four Cases Case ONE 1: Mass processing on Amazon Use of GridWay to command IaaS provisioning from Amazon 2: Dissemination and Processing on Hetzner Use of ONE to host private IaaS on dedicated hosting services 3: Exploitation platform with Helix Nebula Use of ONE in resource providers to EGI Federated Cloud 4: A sandbox service for Science Use of ONE for hybrid cloud provision ESA + Helix Nebula Development of drivers for Zimory and Jclouds ONE Conference 2013 - 26th September 2013
  • 25. Page 25ONE Conference 2013 - 26th September 2013 Agenda 1.Background and Objectives 2.ESA Experiences 1. EO Re-processing on Amazon 2. Dissemination and Processing on Hetzner 3. SuperSites Exploitation platform with Helix Nebula 4. A sandbox service for Science 3.Summary of lessons learnt 4.Future prospects
  • 26. Page 26 Lessons Learnt: ICT provisioning 1. As soon as ICT needs can be predicted and planned, IaaS is more expensive than other hosting solutions like (rental, dedicated hosting). 2. Flexibility of Public IaaS is less appealing when internal resources are pooled, virtualized and managed as an internal cloud. 3. On the other hand, IaaS services allow to size down internal ICT resources to the “fixed” need and ensure their maximum utilization; e.g. using external provisioning for the “variable” need. Hybrid ICT provisioning ONE Conference 2013 - 26th September 2013
  • 27. Page 27 Lessons Learnt: Service Levels • Terms & Conditions in Public Clouds express surprising low commitment. • Cloud opportunities can become risks when applied to critical systems. Develop multi-sourcing Plan contingency scenarios for services hosted in Public Clouds ONE Conference 2013 - 26th September 2013
  • 28. Page 28 Lessons Learnt: Application Areas • Dissemination and on-demand processing – because are very variable (depending on user demand) • Secondary archive and re-processing – because are limited in time • Temporary resources for integration, testing and demonstration – because are limited in time • System sizing – because needs are unknown Important areas when remote sensing services can gain from Cloud Computing ONE Conference 2013 - 26th September 2013
  • 29. Page 29 Lessons learnt: User expectations • Open Data – All data are discoverable, accessible online and free – Data is arranged on long time series of coherent data from different providers. • Open Computing – Users will be able to perform processing directly on the cloud using virtual servers. – Users can choose their preferred cloud provider • Open Source Software – All basic/platform software is open and freely available – Application can be easily ported across clouds • Open Collaboration – Data and applications can be easily shared with other users Be up to the users expectations ONE Conference 2013 - 26th September 2013
  • 30. Page 30 Lessons learnt and OpenNebula ONE Conference 2013 - 26th September 2013 OK, Case 4 OK, Case 3 Lesson learnt ONE Hybrid cloud OK, Case4: A sandbox service for Science Cloud multi-sourcing OK, Case3: Exploitation platform with Helix Nebula Application Areas N/A User Expectations OK, Open Source Software
  • 31. Page 31ONE Conference 2013 - 26th September 2013 Agenda 1.Background and Objectives 2.ESA Experiences 1. EO Re-processing on Amazon 2. Dissemination and Processing on Hetzner 3. SuperSites Exploitation platform with Helix Nebula 4. A sandbox service for Science 3.Summary of lessons learnt 4.Future prospects
  • 32. Page 32 Future Prospects (1) ICT • Set-up mid-term relation with 2-4 cloud providers; similarly to current agreements with network providers. • Cloudify present corporate computing resources • Establish common ICT provisioning service based on: – Hybrid and multi-sourced resources – Based on in-house brokering layer ONE Conference 2013 - 26th September 2013
  • 33. Page 33 Future Prospects (2) POLICY • Mandate cloud hosting for specific activities: Integration & Validation, Demonstrators, R&D … • Promote the use of cloud computing solutions among ESA service providers, in benefit for their competiveness • Continue to launch specific flagship projects based on public clouds – SuperSites exploitation platform (continuation) – Thematic Exploitation platforms ONE Conference 2013 - 26th September 2013
  • 34. Page 34 Future Prospects (3) 1. Adopt and promote Open data policy – ESA Data Policy – Data access agreements 2. Adopt and promote interoperability Standards – Data discovery & ordering – Processing discovery & ordering – Data / Results access – RAAA – Registration, Authentification, Authorisation, Accounting 3. Provide Open Source Software – EO Toolboxes – Data discovery/catalogue tools – EO Data management and access 4. Promote Open computing infrastructures – Cloud computing paradigm – Standards for IaaS and brokering services – Software Licensing Agreements
  • 35. Page 35 In summary: Manifest + Open Data + Open Software + Open Computing = Increased market development & industrial competiveness ONE Conference 2013 - 26th September 2013