SlideShare a Scribd company logo

Master's Thesis - climateprediction.net: A Cloudy Approach

•
•
K
kabute

The document discusses migrating the climateprediction.net project's computing infrastructure to the cloud to address issues with long simulation times and resource constraints. It outlines measuring simulation times on AWS instances, developing a central control system with a scheduler and dashboard to manage cloud resources, and storing simulation outputs in S3. The objectives of enabling simulations in the cloud, creating metrics and reducing costs were achieved, with next steps including migrating the BOINC server and improving scheduler control of clients.

1 of 26
Download to read offline
Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions climateprediction.net: A Cloudy Approach Master in High Performance Computing Master’s Thesis Diego P´erez Montes advised by Tom´as Fern´andez Pena Juan Antonio A˜nel Cabanelas July 1, 2014 Diego P´erez Montes climateprediction.net: A Cloudy Approach
Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions 1 Problem Background Current Infrastructure Problem Description 2 Computing Infrastructure Migration Measuring the Problem... Infrastructure Redesign 3 Storage 4 Central Control System Backend Components Dashboard Running the Simulation 5 Conclusions Diego P´erez Montes climateprediction.net: A Cloudy Approach
Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Motivation Solve a real problem, useful for someone and that can be expanded in further works. Apply what I’ve learned in the Master courses. I do love large infrastructure problems (and this is a big one!). Diego P´erez Montes climateprediction.net: A Cloudy Approach
Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Current Infrastructure First of all: How does the project currently work? Diego P´erez Montes climateprediction.net: A Cloudy Approach
Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Current Infrastructure Figure : BOINC: High Level Architecture and Workflow Diego P´erez Montes climateprediction.net: A Cloudy Approach
Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Problem Description So, what is the problem then? The need of execution of a new model (HadGEM) The resources requirements are higher (Hardware: Computing and Storage). The current BOINC workunit processing time is 7-9 days , this wants to be reduced. Heterogeneous and unpredictable environment: Can’t manage resources on-demand. Execution time can’t be properly measured. Processed data is missing. Diego P´erez Montes climateprediction.net: A Cloudy Approach
Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Problem Description So, what is the problem then? Need to establish metrics on the project. Rationalization of costs (how much does a simulation really cost?) Diego P´erez Montes climateprediction.net: A Cloudy Approach
Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Project Objectives How is it going to be solved? Conversion to an Infrastructure as a Service (Iaas) in the Cloud (Amazon Web Services AWS: EC2 for Computing and S3 for Storage). Creation of a new abstraction layer, the Central Control System: Infrastructure and resources management. Creation of metrics and statistics. Free Software. Fully documented. Diego P´erez Montes climateprediction.net: A Cloudy Approach
Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Measuring the Problem... Unknown problem real size and how is it going to behave into the new environment with the new parametrization. Initial data from the current infrastructure over BOINC (Computing point of view): A workunit takes in average from 7 to 9 days to be processed. A full simulation is (minimum) 36,000 workunits into sections of 6,000. Diego P´erez Montes climateprediction.net: A Cloudy Approach
Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Measuring the Problem... Initial considerations: Models used on the tests: weather@homeUK floods and and weather@home Australia New Zealand (full and regional: HaDAM3P and HadRM3P) Two representative systems (on EC2) were selected and 10 consecutive executions were done. Diego P´erez Montes climateprediction.net: A Cloudy Approach
Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Measuring the Problem... System #1: Moderate CPU CPU: 2 x Xeon E5-2650 MEM: 8GB (4GB/Core) GPU: No Workunit Time: 7.32 days Workunit Cost: USD 4.464 Full Simulation Cost: USD 160,704 Diego P´erez Montes climateprediction.net: A Cloudy Approach
Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Measuring the Problem... System #2: Intensive CPU&GPU CPU: 16 x Xeon X5570 MEM: 24GB (1.5GB/Core) GPU: 2 x Tesla M2050 Workunit Time: 1.99 days Workunit Cost: USD 100.966 Full Simulation Cost: USD 3,634,776 Diego P´erez Montes climateprediction.net: A Cloudy Approach
Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Measuring the Problem... Diego P´erez Montes climateprediction.net: A Cloudy Approach
Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions How much does it really cost? Diego P´erez Montes climateprediction.net: A Cloudy Approach
Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Going IaaS Figure : Proposed Infrastructure Diego P´erez Montes climateprediction.net: A Cloudy Approach
Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Going IaaS Steps: 1 Template an instance: Install Operating System (Amazon Linux 2014.03.1 64bit) Configure network and firewall. Configure local storage: 16GB Install and configure BOINC to use climateprediction.net Install local client (Simulation Terminator) 2 Contextualize and scale. Diego P´erez Montes climateprediction.net: A Cloudy Approach
Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Storage Every simulation (36,000 workunits) outputs 3.6 TB of data. There are not enough resources (disk space) on the current systems. Figure : Shared Storage Architecture Diego P´erez Montes climateprediction.net: A Cloudy Approach
Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Architecture Figure : Central System Architecture Diego P´erez Montes climateprediction.net: A Cloudy Approach
Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Backend Components Simple Scheduler: Runs and configures simulation with given parameters (start/stop instances). Reaper: Releases resources (terminates instances) when they are powered off. RESTful API: Gives access to configure and run simulations. Diego P´erez Montes climateprediction.net: A Cloudy Approach
Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions API RESTful API Get simulation status. Get metric/statistic data. Set/modify simulation parameters (number of worker nodes/instances). Stop simulation. Diego P´erez Montes climateprediction.net: A Cloudy Approach
Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Dashboard Figure : Dashboard Interface Diego P´erez Montes climateprediction.net: A Cloudy Approach
Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Running the Simulation [Overview of a Live System] Diego P´erez Montes climateprediction.net: A Cloudy Approach
Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Conclusions Objectives Achieved Computing and Storage successfully migrated to the Cloud (EC2 and S3). Simulations were executed, showing that running the model in the cloud is possible. Development and a Central System (scheduler and dashboard). Got costs and metrics of the project. Diego P´erez Montes climateprediction.net: A Cloudy Approach
Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Conclusions What’s Next? Migrate BOINC server. More control/interaction with clients so the scheduler can be improved (and give a full SaaS layer). Costs: “warm up“ stage to dynamically recalculate price. Diego P´erez Montes climateprediction.net: A Cloudy Approach
Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Thanks! Diego P´erez Montes climateprediction.net: A Cloudy Approach
Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Used Icons Links Iconset Windows 8 metro style: https://www.iconfinder. com/iconsets/windows-8-metro-style Link: http://sta.sh/0228t4fyjyjb Diego P´erez Montes climateprediction.net: A Cloudy Approach

Recommended

GRIMES_Visualizing_Telemetry
GRIMES_Visualizing_TelemetryGRIMES_Visualizing_Telemetry
GRIMES_Visualizing_TelemetryKevin Grimes
 
Vizualize 300 tb in less than 5_seconds
Vizualize 300 tb in less than 5_secondsVizualize 300 tb in less than 5_seconds
Vizualize 300 tb in less than 5_seconds
Yaniv Shalev
 
SkyhookDM - Towards an Arrow-Native Storage System
SkyhookDM - Towards an Arrow-Native Storage SystemSkyhookDM - Towards an Arrow-Native Storage System
SkyhookDM - Towards an Arrow-Native Storage System
JayjeetChakraborty
 
Demonstrating a Pre-Exascale, Cost-Effective Multi-Cloud Environment for Scie...
Demonstrating a Pre-Exascale, Cost-Effective Multi-Cloud Environment for Scie...Demonstrating a Pre-Exascale, Cost-Effective Multi-Cloud Environment for Scie...
Demonstrating a Pre-Exascale, Cost-Effective Multi-Cloud Environment for Scie...
Igor Sfiligoi
 
Running a GPU burst for Multi-Messenger Astrophysics with IceCube across all ...
Running a GPU burst for Multi-Messenger Astrophysics with IceCube across all ...Running a GPU burst for Multi-Messenger Astrophysics with IceCube across all ...
Running a GPU burst for Multi-Messenger Astrophysics with IceCube across all ...
Igor Sfiligoi
 
Building a Better TSA Screening Algorithm
Building a Better TSA Screening AlgorithmBuilding a Better TSA Screening Algorithm
Building a Better TSA Screening Algorithm
Yvonne K. Matos
 
"Building and running the cloud GPU vacuum cleaner"
"Building and running the cloud GPU vacuum cleaner""Building and running the cloud GPU vacuum cleaner"
"Building and running the cloud GPU vacuum cleaner"
Frank Wuerthwein
 
"Approaches for Energy Efficient Implementation of Deep Neural Networks," a P...
"Approaches for Energy Efficient Implementation of Deep Neural Networks," a P..."Approaches for Energy Efficient Implementation of Deep Neural Networks," a P...
"Approaches for Energy Efficient Implementation of Deep Neural Networks," a P...
Edge AI and Vision Alliance
 

Recommended

GRIMES_Visualizing_Telemetry
GRIMES_Visualizing_TelemetryGRIMES_Visualizing_Telemetry
GRIMES_Visualizing_TelemetryKevin Grimes
 
Vizualize 300 tb in less than 5_seconds
Vizualize 300 tb in less than 5_secondsVizualize 300 tb in less than 5_seconds
Vizualize 300 tb in less than 5_seconds
Yaniv Shalev
 
SkyhookDM - Towards an Arrow-Native Storage System
SkyhookDM - Towards an Arrow-Native Storage SystemSkyhookDM - Towards an Arrow-Native Storage System
SkyhookDM - Towards an Arrow-Native Storage System
JayjeetChakraborty
 
Demonstrating a Pre-Exascale, Cost-Effective Multi-Cloud Environment for Scie...
Demonstrating a Pre-Exascale, Cost-Effective Multi-Cloud Environment for Scie...Demonstrating a Pre-Exascale, Cost-Effective Multi-Cloud Environment for Scie...
Demonstrating a Pre-Exascale, Cost-Effective Multi-Cloud Environment for Scie...
Igor Sfiligoi
 
Running a GPU burst for Multi-Messenger Astrophysics with IceCube across all ...
Running a GPU burst for Multi-Messenger Astrophysics with IceCube across all ...Running a GPU burst for Multi-Messenger Astrophysics with IceCube across all ...
Running a GPU burst for Multi-Messenger Astrophysics with IceCube across all ...
Igor Sfiligoi
 
Building a Better TSA Screening Algorithm
Building a Better TSA Screening AlgorithmBuilding a Better TSA Screening Algorithm
Building a Better TSA Screening Algorithm
Yvonne K. Matos
 
"Building and running the cloud GPU vacuum cleaner"
"Building and running the cloud GPU vacuum cleaner""Building and running the cloud GPU vacuum cleaner"
"Building and running the cloud GPU vacuum cleaner"
Frank Wuerthwein
 
"Approaches for Energy Efficient Implementation of Deep Neural Networks," a P...
"Approaches for Energy Efficient Implementation of Deep Neural Networks," a P..."Approaches for Energy Efficient Implementation of Deep Neural Networks," a P...
"Approaches for Energy Efficient Implementation of Deep Neural Networks," a P...
Edge AI and Vision Alliance
 
Data-intensive IceCube Cloud Burst
Data-intensive IceCube Cloud BurstData-intensive IceCube Cloud Burst
Data-intensive IceCube Cloud Burst
Igor Sfiligoi
 
Burst data retrieval after 50k GPU Cloud run
Burst data retrieval after 50k GPU Cloud runBurst data retrieval after 50k GPU Cloud run
Burst data retrieval after 50k GPU Cloud run
Igor Sfiligoi
 
NRP Engagement webinar - Running a 51k GPU multi-cloud burst for MMA with Ic...
NRP Engagement webinar - Running a 51k GPU multi-cloud burst for MMA with Ic... NRP Engagement webinar - Running a 51k GPU multi-cloud burst for MMA with Ic...
NRP Engagement webinar - Running a 51k GPU multi-cloud burst for MMA with Ic...
Igor Sfiligoi
 
Deep Learning for Computer Vision - PyconDE 2017
Deep Learning for Computer Vision - PyconDE 2017Deep Learning for Computer Vision - PyconDE 2017
Deep Learning for Computer Vision - PyconDE 2017
Alex Conway
 
Deep recurrent neutral networks for Sequence Learning in Spark
Deep recurrent neutral networks for Sequence Learning in SparkDeep recurrent neutral networks for Sequence Learning in Spark
Deep recurrent neutral networks for Sequence Learning in Spark
DataWorks Summit/Hadoop Summit
 
Big Data in the Real World. Real-time Football Analytics
Big Data in the Real World. Real-time Football AnalyticsBig Data in the Real World. Real-time Football Analytics
Big Data in the Real World. Real-time Football AnalyticsWSO2
 
Message broadcasts and logical time of process
Message broadcasts and logical time of processMessage broadcasts and logical time of process
Message broadcasts and logical time of processJawid Ahmad Baktash
 
PyConZA'17 Deep Learning for Computer Vision
PyConZA'17 Deep Learning for Computer VisionPyConZA'17 Deep Learning for Computer Vision
PyConZA'17 Deep Learning for Computer Vision
Alex Conway
 
Image Classification Done Simply using Keras and TensorFlow
Image Classification Done Simply using Keras and TensorFlow Image Classification Done Simply using Keras and TensorFlow
Image Classification Done Simply using Keras and TensorFlow
Rajiv Shah
 
High Throughput Data Analysis
High Throughput Data AnalysisHigh Throughput Data Analysis
High Throughput Data Analysis
J Singh
 
Machine Learning Tokyo - Deep Neural Networks for Video - NumberBoost
Machine Learning Tokyo - Deep Neural Networks for Video - NumberBoostMachine Learning Tokyo - Deep Neural Networks for Video - NumberBoost
Machine Learning Tokyo - Deep Neural Networks for Video - NumberBoost
Alex Conway
 
Deep Neural Networks for Computer Vision
Deep Neural Networks for Computer VisionDeep Neural Networks for Computer Vision
Deep Neural Networks for Computer Vision
Alex Conway
 
Noha danms13 talk_final
Noha danms13 talk_finalNoha danms13 talk_final
Noha danms13 talk_finalNoha Elprince
 
Convolutional Neural Networks for Image Classification (Cape Town Deep Learni...
Convolutional Neural Networks for Image Classification (Cape Town Deep Learni...Convolutional Neural Networks for Image Classification (Cape Town Deep Learni...
Convolutional Neural Networks for Image Classification (Cape Town Deep Learni...
Alex Conway
 
Real-time Big Data Processing with Storm
Real-time Big Data Processing with StormReal-time Big Data Processing with Storm
Real-time Big Data Processing with Storm
viirya
 
PyDresden 20170824 - Deep Learning for Computer Vision
PyDresden 20170824 - Deep Learning for Computer VisionPyDresden 20170824 - Deep Learning for Computer Vision
PyDresden 20170824 - Deep Learning for Computer Vision
Alex Conway
 
KEK helps scientists uncover the mysteries of the universe with Lenovo superc...
KEK helps scientists uncover the mysteries of the universe with Lenovo superc...KEK helps scientists uncover the mysteries of the universe with Lenovo superc...
KEK helps scientists uncover the mysteries of the universe with Lenovo superc...
Lenovo Data Center
 
Automatic Features Generation And Model Training On Spark: A Bayesian Approach
Automatic Features Generation And Model Training On Spark: A Bayesian ApproachAutomatic Features Generation And Model Training On Spark: A Bayesian Approach
Automatic Features Generation And Model Training On Spark: A Bayesian Approach
Spark Summit
 
deep reinforcement learning with double q learning
deep reinforcement learning with double q learningdeep reinforcement learning with double q learning
deep reinforcement learning with double q learning
SeungHyeok Baek
 
Deep Reinforcement Learning: Q-Learning
Deep Reinforcement Learning: Q-LearningDeep Reinforcement Learning: Q-Learning
Deep Reinforcement Learning: Q-Learning
Kai-Wen Zhao
 
Elm 0.17 at Dublin Elm Meetup May 2016
Elm 0.17 at Dublin Elm Meetup May 2016Elm 0.17 at Dublin Elm Meetup May 2016
Elm 0.17 at Dublin Elm Meetup May 2016
Michael Twomey
 
Mesilane Lendab.Multifilm
Mesilane Lendab.MultifilmMesilane Lendab.Multifilm
Mesilane Lendab.Multifilm
pillilood53
 

More Related Content

What's hot

Data-intensive IceCube Cloud Burst
Data-intensive IceCube Cloud BurstData-intensive IceCube Cloud Burst
Data-intensive IceCube Cloud Burst
Igor Sfiligoi
 
Burst data retrieval after 50k GPU Cloud run
Burst data retrieval after 50k GPU Cloud runBurst data retrieval after 50k GPU Cloud run
Burst data retrieval after 50k GPU Cloud run
Igor Sfiligoi
 
NRP Engagement webinar - Running a 51k GPU multi-cloud burst for MMA with Ic...
NRP Engagement webinar - Running a 51k GPU multi-cloud burst for MMA with Ic... NRP Engagement webinar - Running a 51k GPU multi-cloud burst for MMA with Ic...
NRP Engagement webinar - Running a 51k GPU multi-cloud burst for MMA with Ic...
Igor Sfiligoi
 
Deep Learning for Computer Vision - PyconDE 2017
Deep Learning for Computer Vision - PyconDE 2017Deep Learning for Computer Vision - PyconDE 2017
Deep Learning for Computer Vision - PyconDE 2017
Alex Conway
 
Deep recurrent neutral networks for Sequence Learning in Spark
Deep recurrent neutral networks for Sequence Learning in SparkDeep recurrent neutral networks for Sequence Learning in Spark
Deep recurrent neutral networks for Sequence Learning in Spark
DataWorks Summit/Hadoop Summit
 
Big Data in the Real World. Real-time Football Analytics
Big Data in the Real World. Real-time Football AnalyticsBig Data in the Real World. Real-time Football Analytics
Big Data in the Real World. Real-time Football AnalyticsWSO2
 
Message broadcasts and logical time of process
Message broadcasts and logical time of processMessage broadcasts and logical time of process
Message broadcasts and logical time of processJawid Ahmad Baktash
 
PyConZA'17 Deep Learning for Computer Vision
PyConZA'17 Deep Learning for Computer VisionPyConZA'17 Deep Learning for Computer Vision
PyConZA'17 Deep Learning for Computer Vision
Alex Conway
 
Image Classification Done Simply using Keras and TensorFlow
Image Classification Done Simply using Keras and TensorFlow Image Classification Done Simply using Keras and TensorFlow
Image Classification Done Simply using Keras and TensorFlow
Rajiv Shah
 
High Throughput Data Analysis
High Throughput Data AnalysisHigh Throughput Data Analysis
High Throughput Data Analysis
J Singh
 
Machine Learning Tokyo - Deep Neural Networks for Video - NumberBoost
Machine Learning Tokyo - Deep Neural Networks for Video - NumberBoostMachine Learning Tokyo - Deep Neural Networks for Video - NumberBoost
Machine Learning Tokyo - Deep Neural Networks for Video - NumberBoost
Alex Conway
 
Deep Neural Networks for Computer Vision
Deep Neural Networks for Computer VisionDeep Neural Networks for Computer Vision
Deep Neural Networks for Computer Vision
Alex Conway
 
Noha danms13 talk_final
Noha danms13 talk_finalNoha danms13 talk_final
Noha danms13 talk_finalNoha Elprince
 
Convolutional Neural Networks for Image Classification (Cape Town Deep Learni...
Convolutional Neural Networks for Image Classification (Cape Town Deep Learni...Convolutional Neural Networks for Image Classification (Cape Town Deep Learni...
Convolutional Neural Networks for Image Classification (Cape Town Deep Learni...
Alex Conway
 
Real-time Big Data Processing with Storm
Real-time Big Data Processing with StormReal-time Big Data Processing with Storm
Real-time Big Data Processing with Storm
viirya
 
PyDresden 20170824 - Deep Learning for Computer Vision
PyDresden 20170824 - Deep Learning for Computer VisionPyDresden 20170824 - Deep Learning for Computer Vision
PyDresden 20170824 - Deep Learning for Computer Vision
Alex Conway
 
KEK helps scientists uncover the mysteries of the universe with Lenovo superc...
KEK helps scientists uncover the mysteries of the universe with Lenovo superc...KEK helps scientists uncover the mysteries of the universe with Lenovo superc...
KEK helps scientists uncover the mysteries of the universe with Lenovo superc...
Lenovo Data Center
 
Automatic Features Generation And Model Training On Spark: A Bayesian Approach
Automatic Features Generation And Model Training On Spark: A Bayesian ApproachAutomatic Features Generation And Model Training On Spark: A Bayesian Approach
Automatic Features Generation And Model Training On Spark: A Bayesian Approach
Spark Summit
 
deep reinforcement learning with double q learning
deep reinforcement learning with double q learningdeep reinforcement learning with double q learning
deep reinforcement learning with double q learning
SeungHyeok Baek
 
Deep Reinforcement Learning: Q-Learning
Deep Reinforcement Learning: Q-LearningDeep Reinforcement Learning: Q-Learning
Deep Reinforcement Learning: Q-Learning
Kai-Wen Zhao
 

What's hot (20)

Data-intensive IceCube Cloud Burst
Data-intensive IceCube Cloud BurstData-intensive IceCube Cloud Burst
Data-intensive IceCube Cloud Burst
 
Burst data retrieval after 50k GPU Cloud run
Burst data retrieval after 50k GPU Cloud runBurst data retrieval after 50k GPU Cloud run
Burst data retrieval after 50k GPU Cloud run
 
NRP Engagement webinar - Running a 51k GPU multi-cloud burst for MMA with Ic...
NRP Engagement webinar - Running a 51k GPU multi-cloud burst for MMA with Ic... NRP Engagement webinar - Running a 51k GPU multi-cloud burst for MMA with Ic...
NRP Engagement webinar - Running a 51k GPU multi-cloud burst for MMA with Ic...
 
Deep Learning for Computer Vision - PyconDE 2017
Deep Learning for Computer Vision - PyconDE 2017Deep Learning for Computer Vision - PyconDE 2017
Deep Learning for Computer Vision - PyconDE 2017
 
Deep recurrent neutral networks for Sequence Learning in Spark
Deep recurrent neutral networks for Sequence Learning in SparkDeep recurrent neutral networks for Sequence Learning in Spark
Deep recurrent neutral networks for Sequence Learning in Spark
 
Big Data in the Real World. Real-time Football Analytics
Big Data in the Real World. Real-time Football AnalyticsBig Data in the Real World. Real-time Football Analytics
Big Data in the Real World. Real-time Football Analytics
 
Message broadcasts and logical time of process
Message broadcasts and logical time of processMessage broadcasts and logical time of process
Message broadcasts and logical time of process
 
PyConZA'17 Deep Learning for Computer Vision
PyConZA'17 Deep Learning for Computer VisionPyConZA'17 Deep Learning for Computer Vision
PyConZA'17 Deep Learning for Computer Vision
 
Image Classification Done Simply using Keras and TensorFlow
Image Classification Done Simply using Keras and TensorFlow Image Classification Done Simply using Keras and TensorFlow
Image Classification Done Simply using Keras and TensorFlow
 
High Throughput Data Analysis
High Throughput Data AnalysisHigh Throughput Data Analysis
High Throughput Data Analysis
 
Machine Learning Tokyo - Deep Neural Networks for Video - NumberBoost
Machine Learning Tokyo - Deep Neural Networks for Video - NumberBoostMachine Learning Tokyo - Deep Neural Networks for Video - NumberBoost
Machine Learning Tokyo - Deep Neural Networks for Video - NumberBoost
 
Deep Neural Networks for Computer Vision
Deep Neural Networks for Computer VisionDeep Neural Networks for Computer Vision
Deep Neural Networks for Computer Vision
 
Noha danms13 talk_final
Noha danms13 talk_finalNoha danms13 talk_final
Noha danms13 talk_final
 
Convolutional Neural Networks for Image Classification (Cape Town Deep Learni...
Convolutional Neural Networks for Image Classification (Cape Town Deep Learni...Convolutional Neural Networks for Image Classification (Cape Town Deep Learni...
Convolutional Neural Networks for Image Classification (Cape Town Deep Learni...
 
Real-time Big Data Processing with Storm
Real-time Big Data Processing with StormReal-time Big Data Processing with Storm
Real-time Big Data Processing with Storm
 
PyDresden 20170824 - Deep Learning for Computer Vision
PyDresden 20170824 - Deep Learning for Computer VisionPyDresden 20170824 - Deep Learning for Computer Vision
PyDresden 20170824 - Deep Learning for Computer Vision
 
KEK helps scientists uncover the mysteries of the universe with Lenovo superc...
KEK helps scientists uncover the mysteries of the universe with Lenovo superc...KEK helps scientists uncover the mysteries of the universe with Lenovo superc...
KEK helps scientists uncover the mysteries of the universe with Lenovo superc...
 
Automatic Features Generation And Model Training On Spark: A Bayesian Approach
Automatic Features Generation And Model Training On Spark: A Bayesian ApproachAutomatic Features Generation And Model Training On Spark: A Bayesian Approach
Automatic Features Generation And Model Training On Spark: A Bayesian Approach
 
deep reinforcement learning with double q learning
deep reinforcement learning with double q learningdeep reinforcement learning with double q learning
deep reinforcement learning with double q learning
 
Deep Reinforcement Learning: Q-Learning
Deep Reinforcement Learning: Q-LearningDeep Reinforcement Learning: Q-Learning
Deep Reinforcement Learning: Q-Learning
 

Viewers also liked

Elm 0.17 at Dublin Elm Meetup May 2016
Elm 0.17 at Dublin Elm Meetup May 2016Elm 0.17 at Dublin Elm Meetup May 2016
Elm 0.17 at Dublin Elm Meetup May 2016
Michael Twomey
 
Mesilane Lendab.Multifilm
Mesilane Lendab.MultifilmMesilane Lendab.Multifilm
Mesilane Lendab.Multifilm
pillilood53
 
PyCon 2010
PyCon 2010PyCon 2010
PyCon 2010
Michael Twomey
 
Elm @ DublinJS
Elm @ DublinJSElm @ DublinJS
Elm @ DublinJS
Michael Twomey
 
Local Business Online Marketing 101
Local Business Online Marketing 101Local Business Online Marketing 101
Local Business Online Marketing 101
Nick Allen
 
LinkedIn Groups - The ins and outs
LinkedIn Groups - The ins and outsLinkedIn Groups - The ins and outs
LinkedIn Groups - The ins and outs
Nick Allen
 
Mechanics of materials_solution_manual_3rd_ed__by_beer_johnston__dewolf
Mechanics of materials_solution_manual_3rd_ed__by_beer_johnston__dewolfMechanics of materials_solution_manual_3rd_ed__by_beer_johnston__dewolf
Mechanics of materials_solution_manual_3rd_ed__by_beer_johnston__dewolfUniversiti Kuala Lumpur
 

Viewers also liked (9)

Elm 0.17 at Dublin Elm Meetup May 2016
Elm 0.17 at Dublin Elm Meetup May 2016Elm 0.17 at Dublin Elm Meetup May 2016
Elm 0.17 at Dublin Elm Meetup May 2016
 
Mesilane Lendab.Multifilm
Mesilane Lendab.MultifilmMesilane Lendab.Multifilm
Mesilane Lendab.Multifilm
 
PyCon 2010
PyCon 2010PyCon 2010
PyCon 2010
 
Mesilane Lendab
Mesilane LendabMesilane Lendab
Mesilane Lendab
 
Elm @ DublinJS
Elm @ DublinJSElm @ DublinJS
Elm @ DublinJS
 
Local Business Online Marketing 101
Local Business Online Marketing 101Local Business Online Marketing 101
Local Business Online Marketing 101
 
LinkedIn Groups - The ins and outs
LinkedIn Groups - The ins and outsLinkedIn Groups - The ins and outs
LinkedIn Groups - The ins and outs
 
Mechanics of materials_solution_manual_3rd_ed__by_beer_johnston__dewolf
Mechanics of materials_solution_manual_3rd_ed__by_beer_johnston__dewolfMechanics of materials_solution_manual_3rd_ed__by_beer_johnston__dewolf
Mechanics of materials_solution_manual_3rd_ed__by_beer_johnston__dewolf
 
Personality
PersonalityPersonality
Personality
 

Similar to Master's Thesis - climateprediction.net: A Cloudy Approach

Google Cloud Computing on Google Developer 2008 Day
Google Cloud Computing on Google Developer 2008 DayGoogle Cloud Computing on Google Developer 2008 Day
Google Cloud Computing on Google Developer 2008 Dayprogrammermag
 
Apache Storm
Apache StormApache Storm
Apache StormEdureka!
 
High Performance & High Throughput Computing - EUDAT Summer School (Giuseppe ...
High Performance & High Throughput Computing - EUDAT Summer School (Giuseppe ...High Performance & High Throughput Computing - EUDAT Summer School (Giuseppe ...
High Performance & High Throughput Computing - EUDAT Summer School (Giuseppe ...
EUDAT
 
Intelligent Monitoring
Intelligent MonitoringIntelligent Monitoring
Intelligent Monitoring
Intelie
 
Using Grid Technologies in the Cloud for High Scalability
Using Grid Technologies in the Cloud for High ScalabilityUsing Grid Technologies in the Cloud for High Scalability
Using Grid Technologies in the Cloud for High Scalability
mabuhr
 
Ssbse10.ppt
Ssbse10.pptSsbse10.ppt
Ssbse10.ppt
Yann-Gaël Guéhéneuc
 
Exascale Capabl
Exascale CapablExascale Capabl
Exascale Capabl
Sagar Dolas
 
Fast optimization intevacoct6_3final
Fast optimization intevacoct6_3finalFast optimization intevacoct6_3final
Fast optimization intevacoct6_3finaleArtius, Inc.
 
Slide 1
Slide 1Slide 1
Slide 1butest
 
Multi-Objective Optimization of Solar Cells Thermal Uniformity Using Combined...
Multi-Objective Optimization of Solar Cells Thermal Uniformity Using Combined...Multi-Objective Optimization of Solar Cells Thermal Uniformity Using Combined...
Multi-Objective Optimization of Solar Cells Thermal Uniformity Using Combined...eArtius, Inc.
 
Iwsm2014 performance measurement for cloud computing applications using iso...
Iwsm2014 performance measurement for cloud computing applications using iso...Iwsm2014 performance measurement for cloud computing applications using iso...
Iwsm2014 performance measurement for cloud computing applications using iso...
Nesma
 
Forecasting database performance
Forecasting database performanceForecasting database performance
Forecasting database performance
Shenglin Du
 
129471717 unit-v
129471717 unit-v129471717 unit-v
129471717 unit-v
homeworkping8
 
Embedded Intro India05
Embedded Intro India05Embedded Intro India05
Embedded Intro India05
Rajesh Gupta
 
Maximizing Data Lake ROI with Data Virtualization: A Technical Demonstration
Maximizing Data Lake ROI with Data Virtualization: A Technical DemonstrationMaximizing Data Lake ROI with Data Virtualization: A Technical Demonstration
Maximizing Data Lake ROI with Data Virtualization: A Technical Demonstration
Denodo
 
Programmable Exascale Supercomputer
Programmable Exascale SupercomputerProgrammable Exascale Supercomputer
Programmable Exascale Supercomputer
Sagar Dolas
 
Apache Beam: A unified model for batch and stream processing data
Apache Beam: A unified model for batch and stream processing dataApache Beam: A unified model for batch and stream processing data
Apache Beam: A unified model for batch and stream processing data
DataWorks Summit/Hadoop Summit
 
Big Data Day LA 2016/ Big Data Track - Portable Stream and Batch Processing w...
Big Data Day LA 2016/ Big Data Track - Portable Stream and Batch Processing w...Big Data Day LA 2016/ Big Data Track - Portable Stream and Batch Processing w...
Big Data Day LA 2016/ Big Data Track - Portable Stream and Batch Processing w...
Data Con LA
 
The Next Generation of Data Processing and Open Source
The Next Generation of Data Processing and Open SourceThe Next Generation of Data Processing and Open Source
The Next Generation of Data Processing and Open Source
DataWorks Summit/Hadoop Summit
 

Similar to Master's Thesis - climateprediction.net: A Cloudy Approach (20)

Google Cloud Computing on Google Developer 2008 Day
Google Cloud Computing on Google Developer 2008 DayGoogle Cloud Computing on Google Developer 2008 Day
Google Cloud Computing on Google Developer 2008 Day
 
Apache Storm
Apache StormApache Storm
Apache Storm
 
High Performance & High Throughput Computing - EUDAT Summer School (Giuseppe ...
High Performance & High Throughput Computing - EUDAT Summer School (Giuseppe ...High Performance & High Throughput Computing - EUDAT Summer School (Giuseppe ...
High Performance & High Throughput Computing - EUDAT Summer School (Giuseppe ...
 
Handout3o
Handout3oHandout3o
Handout3o
 
Intelligent Monitoring
Intelligent MonitoringIntelligent Monitoring
Intelligent Monitoring
 
Using Grid Technologies in the Cloud for High Scalability
Using Grid Technologies in the Cloud for High ScalabilityUsing Grid Technologies in the Cloud for High Scalability
Using Grid Technologies in the Cloud for High Scalability
 
Ssbse10.ppt
Ssbse10.pptSsbse10.ppt
Ssbse10.ppt
 
Exascale Capabl
Exascale CapablExascale Capabl
Exascale Capabl
 
Fast optimization intevacoct6_3final
Fast optimization intevacoct6_3finalFast optimization intevacoct6_3final
Fast optimization intevacoct6_3final
 
Slide 1
Slide 1Slide 1
Slide 1
 
Multi-Objective Optimization of Solar Cells Thermal Uniformity Using Combined...
Multi-Objective Optimization of Solar Cells Thermal Uniformity Using Combined...Multi-Objective Optimization of Solar Cells Thermal Uniformity Using Combined...
Multi-Objective Optimization of Solar Cells Thermal Uniformity Using Combined...
 
Iwsm2014 performance measurement for cloud computing applications using iso...
Iwsm2014 performance measurement for cloud computing applications using iso...Iwsm2014 performance measurement for cloud computing applications using iso...
Iwsm2014 performance measurement for cloud computing applications using iso...
 
Forecasting database performance
Forecasting database performanceForecasting database performance
Forecasting database performance
 
129471717 unit-v
129471717 unit-v129471717 unit-v
129471717 unit-v
 
Embedded Intro India05
Embedded Intro India05Embedded Intro India05
Embedded Intro India05
 
Maximizing Data Lake ROI with Data Virtualization: A Technical Demonstration
Maximizing Data Lake ROI with Data Virtualization: A Technical DemonstrationMaximizing Data Lake ROI with Data Virtualization: A Technical Demonstration
Maximizing Data Lake ROI with Data Virtualization: A Technical Demonstration
 
Programmable Exascale Supercomputer
Programmable Exascale SupercomputerProgrammable Exascale Supercomputer
Programmable Exascale Supercomputer
 
Apache Beam: A unified model for batch and stream processing data
Apache Beam: A unified model for batch and stream processing dataApache Beam: A unified model for batch and stream processing data
Apache Beam: A unified model for batch and stream processing data
 
Big Data Day LA 2016/ Big Data Track - Portable Stream and Batch Processing w...
Big Data Day LA 2016/ Big Data Track - Portable Stream and Batch Processing w...Big Data Day LA 2016/ Big Data Track - Portable Stream and Batch Processing w...
Big Data Day LA 2016/ Big Data Track - Portable Stream and Batch Processing w...
 
The Next Generation of Data Processing and Open Source
The Next Generation of Data Processing and Open SourceThe Next Generation of Data Processing and Open Source
The Next Generation of Data Processing and Open Source
 

Recently uploaded

Fundamentals of Induction Motor Drives.pptx
Fundamentals of Induction Motor Drives.pptxFundamentals of Induction Motor Drives.pptx
Fundamentals of Induction Motor Drives.pptx
manasideore6
 
在线办理(ANU毕业证书)澳洲国立大学毕业证录取通知书一模一样
在线办理(ANU毕业证书)澳洲国立大学毕业证录取通知书一模一样在线办理(ANU毕业证书)澳洲国立大学毕业证录取通知书一模一样
在线办理(ANU毕业证书)澳洲国立大学毕业证录取通知书一模一样
obonagu
 
Water Industry Process Automation and Control Monthly - May 2024.pdf
Water Industry Process Automation and Control Monthly - May 2024.pdfWater Industry Process Automation and Control Monthly - May 2024.pdf
Water Industry Process Automation and Control Monthly - May 2024.pdf
Water Industry Process Automation & Control
 
Forklift Classes Overview by Intella Parts
Forklift Classes Overview by Intella PartsForklift Classes Overview by Intella Parts
Forklift Classes Overview by Intella Parts
Intella Parts
 
KuberTENes Birthday Bash Guadalajara - K8sGPT first impressions
KuberTENes Birthday Bash Guadalajara - K8sGPT first impressionsKuberTENes Birthday Bash Guadalajara - K8sGPT first impressions
KuberTENes Birthday Bash Guadalajara - K8sGPT first impressions
Victor Morales
 
DfMAy 2024 - key insights and contributions
DfMAy 2024 - key insights and contributionsDfMAy 2024 - key insights and contributions
DfMAy 2024 - key insights and contributions
gestioneergodomus
 
AKS UNIVERSITY Satna Final Year Project By OM Hardaha.pdf
AKS UNIVERSITY Satna Final Year Project By OM Hardaha.pdfAKS UNIVERSITY Satna Final Year Project By OM Hardaha.pdf
AKS UNIVERSITY Satna Final Year Project By OM Hardaha.pdf
SamSarthak3
 
RAT: Retrieval Augmented Thoughts Elicit Context-Aware Reasoning in Long-Hori...
RAT: Retrieval Augmented Thoughts Elicit Context-Aware Reasoning in Long-Hori...RAT: Retrieval Augmented Thoughts Elicit Context-Aware Reasoning in Long-Hori...
RAT: Retrieval Augmented Thoughts Elicit Context-Aware Reasoning in Long-Hori...
thanhdowork
 
一比一原版(IIT毕业证)伊利诺伊理工大学毕业证成绩单专业办理
一比一原版(IIT毕业证)伊利诺伊理工大学毕业证成绩单专业办理一比一原版(IIT毕业证)伊利诺伊理工大学毕业证成绩单专业办理
一比一原版(IIT毕业证)伊利诺伊理工大学毕业证成绩单专业办理
zwunae
 
一比一原版(UofT毕业证)多伦多大学毕业证成绩单如何办理
一比一原版(UofT毕业证)多伦多大学毕业证成绩单如何办理一比一原版(UofT毕业证)多伦多大学毕业证成绩单如何办理
一比一原版(UofT毕业证)多伦多大学毕业证成绩单如何办理
ydteq
 
Harnessing WebAssembly for Real-time Stateless Streaming Pipelines
Harnessing WebAssembly for Real-time Stateless Streaming PipelinesHarnessing WebAssembly for Real-time Stateless Streaming Pipelines
Harnessing WebAssembly for Real-time Stateless Streaming Pipelines
Christina Lin
 
Literature Review Basics and Understanding Reference Management.pptx
Literature Review Basics and Understanding Reference Management.pptxLiterature Review Basics and Understanding Reference Management.pptx
Literature Review Basics and Understanding Reference Management.pptx
Dr Ramhari Poudyal
 
14 Template Contractual Notice - EOT Application
14 Template Contractual Notice - EOT Application14 Template Contractual Notice - EOT Application
14 Template Contractual Notice - EOT Application
SyedAbiiAzazi1
 
Governing Equations for Fundamental Aerodynamics_Anderson2010.pdf
Governing Equations for Fundamental Aerodynamics_Anderson2010.pdfGoverning Equations for Fundamental Aerodynamics_Anderson2010.pdf
Governing Equations for Fundamental Aerodynamics_Anderson2010.pdf
WENKENLI1
 
6th International Conference on Machine Learning & Applications (CMLA 2024)
6th International Conference on Machine Learning & Applications (CMLA 2024)6th International Conference on Machine Learning & Applications (CMLA 2024)
6th International Conference on Machine Learning & Applications (CMLA 2024)
ClaraZara1
 
An Approach to Detecting Writing Styles Based on Clustering Techniques
An Approach to Detecting Writing Styles Based on Clustering TechniquesAn Approach to Detecting Writing Styles Based on Clustering Techniques
An Approach to Detecting Writing Styles Based on Clustering Techniques
ambekarshweta25
 
一比一原版(Otago毕业证)奥塔哥大学毕业证成绩单如何办理
一比一原版(Otago毕业证)奥塔哥大学毕业证成绩单如何办理一比一原版(Otago毕业证)奥塔哥大学毕业证成绩单如何办理
一比一原版(Otago毕业证)奥塔哥大学毕业证成绩单如何办理
dxobcob
 
PPT on GRP pipes manufacturing and testing
PPT on GRP pipes manufacturing and testingPPT on GRP pipes manufacturing and testing
PPT on GRP pipes manufacturing and testing
anoopmanoharan2
 
Heap Sort (SS).ppt FOR ENGINEERING GRADUATES, BCA, MCA, MTECH, BSC STUDENTS
Heap Sort (SS).ppt FOR ENGINEERING GRADUATES, BCA, MCA, MTECH, BSC STUDENTSHeap Sort (SS).ppt FOR ENGINEERING GRADUATES, BCA, MCA, MTECH, BSC STUDENTS
Heap Sort (SS).ppt FOR ENGINEERING GRADUATES, BCA, MCA, MTECH, BSC STUDENTS
Soumen Santra
 
Understanding Inductive Bias in Machine Learning
Understanding Inductive Bias in Machine LearningUnderstanding Inductive Bias in Machine Learning
Understanding Inductive Bias in Machine Learning
SUTEJAS
 

Recently uploaded (20)

Fundamentals of Induction Motor Drives.pptx
Fundamentals of Induction Motor Drives.pptxFundamentals of Induction Motor Drives.pptx
Fundamentals of Induction Motor Drives.pptx
 
在线办理(ANU毕业证书)澳洲国立大学毕业证录取通知书一模一样
在线办理(ANU毕业证书)澳洲国立大学毕业证录取通知书一模一样在线办理(ANU毕业证书)澳洲国立大学毕业证录取通知书一模一样
在线办理(ANU毕业证书)澳洲国立大学毕业证录取通知书一模一样
 
Water Industry Process Automation and Control Monthly - May 2024.pdf
Water Industry Process Automation and Control Monthly - May 2024.pdfWater Industry Process Automation and Control Monthly - May 2024.pdf
Water Industry Process Automation and Control Monthly - May 2024.pdf
 
Forklift Classes Overview by Intella Parts
Forklift Classes Overview by Intella PartsForklift Classes Overview by Intella Parts
Forklift Classes Overview by Intella Parts
 
KuberTENes Birthday Bash Guadalajara - K8sGPT first impressions
KuberTENes Birthday Bash Guadalajara - K8sGPT first impressionsKuberTENes Birthday Bash Guadalajara - K8sGPT first impressions
KuberTENes Birthday Bash Guadalajara - K8sGPT first impressions
 
DfMAy 2024 - key insights and contributions
DfMAy 2024 - key insights and contributionsDfMAy 2024 - key insights and contributions
DfMAy 2024 - key insights and contributions
 
AKS UNIVERSITY Satna Final Year Project By OM Hardaha.pdf
AKS UNIVERSITY Satna Final Year Project By OM Hardaha.pdfAKS UNIVERSITY Satna Final Year Project By OM Hardaha.pdf
AKS UNIVERSITY Satna Final Year Project By OM Hardaha.pdf
 
RAT: Retrieval Augmented Thoughts Elicit Context-Aware Reasoning in Long-Hori...
RAT: Retrieval Augmented Thoughts Elicit Context-Aware Reasoning in Long-Hori...RAT: Retrieval Augmented Thoughts Elicit Context-Aware Reasoning in Long-Hori...
RAT: Retrieval Augmented Thoughts Elicit Context-Aware Reasoning in Long-Hori...
 
一比一原版(IIT毕业证)伊利诺伊理工大学毕业证成绩单专业办理
一比一原版(IIT毕业证)伊利诺伊理工大学毕业证成绩单专业办理一比一原版(IIT毕业证)伊利诺伊理工大学毕业证成绩单专业办理
一比一原版(IIT毕业证)伊利诺伊理工大学毕业证成绩单专业办理
 
一比一原版(UofT毕业证)多伦多大学毕业证成绩单如何办理
一比一原版(UofT毕业证)多伦多大学毕业证成绩单如何办理一比一原版(UofT毕业证)多伦多大学毕业证成绩单如何办理
一比一原版(UofT毕业证)多伦多大学毕业证成绩单如何办理
 
Harnessing WebAssembly for Real-time Stateless Streaming Pipelines
Harnessing WebAssembly for Real-time Stateless Streaming PipelinesHarnessing WebAssembly for Real-time Stateless Streaming Pipelines
Harnessing WebAssembly for Real-time Stateless Streaming Pipelines
 
Literature Review Basics and Understanding Reference Management.pptx
Literature Review Basics and Understanding Reference Management.pptxLiterature Review Basics and Understanding Reference Management.pptx
Literature Review Basics and Understanding Reference Management.pptx
 
14 Template Contractual Notice - EOT Application
14 Template Contractual Notice - EOT Application14 Template Contractual Notice - EOT Application
14 Template Contractual Notice - EOT Application
 
Governing Equations for Fundamental Aerodynamics_Anderson2010.pdf
Governing Equations for Fundamental Aerodynamics_Anderson2010.pdfGoverning Equations for Fundamental Aerodynamics_Anderson2010.pdf
Governing Equations for Fundamental Aerodynamics_Anderson2010.pdf
 
6th International Conference on Machine Learning & Applications (CMLA 2024)
6th International Conference on Machine Learning & Applications (CMLA 2024)6th International Conference on Machine Learning & Applications (CMLA 2024)
6th International Conference on Machine Learning & Applications (CMLA 2024)
 
An Approach to Detecting Writing Styles Based on Clustering Techniques
An Approach to Detecting Writing Styles Based on Clustering TechniquesAn Approach to Detecting Writing Styles Based on Clustering Techniques
An Approach to Detecting Writing Styles Based on Clustering Techniques
 
一比一原版(Otago毕业证)奥塔哥大学毕业证成绩单如何办理
一比一原版(Otago毕业证)奥塔哥大学毕业证成绩单如何办理一比一原版(Otago毕业证)奥塔哥大学毕业证成绩单如何办理
一比一原版(Otago毕业证)奥塔哥大学毕业证成绩单如何办理
 
PPT on GRP pipes manufacturing and testing
PPT on GRP pipes manufacturing and testingPPT on GRP pipes manufacturing and testing
PPT on GRP pipes manufacturing and testing
 
Heap Sort (SS).ppt FOR ENGINEERING GRADUATES, BCA, MCA, MTECH, BSC STUDENTS
Heap Sort (SS).ppt FOR ENGINEERING GRADUATES, BCA, MCA, MTECH, BSC STUDENTSHeap Sort (SS).ppt FOR ENGINEERING GRADUATES, BCA, MCA, MTECH, BSC STUDENTS
Heap Sort (SS).ppt FOR ENGINEERING GRADUATES, BCA, MCA, MTECH, BSC STUDENTS
 
Understanding Inductive Bias in Machine Learning
Understanding Inductive Bias in Machine LearningUnderstanding Inductive Bias in Machine Learning
Understanding Inductive Bias in Machine Learning
 

Master's Thesis - climateprediction.net: A Cloudy Approach

  • 1. Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions climateprediction.net: A Cloudy Approach Master in High Performance Computing Master’s Thesis Diego P´erez Montes advised by Tom´as Fern´andez Pena Juan Antonio A˜nel Cabanelas July 1, 2014 Diego P´erez Montes climateprediction.net: A Cloudy Approach
  • 2. Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions 1 Problem Background Current Infrastructure Problem Description 2 Computing Infrastructure Migration Measuring the Problem... Infrastructure Redesign 3 Storage 4 Central Control System Backend Components Dashboard Running the Simulation 5 Conclusions Diego P´erez Montes climateprediction.net: A Cloudy Approach
  • 3. Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Motivation Solve a real problem, useful for someone and that can be expanded in further works. Apply what I’ve learned in the Master courses. I do love large infrastructure problems (and this is a big one!). Diego P´erez Montes climateprediction.net: A Cloudy Approach
  • 4. Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Current Infrastructure First of all: How does the project currently work? Diego P´erez Montes climateprediction.net: A Cloudy Approach
  • 5. Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Current Infrastructure Figure : BOINC: High Level Architecture and Workflow Diego P´erez Montes climateprediction.net: A Cloudy Approach
  • 6. Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Problem Description So, what is the problem then? The need of execution of a new model (HadGEM) The resources requirements are higher (Hardware: Computing and Storage). The current BOINC workunit processing time is 7-9 days , this wants to be reduced. Heterogeneous and unpredictable environment: Can’t manage resources on-demand. Execution time can’t be properly measured. Processed data is missing. Diego P´erez Montes climateprediction.net: A Cloudy Approach
  • 7. Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Problem Description So, what is the problem then? Need to establish metrics on the project. Rationalization of costs (how much does a simulation really cost?) Diego P´erez Montes climateprediction.net: A Cloudy Approach
  • 8. Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Project Objectives How is it going to be solved? Conversion to an Infrastructure as a Service (Iaas) in the Cloud (Amazon Web Services AWS: EC2 for Computing and S3 for Storage). Creation of a new abstraction layer, the Central Control System: Infrastructure and resources management. Creation of metrics and statistics. Free Software. Fully documented. Diego P´erez Montes climateprediction.net: A Cloudy Approach
  • 9. Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Measuring the Problem... Unknown problem real size and how is it going to behave into the new environment with the new parametrization. Initial data from the current infrastructure over BOINC (Computing point of view): A workunit takes in average from 7 to 9 days to be processed. A full simulation is (minimum) 36,000 workunits into sections of 6,000. Diego P´erez Montes climateprediction.net: A Cloudy Approach
  • 10. Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Measuring the Problem... Initial considerations: Models used on the tests: weather@homeUK floods and and weather@home Australia New Zealand (full and regional: HaDAM3P and HadRM3P) Two representative systems (on EC2) were selected and 10 consecutive executions were done. Diego P´erez Montes climateprediction.net: A Cloudy Approach
  • 11. Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Measuring the Problem... System #1: Moderate CPU CPU: 2 x Xeon E5-2650 MEM: 8GB (4GB/Core) GPU: No Workunit Time: 7.32 days Workunit Cost: USD 4.464 Full Simulation Cost: USD 160,704 Diego P´erez Montes climateprediction.net: A Cloudy Approach
  • 12. Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Measuring the Problem... System #2: Intensive CPU&GPU CPU: 16 x Xeon X5570 MEM: 24GB (1.5GB/Core) GPU: 2 x Tesla M2050 Workunit Time: 1.99 days Workunit Cost: USD 100.966 Full Simulation Cost: USD 3,634,776 Diego P´erez Montes climateprediction.net: A Cloudy Approach
  • 13. Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Measuring the Problem... Diego P´erez Montes climateprediction.net: A Cloudy Approach
  • 14. Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions How much does it really cost? Diego P´erez Montes climateprediction.net: A Cloudy Approach
  • 15. Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Going IaaS Figure : Proposed Infrastructure Diego P´erez Montes climateprediction.net: A Cloudy Approach
  • 16. Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Going IaaS Steps: 1 Template an instance: Install Operating System (Amazon Linux 2014.03.1 64bit) Configure network and firewall. Configure local storage: 16GB Install and configure BOINC to use climateprediction.net Install local client (Simulation Terminator) 2 Contextualize and scale. Diego P´erez Montes climateprediction.net: A Cloudy Approach
  • 17. Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Storage Every simulation (36,000 workunits) outputs 3.6 TB of data. There are not enough resources (disk space) on the current systems. Figure : Shared Storage Architecture Diego P´erez Montes climateprediction.net: A Cloudy Approach
  • 18. Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Architecture Figure : Central System Architecture Diego P´erez Montes climateprediction.net: A Cloudy Approach
  • 19. Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Backend Components Simple Scheduler: Runs and configures simulation with given parameters (start/stop instances). Reaper: Releases resources (terminates instances) when they are powered off. RESTful API: Gives access to configure and run simulations. Diego P´erez Montes climateprediction.net: A Cloudy Approach
  • 20. Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions API RESTful API Get simulation status. Get metric/statistic data. Set/modify simulation parameters (number of worker nodes/instances). Stop simulation. Diego P´erez Montes climateprediction.net: A Cloudy Approach
  • 21. Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Dashboard Figure : Dashboard Interface Diego P´erez Montes climateprediction.net: A Cloudy Approach
  • 22. Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Running the Simulation [Overview of a Live System] Diego P´erez Montes climateprediction.net: A Cloudy Approach
  • 23. Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Conclusions Objectives Achieved Computing and Storage successfully migrated to the Cloud (EC2 and S3). Simulations were executed, showing that running the model in the cloud is possible. Development and a Central System (scheduler and dashboard). Got costs and metrics of the project. Diego P´erez Montes climateprediction.net: A Cloudy Approach
  • 24. Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Conclusions What’s Next? Migrate BOINC server. More control/interaction with clients so the scheduler can be improved (and give a full SaaS layer). Costs: “warm up“ stage to dynamically recalculate price. Diego P´erez Montes climateprediction.net: A Cloudy Approach
  • 25. Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Thanks! Diego P´erez Montes climateprediction.net: A Cloudy Approach
  • 26. Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Used Icons Links Iconset Windows 8 metro style: https://www.iconfinder. com/iconsets/windows-8-metro-style Link: http://sta.sh/0228t4fyjyjb Diego P´erez Montes climateprediction.net: A Cloudy Approach