SlideShare a Scribd company logo

Master's Thesis - climateprediction.net: A Cloudy Approach

•
•
K
kabute

Master's Thesis - climateprediction.net: A Cloudy Approach

EngineeringTechnologyBusiness
1 of 26
Download to read offline
Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions climateprediction.net: A Cloudy Approach Master in High Performance Computing Master’s Thesis Diego P´erez Montes advised by Tom´as Fern´andez Pena Juan Antonio A˜nel Cabanelas July 1, 2014 Diego P´erez Montes climateprediction.net: A Cloudy Approach
Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions 1 Problem Background Current Infrastructure Problem Description 2 Computing Infrastructure Migration Measuring the Problem... Infrastructure Redesign 3 Storage 4 Central Control System Backend Components Dashboard Running the Simulation 5 Conclusions Diego P´erez Montes climateprediction.net: A Cloudy Approach
Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Motivation Solve a real problem, useful for someone and that can be expanded in further works. Apply what I’ve learned in the Master courses. I do love large infrastructure problems (and this is a big one!). Diego P´erez Montes climateprediction.net: A Cloudy Approach
Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Current Infrastructure First of all: How does the project currently work? Diego P´erez Montes climateprediction.net: A Cloudy Approach
Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Current Infrastructure Figure : BOINC: High Level Architecture and Workflow Diego P´erez Montes climateprediction.net: A Cloudy Approach
Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Problem Description So, what is the problem then? The need of execution of a new model (HadGEM) The resources requirements are higher (Hardware: Computing and Storage). The current BOINC workunit processing time is 7-9 days , this wants to be reduced. Heterogeneous and unpredictable environment: Can’t manage resources on-demand. Execution time can’t be properly measured. Processed data is missing. Diego P´erez Montes climateprediction.net: A Cloudy Approach
Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Problem Description So, what is the problem then? Need to establish metrics on the project. Rationalization of costs (how much does a simulation really cost?) Diego P´erez Montes climateprediction.net: A Cloudy Approach
Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Project Objectives How is it going to be solved? Conversion to an Infrastructure as a Service (Iaas) in the Cloud (Amazon Web Services AWS: EC2 for Computing and S3 for Storage). Creation of a new abstraction layer, the Central Control System: Infrastructure and resources management. Creation of metrics and statistics. Free Software. Fully documented. Diego P´erez Montes climateprediction.net: A Cloudy Approach
Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Measuring the Problem... Unknown problem real size and how is it going to behave into the new environment with the new parametrization. Initial data from the current infrastructure over BOINC (Computing point of view): A workunit takes in average from 7 to 9 days to be processed. A full simulation is (minimum) 36,000 workunits into sections of 6,000. Diego P´erez Montes climateprediction.net: A Cloudy Approach
Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Measuring the Problem... Initial considerations: Models used on the tests: weather@homeUK floods and and weather@home Australia New Zealand (full and regional: HaDAM3P and HadRM3P) Two representative systems (on EC2) were selected and 10 consecutive executions were done. Diego P´erez Montes climateprediction.net: A Cloudy Approach
Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Measuring the Problem... System #1: Moderate CPU CPU: 2 x Xeon E5-2650 MEM: 8GB (4GB/Core) GPU: No Workunit Time: 7.32 days Workunit Cost: USD 4.464 Full Simulation Cost: USD 160,704 Diego P´erez Montes climateprediction.net: A Cloudy Approach
Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Measuring the Problem... System #2: Intensive CPU&GPU CPU: 16 x Xeon X5570 MEM: 24GB (1.5GB/Core) GPU: 2 x Tesla M2050 Workunit Time: 1.99 days Workunit Cost: USD 100.966 Full Simulation Cost: USD 3,634,776 Diego P´erez Montes climateprediction.net: A Cloudy Approach
Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Measuring the Problem... Diego P´erez Montes climateprediction.net: A Cloudy Approach
Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions How much does it really cost? Diego P´erez Montes climateprediction.net: A Cloudy Approach
Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Going IaaS Figure : Proposed Infrastructure Diego P´erez Montes climateprediction.net: A Cloudy Approach
Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Going IaaS Steps: 1 Template an instance: Install Operating System (Amazon Linux 2014.03.1 64bit) Configure network and firewall. Configure local storage: 16GB Install and configure BOINC to use climateprediction.net Install local client (Simulation Terminator) 2 Contextualize and scale. Diego P´erez Montes climateprediction.net: A Cloudy Approach
Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Storage Every simulation (36,000 workunits) outputs 3.6 TB of data. There are not enough resources (disk space) on the current systems. Figure : Shared Storage Architecture Diego P´erez Montes climateprediction.net: A Cloudy Approach
Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Architecture Figure : Central System Architecture Diego P´erez Montes climateprediction.net: A Cloudy Approach
Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Backend Components Simple Scheduler: Runs and configures simulation with given parameters (start/stop instances). Reaper: Releases resources (terminates instances) when they are powered off. RESTful API: Gives access to configure and run simulations. Diego P´erez Montes climateprediction.net: A Cloudy Approach
Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions API RESTful API Get simulation status. Get metric/statistic data. Set/modify simulation parameters (number of worker nodes/instances). Stop simulation. Diego P´erez Montes climateprediction.net: A Cloudy Approach
Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Dashboard Figure : Dashboard Interface Diego P´erez Montes climateprediction.net: A Cloudy Approach
Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Running the Simulation [Overview of a Live System] Diego P´erez Montes climateprediction.net: A Cloudy Approach
Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Conclusions Objectives Achieved Computing and Storage successfully migrated to the Cloud (EC2 and S3). Simulations were executed, showing that running the model in the cloud is possible. Development and a Central System (scheduler and dashboard). Got costs and metrics of the project. Diego P´erez Montes climateprediction.net: A Cloudy Approach
Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Conclusions What’s Next? Migrate BOINC server. More control/interaction with clients so the scheduler can be improved (and give a full SaaS layer). Costs: “warm up“ stage to dynamically recalculate price. Diego P´erez Montes climateprediction.net: A Cloudy Approach
Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Thanks! Diego P´erez Montes climateprediction.net: A Cloudy Approach
Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Used Icons Links Iconset Windows 8 metro style: https://www.iconfinder. com/iconsets/windows-8-metro-style Link: http://sta.sh/0228t4fyjyjb Diego P´erez Montes climateprediction.net: A Cloudy Approach

Recommended

GRIMES_Visualizing_Telemetry
GRIMES_Visualizing_TelemetryGRIMES_Visualizing_Telemetry
GRIMES_Visualizing_TelemetryKevin Grimes
 
Vizualize 300 tb in less than 5_seconds
Vizualize 300 tb in less than 5_secondsVizualize 300 tb in less than 5_seconds
Vizualize 300 tb in less than 5_secondsYaniv Shalev
 
SkyhookDM - Towards an Arrow-Native Storage System
SkyhookDM - Towards an Arrow-Native Storage SystemSkyhookDM - Towards an Arrow-Native Storage System
SkyhookDM - Towards an Arrow-Native Storage SystemJayjeetChakraborty
 
Demonstrating a Pre-Exascale, Cost-Effective Multi-Cloud Environment for Scie...
Demonstrating a Pre-Exascale, Cost-Effective Multi-Cloud Environment for Scie...Demonstrating a Pre-Exascale, Cost-Effective Multi-Cloud Environment for Scie...
Demonstrating a Pre-Exascale, Cost-Effective Multi-Cloud Environment for Scie...Igor Sfiligoi
 
Running a GPU burst for Multi-Messenger Astrophysics with IceCube across all ...
Running a GPU burst for Multi-Messenger Astrophysics with IceCube across all ...Running a GPU burst for Multi-Messenger Astrophysics with IceCube across all ...
Running a GPU burst for Multi-Messenger Astrophysics with IceCube across all ...Igor Sfiligoi
 
Building a Better TSA Screening Algorithm
Building a Better TSA Screening AlgorithmBuilding a Better TSA Screening Algorithm
Building a Better TSA Screening AlgorithmYvonne K. Matos
 
"Building and running the cloud GPU vacuum cleaner"
"Building and running the cloud GPU vacuum cleaner""Building and running the cloud GPU vacuum cleaner"
"Building and running the cloud GPU vacuum cleaner"Frank Wuerthwein
 
"Approaches for Energy Efficient Implementation of Deep Neural Networks," a P...
"Approaches for Energy Efficient Implementation of Deep Neural Networks," a P..."Approaches for Energy Efficient Implementation of Deep Neural Networks," a P...
"Approaches for Energy Efficient Implementation of Deep Neural Networks," a P...Edge AI and Vision Alliance
 

Recommended

GRIMES_Visualizing_Telemetry
GRIMES_Visualizing_TelemetryGRIMES_Visualizing_Telemetry
GRIMES_Visualizing_TelemetryKevin Grimes
 
Vizualize 300 tb in less than 5_seconds
Vizualize 300 tb in less than 5_secondsVizualize 300 tb in less than 5_seconds
Vizualize 300 tb in less than 5_secondsYaniv Shalev
 
SkyhookDM - Towards an Arrow-Native Storage System
SkyhookDM - Towards an Arrow-Native Storage SystemSkyhookDM - Towards an Arrow-Native Storage System
SkyhookDM - Towards an Arrow-Native Storage SystemJayjeetChakraborty
 
Demonstrating a Pre-Exascale, Cost-Effective Multi-Cloud Environment for Scie...
Demonstrating a Pre-Exascale, Cost-Effective Multi-Cloud Environment for Scie...Demonstrating a Pre-Exascale, Cost-Effective Multi-Cloud Environment for Scie...
Demonstrating a Pre-Exascale, Cost-Effective Multi-Cloud Environment for Scie...Igor Sfiligoi
 
Running a GPU burst for Multi-Messenger Astrophysics with IceCube across all ...
Running a GPU burst for Multi-Messenger Astrophysics with IceCube across all ...Running a GPU burst for Multi-Messenger Astrophysics with IceCube across all ...
Running a GPU burst for Multi-Messenger Astrophysics with IceCube across all ...Igor Sfiligoi
 
Building a Better TSA Screening Algorithm
Building a Better TSA Screening AlgorithmBuilding a Better TSA Screening Algorithm
Building a Better TSA Screening AlgorithmYvonne K. Matos
 
"Building and running the cloud GPU vacuum cleaner"
"Building and running the cloud GPU vacuum cleaner""Building and running the cloud GPU vacuum cleaner"
"Building and running the cloud GPU vacuum cleaner"Frank Wuerthwein
 
"Approaches for Energy Efficient Implementation of Deep Neural Networks," a P...
"Approaches for Energy Efficient Implementation of Deep Neural Networks," a P..."Approaches for Energy Efficient Implementation of Deep Neural Networks," a P...
"Approaches for Energy Efficient Implementation of Deep Neural Networks," a P...Edge AI and Vision Alliance
 
Data-intensive IceCube Cloud Burst
Data-intensive IceCube Cloud BurstData-intensive IceCube Cloud Burst
Data-intensive IceCube Cloud BurstIgor Sfiligoi
 
Burst data retrieval after 50k GPU Cloud run
Burst data retrieval after 50k GPU Cloud runBurst data retrieval after 50k GPU Cloud run
Burst data retrieval after 50k GPU Cloud runIgor Sfiligoi
 
NRP Engagement webinar - Running a 51k GPU multi-cloud burst for MMA with Ic...
NRP Engagement webinar - Running a 51k GPU multi-cloud burst for MMA with Ic... NRP Engagement webinar - Running a 51k GPU multi-cloud burst for MMA with Ic...
NRP Engagement webinar - Running a 51k GPU multi-cloud burst for MMA with Ic...Igor Sfiligoi
 
Deep Learning for Computer Vision - PyconDE 2017
Deep Learning for Computer Vision - PyconDE 2017Deep Learning for Computer Vision - PyconDE 2017
Deep Learning for Computer Vision - PyconDE 2017Alex Conway
 
Deep recurrent neutral networks for Sequence Learning in Spark
Deep recurrent neutral networks for Sequence Learning in SparkDeep recurrent neutral networks for Sequence Learning in Spark
Deep recurrent neutral networks for Sequence Learning in SparkDataWorks Summit/Hadoop Summit
 
Big Data in the Real World. Real-time Football Analytics
Big Data in the Real World. Real-time Football AnalyticsBig Data in the Real World. Real-time Football Analytics
Big Data in the Real World. Real-time Football AnalyticsWSO2
 
Message broadcasts and logical time of process
Message broadcasts and logical time of processMessage broadcasts and logical time of process
Message broadcasts and logical time of processJawid Ahmad Baktash
 
PyConZA'17 Deep Learning for Computer Vision
PyConZA'17 Deep Learning for Computer VisionPyConZA'17 Deep Learning for Computer Vision
PyConZA'17 Deep Learning for Computer VisionAlex Conway
 
Image Classification Done Simply using Keras and TensorFlow
Image Classification Done Simply using Keras and TensorFlow Image Classification Done Simply using Keras and TensorFlow
Image Classification Done Simply using Keras and TensorFlow Rajiv Shah
 
High Throughput Data Analysis
High Throughput Data AnalysisHigh Throughput Data Analysis
High Throughput Data AnalysisJ Singh
 
Machine Learning Tokyo - Deep Neural Networks for Video - NumberBoost
Machine Learning Tokyo - Deep Neural Networks for Video - NumberBoostMachine Learning Tokyo - Deep Neural Networks for Video - NumberBoost
Machine Learning Tokyo - Deep Neural Networks for Video - NumberBoostAlex Conway
 
Deep Neural Networks for Computer Vision
Deep Neural Networks for Computer VisionDeep Neural Networks for Computer Vision
Deep Neural Networks for Computer VisionAlex Conway
 
Noha danms13 talk_final
Noha danms13 talk_finalNoha danms13 talk_final
Noha danms13 talk_finalNoha Elprince
 
Convolutional Neural Networks for Image Classification (Cape Town Deep Learni...
Convolutional Neural Networks for Image Classification (Cape Town Deep Learni...Convolutional Neural Networks for Image Classification (Cape Town Deep Learni...
Convolutional Neural Networks for Image Classification (Cape Town Deep Learni...Alex Conway
 
Real-time Big Data Processing with Storm
Real-time Big Data Processing with StormReal-time Big Data Processing with Storm
Real-time Big Data Processing with Stormviirya
 
PyDresden 20170824 - Deep Learning for Computer Vision
PyDresden 20170824 - Deep Learning for Computer VisionPyDresden 20170824 - Deep Learning for Computer Vision
PyDresden 20170824 - Deep Learning for Computer VisionAlex Conway
 
KEK helps scientists uncover the mysteries of the universe with Lenovo superc...
KEK helps scientists uncover the mysteries of the universe with Lenovo superc...KEK helps scientists uncover the mysteries of the universe with Lenovo superc...
KEK helps scientists uncover the mysteries of the universe with Lenovo superc...Lenovo Data Center
 
Automatic Features Generation And Model Training On Spark: A Bayesian Approach
Automatic Features Generation And Model Training On Spark: A Bayesian ApproachAutomatic Features Generation And Model Training On Spark: A Bayesian Approach
Automatic Features Generation And Model Training On Spark: A Bayesian ApproachSpark Summit
 
deep reinforcement learning with double q learning
deep reinforcement learning with double q learningdeep reinforcement learning with double q learning
deep reinforcement learning with double q learningSeungHyeok Baek
 
Deep Reinforcement Learning: Q-Learning
Deep Reinforcement Learning: Q-LearningDeep Reinforcement Learning: Q-Learning
Deep Reinforcement Learning: Q-LearningKai-Wen Zhao
 
Elm 0.17 at Dublin Elm Meetup May 2016
Elm 0.17 at Dublin Elm Meetup May 2016Elm 0.17 at Dublin Elm Meetup May 2016
Elm 0.17 at Dublin Elm Meetup May 2016Michael Twomey
 
Mesilane Lendab.Multifilm
Mesilane Lendab.MultifilmMesilane Lendab.Multifilm
Mesilane Lendab.Multifilmpillilood53
 

More Related Content

What's hot

Data-intensive IceCube Cloud Burst
Data-intensive IceCube Cloud BurstData-intensive IceCube Cloud Burst
Data-intensive IceCube Cloud BurstIgor Sfiligoi
 
Burst data retrieval after 50k GPU Cloud run
Burst data retrieval after 50k GPU Cloud runBurst data retrieval after 50k GPU Cloud run
Burst data retrieval after 50k GPU Cloud runIgor Sfiligoi
 
NRP Engagement webinar - Running a 51k GPU multi-cloud burst for MMA with Ic...
NRP Engagement webinar - Running a 51k GPU multi-cloud burst for MMA with Ic... NRP Engagement webinar - Running a 51k GPU multi-cloud burst for MMA with Ic...
NRP Engagement webinar - Running a 51k GPU multi-cloud burst for MMA with Ic...Igor Sfiligoi
 
Deep Learning for Computer Vision - PyconDE 2017
Deep Learning for Computer Vision - PyconDE 2017Deep Learning for Computer Vision - PyconDE 2017
Deep Learning for Computer Vision - PyconDE 2017Alex Conway
 
Deep recurrent neutral networks for Sequence Learning in Spark
Deep recurrent neutral networks for Sequence Learning in SparkDeep recurrent neutral networks for Sequence Learning in Spark
Deep recurrent neutral networks for Sequence Learning in SparkDataWorks Summit/Hadoop Summit
 
Big Data in the Real World. Real-time Football Analytics
Big Data in the Real World. Real-time Football AnalyticsBig Data in the Real World. Real-time Football Analytics
Big Data in the Real World. Real-time Football AnalyticsWSO2
 
Message broadcasts and logical time of process
Message broadcasts and logical time of processMessage broadcasts and logical time of process
Message broadcasts and logical time of processJawid Ahmad Baktash
 
PyConZA'17 Deep Learning for Computer Vision
PyConZA'17 Deep Learning for Computer VisionPyConZA'17 Deep Learning for Computer Vision
PyConZA'17 Deep Learning for Computer VisionAlex Conway
 
Image Classification Done Simply using Keras and TensorFlow
Image Classification Done Simply using Keras and TensorFlow Image Classification Done Simply using Keras and TensorFlow
Image Classification Done Simply using Keras and TensorFlow Rajiv Shah
 
High Throughput Data Analysis
High Throughput Data AnalysisHigh Throughput Data Analysis
High Throughput Data AnalysisJ Singh
 
Machine Learning Tokyo - Deep Neural Networks for Video - NumberBoost
Machine Learning Tokyo - Deep Neural Networks for Video - NumberBoostMachine Learning Tokyo - Deep Neural Networks for Video - NumberBoost
Machine Learning Tokyo - Deep Neural Networks for Video - NumberBoostAlex Conway
 
Deep Neural Networks for Computer Vision
Deep Neural Networks for Computer VisionDeep Neural Networks for Computer Vision
Deep Neural Networks for Computer VisionAlex Conway
 
Noha danms13 talk_final
Noha danms13 talk_finalNoha danms13 talk_final
Noha danms13 talk_finalNoha Elprince
 
Convolutional Neural Networks for Image Classification (Cape Town Deep Learni...
Convolutional Neural Networks for Image Classification (Cape Town Deep Learni...Convolutional Neural Networks for Image Classification (Cape Town Deep Learni...
Convolutional Neural Networks for Image Classification (Cape Town Deep Learni...Alex Conway
 
Real-time Big Data Processing with Storm
Real-time Big Data Processing with StormReal-time Big Data Processing with Storm
Real-time Big Data Processing with Stormviirya
 
PyDresden 20170824 - Deep Learning for Computer Vision
PyDresden 20170824 - Deep Learning for Computer VisionPyDresden 20170824 - Deep Learning for Computer Vision
PyDresden 20170824 - Deep Learning for Computer VisionAlex Conway
 
KEK helps scientists uncover the mysteries of the universe with Lenovo superc...
KEK helps scientists uncover the mysteries of the universe with Lenovo superc...KEK helps scientists uncover the mysteries of the universe with Lenovo superc...
KEK helps scientists uncover the mysteries of the universe with Lenovo superc...Lenovo Data Center
 
Automatic Features Generation And Model Training On Spark: A Bayesian Approach
Automatic Features Generation And Model Training On Spark: A Bayesian ApproachAutomatic Features Generation And Model Training On Spark: A Bayesian Approach
Automatic Features Generation And Model Training On Spark: A Bayesian ApproachSpark Summit
 
deep reinforcement learning with double q learning
deep reinforcement learning with double q learningdeep reinforcement learning with double q learning
deep reinforcement learning with double q learningSeungHyeok Baek
 
Deep Reinforcement Learning: Q-Learning
Deep Reinforcement Learning: Q-LearningDeep Reinforcement Learning: Q-Learning
Deep Reinforcement Learning: Q-LearningKai-Wen Zhao
 

What's hot (20)

Data-intensive IceCube Cloud Burst
Data-intensive IceCube Cloud BurstData-intensive IceCube Cloud Burst
Data-intensive IceCube Cloud Burst
 
Burst data retrieval after 50k GPU Cloud run
Burst data retrieval after 50k GPU Cloud runBurst data retrieval after 50k GPU Cloud run
Burst data retrieval after 50k GPU Cloud run
 
NRP Engagement webinar - Running a 51k GPU multi-cloud burst for MMA with Ic...
NRP Engagement webinar - Running a 51k GPU multi-cloud burst for MMA with Ic... NRP Engagement webinar - Running a 51k GPU multi-cloud burst for MMA with Ic...
NRP Engagement webinar - Running a 51k GPU multi-cloud burst for MMA with Ic...
 
Deep Learning for Computer Vision - PyconDE 2017
Deep Learning for Computer Vision - PyconDE 2017Deep Learning for Computer Vision - PyconDE 2017
Deep Learning for Computer Vision - PyconDE 2017
 
Deep recurrent neutral networks for Sequence Learning in Spark
Deep recurrent neutral networks for Sequence Learning in SparkDeep recurrent neutral networks for Sequence Learning in Spark
Deep recurrent neutral networks for Sequence Learning in Spark
 
Big Data in the Real World. Real-time Football Analytics
Big Data in the Real World. Real-time Football AnalyticsBig Data in the Real World. Real-time Football Analytics
Big Data in the Real World. Real-time Football Analytics
 
Message broadcasts and logical time of process
Message broadcasts and logical time of processMessage broadcasts and logical time of process
Message broadcasts and logical time of process
 
PyConZA'17 Deep Learning for Computer Vision
PyConZA'17 Deep Learning for Computer VisionPyConZA'17 Deep Learning for Computer Vision
PyConZA'17 Deep Learning for Computer Vision
 
Image Classification Done Simply using Keras and TensorFlow
Image Classification Done Simply using Keras and TensorFlow Image Classification Done Simply using Keras and TensorFlow
Image Classification Done Simply using Keras and TensorFlow
 
High Throughput Data Analysis
High Throughput Data AnalysisHigh Throughput Data Analysis
High Throughput Data Analysis
 
Machine Learning Tokyo - Deep Neural Networks for Video - NumberBoost
Machine Learning Tokyo - Deep Neural Networks for Video - NumberBoostMachine Learning Tokyo - Deep Neural Networks for Video - NumberBoost
Machine Learning Tokyo - Deep Neural Networks for Video - NumberBoost
 
Deep Neural Networks for Computer Vision
Deep Neural Networks for Computer VisionDeep Neural Networks for Computer Vision
Deep Neural Networks for Computer Vision
 
Noha danms13 talk_final
Noha danms13 talk_finalNoha danms13 talk_final
Noha danms13 talk_final
 
Convolutional Neural Networks for Image Classification (Cape Town Deep Learni...
Convolutional Neural Networks for Image Classification (Cape Town Deep Learni...Convolutional Neural Networks for Image Classification (Cape Town Deep Learni...
Convolutional Neural Networks for Image Classification (Cape Town Deep Learni...
 
Real-time Big Data Processing with Storm
Real-time Big Data Processing with StormReal-time Big Data Processing with Storm
Real-time Big Data Processing with Storm
 
PyDresden 20170824 - Deep Learning for Computer Vision
PyDresden 20170824 - Deep Learning for Computer VisionPyDresden 20170824 - Deep Learning for Computer Vision
PyDresden 20170824 - Deep Learning for Computer Vision
 
KEK helps scientists uncover the mysteries of the universe with Lenovo superc...
KEK helps scientists uncover the mysteries of the universe with Lenovo superc...KEK helps scientists uncover the mysteries of the universe with Lenovo superc...
KEK helps scientists uncover the mysteries of the universe with Lenovo superc...
 
Automatic Features Generation And Model Training On Spark: A Bayesian Approach
Automatic Features Generation And Model Training On Spark: A Bayesian ApproachAutomatic Features Generation And Model Training On Spark: A Bayesian Approach
Automatic Features Generation And Model Training On Spark: A Bayesian Approach
 
deep reinforcement learning with double q learning
deep reinforcement learning with double q learningdeep reinforcement learning with double q learning
deep reinforcement learning with double q learning
 
Deep Reinforcement Learning: Q-Learning
Deep Reinforcement Learning: Q-LearningDeep Reinforcement Learning: Q-Learning
Deep Reinforcement Learning: Q-Learning
 

Viewers also liked

Elm 0.17 at Dublin Elm Meetup May 2016
Elm 0.17 at Dublin Elm Meetup May 2016Elm 0.17 at Dublin Elm Meetup May 2016
Elm 0.17 at Dublin Elm Meetup May 2016Michael Twomey
 
Mesilane Lendab.Multifilm
Mesilane Lendab.MultifilmMesilane Lendab.Multifilm
Mesilane Lendab.Multifilmpillilood53
 
Local Business Online Marketing 101
Local Business Online Marketing 101Local Business Online Marketing 101
Local Business Online Marketing 101Nick Allen
 
LinkedIn Groups - The ins and outs
LinkedIn Groups - The ins and outsLinkedIn Groups - The ins and outs
LinkedIn Groups - The ins and outsNick Allen
 
Mechanics of materials_solution_manual_3rd_ed__by_beer_johnston__dewolf
Mechanics of materials_solution_manual_3rd_ed__by_beer_johnston__dewolfMechanics of materials_solution_manual_3rd_ed__by_beer_johnston__dewolf
Mechanics of materials_solution_manual_3rd_ed__by_beer_johnston__dewolfUniversiti Kuala Lumpur
 

Viewers also liked (9)

Elm 0.17 at Dublin Elm Meetup May 2016
Elm 0.17 at Dublin Elm Meetup May 2016Elm 0.17 at Dublin Elm Meetup May 2016
Elm 0.17 at Dublin Elm Meetup May 2016
 
Mesilane Lendab.Multifilm
Mesilane Lendab.MultifilmMesilane Lendab.Multifilm
Mesilane Lendab.Multifilm
 
PyCon 2010
PyCon 2010PyCon 2010
PyCon 2010
 
Mesilane Lendab
Mesilane LendabMesilane Lendab
Mesilane Lendab
 
Elm @ DublinJS
Elm @ DublinJSElm @ DublinJS
Elm @ DublinJS
 
Local Business Online Marketing 101
Local Business Online Marketing 101Local Business Online Marketing 101
Local Business Online Marketing 101
 
LinkedIn Groups - The ins and outs
LinkedIn Groups - The ins and outsLinkedIn Groups - The ins and outs
LinkedIn Groups - The ins and outs
 
Mechanics of materials_solution_manual_3rd_ed__by_beer_johnston__dewolf
Mechanics of materials_solution_manual_3rd_ed__by_beer_johnston__dewolfMechanics of materials_solution_manual_3rd_ed__by_beer_johnston__dewolf
Mechanics of materials_solution_manual_3rd_ed__by_beer_johnston__dewolf
 
Personality
PersonalityPersonality
Personality
 

Similar to Master's Thesis - climateprediction.net: A Cloudy Approach

Google Cloud Computing on Google Developer 2008 Day
Google Cloud Computing on Google Developer 2008 DayGoogle Cloud Computing on Google Developer 2008 Day
Google Cloud Computing on Google Developer 2008 Dayprogrammermag
 
Apache Storm
Apache StormApache Storm
Apache StormEdureka!
 
High Performance & High Throughput Computing - EUDAT Summer School (Giuseppe ...
High Performance & High Throughput Computing - EUDAT Summer School (Giuseppe ...High Performance & High Throughput Computing - EUDAT Summer School (Giuseppe ...
High Performance & High Throughput Computing - EUDAT Summer School (Giuseppe ...EUDAT
 
Intelligent Monitoring
Intelligent MonitoringIntelligent Monitoring
Intelligent MonitoringIntelie
 
Using Grid Technologies in the Cloud for High Scalability
Using Grid Technologies in the Cloud for High ScalabilityUsing Grid Technologies in the Cloud for High Scalability
Using Grid Technologies in the Cloud for High Scalabilitymabuhr
 
Fast optimization intevacoct6_3final
Fast optimization intevacoct6_3finalFast optimization intevacoct6_3final
Fast optimization intevacoct6_3finaleArtius, Inc.
 
Slide 1
Slide 1Slide 1
Slide 1butest
 
Multi-Objective Optimization of Solar Cells Thermal Uniformity Using Combined...
Multi-Objective Optimization of Solar Cells Thermal Uniformity Using Combined...Multi-Objective Optimization of Solar Cells Thermal Uniformity Using Combined...
Multi-Objective Optimization of Solar Cells Thermal Uniformity Using Combined...eArtius, Inc.
 
Iwsm2014 performance measurement for cloud computing applications using iso...
Iwsm2014 performance measurement for cloud computing applications using iso...Iwsm2014 performance measurement for cloud computing applications using iso...
Iwsm2014 performance measurement for cloud computing applications using iso...Nesma
 
Forecasting database performance
Forecasting database performanceForecasting database performance
Forecasting database performanceShenglin Du
 
Embedded Intro India05
Embedded Intro India05Embedded Intro India05
Embedded Intro India05Rajesh Gupta
 
Maximizing Data Lake ROI with Data Virtualization: A Technical Demonstration
Maximizing Data Lake ROI with Data Virtualization: A Technical DemonstrationMaximizing Data Lake ROI with Data Virtualization: A Technical Demonstration
Maximizing Data Lake ROI with Data Virtualization: A Technical DemonstrationDenodo
 
Programmable Exascale Supercomputer
Programmable Exascale SupercomputerProgrammable Exascale Supercomputer
Programmable Exascale SupercomputerSagar Dolas
 
Apache Beam: A unified model for batch and stream processing data
Apache Beam: A unified model for batch and stream processing dataApache Beam: A unified model for batch and stream processing data
Apache Beam: A unified model for batch and stream processing dataDataWorks Summit/Hadoop Summit
 
Big Data Day LA 2016/ Big Data Track - Portable Stream and Batch Processing w...
Big Data Day LA 2016/ Big Data Track - Portable Stream and Batch Processing w...Big Data Day LA 2016/ Big Data Track - Portable Stream and Batch Processing w...
Big Data Day LA 2016/ Big Data Track - Portable Stream and Batch Processing w...Data Con LA
 
The Next Generation of Data Processing and Open Source
The Next Generation of Data Processing and Open SourceThe Next Generation of Data Processing and Open Source
The Next Generation of Data Processing and Open SourceDataWorks Summit/Hadoop Summit
 

Similar to Master's Thesis - climateprediction.net: A Cloudy Approach (20)

Google Cloud Computing on Google Developer 2008 Day
Google Cloud Computing on Google Developer 2008 DayGoogle Cloud Computing on Google Developer 2008 Day
Google Cloud Computing on Google Developer 2008 Day
 
Apache Storm
Apache StormApache Storm
Apache Storm
 
High Performance & High Throughput Computing - EUDAT Summer School (Giuseppe ...
High Performance & High Throughput Computing - EUDAT Summer School (Giuseppe ...High Performance & High Throughput Computing - EUDAT Summer School (Giuseppe ...
High Performance & High Throughput Computing - EUDAT Summer School (Giuseppe ...
 
Handout3o
Handout3oHandout3o
Handout3o
 
Intelligent Monitoring
Intelligent MonitoringIntelligent Monitoring
Intelligent Monitoring
 
Using Grid Technologies in the Cloud for High Scalability
Using Grid Technologies in the Cloud for High ScalabilityUsing Grid Technologies in the Cloud for High Scalability
Using Grid Technologies in the Cloud for High Scalability
 
Ssbse10.ppt
Ssbse10.pptSsbse10.ppt
Ssbse10.ppt
 
Exascale Capabl
Exascale CapablExascale Capabl
Exascale Capabl
 
Fast optimization intevacoct6_3final
Fast optimization intevacoct6_3finalFast optimization intevacoct6_3final
Fast optimization intevacoct6_3final
 
Slide 1
Slide 1Slide 1
Slide 1
 
Multi-Objective Optimization of Solar Cells Thermal Uniformity Using Combined...
Multi-Objective Optimization of Solar Cells Thermal Uniformity Using Combined...Multi-Objective Optimization of Solar Cells Thermal Uniformity Using Combined...
Multi-Objective Optimization of Solar Cells Thermal Uniformity Using Combined...
 
Iwsm2014 performance measurement for cloud computing applications using iso...
Iwsm2014 performance measurement for cloud computing applications using iso...Iwsm2014 performance measurement for cloud computing applications using iso...
Iwsm2014 performance measurement for cloud computing applications using iso...
 
Forecasting database performance
Forecasting database performanceForecasting database performance
Forecasting database performance
 
129471717 unit-v
129471717 unit-v129471717 unit-v
129471717 unit-v
 
Embedded Intro India05
Embedded Intro India05Embedded Intro India05
Embedded Intro India05
 
Maximizing Data Lake ROI with Data Virtualization: A Technical Demonstration
Maximizing Data Lake ROI with Data Virtualization: A Technical DemonstrationMaximizing Data Lake ROI with Data Virtualization: A Technical Demonstration
Maximizing Data Lake ROI with Data Virtualization: A Technical Demonstration
 
Programmable Exascale Supercomputer
Programmable Exascale SupercomputerProgrammable Exascale Supercomputer
Programmable Exascale Supercomputer
 
Apache Beam: A unified model for batch and stream processing data
Apache Beam: A unified model for batch and stream processing dataApache Beam: A unified model for batch and stream processing data
Apache Beam: A unified model for batch and stream processing data
 
Big Data Day LA 2016/ Big Data Track - Portable Stream and Batch Processing w...
Big Data Day LA 2016/ Big Data Track - Portable Stream and Batch Processing w...Big Data Day LA 2016/ Big Data Track - Portable Stream and Batch Processing w...
Big Data Day LA 2016/ Big Data Track - Portable Stream and Batch Processing w...
 
The Next Generation of Data Processing and Open Source
The Next Generation of Data Processing and Open SourceThe Next Generation of Data Processing and Open Source
The Next Generation of Data Processing and Open Source
 

Recently uploaded

main PPT.pptx of girls hostel security using rfid
main PPT.pptx of girls hostel security using rfidmain PPT.pptx of girls hostel security using rfid
main PPT.pptx of girls hostel security using rfidNikhilNagaraju
 
Introduction to Machine Learning Unit-3 for II MECH
Introduction to Machine Learning Unit-3 for II MECHIntroduction to Machine Learning Unit-3 for II MECH
Introduction to Machine Learning Unit-3 for II MECHC Sai Kiran
 
Study on Air-Water & Water-Water Heat Exchange in a Finned Tube Exchanger
Study on Air-Water & Water-Water Heat Exchange in a Finned Tube ExchangerStudy on Air-Water & Water-Water Heat Exchange in a Finned Tube Exchanger
Study on Air-Water & Water-Water Heat Exchange in a Finned Tube ExchangerAnamika Sarkar
 
Electronically Controlled suspensions system .pdf
Electronically Controlled suspensions system .pdfElectronically Controlled suspensions system .pdf
Electronically Controlled suspensions system .pdfme23b1001
 
EduAI - E learning Platform integrated with AI
EduAI - E learning Platform integrated with AIEduAI - E learning Platform integrated with AI
EduAI - E learning Platform integrated with AIkoyaldeepu123
 
Heart Disease Prediction using machine learning.pptx
Heart Disease Prediction using machine learning.pptxHeart Disease Prediction using machine learning.pptx
Heart Disease Prediction using machine learning.pptxPoojaBan
 
Risk Assessment For Installation of Drainage Pipes.pdf
Risk Assessment For Installation of Drainage Pipes.pdfRisk Assessment For Installation of Drainage Pipes.pdf
Risk Assessment For Installation of Drainage Pipes.pdfROCENODodongVILLACER
 
Call Girls Narol 7397865700 Independent Call Girls
Call Girls Narol 7397865700 Independent Call GirlsCall Girls Narol 7397865700 Independent Call Girls
Call Girls Narol 7397865700 Independent Call Girlsssuser7cb4ff
 
Arduino_CSE ece ppt for working and principal of arduino.ppt
Arduino_CSE ece ppt for working and principal of arduino.pptArduino_CSE ece ppt for working and principal of arduino.ppt
Arduino_CSE ece ppt for working and principal of arduino.pptSAURABHKUMAR892774
 
Oxy acetylene welding presentation note.
Oxy acetylene welding presentation note.Oxy acetylene welding presentation note.
Oxy acetylene welding presentation note.eptoze12
 
Gurgaon ✡️9711147426✨Call In girls Gurgaon Sector 51 escort service
Gurgaon ✡️9711147426✨Call In girls Gurgaon Sector 51 escort serviceGurgaon ✡️9711147426✨Call In girls Gurgaon Sector 51 escort service
Gurgaon ✡️9711147426✨Call In girls Gurgaon Sector 51 escort servicejennyeacort
 
CCS355 Neural Network & Deep Learning Unit II Notes with Question bank .pdf
CCS355 Neural Network & Deep Learning Unit II Notes with Question bank .pdfCCS355 Neural Network & Deep Learning Unit II Notes with Question bank .pdf
CCS355 Neural Network & Deep Learning Unit II Notes with Question bank .pdfAsst.prof M.Gokilavani
 
An experimental study in using natural admixture as an alternative for chemic...
An experimental study in using natural admixture as an alternative for chemic...An experimental study in using natural admixture as an alternative for chemic...
An experimental study in using natural admixture as an alternative for chemic...Chandu841456
 
INFLUENCE OF NANOSILICA ON THE PROPERTIES OF CONCRETE
INFLUENCE OF NANOSILICA ON THE PROPERTIES OF CONCRETEINFLUENCE OF NANOSILICA ON THE PROPERTIES OF CONCRETE
INFLUENCE OF NANOSILICA ON THE PROPERTIES OF CONCRETEroselinkalist12
 
What are the advantages and disadvantages of membrane structures.pptx
What are the advantages and disadvantages of membrane structures.pptxWhat are the advantages and disadvantages of membrane structures.pptx
What are the advantages and disadvantages of membrane structures.pptxwendy cai
 
Internship report on mechanical engineering
Internship report on mechanical engineeringInternship report on mechanical engineering
Internship report on mechanical engineeringmalavadedarshan25
 
UNIT III ANALOG ELECTRONICS (BASIC ELECTRONICS)
UNIT III ANALOG ELECTRONICS (BASIC ELECTRONICS)UNIT III ANALOG ELECTRONICS (BASIC ELECTRONICS)
UNIT III ANALOG ELECTRONICS (BASIC ELECTRONICS)Dr SOUNDIRARAJ N
 

Recently uploaded (20)

main PPT.pptx of girls hostel security using rfid
main PPT.pptx of girls hostel security using rfidmain PPT.pptx of girls hostel security using rfid
main PPT.pptx of girls hostel security using rfid
 
Introduction to Machine Learning Unit-3 for II MECH
Introduction to Machine Learning Unit-3 for II MECHIntroduction to Machine Learning Unit-3 for II MECH
Introduction to Machine Learning Unit-3 for II MECH
 
Study on Air-Water & Water-Water Heat Exchange in a Finned Tube Exchanger
Study on Air-Water & Water-Water Heat Exchange in a Finned Tube ExchangerStudy on Air-Water & Water-Water Heat Exchange in a Finned Tube Exchanger
Study on Air-Water & Water-Water Heat Exchange in a Finned Tube Exchanger
 
Electronically Controlled suspensions system .pdf
Electronically Controlled suspensions system .pdfElectronically Controlled suspensions system .pdf
Electronically Controlled suspensions system .pdf
 
EduAI - E learning Platform integrated with AI
EduAI - E learning Platform integrated with AIEduAI - E learning Platform integrated with AI
EduAI - E learning Platform integrated with AI
 
🔝9953056974🔝!!-YOUNG call girls in Rajendra Nagar Escort rvice Shot 2000 nigh...
🔝9953056974🔝!!-YOUNG call girls in Rajendra Nagar Escort rvice Shot 2000 nigh...🔝9953056974🔝!!-YOUNG call girls in Rajendra Nagar Escort rvice Shot 2000 nigh...
🔝9953056974🔝!!-YOUNG call girls in Rajendra Nagar Escort rvice Shot 2000 nigh...
 
Heart Disease Prediction using machine learning.pptx
Heart Disease Prediction using machine learning.pptxHeart Disease Prediction using machine learning.pptx
Heart Disease Prediction using machine learning.pptx
 
Risk Assessment For Installation of Drainage Pipes.pdf
Risk Assessment For Installation of Drainage Pipes.pdfRisk Assessment For Installation of Drainage Pipes.pdf
Risk Assessment For Installation of Drainage Pipes.pdf
 
Call Girls Narol 7397865700 Independent Call Girls
Call Girls Narol 7397865700 Independent Call GirlsCall Girls Narol 7397865700 Independent Call Girls
Call Girls Narol 7397865700 Independent Call Girls
 
Arduino_CSE ece ppt for working and principal of arduino.ppt
Arduino_CSE ece ppt for working and principal of arduino.pptArduino_CSE ece ppt for working and principal of arduino.ppt
Arduino_CSE ece ppt for working and principal of arduino.ppt
 
Oxy acetylene welding presentation note.
Oxy acetylene welding presentation note.Oxy acetylene welding presentation note.
Oxy acetylene welding presentation note.
 
Gurgaon ✡️9711147426✨Call In girls Gurgaon Sector 51 escort service
Gurgaon ✡️9711147426✨Call In girls Gurgaon Sector 51 escort serviceGurgaon ✡️9711147426✨Call In girls Gurgaon Sector 51 escort service
Gurgaon ✡️9711147426✨Call In girls Gurgaon Sector 51 escort service
 
9953056974 Call Girls In South Ex, Escorts (Delhi) NCR.pdf
9953056974 Call Girls In South Ex, Escorts (Delhi) NCR.pdf9953056974 Call Girls In South Ex, Escorts (Delhi) NCR.pdf
9953056974 Call Girls In South Ex, Escorts (Delhi) NCR.pdf
 
CCS355 Neural Network & Deep Learning Unit II Notes with Question bank .pdf
CCS355 Neural Network & Deep Learning Unit II Notes with Question bank .pdfCCS355 Neural Network & Deep Learning Unit II Notes with Question bank .pdf
CCS355 Neural Network & Deep Learning Unit II Notes with Question bank .pdf
 
An experimental study in using natural admixture as an alternative for chemic...
An experimental study in using natural admixture as an alternative for chemic...An experimental study in using natural admixture as an alternative for chemic...
An experimental study in using natural admixture as an alternative for chemic...
 
INFLUENCE OF NANOSILICA ON THE PROPERTIES OF CONCRETE
INFLUENCE OF NANOSILICA ON THE PROPERTIES OF CONCRETEINFLUENCE OF NANOSILICA ON THE PROPERTIES OF CONCRETE
INFLUENCE OF NANOSILICA ON THE PROPERTIES OF CONCRETE
 
Call Us -/9953056974- Call Girls In Vikaspuri-/- Delhi NCR
Call Us -/9953056974- Call Girls In Vikaspuri-/- Delhi NCRCall Us -/9953056974- Call Girls In Vikaspuri-/- Delhi NCR
Call Us -/9953056974- Call Girls In Vikaspuri-/- Delhi NCR
 
What are the advantages and disadvantages of membrane structures.pptx
What are the advantages and disadvantages of membrane structures.pptxWhat are the advantages and disadvantages of membrane structures.pptx
What are the advantages and disadvantages of membrane structures.pptx
 
Internship report on mechanical engineering
Internship report on mechanical engineeringInternship report on mechanical engineering
Internship report on mechanical engineering
 
UNIT III ANALOG ELECTRONICS (BASIC ELECTRONICS)
UNIT III ANALOG ELECTRONICS (BASIC ELECTRONICS)UNIT III ANALOG ELECTRONICS (BASIC ELECTRONICS)
UNIT III ANALOG ELECTRONICS (BASIC ELECTRONICS)
 

Master's Thesis - climateprediction.net: A Cloudy Approach

  • 1. Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions climateprediction.net: A Cloudy Approach Master in High Performance Computing Master’s Thesis Diego P´erez Montes advised by Tom´as Fern´andez Pena Juan Antonio A˜nel Cabanelas July 1, 2014 Diego P´erez Montes climateprediction.net: A Cloudy Approach
  • 2. Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions 1 Problem Background Current Infrastructure Problem Description 2 Computing Infrastructure Migration Measuring the Problem... Infrastructure Redesign 3 Storage 4 Central Control System Backend Components Dashboard Running the Simulation 5 Conclusions Diego P´erez Montes climateprediction.net: A Cloudy Approach
  • 3. Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Motivation Solve a real problem, useful for someone and that can be expanded in further works. Apply what I’ve learned in the Master courses. I do love large infrastructure problems (and this is a big one!). Diego P´erez Montes climateprediction.net: A Cloudy Approach
  • 4. Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Current Infrastructure First of all: How does the project currently work? Diego P´erez Montes climateprediction.net: A Cloudy Approach
  • 5. Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Current Infrastructure Figure : BOINC: High Level Architecture and Workflow Diego P´erez Montes climateprediction.net: A Cloudy Approach
  • 6. Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Problem Description So, what is the problem then? The need of execution of a new model (HadGEM) The resources requirements are higher (Hardware: Computing and Storage). The current BOINC workunit processing time is 7-9 days , this wants to be reduced. Heterogeneous and unpredictable environment: Can’t manage resources on-demand. Execution time can’t be properly measured. Processed data is missing. Diego P´erez Montes climateprediction.net: A Cloudy Approach
  • 7. Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Problem Description So, what is the problem then? Need to establish metrics on the project. Rationalization of costs (how much does a simulation really cost?) Diego P´erez Montes climateprediction.net: A Cloudy Approach
  • 8. Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Project Objectives How is it going to be solved? Conversion to an Infrastructure as a Service (Iaas) in the Cloud (Amazon Web Services AWS: EC2 for Computing and S3 for Storage). Creation of a new abstraction layer, the Central Control System: Infrastructure and resources management. Creation of metrics and statistics. Free Software. Fully documented. Diego P´erez Montes climateprediction.net: A Cloudy Approach
  • 9. Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Measuring the Problem... Unknown problem real size and how is it going to behave into the new environment with the new parametrization. Initial data from the current infrastructure over BOINC (Computing point of view): A workunit takes in average from 7 to 9 days to be processed. A full simulation is (minimum) 36,000 workunits into sections of 6,000. Diego P´erez Montes climateprediction.net: A Cloudy Approach
  • 10. Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Measuring the Problem... Initial considerations: Models used on the tests: weather@homeUK floods and and weather@home Australia New Zealand (full and regional: HaDAM3P and HadRM3P) Two representative systems (on EC2) were selected and 10 consecutive executions were done. Diego P´erez Montes climateprediction.net: A Cloudy Approach
  • 11. Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Measuring the Problem... System #1: Moderate CPU CPU: 2 x Xeon E5-2650 MEM: 8GB (4GB/Core) GPU: No Workunit Time: 7.32 days Workunit Cost: USD 4.464 Full Simulation Cost: USD 160,704 Diego P´erez Montes climateprediction.net: A Cloudy Approach
  • 12. Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Measuring the Problem... System #2: Intensive CPU&GPU CPU: 16 x Xeon X5570 MEM: 24GB (1.5GB/Core) GPU: 2 x Tesla M2050 Workunit Time: 1.99 days Workunit Cost: USD 100.966 Full Simulation Cost: USD 3,634,776 Diego P´erez Montes climateprediction.net: A Cloudy Approach
  • 13. Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Measuring the Problem... Diego P´erez Montes climateprediction.net: A Cloudy Approach
  • 14. Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions How much does it really cost? Diego P´erez Montes climateprediction.net: A Cloudy Approach
  • 15. Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Going IaaS Figure : Proposed Infrastructure Diego P´erez Montes climateprediction.net: A Cloudy Approach
  • 16. Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Going IaaS Steps: 1 Template an instance: Install Operating System (Amazon Linux 2014.03.1 64bit) Configure network and firewall. Configure local storage: 16GB Install and configure BOINC to use climateprediction.net Install local client (Simulation Terminator) 2 Contextualize and scale. Diego P´erez Montes climateprediction.net: A Cloudy Approach
  • 17. Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Storage Every simulation (36,000 workunits) outputs 3.6 TB of data. There are not enough resources (disk space) on the current systems. Figure : Shared Storage Architecture Diego P´erez Montes climateprediction.net: A Cloudy Approach
  • 18. Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Architecture Figure : Central System Architecture Diego P´erez Montes climateprediction.net: A Cloudy Approach
  • 19. Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Backend Components Simple Scheduler: Runs and configures simulation with given parameters (start/stop instances). Reaper: Releases resources (terminates instances) when they are powered off. RESTful API: Gives access to configure and run simulations. Diego P´erez Montes climateprediction.net: A Cloudy Approach
  • 20. Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions API RESTful API Get simulation status. Get metric/statistic data. Set/modify simulation parameters (number of worker nodes/instances). Stop simulation. Diego P´erez Montes climateprediction.net: A Cloudy Approach
  • 21. Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Dashboard Figure : Dashboard Interface Diego P´erez Montes climateprediction.net: A Cloudy Approach
  • 22. Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Running the Simulation [Overview of a Live System] Diego P´erez Montes climateprediction.net: A Cloudy Approach
  • 23. Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Conclusions Objectives Achieved Computing and Storage successfully migrated to the Cloud (EC2 and S3). Simulations were executed, showing that running the model in the cloud is possible. Development and a Central System (scheduler and dashboard). Got costs and metrics of the project. Diego P´erez Montes climateprediction.net: A Cloudy Approach
  • 24. Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Conclusions What’s Next? Migrate BOINC server. More control/interaction with clients so the scheduler can be improved (and give a full SaaS layer). Costs: “warm up“ stage to dynamically recalculate price. Diego P´erez Montes climateprediction.net: A Cloudy Approach
  • 25. Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Thanks! Diego P´erez Montes climateprediction.net: A Cloudy Approach
  • 26. Outline Problem Background Computing Infrastructure Migration Storage Central Control System Conclusions Used Icons Links Iconset Windows 8 metro style: https://www.iconfinder. com/iconsets/windows-8-metro-style Link: http://sta.sh/0228t4fyjyjb Diego P´erez Montes climateprediction.net: A Cloudy Approach