SlideShare a Scribd company logo

Mapping Life Science Informatics to the Cloud

Chris Dagdigian
Chris Dagdigian
Chris DagdigianCo-founder; Principal Consultant at The BioTeam, Inc.

Infrastructure cloud platforms such as those offered by Amazon Web Services are not designed and built with scientific research as the primary use case. These presentation slides cover the current state of mapping life science research and HPC technique onto “the cloud” and how to work around the common engineering, orchestration and data movement problems. [Note: I've replaced the 2011 version of this talk deck with a slightly updated version as delivered at the AIRI Petabyte Challenge Meeting]

Mapping Life Science Informatics to the Cloud

1 of 82
Download to read offline
Mapping Informatics To the Cloud


                    2012 AIRI Petabyte Challenge
                                  Chris Dagdigian
                               chris@bioteam.net
I‟m Chris.

I‟m an infrastructure
geek.

I work for the
BioTeam.
The “C” Word.
When I say “cloud”
 I‟m talking IaaS.
Amazon AWS
      Is the IaaS cloud.
         Most others are fooling themselves.
(Has-beens, also-rans & delusional marketing zombies)
A message for the
  pretenders…
Ad

Recommended

Multi-Tenant Pharma HPC Clusters
Multi-Tenant Pharma HPC ClustersMulti-Tenant Pharma HPC Clusters
Multi-Tenant Pharma HPC ClustersChris Dagdigian
 
BioIT World 2016 - HPC Trends from the Trenches
BioIT World 2016 - HPC Trends from the TrenchesBioIT World 2016 - HPC Trends from the Trenches
BioIT World 2016 - HPC Trends from the TrenchesChris Dagdigian
 
BioIT Trends - 2014 Internet2 Technology Exchange
BioIT Trends - 2014 Internet2 Technology ExchangeBioIT Trends - 2014 Internet2 Technology Exchange
BioIT Trends - 2014 Internet2 Technology ExchangeChris Dagdigian
 
2015 CDC Workshop on ScienceDMZ
2015 CDC Workshop on ScienceDMZ2015 CDC Workshop on ScienceDMZ
2015 CDC Workshop on ScienceDMZChris Dagdigian
 
Taming Big Science Data Growth with Converged Infrastructure
Taming Big Science Data Growth with Converged InfrastructureTaming Big Science Data Growth with Converged Infrastructure
Taming Big Science Data Growth with Converged InfrastructureThe BioTeam Inc.
 
2015 Bio-IT Trends From the Trenches
2015 Bio-IT Trends From the Trenches2015 Bio-IT Trends From the Trenches
2015 Bio-IT Trends From the TrenchesChris Dagdigian
 
2013: Trends from the Trenches
2013: Trends from the Trenches2013: Trends from the Trenches
2013: Trends from the TrenchesChris Dagdigian
 
2014 BioIT World - Trends from the trenches - Annual presentation
2014 BioIT World - Trends from the trenches - Annual presentation2014 BioIT World - Trends from the trenches - Annual presentation
2014 BioIT World - Trends from the trenches - Annual presentationChris Dagdigian
 

More Related Content

What's hot

Cloud Security for Life Science R&D
Cloud Security for Life Science R&DCloud Security for Life Science R&D
Cloud Security for Life Science R&DChris Dagdigian
 
Facilitating Collaborative Life Science Research in Commercial & Enterprise E...
Facilitating Collaborative Life Science Research in Commercial & Enterprise E...Facilitating Collaborative Life Science Research in Commercial & Enterprise E...
Facilitating Collaborative Life Science Research in Commercial & Enterprise E...Chris Dagdigian
 
Bio-IT & Cloud Sobriety: 2013 Beyond The Genome Meeting
Bio-IT & Cloud Sobriety: 2013 Beyond The Genome MeetingBio-IT & Cloud Sobriety: 2013 Beyond The Genome Meeting
Bio-IT & Cloud Sobriety: 2013 Beyond The Genome MeetingChris Dagdigian
 
Bio-IT for Core Facility Managers
Bio-IT for Core Facility ManagersBio-IT for Core Facility Managers
Bio-IT for Core Facility ManagersChris Dagdigian
 
2015 04 bio it world
2015 04 bio it world2015 04 bio it world
2015 04 bio it worldChris Dwan
 
Cloud Sobriety for Life Science IT Leadership (2018 Edition)
Cloud Sobriety for Life Science IT Leadership (2018 Edition)Cloud Sobriety for Life Science IT Leadership (2018 Edition)
Cloud Sobriety for Life Science IT Leadership (2018 Edition)Chris Dagdigian
 
Trends from the Trenches: 2019
Trends from the Trenches: 2019Trends from the Trenches: 2019
Trends from the Trenches: 2019Chris Dagdigian
 
Bio-IT Trends From The Trenches (digital edition)
Bio-IT Trends From The Trenches (digital edition)Bio-IT Trends From The Trenches (digital edition)
Bio-IT Trends From The Trenches (digital edition)Chris Dagdigian
 
Practical Petabyte Pushing
Practical Petabyte PushingPractical Petabyte Pushing
Practical Petabyte PushingChris Dagdigian
 
Disruptive Innovation: how do you use these theories to manage your IT?
Disruptive Innovation: how do you use these theories to manage your IT?Disruptive Innovation: how do you use these theories to manage your IT?
Disruptive Innovation: how do you use these theories to manage your IT?mark madsen
 
Big Data and Bad Analogies
Big Data and Bad AnalogiesBig Data and Bad Analogies
Big Data and Bad Analogiesmark madsen
 
Lean approach to IT development
Lean approach to IT developmentLean approach to IT development
Lean approach to IT developmentMark Krebs
 
Everything has changed except us
Everything has changed except usEverything has changed except us
Everything has changed except usmark madsen
 
2016 05 sanger
2016 05 sanger2016 05 sanger
2016 05 sangerChris Dwan
 
BioTeam Trends from the Trenches - NIH, April 2014
BioTeam Trends from the Trenches - NIH, April 2014BioTeam Trends from the Trenches - NIH, April 2014
BioTeam Trends from the Trenches - NIH, April 2014Ari Berman
 
frog IoT Big Design IoT World Congress 2015
frog IoT Big Design IoT World Congress 2015frog IoT Big Design IoT World Congress 2015
frog IoT Big Design IoT World Congress 2015Patrick Kalaher
 
Mini-course "Practices of the Web Giants" at Global Code - São Paulo
Mini-course "Practices of the Web Giants" at Global Code - São PauloMini-course "Practices of the Web Giants" at Global Code - São Paulo
Mini-course "Practices of the Web Giants" at Global Code - São PauloOCTO Technology
 
Humans in a loop: Jupyter notebooks as a front-end for AI
Humans in a loop: Jupyter notebooks as a front-end for AIHumans in a loop: Jupyter notebooks as a front-end for AI
Humans in a loop: Jupyter notebooks as a front-end for AIPaco Nathan
 
Trends from the Trenches (Singapore Edition)
Trends from the Trenches (Singapore Edition)Trends from the Trenches (Singapore Edition)
Trends from the Trenches (Singapore Edition)Chris Dagdigian
 
Innovation med big data – chr. hansens erfaringer
Innovation med big data – chr. hansens erfaringerInnovation med big data – chr. hansens erfaringer
Innovation med big data – chr. hansens erfaringerMicrosoft
 

What's hot (20)

Cloud Security for Life Science R&D
Cloud Security for Life Science R&DCloud Security for Life Science R&D
Cloud Security for Life Science R&D
 
Facilitating Collaborative Life Science Research in Commercial & Enterprise E...
Facilitating Collaborative Life Science Research in Commercial & Enterprise E...Facilitating Collaborative Life Science Research in Commercial & Enterprise E...
Facilitating Collaborative Life Science Research in Commercial & Enterprise E...
 
Bio-IT & Cloud Sobriety: 2013 Beyond The Genome Meeting
Bio-IT & Cloud Sobriety: 2013 Beyond The Genome MeetingBio-IT & Cloud Sobriety: 2013 Beyond The Genome Meeting
Bio-IT & Cloud Sobriety: 2013 Beyond The Genome Meeting
 
Bio-IT for Core Facility Managers
Bio-IT for Core Facility ManagersBio-IT for Core Facility Managers
Bio-IT for Core Facility Managers
 
2015 04 bio it world
2015 04 bio it world2015 04 bio it world
2015 04 bio it world
 
Cloud Sobriety for Life Science IT Leadership (2018 Edition)
Cloud Sobriety for Life Science IT Leadership (2018 Edition)Cloud Sobriety for Life Science IT Leadership (2018 Edition)
Cloud Sobriety for Life Science IT Leadership (2018 Edition)
 
Trends from the Trenches: 2019
Trends from the Trenches: 2019Trends from the Trenches: 2019
Trends from the Trenches: 2019
 
Bio-IT Trends From The Trenches (digital edition)
Bio-IT Trends From The Trenches (digital edition)Bio-IT Trends From The Trenches (digital edition)
Bio-IT Trends From The Trenches (digital edition)
 
Practical Petabyte Pushing
Practical Petabyte PushingPractical Petabyte Pushing
Practical Petabyte Pushing
 
Disruptive Innovation: how do you use these theories to manage your IT?
Disruptive Innovation: how do you use these theories to manage your IT?Disruptive Innovation: how do you use these theories to manage your IT?
Disruptive Innovation: how do you use these theories to manage your IT?
 
Big Data and Bad Analogies
Big Data and Bad AnalogiesBig Data and Bad Analogies
Big Data and Bad Analogies
 
Lean approach to IT development
Lean approach to IT developmentLean approach to IT development
Lean approach to IT development
 
Everything has changed except us
Everything has changed except usEverything has changed except us
Everything has changed except us
 
2016 05 sanger
2016 05 sanger2016 05 sanger
2016 05 sanger
 
BioTeam Trends from the Trenches - NIH, April 2014
BioTeam Trends from the Trenches - NIH, April 2014BioTeam Trends from the Trenches - NIH, April 2014
BioTeam Trends from the Trenches - NIH, April 2014
 
frog IoT Big Design IoT World Congress 2015
frog IoT Big Design IoT World Congress 2015frog IoT Big Design IoT World Congress 2015
frog IoT Big Design IoT World Congress 2015
 
Mini-course "Practices of the Web Giants" at Global Code - São Paulo
Mini-course "Practices of the Web Giants" at Global Code - São PauloMini-course "Practices of the Web Giants" at Global Code - São Paulo
Mini-course "Practices of the Web Giants" at Global Code - São Paulo
 
Humans in a loop: Jupyter notebooks as a front-end for AI
Humans in a loop: Jupyter notebooks as a front-end for AIHumans in a loop: Jupyter notebooks as a front-end for AI
Humans in a loop: Jupyter notebooks as a front-end for AI
 
Trends from the Trenches (Singapore Edition)
Trends from the Trenches (Singapore Edition)Trends from the Trenches (Singapore Edition)
Trends from the Trenches (Singapore Edition)
 
Innovation med big data – chr. hansens erfaringer
Innovation med big data – chr. hansens erfaringerInnovation med big data – chr. hansens erfaringer
Innovation med big data – chr. hansens erfaringer
 

Similar to Mapping Life Science Informatics to the Cloud

Cloud Computing: The Hard Problems Never Go Away
Cloud Computing: The Hard Problems Never Go AwayCloud Computing: The Hard Problems Never Go Away
Cloud Computing: The Hard Problems Never Go AwayZendCon
 
Dev/Test in the Cloud - F
Dev/Test in the Cloud - FDev/Test in the Cloud - F
Dev/Test in the Cloud - FChris Riley ☁
 
Big Data Strategy for the Relational World
Big Data Strategy for the Relational World Big Data Strategy for the Relational World
Big Data Strategy for the Relational World Andrew Brust
 
SQL 2014 hybrid platform - Azure and on premise
SQL 2014 hybrid platform - Azure and on premise SQL 2014 hybrid platform - Azure and on premise
SQL 2014 hybrid platform - Azure and on premise Shy Engelberg
 
Serverless Toronto helps Startups
Serverless Toronto helps StartupsServerless Toronto helps Startups
Serverless Toronto helps StartupsDaniel Zivkovic
 
When small problems become big problems
When small problems become big problemsWhen small problems become big problems
When small problems become big problemsAdrian Cole
 
Cloud Big Data Architectures
Cloud Big Data ArchitecturesCloud Big Data Architectures
Cloud Big Data ArchitecturesLynn Langit
 
Building a Service Provider Cloud Offering - MVMUG Sept2013
Building a Service Provider Cloud Offering - MVMUG Sept2013Building a Service Provider Cloud Offering - MVMUG Sept2013
Building a Service Provider Cloud Offering - MVMUG Sept2013Arron Stebbing
 
Scaling a High Traffic Web Application: Our Journey from Java to PHP
Scaling a High Traffic Web Application: Our Journey from Java to PHPScaling a High Traffic Web Application: Our Journey from Java to PHP
Scaling a High Traffic Web Application: Our Journey from Java to PHP120bi
 
Scaling High Traffic Web Applications
Scaling High Traffic Web ApplicationsScaling High Traffic Web Applications
Scaling High Traffic Web ApplicationsAchievers Tech
 
Greenfields tech decisions
Greenfields tech decisionsGreenfields tech decisions
Greenfields tech decisionsTrent Hornibrook
 
Performance Optimization of Cloud Based Applications by Peter Smith, ACL
Performance Optimization of Cloud Based Applications by Peter Smith, ACLPerformance Optimization of Cloud Based Applications by Peter Smith, ACL
Performance Optimization of Cloud Based Applications by Peter Smith, ACLTriNimbus
 
SpringPeople - Introduction to Cloud Computing
SpringPeople - Introduction to Cloud ComputingSpringPeople - Introduction to Cloud Computing
SpringPeople - Introduction to Cloud ComputingSpringPeople
 
"Portrait of the developer as The Artist" Lockheed Architect Workshop
"Portrait of the developer as The Artist" Lockheed Architect Workshop"Portrait of the developer as The Artist" Lockheed Architect Workshop
"Portrait of the developer as The Artist" Lockheed Architect WorkshopPatrick Chanezon
 
Sage Summit 2012: Cloud Computing for Accountants
Sage Summit 2012: Cloud Computing for AccountantsSage Summit 2012: Cloud Computing for Accountants
Sage Summit 2012: Cloud Computing for AccountantsGrant M Howe
 
First Hive Meetup London 2012-07-10 - Tomas Cervenka - VisualDNA
First Hive Meetup London 2012-07-10 - Tomas Cervenka - VisualDNAFirst Hive Meetup London 2012-07-10 - Tomas Cervenka - VisualDNA
First Hive Meetup London 2012-07-10 - Tomas Cervenka - VisualDNATomas Cervenka
 
Why choose cloud based servers over physical hardware?
Why choose cloud based servers over physical hardware?Why choose cloud based servers over physical hardware?
Why choose cloud based servers over physical hardware?Michael Lobb
 
Itlc hanoi lesson learned-open-stack - pham tung duong
Itlc hanoi lesson learned-open-stack - pham tung duongItlc hanoi lesson learned-open-stack - pham tung duong
Itlc hanoi lesson learned-open-stack - pham tung duongLe Cuong
 

Similar to Mapping Life Science Informatics to the Cloud (20)

Cloud Computing: The Hard Problems Never Go Away
Cloud Computing: The Hard Problems Never Go AwayCloud Computing: The Hard Problems Never Go Away
Cloud Computing: The Hard Problems Never Go Away
 
Dev/Test in the Cloud - F
Dev/Test in the Cloud - FDev/Test in the Cloud - F
Dev/Test in the Cloud - F
 
Big Data Strategy for the Relational World
Big Data Strategy for the Relational World Big Data Strategy for the Relational World
Big Data Strategy for the Relational World
 
SQL 2014 hybrid platform - Azure and on premise
SQL 2014 hybrid platform - Azure and on premise SQL 2014 hybrid platform - Azure and on premise
SQL 2014 hybrid platform - Azure and on premise
 
Serverless Toronto helps Startups
Serverless Toronto helps StartupsServerless Toronto helps Startups
Serverless Toronto helps Startups
 
When small problems become big problems
When small problems become big problemsWhen small problems become big problems
When small problems become big problems
 
Cloud Big Data Architectures
Cloud Big Data ArchitecturesCloud Big Data Architectures
Cloud Big Data Architectures
 
Building a Service Provider Cloud Offering - MVMUG Sept2013
Building a Service Provider Cloud Offering - MVMUG Sept2013Building a Service Provider Cloud Offering - MVMUG Sept2013
Building a Service Provider Cloud Offering - MVMUG Sept2013
 
Scaling a High Traffic Web Application: Our Journey from Java to PHP
Scaling a High Traffic Web Application: Our Journey from Java to PHPScaling a High Traffic Web Application: Our Journey from Java to PHP
Scaling a High Traffic Web Application: Our Journey from Java to PHP
 
Scaling High Traffic Web Applications
Scaling High Traffic Web ApplicationsScaling High Traffic Web Applications
Scaling High Traffic Web Applications
 
Greenfields tech decisions
Greenfields tech decisionsGreenfields tech decisions
Greenfields tech decisions
 
Performance Optimization of Cloud Based Applications by Peter Smith, ACL
Performance Optimization of Cloud Based Applications by Peter Smith, ACLPerformance Optimization of Cloud Based Applications by Peter Smith, ACL
Performance Optimization of Cloud Based Applications by Peter Smith, ACL
 
SpringPeople - Introduction to Cloud Computing
SpringPeople - Introduction to Cloud ComputingSpringPeople - Introduction to Cloud Computing
SpringPeople - Introduction to Cloud Computing
 
Dev Ops without the Ops
Dev Ops without the OpsDev Ops without the Ops
Dev Ops without the Ops
 
"Portrait of the developer as The Artist" Lockheed Architect Workshop
"Portrait of the developer as The Artist" Lockheed Architect Workshop"Portrait of the developer as The Artist" Lockheed Architect Workshop
"Portrait of the developer as The Artist" Lockheed Architect Workshop
 
Sage Summit 2012: Cloud Computing for Accountants
Sage Summit 2012: Cloud Computing for AccountantsSage Summit 2012: Cloud Computing for Accountants
Sage Summit 2012: Cloud Computing for Accountants
 
First Hive Meetup London 2012-07-10 - Tomas Cervenka - VisualDNA
First Hive Meetup London 2012-07-10 - Tomas Cervenka - VisualDNAFirst Hive Meetup London 2012-07-10 - Tomas Cervenka - VisualDNA
First Hive Meetup London 2012-07-10 - Tomas Cervenka - VisualDNA
 
Why choose cloud based servers over physical hardware?
Why choose cloud based servers over physical hardware?Why choose cloud based servers over physical hardware?
Why choose cloud based servers over physical hardware?
 
Architecting Your First Big Data Implementation
Architecting Your First Big Data ImplementationArchitecting Your First Big Data Implementation
Architecting Your First Big Data Implementation
 
Itlc hanoi lesson learned-open-stack - pham tung duong
Itlc hanoi lesson learned-open-stack - pham tung duongItlc hanoi lesson learned-open-stack - pham tung duong
Itlc hanoi lesson learned-open-stack - pham tung duong
 

Recently uploaded

Cfgmgmtcamp 2024 — eBPF-based Security Observability & Runtime Enforcement wi...
Cfgmgmtcamp 2024 — eBPF-based Security Observability & Runtime Enforcement wi...Cfgmgmtcamp 2024 — eBPF-based Security Observability & Runtime Enforcement wi...
Cfgmgmtcamp 2024 — eBPF-based Security Observability & Runtime Enforcement wi...Raphaël PINSON
 
Manual Eurotronic Thermostatic Valve Comry Z-Wave
Manual Eurotronic Thermostatic Valve Comry Z-WaveManual Eurotronic Thermostatic Valve Comry Z-Wave
Manual Eurotronic Thermostatic Valve Comry Z-WaveDomotica daVinci
 
Z-Wave Fan coil Thermostat Heltun_HE-HT01_User_Manual.pdf
Z-Wave Fan coil Thermostat Heltun_HE-HT01_User_Manual.pdfZ-Wave Fan coil Thermostat Heltun_HE-HT01_User_Manual.pdf
Z-Wave Fan coil Thermostat Heltun_HE-HT01_User_Manual.pdfDomotica daVinci
 
Semiconductor Review Magazine Feature.pdf
Semiconductor Review Magazine Feature.pdfSemiconductor Review Magazine Feature.pdf
Semiconductor Review Magazine Feature.pdfkeyaramicrochipusa
 
Evolution of Chatbots: From Custom AI Chatbots and AI Chatbots for Websites.pptx
Evolution of Chatbots: From Custom AI Chatbots and AI Chatbots for Websites.pptxEvolution of Chatbots: From Custom AI Chatbots and AI Chatbots for Websites.pptx
Evolution of Chatbots: From Custom AI Chatbots and AI Chatbots for Websites.pptxKyle Willson
 
"Journey of Aspiration: Unveiling the Path to Becoming a Technocrat and Entre...
"Journey of Aspiration: Unveiling the Path to Becoming a Technocrat and Entre..."Journey of Aspiration: Unveiling the Path to Becoming a Technocrat and Entre...
"Journey of Aspiration: Unveiling the Path to Becoming a Technocrat and Entre...shaiyuvasv
 
Zi-Stick UBS Dongle ZIgbee from Aeotec manual
Zi-Stick UBS Dongle ZIgbee from  Aeotec manualZi-Stick UBS Dongle ZIgbee from  Aeotec manual
Zi-Stick UBS Dongle ZIgbee from Aeotec manualDomotica daVinci
 
zigbee motion sensor user manual NAS-PD07B2.pdf
zigbee motion sensor user manual NAS-PD07B2.pdfzigbee motion sensor user manual NAS-PD07B2.pdf
zigbee motion sensor user manual NAS-PD07B2.pdfDomotica daVinci
 
2024 February Patch Tuesday
2024 February Patch Tuesday2024 February Patch Tuesday
2024 February Patch TuesdayIvanti
 
My self introduction to know others abut me
My self  introduction to know others abut meMy self  introduction to know others abut me
My self introduction to know others abut meManoj Prabakar B
 
Importance of magazines in education ppt
Importance of magazines in education pptImportance of magazines in education ppt
Importance of magazines in education pptsafnarafeek2002
 
Navigating the Never Normal Strategies for Portfolio Leaders
Navigating the Never Normal Strategies for Portfolio LeadersNavigating the Never Normal Strategies for Portfolio Leaders
Navigating the Never Normal Strategies for Portfolio LeadersOnePlan Solutions
 
Traffic Signboard Classification with Voice alert to the driver.pptx
Traffic Signboard Classification with Voice alert to the driver.pptxTraffic Signboard Classification with Voice alert to the driver.pptx
Traffic Signboard Classification with Voice alert to the driver.pptxharimaxwell0712
 
Introduction to Serverless with AWS Lambda in C#.pptx
Introduction to Serverless with AWS Lambda in C#.pptxIntroduction to Serverless with AWS Lambda in C#.pptx
Introduction to Serverless with AWS Lambda in C#.pptxBrandon Minnick, MBA
 
Put a flag on it. A busy developer's guide to feature toggles.
Put a flag on it. A busy developer's guide to feature toggles.Put a flag on it. A busy developer's guide to feature toggles.
Put a flag on it. A busy developer's guide to feature toggles.Mateusz Kwasniewski
 
AWS for the beginning is cloud computing
AWS for the beginning  is  cloud computingAWS for the beginning  is  cloud computing
AWS for the beginning is cloud computingkajalghule1
 

Recently uploaded (20)

Cfgmgmtcamp 2024 — eBPF-based Security Observability & Runtime Enforcement wi...
Cfgmgmtcamp 2024 — eBPF-based Security Observability & Runtime Enforcement wi...Cfgmgmtcamp 2024 — eBPF-based Security Observability & Runtime Enforcement wi...
Cfgmgmtcamp 2024 — eBPF-based Security Observability & Runtime Enforcement wi...
 
Manual Eurotronic Thermostatic Valve Comry Z-Wave
Manual Eurotronic Thermostatic Valve Comry Z-WaveManual Eurotronic Thermostatic Valve Comry Z-Wave
Manual Eurotronic Thermostatic Valve Comry Z-Wave
 
Z-Wave Fan coil Thermostat Heltun_HE-HT01_User_Manual.pdf
Z-Wave Fan coil Thermostat Heltun_HE-HT01_User_Manual.pdfZ-Wave Fan coil Thermostat Heltun_HE-HT01_User_Manual.pdf
Z-Wave Fan coil Thermostat Heltun_HE-HT01_User_Manual.pdf
 
Semiconductor Review Magazine Feature.pdf
Semiconductor Review Magazine Feature.pdfSemiconductor Review Magazine Feature.pdf
Semiconductor Review Magazine Feature.pdf
 
Evolution of Chatbots: From Custom AI Chatbots and AI Chatbots for Websites.pptx
Evolution of Chatbots: From Custom AI Chatbots and AI Chatbots for Websites.pptxEvolution of Chatbots: From Custom AI Chatbots and AI Chatbots for Websites.pptx
Evolution of Chatbots: From Custom AI Chatbots and AI Chatbots for Websites.pptx
 
"Journey of Aspiration: Unveiling the Path to Becoming a Technocrat and Entre...
"Journey of Aspiration: Unveiling the Path to Becoming a Technocrat and Entre..."Journey of Aspiration: Unveiling the Path to Becoming a Technocrat and Entre...
"Journey of Aspiration: Unveiling the Path to Becoming a Technocrat and Entre...
 
Zi-Stick UBS Dongle ZIgbee from Aeotec manual
Zi-Stick UBS Dongle ZIgbee from  Aeotec manualZi-Stick UBS Dongle ZIgbee from  Aeotec manual
Zi-Stick UBS Dongle ZIgbee from Aeotec manual
 
zigbee motion sensor user manual NAS-PD07B2.pdf
zigbee motion sensor user manual NAS-PD07B2.pdfzigbee motion sensor user manual NAS-PD07B2.pdf
zigbee motion sensor user manual NAS-PD07B2.pdf
 
2024 February Patch Tuesday
2024 February Patch Tuesday2024 February Patch Tuesday
2024 February Patch Tuesday
 
5 Tech Trend to Notice in ESG Landscape- 47Billion
5 Tech Trend to Notice in ESG Landscape- 47Billion5 Tech Trend to Notice in ESG Landscape- 47Billion
5 Tech Trend to Notice in ESG Landscape- 47Billion
 
COE AI Lab Universities
COE AI Lab UniversitiesCOE AI Lab Universities
COE AI Lab Universities
 
My self introduction to know others abut me
My self  introduction to know others abut meMy self  introduction to know others abut me
My self introduction to know others abut me
 
GTA 6.pdf
GTA 6.pdfGTA 6.pdf
GTA 6.pdf
 
My Country Mobile
My Country MobileMy Country Mobile
My Country Mobile
 
Importance of magazines in education ppt
Importance of magazines in education pptImportance of magazines in education ppt
Importance of magazines in education ppt
 
Navigating the Never Normal Strategies for Portfolio Leaders
Navigating the Never Normal Strategies for Portfolio LeadersNavigating the Never Normal Strategies for Portfolio Leaders
Navigating the Never Normal Strategies for Portfolio Leaders
 
Traffic Signboard Classification with Voice alert to the driver.pptx
Traffic Signboard Classification with Voice alert to the driver.pptxTraffic Signboard Classification with Voice alert to the driver.pptx
Traffic Signboard Classification with Voice alert to the driver.pptx
 
Introduction to Serverless with AWS Lambda in C#.pptx
Introduction to Serverless with AWS Lambda in C#.pptxIntroduction to Serverless with AWS Lambda in C#.pptx
Introduction to Serverless with AWS Lambda in C#.pptx
 
Put a flag on it. A busy developer's guide to feature toggles.
Put a flag on it. A busy developer's guide to feature toggles.Put a flag on it. A busy developer's guide to feature toggles.
Put a flag on it. A busy developer's guide to feature toggles.
 
AWS for the beginning is cloud computing
AWS for the beginning  is  cloud computingAWS for the beginning  is  cloud computing
AWS for the beginning is cloud computing
 

Mapping Life Science Informatics to the Cloud

  • 1. Mapping Informatics To the Cloud 2012 AIRI Petabyte Challenge Chris Dagdigian chris@bioteam.net
  • 2. I‟m Chris. I‟m an infrastructure geek. I work for the BioTeam.
  • 4. When I say “cloud” I‟m talking IaaS.
  • 5. Amazon AWS Is the IaaS cloud. Most others are fooling themselves. (Has-beens, also-rans & delusional marketing zombies)
  • 6. A message for the pretenders…
  • 7. No APIs? Not a cloud.
  • 9. I have to email a human? Not a cloud.
  • 10. ~50% failure rate when provisioning new servers? Stupid cloud.
  • 11. Block storage and virtual servers only? (barely) a cloud;
  • 13. Private Clouds in 2012: • Hype vs. Reality ratio still wacky • Sensible only for certain shops • Have you seen what you have to do to your networks & gear? • There are easier ways
  • 14. Private Clouds: My Advice for „12 • Remain cynical (test vendor claims) • Due Diligence still essential • I personally would not deploy/buy anything that does not explicitly provide Amazon API compatibility
  • 15. Private Clouds: My Advice for „12 • Most people are better off: • Adding VM platforms to existing HPC clusters & environments • Extending enterprise VM platforms to allow user self-service & server catalogs
  • 18. HPC & Clouds: Whole New World
  • 19. • We have spent decades learning to tune research HPC systems for shared access & many users. • The cloud upends this model
  • 20. • Far more common to see … • Dedicated cloud resources spun up for each app or use case • Each system gets individually tuned & optimized
  • 22. Hybrid Clouds & Cloud Bursting
  • 23. • Lots of aggressive marketing • Lots of carefully constructed “case studies” and prototypes • The truth? • Less usable than you‟ve been told • Possible? Heck yeah. • Practical? Only sometimes.
  • 24. • Advice • Be cynical • Demand proof • Test carefully
  • 25. • Still want to do it? • Buy it, don‟t build it • Cycle Computing • Univa • BrightComputing • …
  • 26. • Follow the crowd • In the real world we see: • Separation between local and cloud HPC resources • Send your work to the system most suitable
  • 28. You can‟t rewrite EVERYTHING.
  • 29. • Salesfolk will just glibly tell you to rewrite your apps so you can use whatever big data analysis framework they happen to be selling today
  • 30. • They have no clue.
  • 31. • In life science informatics we have hundreds of codes that will never be rewritten. • We‟ll be needing them for years to come.
  • 32. • Advice: • MapReduceish methods are the future for big-data informatics • It will take years to get there • We still have to deal with legacy algorithms and codes
  • 33. • You will need: • A process for figuring out when it‟s worthwhile to rewrite/re-architect • Tested cloud strategies for handling three use cases
  • 34. You need 3 cloud architectures: 1. Legacy HPC 2. “Cloudy” HPC 3. Big Data HPC (Hadoop)
  • 35. Legacy HPC on the cloud • MIT StarCluster • http://web.mit.edu/star/cluster/ • This is your baseline • Extend as needed
  • 36. “Cloudy” HPC • Use this method when … • It makes sense to rewrite or rearchitect an HPC workflow to better leverage modern cloud capabilities
  • 37. “Cloudy” HPC, continued • Ditch the legacy compute farm model • Leverage elastic scale-out tools (***) • Spot Instances for elastic & cheap compute • SimpleDB for job statekeeping • SQS for job queues & workrflow “glue” • SNS for message passing & monitoring • S3 for input & output data • Etc.
  • 38. Big Data HPC • It‟s gonna be a MapReduce world • Little need to roll your own • Ecosystem already healthy • Multiple providers today • Often a slam-dunk cloud use case
  • 40. The Cloud was not designed for “us”
  • 41. • HPC is an edge case for the hyperscale IaaS clouds • We need to deal with this and engineer around it.
  • 42. • Many examples • Eventual consistency • Networking & subnets • Latency • Node placement
  • 43. • Advice • Manage expectations • Benchmark & test • Evangelize • (pester the cloud sales reps …)
  • 45. Data Movement Is Still Hard
  • 46. • Consistently getting easier • Amazon is not a bottleneck • AWS Import/Export • AWS Direct Connect • Aspera has some amazing stuff out right now
  • 47. • Advice • AWS Import/Export works well • Size of pipe is not everything • Sweat the small stuff • Tracking, checksums, disk speed • Dedicated workstations • Secure media storage
  • 50. Don‟t overlook media storage …
  • 51. • Advice for 2012 • BioTeam is dialing down our advocacy of physical data ingestion into the cloud • Why? • Operationally hard, expensive and no longer strictly needed
  • 52. Real world cross-country internet-based data movement March 2012
  • 53. 700Mb/sec into Amazon, stress-free & zero tuning March 2012
  • 54. • People trying to move data via physical media quickly realize the operational difficulties • Bandwidth is cheaper than hiring another body to manage physical data ingestion & movement • In 2012 we strongly recommend network-based data movement when at all possible
  • 55. u r doing it wrong
  • 59. Big shared storage. Still hard.
  • 60. • Not much we can do except engineer around it • AWS compute cluster instances are a huge step forward • AWS competitors take note
  • 61. • We are not database nerds • We care about more than just random IO performance • We need it all • Random I/O • Long sequential read/write
  • 62. • Faster Storage Options • Software RAID on EBS • Various GlusterFS options • Even if you optimize everything, the virtual NICs are still a bottleneck
  • 63. • Big Shared Storage • 10GbE nodes and NFS • Software RAID sets • GlusterFS or similar • 2012: pNFS finally?
  • 65. Things fail differently in the cloud.
  • 66. • Stuff breaks • It breaks in weird ways • Transient/temporary issues more common than what we see “at home”
  • 67. • Advice • Pessimism is good • Design for failure • Think hard about • How will you detect? • How will you respond?
  • 68. • Advice • Remove humans from loop • Automate recovery • Automate your backups
  • 71. • Loosely coupled workflows are ideal • Break the pipeline into discrete components • Components should be able to scale up|down independently
  • 72. • Component = Opportunity to: • … Make a scaling decision • (# nodes in use) • … Make sizing decision • (instance type in use)
  • 74. … independent loosely connected components that can self-scale and communicate asynchronously
  • 75. Advice: • Many people already doing this • Best practices are well known • Steal from the best: • RightScale, Opscode & Cycle Computing
  • 76. Phew. Think I‟m done now.
  • 77. Questions? Slides available at http://slideshare.net/chrisdag/
  • 78. End;
  • 80. Private Clouds: Pick Your Poison • OpenStack - http://openstack.org • Pro: Super smart developers; significant mindshare; True Open Source • Con: Commitment to AWS API compatibility (?) & stability
  • 81. Private Clouds: Pick Your Poison • CloudStack- http://cloudstack.org • Pro: Explicit AWS API support; very recent move away from “open-core” model; usability • Con: Developer mindshare? Sudden switch to Apache
  • 82. Private Clouds: Pick Your Poison • Eucalyptus- http://eucalyptus.com • Pro: Direct AWS API compatibility; lots of hypervisor support • Con: Open-core model; mindshare; Recent ressurection