SlideShare a Scribd company logo
1 of 29
Deliver Best-in-Class HPC Cloud Solutions
Without Losing Your Mind
WEBINAR
April 13, 2016, 11:00 AM ET
Housekeeping
• Audio help
• Attachments
• Questions
• Rating
Today’s Speakers
Rick Friedman
Vice President, Solution
Development
Cycle Computing
Scott Jeschonek
Director of Product
Management, Cloud
Avere Systems
Agenda
• Discuss the current state of HPC
• Clouds and their impact on your HPC world
• Reasons why you aren’t 100% cloud-based already
• The Hybrid Cloud and HPC
• Possible implementations
• Delivering File Systems using Avere Systems
• Orchestration using Cycle Computing
HPC Today (and Yesterday, and
Tomorrow)
What Drives Today’s Needs
• Data
– Who, what, when, how much, where?
• Datacenter limitations
– Can I defy physics?
• User expectations
– Can we even do that?
• Technology shifts
– What is the “best practice”?
Big Compute Workloads: How are they handled?
Compute Demand vs. Cluster Size
Cluster Size
Compute
Demand
Missed
Opportunity
Wasted
Resources
• Internal infrastructure has huge value and
some limitations
• Access, not capacity, is the barrier to
continued growth
• Perception limits scale of problem solving
• Public cloud = cost-effective, readily
available resources to users with problems
& deadlines.
• Financial services, manufacturing and life
sciences are leading the way.
Basic HPC Environment Requirements
Resource
Manager
Jobs Manager /
Scheduler
Workload
NAS Storage
Lots of compute resources (“Grid”)
Advantages of Clouds
Significantly reduce
infrastructure
management costs both in
money and time
Maintain operational
flexibility during scale-out
jobs…let the provider deal
with scale challenges
Why the Cloud for Big Compute?
• Scientist / Engineer User perspective
– Zero queue times, capacity in minutes
– Scale compute to problems size, not vice versa
– Try / support new computational approaches and software quickly
• SysArchitect perspective
– Dynamically adjust workloads to “lowest cost/impact” provider
– Focus on computational excellence, not hardware management
– Support a wide range of user types efficiently
• Organizational perspective
– Match spending to actual consumption
– Increase responsiveness to business dynamics
– Grow user base without hardware limitations
Clouds Have Awesome New Capabilities
• Big Data
– Analytics Tools
– Massively scalable NoSQL
– Data warehousing
• Machine Learning
– Voice/Vision/Speech
– Early days
So…why isn’t everything in the cloud?
• Current infrastructure investment (capex)
• Cloud costs not yet completely in line
• Software infrastructure in place
– Costs to refactor, dependencies to consider
• Data environment in one or more data centers
• Orchestration and management of cloud clusters is hard
• Network bandwidth / latency concerns
• Business Continuity
Other Reasons You’re Not 100% in the Cloud
• Corporate budgets
• Corporate policies
• Corporate politics
• Education / awareness
• Government regulations
• Interest groups
• Vendor relationships
Near Future, Hybrid Cloud
Tokyo office London office
Analysts
Analysts
NYC office
Analysts AnalystsAnalysts
Analysts AnalystsAnalysts
AnalystsAnalysts
Hong Kong office
• Adoption of one or more cloud providers
• > 1 hedge on price and SLA
• Mix of on-prem and cloud resources
• Regulatory, proprietary and/or security
characteristics will likely keep data in the DC
NAS
Primary
DC
Cloud
Provider
1
Cloud
Provider
2
NAS
Secondary
DC
Cloud Compute
Environment
Data
HPC in the Cloud
Cloud Compute
API
Scheduler
NAS Storage
Analysts
Scheduler
AnalystsAnalysts Analysts Analysts Analysts
Jobs
On-Premises
Data Center
Cloud Compute
Environment
HPC in the Cloud, “Grids on Demand”
Cloud Compute
API
Data
NAS Storage
Analysts
Scheduler
AnalystsAnalysts Analysts Analysts Analysts
Jobs
On-Premises
Data Center
Scheduler1 Scheduler2
Scheduler3 Scheduler4
Challenges with HPC in the Cloud
• How do you get the data close to your compute nodes?
• How do you orchestrate on-demand clusters/grids of compute
nodes?
• How does this all come together??
Cloud Compute
Environment
Data Access Layer
Cloud Compute
API
Scheduler1
Data
NAS Storage
Analysts
Scheduler
AnalystsAnalysts Analysts Analysts Analysts
Jobs
On-Premises
Data Center
Data Access Layer
Scheduler2 Scheduler3 Scheduler4
• File System
• Caching Layer
• Only load necessary
blocks of files
• Opaque to compute
nodes
Advantages of Data Access / Cache Layer
• Keep your data on prem! – Data in cloud is only there while the
compute nodes work the jobs.
– Reduce the security objections, simplify the move to cloud
• Increase cloud compute performance – using file system caching,
most of the data will be in RAM, close to the nodes
– Avoids ingest latencies and slashes transit latency after first read
• Scale out – Using solution that facilitates 10s of 1000s of core file
system connections
Typical File Access in Hadoop Cluster
Caching files will work for
certain types of jobs
Where typical file is accessed
By multiple clients
source: http://blog.cloudera.com/blog/2012/09/what-do-real-life-hadoop-workloads-look-like/
Hybrid Cloud using Avere FXT and vFXT Edge Filers
Cloud
Compute
On-Prem
Compute
Cloud
Storage
On-Prem
Storage
NAS
Object
Bucket 1 Bucket 2
Bucket n
Virtual Compute
Farm
Virtual
FXT
File Storage for
Private Object
NAS
Optimization
Cloud NAS
Physical
FXT
The “Edge” = locating your data
Close to your compute
Without truly moving it from your
NAS environment
Avere Building Blocks
“Avere is uniquely positioned to offer scale
across tens of thousands of cloud compute
cores while leaving the data where it
originates, on premises, with it’s global file
system and caching capabilities.”
- Unnamed CTO
Cloud Compute
Virtual FXT
NAS
Object
Physical
FXT
Cloud
On-Premises
File Acceleration
Cloud Compute
Environment
Orchestration and Management Layer
Cloud Compute
API
Data
On-Premises
Data Center
Scheduler1 Scheduler2
Scheduler3 Scheduler4
NAS Storage
Analysts
Scheduler
AnalystsAnalysts Analysts Analysts Analysts
Jobs
Optimization
• Benchmark instances
• Make Workflow UI
• Human workflow
Provisioning
• Workload placement
Optimal scale
• Cost optimization
• Data scheduling
Cluster Configuration
• Multi-cloud, without changes
• Pre-set or User-defined “types”
• Abstraction for all cluster data,
attributes (roles, OS, etc)
Monitoring
• Auto-scaling
• Usage tracking
• Error Handling
• Reporting
Internal
File: Declarative
Cluster Definition
Packages, Installers
Containers, Data
Admin
Scope Configure
Run on
Cloud
Optimize
User
Complete Multi-Cloud Workflow Control
User
Web
UI API
CMD
Line
Job & Data
Workflow
Automated
Job Placement,
Cost optimization
Auto-scaling,
Benchmarking,
Compliance,
Reporting tools
Multi-cloud
Without Changes
Internal
Cluster
How Cycle Makes Cloud Productive
• Scientist / Engineer productivity:
– Simple workflows
– Zero queue time
– Auto-scaling
• SysAdmin productivity:
– Instant access to additional resources
– Workflows linking internal and multiple clouds
– Simple reliable tools to enable apps with
special requirements
• Organizational productivity:
– Secure, consistent cloud access
– Usage tracking
– Ability to leverage multiple providers
Big Data w/o Disrupting Production
• Challenge
– Estimate the carbon stored in Saharan biomass
– Rapidly establish a baseline for later research using large
amounts of high-resolution remote sensing data
– Existing internal compute resources fully committed
– Limited window to complete processing
• Cycle solution
– Full workflow including data management between internal
data capture and cloud processing
– Leverage spot pricing to minimize cost while maximizing
computation
• Results
– Linearly scalable, predictable enabling plan for next steps
– Science being done that could not be done otherwise
– 1 month start to initial runs
26
Overall Architecture – Data In-House
Cloud Compute
Scheduler
Avere FXT Edge Filer
Avere FXT
Workload
Cloud API
NAS Storage
Scheduler
Cloud Storage
What We Covered…
• The Current State of HPC
• Clouds and Their Impact on Your HPC World
• Reasons Why You aren’t 100% Cloud-based Already
• The Hybrid Cloud and HPC
• Possible Implementations
• Delivering File Systems Using Avere Systems
• Orchestration Using Cycle Computing
Thank you!
Cycle Computing Contact Info: More about Avere Systems:
askavere@averesystems.com
www.averesystems.com
1-888.88.AVERE
https://twitter.com/averesystems
https://www.youtube.com/user/AvereSystems
https://www.linkedin.com/company/589037
info@cyclecomputing.com
www.cyclecomputing.com
888.292.5320
https://twitter.com/cyclecomputing
https://www.youtube.com/user/CycleComputing
https://www.linkedin.com/company/692068

More Related Content

What's hot

Suitability of Commercial Clouds for NASA's HPC Applications
Suitability of Commercial Clouds for NASA's HPC ApplicationsSuitability of Commercial Clouds for NASA's HPC Applications
Suitability of Commercial Clouds for NASA's HPC Applications
inside-BigData.com
 

What's hot (20)

Cloud
CloudCloud
Cloud
 
Optimizing Your Cloud Applications in RightScale
Optimizing Your Cloud Applications in RightScaleOptimizing Your Cloud Applications in RightScale
Optimizing Your Cloud Applications in RightScale
 
AWS re:Invent 2013 Recap
AWS re:Invent 2013 RecapAWS re:Invent 2013 Recap
AWS re:Invent 2013 Recap
 
Suitability of Commercial Clouds for NASA's HPC Applications
Suitability of Commercial Clouds for NASA's HPC ApplicationsSuitability of Commercial Clouds for NASA's HPC Applications
Suitability of Commercial Clouds for NASA's HPC Applications
 
Amazon EMR
Amazon EMRAmazon EMR
Amazon EMR
 
AWS Webcast - How to Migrate On-premise NAS Storage to Cloud NAS Storage
AWS Webcast - How to Migrate On-premise NAS Storage to Cloud NAS StorageAWS Webcast - How to Migrate On-premise NAS Storage to Cloud NAS Storage
AWS Webcast - How to Migrate On-premise NAS Storage to Cloud NAS Storage
 
Optimizing Storage for Big Data Workloads
Optimizing Storage for Big Data WorkloadsOptimizing Storage for Big Data Workloads
Optimizing Storage for Big Data Workloads
 
Backup and Archiving in the AWS Cloud
Backup and Archiving in the AWS CloudBackup and Archiving in the AWS Cloud
Backup and Archiving in the AWS Cloud
 
(BDT201) Big Data and HPC State of the Union | AWS re:Invent 2014
(BDT201) Big Data and HPC State of the Union | AWS re:Invent 2014(BDT201) Big Data and HPC State of the Union | AWS re:Invent 2014
(BDT201) Big Data and HPC State of the Union | AWS re:Invent 2014
 
Beyond Relational
Beyond RelationalBeyond Relational
Beyond Relational
 
DATA LAKE AND THE RISE OF THE MICROSERVICES - ALEX BORDEI
DATA LAKE AND THE RISE OF THE MICROSERVICES - ALEX BORDEIDATA LAKE AND THE RISE OF THE MICROSERVICES - ALEX BORDEI
DATA LAKE AND THE RISE OF THE MICROSERVICES - ALEX BORDEI
 
Cloud Migration
Cloud MigrationCloud Migration
Cloud Migration
 
Part 2: Cloudera’s Operational Database: Unlocking New Benefits in the Cloud
Part 2: Cloudera’s Operational Database: Unlocking New Benefits in the CloudPart 2: Cloudera’s Operational Database: Unlocking New Benefits in the Cloud
Part 2: Cloudera’s Operational Database: Unlocking New Benefits in the Cloud
 
(CMP202) Engineering Simulation and Analysis in the Cloud
(CMP202) Engineering Simulation and Analysis in the Cloud(CMP202) Engineering Simulation and Analysis in the Cloud
(CMP202) Engineering Simulation and Analysis in the Cloud
 
4 C’s for Using Cloud to Support Scientific Research
4 C’s for Using Cloud to Support Scientific Research4 C’s for Using Cloud to Support Scientific Research
4 C’s for Using Cloud to Support Scientific Research
 
Intro to High Performance Computing in the AWS Cloud
Intro to High Performance Computing in the AWS CloudIntro to High Performance Computing in the AWS Cloud
Intro to High Performance Computing in the AWS Cloud
 
How To Build A Stable And Robust Base For a “Cloud”
How To Build A Stable And Robust Base For a “Cloud”How To Build A Stable And Robust Base For a “Cloud”
How To Build A Stable And Robust Base For a “Cloud”
 
ADDO 2021: Why and how to include database changes in the deployment pipeline
ADDO 2021: Why and how to include database changes in the deployment pipelineADDO 2021: Why and how to include database changes in the deployment pipeline
ADDO 2021: Why and how to include database changes in the deployment pipeline
 
Deep Learning in the Cloud at Scale: A Data Orchestration Story
Deep Learning in the Cloud at Scale: A Data Orchestration StoryDeep Learning in the Cloud at Scale: A Data Orchestration Story
Deep Learning in the Cloud at Scale: A Data Orchestration Story
 
Running your database in the cloud presentation
Running your database in the cloud presentationRunning your database in the cloud presentation
Running your database in the cloud presentation
 

Viewers also liked

Viewers also liked (6)

MT09 Using Dell’s HPC Cloud Solutions to maximize HPC utilization while reduc...
MT09 Using Dell’s HPC Cloud Solutions to maximize HPC utilization while reduc...MT09 Using Dell’s HPC Cloud Solutions to maximize HPC utilization while reduc...
MT09 Using Dell’s HPC Cloud Solutions to maximize HPC utilization while reduc...
 
Nasa HPC in the Cloud
Nasa HPC in the CloudNasa HPC in the Cloud
Nasa HPC in the Cloud
 
Measuring HPC: Performance, Cost, & Value
Measuring HPC: Performance, Cost, & ValueMeasuring HPC: Performance, Cost, & Value
Measuring HPC: Performance, Cost, & Value
 
IDC HPC Market Update
IDC HPC Market UpdateIDC HPC Market Update
IDC HPC Market Update
 
2016 IDC HPC Market Update
2016 IDC HPC Market Update2016 IDC HPC Market Update
2016 IDC HPC Market Update
 
HPC Market Update from IDC
HPC Market Update from IDCHPC Market Update from IDC
HPC Market Update from IDC
 

Similar to Deliver Best-in-Class HPC Cloud Solutions Without Losing Your Mind

Engineering Machine Learning Data Pipelines Series: Streaming New Data as It ...
Engineering Machine Learning Data Pipelines Series: Streaming New Data as It ...Engineering Machine Learning Data Pipelines Series: Streaming New Data as It ...
Engineering Machine Learning Data Pipelines Series: Streaming New Data as It ...
Precisely
 
November 2013 HUG: Cyber Security with Hadoop
November 2013 HUG: Cyber Security with HadoopNovember 2013 HUG: Cyber Security with Hadoop
November 2013 HUG: Cyber Security with Hadoop
Yahoo Developer Network
 
Journey to the Programmable Data Center
Journey to the Programmable Data CenterJourney to the Programmable Data Center
Journey to the Programmable Data Center
Toby Weiss
 
Data Warehouse Modernization - Big Data in the Cloud Success with Qubole on O...
Data Warehouse Modernization - Big Data in the Cloud Success with Qubole on O...Data Warehouse Modernization - Big Data in the Cloud Success with Qubole on O...
Data Warehouse Modernization - Big Data in the Cloud Success with Qubole on O...
Qubole
 

Similar to Deliver Best-in-Class HPC Cloud Solutions Without Losing Your Mind (20)

Engineering Machine Learning Data Pipelines Series: Streaming New Data as It ...
Engineering Machine Learning Data Pipelines Series: Streaming New Data as It ...Engineering Machine Learning Data Pipelines Series: Streaming New Data as It ...
Engineering Machine Learning Data Pipelines Series: Streaming New Data as It ...
 
Meta scale kognitio hadoop webinar
Meta scale kognitio hadoop webinarMeta scale kognitio hadoop webinar
Meta scale kognitio hadoop webinar
 
November 2013 HUG: Cyber Security with Hadoop
November 2013 HUG: Cyber Security with HadoopNovember 2013 HUG: Cyber Security with Hadoop
November 2013 HUG: Cyber Security with Hadoop
 
IBM Aspera for high-speed data migration to your AWS Cloud - DEM02-S - New Yo...
IBM Aspera for high-speed data migration to your AWS Cloud - DEM02-S - New Yo...IBM Aspera for high-speed data migration to your AWS Cloud - DEM02-S - New Yo...
IBM Aspera for high-speed data migration to your AWS Cloud - DEM02-S - New Yo...
 
IBM Aspera for High Speed Data Migration to Your AWS Cloud - DEM06-S - Anahei...
IBM Aspera for High Speed Data Migration to Your AWS Cloud - DEM06-S - Anahei...IBM Aspera for High Speed Data Migration to Your AWS Cloud - DEM06-S - Anahei...
IBM Aspera for High Speed Data Migration to Your AWS Cloud - DEM06-S - Anahei...
 
Journey to the Programmable Data Center
Journey to the Programmable Data CenterJourney to the Programmable Data Center
Journey to the Programmable Data Center
 
Estimating the Total Costs of Your Cloud Analytics Platform
Estimating the Total Costs of Your Cloud Analytics PlatformEstimating the Total Costs of Your Cloud Analytics Platform
Estimating the Total Costs of Your Cloud Analytics Platform
 
Lecture 3.31 3.32.pptx
Lecture 3.31  3.32.pptxLecture 3.31  3.32.pptx
Lecture 3.31 3.32.pptx
 
What is Cloud computing?
What is Cloud computing?What is Cloud computing?
What is Cloud computing?
 
Migration Recipes for Success - AWS Summit Cape Town 2017
Migration Recipes for Success - AWS Summit Cape Town 2017 Migration Recipes for Success - AWS Summit Cape Town 2017
Migration Recipes for Success - AWS Summit Cape Town 2017
 
Managing Performance in the Cloud
Managing Performance in the CloudManaging Performance in the Cloud
Managing Performance in the Cloud
 
An overview of modern scalable web development
An overview of modern scalable web developmentAn overview of modern scalable web development
An overview of modern scalable web development
 
Enabling big data & AI workloads on the object store at DBS
Enabling big data & AI workloads on the object store at DBS Enabling big data & AI workloads on the object store at DBS
Enabling big data & AI workloads on the object store at DBS
 
A Successful Journey to the Cloud with Data Virtualization
A Successful Journey to the Cloud with Data VirtualizationA Successful Journey to the Cloud with Data Virtualization
A Successful Journey to the Cloud with Data Virtualization
 
Hpc lunch and learn
Hpc lunch and learnHpc lunch and learn
Hpc lunch and learn
 
How Data Drives Business at Choice Hotels
How Data Drives Business at Choice HotelsHow Data Drives Business at Choice Hotels
How Data Drives Business at Choice Hotels
 
(STG308) How EA, State Of Texas & H3 Biomedicine Protect Data
(STG308) How EA, State Of Texas & H3 Biomedicine Protect Data(STG308) How EA, State Of Texas & H3 Biomedicine Protect Data
(STG308) How EA, State Of Texas & H3 Biomedicine Protect Data
 
ADV Slides: When and How Data Lakes Fit into a Modern Data Architecture
ADV Slides: When and How Data Lakes Fit into a Modern Data ArchitectureADV Slides: When and How Data Lakes Fit into a Modern Data Architecture
ADV Slides: When and How Data Lakes Fit into a Modern Data Architecture
 
Nairobi OpenStack Meetup - July 2013
Nairobi OpenStack Meetup - July 2013Nairobi OpenStack Meetup - July 2013
Nairobi OpenStack Meetup - July 2013
 
Data Warehouse Modernization - Big Data in the Cloud Success with Qubole on O...
Data Warehouse Modernization - Big Data in the Cloud Success with Qubole on O...Data Warehouse Modernization - Big Data in the Cloud Success with Qubole on O...
Data Warehouse Modernization - Big Data in the Cloud Success with Qubole on O...
 

More from Avere Systems

More from Avere Systems (18)

Scaling Security Workflows in Government Agencies
Scaling Security Workflows in Government AgenciesScaling Security Workflows in Government Agencies
Scaling Security Workflows in Government Agencies
 
Hedge Fund IT Challenges Financial Survey
Hedge Fund IT Challenges Financial SurveyHedge Fund IT Challenges Financial Survey
Hedge Fund IT Challenges Financial Survey
 
Cloud Bursting 101: What to do When Cloud Computing Demand Exceeds Capacity
Cloud Bursting 101: What to do When Cloud Computing Demand Exceeds CapacityCloud Bursting 101: What to do When Cloud Computing Demand Exceeds Capacity
Cloud Bursting 101: What to do When Cloud Computing Demand Exceeds Capacity
 
Solving enterprise challenges through scale out storage & big compute final
Solving enterprise challenges through scale out storage & big compute finalSolving enterprise challenges through scale out storage & big compute final
Solving enterprise challenges through scale out storage & big compute final
 
Compute Cloud for Rendering
Compute Cloud for RenderingCompute Cloud for Rendering
Compute Cloud for Rendering
 
Rendering Takes Flight
Rendering Takes FlightRendering Takes Flight
Rendering Takes Flight
 
Three Steps to Modern Media Asset Management with Active Archive
Three Steps to Modern Media Asset Management with Active ArchiveThree Steps to Modern Media Asset Management with Active Archive
Three Steps to Modern Media Asset Management with Active Archive
 
Cloud Computing Gets Put to the Test
Cloud Computing Gets Put to the TestCloud Computing Gets Put to the Test
Cloud Computing Gets Put to the Test
 
Scientific Computing in the Cloud: Speeding Access for Drug Discovery
Scientific Computing in the Cloud: Speeding Access for Drug DiscoveryScientific Computing in the Cloud: Speeding Access for Drug Discovery
Scientific Computing in the Cloud: Speeding Access for Drug Discovery
 
Build a Cloud Render-Ready Infrastructure
Build a Cloud Render-Ready InfrastructureBuild a Cloud Render-Ready Infrastructure
Build a Cloud Render-Ready Infrastructure
 
Share on LinkedIn Share on Twitter Share on Facebook Share on Google+ Share b...
Share on LinkedIn Share on Twitter Share on Facebook Share on Google+ Share b...Share on LinkedIn Share on Twitter Share on Facebook Share on Google+ Share b...
Share on LinkedIn Share on Twitter Share on Facebook Share on Google+ Share b...
 
Avere & AWS Enterprise Solution with Special Bundle Pricing Offer
Avere & AWS Enterprise Solution with Special Bundle Pricing OfferAvere & AWS Enterprise Solution with Special Bundle Pricing Offer
Avere & AWS Enterprise Solution with Special Bundle Pricing Offer
 
Enable Enterprise Hybrid Cloud NAS
Enable Enterprise Hybrid Cloud NASEnable Enterprise Hybrid Cloud NAS
Enable Enterprise Hybrid Cloud NAS
 
Avere Cloud NAS
Avere Cloud NASAvere Cloud NAS
Avere Cloud NAS
 
Clouds in Your Coffee Session with Cleversafe & Avere
Clouds in Your Coffee Session with Cleversafe & AvereClouds in Your Coffee Session with Cleversafe & Avere
Clouds in Your Coffee Session with Cleversafe & Avere
 
Are you ready for Avere Cloud NAS?
Are you ready for Avere Cloud NAS?Are you ready for Avere Cloud NAS?
Are you ready for Avere Cloud NAS?
 
Optimizing the Upstreaming Workflow: Flexibly Scale Storage for Seismic Proce...
Optimizing the Upstreaming Workflow: Flexibly Scale Storage for Seismic Proce...Optimizing the Upstreaming Workflow: Flexibly Scale Storage for Seismic Proce...
Optimizing the Upstreaming Workflow: Flexibly Scale Storage for Seismic Proce...
 
Webinar: Untethering Compute from Storage
Webinar: Untethering Compute from StorageWebinar: Untethering Compute from Storage
Webinar: Untethering Compute from Storage
 

Recently uploaded

Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...
FIDO Alliance
 
CORS (Kitworks Team Study 양다윗 발표자료 240510)
CORS (Kitworks Team Study 양다윗 발표자료 240510)CORS (Kitworks Team Study 양다윗 발표자료 240510)
CORS (Kitworks Team Study 양다윗 발표자료 240510)
Wonjun Hwang
 
Tales from a Passkey Provider Progress from Awareness to Implementation.pptx
Tales from a Passkey Provider  Progress from Awareness to Implementation.pptxTales from a Passkey Provider  Progress from Awareness to Implementation.pptx
Tales from a Passkey Provider Progress from Awareness to Implementation.pptx
FIDO Alliance
 
Microsoft BitLocker Bypass Attack Method.pdf
Microsoft BitLocker Bypass Attack Method.pdfMicrosoft BitLocker Bypass Attack Method.pdf
Microsoft BitLocker Bypass Attack Method.pdf
Overkill Security
 

Recently uploaded (20)

JohnPollard-hybrid-app-RailsConf2024.pptx
JohnPollard-hybrid-app-RailsConf2024.pptxJohnPollard-hybrid-app-RailsConf2024.pptx
JohnPollard-hybrid-app-RailsConf2024.pptx
 
Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...
 
ADP Passwordless Journey Case Study.pptx
ADP Passwordless Journey Case Study.pptxADP Passwordless Journey Case Study.pptx
ADP Passwordless Journey Case Study.pptx
 
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
 
Portal Kombat : extension du réseau de propagande russe
Portal Kombat : extension du réseau de propagande russePortal Kombat : extension du réseau de propagande russe
Portal Kombat : extension du réseau de propagande russe
 
Frisco Automating Purchase Orders with MuleSoft IDP- May 10th, 2024.pptx.pdf
Frisco Automating Purchase Orders with MuleSoft IDP- May 10th, 2024.pptx.pdfFrisco Automating Purchase Orders with MuleSoft IDP- May 10th, 2024.pptx.pdf
Frisco Automating Purchase Orders with MuleSoft IDP- May 10th, 2024.pptx.pdf
 
The Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightThe Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and Insight
 
Introduction to FIDO Authentication and Passkeys.pptx
Introduction to FIDO Authentication and Passkeys.pptxIntroduction to FIDO Authentication and Passkeys.pptx
Introduction to FIDO Authentication and Passkeys.pptx
 
AI mind or machine power point presentation
AI mind or machine power point presentationAI mind or machine power point presentation
AI mind or machine power point presentation
 
Design Guidelines for Passkeys 2024.pptx
Design Guidelines for Passkeys 2024.pptxDesign Guidelines for Passkeys 2024.pptx
Design Guidelines for Passkeys 2024.pptx
 
Overview of Hyperledger Foundation
Overview of Hyperledger FoundationOverview of Hyperledger Foundation
Overview of Hyperledger Foundation
 
CORS (Kitworks Team Study 양다윗 발표자료 240510)
CORS (Kitworks Team Study 양다윗 발표자료 240510)CORS (Kitworks Team Study 양다윗 발표자료 240510)
CORS (Kitworks Team Study 양다윗 발표자료 240510)
 
Top 10 CodeIgniter Development Companies
Top 10 CodeIgniter Development CompaniesTop 10 CodeIgniter Development Companies
Top 10 CodeIgniter Development Companies
 
2024 May Patch Tuesday
2024 May Patch Tuesday2024 May Patch Tuesday
2024 May Patch Tuesday
 
Continuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
Continuing Bonds Through AI: A Hermeneutic Reflection on ThanabotsContinuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
Continuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
 
Simplifying Mobile A11y Presentation.pptx
Simplifying Mobile A11y Presentation.pptxSimplifying Mobile A11y Presentation.pptx
Simplifying Mobile A11y Presentation.pptx
 
Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...
Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...
Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...
 
Tales from a Passkey Provider Progress from Awareness to Implementation.pptx
Tales from a Passkey Provider  Progress from Awareness to Implementation.pptxTales from a Passkey Provider  Progress from Awareness to Implementation.pptx
Tales from a Passkey Provider Progress from Awareness to Implementation.pptx
 
Intro to Passkeys and the State of Passwordless.pptx
Intro to Passkeys and the State of Passwordless.pptxIntro to Passkeys and the State of Passwordless.pptx
Intro to Passkeys and the State of Passwordless.pptx
 
Microsoft BitLocker Bypass Attack Method.pdf
Microsoft BitLocker Bypass Attack Method.pdfMicrosoft BitLocker Bypass Attack Method.pdf
Microsoft BitLocker Bypass Attack Method.pdf
 

Deliver Best-in-Class HPC Cloud Solutions Without Losing Your Mind

  • 1. Deliver Best-in-Class HPC Cloud Solutions Without Losing Your Mind WEBINAR April 13, 2016, 11:00 AM ET
  • 2. Housekeeping • Audio help • Attachments • Questions • Rating
  • 3. Today’s Speakers Rick Friedman Vice President, Solution Development Cycle Computing Scott Jeschonek Director of Product Management, Cloud Avere Systems
  • 4. Agenda • Discuss the current state of HPC • Clouds and their impact on your HPC world • Reasons why you aren’t 100% cloud-based already • The Hybrid Cloud and HPC • Possible implementations • Delivering File Systems using Avere Systems • Orchestration using Cycle Computing
  • 5. HPC Today (and Yesterday, and Tomorrow)
  • 6. What Drives Today’s Needs • Data – Who, what, when, how much, where? • Datacenter limitations – Can I defy physics? • User expectations – Can we even do that? • Technology shifts – What is the “best practice”?
  • 7. Big Compute Workloads: How are they handled? Compute Demand vs. Cluster Size Cluster Size Compute Demand Missed Opportunity Wasted Resources • Internal infrastructure has huge value and some limitations • Access, not capacity, is the barrier to continued growth • Perception limits scale of problem solving • Public cloud = cost-effective, readily available resources to users with problems & deadlines. • Financial services, manufacturing and life sciences are leading the way.
  • 8. Basic HPC Environment Requirements Resource Manager Jobs Manager / Scheduler Workload NAS Storage Lots of compute resources (“Grid”)
  • 9. Advantages of Clouds Significantly reduce infrastructure management costs both in money and time Maintain operational flexibility during scale-out jobs…let the provider deal with scale challenges
  • 10. Why the Cloud for Big Compute? • Scientist / Engineer User perspective – Zero queue times, capacity in minutes – Scale compute to problems size, not vice versa – Try / support new computational approaches and software quickly • SysArchitect perspective – Dynamically adjust workloads to “lowest cost/impact” provider – Focus on computational excellence, not hardware management – Support a wide range of user types efficiently • Organizational perspective – Match spending to actual consumption – Increase responsiveness to business dynamics – Grow user base without hardware limitations
  • 11. Clouds Have Awesome New Capabilities • Big Data – Analytics Tools – Massively scalable NoSQL – Data warehousing • Machine Learning – Voice/Vision/Speech – Early days
  • 12. So…why isn’t everything in the cloud? • Current infrastructure investment (capex) • Cloud costs not yet completely in line • Software infrastructure in place – Costs to refactor, dependencies to consider • Data environment in one or more data centers • Orchestration and management of cloud clusters is hard • Network bandwidth / latency concerns • Business Continuity
  • 13. Other Reasons You’re Not 100% in the Cloud • Corporate budgets • Corporate policies • Corporate politics • Education / awareness • Government regulations • Interest groups • Vendor relationships
  • 14. Near Future, Hybrid Cloud Tokyo office London office Analysts Analysts NYC office Analysts AnalystsAnalysts Analysts AnalystsAnalysts AnalystsAnalysts Hong Kong office • Adoption of one or more cloud providers • > 1 hedge on price and SLA • Mix of on-prem and cloud resources • Regulatory, proprietary and/or security characteristics will likely keep data in the DC NAS Primary DC Cloud Provider 1 Cloud Provider 2 NAS Secondary DC
  • 15. Cloud Compute Environment Data HPC in the Cloud Cloud Compute API Scheduler NAS Storage Analysts Scheduler AnalystsAnalysts Analysts Analysts Analysts Jobs On-Premises Data Center
  • 16. Cloud Compute Environment HPC in the Cloud, “Grids on Demand” Cloud Compute API Data NAS Storage Analysts Scheduler AnalystsAnalysts Analysts Analysts Analysts Jobs On-Premises Data Center Scheduler1 Scheduler2 Scheduler3 Scheduler4
  • 17. Challenges with HPC in the Cloud • How do you get the data close to your compute nodes? • How do you orchestrate on-demand clusters/grids of compute nodes? • How does this all come together??
  • 18. Cloud Compute Environment Data Access Layer Cloud Compute API Scheduler1 Data NAS Storage Analysts Scheduler AnalystsAnalysts Analysts Analysts Analysts Jobs On-Premises Data Center Data Access Layer Scheduler2 Scheduler3 Scheduler4 • File System • Caching Layer • Only load necessary blocks of files • Opaque to compute nodes
  • 19. Advantages of Data Access / Cache Layer • Keep your data on prem! – Data in cloud is only there while the compute nodes work the jobs. – Reduce the security objections, simplify the move to cloud • Increase cloud compute performance – using file system caching, most of the data will be in RAM, close to the nodes – Avoids ingest latencies and slashes transit latency after first read • Scale out – Using solution that facilitates 10s of 1000s of core file system connections
  • 20. Typical File Access in Hadoop Cluster Caching files will work for certain types of jobs Where typical file is accessed By multiple clients source: http://blog.cloudera.com/blog/2012/09/what-do-real-life-hadoop-workloads-look-like/
  • 21. Hybrid Cloud using Avere FXT and vFXT Edge Filers Cloud Compute On-Prem Compute Cloud Storage On-Prem Storage NAS Object Bucket 1 Bucket 2 Bucket n Virtual Compute Farm Virtual FXT File Storage for Private Object NAS Optimization Cloud NAS Physical FXT The “Edge” = locating your data Close to your compute Without truly moving it from your NAS environment
  • 22. Avere Building Blocks “Avere is uniquely positioned to offer scale across tens of thousands of cloud compute cores while leaving the data where it originates, on premises, with it’s global file system and caching capabilities.” - Unnamed CTO Cloud Compute Virtual FXT NAS Object Physical FXT Cloud On-Premises File Acceleration
  • 23. Cloud Compute Environment Orchestration and Management Layer Cloud Compute API Data On-Premises Data Center Scheduler1 Scheduler2 Scheduler3 Scheduler4 NAS Storage Analysts Scheduler AnalystsAnalysts Analysts Analysts Analysts Jobs
  • 24. Optimization • Benchmark instances • Make Workflow UI • Human workflow Provisioning • Workload placement Optimal scale • Cost optimization • Data scheduling Cluster Configuration • Multi-cloud, without changes • Pre-set or User-defined “types” • Abstraction for all cluster data, attributes (roles, OS, etc) Monitoring • Auto-scaling • Usage tracking • Error Handling • Reporting Internal File: Declarative Cluster Definition Packages, Installers Containers, Data Admin Scope Configure Run on Cloud Optimize User Complete Multi-Cloud Workflow Control
  • 25. User Web UI API CMD Line Job & Data Workflow Automated Job Placement, Cost optimization Auto-scaling, Benchmarking, Compliance, Reporting tools Multi-cloud Without Changes Internal Cluster How Cycle Makes Cloud Productive • Scientist / Engineer productivity: – Simple workflows – Zero queue time – Auto-scaling • SysAdmin productivity: – Instant access to additional resources – Workflows linking internal and multiple clouds – Simple reliable tools to enable apps with special requirements • Organizational productivity: – Secure, consistent cloud access – Usage tracking – Ability to leverage multiple providers
  • 26. Big Data w/o Disrupting Production • Challenge – Estimate the carbon stored in Saharan biomass – Rapidly establish a baseline for later research using large amounts of high-resolution remote sensing data – Existing internal compute resources fully committed – Limited window to complete processing • Cycle solution – Full workflow including data management between internal data capture and cloud processing – Leverage spot pricing to minimize cost while maximizing computation • Results – Linearly scalable, predictable enabling plan for next steps – Science being done that could not be done otherwise – 1 month start to initial runs 26
  • 27. Overall Architecture – Data In-House Cloud Compute Scheduler Avere FXT Edge Filer Avere FXT Workload Cloud API NAS Storage Scheduler Cloud Storage
  • 28. What We Covered… • The Current State of HPC • Clouds and Their Impact on Your HPC World • Reasons Why You aren’t 100% Cloud-based Already • The Hybrid Cloud and HPC • Possible Implementations • Delivering File Systems Using Avere Systems • Orchestration Using Cycle Computing
  • 29. Thank you! Cycle Computing Contact Info: More about Avere Systems: askavere@averesystems.com www.averesystems.com 1-888.88.AVERE https://twitter.com/averesystems https://www.youtube.com/user/AvereSystems https://www.linkedin.com/company/589037 info@cyclecomputing.com www.cyclecomputing.com 888.292.5320 https://twitter.com/cyclecomputing https://www.youtube.com/user/CycleComputing https://www.linkedin.com/company/692068

Editor's Notes

  1. Title Slide
  2. Cost-effective internal infrastructure has enabled users to solve increasingly larger problems over the last 15 years while also highlighting some inefficiencies The barrier to sustaining that growth is an access and allocation problem, not a compute problem Big Compute users are limiting the size of the problems they tackle, to the infrastructure they think they can access. Public cloud represents an opportunity to allocate cost-effective, readily available resources to users who value the ability to solve a problem within a deadline. Financial services, manufacturing and life sciences are leading the way; their problem is most acute and solving it has measureable business benefit
  3. Title Slide