Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
WEBINAR
Solving Enterprise Business Challenges
Through Scale-Out Storage & Big Compute
Michael Basilyan, Product Manager, ...
Housekeeping
• Slides
• Questions
• Recording
• Attachments
Presenters
Michael Basilyan
Product Manager
Scott Jeschonek
Director of Cloud Products
Rob Futrick
CTO
Introduction to Google Cloud Platform
Focusing on Compute Engine & Storage
Michael Basilyan
basilyan@google.com
Product Ma...
Agenda
• Google Cloud Overview
• Compute Engine VMs:
• GCE VM Instances & Managed Infrastructure
• Storage:
• Block Storag...
What is Google Cloud
Platform?
7
Google Cloud Platform Services
VIRTUAL NETWORK
LOAD BALANCING
CDN
DNS
INTERCONNECT
Management Compute Storage Networking...
8
Google Cloud Platform Services
VIRTUAL NETWORK
LOAD BALANCING
CDN
DNS
INTERCONNECT
Management Compute Storage Networking...
GCE: Compute & VM Features
VM Live Migration = No Downtime
Custom Machine Types
Average Savings: 19%
Create VMs shaped for your workloads instead of shaping your workloads to fit pr...
Preemptible VMs
Ideal for batch, grid, and fault-tolerant workloads
Save 80% off regular VM list prices: flat $0.01 per co...
Managed Infrastructure - zero devops for IaaS
Create Groups of
Instances
- Define Instance Template
- Deploy Docker contai...
Ways we save you money
● Preemptible VMs
● Custom Machine Types
● Per-minute billing
● Sustained Use Discount
○ The more y...
Block & Object Storage
Cloud
Storage
Cloud
Bigtable
Cloud
Datastore
Cloud
SQL
Good for:
Binary or object
data (BLOB)
Such as:
Media, analytics,
a...
Cloud
Storage
Cloud
Bigtable
Cloud
Datastore
Cloud
SQL
Good for:
Binary or object
data (BLOB)
Such as:
Media, analytics,
a...
Block Storage
Reliable, high-performance block storage for virtual machine instances on GCE
Standard Persistent
Disk
SSD P...
Cloud Storage: Object/Blog store
● Google Cloud Storage is a scalable
object storage service suitable for
all kinds of uns...
Regions and Zones
–––– 2018
2018
Current regions and number of zones
Edge points of presence
Network
Committed regions for 2017 and number
o...
Cloud HPC:
Data Access Challenges
Scott Jeschonek, Director of Cloud Products
HPC in the Cloud
• Bring 100s or 1000s of cores online, quickly and efficiently
• Networking within the Cloud Compute envi...
“Pure” Cloud HPC
• Entire grid in Compute
Cloud
• Data is located locally
•
Cloud Storage options
may be used
• 3rd party ...
Hybrid HPC
Existing HPC clusters:
Capital investment
- Possibly sunk cost already
Logical investment:
- Hardware Tuned
- S...
Hybrid HPC
Grids On-Demand
Latency “Kills”
• Access to Data is the main challenge for HPC
• Amplified in the cloud:
- Data has to be located on or ne...
Using a Data Access Layer
Advantages of Data Access Layer
Keep your data on prem! – Data in cloud is only there while the compute
nodes work the job...
Hybrid Cloud / Hybrid HPC Using Avere Technology
Customer Needs Avere Delivers
Low-latency file access Edge-Core Architect...
Lessons Learned from 10 Years
How Cloud Changes Big Compute
Rob Futrick, CTO
33
The Broad
Institute
Need: 270,000 hours of
computing
Why: Machine learning to
map relationships among
cancer datasets
© Copyright Cycle Computing LLC | All Rights Reserved
PAGE 34
Internal cluster queue too
long
Up & running in 1 hour,
scal...
© Copyright Cycle Computing LLC | All Rights Reserved
PAGE 35
Manufacturing
& Electronics
Pharma &
Biotech
Financial &
Ins...
© Copyright Cycle Computing LLC | All Rights Reserved
PAGE 36
The Challenges: Cloud & HPC Big Compute
User
Inputs
Existing...
© Copyright Cycle Computing LLC | All Rights Reserved
PAGE 37
The Solution: CycleCloud for Cloud HPC & Big Compute
User
In...
© Copyright Cycle Computing LLC | All Rights Reserved
PAGE 38
Who is Cycle Computing?
• Leader in Cloud Big Compute/HPC
• ...
© 2016 Copyright | All rights reserved
7 Lessons Learned
© Copyright Cycle Computing LLC | All Rights Reserved
PAGE 40
#1 – Zero waiting in line for compute
© Copyright Cycle Computing LLC | All Rights Reserved
PAGE 41
#2 – Ask questions of any scale
Ask the right question,
rega...
© Copyright Cycle Computing LLC | All Rights Reserved
PAGE 42
#3 – Users with unique requirements are OK
Trivial to suppor...
© Copyright Cycle Computing LLC | All Rights Reserved
PAGE 43
#4 – Cloud gets faster/cheaper over time
© Copyright Cycle Computing LLC | All Rights Reserved
PAGE 44
#5 – Time & cost are the sole metrics that matter
© Copyright Cycle Computing LLC | All Rights Reserved
PAGE 45
Everything
you don’t
think about!
© Copyright Cycle Computing LLC | All Rights Reserved
PAGE 46
#6 – Accelerating answers, accelerates people
720 (hours) 72...
© Copyright Cycle Computing LLC | All Rights Reserved
PAGE 47
#6 – Accelerating answers, accelerates people
720 (hours) 72...
© Copyright Cycle Computing LLC | All Rights Reserved
PAGE 48
#7 – Every smart person gets their own workspace
Old: Shared...
© Copyright Cycle Computing LLC | All Rights Reserved
PAGE 49
Lessons Learned Summary
1. Zero Queue Wait for
computing
2. ...
© Copyright Cycle Computing LLC | All Rights Reserved
PAGE 50
The Solution: CycleCloud for Cloud HPC & Big Compute
User
In...
Questions
& Answers
Contact Information
Michael Basilyan
Product Manager
basilyan@google.com
cloud.google.com
Scott Jeschonek
Director of Clou...
Upcoming SlideShare
Loading in …5
×

Solving enterprise challenges through scale out storage & big compute final

Google Cloud Platform, Avere Systems, and Cycle Computing experts will share best practices for advancing solutions to big challenges faced by enterprises with growing compute and storage needs. In this “best practices” webinar, you’ll hear how these companies are working to improve results that drive businesses forward through scalability, performance, and ease of management.

The slides were from a webinar presented January 24, 2017. The audience learned:
- How enterprises are using Google Cloud Platform to gain compute and storage capacity on-demand
- Best practices for efficient use of cloud compute and storage resources
- Overcoming the need for file systems within a hybrid cloud environment
- Understand how to eliminate latency between cloud and data center architectures
- Learn how to best manage simulation, analytics, and big data workloads in dynamic environments
- Look at market dynamics drawing companies to new storage models over the next several years

Presenters communicated a foundation to build infrastructure to support ongoing demand growth.

  • Login to see the comments

  • Be the first to like this

Solving enterprise challenges through scale out storage & big compute final

  1. 1. WEBINAR Solving Enterprise Business Challenges Through Scale-Out Storage & Big Compute Michael Basilyan, Product Manager, Google Cloud Platform Scott Jeschonek, Director of Cloud Products, Avere Systems Rob Futrick, CTO, Cycle Computing
  2. 2. Housekeeping • Slides • Questions • Recording • Attachments
  3. 3. Presenters Michael Basilyan Product Manager Scott Jeschonek Director of Cloud Products Rob Futrick CTO
  4. 4. Introduction to Google Cloud Platform Focusing on Compute Engine & Storage Michael Basilyan basilyan@google.com Product Manager, GCE
  5. 5. Agenda • Google Cloud Overview • Compute Engine VMs: • GCE VM Instances & Managed Infrastructure • Storage: • Block Storage • Cloud Storage
  6. 6. What is Google Cloud Platform?
  7. 7. 7 Google Cloud Platform Services VIRTUAL NETWORK LOAD BALANCING CDN DNS INTERCONNECT Management Compute Storage Networking Data Machine Learning STACKDRIVER IDENTITY AND ACCESS MANAGEMENT CLOUD ML SPEECH API VISION API TRANSLATE API NATURAL LANGUAGE API
  8. 8. 8 Google Cloud Platform Services VIRTUAL NETWORK LOAD BALANCING CDN DNS INTERCONNECT Management Compute Storage Networking Data Machine Learning STACKDRIVER IDENTITY AND ACCESS MANAGEMENT CLOUD ML SPEECH API VISION API TRANSLATE API NATURAL LANGUAGE API
  9. 9. GCE: Compute & VM Features
  10. 10. VM Live Migration = No Downtime
  11. 11. Custom Machine Types Average Savings: 19% Create VMs shaped for your workloads instead of shaping your workloads to fit pre-defined VMs.
  12. 12. Preemptible VMs Ideal for batch, grid, and fault-tolerant workloads Save 80% off regular VM list prices: flat $0.01 per core hour Flat pricing with no complex bidding or competition Same performance (CPU, I/O, Net) as regular VMs Example uses: Hadoop, Rendering/Transcoding, Genomics, Monte Carlo Simulations, etc.
  13. 13. Managed Infrastructure - zero devops for IaaS Create Groups of Instances - Define Instance Template - Deploy Docker containers or apps directly - Automatically connect new instances to load balancer Autoheal - Use app level healthcheck to signal issue - Get machine recreated or restarted Autoscale - Add/Remove instances automatically based on scaling policy (CPU utilization, LB load, Custom Metrics) - Scale pool of workers with task queue Update - Deploy new version of your software with rolling update while serving traffic - Do cannary, % rollout, control pace, roll-back - Recreate in place or surge instances
  14. 14. Ways we save you money ● Preemptible VMs ● Custom Machine Types ● Per-minute billing ● Sustained Use Discount ○ The more you use, the bigger the discount. Automatically. ● Instance right-sizing ○ Instance recommendations displayed on VM Instances Page ○ Single Button Actuation
  15. 15. Block & Object Storage
  16. 16. Cloud Storage Cloud Bigtable Cloud Datastore Cloud SQL Good for: Binary or object data (BLOB) Such as: Media, analytics, archive/backup Good for: Hierarchical, mobile, web Such as: User profiles, Game State Good for: Web frameworks Such as: CMS, eCommerce Good for: Heavy read + write, events, Such as: AdTech, Financial, IoT Where do I store my data? Big Query Good for: Data Warehouse Such as: Analytics, Dashboards Relational NoSQL Object Warehouse Good for: Local VM file storage Such as: Application data/binaries Block Persistent Disk (GCE)
  17. 17. Cloud Storage Cloud Bigtable Cloud Datastore Cloud SQL Good for: Binary or object data (BLOB) Such as: Media, analytics, archive/backup Good for: Hierarchical, mobile, web Such as: User profiles, Game State Good for: Web frameworks Such as: CMS, eCommerce Good for: Heavy read + write, events, Such as: AdTech, Financial, IoT Where do I store my data? Big Query Good for: Data Warehouse Such as: Analytics, Dashboards Relational NoSQL Object Warehouse Good for: Local VM file storage Such as: Application data/binaries Block Persistent Disk (GCE)
  18. 18. Block Storage Reliable, high-performance block storage for virtual machine instances on GCE Standard Persistent Disk SSD Persistent Disk Local SSD Target scenarios Large data processing workloads and some enterprise applications Genomics processing, video transcoding in GCE High performance database and enterprise applications MySQL, SQL Server, Oracle In-memory databases High-performance scratch space Features Persistent storage Cost sensitive ($.04 GB) Persistent storage Performance sensitive ($0.17GB) Ephemeral storage Highest-performance ($0.218 GB) Encryption, Snapshots 64 TB, Disk Size sets performance (Attach larger VMS for max SSD performance) Encryption 3TB
  19. 19. Cloud Storage: Object/Blog store ● Google Cloud Storage is a scalable object storage service suitable for all kinds of unstructured data. ● Cloud Storage vs Perst. Disk: ○ Scales to exabytes. ○ Accessible from anywhere. ○ REST interface; higher latency than locally attached block storage (PD) ○ Write semantics include insert and overwrite file only. ○ Offers versioning. ○ Cheaper! ● Lots of guidelines on picking storage on our site.
  20. 20. Regions and Zones
  21. 21. –––– 2018 2018 Current regions and number of zones Edge points of presence Network Committed regions for 2017 and number of zones # # https://peering.google.com https://cloud.google.com/compute/docs/regions-zones/regions-zones Google Cloud Platform Infrastructure Google Cloud Platform is built on a datacenter network infrastructure that supports Google scale, performance, and availability 2 3 Singapore2 S Carolina N Virginia Belgium London Tokyo Taiwan Mumbai Sydney Oregon Iowa Frankfurt São Paulo Finland 3 3 3 3 3 3 2 4 3 3 3
  22. 22. Cloud HPC: Data Access Challenges Scott Jeschonek, Director of Cloud Products
  23. 23. HPC in the Cloud • Bring 100s or 1000s of cores online, quickly and efficiently • Networking within the Cloud Compute environment minimizes compute latency • Creative use of preemptible / spot market VM instances allow large numbers of worker nodes at reasonable cost
  24. 24. “Pure” Cloud HPC • Entire grid in Compute Cloud • Data is located locally • Cloud Storage options may be used • 3rd party Data may be incorporated (from their cloud storage)
  25. 25. Hybrid HPC Existing HPC clusters: Capital investment - Possibly sunk cost already Logical investment: - Hardware Tuned - Storage optimized - Network optimized - Daily ops dependent on status quo Cloud HPC Clusters: Transient investment: - Can build on demand infrastructure Expand on-prem: - Use orchestration and grid management to extend jobs into cloud - Schedule jobs based on performance / cost requirements
  26. 26. Hybrid HPC
  27. 27. Grids On-Demand
  28. 28. Latency “Kills” • Access to Data is the main challenge for HPC • Amplified in the cloud: - Data has to be located on or near the worker nodes - Data may be in your datacenter - Copy it all to the cloud? - Costs for workers grows if data has to be copied to local disks - Pipelines may require multiple writes (of results) - Writes to local storage increases consistency risks - Writes back to on-prem storage introduces significant latency
  29. 29. Using a Data Access Layer
  30. 30. Advantages of Data Access Layer Keep your data on prem! – Data in cloud is only there while the compute nodes work the jobs. - Reduce the security objections, simplify the move to cloud Increase cloud compute performance – using file system caching, most of the data will be in RAM, close to the nodes - Avoids ingest latencies and slashes transit latency after first read Scale out – Using solution that facilitates 10s of 1000s of core file system connections
  31. 31. Hybrid Cloud / Hybrid HPC Using Avere Technology Customer Needs Avere Delivers Low-latency file access Edge-Core Architecture Scalable Performance and Availability Scale-out Clustering NFS & SMB interfaces FlashCloud File System for Object Single pool of storage Global Namespace High Security AES-256 Encryption, KMIP Flexibility Physical and virtual products
  32. 32. Lessons Learned from 10 Years How Cloud Changes Big Compute Rob Futrick, CTO
  33. 33. 33 The Broad Institute Need: 270,000 hours of computing Why: Machine learning to map relationships among cancer datasets
  34. 34. © Copyright Cycle Computing LLC | All Rights Reserved PAGE 34 Internal cluster queue too long Up & running in 1 hour, scaled and completed project in 2 weeks 30 years of Computing in 6 hours! Submit jobs, orchestrate ML application Encrypt, route data to Cloud, return results 51,200 cores To run R ML framework Secure Cluster Cell Line Data, RNA, DNA Scaling Machine Learning @ The Broad Institute
  35. 35. © Copyright Cycle Computing LLC | All Rights Reserved PAGE 35 Manufacturing & Electronics Pharma & Biotech Financial & Insurance Media & Entertainment Oil & Gas 65% of G2000 are limited by access to Big Compute
  36. 36. © Copyright Cycle Computing LLC | All Rights Reserved PAGE 36 The Challenges: Cloud & HPC Big Compute User Inputs Existing Workflows Data Dependencies Instance types Applications Scalability Budget Controls AuthorizationSecurity Stack IT LOB Inputs Job scripts & data Cloud accounts Storage / Data sources OS variations AD / LDAP Authorization
  37. 37. © Copyright Cycle Computing LLC | All Rights Reserved PAGE 37 The Solution: CycleCloud for Cloud HPC & Big Compute User Inputs Existing Workflows Data Dependencies Instance types Applications Scalability Budget Controls AuthorizationSecurity Stack IT LOB Inputs Job scripts & data Cloud accounts Storage / Data sources OS variations AD / LDAP Audit/Compliance data Usage data (User, Group, App) Job run-time by instance data AppServer platform Internal
  38. 38. © Copyright Cycle Computing LLC | All Rights Reserved PAGE 38 Who is Cycle Computing? • Leader in Cloud Big Compute/HPC • Pioneering Cloud Management Software for 10 years • 370M compute-hours managed • Compute hour growth: 7x every 2 years • CycleCloud Value Proposition • Simple Managed Access to Big Compute • Accelerating Innovation for the Enterprise => Faster time to result, with cost control • Our customers • Fortune 500, startups, and public sector • Life sciences & pharma, financial services, manufacturing, insurance, electronics
  39. 39. © 2016 Copyright | All rights reserved 7 Lessons Learned
  40. 40. © Copyright Cycle Computing LLC | All Rights Reserved PAGE 40 #1 – Zero waiting in line for compute
  41. 41. © Copyright Cycle Computing LLC | All Rights Reserved PAGE 41 #2 – Ask questions of any scale Ask the right question, regardless of scale Think about the problem first Then the system
  42. 42. © Copyright Cycle Computing LLC | All Rights Reserved PAGE 42 #3 – Users with unique requirements are OK Trivial to support different use cases Different GPU, RAM, SSD, OS needs can be created easily Move workloads that don’t fit internally to Cloud
  43. 43. © Copyright Cycle Computing LLC | All Rights Reserved PAGE 43 #4 – Cloud gets faster/cheaper over time
  44. 44. © Copyright Cycle Computing LLC | All Rights Reserved PAGE 44 #5 – Time & cost are the sole metrics that matter
  45. 45. © Copyright Cycle Computing LLC | All Rights Reserved PAGE 45 Everything you don’t think about!
  46. 46. © Copyright Cycle Computing LLC | All Rights Reserved PAGE 46 #6 – Accelerating answers, accelerates people 720 (hours) 720 720 Computing Analysis 2880 hours / 120 Days to Decision Computing 720 Analysis SCALABLE COMPUTING (in hours) 720 Computing Analysis Analysis 1456 hours / 60.6 Days to Decision 7208 Computing ANTICIPATED BENEFIT (in hours) 8
  47. 47. © Copyright Cycle Computing LLC | All Rights Reserved PAGE 47 #6 – Accelerating answers, accelerates people 720 (hours) 720 720 Computing Analysis 2880 hours / 120 Days to Decision Computing 720 Analysis SCALABLE COMPUTING (in hours) Higher Quality Output, Iterative Analysis, Less Context Switching Computing & Analysis POST ADOPTION: AGILE DESIGN PROCESS 8
  48. 48. © Copyright Cycle Computing LLC | All Rights Reserved PAGE 48 #7 – Every smart person gets their own workspace Old: Shared internal cluster • Competition for resources • Waiting in line for compute • Zero sum game between users New: Cluster Per Researcher • Remove bottlenecks • Cost controls to manage $ • No waiting = 2x faster users User User User UserUser User UserUserUser User User User
  49. 49. © Copyright Cycle Computing LLC | All Rights Reserved PAGE 49 Lessons Learned Summary 1. Zero Queue Wait for computing 2. Any scale, Any time 3. Users with unique requirements are ok 4. Performance goes up over time, same cost 5. Time and Cost are the sole metrics 6. Faster iterations 7. Every researcher gets their own workspace 49
  50. 50. © Copyright Cycle Computing LLC | All Rights Reserved PAGE 50 The Solution: CycleCloud for Cloud HPC & Big Compute User Inputs Existing Workflows Data Dependencies Instance types Applications Scalability Budget Controls AuthorizationSecurity Stack IT LOB Inputs Job scripts & data Cloud accounts Storage / Data sources OS variations AD / LDAP Audit/Compliance data Usage data (User, Group, App) Job run-time by instance data AppServer platform Internal
  51. 51. Questions & Answers
  52. 52. Contact Information Michael Basilyan Product Manager basilyan@google.com cloud.google.com Scott Jeschonek Director of Cloud Products scottj@averesystems.com AvereSystems.com Rob Futrick CTO rfutrick@cyclecomputing.com CycleComputing.com

×