ACG_Rackspace.pdf
Upcoming SlideShare
Loading in...5
×
 

ACG_Rackspace.pdf

on

  • 582 views

true

true

Statistics

Views

Total Views
582
Slideshare-icon Views on SlideShare
582
Embed Views
0

Actions

Likes
1
Downloads
35
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    ACG_Rackspace.pdf ACG_Rackspace.pdf Presentation Transcript

    • Analytical Compute Grid(ACG)Elastic “Big Data” InfrastructureRackspace® Private Cloud powered by OpenStack® Use Case by Natasha GajicOctober 17, 2012
    • Rackspace’s EBI EnvironmentCurrent Environment “Big Data” Problem  Windows and Linux  Cost of purchasing operating systems additional licenses  Oracle and Microsoft  Time required to set up databases solutions new hardware  Microsoft and Oracle  Increased demand for DBA replication technology resources  SSIS  System performance  Informatica  System scalability  Dedicated servers  Capacity  Rapid data set growth RACKSPACE® HOSTING | WWW.RACKSPACE.COM 2
    • Analytical Compute Grid (ACG) Features• Host ever growing set of data• Quick data collection and retrieval• Rapid scalability• Ease of maintenance• Provide standard data access API RACKSPACE® HOSTING | WWW.RACKSPACE.COM 3
    • Analytical Compute Grid (ACG) Features• Ability to provide variety of storage types: • Columnar • Relational • HDFS• Enable users to select optimal storage type for information collected• Leverage Rackspace® Private Cloud powered by OpenStack® and open source technology RACKSPACE® HOSTING | WWW.RACKSPACE.COM 4
    • Analytical Compute Grid (ACG) Quality Attributes RACKSPACE® HOSTING | WWW.RACKSPACE.COM 5
    • ACG on Rackspace® PrivateCloud powered by OpenStack® High Level Architecture RACKSPACE® HOSTING | WWW.RACKSPACE.COM 6
    • ACG on Rackspace® Private Cloud powered by OpenStack® 8 Hypervisor Servers each: Dual Socket Six Core 2.4GHz Processors 96GB RAM Terabytes of Storage *The environment will grow significantly next year RACKSPACE® HOSTING | WWW.RACKSPACE.COM 7
    • ACG on Rackspace® Private Cloud powered by OpenStack®Image RACKSPACE® HOSTING | WWW.RACKSPACE.COM 8
    • ACG on Rackspace® Private Cloud powered by OpenStack®Database Engine Selection Columnar Cassandra Relational PostgreSQL HDFS Hadoop RACKSPACE® HOSTING | WWW.RACKSPACE.COM 9
    • ACG on Rackspace® Private Cloud powered by OpenStack®Node RACKSPACE® HOSTING | WWW.RACKSPACE.COM 10
    • ACG on Rackspace® Private Cloud powered by OpenStack®Node RACKSPACE® HOSTING | WWW.RACKSPACE.COM 11
    • ACG on Rackspace® Private Cloud powered by OpenStack®Node RACKSPACE® HOSTING | WWW.RACKSPACE.COM 12
    • ACG on Rackspace® Private Cloud powered by OpenStack®Node RACKSPACE® HOSTING | WWW.RACKSPACE.COM 13
    • ACG on Rackspace® Private Cloud powered by OpenStack®Controller RACKSPACE® HOSTING | WWW.RACKSPACE.COM 14
    • ACG on Rackspace® Private Cloud powered by OpenStack®Controller RACKSPACE® HOSTING | WWW.RACKSPACE.COM 15
    • ACG on Rackspace® Private Cloud powered by OpenStack®Controller RACKSPACE® HOSTING | WWW.RACKSPACE.COM 16
    • ACG on Rackspace® Private Cloud powered by OpenStack®API RACKSPACE® HOSTING | WWW.RACKSPACE.COM 17
    • ACG on Rackspace® PrivateCloud powered by OpenStack® Indexing Structure RACKSPACE® HOSTING | WWW.RACKSPACE.COM 18
    • ACG on Rackspace® Private Cloud powered by OpenStack®Indexing Structure• ACG Indexing Structure: • Resides on a set of Rackspace® Private Cloud powered by OpenStack® instances RACKSPACE® HOSTING | WWW.RACKSPACE.COM 19
    • ACG on Rackspace® Private Cloud powered by OpenStack®Indexing Structure• ACG Indexing Structure: • Resides on a set of Rackspace® Private Cloud powered by OpenStack® instances • It is a set of pointers ultimately addressing database entities RACKSPACE® HOSTING | WWW.RACKSPACE.COM 20
    • ACG on Rackspace® Private Cloud powered by OpenStack®Indexing Structure• ACG Indexing Structure: • Resides on a set of Rackspace® Private Cloud powered by OpenStack® instances • It is a set of pointers ultimately addressing database entities • ACG Controller manages Indexing Structure RACKSPACE® HOSTING | WWW.RACKSPACE.COM 21
    • ACG on Rackspace® Private Cloud powered by OpenStack®Indexing Structure• ACG Indexing Structure: • Resides on a set of Rackspace® Private Cloud powered by OpenStack® instances • It is a set of pointers ultimately addressing database entities • ACG Controller manages Indexing Structure • Dynamically expands vertically and horizontally to address a growing data set RACKSPACE® HOSTING | WWW.RACKSPACE.COM 22
    • ACG on Rackspace® Private Cloud powered by OpenStack®Indexing Structure• ACG Indexing Structure Enables: • Distribution of data bases across many instances RACKSPACE® HOSTING | WWW.RACKSPACE.COM 23
    • ACG on Rackspace® Private Cloud powered by OpenStack®Indexing Structure• ACG Indexing Structure Enables: • Distribution of data bases across many instances • Splitting large data sets across many instances RACKSPACE® HOSTING | WWW.RACKSPACE.COM 24
    • ACG on Rackspace® Private Cloud powered by OpenStack®Indexing Structure• ACG Indexing Structure Enables: • Distribution of data bases across many instances • Splitting large data sets across many instances • Parallelization of large data set queries RACKSPACE® HOSTING | WWW.RACKSPACE.COM 25
    • ACG on Rackspace® Private Cloud powered by OpenStack®Indexing Structure• ACG Indexing Structure Enables: • Distribution of data bases across many instances • Splitting large data sets across many instances • Parallelization of large data set queries • Deploying data stores with optimal configuration, minimizing maintenance RACKSPACE® HOSTING | WWW.RACKSPACE.COM 26
    • ACG on Rackspace® Private Cloud powered by OpenStack®Indexing Structure• ACG Indexing Structure Enables: • Distribution of data bases across many instances • Splitting large data sets across many instances • Parallelization of large data set queries • Deploying data stores with optimal configuration, minimizing maintenance • Accessing data residing in variety of storage types via uniform interface RACKSPACE® HOSTING | WWW.RACKSPACE.COM 27
    • ACG on Rackspace® Private Cloud powered by OpenStack®Sorter & Aggregator• ACG Sorter & Aggregator Enables: • Joining the results from multiple ACG nodes RACKSPACE® HOSTING | WWW.RACKSPACE.COM 28
    • ACG on Rackspace® Private Cloud powered by OpenStack®Sorter & Aggregator• ACG Sorter & Aggregator Enables: • Joining the results from multiple ACG nodes • Result sorting and aggregation RACKSPACE® HOSTING | WWW.RACKSPACE.COM 29
    • ACG on Rackspace® Private Cloud powered by OpenStack®Sorter & Aggregator• ACG Sorter & Aggregator Enables: • Joining the results from multiple ACG nodes • Result sorting and aggregation • Together with temporary segment it will support joining heterogeneous data sets RACKSPACE® HOSTING | WWW.RACKSPACE.COM 30
    • ACG on Rackspace® PrivateCloud powered by OpenStack® Quality Attributes RACKSPACE® HOSTING | WWW.RACKSPACE.COM 31
    • ACG on Rackspace® Private Cloud powered by OpenStack®Quality Attributes - PerformanceRackspace® Private Cloud ACGpowered by OpenStack® RACKSPACE® HOSTING | WWW.RACKSPACE.COM 32
    • ACG on Rackspace® Private Cloud powered by OpenStack®Quality Attributes - PerformanceRackspace® Private Cloud ACGpowered by OpenStack®Creates ACG node in 30 seconds RACKSPACE® HOSTING | WWW.RACKSPACE.COM 33
    • ACG on Rackspace® Private Cloud powered by OpenStack®Quality Attributes - PerformanceRackspace® Private Cloud ACGpowered by OpenStack®Creates ACG node in 30 secondsCreates ACG nodes concurrently RACKSPACE® HOSTING | WWW.RACKSPACE.COM 34
    • ACG on Rackspace® Private Cloud powered by OpenStack®Quality Attributes - PerformanceRackspace® Private Cloud ACGpowered by OpenStack® Controlled data set size resultingCreates ACG node in 30 seconds in:Creates ACG nodes concurrently RACKSPACE® HOSTING | WWW.RACKSPACE.COM 35
    • ACG on Rackspace® Private Cloud powered by OpenStack®Quality Attributes - PerformanceRackspace® Private Cloud ACGpowered by OpenStack® Controlled data set size resultingCreates ACG node in 30 seconds in: Quick data distributionCreates ACG nodes concurrently RACKSPACE® HOSTING | WWW.RACKSPACE.COM 36
    • ACG on Rackspace® Private Cloud powered by OpenStack®Quality Attributes - PerformanceRackspace® Private Cloud ACGpowered by OpenStack® Controlled data set size resultingCreates ACG node in 30 seconds in: Quick data distributionCreates ACG nodes concurrently Query parallelization RACKSPACE® HOSTING | WWW.RACKSPACE.COM 37
    • ACG on Rackspace® Private Cloud powered by OpenStack®Quality Attributes - PerformanceRackspace® Private Cloud ACGpowered by OpenStack® Controlled data set size resultingCreates ACG node in 30 seconds in: Quick data distributionCreates ACG nodes concurrently Query parallelization Fast data retrieval RACKSPACE® HOSTING | WWW.RACKSPACE.COM 38
    • ACG on Rackspace® Private Cloud powered by OpenStack®Quality Attributes – ScalabilityRackspace® Private Cloud ACGpowered by OpenStack® RACKSPACE® HOSTING | WWW.RACKSPACE.COM 39
    • ACG on Rackspace® Private Cloud powered by OpenStack®Quality Attributes – ScalabilityRackspace® Private Cloud ACGpowered by OpenStack®Quick and concurrent ACG nodecreation RACKSPACE® HOSTING | WWW.RACKSPACE.COM 40
    • ACG on Rackspace® Private Cloud powered by OpenStack®Quality Attributes – ScalabilityRackspace® Private Cloud ACGpowered by OpenStack®Quick and concurrent ACG nodecreationAbility to re-size existing nodes RACKSPACE® HOSTING | WWW.RACKSPACE.COM 41
    • ACG on Rackspace® Private Cloud powered by OpenStack®Quality Attributes – ScalabilityRackspace® Private Cloud ACGpowered by OpenStack®Quick and concurrent ACG nodecreationAbility to re-size existing nodesAbility to remove nodes RACKSPACE® HOSTING | WWW.RACKSPACE.COM 42
    • ACG on Rackspace® Private Cloud powered by OpenStack®Quality Attributes – ScalabilityRackspace® Private Cloud ACGpowered by OpenStack® Indexing structure and controlledQuick and concurrent ACG node data set size allow ACG tocreation stabilize quickly as it expands orAbility to re-size existing nodes contractsAbility to remove nodes RACKSPACE® HOSTING | WWW.RACKSPACE.COM 43
    • ACG on Rackspace® Private Cloud powered by OpenStack®Quality Attributes – AvailabilityRackspace® Private Cloud ACGpowered by OpenStack® RACKSPACE® HOSTING | WWW.RACKSPACE.COM 44
    • ACG on Rackspace® Private Cloud powered by OpenStack®Quality Attributes – AvailabilityRackspace® Private Cloud ACGpowered by OpenStack®Rapidly replace failed ACG nodes RACKSPACE® HOSTING | WWW.RACKSPACE.COM 45
    • ACG on Rackspace® Private Cloud powered by OpenStack®Quality Attributes – AvailabilityRackspace® Private Cloud ACGpowered by OpenStack® Deploys data store nativeRapidly replace failed ACG nodes availability mechanisms (replication, data distribution…) RACKSPACE® HOSTING | WWW.RACKSPACE.COM 46
    • ACG on Rackspace® Private Cloud powered by OpenStack®Quality Attributes – MaintainabilityRackspace® Private Cloud ACGpowered by OpenStack® RACKSPACE® HOSTING | WWW.RACKSPACE.COM 47
    • ACG on Rackspace® Private Cloud powered by OpenStack®Quality Attributes – MaintainabilityRackspace® Private Cloud ACGpowered by OpenStack®Adding ACG nodes expands: RACKSPACE® HOSTING | WWW.RACKSPACE.COM 48
    • ACG on Rackspace® Private Cloud powered by OpenStack®Quality Attributes – MaintainabilityRackspace® Private Cloud ACGpowered by OpenStack®Adding ACG nodes expands: Storage capacity RACKSPACE® HOSTING | WWW.RACKSPACE.COM 49
    • ACG on Rackspace® Private Cloud powered by OpenStack®Quality Attributes – MaintainabilityRackspace® Private Cloud ACGpowered by OpenStack®Adding ACG nodes expands: Storage capacity CPU power RACKSPACE® HOSTING | WWW.RACKSPACE.COM 50
    • ACG on Rackspace® Private Cloud powered by OpenStack®Quality Attributes – MaintainabilityRackspace® Private Cloud ACGpowered by OpenStack®Adding ACG nodes expands: Storage capacity CPU power RAM RACKSPACE® HOSTING | WWW.RACKSPACE.COM 51
    • ACG on Rackspace® Private Cloud powered by OpenStack®Quality Attributes – MaintainabilityRackspace® Private Cloud ACGpowered by OpenStack®Adding ACG nodes expands: Storage capacity CPU power RAMNo DBA or system administratorsactivity required RACKSPACE® HOSTING | WWW.RACKSPACE.COM 52
    • ACG on Rackspace® Private Cloud powered by OpenStack®Quality Attributes – MaintainabilityRackspace® Private Cloud ACGpowered by OpenStack® Controlled data set size enables:Adding ACG nodes expands: Storage capacity CPU power RAMNo DBA or system administratorsactivity required RACKSPACE® HOSTING | WWW.RACKSPACE.COM 53
    • ACG on Rackspace® Private Cloud powered by OpenStack®Quality Attributes – MaintainabilityRackspace® Private Cloud ACGpowered by OpenStack® Controlled data set size enables:Adding ACG nodes expands: Optimal and stable data store Storage capacity configuration CPU power RAMNo DBA or system administratorsactivity required RACKSPACE® HOSTING | WWW.RACKSPACE.COM 54
    • ACG on Rackspace® Private Cloud powered by OpenStack®Quality Attributes – MaintainabilityRackspace® Private Cloud ACGpowered by OpenStack® Controlled data set size enables:Adding ACG nodes expands: Optimal and stable data store Storage capacity configuration CPU power Reducing demand for managing RAM data store objectsNo DBA or system administratorsactivity required RACKSPACE® HOSTING | WWW.RACKSPACE.COM 55
    • ACG on Rackspace® Private Cloud powered by OpenStack®Quality Attributes – MaintainabilityRackspace® Private Cloud ACGpowered by OpenStack® Controlled data set size enables:Adding ACG nodes expands: Optimal and stable data store Storage capacity configuration CPU power Reducing demand for managing RAM data store objectsNo DBA or system administrators Stable query execution plansactivity required RACKSPACE® HOSTING | WWW.RACKSPACE.COM 56
    • ACG on Rackspace® Private Cloud powered by OpenStack®Quality Attributes – Flexibility ACG RACKSPACE® HOSTING | WWW.RACKSPACE.COM 57
    • ACG on Rackspace® Private Cloud powered by OpenStack®Quality Attributes – Flexibility ACGVariety of storage types: RACKSPACE® HOSTING | WWW.RACKSPACE.COM 58
    • ACG on Rackspace® Private Cloud powered by OpenStack®Quality Attributes – Flexibility ACGVariety of storage types:Columnar – Cassandra : time series data RACKSPACE® HOSTING | WWW.RACKSPACE.COM 59
    • ACG on Rackspace® Private Cloud powered by OpenStack®Quality Attributes – Flexibility ACGVariety of storage types:Columnar – Cassandra : time series dataRelational – PostgreSQL : relational data RACKSPACE® HOSTING | WWW.RACKSPACE.COM 60
    • ACG on Rackspace® Private Cloud powered by OpenStack®Quality Attributes – Flexibility ACGVariety of storage types:Columnar – Cassandra : time series dataRelational – PostgreSQL : relational dataHDFS – Hadoop : un-structured data RACKSPACE® HOSTING | WWW.RACKSPACE.COM 61
    • ACG on Rackspace® Private Cloud powered by OpenStack®Quality Attributes – Flexibility ACGVariety of storage types: Ability to select optimal storageColumnar – Cassandra : time series data type for individual use caseRelational – PostgreSQL : relational dataHDFS – Hadoop : un-structured data RACKSPACE® HOSTING | WWW.RACKSPACE.COM 62
    • ACG on Rackspace® Private Cloud powered by OpenStack®Quality Attributes – Usability ACG RACKSPACE® HOSTING | WWW.RACKSPACE.COM 63
    • ACG on Rackspace® Private Cloud powered by OpenStack®Quality Attributes – Usability ACGStandard interfaces: RACKSPACE® HOSTING | WWW.RACKSPACE.COM 64
    • ACG on Rackspace® Private Cloud powered by OpenStack®Quality Attributes – Usability ACGStandard interfaces: SQL language RACKSPACE® HOSTING | WWW.RACKSPACE.COM 65
    • ACG on Rackspace® Private Cloud powered by OpenStack®Quality Attributes – Usability ACGStandard interfaces: SQL language JDBC API RACKSPACE® HOSTING | WWW.RACKSPACE.COM 66
    • ACG on Rackspace® Private Cloud powered by OpenStack®Quality Attributes – Usability ACGStandard interfaces: SQL language JDBC API Data store native calls RACKSPACE® HOSTING | WWW.RACKSPACE.COM 67
    • ACG on Rackspace® Private Cloud powered by OpenStack®Quality Attributes – Usability ACGStandard interfaces: Native bulk loader utility SQL language JDBC API Data store native calls RACKSPACE® HOSTING | WWW.RACKSPACE.COM 68
    • ACG on Rackspace® Private Cloud powered by OpenStack®Quality Attributes – Usability ACGStandard interfaces: Native bulk loader utility SQL language JDBC API ACG will support joining Data store native calls heterogeneous data sets RACKSPACE® HOSTING | WWW.RACKSPACE.COM 69
    • ACG on Rackspace® PrivateCloud powered by OpenStack® Rackspace Use Case RACKSPACE® HOSTING | WWW.RACKSPACE.COM 70
    • ACG on Rackspace® Private Cloud powered by OpenStack®Rackspace Use Case• Subject: • Complex availability calculation sourcing 3 months of monitoring data and creating 1 billion records in initial calculation RACKSPACE® HOSTING | WWW.RACKSPACE.COM 71
    • ACG on Rackspace® Private Cloud powered by OpenStack®Rackspace Use Case• Environment 1 • Data Warehouse Microsoft SQL server database • SSIS data loading • SQL server with 24 CPUs and 250GB RAM was dedicated to the initial calculation • SQL server stored procedure performed the calculation • Source and result are stored in traditional data warehouse structure RACKSPACE® HOSTING | WWW.RACKSPACE.COM 72
    • ACG on Rackspace® Private Cloud powered by OpenStack®Rackspace Use Case• Environment 2 • In 30 seconds, ACG Node Manager instantiated new columnar data store consisting of 4 Cassandra nodes, and registered it in ACG Indexing Structure • Each ACG node has 2CPUs and 8GB RAM • Informatica data loading • Calculation developed in Java • Source and result are stored in columnar structure suitable for time series data RACKSPACE® HOSTING | WWW.RACKSPACE.COM 73
    • ACG on Rackspace® Private Cloud powered by OpenStack®Rackspace Use Case - Result• Calculation Duration •Microsoft SQL Server lasted 5 days •ACG calculation completed in 3.5 hours• Storage Size • Microsoft SQL server 500GB •ACG 20 GB• Complexity of the calculation •Columnar data store is optimal for time series data. Sourcing from columnar data store resulted in relatively simple Java calculation process comparing to SQL server stored procedure RACKSPACE® HOSTING | WWW.RACKSPACE.COM 74
    • ACG on Rackspace® Private Cloud powered by OpenStack®Rackspace Use Case - Conclusion • Selecting optimal data store for use case resulted in: • Substantial performance improvement RACKSPACE® HOSTING | WWW.RACKSPACE.COM 75
    • ACG on Rackspace® Private Cloud powered by OpenStack®Rackspace Use Case - Conclusion • Selecting optimal data store for use case resulted in: • Substantial performance improvement • Reduced storage demand RACKSPACE® HOSTING | WWW.RACKSPACE.COM 76
    • ACG on Rackspace® Private Cloud powered by OpenStack®Rackspace Use Case - Conclusion • Selecting optimal data store for use case resulted in: • Substantial performance improvement • Reduced storage demand •Simplified processes RACKSPACE® HOSTING | WWW.RACKSPACE.COM 77
    • ACG on Rackspace® Private Cloud powered by OpenStack®Rackspace Use Case - Conclusion • Selecting optimal data store for use case resulted in: • Substantial performance improvement • Reduced storage demand •Simplified processes •Ability to process terabytes of data per day close to real-time and on-demand RACKSPACE® HOSTING | WWW.RACKSPACE.COM 78
    • ACG on Rackspace® Private Cloud powered by OpenStack®Rackspace Use Case - Conclusion • Selecting optimal data store for use case resulted in: • Substantial performance improvement • Reduced storage demand •Simplified processes •Ability to process terabytes of data per day close to real-time and on-demand •Improved trending and reporting: • enhances support capabilities • improved Rackspace customer experience RACKSPACE® HOSTING | WWW.RACKSPACE.COM 79
    • ACG on Rackspace® Private Cloud powered by OpenStack®Rackspace Use Case - Conclusion • Selecting optimal data store for use case resulted in: • Substantial performance improvement • Reduced storage demand • Simplified processes • Ability to process terabytes of data per day close to real-time and on-demand • Improved trending and reporting: • enhances support capabilities • improved Rackspace customer experience • Significant cost reduction RACKSPACE® HOSTING | WWW.RACKSPACE.COM 80
    • RACKSPACE® HOSTING | 5000 WALZEM ROAD | SAN ANTONIO, TX 78218 US SALES: 1-800-961-2888 | US SUPPORT: 1-800-961-4454 | WWW.RACKSPACE.COMRACKSPACE® HOSTING | © RACKSPACE US, INC. | RACKSPACE® AND FANATICAL SUPPORT® ARE SERVICE MARKS OF RACKSPACE US, INC. REGISTERED IN TH E UNITED STATES AND OTHER COUNTRIES. | WWW.RACKSPACE.COM 81