ACG_Rackspace.pdf

568 views
501 views

Published on

true

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
568
On SlideShare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
38
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

ACG_Rackspace.pdf

  1. 1. Analytical Compute Grid(ACG)Elastic “Big Data” InfrastructureRackspace® Private Cloud powered by OpenStack® Use Case by Natasha GajicOctober 17, 2012
  2. 2. Rackspace’s EBI EnvironmentCurrent Environment “Big Data” Problem  Windows and Linux  Cost of purchasing operating systems additional licenses  Oracle and Microsoft  Time required to set up databases solutions new hardware  Microsoft and Oracle  Increased demand for DBA replication technology resources  SSIS  System performance  Informatica  System scalability  Dedicated servers  Capacity  Rapid data set growth RACKSPACE® HOSTING | WWW.RACKSPACE.COM 2
  3. 3. Analytical Compute Grid (ACG) Features• Host ever growing set of data• Quick data collection and retrieval• Rapid scalability• Ease of maintenance• Provide standard data access API RACKSPACE® HOSTING | WWW.RACKSPACE.COM 3
  4. 4. Analytical Compute Grid (ACG) Features• Ability to provide variety of storage types: • Columnar • Relational • HDFS• Enable users to select optimal storage type for information collected• Leverage Rackspace® Private Cloud powered by OpenStack® and open source technology RACKSPACE® HOSTING | WWW.RACKSPACE.COM 4
  5. 5. Analytical Compute Grid (ACG) Quality Attributes RACKSPACE® HOSTING | WWW.RACKSPACE.COM 5
  6. 6. ACG on Rackspace® PrivateCloud powered by OpenStack® High Level Architecture RACKSPACE® HOSTING | WWW.RACKSPACE.COM 6
  7. 7. ACG on Rackspace® Private Cloud powered by OpenStack® 8 Hypervisor Servers each: Dual Socket Six Core 2.4GHz Processors 96GB RAM Terabytes of Storage *The environment will grow significantly next year RACKSPACE® HOSTING | WWW.RACKSPACE.COM 7
  8. 8. ACG on Rackspace® Private Cloud powered by OpenStack®Image RACKSPACE® HOSTING | WWW.RACKSPACE.COM 8
  9. 9. ACG on Rackspace® Private Cloud powered by OpenStack®Database Engine Selection Columnar Cassandra Relational PostgreSQL HDFS Hadoop RACKSPACE® HOSTING | WWW.RACKSPACE.COM 9
  10. 10. ACG on Rackspace® Private Cloud powered by OpenStack®Node RACKSPACE® HOSTING | WWW.RACKSPACE.COM 10
  11. 11. ACG on Rackspace® Private Cloud powered by OpenStack®Node RACKSPACE® HOSTING | WWW.RACKSPACE.COM 11
  12. 12. ACG on Rackspace® Private Cloud powered by OpenStack®Node RACKSPACE® HOSTING | WWW.RACKSPACE.COM 12
  13. 13. ACG on Rackspace® Private Cloud powered by OpenStack®Node RACKSPACE® HOSTING | WWW.RACKSPACE.COM 13
  14. 14. ACG on Rackspace® Private Cloud powered by OpenStack®Controller RACKSPACE® HOSTING | WWW.RACKSPACE.COM 14
  15. 15. ACG on Rackspace® Private Cloud powered by OpenStack®Controller RACKSPACE® HOSTING | WWW.RACKSPACE.COM 15
  16. 16. ACG on Rackspace® Private Cloud powered by OpenStack®Controller RACKSPACE® HOSTING | WWW.RACKSPACE.COM 16
  17. 17. ACG on Rackspace® Private Cloud powered by OpenStack®API RACKSPACE® HOSTING | WWW.RACKSPACE.COM 17
  18. 18. ACG on Rackspace® PrivateCloud powered by OpenStack® Indexing Structure RACKSPACE® HOSTING | WWW.RACKSPACE.COM 18
  19. 19. ACG on Rackspace® Private Cloud powered by OpenStack®Indexing Structure• ACG Indexing Structure: • Resides on a set of Rackspace® Private Cloud powered by OpenStack® instances RACKSPACE® HOSTING | WWW.RACKSPACE.COM 19
  20. 20. ACG on Rackspace® Private Cloud powered by OpenStack®Indexing Structure• ACG Indexing Structure: • Resides on a set of Rackspace® Private Cloud powered by OpenStack® instances • It is a set of pointers ultimately addressing database entities RACKSPACE® HOSTING | WWW.RACKSPACE.COM 20
  21. 21. ACG on Rackspace® Private Cloud powered by OpenStack®Indexing Structure• ACG Indexing Structure: • Resides on a set of Rackspace® Private Cloud powered by OpenStack® instances • It is a set of pointers ultimately addressing database entities • ACG Controller manages Indexing Structure RACKSPACE® HOSTING | WWW.RACKSPACE.COM 21
  22. 22. ACG on Rackspace® Private Cloud powered by OpenStack®Indexing Structure• ACG Indexing Structure: • Resides on a set of Rackspace® Private Cloud powered by OpenStack® instances • It is a set of pointers ultimately addressing database entities • ACG Controller manages Indexing Structure • Dynamically expands vertically and horizontally to address a growing data set RACKSPACE® HOSTING | WWW.RACKSPACE.COM 22
  23. 23. ACG on Rackspace® Private Cloud powered by OpenStack®Indexing Structure• ACG Indexing Structure Enables: • Distribution of data bases across many instances RACKSPACE® HOSTING | WWW.RACKSPACE.COM 23
  24. 24. ACG on Rackspace® Private Cloud powered by OpenStack®Indexing Structure• ACG Indexing Structure Enables: • Distribution of data bases across many instances • Splitting large data sets across many instances RACKSPACE® HOSTING | WWW.RACKSPACE.COM 24
  25. 25. ACG on Rackspace® Private Cloud powered by OpenStack®Indexing Structure• ACG Indexing Structure Enables: • Distribution of data bases across many instances • Splitting large data sets across many instances • Parallelization of large data set queries RACKSPACE® HOSTING | WWW.RACKSPACE.COM 25
  26. 26. ACG on Rackspace® Private Cloud powered by OpenStack®Indexing Structure• ACG Indexing Structure Enables: • Distribution of data bases across many instances • Splitting large data sets across many instances • Parallelization of large data set queries • Deploying data stores with optimal configuration, minimizing maintenance RACKSPACE® HOSTING | WWW.RACKSPACE.COM 26
  27. 27. ACG on Rackspace® Private Cloud powered by OpenStack®Indexing Structure• ACG Indexing Structure Enables: • Distribution of data bases across many instances • Splitting large data sets across many instances • Parallelization of large data set queries • Deploying data stores with optimal configuration, minimizing maintenance • Accessing data residing in variety of storage types via uniform interface RACKSPACE® HOSTING | WWW.RACKSPACE.COM 27
  28. 28. ACG on Rackspace® Private Cloud powered by OpenStack®Sorter & Aggregator• ACG Sorter & Aggregator Enables: • Joining the results from multiple ACG nodes RACKSPACE® HOSTING | WWW.RACKSPACE.COM 28
  29. 29. ACG on Rackspace® Private Cloud powered by OpenStack®Sorter & Aggregator• ACG Sorter & Aggregator Enables: • Joining the results from multiple ACG nodes • Result sorting and aggregation RACKSPACE® HOSTING | WWW.RACKSPACE.COM 29
  30. 30. ACG on Rackspace® Private Cloud powered by OpenStack®Sorter & Aggregator• ACG Sorter & Aggregator Enables: • Joining the results from multiple ACG nodes • Result sorting and aggregation • Together with temporary segment it will support joining heterogeneous data sets RACKSPACE® HOSTING | WWW.RACKSPACE.COM 30
  31. 31. ACG on Rackspace® PrivateCloud powered by OpenStack® Quality Attributes RACKSPACE® HOSTING | WWW.RACKSPACE.COM 31
  32. 32. ACG on Rackspace® Private Cloud powered by OpenStack®Quality Attributes - PerformanceRackspace® Private Cloud ACGpowered by OpenStack® RACKSPACE® HOSTING | WWW.RACKSPACE.COM 32
  33. 33. ACG on Rackspace® Private Cloud powered by OpenStack®Quality Attributes - PerformanceRackspace® Private Cloud ACGpowered by OpenStack®Creates ACG node in 30 seconds RACKSPACE® HOSTING | WWW.RACKSPACE.COM 33
  34. 34. ACG on Rackspace® Private Cloud powered by OpenStack®Quality Attributes - PerformanceRackspace® Private Cloud ACGpowered by OpenStack®Creates ACG node in 30 secondsCreates ACG nodes concurrently RACKSPACE® HOSTING | WWW.RACKSPACE.COM 34
  35. 35. ACG on Rackspace® Private Cloud powered by OpenStack®Quality Attributes - PerformanceRackspace® Private Cloud ACGpowered by OpenStack® Controlled data set size resultingCreates ACG node in 30 seconds in:Creates ACG nodes concurrently RACKSPACE® HOSTING | WWW.RACKSPACE.COM 35
  36. 36. ACG on Rackspace® Private Cloud powered by OpenStack®Quality Attributes - PerformanceRackspace® Private Cloud ACGpowered by OpenStack® Controlled data set size resultingCreates ACG node in 30 seconds in: Quick data distributionCreates ACG nodes concurrently RACKSPACE® HOSTING | WWW.RACKSPACE.COM 36
  37. 37. ACG on Rackspace® Private Cloud powered by OpenStack®Quality Attributes - PerformanceRackspace® Private Cloud ACGpowered by OpenStack® Controlled data set size resultingCreates ACG node in 30 seconds in: Quick data distributionCreates ACG nodes concurrently Query parallelization RACKSPACE® HOSTING | WWW.RACKSPACE.COM 37
  38. 38. ACG on Rackspace® Private Cloud powered by OpenStack®Quality Attributes - PerformanceRackspace® Private Cloud ACGpowered by OpenStack® Controlled data set size resultingCreates ACG node in 30 seconds in: Quick data distributionCreates ACG nodes concurrently Query parallelization Fast data retrieval RACKSPACE® HOSTING | WWW.RACKSPACE.COM 38
  39. 39. ACG on Rackspace® Private Cloud powered by OpenStack®Quality Attributes – ScalabilityRackspace® Private Cloud ACGpowered by OpenStack® RACKSPACE® HOSTING | WWW.RACKSPACE.COM 39
  40. 40. ACG on Rackspace® Private Cloud powered by OpenStack®Quality Attributes – ScalabilityRackspace® Private Cloud ACGpowered by OpenStack®Quick and concurrent ACG nodecreation RACKSPACE® HOSTING | WWW.RACKSPACE.COM 40
  41. 41. ACG on Rackspace® Private Cloud powered by OpenStack®Quality Attributes – ScalabilityRackspace® Private Cloud ACGpowered by OpenStack®Quick and concurrent ACG nodecreationAbility to re-size existing nodes RACKSPACE® HOSTING | WWW.RACKSPACE.COM 41
  42. 42. ACG on Rackspace® Private Cloud powered by OpenStack®Quality Attributes – ScalabilityRackspace® Private Cloud ACGpowered by OpenStack®Quick and concurrent ACG nodecreationAbility to re-size existing nodesAbility to remove nodes RACKSPACE® HOSTING | WWW.RACKSPACE.COM 42
  43. 43. ACG on Rackspace® Private Cloud powered by OpenStack®Quality Attributes – ScalabilityRackspace® Private Cloud ACGpowered by OpenStack® Indexing structure and controlledQuick and concurrent ACG node data set size allow ACG tocreation stabilize quickly as it expands orAbility to re-size existing nodes contractsAbility to remove nodes RACKSPACE® HOSTING | WWW.RACKSPACE.COM 43
  44. 44. ACG on Rackspace® Private Cloud powered by OpenStack®Quality Attributes – AvailabilityRackspace® Private Cloud ACGpowered by OpenStack® RACKSPACE® HOSTING | WWW.RACKSPACE.COM 44
  45. 45. ACG on Rackspace® Private Cloud powered by OpenStack®Quality Attributes – AvailabilityRackspace® Private Cloud ACGpowered by OpenStack®Rapidly replace failed ACG nodes RACKSPACE® HOSTING | WWW.RACKSPACE.COM 45
  46. 46. ACG on Rackspace® Private Cloud powered by OpenStack®Quality Attributes – AvailabilityRackspace® Private Cloud ACGpowered by OpenStack® Deploys data store nativeRapidly replace failed ACG nodes availability mechanisms (replication, data distribution…) RACKSPACE® HOSTING | WWW.RACKSPACE.COM 46
  47. 47. ACG on Rackspace® Private Cloud powered by OpenStack®Quality Attributes – MaintainabilityRackspace® Private Cloud ACGpowered by OpenStack® RACKSPACE® HOSTING | WWW.RACKSPACE.COM 47
  48. 48. ACG on Rackspace® Private Cloud powered by OpenStack®Quality Attributes – MaintainabilityRackspace® Private Cloud ACGpowered by OpenStack®Adding ACG nodes expands: RACKSPACE® HOSTING | WWW.RACKSPACE.COM 48
  49. 49. ACG on Rackspace® Private Cloud powered by OpenStack®Quality Attributes – MaintainabilityRackspace® Private Cloud ACGpowered by OpenStack®Adding ACG nodes expands: Storage capacity RACKSPACE® HOSTING | WWW.RACKSPACE.COM 49
  50. 50. ACG on Rackspace® Private Cloud powered by OpenStack®Quality Attributes – MaintainabilityRackspace® Private Cloud ACGpowered by OpenStack®Adding ACG nodes expands: Storage capacity CPU power RACKSPACE® HOSTING | WWW.RACKSPACE.COM 50
  51. 51. ACG on Rackspace® Private Cloud powered by OpenStack®Quality Attributes – MaintainabilityRackspace® Private Cloud ACGpowered by OpenStack®Adding ACG nodes expands: Storage capacity CPU power RAM RACKSPACE® HOSTING | WWW.RACKSPACE.COM 51
  52. 52. ACG on Rackspace® Private Cloud powered by OpenStack®Quality Attributes – MaintainabilityRackspace® Private Cloud ACGpowered by OpenStack®Adding ACG nodes expands: Storage capacity CPU power RAMNo DBA or system administratorsactivity required RACKSPACE® HOSTING | WWW.RACKSPACE.COM 52
  53. 53. ACG on Rackspace® Private Cloud powered by OpenStack®Quality Attributes – MaintainabilityRackspace® Private Cloud ACGpowered by OpenStack® Controlled data set size enables:Adding ACG nodes expands: Storage capacity CPU power RAMNo DBA or system administratorsactivity required RACKSPACE® HOSTING | WWW.RACKSPACE.COM 53
  54. 54. ACG on Rackspace® Private Cloud powered by OpenStack®Quality Attributes – MaintainabilityRackspace® Private Cloud ACGpowered by OpenStack® Controlled data set size enables:Adding ACG nodes expands: Optimal and stable data store Storage capacity configuration CPU power RAMNo DBA or system administratorsactivity required RACKSPACE® HOSTING | WWW.RACKSPACE.COM 54
  55. 55. ACG on Rackspace® Private Cloud powered by OpenStack®Quality Attributes – MaintainabilityRackspace® Private Cloud ACGpowered by OpenStack® Controlled data set size enables:Adding ACG nodes expands: Optimal and stable data store Storage capacity configuration CPU power Reducing demand for managing RAM data store objectsNo DBA or system administratorsactivity required RACKSPACE® HOSTING | WWW.RACKSPACE.COM 55
  56. 56. ACG on Rackspace® Private Cloud powered by OpenStack®Quality Attributes – MaintainabilityRackspace® Private Cloud ACGpowered by OpenStack® Controlled data set size enables:Adding ACG nodes expands: Optimal and stable data store Storage capacity configuration CPU power Reducing demand for managing RAM data store objectsNo DBA or system administrators Stable query execution plansactivity required RACKSPACE® HOSTING | WWW.RACKSPACE.COM 56
  57. 57. ACG on Rackspace® Private Cloud powered by OpenStack®Quality Attributes – Flexibility ACG RACKSPACE® HOSTING | WWW.RACKSPACE.COM 57
  58. 58. ACG on Rackspace® Private Cloud powered by OpenStack®Quality Attributes – Flexibility ACGVariety of storage types: RACKSPACE® HOSTING | WWW.RACKSPACE.COM 58
  59. 59. ACG on Rackspace® Private Cloud powered by OpenStack®Quality Attributes – Flexibility ACGVariety of storage types:Columnar – Cassandra : time series data RACKSPACE® HOSTING | WWW.RACKSPACE.COM 59
  60. 60. ACG on Rackspace® Private Cloud powered by OpenStack®Quality Attributes – Flexibility ACGVariety of storage types:Columnar – Cassandra : time series dataRelational – PostgreSQL : relational data RACKSPACE® HOSTING | WWW.RACKSPACE.COM 60
  61. 61. ACG on Rackspace® Private Cloud powered by OpenStack®Quality Attributes – Flexibility ACGVariety of storage types:Columnar – Cassandra : time series dataRelational – PostgreSQL : relational dataHDFS – Hadoop : un-structured data RACKSPACE® HOSTING | WWW.RACKSPACE.COM 61
  62. 62. ACG on Rackspace® Private Cloud powered by OpenStack®Quality Attributes – Flexibility ACGVariety of storage types: Ability to select optimal storageColumnar – Cassandra : time series data type for individual use caseRelational – PostgreSQL : relational dataHDFS – Hadoop : un-structured data RACKSPACE® HOSTING | WWW.RACKSPACE.COM 62
  63. 63. ACG on Rackspace® Private Cloud powered by OpenStack®Quality Attributes – Usability ACG RACKSPACE® HOSTING | WWW.RACKSPACE.COM 63
  64. 64. ACG on Rackspace® Private Cloud powered by OpenStack®Quality Attributes – Usability ACGStandard interfaces: RACKSPACE® HOSTING | WWW.RACKSPACE.COM 64
  65. 65. ACG on Rackspace® Private Cloud powered by OpenStack®Quality Attributes – Usability ACGStandard interfaces: SQL language RACKSPACE® HOSTING | WWW.RACKSPACE.COM 65
  66. 66. ACG on Rackspace® Private Cloud powered by OpenStack®Quality Attributes – Usability ACGStandard interfaces: SQL language JDBC API RACKSPACE® HOSTING | WWW.RACKSPACE.COM 66
  67. 67. ACG on Rackspace® Private Cloud powered by OpenStack®Quality Attributes – Usability ACGStandard interfaces: SQL language JDBC API Data store native calls RACKSPACE® HOSTING | WWW.RACKSPACE.COM 67
  68. 68. ACG on Rackspace® Private Cloud powered by OpenStack®Quality Attributes – Usability ACGStandard interfaces: Native bulk loader utility SQL language JDBC API Data store native calls RACKSPACE® HOSTING | WWW.RACKSPACE.COM 68
  69. 69. ACG on Rackspace® Private Cloud powered by OpenStack®Quality Attributes – Usability ACGStandard interfaces: Native bulk loader utility SQL language JDBC API ACG will support joining Data store native calls heterogeneous data sets RACKSPACE® HOSTING | WWW.RACKSPACE.COM 69
  70. 70. ACG on Rackspace® PrivateCloud powered by OpenStack® Rackspace Use Case RACKSPACE® HOSTING | WWW.RACKSPACE.COM 70
  71. 71. ACG on Rackspace® Private Cloud powered by OpenStack®Rackspace Use Case• Subject: • Complex availability calculation sourcing 3 months of monitoring data and creating 1 billion records in initial calculation RACKSPACE® HOSTING | WWW.RACKSPACE.COM 71
  72. 72. ACG on Rackspace® Private Cloud powered by OpenStack®Rackspace Use Case• Environment 1 • Data Warehouse Microsoft SQL server database • SSIS data loading • SQL server with 24 CPUs and 250GB RAM was dedicated to the initial calculation • SQL server stored procedure performed the calculation • Source and result are stored in traditional data warehouse structure RACKSPACE® HOSTING | WWW.RACKSPACE.COM 72
  73. 73. ACG on Rackspace® Private Cloud powered by OpenStack®Rackspace Use Case• Environment 2 • In 30 seconds, ACG Node Manager instantiated new columnar data store consisting of 4 Cassandra nodes, and registered it in ACG Indexing Structure • Each ACG node has 2CPUs and 8GB RAM • Informatica data loading • Calculation developed in Java • Source and result are stored in columnar structure suitable for time series data RACKSPACE® HOSTING | WWW.RACKSPACE.COM 73
  74. 74. ACG on Rackspace® Private Cloud powered by OpenStack®Rackspace Use Case - Result• Calculation Duration •Microsoft SQL Server lasted 5 days •ACG calculation completed in 3.5 hours• Storage Size • Microsoft SQL server 500GB •ACG 20 GB• Complexity of the calculation •Columnar data store is optimal for time series data. Sourcing from columnar data store resulted in relatively simple Java calculation process comparing to SQL server stored procedure RACKSPACE® HOSTING | WWW.RACKSPACE.COM 74
  75. 75. ACG on Rackspace® Private Cloud powered by OpenStack®Rackspace Use Case - Conclusion • Selecting optimal data store for use case resulted in: • Substantial performance improvement RACKSPACE® HOSTING | WWW.RACKSPACE.COM 75
  76. 76. ACG on Rackspace® Private Cloud powered by OpenStack®Rackspace Use Case - Conclusion • Selecting optimal data store for use case resulted in: • Substantial performance improvement • Reduced storage demand RACKSPACE® HOSTING | WWW.RACKSPACE.COM 76
  77. 77. ACG on Rackspace® Private Cloud powered by OpenStack®Rackspace Use Case - Conclusion • Selecting optimal data store for use case resulted in: • Substantial performance improvement • Reduced storage demand •Simplified processes RACKSPACE® HOSTING | WWW.RACKSPACE.COM 77
  78. 78. ACG on Rackspace® Private Cloud powered by OpenStack®Rackspace Use Case - Conclusion • Selecting optimal data store for use case resulted in: • Substantial performance improvement • Reduced storage demand •Simplified processes •Ability to process terabytes of data per day close to real-time and on-demand RACKSPACE® HOSTING | WWW.RACKSPACE.COM 78
  79. 79. ACG on Rackspace® Private Cloud powered by OpenStack®Rackspace Use Case - Conclusion • Selecting optimal data store for use case resulted in: • Substantial performance improvement • Reduced storage demand •Simplified processes •Ability to process terabytes of data per day close to real-time and on-demand •Improved trending and reporting: • enhances support capabilities • improved Rackspace customer experience RACKSPACE® HOSTING | WWW.RACKSPACE.COM 79
  80. 80. ACG on Rackspace® Private Cloud powered by OpenStack®Rackspace Use Case - Conclusion • Selecting optimal data store for use case resulted in: • Substantial performance improvement • Reduced storage demand • Simplified processes • Ability to process terabytes of data per day close to real-time and on-demand • Improved trending and reporting: • enhances support capabilities • improved Rackspace customer experience • Significant cost reduction RACKSPACE® HOSTING | WWW.RACKSPACE.COM 80
  81. 81. RACKSPACE® HOSTING | 5000 WALZEM ROAD | SAN ANTONIO, TX 78218 US SALES: 1-800-961-2888 | US SUPPORT: 1-800-961-4454 | WWW.RACKSPACE.COMRACKSPACE® HOSTING | © RACKSPACE US, INC. | RACKSPACE® AND FANATICAL SUPPORT® ARE SERVICE MARKS OF RACKSPACE US, INC. REGISTERED IN TH E UNITED STATES AND OTHER COUNTRIES. | WWW.RACKSPACE.COM 81

×