Rackspace Analytical Compute Grid (ACG)

Big Data on Open Cloud
Analytical Compute Grid (ACG)
Elastic “Big Data” Infrastructure
by Natasha Gajic

March 1, 2013

Rackspace’s EBI Environment

Current Environment “Big Data” Problem
 Windows and Linux  Cost of purchasing
operating systems additional licenses
 Oracle and Microsoft  Time required to set up
databases solutions new hardware
 Microsoft and Oracle  Increased demand for DBA
replication technology resources
 SSIS  System performance

 Informatica  System scalability

 Dedicated servers  Capacity

 Rapid data set growth

RACKSPACE® HOSTING | WWW.RACKSPACE.COM
2

Analytical Compute Grid (ACG) Features

• Host ever growing set of data
• Quick data collection and retrieval
• Rapid scalability
• Ease of maintenance
• Provide standard data access API

3

Analytical Compute Grid (ACG) Features
• Ability to provide variety of storage types:
• Columnar
• Relational
• HDFS
• Enable users to select optimal storage
type for information collected
• Leverage Rackspace® Private Cloud
powered by OpenStack® and open
source technology

4

Analytical Compute Grid (ACG) Quality Attributes

5

ACG on Rackspace® Private
Cloud powered by OpenStack®
High Level Architecture

6

ACG on Rackspace® Private Cloud powered by OpenStack®

7

Image

8

Database Engine Selection

Columnar Cassandra
Relational PostgreSQL
HDFS Hadoop

9

Node

10

Node

11

Node

12

Node

13

Controller

14

Controller

15

Controller

16

API

17

Indexing Structure

18

Indexing Structure

19

Indexing Structure
What is ACG Indexing
Structure?
• System entry point

• Set of pointers ultimately
addressing database
entities

20

Indexing Structure
What is ACG Indexing
Structure?

• System entry point
• Set of pointers ultimately
addressing database
entities

Where is Indexing Structure
Located?
• It is a part of ACG so it
resides on Open Cloud
• ACG Controller manages
Indexing Structure

21

Indexing Structure
What ACG Indexing Structure
Enables?
• Splitting of large data sets
across many instances
• Query parallelization
• Controlled data store size
• Optimal data store
configuration
• Uniform access to data
residing in various storage
types
• System scalability as it
expands horizontally and
vertically to address ever
growing data set

22

Quality Attributes

23

Quality Attributes - Performance

Rackspace® Private Cloud
powered by OpenStack®
Creates ACG node in 30 seconds
Creates ACG nodes concurrently
Re-size ACG nodes adding CPUs

24

Quality Attributes - Performance

Creates ACG node in 30 seconds
Creates ACG nodes concurrently
Re-size ACG nodes adding CPUs

ACG
Indexing structure and controlled
data set size allow for:
Quick data distribution
Query parallelization

25

Quality Attributes – Availability

Rapidly replace failed ACG nodes

26

Quality Attributes – Availability

Rapidly replace failed ACG nodes

ACG
Deploys data store native
availability mechanisms
(replication, data distribution…)

27

Quality Attributes – Maintainability

Adding ACG nodes expands:
Storage capacity
CPU power
Memory
No DBA or system administrators
activity required

28

Quality Attributes – Maintainability

Adding ACG nodes expands:
Storage capacity
CPU power
RAM
No DBA or system administrators
activity required
ACG
Controlled data set size enables:
Optimal and stable data store
configuration
Reducing demand for managing
data store objects
Stable query execution plans

29

Quality Attributes – Flexibility

ACG
Variety of storage types:
Columnar – Cassandra : time series
data
Relational – PostgreSQL : relational data
HDFS – Hadoop : un-structured data

Ability to select optimal storage type
for individual use case

30

Quality Attributes – Usability

ACG
Standard interfaces:
SQL language
JDBC API
ODBC

ACG Management Console

ACG Monitoring Console

Loader utility implementing:
Bulk Loader
Insert Loader

31

Current State

32

Current State

Columnar Relational HDFS
ACG Controller
Implementation Implementation Implementation
• ACG Manager • Data Store • Data Store • Will start soon
• Rule Engine Controller Controller
• Node • JDBC • JDBC driver
Manager extended to extended with
• ACG work with distributed
Management supercolumn query rewrite
Console • Loader • Loader
• ACG integrated integrated
Monitoring with with
Informatica Informatica
• ODBC (In
Progress)

33

Rackspace Use Case

34

Rackspace Use Case
• Subject:
• Complex availability calculation sourcing 3
months of monitoring data and creating 1 billion
records in initial calculation

35

Rackspace Use Case
• Environment 1
• Data Warehouse Microsoft SQL server database
• SSIS data loading
• SQL server with 24 CPUs and 250GB RAM was
dedicated to the initial calculation
• SQL server stored procedure performed the
calculation
• Source and result are stored in traditional data
warehouse structure

36

Rackspace Use Case
• Environment 2
• ACG running two Cassandra clusters 4 nodes
each
• Informatica with Cassandra bulk loader
• Each ACG node has 2CPUs and 8GB RAM
• Java program running on instance with 4CPUs
and 8GB RAM
• Source and result are stored in columnar
structure suitable for time series data

37

Rackspace Use Case - Result
• Calculation Duration
•Microsoft SQL Server lasted 5 days
•ACG calculation completed in 3.5 hours
• Storage Size
• Microsoft SQL server 500GB
•ACG 20 GB
• Complexity of the calculation
•Columnar data store is optimal for time series data.
Sourcing from columnar data store resulted in relatively
simple Java calculation process comparing to SQL
server stored procedure
38

Rackspace Use Case - Conclusion
• Selecting optimal data store for use case resulted in:
• Substantial performance improvement
• Reduced storage demand
•Simplified processes
•Ability to process terabytes of data per day close to
real-time and on-demand
•Improved trending and reporting:
• enhances support capabilities
• improved Rackspace customer experience
• Significant cost reduction
39

ACG UI

41

ACG UI

42

ACG UI

43

Rackspace Analytical Compute Grid (ACG)

More Related Content

What's hot

Viewers also liked

Similar to Rackspace Analytical Compute Grid (ACG)

More from Rackspace

Recently uploaded

Rackspace Analytical Compute Grid (ACG)