We are working on building Hybrid Cloud for research and development purpose. Our project goal is to realize managing not only Public Cloud but also Private Cloud by making operations even easier. We are managing Amazon EC2, and our Private Cloud by making our own Cloud management tool by Drupal, which we call Clanavi beyond Drupal as a Content Management System. --- Drupal as a fundamental of PaaS (Platform as a Service).
We are happy to introduce our Clanavi including its requirements, architecture design and business value. We would like to show how Drupal can define to manage multiple Cloud infrastructures and why Drupal can be used as Web Application Framework.
Key Points Covered:
- Cloud Computing Overview (Definition)
- Private Cloud Requiremetns
- Goal, Design and Architecture
- Operation Problems in-the-Cloud
- Business Value by Clanavi
- Future Direction
- Q & A
2. The Purpose of This
Presentation
• Learn about Cloud Computing
• Introduce Virtual Infra Manager
• Share Business Value
• Feedback from YOU!
Biz ValueDemoVirtual InfraCloud
3. Agenda
• Cloud Computing Overview
• Virtual Infra Manager
• Demo
• Business Value
• Q&A
Biz ValueDemoVirtual InfraCloud
7. Centralized to the
Internet
UsersUsers
Cloud ServiceService
ProvidersProviders
Low Price
PCs
Server
VirtualizationBroadband
Network-centric
Services
Develop
Services
Provide
Services
Biz ValueDemoVirtual InfraCloud
9. Cloud Computing
Architecture
Cloud Application : Software as a Service
Photo
Virtual
Desktop
BackupMailSearch
Cloud Infrastructure : Infrastructure as a Service
Cloud Management Infrastructure : Platform as a Service
PaaS
SaaS
IaaS
Log AnalysisMovie …Maps Chat
Virtual Infra
Management
Security Auto-scaling
Load
Balancing
…
Persistent
Storage
Message
Queue
Distributed
File System
Virtual
Server
…
Biz ValueDemoVirtual InfraCloud
11. Why Cloud for Service
Providers?
• Problems
• Manage Tons of Servers
• Manage Peta-byte Class Data
• Requirements
• Downsize and Reduce the Cost
• Flexible Capacity Planning
Biz ValueDemoVirtual InfraCloud
12. Why Cloud for Service
Providers?
• Problems
• Manage Tons of Servers
• Manage Peta-byte Class Data
• Requirements
• Cost Reduction
• Flexible Resource Management
Biz ValueDemoVirtual InfraCloud
13. Why Cloud for Service
Providers?
• Cost Reduction
• Open Source
• Browser-based Easy Operations
• Flexible Resource Management
• Multiple Cloud Access
• Private
• Public
Biz ValueDemoVirtual InfraCloud
18. Private Cloud
Requirements
• Security
•User Authentication
•Permissions
• Monitoring
•Physical Servers / Cloud
Nodes
•Virtual Machines
•Applications
• Open Source
• Browser-based UI
•Unified Access for Multiple
Cloud
• Management
•Linux & Windows
•Virtual Machine Image
•Server Template
•Network
•Audit Log and Reports
•Backup
Biz ValueDemoVirtual InfraCloud
19. The Goal of
Virtual Infra Manager
Multiple Public Cloud
Users
FW
C6
C8C7
C5
C2
C4C3
C1- Amazon
- Google
- Windows
Azure
- etc.Multiple Private Cloud
C2
C4C3
C1 C6
C8C7
C5
• Unified Management
Console
• Flexible Resource
Management
Biz ValueDemoVirtual InfraCloud
20. The Goal of
Virtual Infra Manager
Multiple Public Cloud
Users
FW
C6
C8C7
C5
HybridHybrid
Cloud PortalCloud Portal C2
C4C3
C1- Amazon
- Google
- Windows
Azure
- etc.Multiple Private Cloud
C2
C4C3
C1 C6
C8C7
C5
• Unified Management
Console
• Flexible Resource
Management
• Monitoring
• Security
• Backup
• Failover, etc.
Biz ValueDemoVirtual InfraCloud
21. Architecture Overview
Home
Control
Life Logs
Data
Analysis
Healthcare
Location
Service
…
Private Cloud
Eucaly
ptus
XCP
Public Cloud
Amazon
User
AuthN
Permissions
User
Management
Access LogPortal UI
Job
Queue
Virtual Infra
Management
Server
Cloning
Load
Balancing
DatastoreFailover
Storage
Snapshot
User Activity
Auditing
Batch Job
Execution
Provisioning
Monitoring
Server
Image Backup
Billing
Persistent
Storage
Message
Queue
Distributed
File System
memcached
Biz ValueDemoVirtual InfraCloud
SaaSPaaSIaaS
23. Why Drupal?
Biz ValueDemoVirtual InfraCloud
Drupal + Custom
Modules
Enomaly
(Until Feb. 2010)
oVirt
(As of Mar. 2010)
OpenXen
Manager
Developer - Enomaly Red Hat Citrix
License GPL2 AGPL3 GPL2 GPL2
Browser-based Yes Yes Yes Yes (Recently)
Programming
Language
PHP Python
Ruby on Rails
Python
Python
Database MySQL MySQL PostgreSQL N/A
Hypervisor
Support
(Any)
KVM / Xen / etc
(thru libvirt)
KVM / Xen /etc
(thru libvirt)
Xen Only
Install & Setup Easy Difficult Difficult Easy
Functionality (Yes) Yes w/ Limitations (Yes) Yes
API Drupal / REST REST
REST
(Not Exposed)
XAPI / XML-RPC
Perspective Users (and Admin) Admin (and Users) Admin and Users Admin
Stability D6 Core Stable
Sometimes
Database Corrupt
Implementation
Immature
Sometimes
Hang-up
24. Why Drupal?
Biz ValueDemoVirtual InfraCloud
Drupal + Custom
Modules
Enomaly
(Until Feb. 2010)
oVirt
(As of Mar. 2010)
OpenXen
Manager
Developer - Enomaly Red Hat Citrix
License GPL2 AGPL3 GPL2 GPL2
Browser-based Yes Yes Yes Yes (Recently)
Programming
Language
PHP Python
Ruby on Rails
Python
Python
Database MySQL MySQL PostgreSQL N/A
Hypervisor
Support
(Any)
KVM / Xen / etc
(thru libvirt)
KVM / Xen /etc
(thru libvirt)
Xen Only
Install & Setup Easy Difficult Difficult Easy
Functionality (Yes) Yes w/ Limitations (Yes) Yes
API Drupal / REST REST
REST
(Not Exposed)
XAPI / XML-RPC
Perspective Users (and Admin) Admin (and Users) Admin and Users Admin
Stability D6 Core Stable
Sometimes
Database Corrupt
Implementation
Immature
Sometimes
Hang-up
25. Why Drupal?
Biz ValueDemoVirtual InfraCloud
Drupal + Custom
Modules
Enomaly
(Until Feb. 2010)
oVirt
(As of Mar. 2010)
OpenXen
Manager
Developer - Enomaly Red Hat Citrix
License GPL2 AGPL3 GPL2 GPL2
Browser-based Yes Yes Yes Yes (Recently)
Programming
Language
PHP Python
Ruby on Rails
Python
Python
Database MySQL MySQL PostgreSQL N/A
Hypervisor
Support
(Any)
KVM / Xen / etc
(thru libvirt)
KVM / Xen /etc
(thru libvirt)
Xen Only
Install & Setup Easy Difficult Difficult Easy
Functionality (Yes) Yes w/ Limitations (Yes) Yes
API Drupal / REST REST
REST
(Not Exposed)
XAPI / XML-RPC
Perspective Users (and Admin) Admin (and Users) Admin and Users Admin
Stability D6 Core Stable
Sometimes
Database Corrupt
Implementation
Immature
Sometimes
Hang-up
26. Why Drupal?
• Web Application Framework
• Highly-Modularized Architecture
• User Authentication & Management
• Permissions
• Key-value Pair Programming Style (Array)
• Queuing System Support (D7)
• NoSQL Support (D7)
Biz ValueDemoVirtual InfraCloud
27. Why Drupal?
• Web Application Framework
• Highly-Modularized Architecture
• User Authentication & Management
• Permissions
• Key-value Pair Programming Style (Array)
• Queuing System Support (D7)
• NoSQL Support (D7)
Biz ValueDemoVirtual InfraCloud
28. Why Drupal?
• Web Application Framework
• Highly-Modularized Architecture
• User Authentication & Management
• Permissions
• Key-value Pair Programming Style (Array)
• Queuing System Support (D7)
• NoSQL Support (D7)
Biz ValueDemoVirtual InfraCloud
29. Architecture Overview
Home
Control
Life Logs
Data
Analysis
Healthcare
Location
Service
…
Private Cloud
Eucaly
ptus
XCP
Public Cloud
Amazon
User
AuthN
Permissions
User
Management
Access LogPortal UI
Job
Queue
Virtual Infra
Management
Server
Cloning
Load
Balancing
DatastoreFailover
Storage
Snapshot
User Activity
Auditing
Batch Job
Execution
Provisioning
Monitoring
Server
Image Backup
Billing
Persistent
Storage
Message
Queue
Distributed
File System
memcached
Biz ValueDemoVirtual InfraCloud
SaaSPaaSIaaS
30. Architecture Overview
Home
Control
Life Logs
Data
Analysis
Healthcare
Location
Service
…
Private Cloud
Eucaly
ptus
XCP
Public Cloud
Amazon
User
AuthN
Permissions
User
Management
Access LogPortal UI
Job
Queue
Virtual Infra
Management
Server
Cloning
Load
Balancing
DatastoreFailover
Storage
Snapshot
User Activity
Auditing
Batch Job
Execution
Provisioning
Monitoring
Server
Image Backup
Billing
Persistent
Storage
Message
Queue
Distributed
File System
memcached
Biz ValueDemoVirtual InfraCloud
SaaSPaaSIaaS Drupal Core
31. Virtual Infra Manager
• Works with Multiple Cloud
• Amazon EC2
• Eucalyptus
• XCP (Xen Cloud Platform)
Biz ValueDemoVirtual InfraCloud
32. Architecture
Virtual Infra Manager
Development
Module
Drupal Core Drupal Modules
IaaS
Eucalyptus
IaaS
XCP
IaaS
Amazon EC2
Drupal
•Drupal Core as Web Application Framework
Hybrid Cloud Portal
Public
Private
Private
UI Module
Biz ValueDemoVirtual InfraCloud
Existing
Component
33. Architecture
Virtual Infra Manager
Development
Module
Drupal Core Drupal Modules
IaaS
Eucalyptus
IaaS
XCP
IaaS
Amazon EC2
AWS
Drupal
•Drupal Core as Web Application Framework
•Drupal Custom Modules for Each Cloud, Templates, etc.
Hybrid Cloud Portal
Public
Private
Private
Eucalyptus
XCP
UI Module
Biz ValueDemoVirtual InfraCloud
Existing
Component
34. Architecture
Virtual Infra Manager
Development
Module
Drupal Core Drupal Modules
IaaS
Eucalyptus
IaaS
XCP
IaaS
Amazon EC2
AWS
Drupal
Existing
Component
•Drupal Core as Web Application Framework
•Drupal Custom Modules for Each Cloud, Templates, etc.
•Failover by Monitoring Virtual Machines
Hybrid Cloud Portal
Public
Private
Private
Monitoring
Backup
Failover
Eucalyptus
XCP
UI Module
Biz ValueDemoVirtual InfraCloud
35. Deployment
Private Cloud
• User AuthN
• Permissions
• User Activity Logs
• VM Image Management
• Provisioning
• Server Template
• Snapshot / Backup
• Monitoring
Virtual Infra
Manager
REST
XML-RPC
Active
Directory
LDAP
HTTPS
Eucalyptus Cloud
Cloud Controller + Cluster Controller
Node Controller VM VM VM VM
Node Controller VM VM VM VM
XCP Cloud
Pool Master VM VM VM
VM VM VM VM
Slave VM VM VM
VM VM VM VM
Slave VM VM VM
VM VM VM VM
Storage
VM
Image
• Provisioning
• Monitoring
• Alert
Physical
Server
Maintenance
System
Biz ValueDemoVirtual InfraCloud
36. Systems Requirements
• Drupal 6.19
• Ubuntu 10.04
• Apache 2.2
• MySQL 5.1
• PHP 5.3
• Collectd 4.8
• RRDTool 1.3
• Nagios 3.2
Biz ValueDemoVirtual InfraCloud
37. Modules
Category Module Name Description
Cloud
Management
Common •System Common Library
Amazon •Manage Amazon EC2
Eucalyptus •Manage Eucalyptus Cloud
XCP •Manage XCP Cloud
Biz ValueDemoVirtual InfraCloud
38. Modules
Category Module Name Description
Cloud
Management
Common •System Common Library
Amazon •Manage Amazon EC2
Eucalyptus •Manage Eucalyptus Cloud
XCP •Manage XCP Cloud
Provisioning
Registration •Manage Virtual Server Images
Server Templates •Manage Server Templates
Cluster •Manage Server Template Collection
Scripting •Manage Scripts for Automation
Input Parameters •Input Parameters for Running Virtual Server
Biz ValueDemoVirtual InfraCloud
39. Modules
Category Module Name Description
Cloud
Management
Common •System Common Library
Amazon •Manage Amazon EC2
Eucalyptus •Manage Eucalyptus Cloud
XCP •Manage XCP Cloud
Provisioning
Registration •Manage Virtual Server Images
Server Templates •Manage Server Templates
Cluster •Manage Server Template Collection
Scripting •Manage Scripts for Automation
Input Parameters •Input Parameters for Running Virtual Server
Reporting
Audit •User Activity Log
Usage Report •Cost Estimation Depending on Hours of Usage
Alerts •Provides Alert Functionality
Biz ValueDemoVirtual InfraCloud
40. Modules
Category Module Name Description
Cloud
Management
Common •System Common Library
Amazon •Manage Amazon EC2
Eucalyptus •Manage Eucalyptus Cloud
XCP •Manage XCP Cloud
Provisioning
Registration •Manage Virtual Server Images
Server Templates •Manage Server Templates
Cluster •Manage Server Template Collection
Scripting •Manage Scripts for Automation
Input Parameters •Input Parameters for Running Virtual Server
Reporting
Audit •User Activity Log
Usage Report •Cost Estimation Depending on Hours of Usage
Alerts •Provides Alert Functionality
Reliability
Failover •Provide Failover Functionality
Backup •Provide Backup Functionality
Biz ValueDemoVirtual InfraCloud
43. Problem in-the-Cloud
• Async Operations for VMs on Virtual Infra
• Boot
• Shutdown
• Suspend / Resume
• VM Backup
Biz ValueDemoVirtual InfraCloud
44. XCP Cloud
Master VM VM VM
VM VM VM VM
Slave-1 VM VM VM
VM VM VM VM
Virtual Infra Manager
Virtual Infra Operation
Boot
XCP Module
Drupal Core
Browser
Slave-2 VM VM VM
VM VM VM VM
Biz ValueDemoVirtual InfraCloud
45. XCP Cloud
Master VM VM VM
VM VM VM VM
Slave-1 VM VM VM
VM VM VM VM
Virtual Infra Manager
Virtual Infra Operation
Boot
Shutdown
XCP Module
Drupal Core
Browser
Slave-2 VM VM VM
VM VM VM VM
Biz ValueDemoVirtual InfraCloud
46. XCP Cloud
Master VM VM VM
VM VM VM VM
Slave-1 VM VM VM
VM VM VM VM
Slave-2 VM VM VM
VM VM VM VM
Virtual Infra Manager
Virtual Infra Operation
Boot
Shutdown
Backup
XCP Module
Drupal Core
Browser
Biz ValueDemoVirtual InfraCloud
47. XCP Cloud
Master VM VM VM
VM VM VM VM
Slave-1 VM VM VM
VM VM VM VM
Virtual Infra Manager
Virtual Infra Operation
XCP Module
Drupal Core Boot
Shutdown
Backup
Browser
Slave-2 VM VM VM
VM VM VM VM
Shutdown
Biz ValueDemoVirtual InfraCloud
48. XCP Cloud
Master VM VM VM
VM VM VM VM
Slave-1 VM VM VM
VM VM VM VM
Virtual Infra Manager
Virtual Infra Operation
VM VM VM
VM VM VM VM
Bottleneck
XCP Module
Drupal Core Boot
Shutdown
Backup
Browser
Slave-2 VM VM VM
VM VM VM VM
Shutdown
Biz ValueDemoVirtual InfraCloud
49. Virtual Infra Manager
Virtual Infra Operation
Using Drupal Queue
XCP Module
Browser
Drupal Core
XCP Cloud
Master VM VM VM
VM VM VM VM
Slave-1 VM VM VM
VM VM VM VM
Slave-2 VM VM VM
VM VM VM VM
Dequeue
Drupal Queue
Cron
Enqueue
Drupal Queue
Boot
Shutdown
Suspend
Boot
Biz ValueDemoVirtual InfraCloud
50. Virtual Infra Manager
Virtual Infra Operation
Using Drupal Queue
XCP Module
Browser
Drupal Core
XCP Cloud
Master VM VM VM
VM VM VM VM
Slave-1 VM VM VM
VM VM VM VM
Slave-2 VM VM VM
VM VM VM VM
Dequeue
Drupal Queue
Cron
Enqueue
Drupal Queue
Boot
Shutdown
Suspend
Boot
Biz ValueDemoVirtual InfraCloud
51. Virtual Infra Manager
VM Image Backup
Using Drupal Queue
XCP Module
Drupal Core
Enqueue
Dequeue
Drupal Queue
Drupal Queue
Cron
W1
Persistent Storage
Image Repository
UploadImage
3 – 20 min+ 3 – 10 min+
1. Shutdown VM
2. Bundle Image
3. Upload Image
Browser
Wn …Worker Process
Biz ValueDemoVirtual InfraCloud
52. Virtual Infra Manager
VM Image Backup
Using Drupal Queue
XCP Module
Drupal Core
Enqueue
Dequeue
Drupal Queue
Drupal Queue
Cron
W1
W2
Persistent Storage
Image Repository
UploadImage
3 – 20 min+ 3 – 10 min+
1. Shutdown VM
2. Bundle Image
3. Upload Image
Browser
Wn …Worker Process
Biz ValueDemoVirtual InfraCloud
53. Virtual Infra Manager
VM Image Backup
Using Drupal Queue
XCP Module
Drupal Core
Enqueue
Dequeue
Drupal Queue
Drupal Queue
Cron
W3
W1
W2
Persistent Storage
Image Repository
UploadImage
3 – 20 min+ 3 – 10 min+
1. Shutdown VM
2. Bundle Image
3. Upload Image
Browser
Wn …Worker Process
Biz ValueDemoVirtual InfraCloud
62. Cost Reduction
Printer Setup
Software Installation
Network Setup
Windows Update
Install Windows
TraditionalTraditional
Biz ValueDemoVirtual InfraCloud
63. Cost Reduction
Network Setup
Just Copy
a Pre-configured VM Image
Virtual InfraVirtual Infra
Biz ValueDemoVirtual InfraCloud
Printer Setup
Software Installation
Network Setup
Windows Update
Install Windows
TraditionalTraditional
64. Business Value (1/2)
• Virtual Infra Manager
• No Promising OSS
• Reference Model and Implementation for
Cloud Management
• Multiple Cloud Support
• Security
• Reliability
• Scalability
Biz ValueDemoVirtual InfraCloud
65. Business Value (2/2)
• Virtual Infra Manager
• Effective Cost Reduction of Cloud
Management
• Unified Access to Multiple Cloud
• Easy Provisioning
• Easy Operations
• Open Source
Biz ValueDemoVirtual InfraCloud
66. Future Direction
Mail Maps DocsSearch
Location
Service
…
Drupal 7 Core
User & AuthN
Management
Permissions Report Cache
Custom Modules
Queue
Virtual Infra
Management
Server
Cloning
NoSQL
Provisioning
Storage
Snapshot
User Activity
Auditing
Automation
Failover
Monitoring
Server
Image Backup
Billing
Portal UI
Batch Job
Execution
Private Cloud
Eucaly
ptus
XCP
MySQL
Mem
cached
Varnish
Public Cloud
Amazon Google MongoDB
Persistent
Storage
Biz ValueDemoVirtual InfraCloud
SaaSPaaSIaaS Custom ExistingD7 Core
67. Future Direction
Mail Maps DocsSearch
Location
Service
…
Drupal 7 Core
User & AuthN
Management
Permissions Report Cache
Custom Modules
Queue
Virtual Infra
Management
Server
Cloning
NoSQL
Provisioning
Storage
Snapshot
User Activity
Auditing
Automation
Failover
Monitoring
Server
Image Backup
Billing
Portal UI
Batch Job
Execution
Private Cloud
Eucaly
ptus
XCP
MySQL
Mem
cached
Varnish
Public Cloud
Amazon Google MongoDB
Persistent
Storage
Biz ValueDemoVirtual InfraCloud
SaaSPaaSIaaS
Virtual Infra Manager
More Scalable and Reliable
Cloud Management Platform
on D7
Custom ExistingD7 Core
70. Thank You!
“Coming together is a beginning, keeping
together is a progress; working together is
a success.”
- Henry Ford
Biz ValueVirtual InfraCloudBackground
Yas Naoi
naoi@docomolabs-usa.com
72. Virtual Infra Manager
Hybrid (Public / Private) Cloud Management
Instance Monitoring
Supports Windows and Linux VMs
Biz ValueDemoVirtual InfraCloud
Realize Hybrid Cloud Management for Both Private and Public Cloud
Optimize IT Resources by Providing On-Demand Virtual Servers
Realize Hybrid Cloud Management for Both Private and Public Cloud
Optimize IT Resources by Providing On-Demand Virtual Servers
Running Instance can be stored as Image into Repository
Virtual Infra Manager returns a response to Browser immediately
Queue System takes care of async processing for batch jobs
Running Instance can be stored as Image into Repository
Virtual Infra Manager returns a response to Browser immediately
Queue System takes care of async processing for batch jobs
Running Instance can be stored as Image into Repository
Virtual Infra Manager returns a response to Browser immediately
Queue System takes care of async processing for batch jobs
Running Instance can be stored as Image into Repository
Virtual Infra Manager returns a response to Browser immediately
Queue System takes care of async processing for batch jobs
Running Instance can be stored as Image into Repository
Virtual Infra Manager returns a response to Browser immediately
Queue System takes care of async processing for batch jobs
Running Instance can be stored as Image into Repository
Virtual Infra Manager returns a response to Browser immediately
Queue System takes care of async processing for batch jobs
Running Instance can be stored as Image into Repository
Virtual Infra Manager returns a response to Browser immediately
Queue System takes care of async processing for batch jobs
Running Instance can be stored as Image into Repository
Virtual Infra Manager returns a response to Browser immediately
Queue System takes care of async processing for batch jobs
Running Instance can be stored as Image into Repository
Virtual Infra Manager returns a response to Browser immediately
Queue System takes care of async processing for batch jobs
Running Instance can be stored as Image into Repository
Virtual Infra Manager returns a response to Browser immediately
Queue System takes care of async processing for batch jobs