1.
NEBULA Cloud Computing Platform 2010
How we built Nebula
Cloud Power

2.
NEBULA Cloud Computing Platform 2010
Architecture

3.
NEBULA Cloud Computing Platform 2010
Architecture Drivers
• Reliability
• Availability
• Cost

4.
NEBULA Cloud Computing Platform 2010
Shared Nothing
• Messaging Queue
• State Discovery
• Standard Protocols
Automated
• IPMI
• PXEBoot
• Puppet

5.
NEBULA Cloud Computing Platform 2010
Components
• Cloud Node
• Network Node
• Compute Node
• Volume Node
• Object Node
• (Monitoring / Metering / Logging / Scanning)

6.
NEBULA Cloud Computing Platform 2010
LDAP Data
Store
Ubuntu OS
Puppet
Nova
Cloud
Node
PXE
Cloud Node
RabbitMQ
Redis KVS

7.
NEBULA Cloud Computing Platform 2010
Ubuntu OS
Puppet
KVM
LibVirt
Nova
Compute
Node
802.1(q)
Brctl
PXE
Project VLAN
Running Instance
Compute Node

8.
NEBULA Cloud Computing Platform 2010
Ubuntu OS
Puppet
LVM
AoE
Nova
Volume
Node
PXE
Exported Volume
Volume Node

9.
NEBULA Cloud Computing Platform 2010
Ubuntu OS
Puppet
Nova
Object
Node
PXE
Object Node
Nginx

10.
NEBULA Cloud Computing Platform 2010
Ubuntu OS
Puppet
Nova
Network
Node
802.1(q)
Brctl
PXE
Project
VLAN
Network Node
IPTables
Public
Internet

11.
NEBULA Cloud Computing Platform 2010
Security Groups
• Combination of VLANs and Subnetting
• Can be extended to use physical
network/node separation as well (future)

12.
NEBULA Cloud Computing Platform 2010
Scheduling
• Mixed model (selection + auction)
• Supports instance-type optimizations
• Supports FIPS 199-based selection (future)

13.
NEBULA Cloud Computing Platform 2010
Roadmap (What’s Coming)

14.
NEBULA Cloud Computing Platform 2010
What’s Coming
• SAML authentication and RBAC (Beta)
• Computable Object Store (OS^2) (Beta)
• Auditable Usage Metering (Sept 2010)
• Platform-as-a-Service Prototype (Sept 2010)
• FIPS 199 CIA: Moderate w/ CloudAudit (2011)
• IPv6 and Jumbo Frame support (Late 2011)

15.
NEBULA Cloud Computing Platform 2010
Lessons Learned

16.
NEBULA Cloud Computing Platform 2010
Automate Everything
• No SysAdmin is perfect
• 99% is not good enough
• NEVER make direct system changes
• When in doubt, PXEBoot

17.
NEBULA Cloud Computing Platform 2010
Test Everything
• KVM + Jumbo Frames
• Grinder
• Unit Tests / Cyclometric Complexity
• TransactionID Insertion (Universal Proxy)

18.
NEBULA Cloud Computing Platform 2010
Monitor Everything
• Ganglia
• Munin
• Syslog-NG + PHPSyslog-NG
• Nagios
• Custom Log Parsing (Instance-centric)

19.
NEBULA Cloud Computing Platform 2010
Discussion Points

20.
NEBULA Cloud Computing Platform 2010
SPoF == Availability
• Network Layer
 BGP / DNS LB
 Linux-HA Network Nodes
• Volumes
 DRBD / SAN
• RAID Class
 RAID 6 / RAID 1+0

21.
NEBULA Cloud Computing Platform 2010
What ARC Did
• Train users to embrace cloud model,
• Take advantage of commodity hardware