Rich Miller & Surendra Reddy Lighthouse is a concept for the creation of an intercloud registry service, based on (1) access points established and maintained by cloud instances to disseminate operational metadata; and, (2) the use of publish/subscribe (pub/sub) asynchronous messaging as the dominant means of disseminating operational metadata among the constituents of the intercloud.

1. Lighthouse:
Intercloud Metadata Service
Rich Miller
Surendra Reddy
Infrastructure 2.0 Working Group
January 20, 2010

2. Agenda
• Intercloud & Lighthouse Objectives
• Use cases (as drivers & definition)
• Lighthouse Requirements & Concepts
• Available technologies & standards
• Architectural Guiding Principles
• Call(s) to action

3. Intercloud & Objectives

4. Intercloud
Requires the dissemination &
exchange of operational metadata
- among clouds,
- between cloud services and
consumers of their services.

5. Lighthouse

6. Lighthouse
Where to start?
• Agreement on identification, location
and ID-Loc resolution
• A registry for the discovery and
description of intercloud constituents
• A mechanism for the delivery of cloud
service descriptive & operational data
• A governance structure for
admission & ejection, assurance,
permissions & entitlements

7. Lighthouse
The concept:
• Each member takes responsibility for
its own metadata access services
• Membership in a communal registry of
metadata access services, with
identification – location resolution
• Agreement on mechanisms for
- pub/sub/search/query
- asynchronous message delivery

8. Lighthouse Scope
Scope is limited to providing the
Service Access Point and related
metadata to service Consumers

9. Use Cases

10. Intercloud: Use Case #1
• Customer A, EDA company, seeks a list of
IaaS services which claim to provide:
• cloud data management
• Linux OS image management
• Queries the Intercloud registry,
returns IDs of services that meet criteria
• Searches IaaS service metadata to make selection
• Access the Service Access Point (SAP) of a
vendor to validate claims
• Subscribes to Service Access Point for receipt of
service announcements, rate changes, etc

11. Intercloud: Use Case #2
• Customer B, an insurance company, seeks a
single IaaS provider to continuously satisfy
service requirements (constraints)
• E.g. latencies, geography, SLAs etc.
• Queries the Intercloud registry,
returns IDs of services that meet criteria
• Searches IaaS metadata to make selection
• Access the SAPs of vendor to create
Cloud Service Account Instance
• Subscribes to SAP for receipt of relevant
requirement-specific metadata
• Takes specific actions based on timely notifications
(near realtime alerts) via Service Provider APIs and
management functions

12. Intercloud: Use Case #3
• Customer C, a globally distributed online
service looking for an IaaS Providers in Europe
and in USA with specific SLAs.
• Using the Intercloud registry, locates services
meeting needs in two locations.
• Identifies alternative providers for the business
continuity (DR, backup, …) functions.
• Customer C’s application management system
subscribes to failure events & performance sensors
from the IaaS providers.
• Based monitored event/sensor feeds, C’s service
monitoring application dynamically scales up/down
the resources (computing, networking, and storage)
to their applications

13. Intercloud: Use Case #4
• Customer D, a financial services company,
runs applications that are either (or both)
• latency sensitive
• throughput sensitive
• After selecting IaaS provider:
• Sets up the virtual network between on-premise
data center and the IaaS provider cloud.
• Customer D runs their own application mobility
controller within their data center.
• Application Mobility Controller subscribes to
IaaS and data center metadata related to:
• traffic flows, performance metrics
• log feeds from the IaaS cloud service.

14. Intercloud: Use Case #5
• PaaS E, a security broker service, provides an
anti-phishing service for e-mail:
“whitelist”, analytics and forensics
• Operates on behalf of domain holders
• A list management and forensics for multiple
receiver services (e.g. web mail services)
• After establishing service w/ receiver:
• Each receiver establishes a metadata access
point (MAP) regarding failed email
• PaaS E publishes notifications of phishing
attempts to subs, on behalf of domain holder
• All new events and changes in state/status
distributed as pub/sub metadata

15. Lighthouse:
Requirements & Concepts

16. Lighthouse Requirements
• Defines a dynamically extensible set of
identifiers and metadata
• Automatically aggregates and associates
real-time info from many different sources
• Provides real-time pub/sub/search
mechanism for data regarding cloud instances,
their state and their activities
• Scales for cloud to cloud coordination

17. Lighthouse Concept
Autonomous Metadata Access Point
• All interested and authenticated cloud
services, acting in ‘good faith’, provide
their own Metadata Access Point.
• A Metadata Access Point publishes to
the intercloud community any
information about itself.

18. Lighthouse Concept
A Registry of Registries
• Identity and location of individually and
autonomously managed
Metadata Access Services
• Authoritatively establishes the status of
any individual cloud service and its
standing within the Intercloud
community

19. Lighthouse Concept
Process / Event Coordination
• All 'interested' consumers of a cloud’s
MAP Service may subscribe to
metadata updates that result in a
'property' change
• Many systems can coordinate through
a Metadata Access Protocol with no in-
depth knowledge of each other's APIs

20. Lighthouse Concept
Share operational metadata
• Near Real-time
• Cloud Information Service
+
Cloud Operations Coordination

21. Intercloud Registry: Features
• Discovery of a registry’s specific
interfaces / capabilities
• Auditable logging mechanism
• For element / value changes
• For publishing event

22. Intercloud Registry: Features
Forms of Search & Query
• search and report of items based on
(…)
• comparison of object to ‘checklist’ of
elements and parameters
• ‘standing’ search/query established as
subscription
• query and retrieval of items based on
published / recognized (?) data scheme

23. Intercloud Registry: Operational
• Distributed MAP Servers:
Each Cloud Service is responsible for
establishing and administering
• its own Registry Server, or
• publication of metadata by a trusted party
• Authoritative compilation of Registries
(and, therefore, of Cloud Services)
• Unambiguous identification
• Authentication method associated with ID

24. Available Standards

25. Current Standards/Protocols
Federated UDDI Registry
• Pros:
• Federated UDDI consisting of multiple repositories
that are synchronized periodically.
• Federated UDDI is an efficient solution for service
discovery in distributed service networks.
• Cons:
• too expensive to replicate frequently updated
information
• it is hard to directly utilize this approach to support
discovery of dynamic information
• Governance nightmare…

26. Current Standards/Protocols
Service Location Protocol (SLP)
• Pros:
• agent based service discovery framework
• designed for service discovery in for local area
network
• extensions to SLP proposed aiming to the WAN
environment
• Cons:
• Not suitable for wide area network environment
• unsuitable for Cloud environment due to the scale
and distribution complexities involved.

27. Current Standards/Protocols
IF-MAP
• Pros:
• Client-Server based, real-time pub/sub/search
• Designed to disseminate network security info on
objects & events (dynamic state and activity data)
• Easily extensible to components other than network
and security components
• XML-based SOAP protocol
• Supports standardized, dynamic data interchange
• Provides an uniform mechanism to securely
discover, consume, and manage a single
management domain’s metadata.

28. Current Standards/Protocols
IF-MAP (continued)
• Cons:
• SOAP based only, heavy messaging structure
• Scale for Cloud
• Need lot of extensions to existing metadata model
• IF-MAP access point becomes a central authority
• TBD
• Federation to support intercloud scale?
• Wider range of protocols / RESTful interface?
• A MAP-to-MAP (P2P) approach to bi-directional
pub/sub?
• Asynch messaging queues?
• “Economical” message encoding system ?
hierarchical, binary, self-describing

29. Current Standards/Protocols
Other Standards/Protocols to Consider
• WebDAV/DASL
• DAV Provides Versioned Metadata
Access of Resources
• DASL: Provides Searching and Location

30. Current Standards/Protocols
And, what about asynchronous
messaging?
• AMQP
• Session Initiation Protocol (SIP)
• XMPP
• HTTP
• JMS
Not to mention message encoding…
• ASN.1
• FUDGE
• …

31. Lighthouse: Architectural
Model

32. Lighthouse: Metadata Model

33. Lighthouse: Conceptual Architecture 1
Cloud Service Provider
CSP
CSP
CSP
MAP
Metadata
Access Point
IC-
MAP "
InterCloud
Registry
IC Registry
Metadata
Access Point

34. Lighthouse: Conceptual Architecture 2
Cloud Service Provider
CSP
CSP
CSP IC
MAP InterCloud Registry
"
IC IC-
ROOT
Metadata
Access Point
IC Registry
Metadata
“Root Server”

35. Lighthouse: Call(s) to Action
Rich Miller
Surendra Reddy
Infrastructure 2.0 Working Group