The document discusses the objectives and concepts of the Lighthouse Intercloud Metadata Service. Lighthouse aims to enable the dissemination and exchange of operational metadata among clouds and between cloud services and consumers. It proposes that each cloud provider operates an autonomous Metadata Access Point to publish information about itself. A registry of these access points would then provide a mechanism for discovery and coordination across clouds. The document outlines several use cases and requirements for Lighthouse and reviews existing standards that could potentially be leveraged or extended to support its goals.

1. Lighthouse:
Intercloud Metadata Service
Rich Miller
Surendra Reddy
Infrastructure 2.0 Working Group
January 20, 2010

2. Agenda
• Intercloud & Lighthouse Objectives
• Use cases (as drivers & definition)
• Lighthouse Requirements & Concepts
• Available technologies & standards
• Architectural Guiding Principles
• Call(s) to action

3. Intercloud & Objectives

4. Intercloud
Requires the dissemination &
exchange of operational metadata
- among clouds,
- between cloud services and
consumers of their services.

5. Lighthouse

6. Lighthouse
Where to start?
• Agreement on identification, location
and ID-Loc resolution
• A registry for the discovery and
description of intercloud constituents
• A mechanism for the delivery of cloud
service descriptive & operational data
• A governance structure for
admission & ejection, assurance,
permissions & entitlements

7. Lighthouse
The concept:
• Each member takes responsibility for
its own metadata access services
• Membership in a communal registry of
metadata access services, with
identification – location resolution
• Agreement on mechanisms for
- pub/sub/search/query
- asynchronous message delivery

8. Lighthouse Scope
Scope is limited to providing the
Service Access Point and related
metadata to service Consumers

9. Use Cases

10. Intercloud: Use Case #1
• Customer A, EDA company, seeks a list of
IaaS services which claim to provide:
• cloud data management
• Linux OS image management
• Queries the Intercloud registry,
returns IDs of services that meet criteria
• Searches IaaS service metadata to make selection
• Access the Service Access Point (SAP) of a
vendor to validate claims
• Subscribes to Service Access Point for receipt of
service announcements, rate changes, etc

11. Intercloud: Use Case #2
• Customer B, an insurance company, seeks a
single IaaS provider to continuously satisfy
service requirements (constraints)
• E.g. latencies, geography, SLAs etc.
• Queries the Intercloud registry,
returns IDs of services that meet criteria
• Searches IaaS metadata to make selection
• Access the SAPs of vendor to create
Cloud Service Account Instance
• Subscribes to SAP for receipt of relevant
requirement-specific metadata
• Takes specific actions based on timely notifications
(near realtime alerts) via Service Provider APIs and
management functions

12. Intercloud: Use Case #3
• Customer C, a globally distributed online
service looking for an IaaS Providers in Europe
and in USA with specific SLAs.
• Using the Intercloud registry, locates services
meeting needs in two locations.
• Identifies alternative providers for the business
continuity (DR, backup, …) functions.
• Customer C’s application management system
subscribes to failure events & performance sensors
from the IaaS providers.
• Based monitored event/sensor feeds, C’s service
monitoring application dynamically scales up/down
the resources (computing, networking, and storage)
to their applications

13. Intercloud: Use Case #4
• Customer D, a financial services company,
runs applications that are either (or both)
• latency sensitive
• throughput sensitive
• After selecting IaaS provider:
• Sets up the virtual network between on-premise
data center and the IaaS provider cloud.
• Customer D runs their own application mobility
controller within their data center.
• Application Mobility Controller subscribes to
IaaS and data center metadata related to:
• traffic flows, performance metrics
• log feeds from the IaaS cloud service.

14. Intercloud: Use Case #5
• PaaS E, a security broker service, provides an
anti-phishing service for e-mail:
“whitelist”, analytics and forensics
• Operates on behalf of domain holders
• A list management and forensics for multiple
receiver services (e.g. web mail services)
• After establishing service w/ receiver:
• Each receiver establishes a metadata access
point (MAP) regarding failed email
• PaaS E publishes notifications of phishing
attempts to subs, on behalf of domain holder
• All new events and changes in state/status
distributed as pub/sub metadata

15. Lighthouse:
Requirements & Concepts

16. Lighthouse Requirements
• Defines a dynamically extensible set of
identifiers and metadata
• Automatically aggregates and associates
real-time info from many different sources
• Provides real-time pub/sub/search
mechanism for data regarding cloud instances,
their state and their activities
• Scales for cloud to cloud coordination

17. Lighthouse Concept
Autonomous Metadata Access Point
• All interested and authenticated cloud
services, acting in ‘good faith’, provide
their own Metadata Access Point.
• A Metadata Access Point publishes to
the intercloud community any
information about itself.

18. Lighthouse Concept
A Registry of Registries
• Identity and location of individually and
autonomously managed
Metadata Access Services
• Authoritatively establishes the status of
any individual cloud service and its
standing within the Intercloud
community

19. Lighthouse Concept
Process / Event Coordination
• All 'interested' consumers of a cloud’s
MAP Service may subscribe to
metadata updates that result in a
'property' change
• Many systems can coordinate through
a Metadata Access Protocol with no in-
depth knowledge of each other's APIs

20. Lighthouse Concept
Share operational metadata
• Near Real-time
• Cloud Information Service
+
Cloud Operations Coordination

21. Intercloud Registry: Features
• Discovery of a registry’s specific
interfaces / capabilities
• Auditable logging mechanism
• For element / value changes
• For publishing event

22. Intercloud Registry: Features
Forms of Search & Query
• search and report of items based on
(…)
• comparison of object to ‘checklist’ of
elements and parameters
• ‘standing’ search/query established as
subscription
• query and retrieval of items based on
published / recognized (?) data scheme

23. Intercloud Registry: Operational
• Distributed MAP Servers:
Each Cloud Service is responsible for
establishing and administering
• its own Registry Server, or
• publication of metadata by a trusted party
• Authoritative compilation of Registries
(and, therefore, of Cloud Services)
• Unambiguous identification
• Authentication method associated with ID

24. Available Standards

25. Current Standards/Protocols
Federated UDDI Registry
• Pros:
• Federated UDDI consisting of multiple repositories
that are synchronized periodically.
• Federated UDDI is an efficient solution for service
discovery in distributed service networks.
• Cons:
• too expensive to replicate frequently updated
information
• it is hard to directly utilize this approach to support
discovery of dynamic information
• Governance nightmare…

26. Current Standards/Protocols
Service Location Protocol (SLP)
• Pros:
• agent based service discovery framework
• designed for service discovery in for local area
network
• extensions to SLP proposed aiming to the WAN
environment
• Cons:
• Not suitable for wide area network environment
• unsuitable for Cloud environment due to the scale
and distribution complexities involved.

27. Current Standards/Protocols
IF-MAP
• Pros:
• Client-Server based, real-time pub/sub/search
• Designed to disseminate network security info on
objects & events (dynamic state and activity data)
• Easily extensible to components other than network
and security components
• XML-based SOAP protocol
• Supports standardized, dynamic data interchange
• Provides an uniform mechanism to securely
discover, consume, and manage a single
management domain’s metadata.

28. Current Standards/Protocols
IF-MAP (continued)
• Cons:
• SOAP based only, heavy messaging structure
• Scale for Cloud
• Need lot of extensions to existing metadata model
• IF-MAP access point becomes a central authority
• TBD
• Federation to support intercloud scale?
• Wider range of protocols / RESTful interface?
• A MAP-to-MAP (P2P) approach to bi-directional
pub/sub?
• Asynch messaging queues?
• “Economical” message encoding system ?
hierarchical, binary, self-describing

29. Current Standards/Protocols
Other Standards/Protocols to Consider
• WebDAV/DASL
• DAV Provides Versioned Metadata
Access of Resources
• DASL: Provides Searching and Location

30. Current Standards/Protocols
And, what about asynchronous
messaging?
• AMQP
• Session Initiation Protocol (SIP)
• XMPP
• HTTP
• JMS
Not to mention message encoding…
• ASN.1
• FUDGE
• …

31. Lighthouse: Architectural
Model

32. Lighthouse: Metadata Model

33. Lighthouse: Conceptual Architecture 1
Cloud Service Provider
CSP
CSP
CSP
MAP
Metadata
Access Point
IC-
MAP "
InterCloud
Registry
IC Registry
Metadata
Access Point

34. Lighthouse: Conceptual Architecture 2
Cloud Service Provider
CSP
CSP
CSP IC
MAP InterCloud Registry
"
IC IC-
ROOT
Metadata
Access Point
IC Registry
Metadata
“Root Server”

35. Lighthouse: Call(s) to Action
Rich Miller
Surendra Reddy
Infrastructure 2.0 Working Group