• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content







Total Views
Views on SlideShare
Embed Views



0 Embeds 0

No embeds



Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment
  • Publish/subscribe applications are intended for situations where a single message is required by, and should be distributed to, multiple users. Their big advantage over other delivery methods is that they keep the publisher separated from the subscriber. This means that the publisher in a publish/subscribe application doesn’t need to have any knowledge of either the subscriber’s existence or the applications that may use the published information. Likewise, the subscriber or subscriber applications don’t need to know anything about the publisher application. Answer the key questions: What is a publisher? What is a subscriber?
  • Don’t de-register subscriptions from here in a production environment, write a simple utility that will de-register and log the activity.
  • Thank you
  • Takeaway: The IBM SOA Foundation delivers the capabilities you need to adopt SOA through a comprehensive architecture. These capabilities can be implemented on a build-as-you-go basis, and yet, because of the architecture and its service orientation, capabilities and project level solutions can be easily added as new requirements are addressed over time. Background: The SOA Reference Architecture shows the key capabilities that are required for comprehensive, enterprise wide SOA solutions. Development Services are an essential component of any comprehensive integration architecture. The SOA Architecture includes development tools, used to implement custom artifacts that leverage the infrastructure capabilities, and business performance management tools, used to monitor and manage the runtime implementations at both the IT and business process levels. Development tools allow people to efficiently complete specific tasks and create specific output based on their skills, their expertise, and their role within the enterprise. Business Analysts who analyze business process requirements need modeling tools that allow business processes to be charted and simulated. Software Architects need tool perspectives that allow them to model data, functional flows, system interactions, etc. Integration Specialists require capabilities that allow them to configure specific inter-connections in the integration solution. Programmers need tools that allow them to develop new business logic with little concern for the underlying platform. Yet, while it is important for each person to have a specific set of tool functions based on their role in the enterprise, the tooling environment must provide a framework that promotes joint development, asset management and deep collaboration among all these people. A common repository and functions common across all the developer perspectives (e.g. version control functions, project management functions, etc) are provided in the SOA Reference Architecture through a unified development platform. The Business Innovation & Optimization Services incorporate monitoring capabilities that aggregate operational and process metrics in order to efficiently manage systems and processes. Managing these systems requires a set of capabilities that span the needs of IT operations professionals and business analysts who manage the business operations of the enterprise. These capabilities are delivered through a set of comprehensive services that collect and present both IT and process-level data, allowing business dashboards, administrative dashboards, and other IT level displays to be used to manage system resources and business processes. Through these displays and services, it is possible for LOB and IT personnel to collaborate to determine, for example, what business process paths may not be performing at maximum efficiency, the impact of system problems on specific processes, or the relationship of system performance to business process performance. This collaboration allows IT personnel and assets to be tied more directly to the business success of the enterprise than they traditionally have been. One key feature of the SOA Reference Architecture is the linkage between the Development and the Business Innovation & Optimization Services. The ability to deliver runtime data and statistics into the development environment allows analyses to be completed that drive iterative process re-engineering through a continuous business process improvement cycle. At the core of the SOA Reference Architecture is the Enterprise Service Bus . This architectural construct delivers all the inter-connectivity capabilities required to leverage and use services implemented across the entire architecture. Transport services, event services, and mediation services are all provided through the ESB. Transport services provide the fundamental connection layer; event services allow the system to respond to specific stimuli that are part of a business process; and mediation services allow loose-coupling between interacting services in the system. The ESB is a key factor in enabling the service orientation of the SOA Reference Architecture to be leveraged in implementing service oriented solutions and can be implemented today to meet the quality of service requirements of any integration solution. The SOA Reference Architecture also contains a set of services that are oriented toward the integration of people, processes, and information. These services control the flow of interactions and data among people and automated application services in ways appropriate to the realization of a business process: - Interaction Services provide the capabilities required to deliver IT functions and data to end users, meeting the end-user's specific usage preferences. - Process Services provide the control services required to manage the flow and interactions of multiple services in ways that implement business processes. - Information Services provide the capabilities required to federate, replicate, and transform data sources that may be implemented in a variety of ways. Automated application services, implementations of business logic in automated systems, are a critical part of any integration architecture or solution. Many of these services are provided through existing applications; others are provided in newly implemented components; and others are provided through external connections to third party systems. Existing enterprise applications and enterprise data are accessible from the ESB through a set of access services. These Access Services provide the bridging capabilities between legacy applications, pre-packaged applications, enterprise data stores (including relational, hierarchical and nontraditional, unstructured sources such as XML and Text), etc and the ESB. Using a consistent approach, these access services expose the data and functions of the existing enterprise applications, allowing them to be fully re-used and incorporated into functional flows that represent business processes. Existing enterprise applications and data leverage the Business Application and Data Services of their operating environments such as CICS, IMS, DB2, etc. As these applications and data implementations evolve to become more flexible participants in business processes, enhanced capabilities of their underlying operating environments, for example support of emerging standards, can be fully utilized. The SOA Reference Architecture also contains a set of Business Application Services that provide runtime services required for new application components to be included in the integrated system. These application components provide new business logic required to adapt existing business processes to meet changing competitive and customer demands of the enterprise. Design and implementation of new business logic components for integration enables them to be fully re-useable, allowing them to participate in new and updated business processes over time. The Business Application Services include functions important to the traditional programmer for building maintainable, flexible, and re-useable business logic components. In many enterprise scenarios, business processes involve inter-actions with outside partners and suppliers. Integrating the systems of the partners and suppliers with those of the enterprise improves efficiency of the overall value chain. Partner Services provide the document, protocol, and partner management services required for efficient implementation of business-to-business processes and inter-actions. Underlying all these capabilities of the SOA Reference Architecture is a set of Infrastructure Services which provide security, directory, IT system management, and virtualization functions. The security and directory services include functions involving authentication and authorizations required for implementing, for example, single sign-on capabilities across a distributed and heterogeneous system. IT Services Management Services include functions that relate to scale and performance, for example edge services and clustering services, and the virtualization capabilities allow efficient use of computing resources based on load patterns, etc. The ability to leverage grids and grid computing are also included in infrastructural services. While many of the Infrastructure and IT Service Management services perform functions tied directly to hardware or system implementations, others provide functions that interact directly with integration services provided in other elements of the architecture through the ESB. These interactions typically involve services related to security, directory, and I/T operational systems management. Wrap up: The SOA Reference Architecture is a complete and comprehensive architecture that covers all the integration needs of an enterprise. Its services are well integrated and are delivered in a modular way, allowing SOA implementations to start at a small project level. As each additional project is addressed, new functions can be easily added, incrementally enhancing the scope of integration across the enterprise. In addition to supporting SOA strategies and solutions, the architecture itself is designed using principles of service orientation and function isolation.

WebSphere_UsrGrp_PubSub.ppt WebSphere_UsrGrp_PubSub.ppt Presentation Transcript

  • Publish / Subscribe Tim Pickrell IT Specialist Business Integration Architect, Lab Services
  • Agenda
    • IBM’s Publish / Subscribe Product set
    • Publish / Subscribe Concepts and facilities
    • Ease of implementation
    • Why use Publish / Subscribe?
    • Experiences and some Gotchas
  • Publish / Subscribe Product Set - History
    • MA0C Supportpac
    • MQSeries Integrator V2.0
      • Topic Based Security and contect based filters
      • RFH2 Header
    • MQSeries Integrator V2.0.x
      • Telemetry and Mobile
    • WebSphere MQSeries Event Broker V2.1
      • Real-time transport
    • WebSphere MQSeries Integration Message/Event Broker V5
      • Multicast
  • WebSphere Business Integration Brokers
    • WebSphere Publish / Subscribe
    • WebSphere Business Integrator Event Broker V6.0
    • WebSphere Business Integrator Message Broker V6.0
    • WebSphere Service Integration Bus
  • Publish / Subscribe Concepts and Capabilities
  • What is Publish / Subscribe? Publisher 1 Train Schedules Publisher 2 Flight Schedules Subscriber 1 Train Schedules Subscriber 2 Train and flight Schedules Broker
  • What is a Subscriber?
    • An application that is interested in receiving data of a certain type or category
      • matches a specific category or “topic”
    • Need not know where the data comes from
    • Does not know how the data was delivered to the subscriber
    • Identifies the topic(s) on which it is interested in receiving associated data
    • Example
      • Interested in price of IBM stock
  • What is a Publisher?
    • Application that owns data to be made available to other applications (subscribers)
    • Does not know which or how many subscribers are interested in the data
    • Does not know how to deliver the data to the subscribers
    • Associates the data with a “topic”
    • Example: stock trading
      • New stock price
  • What is a topic?
    • A topic is an arbitrary string
    • Can be part of a hierarchy
      • Levels delimited by /
    • Wildcards are allowed on subscriptions
      • # matches any number of levels
      • + matches a single level
      • One # and any number of + permitted in a subscription
      • Wildcards must be delimited by /
      • # as topic matches everything
    • Support for multiple subscription points (streams) – MQ
      • Similar to high level topic but on a separate tree
  • What is a Topic - Example MarketData RealTime Delayed EQ FI OP SYMBOL EQ FI OP
    • Special characters in topics…
    • Topic level separator
    • Single-level wildcard +
    • Multi-level wildcard #
    e.g Subscription - MD/R/EQ/IBM
  • Publish / Subscribe characteristics
    • Subscriber population changing rapidly
    • All users are interested in an identical message
    • Push applications
    • Minimize management overhead
      • No central control or no control at all
    • Examples of good fit
      • Streaming video
      • Stock tickers
    • Multicast/Broadcast (if used) saves network bandwidth
      • Good for one to many
      • Logging more efficient
  • Publish / Subscribe Application Styles
    • Events
      • Every published message is significant
      • Subscribers receive as available
    • State
      • Information that is being updated/replaced
      • Brokers retain latest publications
      • Subscribers receive when updated and/or when desired
    • Conferencing
      • Publisher also subscriber
      • Option to suppress own publications
  • Programming Interfaces
    • MQI
      • Uses RFH2 headers for pub/sub information
    • JMS
      • Durable versus non-durable
      • API, not a protocol or implementation
    • XMS
    • AMI
      • Recommend you don’t use this for new development
    • WebSphere Message Broker compute node
  • MQI Publish / Subscribe Capabilities
  • Publish / Subscribe using the MQ transport Broker Publisher Subscriber Publish Delete Publication Response Response Register Subscriber DeRegister Subscriber Request Update Publish
  • Simple Publish / Subscribe Publisher Subscriber Broker Input Queue Control Queue 1 - Publish 3 - Publish 4 - Publish 3 - DeRegSub 2 - Subscribe 5 Subscriber Queue
  • Retained publications Publisher Subscriber Broker 1 - Publish 3 - Publish 2 - Subscribe 4 Subscriber Queue Input Queue Control Queue
  • Publish on request Publisher Subscriber Broker 1 - Publish 5 - Publish 2 - Subscribe 6 3 - Publish 4 – Request Update Subscriber Queue Input Queue Control Queue
  • Subscriptions
    • A subscription consists of:
    • One or more topics – wildcards can be used
    • An optional subscription point or Stream
    • An optional filter on the message content
    • A subscriber queue, queue manager and Correlation-id
  • The psc folder – registering a subscriber <psc> <Command> RegSub </Command> <Topic> Sport/Soccer/Score </Topic> <QMgrName> SPORT1 </QMgrName <QName> RESULTS </QName> <PubOpt> CorrelAsId </PubOpt> </psc> The combination of queue manager name, queue name and correlation-Id is known as the subscriber-Id
  • Mapping published messages to subscribers QMGR: SPORT1 Queue: RESULTS Publication message MQMD RFH2 Data <Topic/Sport/Soccer/Score</Topic> Topic: Sport/Events/# QMGR: SPORT1 Queue:Events, CorrelAsId Topic: Sport/Soccer/# Topic: Sport/+/Score QMGR: SPORT1 Queue: RESULTS, CorrelAsId Subscriptions Subscriber-Id MQMD: correlid from subscriber
  • Publishing using the MQ transport
    • Broker needs a Message Flow to tell it how to publish
    • Message flow names input queue for publish message source
    • Recommend failure and log queues
  • Publication Node Properties
  • Shared Subscription Queues
    • Shared subscription queues
      • Correlid sent on subscribe request
      • Published messages have same correlation id
      • Requires use of correlasid option
    • Non-shared subscription queues
      • Multiple subscriptions match same publication
      • Only a single message is returned to subscriber
  • Subscription DeRegistration
    • Subscriptions can be deregistered using the DeRegister subscription command message
    • A subscription may expire
    • A subscription can be removed by an administrator from the subscriptions view of the broker toolkit
    • The broker automatically deregisters subscriptions on closure of a temporary dynamic subscription queue
  • WebSphere Transports High Performance Broker WebSphere MQ Enterprise WebSphere MQ Mobile WebSphere MQ Telemetry WebSphere MQ Real-time WebSphere MQ Enterprise WebSphere MQ Mobile WebSphere MQ Telemetry WebSphere MQ Real-time WebSphere MQ Web Services WebSphere MQ Web Services Send and Receive using Any transport Transports Optimised for your unique application needs WebSphere MQ Multicast
  • JMS and XMS Publish / Subscribe Capabilities
  • Sample JMS Code // Obtain a TCF TopicConnectionFactory tcf = new MQTopicConnectionFactory(); // Obtain MQ-specific reference MQTopicConnectionFactory mqtcf = (MQTopicConnectionFactory)tcf; // Create a topic connection TopicConnection conn = tcf.createTopicConnection(); // start the connection conn.start(); // Create a topic session from the connection TopicSession topicsess = conn.createTopicSession(false, Session.AUTO_ACKNOWLEDGE); // Create the Topic String topicstring = &quot;TEST/TOPIC&quot;; Topic t = topicsess.createTopic(topicstring); // Create the publisher pub = topicsess.createPublisher(t);
  • Sample JMS Code continued // Set up the publication message TextMessage msg = null; msg = topicsess.createTextMessage(); String msgdata = &quot;Sample Publication message&quot;; msg.setText(msgdata); // Publish ! pub.publish(msg); // tidy up by closing the publisher, session and connection objects pub.close(); sess.close(); conn.close();
  • Sample JMS Code
    • package jms11;
    • import javax.naming.*; // The JNDI classes
    • import javax.jms.*; // The JMS 1.1 classes
    • public class JMS11UnifiedTransactionExample {
    • /**
    • * Transfer a message from a source destination
    • * to a target destination. The destinations may
    • * both be queues, both be topics, or may be from a
    • * queue to a topic, or from a topic to a queue.
    • */
    • public void transferMessage(String connectionFactoryName, String sourceName, String targetName)
    • throws NamingException, JMSException {
    • // Get the specified connection factory and destinations
    • Context jndiContext = new InitialContext();
    • ConnectionFactory factory = (ConnectionFactory)
    • jndiContext.lookup(connectionFactoryName);
    • Destination source = (Destination) jndiContext.lookup(sourceName);
    • Destination target = (Destination) jndiContext.lookup(targetName);
    • // Create the connection and session
    • Connection connection = factory.createConnection();
    • Session session = connection.createSession(true,
    • Session.AUTO_ACKNOWLEDGE);
  • Sample JMS Code - continued
    • // Use the session and destinations to
    • // create the consumer and producer
    • MessageConsumer consumer = session.createConsumer(source);
    • MessageProducer producer = session.createProducer(target);
    • // Transfer the next message on the source
    • // destination to the target destination
    • // in a single transaction
    • try {
    • producer.send(consumer.receive());
    • session.commit();
    • }
    • catch (JMSException ex) {
    • session.rollback();
    • }
    • // Release all resources
    • producer.close();
    • consumer.close();
    • session.close();
    • connection.close();
    • }
    • }
  • XMS Client Overview
    • A C++ implementation of the JMS Messaging API
    • Simple API to code, simplifies the programming and manages all connections
    • Common messaging API across IBM integration products
      • Current (WMQ) and future (WPM etc.)
    • High level Message classes (JMS 1.1)
    • Simplified Point-to-Point and Publish/Subscribe messaging
      • Send/Receive messages to/from queues or topics
    • Decoupled Administration
      • Admin tooling to define Connection Factories and Destinations
      • Directory can be LDAP, COS naming, or file system based JNDI
      • XMS and JMS can both use the same administered objects
  • Sample Message Service Client for C/C++ code (IA94) using IBM.XMS; namespace IBM.XMS.LeggiXMS { public class LeggiXMS { /// <summary> /// The main entry point for the application. /// </summary> [STAThread] static void Main(string[] args) { try { Hashtable hash = new Hashtable(); hash.Add(XMSC.IC_URL, &quot;file://C:/JNDI-Directory/.bindings&quot;); IContext ctx = new InitialContext(hash); IConnectionFactory connectionFactory = (IConnectionFactory)ctx.Lookup(&quot;CF_XMS&quot;); IConnection connection = connectionFactory.CreateConnection(); ISession session = connection.CreateSession(false, AcknowledgeMode.AutoAcknowledge); IDestination topic = (IDestination)ctx.Lookup(“TOPIC_XMS&quot;); IMessageConsumer consumer = session.CreateConsumer(topic); IMessage msg = consumer.Receive(0); ITextMessage msgTxt = msg as ITextMessage; if (msgTxt != null) { System.Console.WriteLine( msgTxt.Text ); } else { System.Console.WriteLine( &quot;Empty message&quot; ); } consumer.Close(); queue.Dispose(); session.Close(); connection.Close(); } catch (XMSException e) { System.Console.WriteLine(e.Message); System.Console.WriteLine(e.LinkedException.Message); } }
  • WebSphere Real-time transport
    • High-performance protocol designed for publish/subscribe
      • Developed at IBM Research (Hawthorne)
    • Non-persistent (non-durable) messages only
    • Uses direct TCP/IP connection
      • Does not use WebSphere MQ
    • Supports JMS and XMS clients
    • Integrated in Message Broker V6.0 and Event Broker V6.0
      • First released in WMQ Event Broker V2.1
    • Number of connections may be limited by operating system
      • Requires multiple brokers in collective for more connections
    • Unicast and Multicast
  • Unicast vs Multicast Multiple transmissions to send the same message. Single transmission to send same message to all clients. All clients must be on the same LAN segment. Unicast Multicast 4 Transmissions 1 Transmission
  • Multicast Advantages
    • Scaleable
      • Conserves network bandwidth
      • Minimises broker resource utilisation
      • Particularly effective with high numbers of subscribers
    • Fairness (simultaneous reception)
      • Only within the same LAN segment
    • Minimises latency
    • Neat Mapping with publish / subscribe
  • Publish / Subscribe nodes Receives Subscription and Publication Requests Receives Subscription and Publication requests. Publishes directly. Receives Publication requests that include an RFH header
  • Real-timeOptimizedFlow node
  • Real-Time Input node
  • Real-timeInput Node properties
  • Broker Networks Broker Broker Broker Broker Broker Broker Broker Broker Broker London New York Tokyo
    • Multiple brokers can be interconnected
    • Performance & scalability
    • Availability
    • Geographical distribution
    ConfigMgr – not shown
  • Broker Network configuration
  • Broker Properties
  • Broker – Multicast properties
  • Broker Toolkit - Subscription view
  • Cloned Subscriptions – High Availability QM P QM QM SUBQ Broker PUBQ CTRLQ S Broker PUBQ CTRLQ QM SUBQ S
  • Message Broker – Topic Based Security
  • Best Practices
    • Deciding on a transport
    • Queuing vs messaging
    • Is the application really a pub/sub application?
    • Logging of published messages
    • High fan-out vs high fan-in
  • Why use Publish / Subscribe?
    • All applications are interested in an identical message
    • Subscriber population is dynamic and changes rapidly
  • Market Data Solution Configuration RMMToEB RMMToEB Trade Pipe JMS JMS RMM RMM RMM ITM DJ30,DJ30 Delay,Options, Position blotter, History request, etc Broker 1 Broker 2 WAS Fat Pipe Tick Receiver WAS Real Time Loader MQ Position Server Trade Feed External Trades TAM
  • Design of the Topic Hierarchy
    • Design of the topic hierarchy requires careful consideration !
    • Must be flexible enough to satisfy business requirements when matching publications to subscribers
    • Must provide sufficient granularity for topic based security
    • Has a potential impact on performance – including assignment of multicast groups
  • Design of the Topic Hierarchy Continued /[ R|D ]/[ E|O|F ]/ B[1..100 ]/ Symbol / Exchange Real-time or Delay Service Equities, Fixed-Income or Options Multicast Bucket Symbol e.g. IBM Exchange e.e Nasdaq
  • Real Time Messaging – Example architecture
  • Solution Components
      • IBM Out of the Box Components
      • WMQ
      • Message Broker
      • Replay Server
      • WebSphere Application Server
      • DB2
      • XMS Client API
      • Custom Applications
      • WebSphere Application Server App
      • Heartbeat Monitor App
  • Replay Server Command Handler Solution Server Monitor App S S XMS UDB Broker Replay Proxy
  • WAS Core Components WebSphere Application Server Application Server Cluster ATP Client Clustered Messaging Endpoint Messaging Endpoint Resource Status Table S
  • Register Client Subscription S Replay Proxy: Simple class that subscribes to replay server and publishes to client topics. Serves multiple clients. MDB: Client Command Handler
    • Lookup Table
    • ---------
    • ---------
    Client Command Handler: Implemented as a Message Driven Bean, handles subscription and description requests from the client. Message Broker Replay Server 1 2 3
    • Register subscription request
    • Add reference to lookup table
    • Register client with replay proxy
    Replay Proxy Verb Topic Timestamp Lookup Table: Client to Replay server topic mapping ID/WS 00:34:45 Sub
  • Related material
    • Redbook - MQSeries Publish/Subscribe Applications
      • http://publib-b.boulder.ibm.com/abstracts/sg246282.html
  • WebSphere Event Broker
    • Subset of Message Broker V6
      • Transformation capability removed
        • Message repository
        • Most processing nodes (e.g. compute, filter)
      • Includes Generic XML parser only
        • Content based limited to XML messages
    • Both Message Broker and Event Broker include
      • Gryphon technology from IBM Research for real-time support
      • Multicast support developed by IBM Haifa
  • SOA Reference Architecture Business Innovation & Optimization Services Development Services Interaction Services Process Services Information Services Partner Services Business App Services Access Services Integrated environment for design and creation of solution assets Manage and secure services, applications & resources Facilitates better decision-making with real-time business information Enables collaboration between people, processes & information Orchestrate and automate business processes Manages diverse data and content in a unified manner Connect with trading partners Build on a robust, scaleable, and secure services environment Facilitates interactions with existing information and application assets IT Service Management Infrastructure Services Optimizes throughput, availability and performance Apps & Info Assets ESB Facilitates communication between services