WebSphere Connectivity & Integration: What's New in the Messaging Family?


Published on

WebSphere Connectivity & Integration:
What's New in the Messaging Family?
Hursley Comes To You 2011
Andy Piper
WebSphere Messaging Community Lead

Published in: Business
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

WebSphere Connectivity & Integration: What's New in the Messaging Family?

  1. 1. Hursley Comes To The Nordics – May 2011WebSphere Connectivity & Integration: Whats New in the Messaging Family?Andy Piper WebSphere Messaging Community Lead WebSphere © 2011 IBM Corporation
  2. 2. Hursley Comes to the Nordics – May 2011Coming up in the next hour... Universal Messaging WebSphere MQ 7.0.1.x Files, Security, Telemetry... oh my! Whats new in the community?2 © 2011 IBM Corporation
  3. 3. Hursley Comes to the Nordics – May 2011 The vision: Universal Messaging Any skills Any platform Any QoS Any network Any speed Any data Any delivery level Any device Universal Messaging3 25/05/11 © 2011 IBM Corporation
  4. 4. Hursley Comes to the Nordics – May 2011 Universal MessagingDynamic network that delivers the data you require from wherever it resides to wherever youwant it in whatever way you want it at whatever time you want it Universal Messaging 1. Best Delivery • Choice of service • Resilience, Integrity, Security • Throughput, Latency • High availability 2. Anything Anywhere • Any skills • Any traffic • Any language • Any environment • Any platform 3. Scale Dynamically • Start small • Grow incrementally • Stretch elastically • Scale admin4 25/05/11 © 2011 IBM Corporation
  5. 5. Hursley Comes to the Nordics – May 2011 Messaging and a Smarter Planet Connectivity for Applications Head Offices, Data centres Transport WebSphere MQ REST/HTTP Intelligent Cloud Computing WS* Mission-Critical Connectivity & Intelligence Messages WebSphere MQ Regional Offices Files WebSphere MQ File Transfer Edition Stores, Outlets Stores, Outlets Transform, WebSphere Message Broker Interconnected Enrich, Mediate WebSphere ESB DataPower WebSphere Sensor Events Catalog WebSphere Service Registry & Repository Intelligence WebSphere Business Events Cognos. ILOG, SPSS Instrumented InfoSphere Streams Other InfoSphere and Tivoli productsConnectivity for Smart Devices Remote Systems and Devices Systems and DevicesTransport MQ Telemetry HTTP Multicast Embedded Controllers Sensors Actuators Filtering of duplicate read events, Store- Power meters, weather data Tag printers, status lights, Load based HVAC & lighting controls, Industrial SCADA sensors, pressure, volume, generation, HVAC and lighting, Network Gateways (SCADA) RFID readers, Motion detectors… Valves, switches and pumps…5 © 2011 IBM Corporation
  6. 6. Hursley Comes to the Nordics – May 2011WebSphere MQ Delivery Roadmap (2011) MQ FTE 7.0.4 (11/10) MQ FTE V7.0.3 (11/09) (10/10) MQ FTE V7.0.2 MQ Advanced (06/09) (08/10) Message MQ FTE V7.0.1 Security MQ v7.0.1.3 (08/10) (12/08) MQ Telemetry MQ File Transfer Edition V7 (1H10) (08/09) SupportPacs etc MQ V7.0.1 with (01/09) Multi-Instance Queue Managers MQ V7.0.0.1 with Automatic Client Reconnect Service Def Wizard Enhanced .NET support (05/08) XA-aware API Exit Increased z/OS Capacity MQ V7 with z/OS Group Units of Work Integrated Pub/Sub Rearchitected JMS Extended APIs V7.0.1 EAP Early Access Programs 06/08 12/08 06/09 12/09 06/10 12/106 © 2011 IBM Corporation
  7. 7. Hursley Comes to the Nordics – May 2011IBM WebSphere MQ V7.0.0 - Summary  Enhanced JMS –Underpins many SOA/ESB solutions needing access to messaging –Improved performance & ease-of-use  Enhanced Publish-and-subscribe –Ease-of-use –New support for z/OS  Extended verbs and behaviors for MQI programming interface  Enhanced MQ clients for increased throughput resilience and availability  Web 2.0 support to help create richer user experience7 © 2011 IBM Corporation
  8. 8. Hursley Comes to the Nordics – May 2011 WebSphere MQ V7.0.18 © 2011 IBM Corporation
  9. 9. Hursley Comes to the Nordics – May 2011WebSphere MQ V7.0.1 – Major Themes  Enhanced availability options on Distributed platforms  Constraint relief on z/OS  IBM portfolio exploitation, and extension of reach, for V7 features  Ongoing performance, consumability and serviceability enhancements  Keeping pace with industry evolution in areas such as platforms and SSL9 © 2011 IBM Corporation
  10. 10. Hursley Comes to the Nordics – May 2011Distributed Platforms: Multi-instance Queue Managers  Basic failover support without HA coordinator – Faster takeover: fewer moving parts – Cheaper: no specialised software or administration skills needed – All non-z/OS platforms  Queue manager data is held in networked storage – NAS, NFS, GPFS etc so more than one machine sees the queue manager data – Improves storage management options: formal support for these even without failover config  Multiple (2) instances of a queue manager on different machines – One is “active” instance; other is “standby” instance – Active instance “owns” the queue manager’s files and will accept app connections – Standby instance does not “own” the queue manager’s files and apps cannot connect • If active instance fails, standby performs queue manager restart and becomes active  Instances share data, so it’s the SAME queue manager10 © 2011 IBM Corporation
  11. 11. Hursley Comes to the Nordics – May 2011Multi-instance Queue Managers 1. Normal MQ MQ Execution Client Client network Machine A Machine B QM1 QM1 Active can fail-over Standby instance instance QM1 networked storage Owns the queue manager data11 © 2011 IBM Corporation
  12. 12. Hursley Comes to the Nordics – May 2011Multi-instance Queue Managers 2. Disaster MQ MQ Strikes Client Client networkConnectionsbroken fromclients Machine A Machine B QM1 QM1 Active locks freed Standby instance instance QM1 networked storage12 © 2011 IBM Corporation
  13. 13. Hursley Comes to the Nordics – May 2011Multi-instance Queue Managers 3. Standby MQ MQ ConnectionsComes to Life Client Client still broken network Machine B QM1 Active instance QM1 networked storage Owns the queue manager data13 © 2011 IBM Corporation
  14. 14. Hursley Comes to the Nordics – May 2011Multi-instance Queue Managers 4. Recovery MQ MQ Complete Client Client Clients reconnected. Processing continues. network Machine B QM1 Active instance QM1 networked storage Owns the queue manager data14 © 2011 IBM Corporation
  15. 15. Hursley Comes to the Nordics – May 2011Multi-instance queue managers: Details  MQ is NOT becoming an HA coordinator – Generally, if other resources also required, use an HA coordinator such as HACMP – Service objects can restart applications with qmgr but limited control – Message Broker also integrates with and exploits this MQ function  The IP address is not taken over – Channel config needs all possible addresses unless you use external IPAT or intelligent router – CONNAME(‘host1(port1),host2(port2)’) syntax extension on all platforms including z/OS  Support for networked storage over modern network file system protocols – For example, NFS v4 (not v3) – Tool shipped to validate configuration  New options for crtmqm/strmqm/endmqm to control operations – Cannot guarantee which instance becomes the primary  Removes need for MC91 (withdrawn)15 © 2011 IBM Corporation
  16. 16. Hursley Comes to the Nordics – May 2011Automatic Client Reconnection  Client library provides necessary reconnection logic on detection of a failure  Hides failure from application code QM1 QM1 Application Application QM2 QM2 MQ Client MQ Client QM3 QM316 © 2011 IBM Corporation
  17. 17. Hursley Comes to the Nordics – May 2011Automatic Client Reconnection  Tries to hide queue manager failures by restoring current state automatically – Re-opens queues and other qmgr objects, re-establishes subscriptions – For example, if MQPUT returns error, client reruns MQCONN/MQOPEN/MQPUT internally – Not all MQI is seamless, but majority repaired transparently  Uses the list of addresses in CONNAME to find queue manager – MQSERVER environment variable also understands list – MQSERVER=SYSTEM.DEF.SVRCONN/TCP/host1(1414),host2(1414)  Can reconnect to the same or different Queue Manager  Reconnection interval is backed off exponentially on each unsuccessful retry – Total timeout is configurable – default 30 minutes.  Enabled in application code or ini file – Event Handler callback shows reconnection is happening if app cares17 © 2011 IBM Corporation
  18. 18. Hursley Comes to the Nordics – May 2011Windows Communication Foundation  Windows Communication Foundation (WCF) – Underpins .NET Web Services and Messaging – Built-in Transports – e.g. MSMQ, HTTP(S), Named Pipes, TCP/IP, etc. – Transports can be extended with “custom channels” – Allows alternative transports (like MQ) to be slotted into WCF seamlessly  WCF support now included in product – Previously an Alphaworks prototype  Built on XMS .Net classes – Now shipped as part of the product instead of SupportPac – Internally, it exploits V7 API, in same way as JMS implementation18 © 2011 IBM Corporation
  19. 19. Hursley Comes to the Nordics – May 2011Security & Monitoring  Command and Configuration Events for Distributed platforms – Matching already-available z/OS function – Successful commands – MQSC or PCF – recorded as event messages – Configuration changes report “before” and “after” definitions of objects – Provides a record of who changed what and when display qmgr event AMQ8408: Display Queue Manager details. QMNAME(V7) CMDEV(ENABLED) CONFIGEV(ENABLED) …  SSL OCSP Support – Now commonly used as alternative to LDAP-based CRLs – Simpler to manage as no need to have an LDAP server – Can use details provided in inbound certificate – Also can be configured within queue manager or by application code19 © 2011 IBM Corporation
  20. 20. Hursley Comes to the Nordics – May 2011z/OS Constraint Relief  In V7.0, the queue manager started to exploit 64-bit addressing – New Pub/Sub features  In V7.0.1 more Queue Manager storage moves to 64-bit – 64-bit Queue Indices – 64-bit Lock Manager  Can have more open queues, more messages on indexed queues etc  Small message storage changed – One message per page instead of fitting multiples into single page – May increase DASD use, especially if messages not quickly retrieved – But typical workloads show improved performance and reduced CPU20 © 2011 IBM Corporation
  21. 21. Hursley Comes to the Nordics – May 2011z/OS Log Compression  Can increase the throughput possible for persistent messages  May reduce the size of your logs – Dependent on your message content – Useful if you are DASD constrained.  RLE (run-length encoding) of “insert” log records for private queue messages – Will not compress shared queue log records – SMF 115 records updated to show compression rates achieved etc  Controlled via zPARM option at queue manager level. – COMPLOG(NONE) or COMPLOG(RLE) in CSQ6LOGP – Can also be viewed/controlled via DISPLAY LOG / SET LOG21 © 2011 IBM Corporation
  22. 22. Hursley Comes to the Nordics – May 2011 WebSphere MQ SupportPacs22 © 2011 IBM Corporation
  23. 23. Hursley Comes to the Nordics – May 2011Most recent SupportPacs  MA98: WebSphere MQ - Connection Endpoint Look-up Exit – Find connection information from an LDAP server  MH05: WebSphere MQ - Events Display Tool – Reads and displays events from SYSTEM.ADMIN.*.EVENT queues – Can be triggered from WMQ – Can be written to a file  MO06: WebSphere MQ – Message Log Summary Utility – Summarise and filter Qmgr Message Logs See http://pipr.co/supportpac for the full list of recent SupportPacs23 CSS: S D © 2011 IBM Corporation
  24. 24. Hursley Comes to the Nordics – May 2011 WebSphere MQ Low Latency Messaging24 © 2011 IBM Corporation
  25. 25. Hursley Comes to the Nordics – May 2011WebSphere MQ Low Latency Messaging  Initially built for Financial Markets customers requiring high performance, reliable messaging in the front office for market data and trade data  Included with WebSphere Front Office for Financial Markets  Integrated with Partners such as Data Vendors/ISVs for internal application communication and external application connectivity  Production deployments in market data distribution systems, internal trade execution systems and exchange execution systems  Also applicable in industries with similar quality of service requirements – Defense and Aerospace – Telecommunications – Logistics25 © 2011 IBM Corporation
  26. 26. Hursley Comes to the Nordics – May 2011Positioning of IBM Messaging Transports WebSphere MQ Low Latency WebSphere MQ Messaging Speed – Quality of Time Independent Ultra-Fast Service “As-Soon-As-Possible” “Microseconds” Reliability – Quality of Either non-persistent or persistent Ranges from unreliable to assured Service “Ultra-Reliable, Bet-The-Business” “Variable, based on class of data” Target Market Enterprise Messaging Low Latency Messaging Financial Markets and several others with similar Target Industries All QoS needs Queuing Yes No Standardized Unique / specific APIs (JMS, XMS and MQI) (C, Java, .NET) Linux and Windows on x86, Platform coverage 80 platforms Linux on pSeries & zSeries, Solaris (x86,SPARC), HP-UX, AIX Included in WAS, WMB, WESB, DataPower XI50, DB2, Interoperates with CICS, IMS, etc, etc… WebSphere Front Office Integrated with DataPower26 © 2011 IBM Corporation
  27. 27. Hursley Comes to the Nordics – May 2011Software Overview Peer-to-peer  Fully distributed  No centralized management – No centralized queue for messages – Daemon-less Publish-Subscribe  Transmitter publishes on a topic (RMM) or “queue” (RUM)  Receiver subscribes to a topic or “queue”  Topic usually identified by a unique name. – Can have multiple transmitters/receivers for the same Topic Transports –Ethernet 1GigE and 10 GigE –Infiniband – Native verbs or IB over IP –Shared Memory © 2011 IBM Corporation
  28. 28. Hursley Comes to the Nordics – May 2011 WebSphere MQ File Transfer Edition28 © 2011 IBM Corporation
  29. 29. Hursley Comes to the Nordics – May 2011Introducing WMQ File Transfer Edition Adds file transfer services to WebSphere MQ to enable managed file movement Multi-purpose solution combining file transfer and messaging on a single backbone  Reliable – leveraging the WebSphere MQ transport automate  Multi-purpose – transfers both messages and track audit configure files  Auditable – logging subsystem tracks transfer at source and destination for audit purposes  Centralized – monitoring, control and configuration  No need to program – no MQ skills required  Graphical tooling – visual configuration and status  Moves Massive files – no size limit, Kb, Mb, Gb…  Command line interface – for advanced users WebSphere MQ File Transfer Edition  Scripting support – enables automation via scripting of complex multi-step transfers in XML using Apache Ant  Flexible backbone – moves files from anywhere a b c x y z to everywhere in network  Integration with MQ-enabled apps and ESBs  Automatic file character conversion  Security of file payload using SSL  Support a wide range of Platforms and O/S © 2011 IBM Corporation
  30. 30. Hursley Comes to the Nordics – May 2011WMQ File Transfer Edition 7.0.3 & 7.0.4 Increased web support –REST interfaces –Web transfers from end-users Extended platform support: AIX on Power7, Linux on Power7, Oracle Solaris 10 on Intel x86 and virtualized platforms WebSphere Message Broker integration –FTEInput and FTEOutput nodes in WMB FP2 WebSphere Advanced Message Security integration –Easily encrypt file transfers Sterling Connect:Direct Bridge capability including audit trail 7.0.4 GA June 3 2011 © 2011 IBM Corporation
  31. 31. Hursley Comes to the Nordics – May 2011 WebSphere MQ Advanced Message Security31 © 2011 IBM Corporation
  32. 32. Hursley Comes to the Nordics – May 2011WebSphere MQ Advanced Message Security  End-to-End Message Security - Secures application data even before it is passed to MQ  Extension to base MQ – No changes to existing applications WebSphere MQ standard security: Message data can be encrypted in transport (SSL) but not when it resides in the queues Authentication is based on Operating System identifier of local process Application A Application Z WebSphere MQ Advanced Message Security Supplements WMQs security features: + Assurance that messages have not been WebSphere MQ Advanced Message Security altered in transit WebSphere MQ + Assurance that messages originated from the expected source + Assurance that messages can only be viewed by intended recipient(s) + Administered using queue based policies Securing the data and created from the WMQ Explorer or the applications command line tooling.32 © 2011 IBM Corporation
  33. 33. Hursley Comes to the Nordics – May 2011Advanced Message Security – Key Elements  Message Interceptors  Protection policy for the application queue  Public / private key pairs  Recipient’s public key Sending Application (Alice) Receiving Application (Bob) AMS_QM APP.Q Keystore Keystore Alice Priv Bob Priv Alice Pub Bob Pub Bob Pub Policy APP.Q Privacy Recipient : Bob © 2011 IBM Corporation
  34. 34. Hursley Comes to the Nordics – May 2011WebSphere MQ Advanced Message Security Adds message-level security to existing WebSphere MQ (V6 & V7) Support details: –C MQI and Java clients –Standard WMQ distributed platforms (apart from iSeries), including zOS –Requires a minimum of WMQ v6.0.2.8 or WMQ v7.0.1.2 Constraints –Point to point messaging domain only –On zOS MQI v6 verbs only (no async consume, message properties etc) © 2011 IBM Corporation
  35. 35. Hursley Comes to the Nordics – May 2011 WebSphere MQ Telemetry35 © 2011 IBM Corporation
  36. 36. Hursley Comes to the Nordics – May 2011The Internet of Things Many smart devices instrument our world today Interconnecting these smart devices creates a Central Nervous System36 © 2011 IBM Corporation
  37. 37. Hursley Comes to the Nordics – May 2011Universal Messaging for a Smarter Planet Enterprise Regional Office Offer connectivity capabilities optimised for sensors and devices Branch Outlet Provide Enterprise Access to new sources of data to support your business Provide access to Enterprise data and services from new platforms/businesses Sensor Retail e.g. Store Deliver data to intelligent RFID decision-making assets Zero-Administration for “remote” deployments Pervasive Device Refinery Petrol Forecourt37 25/05/11 © 2011 IBM Corporation
  38. 38. Hursley Comes to the Nordics – May 2011Connectivity with MQTT ■ MQ Telemetry Transport (MQTT) is designed with the following primary intentions: – Publish/subscribe messaging paradigm as required by the majority of SCADA and sensor applications. – Minimise the on-the-wire footprint. – Expect and cater for frequent network disruption. – Cope with slow and poor quality networks – Expect that client applications may have very limited processing resources available. – Provide traditional messaging qualities of service where the environment allows – Publish the protocol for ease of adoption by device vendors and third-party client software. © 2011 IBM Corporation
  39. 39. Hursley Comes to the Nordics – May 2011 Extended Reach Basic Telemetry WebSphere MQ Telemetry V7.0.1 extends IBM’s Client industrial strength enterprise message system Backend Server • Highly scalable – A single queue manager can handle 100,000 concurrently connected devices • Ships with two types of client: WebSphere – Basic MQ • direct connectivity from a device – Advanced: • Acts as a “concentrator/hub” for mini-networks of Telemetry devices • Can connect to multiple backend servers • Can failover to alternate backend server • Can buffer messages MQ Telemetry • Provides rich security Daemon for Devices – Network: SSL – Authentication: JAAS – Authorisation: OAM Advanced Telemetry Client • In addition, any third party, open source or “roll your own” MQTT client can be used39 25/05/11 © 2011 IBM Corporation
  40. 40. Hursley Comes to the Nordics – May 2011WebSphere MQ Telemetry Implementation Fully integrated / interoperable with WMQ – Publishers and subscribers can exchange messages with MQI and JMS applications Telemetry channels enable MQTT connections to queue manager – Supports MQTTv3.1 protocol Administration using WMQ Explorer Initially available on Windows and Linux Ships with reference Java (for MIDP upwards) and C clients Includes WebSphere MQ Telemetry Daemon for Devices – An advanced client which also accepts connections from other clients – Provides ability to store and forward, network local devices. © 2011 IBM Corporation
  41. 41. Hursley Comes to the Nordics – May 2011Broader integration Simple Lightweight (CPU,Mem,**Net) Data-centric Distribution (pub/sub) Range of QoS => developer/community interest! © 2011 IBM Corporation
  42. 42. Hursley Comes to the Nordics – May 2011 Case Study: Smarter Healthcare Medical organization created a remote pace-maker monitoring solution to provide better patient care Client Pains  Physicians needed better monitoring of cardiac patients  Improve efficiency of checkups  Meet healthcare data capture standards Enables higher level of patient care and peace of mind Improves administrative efficiency and maintenance Helps conform to standards and ease integration of data42 © 2011 IBM Corporation
  43. 43. Hursley Comes to the Nordics – May 2011 Case Study: Improving Energy Usage Utility company developing an Intelligent Utility Network offering for optimizing load on electricity grids Business Partner  Needs robust middleware technology to connect to remote smart meters  Needs to be able to rapidly scale solution nationally Able to offer daily energy savings of 15-20% Enables utilities to reduce peaks and avoid punitive charges Helps save electricity through better peak load management43 © 2011 IBM Corporation
  44. 44. Hursley Comes to the Nordics – May 2011 WebSphere MQ Community44 © 2011 IBM Corporation
  45. 45. Hursley Comes to the Nordics – May 2011 We love our users!  developerWorks  MQSeries.net community – http://mqseries.net/  Global WebSphere Community GWC – http://websphereusergroup.org  MQTT – http://mqtt.org  IBMers Blog on Messaging on developerWorks  Twitter – @IBM_WMQ and @IBM_Broker  YouTube – http://youtube.com/mqontv  Store and Forward – T.Robs WMQ Security blog – http://t-rob.net/  Open, vibrant ecosystem – e.g. PyMQI, mosquitto, github projects, etc. etc.4545 © 2011 IBM Corporation
  46. 46. Thank you! Contact: Andy Piper andy.piper@uk.ibm.comhttp://twitter.com/andypiper | http://andypiper.co.uk WebSphere © 2011 IBM Corporation