Today’s MQ Infrastructure & Tomorrows Security & High Availability with MQ 7.1, MQ AMS & MQ FTE Author: A.J. Aronoff Connectivity Practice Director Email: firstname.lastname@example.org Desk: 646-201-4943
Agenda – MQ InfrastructureUniversal Connectivity: The Path to the FutureMQ File Transfer EditionMQ Security – With MQ AMSMQ 7.1 – the latest MQ Infrastructure features Including MQ “Security Policies” 2
Prolifics Wins IBM Awards A Long Record Of IBM Honors Software Sales LeadershipMulti Award-winning: 2010 Lotus Award Best End-User Solution 2010 Lotus Award for Best Industry Solution 2009 Rational Solution Award 2008 Outstanding SOA Solution Award 2008 Overall Technical Excellence Award 2007 Overall Technical Excellence Award 2007 Impact SOA Process Solution Award 2006 Best Portal Solution Lotus Award Technical Innovation 2005 5-Star Partner Award demonstrating Prolifics’ cross-brand sales expertise and Serviced over 1200 IBM software accounts in the past 8 years; implemented over 250 portals certifications. One of only 5 partners world wide to receive the distinction Prolifics boasts more overall certifications than any other of the over 300 SVI partners in the US totaling over 250 J2EE & WebSphere certifications IBM’s highest technical rating (Level 5) IBM Tivoli “AAA Accredited”
by doing great work with Great Customers Financial Services Healthcare Government Retail & Distribution Utilities Insurance Education4
WebSphere MQ Value: Connectivity to, from & within an Enterprise The path to the future Enterprise A Universal Message Bus for access to Regional Office data wherever it exists to support your business Provides a comprehensive range of Messaging capabilities to support your Business requirements for data integration Managed File Transfer Sensor e.g. RFID Messaging integration patterns Branch Reliability and availability QoS Outlet SOA foundation Provides appropriate data access Retail and data privacy controls to help Store meet audit and regulatory requirements WMQ Telemetry is one step in extending the reach of WMQ to a wider world of data relevant to your business Pervasive Device Recent technology demonstration of MQ Refinery Web Messaging using HTML5 WebSockets continues this progress Petrol Mobile Forecourt Phone5 CSS: F S
IBM Universal Messaging Proven, Flexible, Robust business data delivery from anywhere to everywhere IBM UNIVERSAL MESSAGING Business Sense and Respond Transactions MQ MQ Telemetry Leveraging System z Web applications MQ for z/OS MQ HTTP Bridge Managed File Transfer Real-time Awareness MQ File Transfer Edition MQ Low Latency Messaging Extra Data Protection Cloud Platform-as-a-Service MQ Advanced Message Security MQ Hypervisor Edition7
WMQ Family Roadmap – continual delivery of customer value (4Q/09) (2Q/10) (4Q/10) (2Q/11) () MQ LLM V2.3 MQ LLM V2.4 MQ LLM V2.5 MQ LLM V2.6 MQ LLM V2.x msg store late join self-managing improved perf. (4Q/09) (4Q/10) (2Q/11) () MQ FTE V7.0.2 MQ FTE V7.0.3 MQ FTE V7.0.4 MQ FTE V7.x FTP Bridging end-to-end security C:D Integration (4Q/11 ) (1Q/11) MQ V7.1 with(3Q/09) MQ V126.96.36.199 Multi-version Install,MQ V7.0.1 with Pre-Connect Exit Out-of-the-box security, (3Q/10) Multicast capability,Multi-Instance QMgrs, MQ Telemetry V7.0.1Automatic Client Reconnect, Improved Performance,z/OS Availability, Capacity and z/OS Shared Q enhancementsPerformance improvements (2Q/11) (1Q/10) MQ WebSockets Tech Preview. Security SupportPacs and MQ HVE for RHEL ESX and Wizards IBM Workload Deployer (4Q/10) MQ Advanced Message Security () V7.0.1 MQ AMS V7.x200 201 201 201 9 0 1 2 Early Access Programs8 CSS: F S
MQ FTE Quick Overview Directory Monitoring File to Message - Message to File FTP & SFTP Bridging agents9
FTP Spaghetti Infrustructure (haphazard growth)X Unreliable transport mechanisms Each link in a chain reduces reliabilityX No central set-up, logging or monitoringX Poor documentation of overall systemX Expensive, one-off solutionsX High maintenance costs (60 – 70% of a company’s IT budget)X Lack of business agility
Ideal File Transfer Infrastructure Automation & Documented, Centralized Standardized Set-up Solutions Reliable Transport Reliable Reliable Transport Transport Event basedCentralized Centralized Monitoring Logging Reliable Reliable Transport Transport Reliable Reliable Transport Transport
MQ FTE 7.0.2 Protocol BridgeSupport for transferring files located on FTP and SFTP servers The source or destination for a transfer can be an FTP or an SFTP serverFully integrated into graphical, command line and XML scripting interfaces Just looks like another FTE agent…Enables incremental modernization of (S)FTP-based Legacy solutions This helps ease migration from a non-managed (FTP or SFTP) network to a managed network based on WebSphere MQ File Transfer Edition. (I.E. less rip & replace). Ensures reliability of transfers across FTP/SFTP with checkpoint restart Provides auditability of transfers across FTP/SFTP to central audit log Files exchanged between FTE and FTP/SFTP FTE FTP MQ FTE FTP FTP / SFTP FTE FTP network network FTE FTP Bridging FTP/SFTP Agent Server Protocol FTP Bridge Server Audit Agent information
MQ FTE: Use Case 1: Directory Monitor•Three sub directories with the same names of three destination FTE Agents•When a file with an extension of “doc” is added to one of the sub directories … • The Resource monitor detects the file and • creates a file transfer request for the file where the destination agent has the same name as the sub directory.http://www.ibm.com/developerworks/websphere/library/techarticles/0910_bonney/0910_bonney.html•Company in Florida is using the above system and planning to scale up further FTE Receiving Agent Resource OfficeA /incoming/monitor Monitor FTE Receiving Agent OfficeB /A /B /C FTE Sending Agent 1.Doc FTE Receiving Agent OfficeC 14
File & Message Broker Hub: Connect Anything to AnythingIntegration with WebSphere Message Broker for File Processing Tight integration between FTE and WebSphere Message Broker Enables ESB capabilities to be applied to file data Ability to parse and transform files and process into messages, files, events, service requests etc Messages Files Files WMQ FTE MQ, FTE, FTP, Network HTTP, SOAP… WebSpher e Message Broker Enrich, Mediate, Transform… 15
WMB FTEInput and FTEOutput nodes Message Broker Execution Group Message Flow FTE Agent FTE FTE FTE Agent Agent Agent FTEInput FTEOutput FTEInput node Build flows that accepts file transfers from the WMQ FTE network FTEOutput node Build flows that are designed to send a file across a WMQ FTE networkWhen WMQ FTE nodes are used in a flow an FTE agent is automatically stated in the Message Broker Execution Group16
File & Message Hub (HTTP and MQ FTE)Web based File Transfers using the Web GatewayWeb-based File Transfer A RESTful API for sending files into and receiving files from a WMQ FTE network Reliable and secure file transfer option for Web users Auditable transfer and large file support Zero-footprint file transfer support without the need to provision and install code Interfaces for embedding into third party and custom user applications WMQ FTE Serve WMQ FTE r Network HTT P/S17
Options for converting data between files & messagesOne file to one message WMQ One file becomes one message FTEOne file to a group of messages The file can be split WMQ based on: FTE Size Binary delimiterOne message to one file Regular expression WMQ One message becomes one file FTEA group of messages (or all messages on the queue) to one file Optionally, a delimiter can be WMQ inserted between each message FTE used to compose the file 18
End-to-end encryption usingWebSphere MQ Advanced Message Security WMQ FTE already svrconn WebSphere sndr/rcvr WebSphere supports transport level FTE channel MQ channels MQ FTE Agent Queue Queue Agent encryption using SSL Manager Manager Data is encrypted before it is sent over a channel and decrypted when it is received V7.0.3 (when combined with WMQ AMS v7.0.1) allows file svrconn WebSphere sndr/rcvr WebSphere FTE channel MQ channels MQ FTE data to be encrypted at the Agent Queue Queue Agent source system and only Manager Manager decrypted when it reaches the destination system – This helps reduce encryption costs – Data is secure even when at rest on a queue19
Customer Survey: Of the points below:Which point(s) matters most to you? Records complete and detailed audit log of entire file journey Auditable “What went where, when and to whom” File contents not corrupted or partially transmitted Reliable Files only appear at destination whole and intact Files content encrypted during transmission Secure File access authenticated and controlled Eliminates need to manually detect problems and restart transfers Automated Providing scheduling and triggering for event-driven transfers Centralized Remote control and monitoring of file progress from anywhere Able to deploy and re-configure file transfers instantaneously from anywhere Flexible Managing transfers end-to-end across a network – not just between 2 points Any file size No upper limit on the size of file that can be moved Integrated With SOA infrastructure: Messaging, ESBs, Governance, B2B and BPM Cost Provides a consolidated transport for moving both Files and MessagesEffective
WebSphere MQ Advanced Message SecurityWhat is it? New product - WebSphere MQ Advanced Message Security Replaces WebSphere MQ Extended Security Edition Component added to WebSphere MQ V7 or V6Enhances MQ security processing Provides additional security services over and above base QM Designed to assist with requirements such as PCI DSS complianceApplication ---> Application protection for point-to-point messagingIndustry standard asymmetric cryptography used to protect individual messagesUses Public Key Infrastructure (PKI) to protect MQ messages Uses digital certificates (X.509) for applicationsNon-invasive No changes required to MQ applicationsSecurity policies used to define the security level required Administratively controlled policies applied to queues • Command line • Explorer
Message Level ProtectionEnables secure message transfers at application levelAssurance that messages have not been altered in transit When issuing payment information messages, ensure the payment amount does not change before reaching the receiverAssurance that messages originated from the expected source When processing messages, validate the senderAssurance that messages can only be viewed by intended recipient(s When sending confidential information.
WMQ AMS - Key FeaturesSecures sensitive or high-value MQ messagesDetects and removes rogue or unauthorized messages before they are processed by receiving applicationsVerifies that messages are not modified in transit from queue to queueProtects messages not only when they flow across the network but when they are at rest in queuesMessages from existing MQ applications are transparently secured using interceptorsProtects point-to-point messages
WMQ AMS - Key Features (continued)No prereq products Significantly simplified installation and configuration compared to predecessor product Up and running in minutes …Works in conjunction with SSL Can choose to use either or both depending on your requirementsWorks in conjunction with WMQ authorisation model (OAM and SAF)No changes required to WMQ applications Works with local applications and clients, including Java Support for WMQ V6 and V7No changes required to existing object definitionsFine-grained policies to define which queues are protected and how Asymmetric cryptography used to protect individual messagesAdministratively controlled policies Command line MQ Explorer
MQ AMS interceptorsMQ AMS functionality is implemented in interceptors. There are no long running processes or daemons (Except in z/OS).Existing MQ applications do not require changes.Three interceptors are provided:1.Server interceptor for local (bindings mode) MQI API & Java applications. Implemented as queue manager API exit.2. MQI API client interceptor for remote (client mode) MQ API applications. MQ AMS interceptor imbedded in MQ client code.3. Java client interceptor for remote (client mode) MQ JMS and MQ classes for java applications (J2EE and J2SE). MQ AMS interceptor imbedded in MQ java client code. MQ V7.0 java client required. SupportPac MQC7 WebSphere MQ V7.0 clients.
Protecting files transferred with WMQ FTE AMS plugs in on top of / alongside WebSphere MQ File Tranfer Edition, enable file data to be encrypted in transit through the MQ network Apply AMS protection to your WMQ FTE agent data queue its that simple!
Instantly familiar UI and command line: no new tools to learn!
Message integrity policy example setmqspl -m QMThis policy is to enforce integrity protection (signature) -p Q.INTEGRITY for messages put on queue Q.INTEGRITY in queue manager -s SHA1 QM.The message signing algorithm -e NONE is SHA1.Messages can only by signed by -CN=pdmqss,O=tivoli,C=US one authorized application.Messages signed by any other signer are sent to the SYSTEM.PROTECTION.ERROR. QUEUE and error returned to the receiving application.
WebSphere MQ AMS 1.Install AMS Interceptor 2.Create public / private key pairs 3.Copy public key
AMS SummaryWebSphere MQ Advanced Message Security V7.0.1 It is a new member of the WebSphere MQ family. It is a replacement for MQ ESE V6.0 It protects message integrity and/or privacy. It supports MQ V6 and V7. It does not support Pub/Sub. Existing MQ applications do not require changes. MQ AMS uses interceptors, policies, keystores and certificates.
MQ in the cloudMQ Cloud Support: HyperVisor Editions HVE is pre-packaged image of MQ with an operating system For easy configuration deployment into virtualised environments First release included MQ V188.8.131.52 and Red Hat Enterprise Linux x86 64- bit OS Also now available with an AIX flavour Pre-defined patterns for IBM WebSphere Workload Deployer configure deploy HVE Config Pattern CSS: F S
WebSphere MQ V7.1: Feature Summary WebSphere MQ V7.1 Announced: 4 October 2011New Feature Benefits Availability: 11 November 2011 DetailsMulti-Version Install Unix and Windows support for multiple versions of MQ V7.x (AND Makes it easier to deploy and upgrade one copy of MQ V7.0.1) down to fixpack levels.capability on Distributed systems and stage version to version Relocatable installation support.platforms migration Applications can connect to any Qmgr IP address Authorisation capability Simplified Configuration Additional crypto algorithmsEnhanced Security More granular authorisation for non-local queues Enhanced Authorisation and Auditing Application Activity ReportsCloud Support Simplifies and support Cloud deployments Additional HVE images Authorisation on Cluster Q rather than XMIT Q on Dist. PlatformsEnhanced Clustering Improves ease-of-use Bind-on-Group Support MQ Pub/Sub Topic space can now map to multicast Group New messaging QoS provides low latencyMulticast capability with high fan-out capability Addresses Provides direct interoperability with MQ LLM Further exploitation of z196 Code contention reduced to improve multi-processor linear scalingImproved scalability and Use of MQ Datasets rather than DB2 significantly improves “large”availability on z/OS Customer control over CF storage use message capability CF Connectivity Loss improvements Structure rebuild capability for CF Connectivity Loss scenariosImproved Performance on Improved multiprocessor exploitation Various code improvementsDist platforms42 CSS: F S
Scalability & Performance – Distributed platforms Performance measured and improved for a range of scenarios Hardware capabilities have evolved over years to have more CPUs, more memory etc MQ topologies have evolved to have more clients and larger/fewer queue managers “Fastest MQ ever”: better performance than V6 and V7 Multicast faster than traditional non-persistent Over 5x for one-many publications Performance reports to be released on availability CSS: F SCSS: F S
Channel Access Blocking Points Access Control Lists Channel blocking and mapping Listener blocking IP Firewall CSS: F
Blocking at the Listener Single list of IP address patterns NOT A REPLACEMENT FOR AN IP FIREWALL Temporary blocking Blocking until IP firewall updated Shouldn’t be many entries in the list Blocked before any data read from the socket i.e. before SSL Handshake Before channel name or userid is known Avoiding DoS attack Really the place of the IP firewall Simplistic ‘hold’ of inbound connection to avoid reconnect busy loop Network Pingers if blocked don’t raise an alert Immediate close of socket with no data not considered a threat SET CHLAUTH(*) TYPE(BLOCKADDR) ADDRLIST(‘9.20.*’, ‘192.168.2.10’) CSS: F
Channel Access Policy (1) SET CHLAUTH(*) TYPE(ADDRESSMAP) ADDRESS(‘*’) USERSRC(NOACCESS) “We must make sure our system is completely locked down” CSS: F
Channel Access Policy (2) SET CHLAUTH(*) TYPE(ADDRESSMAP) ADDRESS(‘*’) USERSRC(NOACCESS) SET CHLAUTH(BPCHL.*) TYPE(SSLPEERMAP) SSLPEER(‘O=Bank of Shetland’) MCAUSER(BANK123) SET CHLAUTH(BPCHL.*) TYPE(SSLPEERMAP) SSLPEER(‘O=Bank of Orkney’) MCAUSER(BANK456) “Our Business Partners must all connect using SSL, so we will map their access from the certificate DNs” CSS: F
Channel Access Policy (3) SET CHLAUTH(*) TYPE(ADDRESSMAP) ADDRESS(‘*’) USERSRC(NOACCESS) SET CHLAUTH(BPCHL.*) TYPE(SSLPEERMAP) SSLPEER(‘O=Bank of Shetland’) MCAUSER(BANK123) SET CHLAUTH(BPCHL.*) TYPE(SSLPEERMAP) SSLPEER(‘O=Bank of Orkney’) MCAUSER(BANK456) SET CHLAUTH(SYSTEM.ADMIN.SVRCONN) TYPE(ADDRESSMAP) ADDRESS(‘9.20.1-30.*’) MCAUSER(ADMUSER) “Our Administrators connect in using MQ Explorer, but don’t use SSL. We will map their access by IP Address” CSS: F
Channel Access Policy (4) SET CHLAUTH(*) TYPE(ADDRESSMAP) ADDRESS(‘*’) USERSRC(NOACCESS) SET CHLAUTH(BPCHL.*) TYPE(SSLPEERMAP) SSLPEER(‘O=Bank of Shetland’) MCAUSER(BANK123) SET CHLAUTH(BPCHL.*) TYPE(SSLPEERMAP) SSLPEER(‘O=Bank of Orkney’) MCAUSER(BANK456) SET CHLAUTH(SYSTEM.ADMIN.SVRCONN) TYPE(ADDRESSMAP) ADDRESS(‘9.20.1-30.*’) MCAUSER(ADMUSER) SET CHLAUTH(TO.CLUS.*) TYPE(QMGRMAP) QMNAME(CLUSQM*) MCAUSER(CLUSUSR) ADDRESS(‘9.30.*’) “Our internal cluster doesn’t use SSL, but we must ensure only the correct queue managers can connect into the cluster” CSS: F
MQ High Availability: Multi-instance Queue Managers 1. Normal MQ MQ Execution Client Client network 192.168.0.1 192.168.0.2 Machine A Machine B QM1 QM1 Active can fail-over Standby instance instance QM1 networked storage Owns the queue manager data
Multi-instance Queue Managers 2. Disaster MQ MQ Strikes Client Client networkConnectionsbroken fromclients 192.168.0.1 192.168.0.2 Machine A Machine B QM1 QM1 Active locks freed Standby instance instance QM1 networked storage
Multi-instance Queue Managers 3. Standby MQ MQ ConnectionsComes to Life Client Client still broken network 192.168.0.2 Machine B QM1 Active instance QM1 networked storage Owns the queue manager data
Multi-instance Queue Managers4. Recovery MQ MQ Complete Client Client Clients reconnected. Processing continues. network 192.168.0.2 Machine B QM1 Active instance QM1 networked storage Owns the queue manager data
Multi-instance queue managers: How it looks As a graphical example, SupportPac MS0P V7.0.1
Multi-instance queue managers: How it looks Enhanced dspmq New option for dspmq to output English-only text Useful for programmable parsing$ hostnamerockall$ dspmq -xQMNAME(V7) STATUS(Running) INSTANCE(rockall) MODE(Active)QMNAME(V7B) STATUS(Running) INSTANCE(rockall) MODE(Active)QMNAME(V7C) STATUS(Running as standby) INSTANCE(llareggub) MODE(Active) INSTANCE(rockall) MODE(Standby)
Message Broker H.A. using MQ 7.0.1 multi instance queue managers Message Broker exploits MQ 7.0.1 multi-instance queue manager capability Active and stand-by queue managers Start multiple instances of a queue manager on different machines One is “active” instance; other is “standby” instance Shared data is held in shared networked storage but owned by active instance Exploitation by Message Broker If standby instance of the queue manager becomes active, then the newly active MQ instance will start message broker once MQ recovery is complete
Automatic Client Reconnection Client library provides necessary reconnection logic on detection of a failure Hides failure from application code QM1 Application QM2 MQ Client QM3
Automatic Client Reconnection Tries to hide queue manager failures by restoring current state automatically For example, if MQPUT returns error, client reruns MQCONN/MQOPEN/MQPUT internally Uses the list of addresses in CONNAME to find queue manager MQSERVER environment variable also understands list MQSERVER=SYSTEM.DEF.SVRCONN/TCP/host1(1414),host2(1414) Can reconnect to the same or different Queue Manager Re-opens queues and other qmgr objects, re-establishes subscriptions Reconnection interval is backed off exponentially on each unsuccessful retry Total timeout is configurable – default 30 minutes.
Automatic Client Reconnection: Details Enabled in application code or ini file Event Handler callback shows reconnection is happening if app cares Good For Debugging If callback occurs may decide on special handling for following 3 cases. 1. Not all MQI is seamless, but majority repaired transparently • eg a browse cursor would revert to the top of the queue, non-persistent messages will have been lost during restart, non-durable subscriptions may miss some messages, in-flight transactions backed out, hObj values maintained 2. Some MQI options will fail if you have reconnection enabled • Using MQGMO_LOGICAL_ORDER, MQGET gives MQRC_RECONNECT_INCOMPATIBLE 3. Tries to keep dynamic queues with same name • So replies may not be missed Initially just in MQI and JMS – not the other OO classes Requires both client and server to be V7.0.1 level with SHARECNV>0 Server can be z/OS
Resources IBM Page: http://www.ibm.com/webspheremq/filetransfer Getting Started • http://ow.ly/uO9e Blogs: http://cumbers.wordpress.com/tag/wmqfte/ Twitter http://www.twitter.com/ibm_wmq Support Pacs http://www- 01.ibm.com/support/docview.wss?rs=171&uid=swg27007197