Hello. Thanks for joining today’s webcast – we are going to be looking at the use of Messaging in today’s IT environment, and specifically at IBM MQ – what it can do for your infrastructure and the latest updates to it – in the shape of MQ V8 – a very recent announcement. I’m Leif Davidsen and I am the senior product manager for IBM Messaging, including the MQ family.
Every client we talk to today is looking to do more. Now in many cases a lot of it will be doing more with less – but always there is a need to do more – and at the heart of everything is connectivity. Connecting systems, connecting services – and essentially what they are looking to do is to connect sources of information. There are new information sources springing up everywhere – and it is these new sources of data that are changing the world – creating new data to bring value to businesses – but unless this data is moved from where it is created to where it can be consumed to add value to the business then it is simply a burden to the business. So what is really important is that our clients need to connect systems together, ensuring newly created data is delivered to where it can provide value – and that is likely to be as soon as possible – just when it is created – that will create timely business insight to deliver value to the business – and then all this new data is no longer a burden, but instead it is a valuable asset.
Let’s talk about the data itself – businesses are seeing massive growth in data – it might be online purchase information, driven by massive numbers of mobile consumers, it might be sensor data from more and more physical devices. It might simply be the move to live processing rather than batch, creating a new timely stream of information. All of this data is being captured – and the risk is that having gone to the effort of creating it – it then simply is written to disk in the hope it will be analysed later. But the growth of data is likely created more data than can ever be reviewed and acted on – once the business starts to fall behind the flow of data it will never catch up.
The time to process, analyse and act on data is as it is created – it must be moved to where it can be consumed – where it can provide the insight to the business. And given this data must have some value to the business, it needs to be treated as valuable data – with security and integrity – it might well be customer data – not something you want to expose – and of course customer data, captured in the course of a customer interaction gives a very short window to then offer a tailored response based on the customer data – to deliver real business value.
So – for all this data to be created and consumed – what needs to happen. Especially given that just as new sources of data spring up quickly – so do the systems that consume them – and these are also likely to change – grow – and even disappear. So what is the best way to move this data, to deliver the value, but without overly burdening the infrastructure and application teams. Because if we try to move the data in the application, through even simply custom coding, then it will slow the coders down, add complexity and errors, increases costs for development and maintenance. So although you might end up with a solution – it is a solution you will need to keep changing – and the coders better have catered for all the possible failure scenarios for the movement of data. Maybe it is simpler to just dump it to disk – maybe a file or database – but that will slow things down – won’t deliver it as it is created to an application and adds to the problem of data without adding value. A better solution is to use a messaging infrastructure – a common set of APIs for the application to use to package and move the data – data which can be deliver to consuming applications quickly, simply, reliably and securely – even based on filters, and distribution mechanisms like publish subscribe which can ensure the right information can be delivered to the right consumer – helping to reduce the burden of processing all the data – and delivering more data faster.
So in IBM, where we have been meeting our customer’s messaging needs for 20 years we are focusing on this space in 3 core ways. Enterprise Messaging, Application Messaging, and Mobile Messaging with M2M. I’ll just quickly explain these areas, and then the rest of the webcast will be looking at some of the new capabilities we are delivering So enterprise messaging is the core messaging that goes on in your business today – and will continue to do so – it keeps the lights on, if you will. Absolutely class leading – robust, persistent, scalable, secure on all our platforms, on all your platforms. Then there is the exciting and rapidly growing world of Mobile and M2M messaging. You will have heard about big data – and this is where a lot of it is coming from – smart devices, connecting into the business – providing instrumentation and insight – with the potential to massively change the world, and your business opportunities along with it. Then we have a focus on the developers – that’s both existing developers of enterprise messaging, and also now pushing to provide more support and capabilities for developers of new applications, for new use cases – ensuring they too can exploit the power of messaging
So let’s start from the top – what actually is IBM MQ – if you never heard of it – how could I explain it? – well it is the messaging you need, as you need it. As we have discussed, messaging helps keep your applications simple – keeping all the connectivity logic and associated error handling out of the applications. And it is an infrastructure able to scale from a simple low use application – up to billions of messages per day – on any platform from the mainframe to the mobile – and moving any data between any of those endpoints – not really mattering whether the data starts or ends in a application, service or even a file on the file system And because the data is valuable to the business we ensure that MQ is secure – both the messaging infrastructure itself, and the message contents being moved – moving the message once, and once only, able to participate in transactional updates to protect the integrity of the business, and through being a single data movement infrastructure it can help manage the business by providing a single place to see what it going on – both as it happens and as a historical record for audits. Combine that with built-in high availability and you see why it is the messaging you need – as you need it.
As we just covered MQ provides guaranteed, assured messaging that provides transactional integrity with the speed and security that any enterprise will need for mission critical applications. E.g., transactions at a bank, money movement in finance sector, retailer processing of payment card or purchase details, border security and immigration processing, moving data through factories and in and out of ERP systems MQ enables you to connect applications and services together with valuable qualities-of-service. Applications can exchange information without tying themselves up – just like email where people communication a-synchronously.
MQ has been doing all this for years – we have 10000+ customers who have built their businesses around it – and rely on it not just every day – but every second. It helps their businesses be more flexible, more reliable and more secure. Deployments on platforms like System z with its coupling facility helps customers to be robust and recover from failures without interrupting their processing – and we have hundreds if not thousands of partners and skilled practioners to support these deployments – as well as big teams in IBM enhancing the product and delivering major new updates – such as the new IBM MQ V8.
In april 2014 IBM announced our latest major new version of MQ – V8 to be available on all major platforms this quarter. As we have covered – by using messaging solutions like MQ, businesses can keep their applications simple, while delivering data from pretty much any source to any other endpoint, helping the business to streamline, cutting costs and time, and meeting the changing needs of the business. So although thousands of customers are already current with MQ – what can they expect in the new V8 release? Well MQ is broadly common across bother distributed and z/OS platforms but there are some distinctions – on distributed there are updates focused on helping customers to scale their workload deployments – supporting larger workloads, more messages, more queues, and doing that without increasing the tasks on the administration and operation team. Security is of course high on everyone’s agenda and there have been big changes there too – both to enhance security and to ensure it is easier to use and manage. And there are some new functions as well – in support of new standards like JMS2.0 – plus exciting news for customers on IBM I – with MQ AMS available on that platform for the first time. For z/OS it gets the benefit of a lot of the same updates but also some specific new features to really exploit the platform capabilities – helping deployments to scale better, perform better, and with easier administration, boosting reliability and robustness. And of course with z being critical to so many businesses, security again is featured with stronger SHA-2 encryption. Plus as a bonus customers on V8, as well as V7 can now connect MQ clients on other platforms directly to the z/OS MQ without the additional CAF charge – this will help customers choose how to connect and exploit the data movement and business needs far more flexibly.
So that was a very quick set of highlights – but if I am going to describe MQ V8 updates at a very high level – then how should I call out the key points? I would suggest as follows: grouping the improvements into 4 key themes Ease of use, scalability, security, and exploiting the platforms with standards currency. A couple of key bullets on each of those Ease of use – some improvements about getting started on new projects faster – making it simpler and quicker to start with – but then a lot of work has been done on the challenges of then growing those deployments in production to large scale. These operations tasks are now much simpler, and faster – helping to reduce the time and cost of operations Scalability – every release of MQ has seen some boost in scalability and this one has improvements in a number of areas – boost in vertical scaling – exploiting larger SMP machines – and a lot of very common tasks – like queue management have seen real improvements. But there is a big change for pub sub in clusters – a change to allow this to perform much better in this configuration. Security – stronger security everywhere – SHA2 encryption and the AMS offering everywhere. Plus the usability of security is improved – using userid authentication in LDAP or in OS definitions – multiple certificate authority support and more. Exploiting z/OS hardware – larger SMP machines, increasing commonality between platforms and the new feature of JMS 2.0 now supported
So those are all very interesting – but really why do you care – why are those improvements useful to you? Ease of use can really be a major benefit – as the day to day operation of systems can require high skills and consume valuable resources if it takes too long to do common tasks – a focus on improving these day to day tasks can show quick benefit – that then accumulates through the years Security benefits in today’s environment are becoming increasingly important – execs are focusing here due to constant exposures – so being secure and having the security being usable are key benefits Scalability – MQ is already pretty good – but we focus on benefits in specific scenarios – like pubsub – and pubsub in clusters as well as common operations will show goodness. Exploiting hardware is great news for the z community – as is AMS on I for a great cross system story – so hopefully good synergy between MQ and the platforms being deployed on – as well as keeping the product in line across the platforms
Now hopefully some of you on this call may know enough to be dangerous with MQ – and clearly there are a lot more details you can learn about the new features in MQ - here are some broken down into 4 themes again – there is a separate column for z exploitation rather than ease of use, as to be honest the outcome of so many of the individual feature improvements is ease of use – and there are also so many z specific enhancements. I will call out a few notable areas here – and then we will cover more in the next few slides. Performance is always something that catches the eye. As mentioned we have focussed on common tasks – some early measurements of opening and closing queues show these tasks can be up to 5 times faster in the new release. And persistent and non-persistent messaging is showing 25%-30% improvements for some deployments. And just to mention the changes to pub-sub in clusters – a big change here to allow publications and subscriptions to route just to nominated Queue Managers. This should see a major boost for these types of configurations.
Let’s drill down a little bit further into this release, looking mostly at the multiple changes for Security. Probably the biggest one is that MQ can now verify the userid and password against the operating system definitions. On z/OS this can include PassPhrase, and on distributed systems LDAP can be used. This makes admin a lot easier, and should help boost security, both in use and also by encouraging use of security. In 7.1 IBM made major security changes as well including adding CHLAUTH – this used IP address – but we had multiple requests to have this use DNS hostnames as well to simplify tasks when making machine or network changes – we now allow this – although we would still recommend IP address if possible. For when different certificate authorities are used for maybe connecting to a 3rd party or to another part of the business, previously a different Queue Manager was needed. But now, making things easier once again, we allow multiple certificate authorities to be defined in a Queue Manager. Also, MQ AMS is now available on all our major MQ platforms – I will cover this in more depth in a minute – but we are certainly happy it is now been added to the IBM I platform, and also on z/OS the code is now integrated into the core MQ runtime – meaning if you are entitled to use it it doesn’t need to be separately started. On scalability I have already covered a number of the performance improvements – but pub-sub there have been multiple changes – both in performance and in the way it works in clusters as I said previously – this should help you to consider using this in more areas. In standards – we now support the fairly new JMS 2.0 standard and have added additional functionality and extended our API to support this. And lastly on this chart – the multiple cluster transmit queues – which were added to most of the distributed platforms in MQ V7.5 have now been added to IBM I and z/OS – this provides better application isolation for different applications running in clusters and should help the throughput for these applications
64-bit buffer pools In IBM MQ V8 the customer is able to use 64-bit storage for their buffer pools. Workloads with large numbers of messages in queues can now keep these messages in memory instead of writing to pagesets, reducing CPU costs and elapsed time for MQGET. CPU costs for MQGET can be reduced by more than 50% when message is stored in buffer pools. Time taken to retrieve a 8KB message from buffers reduced by 97% compared to reading from a pageset on a low-utilisation DS8800 (Down from 620 microseconds to 17 microseconds)
Benefits: * Ease of administration with lower overheads to define and manage buffer pools * For deep queue workloads, potential for CPU reduction and reduction in elapsed time
Log capacity IBM MQ V8 improves the availability of the queue manager by increasing the period of time before the log needs to be reset. Before V800, the 6 byte log RBA could address up to 256 terabytes of data. V800 the log RBA can be 8 bytes long and the queue manager can now address over 64,000 times as much data (16 exabytes) before the log RBA needs to be reset.
This means that in a system that currently logs 256 TB of data annually (and currently resets the logs on an annual basis) but doubles the amount of data each year, using the 8-byte RBA would mean the logs would not need resetting for 17 years.
Benefits * Reduced outages in queue managers with high volume persistent messages workloads
Channel Compression The latest class of System z hardware introduces hardware compression via zEDC. IBM MQ v800 adds support for hardware compression via the ZLIBFAST channel compression option.
Channel compression typically is used on high latency / low band-width networks to reduce the amount of traffic being flowed but has the effect of increasing CPU costs per message. Using zEDC hardware compression can reduce the message costs by 40% compared to compression performed in software.
Benefits * Reduced cost of compressing messages prior to sending over channels
I will quickly also mention the Flash Express feature – this adds flash memory to the coupling facility, meaning more storage is available for meaningful data to be stored there in the case of a failure, allowing for faster recoverability. And for users of IBM Information replicator – there have been improvements for this use case, lowering CPU usage, and seeing better throughput.
I thought I would also cover the Client Attachment Feature on z/OS as we have made changes here. As I mentioned on the MQ VUE offering – there is no charge to connect MQ clients to the new MQ workload with VUE – but previously to connect MQ Clients to MQ for z/OS there has been a chargeable feature that needed to be installed to allow this. This type of thing can actually hinder take-up of this type of workload – and where we see new workload, we might well want to connect MQ Clients directly. Therefore in MQ V8 we actually removed the CAF – there is no longer a need for it to allow MQ Clients to connect to MQ on z/OS V8 – the CAF simply no longer exists for this release. However for V7 customers, we have also made a change. If you are deploying a new V7.0.1 or V7.1 system, then you can install a PTF which means that going forward you won’t need the CAF on the system to connect MQ Clients. If you have an existing MQ V7 system deployed on z/OS you can install the PTF to remove the CAF and the need for the CAF, but even without this you will see a cost benefit as IBM has set the price of the CAF to zero for MQ V7.0.1 and V7.1 customers with effect from June 1st. Note if you are still on V6 or earlier you continue to need the CAF and to pay for it.
Now we have covered some of the key areas. Lets try to position them in context of how you can get MQ in your organization. but that itself might refer to just the basic MQ messaging infrastructure – where as we have extensions to that which provide added value – offering managed file transfer, advanced message security, and support for MQTT with Telemetry. IBM provides all these separately but it is far more powerful with them packaged together as MQ Advanced – a singe part priced per PVU – clients buying this get entitlement to all the capabilities. Many customers find it is simpler for them to do this rather than try to judge whether they need all the capabilities on all the systems. And we look to start the enablement by providing the capabilities with our offering for developers – which is MQ Advanced for Developers – and it comes with all the functions and products – and thus it helps the developer to start using every feature in their project. Bear in mind as well as MQ Advanced and the capabilities you have on this chart there are other things you might need, like the Idle Standby licensing we have to enable the automatic High Availability for MQ – plus some of the ways you can deploy managed file transfer as Agents or Endpoints but you’ll hear more about them in a minute
First let’s focus on one of the hottest topics of the hour – security. Although MQ itself is designed to be safe, there is one aspect that can’t be addresses directly in the base product – MQ moves as a message exactly what is put in the message – so if unencrypted data is sent, then the message has the data exactly unencrypted. Now we do have SSL while it is moving – but MQ moves from Queue Manager to Queue Manager – and writes the message to a Queue – and at that point it is on a disk unencrypted – at risk. MQ AMS uses policies to determine which message contents need to be encrypted and then does so – between the sending and receiving application – no changes are needed to the application but the data is secure throughout the exchange. You can buy this separately to MQ – but we recommend MQ Advanced which includes this.
MQ Telemetry is the component MQ needs to be able to send and receive MQTT messages – these typically are clients on physical sensor devices or mobile devices – used for small footprint, low power consumption and low bandwidth messages. A growing set of use cases need this – and again the customer can either buy this separately, or if they buy MQ Advanced, then they are able to deploy MQ telemetry to all their MQ Servers – so this is an easy way to get interested in MQ Advanced if you want to start working with MQTT. A change that comes with MQ V8 is that where we previously had MQ Telemetry, and also MQ Telemetry Advanced – which included the MQ Telemetry Daemon for Devices, which allowed multiple local MQTT clients to aggregate messages and provided some persistence. In MQ V8 we have now rolled this into the standard MQ Telemetry offering, removing the separate higher priced MQ Telemetry Advanced offering. Note that MQ Telemetry is to enable MQTT clients to connect to MQ Servers. However some customers who have very large deployments of MQTT clients will be interested in the IBM MessageSight appliance, which is a dedicated appliance enabling connections of up to 1 million concurrent MQTT connections.
Managed File Transfer is like security a very hot topic – this is again something every business needs – after all so much of their data is today held, or indeed trapped in files – and they need to get it out to realize some value. MQ MFT sends and receives the contents of files as messages over the MQ infrastructure – the fundamental component is the MFT Agent – this the endpoint where a file can have its contents turned into a message to be moved and then re-written as a file providing the managed file transfer many customers need. The file is moved as a message, picked up from one file system and moved over MQ’s reliable and secure network, to another file system. However there are real new benefits to this – as we are no longer simply dealing with a file – but rather now with a MQ message – and the contents of that message. So you can move a file and have it be deliver to an application as a message, or even a group of messages, splitting the file up. And of course the other way round – you can take a message, or a group of messages and use them to create and deposit a file. Both of these options can be fundamentally in not simply moving files – but actually making use of the data in the file in a far more timely and dynamic way. In terms of what you need to provide this - we have the Agent packaged as the MFT Endpoint as one component, and then the MFT Service that extends the MQ Server – this one is both a separate component but is included as a part of MQ Advanced.
You might have seen mention on the previous chart that Sterling Control Center could be used for monitoring MQ MFT – what’s an exciting recent change for both MQ 7.5 and MQ V8 is that we have changed how we deliver our MQ MFT capability – rather than having to compare and contrast it with other MFT solutions such as IBM’s own Connect:Direct we have decided to bundle MQ MFT and Connect:Direct together as a single offering, along with a dedicated monitoring solution for both products called Control Center. Now although, as we have covered MQ MFT is far more than just a file movement tool – with file to message and message to file capabilities, and being built as a MQ offering it is ideal for MQ Users but this bundle helps us to sell IBM’s capability rather than needing to make a single choice for purchase. So the separate offering with MQ MFT is now called WebSphere MFT – and in MQ Advanced we have this WebSphere MFT bundle with MQ MFT, Control Center and Connect:Direct. This is also available as IBM Connect:Direct Advanced, depending on which IBM rep you deal with, but it is exactly the same product and package either way. Plus of course the complete same package is a part of MQ Advanced including both Connect:Direct and Control Center.
If you are an existing MQ customer then you very much need to know we have announced a EOS date for MQ V7.0.1 on distributed and also on z/OS – this is going to be September 30th 2015. You this gives you some time to plan ahead. But also see this as a way to start to think about some of the new features in V8 – or indeed some of the great features we added in V7.1 or V7.5 if you are still on 7.0.1 . Think about it – you can get your developers started today with the MQ Advanced for developers offering – which is free to download from developerworks today. And zOS will be available from 13th June – which is great news
So a quick summary – the growth of connectivity and data means the need for messaging is growing faster than ever. MQ has been evolving and growing with this need – and we covered the key themes of V8 that help here. And there are great new opportunities around MQ VUE on z/OS, with MQ AMS for end to end encryption and with better Managed File Transfer – including the new bundled capabilities.
IBM has continued to enhance MQ for more than 20 years – as you can see we have been adding new functions, capabilities and product deliverables throughout that time. Rest assured that now we have delivered V8 the team in Hursley are already busy working on the future, to help make IBM MQ and other messaging offerings better. You can have your say about what you want to see us do with the RFE system – where you can submit and vote on enhancement requests, and help us deliver the right product in the future as well as making the best of today’s products.
To get started you might want to read a summary of the changes in the V8 release – you can download a short paper highlighting some of the changes. But there is a lot going on in the wider community – we have an excellent site on developerworks – and many of our developers blog there about the details of the latest release – and you can always comment and contribute. One of the many valuable things you will find there is the MQ Advanced for developers V8 download – that’s right – you can download V8 for free for development use today. What’s stopping you – as soon as this webcast is finished of course….