Messaging architectures in any environment, from local standalone deployments through to public clouds, must provide the highest reliability yet maximize their performance. This session gives you an insight into IBM MQ and how applications can be made to perform to their absolute best while maintaining the data integrity that IBM MQ is renowned for. We'll see how this can be achieved through a combination of good application design, system tuning and architectural patterns.
Keeping track of your messaging infrastructure is essential to be able to rapidly respond to your changing business demands whilst complying with your constantly changing audit demands. This session will provide illustrations and insight on the techniques and capabilities available in IBM MQ to help you meet these monitoring and auditing demands.
HHM-2833: Where is My Message?: Using IBM MQ Tools to Work Out What Applicati...Matt Leming
Every MQ infrastructure team member has been asked this question, and most developers who have worked with MQ have asked it:
"Where is my message?" In this session, we look into the tools that MQ provides to find your messages.
We demonstrate how to analyze the MQ recovery log on distributed platforms to find out what happened to your persistent messages,
with the assistance of a new tool. We also look at how to trace the route messages take through your MQ infrastructure, and how to generate
and analyze activity reports showing the behavior of MQ applications.
HHM 6887 Managing Your Scalable Applications in an MQ Hybrid Cloud Worldmatthew1001
This document provides an overview of IBM MQ sessions at the InterConnect 2017 conference, including:
- A schedule of MQ sessions occurring each day of the conference across topics like the MQ Appliance, hybrid cloud, and security.
- A brief description of each session including titles and speakers.
- A notice that the information provided is subject to change and limitations on warranties and liability.
IBM MQ Version 8.0.0.4 includes new features such as message expiry caps, event formatting samples, command and configuration events for security changes, and integration of MQLight. It also improves SSL/TLS configuration verification and provides a relocatable MQ client. Future plans include more frequent delivery of incremental changes.
This document discusses IBM MQ clustering and how it provides availability, scalability, and workload balancing. It describes how to set up a basic two node cluster with full repository queue managers and cluster receiver and sender channels. It then discusses more advanced clustering capabilities like supporting multiple applications and global deployments spanning different regions.
IBM MQ - better application performanceMarkTaylorIBM
Presented in Feb 2015 at Interconnect
This presentation is aimed at helping application developers understand how to best use MQ features for higher performance.
Monitoring and problem determination of your mq distributed systemsmatthew1001
The document discusses various tools for monitoring MQ distributed systems over time, including accounting and statistics messages, and system topics. Accounting and statistics messages provide overlapping information and are written to dedicated queues when enabled. System topics do not require enabling and provide more granular data on MQ resources as well as broader system metrics, making them better for multiple concurrent consumers.
Keeping track of your messaging infrastructure is essential to be able to rapidly respond to your changing business demands whilst complying with your constantly changing audit demands. This session will provide illustrations and insight on the techniques and capabilities available in IBM MQ to help you meet these monitoring and auditing demands.
HHM-2833: Where is My Message?: Using IBM MQ Tools to Work Out What Applicati...Matt Leming
Every MQ infrastructure team member has been asked this question, and most developers who have worked with MQ have asked it:
"Where is my message?" In this session, we look into the tools that MQ provides to find your messages.
We demonstrate how to analyze the MQ recovery log on distributed platforms to find out what happened to your persistent messages,
with the assistance of a new tool. We also look at how to trace the route messages take through your MQ infrastructure, and how to generate
and analyze activity reports showing the behavior of MQ applications.
HHM 6887 Managing Your Scalable Applications in an MQ Hybrid Cloud Worldmatthew1001
This document provides an overview of IBM MQ sessions at the InterConnect 2017 conference, including:
- A schedule of MQ sessions occurring each day of the conference across topics like the MQ Appliance, hybrid cloud, and security.
- A brief description of each session including titles and speakers.
- A notice that the information provided is subject to change and limitations on warranties and liability.
IBM MQ Version 8.0.0.4 includes new features such as message expiry caps, event formatting samples, command and configuration events for security changes, and integration of MQLight. It also improves SSL/TLS configuration verification and provides a relocatable MQ client. Future plans include more frequent delivery of incremental changes.
This document discusses IBM MQ clustering and how it provides availability, scalability, and workload balancing. It describes how to set up a basic two node cluster with full repository queue managers and cluster receiver and sender channels. It then discusses more advanced clustering capabilities like supporting multiple applications and global deployments spanning different regions.
IBM MQ - better application performanceMarkTaylorIBM
Presented in Feb 2015 at Interconnect
This presentation is aimed at helping application developers understand how to best use MQ features for higher performance.
Monitoring and problem determination of your mq distributed systemsmatthew1001
The document discusses various tools for monitoring MQ distributed systems over time, including accounting and statistics messages, and system topics. Accounting and statistics messages provide overlapping information and are written to dedicated queues when enabled. System topics do not require enabling and provide more granular data on MQ resources as well as broader system metrics, making them better for multiple concurrent consumers.
Presentation from Robin Wiley.
DataPower can connect to MQ as a client, sending and receiving messages to and from queues. Unlike traditional MQ client programming, the DataPower client interface has certain predefined functions that limit the scope of activities that can be performed. This session will explain what you can and can't do when integrating DataPower into your MQ infrastructure. Topics covered will include syncpoint processing, message grouping, message properties, JMS considerations, connec-tion options, and many more.
The document discusses the benefits of meditation for reducing stress and anxiety. Regular meditation practice can help calm the mind and body by lowering heart rate and blood pressure. Studies have shown that meditating for just 10-20 minutes per day can have significant positive impacts on both mental and physical health over time.
IBM MQ for z/OS The Latest and Greatest EnhancementsPete Siddall
IBM MQ for z/OS version 9.0 includes several new features and enhancements:
- Pageset statistics collected in SMF records allow for improved capacity planning, problem resolution, and system management related to pageset storage usage.
- Additional active log capacity of up to 310 x 4GB active logs improves resilience to issues affecting log archiving.
- Updated support for Unicode 8.0 includes all defined characters for improved globalization.
- Sample z/OS Management Facility workflows provide simplified provisioning and management of MQ resources.
- Performance enhancements in Advanced Message Security reduce encryption/decryption costs for streaming messages.
- The z/OS Connect service provider allows existing MQ-based services to be
1) The patches add support for managing CPU idle states using the generic PM domain framework and runtime PM. This provides a unified approach for idle management across all devices.
2) Key aspects include extending PM domains to support multiple idle levels, initializing CPU PM domains from device tree, and adding a governor to determine idle states based on wakeup times and QoS.
3) The changes allow the Linux kernel to directly control CPU and cluster idle states when firmware supports the OS-initiated suspend mode in the PSCI standard.
HHM-3481: IBM MQ for z/OS: Enhancing Application and Messaging Connectivity ...Matt Leming
Today's business environment is driving re-engineering of business systems using new techniques and architectures. This session will elaborate on the new environments MQ for z/OS is enabling for Java JMS applications, and enhanced interconnectivity capabilities that put MQ for z/OS at the vital heart of a hybrid messaging world.
Improving Kafka at-least-once performance at UberYing Zheng
At Uber, we are seeing an increasing demand for Kafka at-least-once delivery (asks=all). So far, we are running a dedicated at-least-once Kafka cluster with special settings. With a very low workload, the dedicated at-least-once cluster has been working well for more than a year. When trying to allow at-least-once producing on the regular Kafka clusters, the producing performance was the main concern. We spent some effort on this issue in the recent months, and managed to reduce at-least-once producer latency by about 80% with code changes and configuration tuning. When acks=0, these improvements also help increasing Kafka throughput and reducing Kafka end-to-end latency.
1) Sockets provide an interface between an application process and the transport layer to allow communication between processes locally or remotely. The main types of sockets are TCP, UDP, and raw sockets.
2) Socket programming in Java uses I/O streams, readers/writers, and socket classes like ServerSocket and Socket to create TCP and UDP sockets for communication. TCP sockets use streams for reliable connection-oriented communication while UDP uses datagram packets for unreliable connectionless communication.
3) The document discusses how two friends could use TCP or UDP sockets in Java to chat between their computers after their messaging apps were uninstalled, providing examples of one-way and two-way communication implementations.
This document provides guidelines for capturing and formatting test content for popular applications to be used on the Mu Dynamics test platform. It describes how to capture packet capture (PCAP) files using Wireshark for non-HTTP applications, and HTTP Archive (HAR) files using Firebug for HTTP-based applications. The steps include installing the necessary software, capturing representative application traffic, filtering the captures, generating scenarios in the Mu platform, and validating the scenarios. Standards are also defined for naming, formatting and describing the scenario files, JSON metadata files and PCAP/HAR captures to ensure consistency.
IBM WebSphere MQ: Managing Workloads, Scaling and Availability with MQ ClustersDavid Ware
IBM WebSphere MQ Clustering can be used to solve many problems, from simplified administration and workload management in an MQ network, to horizontal scalability and continuous availability of messaging applications. This session will show the full range of uses of MQ Clusters to solve real problems, highlighting the underlying technology being used.
This has been superseded by http://www.slideshare.net/DavidWare1/ame-2273-mq-clustering-pdf
The talk first walks over some of the security issues of the older versions of the SSL/TLS protocol. Then It introduces the upcomint TLS 1.3 version, presenting its new features and adoption status.
Training Slides: 104 - Basics - Working With Command Line ToolsContinuent
This 62min training session takes an in-depth look at the command line tools used in conjunction with Tungsten Clustering.
TOPICS COVERED
- Re-cap the previous Installation
- Explore the main Command Line Tools
- tpm
- cctrl
- trepctl
- thl
Intro to Packet Analysis - pfSense Hangout May 2014Netgate
This document summarizes an introduction to packet analysis presentation. It covers the basics of analyzing traffic at layers 2 through 4, including MAC addresses, IP addresses, TCP and UDP ports and protocols. Specific examples are provided of capturing TCP and UDP traffic in different states like established connections, rejected attempts and responses. ICMP traffic analysis is also introduced through ping examples. Methods for capturing traffic are demonstrated including tcpdump commands and filters. Potential uses of packet analysis like troubleshooting bandwidth usage, VPNs, port forwarding and routing are listed. The presentation ends with case studies on analyzing a DDoS bot and a TCP window size of 0 issue.
MaxScale uses an asynchronous and multi-threaded architecture to route client queries to backend database servers. Each thread creates its own epoll instance to monitor file descriptors for I/O events, avoiding locking between threads. Listening sockets are added to a global epoll file descriptor that notifies threads when clients connect, allowing connections to be distributed evenly across threads. This architecture improves performance over the previous single epoll instance approach.
IBM WebSphere MQ (MQ Series) is middleware for messaging and queuing. It uses the Message Queue Interface (MQI) to communicate with Message Queue Managers (MQMs) which manage queues and messages. MQ Series supports persistent and non-persistent messages up to 100 MB in size that can be segmented, grouped, sent to distribution lists, and assigned types like datagram, request, reply, or report. MQMs transfer messages between queues using channels and coordinate updates using two-phase commit.
Server Load Balancing on pfSense 2.4 - pfSense Hangout July 2017Netgate
- relayd and HAProxy can both be used for load balancing on pfSense, but HAProxy is more powerful, flexible, and reliable as a true proxy.
- An example setup demonstrated configuring relayd and HAProxy for load balancing multiple web servers, with HAProxy able to offload SSL and use ACLs to route requests based on hostname.
- Let's Encrypt certificates can be automated for HAProxy using a Lua script and the ACME package to validate via HTTP.
Connectivity Troubleshooting - pfSense Hangout June 2016Netgate
Connectivity troubleshooting hangout covered diagnostic tools like ping, traceroute, packet captures. Testing methodology was outlined starting from source and stopping at each hop. Common issues discussed included disappearing packets due to incorrect routes, firewall rules not matching, NAT/outbound rules blocking traffic. Subnetting mistakes and incorrect client configurations were also covered. The hangout provided tips for troubleshooting connectivity issues end-to-end from clients to firewall to internet, and checking firewall, DNS, and client settings. Other potential causes discussed were captive portals, IPsec tunnels, and misconfigured packages.
3425 - Using publish/subscribe to integrate applicationsTimothy McCormick
Publish/subscribe is the messaging model of choice for a wide range of messaging environments. Ranging from traditional MQ applications, applications running in frameworks like JEE and .NET or applications using the new MQ Light and open source AMQP capabilities, either on premise or in a Hybrid cloud. When MQ is at the heart of the solution all these environments can interoperate over a shared publish/subscribe infrastructure. This session will take you from the fundamentals of publish/subscribe through to how you can join these diverse systems together.
Apache Kafka is a distributed messaging system originally developed by LinkedIn to handle high volumes of log data with low latency. It allows for both online and offline data analysis and is highly scalable and efficient. Kafka uses a "pull model" where consumers pull messages from brokers in a distributed, fault-tolerant way coordinated by Zookeeper. Producers push messages to topics which are partitioned across brokers for scalability.
Presentation from Robin Wiley.
DataPower can connect to MQ as a client, sending and receiving messages to and from queues. Unlike traditional MQ client programming, the DataPower client interface has certain predefined functions that limit the scope of activities that can be performed. This session will explain what you can and can't do when integrating DataPower into your MQ infrastructure. Topics covered will include syncpoint processing, message grouping, message properties, JMS considerations, connec-tion options, and many more.
The document discusses the benefits of meditation for reducing stress and anxiety. Regular meditation practice can help calm the mind and body by lowering heart rate and blood pressure. Studies have shown that meditating for just 10-20 minutes per day can have significant positive impacts on both mental and physical health over time.
IBM MQ for z/OS The Latest and Greatest EnhancementsPete Siddall
IBM MQ for z/OS version 9.0 includes several new features and enhancements:
- Pageset statistics collected in SMF records allow for improved capacity planning, problem resolution, and system management related to pageset storage usage.
- Additional active log capacity of up to 310 x 4GB active logs improves resilience to issues affecting log archiving.
- Updated support for Unicode 8.0 includes all defined characters for improved globalization.
- Sample z/OS Management Facility workflows provide simplified provisioning and management of MQ resources.
- Performance enhancements in Advanced Message Security reduce encryption/decryption costs for streaming messages.
- The z/OS Connect service provider allows existing MQ-based services to be
1) The patches add support for managing CPU idle states using the generic PM domain framework and runtime PM. This provides a unified approach for idle management across all devices.
2) Key aspects include extending PM domains to support multiple idle levels, initializing CPU PM domains from device tree, and adding a governor to determine idle states based on wakeup times and QoS.
3) The changes allow the Linux kernel to directly control CPU and cluster idle states when firmware supports the OS-initiated suspend mode in the PSCI standard.
HHM-3481: IBM MQ for z/OS: Enhancing Application and Messaging Connectivity ...Matt Leming
Today's business environment is driving re-engineering of business systems using new techniques and architectures. This session will elaborate on the new environments MQ for z/OS is enabling for Java JMS applications, and enhanced interconnectivity capabilities that put MQ for z/OS at the vital heart of a hybrid messaging world.
Improving Kafka at-least-once performance at UberYing Zheng
At Uber, we are seeing an increasing demand for Kafka at-least-once delivery (asks=all). So far, we are running a dedicated at-least-once Kafka cluster with special settings. With a very low workload, the dedicated at-least-once cluster has been working well for more than a year. When trying to allow at-least-once producing on the regular Kafka clusters, the producing performance was the main concern. We spent some effort on this issue in the recent months, and managed to reduce at-least-once producer latency by about 80% with code changes and configuration tuning. When acks=0, these improvements also help increasing Kafka throughput and reducing Kafka end-to-end latency.
1) Sockets provide an interface between an application process and the transport layer to allow communication between processes locally or remotely. The main types of sockets are TCP, UDP, and raw sockets.
2) Socket programming in Java uses I/O streams, readers/writers, and socket classes like ServerSocket and Socket to create TCP and UDP sockets for communication. TCP sockets use streams for reliable connection-oriented communication while UDP uses datagram packets for unreliable connectionless communication.
3) The document discusses how two friends could use TCP or UDP sockets in Java to chat between their computers after their messaging apps were uninstalled, providing examples of one-way and two-way communication implementations.
This document provides guidelines for capturing and formatting test content for popular applications to be used on the Mu Dynamics test platform. It describes how to capture packet capture (PCAP) files using Wireshark for non-HTTP applications, and HTTP Archive (HAR) files using Firebug for HTTP-based applications. The steps include installing the necessary software, capturing representative application traffic, filtering the captures, generating scenarios in the Mu platform, and validating the scenarios. Standards are also defined for naming, formatting and describing the scenario files, JSON metadata files and PCAP/HAR captures to ensure consistency.
IBM WebSphere MQ: Managing Workloads, Scaling and Availability with MQ ClustersDavid Ware
IBM WebSphere MQ Clustering can be used to solve many problems, from simplified administration and workload management in an MQ network, to horizontal scalability and continuous availability of messaging applications. This session will show the full range of uses of MQ Clusters to solve real problems, highlighting the underlying technology being used.
This has been superseded by http://www.slideshare.net/DavidWare1/ame-2273-mq-clustering-pdf
The talk first walks over some of the security issues of the older versions of the SSL/TLS protocol. Then It introduces the upcomint TLS 1.3 version, presenting its new features and adoption status.
Training Slides: 104 - Basics - Working With Command Line ToolsContinuent
This 62min training session takes an in-depth look at the command line tools used in conjunction with Tungsten Clustering.
TOPICS COVERED
- Re-cap the previous Installation
- Explore the main Command Line Tools
- tpm
- cctrl
- trepctl
- thl
Intro to Packet Analysis - pfSense Hangout May 2014Netgate
This document summarizes an introduction to packet analysis presentation. It covers the basics of analyzing traffic at layers 2 through 4, including MAC addresses, IP addresses, TCP and UDP ports and protocols. Specific examples are provided of capturing TCP and UDP traffic in different states like established connections, rejected attempts and responses. ICMP traffic analysis is also introduced through ping examples. Methods for capturing traffic are demonstrated including tcpdump commands and filters. Potential uses of packet analysis like troubleshooting bandwidth usage, VPNs, port forwarding and routing are listed. The presentation ends with case studies on analyzing a DDoS bot and a TCP window size of 0 issue.
MaxScale uses an asynchronous and multi-threaded architecture to route client queries to backend database servers. Each thread creates its own epoll instance to monitor file descriptors for I/O events, avoiding locking between threads. Listening sockets are added to a global epoll file descriptor that notifies threads when clients connect, allowing connections to be distributed evenly across threads. This architecture improves performance over the previous single epoll instance approach.
IBM WebSphere MQ (MQ Series) is middleware for messaging and queuing. It uses the Message Queue Interface (MQI) to communicate with Message Queue Managers (MQMs) which manage queues and messages. MQ Series supports persistent and non-persistent messages up to 100 MB in size that can be segmented, grouped, sent to distribution lists, and assigned types like datagram, request, reply, or report. MQMs transfer messages between queues using channels and coordinate updates using two-phase commit.
Server Load Balancing on pfSense 2.4 - pfSense Hangout July 2017Netgate
- relayd and HAProxy can both be used for load balancing on pfSense, but HAProxy is more powerful, flexible, and reliable as a true proxy.
- An example setup demonstrated configuring relayd and HAProxy for load balancing multiple web servers, with HAProxy able to offload SSL and use ACLs to route requests based on hostname.
- Let's Encrypt certificates can be automated for HAProxy using a Lua script and the ACME package to validate via HTTP.
Connectivity Troubleshooting - pfSense Hangout June 2016Netgate
Connectivity troubleshooting hangout covered diagnostic tools like ping, traceroute, packet captures. Testing methodology was outlined starting from source and stopping at each hop. Common issues discussed included disappearing packets due to incorrect routes, firewall rules not matching, NAT/outbound rules blocking traffic. Subnetting mistakes and incorrect client configurations were also covered. The hangout provided tips for troubleshooting connectivity issues end-to-end from clients to firewall to internet, and checking firewall, DNS, and client settings. Other potential causes discussed were captive portals, IPsec tunnels, and misconfigured packages.
3425 - Using publish/subscribe to integrate applicationsTimothy McCormick
Publish/subscribe is the messaging model of choice for a wide range of messaging environments. Ranging from traditional MQ applications, applications running in frameworks like JEE and .NET or applications using the new MQ Light and open source AMQP capabilities, either on premise or in a Hybrid cloud. When MQ is at the heart of the solution all these environments can interoperate over a shared publish/subscribe infrastructure. This session will take you from the fundamentals of publish/subscribe through to how you can join these diverse systems together.
Apache Kafka is a distributed messaging system originally developed by LinkedIn to handle high volumes of log data with low latency. It allows for both online and offline data analysis and is highly scalable and efficient. Kafka uses a "pull model" where consumers pull messages from brokers in a distributed, fault-tolerant way coordinated by Zookeeper. Producers push messages to topics which are partitioned across brokers for scalability.
Introducing IBM Message Hub: Cloud-scale messaging based on Apache KafkaAndrew Schofield
IBM Message Hub is a new Bluemix service for messaging in the cloud. It's ideal for linking together microservices to build a scalable, flexible application in the cloud. It's great for feeding data at speed into other services such as analytics. You can also use it to bridge securely from your enterprise MQ systems into the cloud.
IBM Integration Bus and API Management can help enterprises unleash their systems by exposing them as APIs and managing API access. Integration Bus allows connecting different systems and exposing their data through REST APIs. These REST APIs can then be pushed to IBM API Management, which provides tools to securely publish, monitor, and manage access to APIs. The demo showed how a REST API created in Integration Bus can be pushed to API Management to publish it and create pricing plans for external access.
HHM-3540: The IBM MQ Light API: From Developer Laptop to Enterprise Data Cen...Matt Leming
The IBM MQ Light API makes it simple for developers to create responsive applications that are easy to scale without having to become messaging experts. Increasingly, development teams choose from a wide variety of languages, so the MQ Light API is available in a range of popular languages such as Ruby and Python, with the syntax tailored to fit naturally in each. The same API can be used with MQ Light installed on a laptop, with enterprise MQ queue managers, or in the cloud with the Message Hub service, so you can move seamlessly between these environments. Come and see how this API can make your developers more productive.
IBM Integration Bus v10 introduces several changes including simplified packaging and installation, built-in unit testing, and support for flexible MQ topologies where the queue manager is optional. The Web UI becomes the primary administration tool, and MQ dependencies are reduced but MQ remains fully supported. Docker container images are provided for flexible deployment.
Connectors for Integration provide ways to connect IBM Integration Bus to various endpoints and applications. They handle data exchange with external systems and applications, supporting various interaction styles and data volumes. Well-designed connectors understand the business objects and events of an endpoint, can provide event-driven access even if one is not directly available, leverage semantic data models, and have built-in error handling. Connectors also aim to offer efficient, scalable integration for event-driven and data synchronization scenarios, and have flexible deployment that is independent of other integration components.
IBM Messaging Security - Why securing your environment is important : IBM Int...Leif Davidsen
Presentation from IBM InterConnect 2016 . With growth in the number of business applications and exponential growth in connectivity between applications and systems, it is important to understand not just how to implement security, but why it is important to ensure all parts of the business can appreciate it and apply the right levels of security to their messaging system use. - jointly presented by Leif Davidsen and Rob Parker
IBM Integration Bus can be used to develop and deploy REST APIs. A REST API in Integration Bus is defined using the OpenAPI/Swagger specification. Operations in the REST API are implemented as subflows in Integration Bus. The REST API handles routing HTTP requests to the appropriate subflow. Parameters and JSON request bodies are mapped to the subflow message for processing. REST APIs are packaged and deployed like other Integration Bus artifacts.
IBM Integration Bus is a software product that provides integration capabilities for connecting applications, services, systems and devices. It uses a graphical interface to create reusable message flows that can transform and route messages between different platforms and data formats. The product provides extensive connectivity options, scalability, reliability and tools for development, testing and administration. A new IBM Integration Bus on Cloud service is also available, which provides a fully managed integration platform hosted in the cloud.
Java zone 2015 How to make life with kafka easier.Krzysztof Debski
You’ve just set up your Kafka cluster and now you are ready to process tens of thousands events per second. You decoupled your architecture and now all the communication goes via pubSub bus and you can focus only on providing business value. It would be great if that can be true. In real life you need to do a lot of tweaks to have your backbone ready to handle all the traffic you want.
Microservices: Where do they fit within a rapidly evolving integration archit...Kim Clark
Do microservices force us to look differently at the way we lay down and evolve our integration architecture, or are they purely about how we build applications? Are microservices a new concept, or an evolution of the many ideas that came before them? What is the relationship between microservices and other key initiatives such as APIs, SOA, and Agile. In this session, we will unpick what microservices really are, and indeed what they are not. We will consider whether there is something unique about this particular point time in technology that has enables microservice concepts to take hold. Finally, we will look at if, when, where and how an enterprise can take on the benefits of microservices, and what products and technologies are applicable for that journey.
Hands on guide to the nuts and bolts of administering an MQ Appliance and key differences from working with a software MQ installation. (Live presentation was accompanied by demonstration of the MQ Console WebUI capabilities - some screenshots included give a flavor).
Big Data Streams Architectures. Why? What? How?Anton Nazaruk
With a current zoo of technologies and different ways of their interaction it's a big challenge to architect a system (or adopt existed one) that will conform to low-latency BigData analysis requirements. Apache Kafka and Kappa Architecture in particular take more and more attention over classic Hadoop-centric technologies stack. New Consumer API put significant boost in this direction. Microservices-based streaming processing and new Kafka Streams tend to be a synergy in BigData world.
PHP At 5000 Requests Per Second: Hootsuite’s Scaling Storyvanphp
The document describes Hootsuite's scaling journey from using Apache and PHP on one MySQL server to a microservices architecture using multiple technologies like Nginx, PHP-FPM, Memcached, MongoDB, Gearman, and Scala/Akka services communicating via ZeroMQ. Key steps included caching with Memcached to reduce MySQL load, using Gearman for asynchronous tasks, and MongoDB for large datasets. Monitoring with Statsd, Logstash and Elasticsearch was added for visibility. They moved to a service-oriented architecture with independent services to keep scaling their large codebase and engineering team.
Hhm 3474 mq messaging technologies and support for high availability and acti...Pete Siddall
The document discusses concepts of business continuity including high availability, continuous serviceability, and continuous availability across sites. It then discusses how messaging technologies like IBM MQ can provide various levels of business continuity. Specifically, it provides examples of how MQ can enable active-active configurations across multiple sites for continuous availability through data synchronization and workload distribution. This allows no downtime even during planned or unplanned events.
Training Slides: Basics 102: Introduction to Tungsten ClusteringContinuent
This document provides an introduction to Continuent Tungsten clustering. It discusses key benefits like high availability, multi-site deployment, and ease of use. It examines the clustering architecture including topologies, automatic and manual failover, and rolling maintenance procedures. Commands for monitoring and managing the cluster are also reviewed, including cctrl and tpm diag. A demo shows using cctrl to perform a manual failover by promoting a slave to master.
This document discusses Apache Kudu, an open source column-oriented storage system that provides fast analytics on fast data. It describes Kudu's design goals of high throughput for large scans, low latency for short accesses, and database-like semantics. The document outlines Kudu's architecture, including its use of columnar storage, replication for fault tolerance, and integrations with Spark, Impala and other frameworks. It provides examples of using Kudu for IoT and real-time analytics use cases. Performance comparisons show Kudu outperforming other NoSQL systems on analytics and operational workloads.
IBM MQ High Availabillity and Disaster Recovery (2017 version)MarkTaylorIBM
This document discusses high availability and disaster recovery strategies for IBM MQ. It describes technologies like queue manager clusters, multi-instance queue managers, and HA clusters that can be used to provide high availability when failures occur across datacenters and clouds. Multi-instance queue managers provide basic failover of a queue manager between two systems without an HA cluster. HA clusters coordinate failover of resources like the queue manager, shared storage, and IP address across multiple machines for increased reliability. The IBM MQ Appliance also supports high availability between two appliances.
Slash n: Technical Session 2 - Messaging as a Platform - Shashwat Agarwal, V...slashn
This document discusses messaging as a platform (MaaP) for distributed processing of data. MaaP uses messaging services as pipes to connect producer and consumer processes. Messaging services provide capacity buffering, transactions, and reliability. Message brokers dynamically route messages and enable consumers to join or leave processing flexibly. Periodic batching dumps messages to batch processors like HDFS. Notifications then push results back to event-based processing. A process manager distributes process binaries across hosts. MaaP is well-suited for applications like log processing, product data aggregation, and connecting existing solutions at Flipkart.
Software Architecture for Cloud InfrastructureTapio Rautonen
The document discusses software architecture principles for cloud infrastructure, including microservices, distributed computing fallacies, designing for failure, and new design patterns like cache-aside, circuit breaker, and event sourcing. It also covers topics like autoscaling, asynchronous messaging, reactive streams, configuration management, and challenges like software erosion and failures cascading in distributed systems. The overall message is that building distributed systems on cloud infrastructure requires adopting new architectural patterns to deal with failures and improve scalability, performance and resilience.
Apache Big Data 2016: Next Gen Big Data Analytics with Apache ApexApache Apex
Apache Apex is a next gen big data analytics platform. Originally developed at DataTorrent it comes with a powerful stream processing engine, rich set of functional building blocks and an easy to use API for the developer to build real-time and batch applications. Apex runs natively on YARN and HDFS and is used in production in various industries. You will learn about the Apex architecture, including its unique features for scalability, fault tolerance and processing guarantees, programming model and use cases.
http://apachebigdata2016.sched.org/event/6M0L/next-gen-big-data-analytics-with-apache-apex-thomas-weise-datatorrent
Flexible and Real-Time Stream Processing with Apache FlinkDataWorks Summit
This document provides an overview of stream processing with Apache Flink. It discusses the rise of stream processing and how it enables low-latency applications and real-time analysis. It then describes Flink's stream processing capabilities, including pipelining of data, fault tolerance through checkpointing and recovery, and integration with batch processing. The document also summarizes Flink's programming model, state management, and roadmap for further development.
Maria DB Galera Cluster for High AvailabilityOSSCube
Want to understand how to set high availability solutions for MySQL using MariaDB Galera Cluster? Join this webinar, and learn from experts. During this webinar, you will also get guidance on how to implement MariaDB Galera Cluster.
This document discusses configuring and implementing a MariaDB Galera cluster for high availability on 3 Ubuntu servers. It provides steps to install MariaDB with Galera patches, configure the basic Galera settings, and start the cluster across the nodes. Key aspects covered include state transfers methods, Galera architecture, and important status variables for monitoring the cluster.
The need for gleaning answers from unbounded data streams is moving from nicety to a necessity. Netflix is a data driven company, and has a need to process over 1 trillion events a day amounting to 3 PB of data to derive business insights.
To ease extracting insight, we are building a self-serve, scalable, fault-tolerant, multi-tenant "Stream Processing as a Service" platform so the user can focus on data analysis. I'll share our experience using Flink to help build the platform.
AWS re:Invent presentation: Unmeltable Infrastructure at Scale by Loggly SolarWinds Loggly
This document summarizes Loggly's transition from their first generation log management infrastructure to their second generation infrastructure built on Apache Kafka, Twitter Storm, and ElasticSearch on AWS. The first generation faced challenges around tightly coupling event ingestion and indexing. The new system uses Kafka as a persistent queue, Storm for real-time event processing, and ElasticSearch for search and storage. This architecture leverages AWS services like auto-scaling and provisioned IOPS for high availability and scale. The new system provides improved elasticity, multi-tenancy, and a pre-production staging environment.
Citi Tech Talk: Monitoring and Performanceconfluent
The objective of the engagement is for Citi to have an understanding and path forward to monitor their Confluent Platform and
- Platform Monitoring
- Maintenance and Upgrade
Intro to Apache Apex - Next Gen Platform for Ingest and TransformApache Apex
Introduction to Apache Apex - The next generation native Hadoop platform. This talk will cover details about how Apache Apex can be used as a powerful and versatile platform for big data processing. Common usage of Apache Apex includes big data ingestion, streaming analytics, ETL, fast batch alerts, real-time actions, threat detection, etc.
Bio:
Pramod Immaneni is Apache Apex PMC member and senior architect at DataTorrent, where he works on Apache Apex and specializes in big data platform and applications. Prior to DataTorrent, he was a co-founder and CTO of Leaf Networks LLC, eventually acquired by Netgear Inc, where he built products in core networking space and was granted patents in peer-to-peer VPNs.
The document provides an overview of Confluent Control Center and how it can be used to monitor Apache Kafka deployments. It discusses how Control Center provides visibility into key metrics for brokers, topics, consumers and connectors. It also describes how Control Center helps answer important business questions about whether applications are receiving all data, showing the latest data, if the applications or cluster need to scale, and ensures data is not lost. Control Center provides dashboards, alerts and visibility to help operators effectively manage Kafka clusters and identify and address issues.
The document outlines best practices for optimizing performance in the different layers of Oracle E-Business Suite, including the applications tier, concurrent manager, database tier, and applications. It discusses approaches to identifying and resolving performance issues such as defining the problem clearly, gathering the right data to analyze the issue, identifying the root cause, and searching for known solutions. Specific techniques are provided for tuning each layer to maximize throughput and response times.
Apache Kafka - Scalable Message-Processing and more !Guido Schmutz
ndependent of the source of data, the integration of event streams into an Enterprise Architecture gets more and more important in the world of sensors, social media streams and Internet of Things. Events have to be accepted quickly and reliably, they have to be distributed and analysed, often with many consumers or systems interested in all or part of the events. How can me make sure that all these event are accepted and forwarded in an efficient and reliable way? This is where Apache Kafaka comes into play, a distirbuted, highly-scalable messaging broker, build for exchanging huge amount of messages between a source and a target.
This session will start with an introduction into Apache and presents the role of Apache Kafka in a modern data / information architecture and the advantages it brings to the table. Additionally the Kafka ecosystem will be covered as well as the integration of Kafka in the Oracle Stack, with products such as Golden Gate, Service Bus and Oracle Stream Analytics all being able to act as a Kafka consumer or producer.
Similar to 3450 - Writing and optimising applications for performance in a hybrid messaging world (20)
DDS Security Version 1.2 was adopted in 2024. This revision strengthens support for long runnings systems adding new cryptographic algorithms, certificate revocation, and hardness against DoS attacks.
GraphSummit Paris - The art of the possible with Graph TechnologyNeo4j
Sudhir Hasbe, Chief Product Officer, Neo4j
Join us as we explore breakthrough innovations enabled by interconnected data and AI. Discover firsthand how organizations use relationships in data to uncover contextual insights and solve our most pressing challenges – from optimizing supply chains, detecting fraud, and improving customer experiences to accelerating drug discoveries.
Enterprise Resource Planning System includes various modules that reduce any business's workload. Additionally, it organizes the workflows, which drives towards enhancing productivity. Here are a detailed explanation of the ERP modules. Going through the points will help you understand how the software is changing the work dynamics.
To know more details here: https://blogs.nyggs.com/nyggs/enterprise-resource-planning-erp-system-modules/
E-commerce Application Development Company.pdfHornet Dynamics
Your business can reach new heights with our assistance as we design solutions that are specifically appropriate for your goals and vision. Our eCommerce application solutions can digitally coordinate all retail operations processes to meet the demands of the marketplace while maintaining business continuity.
Graspan: A Big Data System for Big Code AnalysisAftab Hussain
We built a disk-based parallel graph system, Graspan, that uses a novel edge-pair centric computation model to compute dynamic transitive closures on very large program graphs.
We implement context-sensitive pointer/alias and dataflow analyses on Graspan. An evaluation of these analyses on large codebases such as Linux shows that their Graspan implementations scale to millions of lines of code and are much simpler than their original implementations.
These analyses were used to augment the existing checkers; these augmented checkers found 132 new NULL pointer bugs and 1308 unnecessary NULL tests in Linux 4.4.0-rc5, PostgreSQL 8.3.9, and Apache httpd 2.2.18.
- Accepted in ASPLOS ‘17, Xi’an, China.
- Featured in the tutorial, Systemized Program Analyses: A Big Data Perspective on Static Analysis Scalability, ASPLOS ‘17.
- Invited for presentation at SoCal PLS ‘16.
- Invited for poster presentation at PLDI SRC ‘16.
Odoo ERP software
Odoo ERP software, a leading open-source software for Enterprise Resource Planning (ERP) and business management, has recently launched its latest version, Odoo 17 Community Edition. This update introduces a range of new features and enhancements designed to streamline business operations and support growth.
The Odoo Community serves as a cost-free edition within the Odoo suite of ERP systems. Tailored to accommodate the standard needs of business operations, it provides a robust platform suitable for organisations of different sizes and business sectors. Within the Odoo Community Edition, users can access a variety of essential features and services essential for managing day-to-day tasks efficiently.
This blog presents a detailed overview of the features available within the Odoo 17 Community edition, and the differences between Odoo 17 community and enterprise editions, aiming to equip you with the necessary information to make an informed decision about its suitability for your business.
Why Mobile App Regression Testing is Critical for Sustained Success_ A Detail...kalichargn70th171
A dynamic process unfolds in the intricate realm of software development, dedicated to crafting and sustaining products that effortlessly address user needs. Amidst vital stages like market analysis and requirement assessments, the heart of software development lies in the meticulous creation and upkeep of source code. Code alterations are inherent, challenging code quality, particularly under stringent deadlines.
Atelier - Innover avec l’IA Générative et les graphes de connaissancesNeo4j
Atelier - Innover avec l’IA Générative et les graphes de connaissances
Allez au-delà du battage médiatique autour de l’IA et découvrez des techniques pratiques pour utiliser l’IA de manière responsable à travers les données de votre organisation. Explorez comment utiliser les graphes de connaissances pour augmenter la précision, la transparence et la capacité d’explication dans les systèmes d’IA générative. Vous partirez avec une expérience pratique combinant les relations entre les données et les LLM pour apporter du contexte spécifique à votre domaine et améliorer votre raisonnement.
Amenez votre ordinateur portable et nous vous guiderons sur la mise en place de votre propre pile d’IA générative, en vous fournissant des exemples pratiques et codés pour démarrer en quelques minutes.
Flutter is a popular open source, cross-platform framework developed by Google. In this webinar we'll explore Flutter and its architecture, delve into the Flutter Embedder and Flutter’s Dart language, discover how to leverage Flutter for embedded device development, learn about Automotive Grade Linux (AGL) and its consortium and understand the rationale behind AGL's choice of Flutter for next-gen IVI systems. Don’t miss this opportunity to discover whether Flutter is right for your project.
Revolutionizing Visual Effects Mastering AI Face Swaps.pdfUndress Baby
The quest for the best AI face swap solution is marked by an amalgamation of technological prowess and artistic finesse, where cutting-edge algorithms seamlessly replace faces in images or videos with striking realism. Leveraging advanced deep learning techniques, the best AI face swap tools meticulously analyze facial features, lighting conditions, and expressions to execute flawless transformations, ensuring natural-looking results that blur the line between reality and illusion, captivating users with their ingenuity and sophistication.
Web:- https://undressbaby.com/
May Marketo Masterclass, London MUG May 22 2024.pdfAdele Miller
Can't make Adobe Summit in Vegas? No sweat because the EMEA Marketo Engage Champions are coming to London to share their Summit sessions, insights and more!
This is a MUG with a twist you don't want to miss.
Utilocate offers a comprehensive solution for locate ticket management by automating and streamlining the entire process. By integrating with Geospatial Information Systems (GIS), it provides accurate mapping and visualization of utility locations, enhancing decision-making and reducing the risk of errors. The system's advanced data analytics tools help identify trends, predict potential issues, and optimize resource allocation, making the locate ticket management process smarter and more efficient. Additionally, automated ticket management ensures consistency and reduces human error, while real-time notifications keep all relevant personnel informed and ready to respond promptly.
The system's ability to streamline workflows and automate ticket routing significantly reduces the time taken to process each ticket, making the process faster and more efficient. Mobile access allows field technicians to update ticket information on the go, ensuring that the latest information is always available and accelerating the locate process. Overall, Utilocate not only enhances the efficiency and accuracy of locate ticket management but also improves safety by minimizing the risk of utility damage through precise and timely locates.
Transform Your Communication with Cloud-Based IVR SolutionsTheSMSPoint
Discover the power of Cloud-Based IVR Solutions to streamline communication processes. Embrace scalability and cost-efficiency while enhancing customer experiences with features like automated call routing and voice recognition. Accessible from anywhere, these solutions integrate seamlessly with existing systems, providing real-time analytics for continuous improvement. Revolutionize your communication strategy today with Cloud-Based IVR Solutions. Learn more at: https://thesmspoint.com/channel/cloud-telephony
Transform Your Communication with Cloud-Based IVR Solutions
3450 - Writing and optimising applications for performance in a hybrid messaging world
1. 3450: Writing and optimising Applications for
performance in a hybrid messaging world
Tim McCormick
IBM MQ Service Architect
timmccor@uk.ibm.com
2. Agenda
• Some performance numbers
• Structure of the Queue Manager
• Function Walkthroughs
• Client Channels
• Logging and Recovery
3. Agenda
• Some performance numbers
• Structure of the Queue Manager
• Function Walkthroughs
• Client Channels
• Logging and Recovery
4. Performance Bottlenecks
• Business Systems are complex
– Often no single bottleneck limiting performance
– Performance can mean different things to different people
– Throughput
• Scalability
• Low resource usage
• Not only limited by physical resources
– Application design, such as parallelism can have major effect
• Performance Reports (from SupportPac site) show range of scenarios
RAM CPU NetworkI/O
Bottlenecks
8. Distributed Performance in V8
• Improvements to distributed queue manager scaling
– Improve efficiency
– Better exploitation of wider SMP machines
– Looking at customer-provided application patterns not just benchmarks
• Always happy to have more customer examples
• Multiplexed client performance
– Increase the performance of multiplexed client channels (SHRCONV > 0)
– Especially for SHRCONV=1
• Other areas that helped:
– Cache alignment for internal structures
– Extended 64-bit exploitation for locking primitives
– RFH2 handling, particularly for waiting-getter
• Fewer copies of data are needed
– Better compiler optimisations including feedback-directed optimisation
– Faster data conversion (especially for 1208)
• Many messages are in 1208 codepage
• Optimised handling when the queue manager needs to convert them
9. Agenda
• What is distributed WebSphere MQ?
• Structure of the Queue Manager
• Function Walkthroughs
• Client Channels
• Logging and Recovery
12. Agenda
• What is distributed WebSphere MQ?
• Structure of the Queue Manager
• Function Walkthroughs
• Client Channels
• Logging and Recovery
13. MQI Stub
Application
IPCC
Execution
Controller
IPCC
IPCC
DAP
Kernel
Agent
IPCC
Application (MQI Stub)
Verify parameters and handles
Construct a Connect message
Call API crossing exit
Send a message to the EC
Application (MQI Stub)
Receive the reply
Construct an IPCC message
Send the message to the agent
Application (MQI Stub)
Receive reply and call API crossing exit
Return HCONN
Execution Controller
Choose an agent or start a new one
Construct a reply IPCC message
Return reply to application
Agent
Check App's permission to connect
Allocate and assign agent resources
Send IPCC reply back to application
MQCONN
14. Performance Implications: Connection Binding
• Fastpath binding removes inter-process communications
– Implies that the application is 'trusted'
– MQCONNX option MQCNO_FASTPATH_BINDING
– Application failure can corrupt queue manager
• Primary benefit is for non-persistent message processing
– Use for MCAs, Broker
• - 30% CPU saving
MQ
Appl
Agent
Memory
Log
IPC
Standard
Binding
MQ
Appl
Agent
Memory
Log
FASTPATH
Binding
16. Application (MQI Stub)
Verify hConn
Call API crossing exit
Verify parameter addressability
Place parameters in an Open message
Send an IPCC message to the agent
MQI Stub
Application
IPCC
DAP
Kernel
Agent
IPCC
Agent
Application interface
Verify open parameters
Kernel
Verify operation validity
Resolve target – including cluster lookup
Check permissions on the queue
DAP
Load the queue ready for gets and puts if required
ƒ This is the part that can use system resources
Kernel
Generate handle to object for application
Generate responses and event messages
IPCC
Send reply back to application
Application (MQI Stub)
Receive reply
Call API crossing exit
Return HOBJ
MQOPEN of a queue
17. MQOPEN internal changes for V8
• The object catalog is the list of objects and a map to the real files
– Shows that queue 'X.Y' is stored in "/var/mqm/qmgrs/<qm>/queues/X!Y"
– dspmqfls command shows the map
– Updated whenever an object is created or deleted
• In MQ V7 there were just two locks associated with the object catalog
– A significant implication of this scheme was that an MQOPEN could not overlap with the
creation/deletion of a queue (or any other object)
– Ghost queues introduced in MQ V5.2 went some way to reducing the impact, particularly for TDQ’s, but
very significant serialization implications remained.
• MQ V8 uses finer grained locks
– The master mutex need not be owned while waiting for the object mutex.
• The most significant impact of this serialization change is that dynamic queues can now be
created and destroyed concurrently with MQOPEN activity
26
18. Performance Implications: Heavyweight MQI Calls
• MQCONN is a “heavy” operation
– Don’t let your application do lots of them
– Wrappers and OO interfaces can sometimes hide what’s really happening
– Lots of MQCONNs can drop throughput from 1000s Msgs/Sec to 10s Msgs/Sec
• MQOPEN is also ‘heavy’ compared to MQPUT/MQGET
– Depends on the type of queue and whether first use
• Loading pre-existing queue; creating dynamic queue
– It’s where we do the security check
• Try to cache queue handles if more than one message
– If you’re only putting one message consider using MQPUT1
• Particularly client bindings
• Try to avoid exclusive access to the Queue
– Makes it harder to scale the solution
• For example adding more instances of application
– Implies that reliance on message order is not required
• Partition the data to allow parallel processing?
19. Performance Implications: Persistence
• Log bandwidth is going to restrict throughput
– Put the log files on the fastest disks you have, separate from queue file
– Persistent messages are the main things requiring recovery after an outage
– Can significantly affect restart times
• Why use persistence?
– False assumption that persistence is for "important" data and nonpersistent for when you don't care
– The real reason for persistent messages is to reduce application complexity
– With persistent, apps do not need logic to detect and deal with lost messages
– If your app (or operational procedures) can detect and deal with lost messages, then you do not need to
use persistent messages
• Understand which apps use persistence
MQPUT Queue File
Log
20. Kernel
Verify operation validity
(Resolve cluster queue destination)
DAP
Reserve space for the message data
If (persistent message)
Write log records for the update
(Wait for log records to reach the disk if outside syncpoint)
Write the message to the queue file
Else (non-persistent)
If (space available in queue buffer)
Copy the message data into the buffer
Else
Write the message to the queue file without logging
Maintain queue statistics such as queue depth
Kernel
Generate responses and events, wakeup getters/drive async consumers
Serialised
MQPUT Walkthrough
Also check for "if waiting getter"
Tip!
Use persistent messages with
syncpoint
21. Put to a waiting getter
• MQPUT most efficient if there is getting application waiting
– Having multiple applications processing queue increases percentage
– May not appear ‘balanced’ – May keep one ‘hot’
• Only for out of syncpoint messages
– Both persistent and non-persistent
– If Persistent outside of syncpoint, think carefully about why using persistence
• Got message could be lost if crash before returning to the application!
• No queuing required
– Removes a lot of processing of placing the message onto the queue
• Significantly reduces CPU cost and improves throughput
– Lots of improvements on this area in newest releases (7.1, 7.5)
MQPUT MQGET
MQGET
MQGET
22. Kernel
Verify operation validity
Check message expiry
Wait for message if not available
DAP
Locate a message meeting the requested criteria including
current browse cursor position
priority
message id, correlation id, segment or group conditions
Copy data into the message buffer
If (persistent)
Write log record
(Wait for log record to reach the disk if outside syncpoint)
Move the browse cursor if required
Maintain queue statistics such as queue depth
Kernel
Generate responses and events
Serialised
MQGET Walkthrough
Tip!
In request reply model, get by correlid
more efficient than get by msgid
23. Kernel
Verify operation validity
DAP
Write log record to end transaction
Wait for this log record to reach the disk
Lock all queues touched in this transaction
For each queue
Make any changes to messages in the transaction visible
Unlock queue
Kernel
Generate responses and events
Wakeup Getters/Drive Async Consumers
MQCMIT Walkthrough
Deal with 2PC protocol if in an
XA transaction
24. Performance Implications: Syncpoint
• Do you need it?
– Yes, when a set of work needs to either all be performed, or all not performed
• Maximum Size of UOW can be limited
– QMGR MAXUMSGS parm
– Set to sensible value to avoid runaway applications
• Make sure you keep the size of your UOWs small
– Don’t forget to end the UOW
• Cheaper to process in syncpoint for persistent messages
– Up to a point, not huge UOWs
– Log not forced after every MQPUT/MQGET
• Useful even when only a single message inside syncpoint
– And running multiple parallel applications
25. Good Application Design - Summary
• Long-running connection
• Opens queues up-front
• Uses syncpoint for persistent operations
• No message affinities so multiple instances can run in parallel
26. Agenda
• What is distributed WebSphere MQ?
• Structure of the Queue Manager
• Function Walkthroughs
• Client Channels
• Logging and Recovery
• Multiple Installation Support
27. Clients from V7
• Many changes to the client protocols in V7
– All can improve performance
• Multiplexing or Shared Conversations
– For multi-threaded applications
– Several connections use the same socket
• Asynchronous Put
– For sending applications
• Read-ahead
– For receiving applications
• New threads inside application for full-duplex comms
– Sharecnv(0) – May be fast but no full-duplex so miss good functionality
– Sharecnv(1) – One socket per hconn, optimised in V8 - recommended value
– Sharecvn(10) – Shared socket for multiple conversations (default value)
30. Client Connection Performance.
• Lots of internal changes at V8
– Channel status table restructure
– Reduce cost of health checking (MQIBindType=FASTPATH)
– Keep a hash table of processes, each with a sub-list of threads.
– Reduce cost of creating/deleting TDQ’s
– Channel process pooling pre-start
• Net effect of all of these changes is significant improvement to the queue manager’s ability to
accept a large number of inbound connections in a relatively short time.
• In one test the time taken to attach 50,000 clients, with each client owning a TDQ, reduced from
over an hour at 7.0.1.6 to under 4 minutes at MQ V8.
58
31. Agenda
• What is distributed WebSphere MQ?
• Structure of the Queue Manager
• Function Walkthroughs
• Channels
• Logging and Recovery
• Multiple Installation Support
32. What's the point of logging?
• A log record is written for each persistent update
– The log record describes the update
• Optimisations to minimise serialisation points
• Write-Ahead Logging
– The log is always more up-to-date than the actual data
• Log is a sequential file
– Sequential I/O is much quicker than random
– Single point of writing rather than to individual object files
• Log and actual data are reconciled during strmqm
– Progress information displayed
• Point of consistency – Checkpoint
– Log control file: amqhlctl.lfh – in log directory
– Checkpoint amqalchk.fil – qmgr directory
– Backup queue managers from MQ V6
Myth!
Triple write integrity does not mean
we write all data 3 times!
33. Looking at logger performance
• Can extract internal information from a service tool
– Lots of MQ performance PMRs turn out to be disk-related. So recording was added to the internal state
• amqldmpa –c H –m <qmgr> -d 8 –n <count> -s <interval> -f <file>
– The amqldmpa program can dump lots of other internal information too
• Includes logger I/O activity
– WriteTimeMax shows maximum time (microseconds) to complete I/O
– WriteSizeMax shows largest (bytes) I/O
– Since qmgr started
• Maintains averages
– WriteSizeShort is short-term (64-sample) weighted average of recent writes
– WriteSizeLong is longer-term (1024-sample) weighted average
– Similarly for WriteTimeShort/Long
• From one PMR:
– WriteTimeMax = 59102377, WriteSizeMax=2097152
– So it has taken nearly 60 seconds to write 2MB
– Implies they need to talk to disk support team!
34. Summary – understand the applications
• Understand the application pattern
– Exclusivity, affinity
• Understand the individual application
– Many times have heard "it's using persistence" when it's not. Or vice versa.
• If you don't have access to application source, tools exist to help
– Application Activity Trace
– Accounting messages
– MQ Trace
37. Notices and Disclaimers Con’t.
67
Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not
tested those products in connection with this publication and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products.
Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. IBM does not warrant the quality of any third-party products, or the
ability of any such third-party products to interoperate with IBM’s products. IBM EXPRESSLY DISCLAIMS ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING BUT
NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.
The provision of the information contained h erein is not intended to, and does not, grant any right or license under any IBM patents, copyrights, trademarks or other intellectual
property right.
IBM, the IBM logo, ibm.com, Aspera®, Bluemix, Blueworks Live, CICS, Clearcase, Cognos®, DOORS®, Emptoris®, Enterprise Document Management System™, FASP®,
FileNet®, Global Business Services ®, Global Technology Services ®, IBM ExperienceOne™, IBM SmartCloud®, IBM Social Business®, Information on Demand, ILOG,
Maximo®, MQIntegrator®, MQSeries®, Netcool®, OMEGAMON, OpenPower, PureAnalytics™, PureApplication®, pureCluster™, PureCoverage®, PureData®,
PureExperience®, PureFlex®, pureQuery®, pureScale®, PureSystems®, QRadar®, Rational®, Rhapsody®, Smarter Commerce®, SoDA, SPSS, Sterling Commerce®,
StoredIQ, Tealeaf®, Tivoli®, Trusteer®, Unica®, urban{code}®, Watson, WebSphere®, Worklight®, X-Force® and System z® Z/OS, are trademarks of International Business
Machines Corporation, registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM
trademarks is available on the Web at "Copyright and trademark information" at: www.ibm.com/legal/copytrade.shtml.
38. Thank You
Your Feedback is Important!
Access the InterConnect 2016 Conference Attendee
Portal to complete your session surveys from your
smartphone,
laptop or conference kiosk.
39. Monday
10:30-11:30 3592 New MQ features
3452 Managing applications
12:00-13:00 2835 MQ on z/OS and Distributed
15:00-16:00 3470 Latest MQ z/OS features
2833 Where is my message?
3544 MQ Light in an MQ infrastructure
16:30-17:30 3573 Hybrid cloud messaging
2941 MQ Advanced
Tuesday
08:30-09:30 3540 The MQ Light API
12:00-13:00 3456 The IBM MQ Appliance
13:15-14:15 3499 Introducing Message Hub
3458 MQ Appliance administration
14:30-15:30 6432 MQ updates and futures (InnerCircle)
2849 Messaging feedback roundtable
16:00-17:00 3544 MQ Light in an MQ infrastructure
3513 MQ hands on lab
Wednesday
08:30-09:30 3602 Managing your MQ environment
12:00-13:00 3613 Designing MQ self service
6408 Hybrid messaging roadmap (InnerCircle)
13:15-14:00 3416 HA and DR with MQ
3433 Why secure your messaging?
15:45-16:30 3429 Securing MQ
2847 Meet the messaging experts
16:00-17:00 3508 MQ Light hands on lab
16:45-17:30 2275 Migrating to the IBM MQ Appliance
Thursday
08:30-09:15 3420 MQ Clustering
2931 Business agility with self service MQ
09:30-10:15 3479 MQ z/OS clusters and shared queue
3450 Optimising MQ applications
2849 Messaging feedback roundtable
10:30-11:15 3465 MQ Appliance high availability
3481 MQ z/OS messaging connectivity
11:30-12:15 3474 Active-active messaging
3537 Monitoring and managing MQ
3425 MQ publish/subscribe
Find us at the EXPO:
Hybrid Integration peds 65-68
Check out the Hybrid Messaging sub topic under the
Hybrid Integration topic for further customer and business
partner sessions
Hybrid Messaging from the IBM experts at InterConnect 2016
Sunday
14:30-15:30 6408 Hybrid messaging roadmap (InnerCircle)