Effective administration of IBM Integration Bus - Sanjay Nagchowdhury

InterConnect
2017
HHI-2151
Effective Administration of
IBM Integration Bus
Sanjay Nagchowdhury
IBM Integration Bus
sanjay_nagchowdhury@uk.ibm.com
1

Effective Administration…
Installation
Configuration
MonitoringDeployment
Migration
2
Security

Installation
Configuration
Deployment
Migration
3
Monitoring
Security

Installation
 Radically Simplified Packaging and Installation
– Full function, simple, single package install
– Developer Operating Systems contain Toolkit and Server
• Total size approx. 1.3 GB
– Server Operating Systems contain only server
 Other changes
– Full entitlement to MQ remains
– MQ no longer packaged
• Default queue manager for IB node for backwards
compatibility
 Built-in Unit Test Environment
– Developer tools have built-in unit test server
• Fixed name of TESTNODE_<userid>
– Started and stopped with tools
– Can still test / deploy to manually created local and remote servers 4
Single install package
for server and toolkit.
Simple unzip on unix.
Local integration node and
server automatically created
when you start toolkit.
tar –xzf…
New Tutorials Gallery.

Installation
Configuration
Deployment
Migration
5
Monitoring
Security

Configuration
TOOLKIT
• Connect to local or remote
Integration Nodes using toolkit
• Configure bar file properties.
• Deploy bar files
• Configure policy sets for web
services
CONSOLE
• New command console
• iib help, iib tools, etc
• Remote administration commands
need to point at the integration
node’s administration port instead of
queue manager.
Integration API
• Java interface for administering IIB
through Java code.
REST API
• Administer IIB resources using
operations defined on a REST
model.
6
WEB UI
• Primary graphical means of
administering IIB.
• Many new features in V10.
• Manage Integration
Servers
• Define policy
• Create configurable
service
• View Statistics
• Workload Management

7
Manage Integration Servers
• Start, stop
• View, Create, Rename, Delete
Manage deployed resources
• Start, stop,
• View, manage, delete
Start/stop statistics and accounting data
Start/stop resource statistics
Start/stop flow monitoring for publishing
events, record and replay and business
transaction monitoring
Configuration
NEW !

Loopback Request Node
• The LoopBack Request node provides a way for IIB
message flow authors to invoke synchronous CRUD
operations to external systems using LoopBack
Connectors.
• LoopBack provides numerous connectors to access
enterprise and other backend data systems such as:
• NoSQL databases.
• Relational databases.
Cloudant
8
NEW !

Windows, Linux x86
Integration Server
Node.js®
MongoDB
Connector
PostgreSQL
Connector
Cloudant
Connector
Non Database
Connectors
Community
Connectors
LoopBack
Cloudant
Data SourcesIntegration Node
MongoDB
9
Configuration
NEW ! New LoopBack Request node provides a
way for IIB message flows to invoke
synchronous CRUD operations to external
systems using LoopBack Connectors.
LoopBack is an open source Node.js
framework that can be extended by
downloading and installing connectors from
a large open source catalog using the ‘npm’
tool.
LoopBack provides connectors for:
• NoSQL databases e.g. MongoDB, IBM
Cloudant
• Relational databases e.g. PostgreSQL,
MySQL
• Other backend services e.g. REST,
SOAP

10
LoopBack Connectors
Contributed by
LoopBack
Community
http://loopback.io/doc/

11
Configuration
MQSI_WORKPATHconnectorsloopbackdatasources.json
MQSI_WORKPATHnode_modules
1
2
NEW !
3 mqsisetdbparms –n loopback::<secid> -u uid –p pwd
npm install loopback-connector-mongodb --save

Installation
Configuration
Deployment
Migration
12
Monitoring
Security

13
Toolkit
BAR files contain
configurable properties which
allow you to deploy your
applications from your QA
system to Production.
Web UI
Message node properties
are configurable.
Workload management
properties are configurable.

Deployment
14
When a REST API is deployed, the Swagger
document for that REST API is automatically
made available over HTTP from the same
server and port that the REST API is hosted in.
The deployed Swagger document is
automatically updated to reflect the server, port,
and HTTP/HTTPS details for the deployed
REST API. You do not have to update it with the
correct details before deployment.
The REST APIs can be pushed to API
Connect where you can secure them,
define Service Level Agreements and
examine visual analytics data.
NEW !

How many Integration Servers should I have?
How many Additional Instances should I add?
Additional Instances
• Results in more processing threads
• Low(er) memory requirement
• Thread level separation
• Can share data between threads
• Scales across multiple servers
Integration Servers
• Results in a new process/address-space
• Increased memory requirement
• Multiple threads including management
• Operational simplicity
• Gives process level separation
• Scales across multiple servers
Recommended Usage
• Check resource constraints on system
• How much memory available?
• How many CPUs?
• Start low (1 server, No additional instances)
• Group applications in a single integration server
• Assign heavy resource users to their own integration server
• Increment integration servers and additional instances one at a time
• Keep checking memory and CPU on machine
• Don’t assume configuration will work the same on different machines
• Different memory and number of CPUs 15
Deployment
Considerations Deployment
How many copies of a
message flow do I need ?
Each message flow is different…..
Each will use certain different level of resources
(CPU, memory, I/O) and have a particular
performance profile
In deciding number of copies to run need to know
1. Availability Requirements
2. Target throughput (messages/second) rate
3. Target response time
Number of copies of each message flow needed will
vary.
So do not decide the same number of copies are
sufficient in all cases.
Ultimately need to
balance resource
manageability &
availability.

Installation
Configuration
Deployment
Migration
16
Monitoring
Security

Monitoring
17
Workload management policy offers
dynamic control over:
• Message rate limits.
• Additional instances.
• Commit count & commit interval.
• Automatic restart in case of
unresponsive message flow.
View message flow and resource
statistics in the Web UI. Use this to
analyse bottlenecks in your flows.

18
IBM Cloud Product Insights is a Bluemix
service that is part of IBM Connect to Cloud.
Monitoring
NEW !

20
Different IBM products can be registered, so
you can create a cross-product inventory and
view product usage metrics across your
portfolio of IBM products in your solution.
Monitoring
If you click on Register a product, you can see
which IBM products are available for
registration.

21
You will see all products that are
available for registration.
If you click on any of these, you will
see instructions on how to register
your product.

22
You can see what is the minimum
supported version of IIB.
Links are provided to the product
web page and knowledge center.
Simple instructions on how to
activate the Bluemix reporting
feature for your Integration Node
and Integration Servers.
Click this to copy the
command and then fill
in the blanks!
apihost and apikey information is
on the Service credentials tab.
Restart your Integration Node
for the changes to take effect.

23
The Service Credentials section is pre-
populated with an apikey which has been
created on your behalf
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
You use the apihost and apikey when
activating the Bluemix reporting feature.

24
You can group your Integration
Nodes and Integration Servers into
groups. For example: Dev, QA,
Stage, Production
Usage data can be seen for a
group of Integration Servers or an
individual Integration Server.

25
Choose between:
CPU time,
Number of active CPUs
Resident Set Size
Usage Details Advisor
View Integration
Node details
View a list of
recommended
services from the
Bluemix catalog.

[sanjayn@sachin iib-10.0.0.8]$ mqsichangebluemixreporting LINUX_NODE -g -l active -r us-south -o
sanjay_nagchowdhury@uk.ibm.com -a "Sanjay Nagchowdhury" -u sanjay_nagchowdhury@uk.ibm.com -p XXXXX -d
Request Bluemix logging tokens: https://logmet.ng.bluemix.net/login
Successful HTTPStatus: 200
Publishing Kibana dashboard
Successful HTTPStatus: 200
Response: Your Kibana 4 dashboards can be accessed at the following location:
https://logmet.ng.bluemix.net/app/#/kibana4
The Kibana 4 dashboards that have been uploaded include: IBM Integration Bus Dashboard
BIP8377I: Current dashboard has been loaded into Kibana for Bluemix logging space 'Sanjay Nagchowdhury'. Dashboard
files are also prepared for manual import into Kibana from '/home/sanjayn/iibconfig/BluemixReporting/LINUX_NODE'.
BIP8071I: Successful command completion.
26
Reporting logging information to a Logmet service in IBM Bluemix
and displaying it in a Kibana dashboard
Run mqsichangebluemixreporting to
configure a connection to the Logmet service
in Bluemix and send it logging information.
Go to this URL in a web browser to see
logging infromation in a dashboard.
Command can be repeated for multiple
Integration Nodes and Integration Servers to see
logging information in a single dashboard.
NEW !

27
Example using Integration Nodes running
on Linux and Windows.
Top 5 hosts that are
generating log events
Top 5 integration
servers that are
generating log events
Top 5 log events
across the Integration
Servers for a timeslot.
All events by
severity.
Custom visual which shows
Integration Servers that are
stopping/starting/changing

28
This spike shows that a sudden increase in
error messages occurred.
You can zoom in and see precise BIP
messages that were generated in this
timeslot.

29
Chart after zooming in on the timeslot to
check the errors.
Bottom table shows the
logging messages that were
recorded from the different
Integration Servers.
You can see errors that
were generated.
You can see errors that
were generated.

Installation
Configuration
Deployment
Migration
30
Monitoring
Security

31
Security
Admin security is optional, it is not enabled by default
– Control user access to Integration Node resources
– Enable through mqsichangebroker / mqsicreatebroker
Administration Security
Authentication Authorisation
“You are who you say you are!” “But what are you allowed to do ?”
Credentials checked by….
Web user account
with a local password.
Local user
authenticated by OS
Integration Node LDAP Server
Web user account
with a password held
in the LDAP Server
Permissions checked using…
Queue-based authorisation File-based authorisation
SYSTEM.BROKER.AUTH
SYSTEM.BROKER.AUTH.EG
MQ INQ
MQ PUT
MQ SET
File Read
File Write
File Execute

33
Security
Set mode to file-based
authorisation
Add web user account
with a local password
Web UI now requires
userid/password
User can view the Integration
Server and its contents but
cannot start/stop/deploy
Set permission to view
Integration Server, but
not start/stop/deploy

34
Security
XXXXXX
Set the LDAP server that you want to use for authentication.
Configure Integration Node to
connect to the LDAP Server.
Add a web user account for a
user that is defined in the LDAP
Server.
Use a wildcard to add all users.
NEW !

Installation
Configuration
Deployment
Migration
35
Monitoring
Security

Supported Migration Paths
You can migrate to IBM Integration Bus Version 10.0 from the following previous versions:
• WebSphere® Message Broker Version 7.0.0.5
• WebSphere Message Broker Version 8.0
• IBM Integration Bus Version 9.0
36
Migration
Co-existence: IIB V10 can co-exist with IIB V9, WMB V8 and V7.
Windows :
• Multiple instances of each of the versions (but different fixpacks) can be installed on the same system.
UNIX and z/OS :
• Multiple instances of each of the versions and fixpacks can be installed on the same system.
Ensure correct mqsiprofile is sourced before starting the Integration nodes to pick up the correct versions.
Multiple versions of IIB Toolkit can be installed on the same machine
• You can not use IIB V10 Toolkit to connect to a V9 broker or vice versa

39
Migration
Migration Options
In-place migration:
• Migrates existing broker and its components immediately to V10
• Should be performed on the same system where broker exists
• Requires stopping the broker / integration node
• Requires running the command: mqsimigratecomponents
• Provides –t option to roll back to previous state
Parallel migration:
• Provides ability to migrate on same or a different system Does
not overlap with the existing broker, so no need to stop the broker
• Create new Integration node and deploy artifacts to it
• Requires the Integration node / server properties and
Administration security to be reconfigured on the newly created
components
1
2
3
4
Install V10
Stop Broker
mqsimigratecomponents
Start Integration Node
1
2
3
4
Install V10
Create new
Integration Node
Deploy artefacts
Stop old Broker
Backup resources first!

MongoDB
Switch Server
LINUX_NODE
WINDOWS_NODE

Installation
Configuration
Deployment
Migration
43
Monitoring
Security

IIB Sessions at Interconnect 2017
Session Who Time
2110A What's New in IBM Integration Bus BT Monday 16:15 – 17:00
2141A IBM Integration Bus Futures and Strategy (Inner Circle only) BT Tuesday 11:30 – 12:15
2158A Technical Introduction to IBM Integration Bus GG Tuesday 13:30 – 14:15
2118A Developing Integrations for IBM Integration Bus on Cloud GG Tuesday 14:30 – 15:15
2144A IBM Integration Bus Customer Roundtable BT Tuesday 15:45 – 16:30
2121A Docker and IBM Integration Bus GG Wednesday 09:00 – 09:45
2151A Effective Administration of IBM Integration Bus SN Wednesday 10:15 – 11:00
2144B IBM Integration Bus Customer Roundtable BT Wednesday 16:15 – 17:00
2124A Operational and Business Monitoring with IBM Integration Bus SN Thursday 09:30 – 10:15
2111A IBM Integration Bus and REST APIs SN Thursday 10:30 – 11:15
2166 IBM Integration Bus Version 10 Hands-On Scheduled Lab GG+SN Monday 13:00 – 14:45
9402 IBM Integration Bus Version 10 Hands-On Open Lab None Any Open Lab Session
44

In case powerpoint isn’t your thing …
 https://developer.ibm.com/integration
 Lots of Blog entries, regular updates and links to product demo
videos! All our recent enablement material is on youtube
IIB and Kibana dashboards https://youtu.be/sCPrT2dHKSs
Running IIB in Bluemix Container Service https://youtu.be/ybGOiPZO3sY
IIB and Kibana dashboards https://youtu.be/sCPrT2dHKSs
IIB and Hybrid Connect https://youtu.be/gWbxIooq3_g
IIB and LDAP https://youtu.be/HrqY9MyfzNs
IIB LoopBack Request node https://youtu.be/rUK_OQ5-Anw
Using IIB to integrate with MongoDB and Cloudant https://youtu.be/Is1pphngUlM
Using IIB for REST, Graphical Mapping & Salesforce: https://youtu.be/XIK6QvNSHdY
IIB, Kafka and Twilio SMS: https://youtu.be/7mCQ_cfGGtU
Using Kafka with IIB https://youtu.be/kYv0crxL86Y
Consuming REST APIs using the IIB REST Request node https://youtu.be/C_6gPlrCHZQ
Easy demo of an IIB App Connect node https://youtu.be/StwPbOiFKzk
45

Notices and disclaimers
Copyright © 2017 by International Business Machines Corporation (IBM).
No part of this document may be reproduced or transmitted in any form
without written permission from IBM.
U.S. Government Users Restricted Rights — use, duplication or
disclosure restricted by GSA ADP Schedule Contract with IBM.
Information in these presentations (including information relating to
products that have not yet been announced by IBM) has been reviewed
for accuracy as of the date of initial publication and could include
unintentional technical or typographical errors. IBM shall have no
responsibility to update this information. This document is distributed
“as is” without any warranty, either express or implied. In no event
shall IBM be liable for any damage arising from the use of this
information, including but not limited to, loss of data, business
interruption, loss of profit or loss of opportunity. IBM products and
services are warranted according to the terms and conditions of the
agreements under which they are provided.
IBM products are manufactured from new parts or new and used parts.
In some cases, a product may not be new and may have been previously
installed. Regardless, our warranty terms apply.”
Any statements regarding IBM's future direction, intent or product
plans are subject to change or withdrawal without notice.
Performance data contained herein was generally obtained in a
controlled, isolated environments. Customer examples are presented
as illustrations of how those customers have used IBM products and
the results they may have achieved. Actual performance, cost, savings or
other results in other operating environments may vary.
References in this document to IBM products, programs, or services
does not imply that IBM intends to make such products, programs or
services available in all countries in which IBM operates or does
business.
Workshops, sessions and associated materials may have been prepared
by independent session speakers, and do not necessarily reflect the
views of IBM. All materials and discussions are provided for informational
purposes only, and are neither intended to, nor shall constitute legal or
other guidance or advice to any individual participant or their specific
situation.
It is the customer’s responsibility to insure its own compliance with legal
requirements and to obtain advice of competent legal counsel as to
the identification and interpretation of any relevant laws and regulatory
requirements that may affect the customer’s business and any actions
the customer may need to take to comply with such laws. IBM does not
provide legal advice or represent or warrant that its services or products
will ensure that the customer is in compliance with any law.
46

Notices and disclaimers
continued
Information concerning non-IBM products was obtained from the
suppliers of those products, their published announcements or other
publicly available sources. IBM has not tested those products in
connection with this publication and cannot confirm the accuracy of
performance, compatibility or any other claims related to non-IBM
products. Questions on the capabilities of non-IBM products should be
addressed to the suppliers of those products. IBM does not warrant the
quality of any third-party products, or the ability of any such third-party
products to interoperate with IBM’s products. IBM expressly disclaims
all warranties, expressed or implied, including but not limited to, the
implied warranties of merchantability and fitness for a particular,
purpose.
The provision of the information contained herein is not intended to, and
does not, grant any right or license under any IBM patents, copyrights,
trademarks or other intellectual property right.
IBM, the IBM logo, ibm.com, Aspera®, Bluemix, Blueworks Live, CICS,
Clearcase, Cognos®, DOORS®, Emptoris®, Enterprise Document
Management System™, FASP®, FileNet®, Global Business Services®,
Global Technology Services®, IBM ExperienceOne™, IBM SmartCloud®,
IBM Social Business®, Information on Demand, ILOG, Maximo®,
MQIntegrator®, MQSeries®, Netcool®, OMEGAMON, OpenPower,
PureAnalytics™, PureApplication®, pureCluster™, PureCoverage®,
PureData®, PureExperience®, PureFlex®, pureQuery®, pureScale®,
PureSystems®, QRadar®, Rational®, Rhapsody®, Smarter Commerce®,
SoDA, SPSS, Sterling Commerce®, StoredIQ, Tealeaf®, Tivoli® Trusteer®,
Unica®, urban{code}®, Watson, WebSphere®, Worklight®, X-Force® and
System z® Z/OS, are trademarks of International Business Machines
Corporation, registered in many jurisdictions worldwide. Other product
and service names might be trademarks of IBM or other companies. A
current list of IBM trademarks is available on the Web at "Copyright and
trademark information" at: www.ibm.com/legal/copytrade.shtml.
47

Topology
Planning
Installation
Configuration
SecurityDeployment
Migration
50
Monitoring
High Availability

51
• What does your SLA say?
• Agree SLAs with the business
(% uptime)
• Scheduled and unscheduled
outages
• Ensure you have the technology
to meet the SLA
• Redundancy in applications,
• Clustering,
• PowerHA (HACMP),
• What about applying service?
Availability % Downtime per year Downtime per month Downtime per week
90 (one nine) 36.5 days 72 hours 16.8 hours
99 3.65 days 7.2 hours 1.68 hours
99.9 8.76 hours 43.2 min 10.10 min
99.99 52.6 min 4.32 min 1.01 min
99.999 (“five nines”) 5.26 min 25.9 secs 6.05 secs
99.9999 31.5 secs 2.59 secs 0.605 secs
A C
B1L1 L2
B2
A CB
L1 L2 89.996%
Availability
[~2.4 hours down per day]
98.996%
Availability
[~14.5 mins down per day]
Example
• Same processing components: A, L1, B, L2, C
• Same levels of availability for each component
• A, C, L1, L2 are 99.999% available.
• B is 90% available
• So duplicate B  B1 and B2
• Individually B1 and B2 are 90% available, but
together they are 99% available.
• Increased availability by duplicating components

• How do I ensure that the integration node is
continually processing messages?
• Active/Active vs. Active/Passive
52
High
Availability
Each Integration
Node operates
independently
Continuous availability of the
service during a failure
Single
instance of IIB
Active & Passive state must have
identical copies of persisted data.

53
High
Availability
Multi-instance queue manager
• Integrated into the IIB and MQ products
• Faster failover than HA cluster
• Delay before queue manager restart is much
shorter
• Runtime performance of networked storage
must be considered
• IP address of standby instance is different to
primary
• No automatic fail-back to primary hardware
when restored
HA cluster
• Capable of handling a wider range of failures
• Failover historically rather slow, but some HA
clusters are improving
• Some customers frustrated by unnecessary
failovers
• Extra product purchase and skills required
IP address of each
machine is different.
Queue Manager
restart quicker.
Single IP address.
Handle wider range of
failures.

54
High
Availability
Planning for disaster recovery
• What would you do if your primary IIB location goes down?
• Distribute IIB to multiple sites if possible
• This introduces data replication and latency concerns
• Keep DR concerns separate from HA!
Disaster Recovery High Availability
Systems at multiple sites with
replicated configurations
Systems at a single site with a single
configuration
DR is unplanned. An HA failover can be a planned
activity.
Take regular
backups!
Write a DR
plan!
Test your DR
plan!

Topology
Planning
Installation
Configuration
SecurityDeployment
Migration
55
Monitoring
High Availability

QM1
Integration
Node 1
QM2
Integration
Node 2
sprayer
MQ requests
Machine A Machine B
 Integration Node 1 has flows
getting/putting to QM1 and QM2
 Integration Node 2 also has flows
getting/putting to QM1 and QM2
Topology
Planning
QM1
Integration
Node 1
QM2
Integration
Node 2
sprayer
MQ requests
Machine A Machine B
 If QM1 goes down, Integration Node
1 and Integration Node 2 are
unaffected. They can continue to
get/put from QM2
QM1
Integration
Node 1
QM2
Integration
Node 2
sprayer
MQ requests
Machine A Machine B
 If Integration Node 1 goes down, but
QM1 is still running, then Integration
Node 2 is unaffected. It can continue
to get/put from QM1 and QM2

Node wide
HTTP listener
SYSTEM.BROKER.
WS.INPUT
http://localhost:7080
https://localhost:7083
http://localhost:7080/ServiceC
http://localhost:7080/ServiceD
HTTP
HTTPS
SYSTEM.BROKER.
WS.REPLY
http://localhost:7080/ServiceA
http://localhost:7080/ServiceB
Integration Server
Integration Server
Integration Node wide HTTP(S) Listener
biphttplistener

Node wide
HTTP listener
SYSTEM.BROKER.
WS.INPUT
SYSTEM.BROKER.
WS.REPLY
Integration Server
Integration Server
Servlet Container
Proxy Servlet
HTTP
HTTPS
biphttplistener

60
Node wide
HTTP listener
SYSTEM.BROKER.
WS.INPUT
SYSTEM.BROKER.
WS.REPLY
Threads
HTTP(S) Listener
Threads
HTTP(S) Listener
Integration Server
Integration Server
HTTP
HTTPS
HTTP
HTTPS
Integration Server
Integration Server
Integration Server
HTTP(S) Listener
Servlet Container
Proxy Servlet
HTTP
HTTPS
biphttplistener

61
Node wide
HTTP listener
SYSTEM.BROKER.
WS.INPUT
SYSTEM.BROKER.
WS.REPLY
Threads
HTTP(S) Listener
Threads
HTTP(S) Listener
Integration Server
Integration Server
HTTP
HTTPS
HTTP
HTTPS
Integration Server
Integration Server
Integration Server
HTTP(S) Listener
Servlet Container
Proxy Servlet
HTTP
HTTPS
biphttplistener
Export port
configurations and use
in an external HTTP
Server
Export port
configurations and use
in an external HTTP
Server

62
• Most resilient and best performing systems are those that are:
• Loosely coupled and have parallel execution (threads and processes)
• Implications for your message flow design will depend on the systems that IIB interacts with
• Vital to conduct performance testing before production in production like environment
• Gives time to evaluate and refactor code if needed
• Top Tips
• Always ensure your environment is reproducible
• Treat DR and HA separate
• Ensure regular backups
• Encourage developers to create message flows that enables operational tweaks to be made
• Schedule regular maintenance windows
Summary

Effective administration of IBM Integration Bus - Sanjay Nagchowdhury

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to Effective administration of IBM Integration Bus - Sanjay Nagchowdhury

Similar to Effective administration of IBM Integration Bus - Sanjay Nagchowdhury (20)

Recently uploaded

Recently uploaded (20)

Effective administration of IBM Integration Bus - Sanjay Nagchowdhury