© 2015 IBM Corporation1
IBM MQ Light :
From monolith to microservices for
speed and scale
Alan Chatt
Product Manager IBM Messaging
alan.chatt@uk.ibm.com
2 © 2015 IBM Corporation
Agenda
• Microservices : what & why
• MQ Light messaging for Microservices
• How are others doing microservices
3 © 2015 IBM Corporation
Microservices
Development teams
Delivering elastically scalable, distributed
applications with unprecedented agility
using a microservices architecture
Agility comes from :
“2 pizza” teams operating completely independently,
owning the whole technology stack, deploying on their own
schedules, writing and owning services that will scale
when demand dictates
Not without challenges :
• Fallacies of distributed computing
• Operational overhead increases with the number of services to deploy
• New skills and tooling required to cope with additional complexities
3
4 © 2015 IBM Corporation
Motivations : Rate of change
More local changes
More GitHub check-ins
Changes go all the
way to Production
5 © 2015 IBM Corporation
Motivations : Flexibility
6 © 2015 IBM Corporation
Motivations : Resilience
Communication can:
1) Re-route
2) Stored for future delivery
3) Return cached/out of date data
1
2 3
7 © 2015 IBM Corporation
Motivations : Scale
Steady state Workload increases Individual components
scale up as required
8 © 2015 IBM Corporation
Migrating from a monolith
• When & why
• Do it because you need to evolve rapidly
• Don’t just re-write
• Chip away at bits and improve
• Techniques
• Emit events from existing application
• Surround with simple service
interfaces that you can call
Monolithic
Application
9 © 2015 IBM Corporation
A few words of caution
Some wisdom shared by people in the trenches :
• Do not start a new implementation with microservices
• Master dev-ops before starting a microservices
deployment
• Cleanly separate platform services from the application
• Do not rush into creating too many microservices
10 © 2015 IBM Corporation
MQ Light
Messaging for microservices
11 © 2015 IBM Corporation
MQ Light : Software and Cloud
• Messaging that application developers will
love to use, helping them make responsive
applications that scale easily
• 3 ways to get it:
• Bluemix service
• MQ Light software download
• MQ Light API support in MQ (Beta)
• Open APIs crafted to feel natural in a growing
range of popular languages
• Tooling that makes modular app development
easy
11
12 © 2015 IBM Corporation
Messaging patterns for loose coupling
Worker offload • Image processing
• Text analytics
Multiple service instances use a shared destination
• Service scaled up by adding an instance
• MQ Light will start to distribute work to new instance
• Service scaled down by “killing” an instance
• MQ Light will stop distributing work to that instance and will re-
distribute unacknowledged messages
• Service can rate limit inbound work
• Use link credit to restrict amount of work currently being processed
13 © 2015 IBM Corporation
Messaging patterns for loose coupling
Event driven /
Fire and forget
• Posting video to multiple social
sites after transcoding
• Respond to external events
Using multiple MQ Light destinations subscribed to the same topic
• Cuts down unnecessary “polling” traffic between services
• Deploying multiple versions of a service simultaneously
• Allows duel running for a period of time
• Create new extensions driven by existing data
• New subscribers just listen to an existing topic, no need to update
publisher
• i.e. create new service which performs analytics on existing data flows
14 © 2015 IBM Corporation
Messaging for loose coupling
• Isolating your application from the availability and performance
of an external system
• Replace part of existing system by intercepting events and
running new logic
• MQ Light API support in MQ
• Subscribe to data already flowing between existing systems
Connecting external
systems
• Updating external booking system
• Running logic when existing CRM
system updates contact address
15 © 2015 IBM Corporation
Using open standards for technology freedom
• Using AMQP1.0 as standard protocol
• Open source clients from the Apache Qpid Proton
project
• Example for this for PHP :
https://developer.ibm.com/messaging/2014/07/01/using-php-mq-light-bluemix/
• Easily Integrated with microservices frameworks :
https://developer.ibm.com/messaging/2015/05/06/microservices-with-seneca-and-mq-light/
• MQ Light specific clients reduce quantity and
complexity of code
• Node.js, Java, Ruby, Python
16 © 2015 IBM Corporation
Designed for development
• MQ Light download quick and easy to run on a
laptop
• Works well with Vagrant & Docker
https://hub.docker.com/r/ibmcom/mqlight/
• No configuration : applications define behaviour
• MQ Light tooling designed for developers
17 © 2015 IBM Corporation
Deployment Options
17
MQ Light Service
for Bluemix
[Managed Service]
WebSphere MQ
[open Beta]
“MQ Light”
18 © 2015 IBM Corporation
Presence Insights
Adopting Microservices within IBM
19 © 2015 IBM Corporation
IBM Presence Insights
Bring the same personalized shopping experience you get
online to “bricks and mortar” stores
IBM SaaS offering built on the cloud using microservices
20 © 2015 IBM Corporation
Technologies used by Presence Insights
21 © 2015 IBM Corporation
Presence Insights : Today
• 22+ Microservices
• 70+ Node.js instances
• Application deployed across Bluemix and Softlayer
• Node.js (Web Tier)
• MQ Light (Messaging)
• Cloudant (Persistence)
• Redis (Caching)
• D3 (Charting)
22 © 2015 IBM Corporation
Presence Insights : Architecture
23 © 2015 IBM Corporation
Presence Insights : Use of MQ Light
• PI use the MQ Light service in Bluemix for communication
between their microservices, using both worker offload and
event driven patterns
• Wildcards : Feeding many separate streams of data to multiple services
• ClientID : Audit trail of which service instances receives which event
• Destination Sharing : Built in load balancer
• Advantage of managed MQ Light service : “…the team does not need
to manage the instances themselves (security compliance and operating system
monitoring/management)”
“MQLight as a managed service is a key element to our Presence Insights
strategy. The capabilities that are provided in the Node.js client libraries and the
MQLight service instance provides a great foundation for our messaging system
around publishing and subscribing to our location sensor events”
24 © 2015 IBM Corporation
Presence Insights : Lessons learned
• Streamlining the build & deployment pipeline is critical
• Developers need to be able to push changed and demo them in the
cloud environment
• Continuous and holistic testing combats the additional
complexities of a distributed application
• Tests run at each level act as gate for changes to be promoted
• End to end tests run every 10 minutes against all environments
• These End to end tests ensure platform services (i.e. Cloudant and
MQ Light)
• PI experiment before switching to node.js and current
technology stack
• Time boxing experimentation is essential
25 © 2015 IBM Corporation
Summary
26 © 2015 IBM Corporation
Summary
• Microservices is a great architecture for large, rapidly
evolving applications that need to scale
• Not a silver bullet that solves all problems
• Using asynchronous messaging patterns between services
supports the goals of a microservice architecture
• Provide workload balancing when scaling up and down
• Provides loose coupling, promoting resilience
• MQ Light works well in a dev-ops environment, essential for
making a microservice architecture work well
• Offline development
• Application defined behaviour, no separate configuration to manage
27 © 2015 IBM Corporation
Resources
• Bluemix PaaS :
https://console.ng.bluemix.net/
• MQ Light :
https://developer.ibm.com/messaging/mq-light/
• Redbook : Microservices from Theory to Practice
https://www.redbooks.ibm.com/redbooks.nsf/RedpieceAbstracts/sg248275.html

GWC : MQ Light - from monolith to Microservices for speed and scale

  • 1.
    © 2015 IBMCorporation1 IBM MQ Light : From monolith to microservices for speed and scale Alan Chatt Product Manager IBM Messaging alan.chatt@uk.ibm.com
  • 2.
    2 © 2015IBM Corporation Agenda • Microservices : what & why • MQ Light messaging for Microservices • How are others doing microservices
  • 3.
    3 © 2015IBM Corporation Microservices Development teams Delivering elastically scalable, distributed applications with unprecedented agility using a microservices architecture Agility comes from : “2 pizza” teams operating completely independently, owning the whole technology stack, deploying on their own schedules, writing and owning services that will scale when demand dictates Not without challenges : • Fallacies of distributed computing • Operational overhead increases with the number of services to deploy • New skills and tooling required to cope with additional complexities 3
  • 4.
    4 © 2015IBM Corporation Motivations : Rate of change More local changes More GitHub check-ins Changes go all the way to Production
  • 5.
    5 © 2015IBM Corporation Motivations : Flexibility
  • 6.
    6 © 2015IBM Corporation Motivations : Resilience Communication can: 1) Re-route 2) Stored for future delivery 3) Return cached/out of date data 1 2 3
  • 7.
    7 © 2015IBM Corporation Motivations : Scale Steady state Workload increases Individual components scale up as required
  • 8.
    8 © 2015IBM Corporation Migrating from a monolith • When & why • Do it because you need to evolve rapidly • Don’t just re-write • Chip away at bits and improve • Techniques • Emit events from existing application • Surround with simple service interfaces that you can call Monolithic Application
  • 9.
    9 © 2015IBM Corporation A few words of caution Some wisdom shared by people in the trenches : • Do not start a new implementation with microservices • Master dev-ops before starting a microservices deployment • Cleanly separate platform services from the application • Do not rush into creating too many microservices
  • 10.
    10 © 2015IBM Corporation MQ Light Messaging for microservices
  • 11.
    11 © 2015IBM Corporation MQ Light : Software and Cloud • Messaging that application developers will love to use, helping them make responsive applications that scale easily • 3 ways to get it: • Bluemix service • MQ Light software download • MQ Light API support in MQ (Beta) • Open APIs crafted to feel natural in a growing range of popular languages • Tooling that makes modular app development easy 11
  • 12.
    12 © 2015IBM Corporation Messaging patterns for loose coupling Worker offload • Image processing • Text analytics Multiple service instances use a shared destination • Service scaled up by adding an instance • MQ Light will start to distribute work to new instance • Service scaled down by “killing” an instance • MQ Light will stop distributing work to that instance and will re- distribute unacknowledged messages • Service can rate limit inbound work • Use link credit to restrict amount of work currently being processed
  • 13.
    13 © 2015IBM Corporation Messaging patterns for loose coupling Event driven / Fire and forget • Posting video to multiple social sites after transcoding • Respond to external events Using multiple MQ Light destinations subscribed to the same topic • Cuts down unnecessary “polling” traffic between services • Deploying multiple versions of a service simultaneously • Allows duel running for a period of time • Create new extensions driven by existing data • New subscribers just listen to an existing topic, no need to update publisher • i.e. create new service which performs analytics on existing data flows
  • 14.
    14 © 2015IBM Corporation Messaging for loose coupling • Isolating your application from the availability and performance of an external system • Replace part of existing system by intercepting events and running new logic • MQ Light API support in MQ • Subscribe to data already flowing between existing systems Connecting external systems • Updating external booking system • Running logic when existing CRM system updates contact address
  • 15.
    15 © 2015IBM Corporation Using open standards for technology freedom • Using AMQP1.0 as standard protocol • Open source clients from the Apache Qpid Proton project • Example for this for PHP : https://developer.ibm.com/messaging/2014/07/01/using-php-mq-light-bluemix/ • Easily Integrated with microservices frameworks : https://developer.ibm.com/messaging/2015/05/06/microservices-with-seneca-and-mq-light/ • MQ Light specific clients reduce quantity and complexity of code • Node.js, Java, Ruby, Python
  • 16.
    16 © 2015IBM Corporation Designed for development • MQ Light download quick and easy to run on a laptop • Works well with Vagrant & Docker https://hub.docker.com/r/ibmcom/mqlight/ • No configuration : applications define behaviour • MQ Light tooling designed for developers
  • 17.
    17 © 2015IBM Corporation Deployment Options 17 MQ Light Service for Bluemix [Managed Service] WebSphere MQ [open Beta] “MQ Light”
  • 18.
    18 © 2015IBM Corporation Presence Insights Adopting Microservices within IBM
  • 19.
    19 © 2015IBM Corporation IBM Presence Insights Bring the same personalized shopping experience you get online to “bricks and mortar” stores IBM SaaS offering built on the cloud using microservices
  • 20.
    20 © 2015IBM Corporation Technologies used by Presence Insights
  • 21.
    21 © 2015IBM Corporation Presence Insights : Today • 22+ Microservices • 70+ Node.js instances • Application deployed across Bluemix and Softlayer • Node.js (Web Tier) • MQ Light (Messaging) • Cloudant (Persistence) • Redis (Caching) • D3 (Charting)
  • 22.
    22 © 2015IBM Corporation Presence Insights : Architecture
  • 23.
    23 © 2015IBM Corporation Presence Insights : Use of MQ Light • PI use the MQ Light service in Bluemix for communication between their microservices, using both worker offload and event driven patterns • Wildcards : Feeding many separate streams of data to multiple services • ClientID : Audit trail of which service instances receives which event • Destination Sharing : Built in load balancer • Advantage of managed MQ Light service : “…the team does not need to manage the instances themselves (security compliance and operating system monitoring/management)” “MQLight as a managed service is a key element to our Presence Insights strategy. The capabilities that are provided in the Node.js client libraries and the MQLight service instance provides a great foundation for our messaging system around publishing and subscribing to our location sensor events”
  • 24.
    24 © 2015IBM Corporation Presence Insights : Lessons learned • Streamlining the build & deployment pipeline is critical • Developers need to be able to push changed and demo them in the cloud environment • Continuous and holistic testing combats the additional complexities of a distributed application • Tests run at each level act as gate for changes to be promoted • End to end tests run every 10 minutes against all environments • These End to end tests ensure platform services (i.e. Cloudant and MQ Light) • PI experiment before switching to node.js and current technology stack • Time boxing experimentation is essential
  • 25.
    25 © 2015IBM Corporation Summary
  • 26.
    26 © 2015IBM Corporation Summary • Microservices is a great architecture for large, rapidly evolving applications that need to scale • Not a silver bullet that solves all problems • Using asynchronous messaging patterns between services supports the goals of a microservice architecture • Provide workload balancing when scaling up and down • Provides loose coupling, promoting resilience • MQ Light works well in a dev-ops environment, essential for making a microservice architecture work well • Offline development • Application defined behaviour, no separate configuration to manage
  • 27.
    27 © 2015IBM Corporation Resources • Bluemix PaaS : https://console.ng.bluemix.net/ • MQ Light : https://developer.ibm.com/messaging/mq-light/ • Redbook : Microservices from Theory to Practice https://www.redbooks.ibm.com/redbooks.nsf/RedpieceAbstracts/sg248275.html