MuleSoft Surat Virtual Meetup#18 - Persistent Queue, Object Store and Persistent Queue

1. [19th June 2021]
16:00 IST (GMT+05:30)
[Surat] MuleSoft Meetup Group
Persistent Queue, VM Connector and
Object Store

2. 2
Organizers
Jitendra Bafna
Senior Solution Architect
Capgemini
About the organizer:
➢ Working as Senior Solution Architect at Capgemini.
➢ Surat MuleSoft Meetup Leader and MuleSoft Ambassador.
➢ 12.5+ Years of Experience in Integrations and API Technologies.
➢ Certified MuleSoft Integration Architect and platform Architect.

3. 3
Organizers
Nitish Jain
Consultant
IBM
About the organizer :
➢ Working as Consultant at IBM.
➢ Surat MuleSoft Meetup Leader.
➢ 2.5+ Years of Experience in Integrations and API Technologies.
➢ Certified MuleSoft Developer and Platform Architect.

4. 4
Speakers
Jitendra Bafna
Senior Solution Architect
Capgemini
About the speaker:
➢ Working as Senior Solution Architect at Capgemini.
➢ Surat MuleSoft Meetup Leader and MuleSoft Ambassador.
➢ 12.5+ Years of Experience in Integrations and API Technologies.
➢ Certified MuleSoft Integration Architect and platform Architect.

5. Agenda
5
❖ Introduction
❖Persistent Queue
❖VM Connector – Request Reply and Reliability Pattern
❖ Object Store
❖Live Demonstration
❖Trivia Quiz

6. Persistent Queue
CloudHub Persistent Queues is a Cloud Service that allows messages that are published to VM queues to
be stored externally to the application. This feature is directly tied into the CloudHub Fabric (Multi-
worker) feature and is part of CloudHub's high availability infrastructure. When CloudHub Persistent
Queues is enabled, it allows a developer to design VM queues in such a way that payloads can be shared
across different workers, for the purposes of load distribution. In the case of a failure of one of the
workers in a multi-worker deployment, messages on the queues are able to be consumed by other
working workers, ensuring continuity of service.
6

7. Persistent Queue – Use Cases
CloudHub Persistent Queues is a specific feature for a specific task.
● It's design is solely to enable the capability to share payloads across multi-worker application
deployments. It should only be used for this purpose.
● Enabled for this use-case, CloudHub Persistent Queues will improve HA resiliency and performance
due to the load distribution and sharing capabilities.
7

8. Persistent Queue – Recommendations
Reliability Patterns. CloudHub Persistent Queues should not be confused with traditional Mule Persistent
VM queues. While Mule Persistent VM queues were intended to be part of a reliability pattern solution,
this was never the intention for CloudHub Persistent Queues.
The strong recommendation is that CloudHub Persistent Queues should not be considered as the feature
to use when designing a Reliability pattern in CloudHub. With that said, CloudHub Persistent queues
should never be enabled for single worker applications. For a true reliability pattern, the queues used
should be an enterprise grade message broker, examples include Anypoint MQ, ActiveMQ, IBM MQ, or
other equivalent JMS or AMQP broker.
8

9. Persistent Queue – Limitations
● Performance. Applications with CloudHub Persistent Queues will process messages up-to or more
than 10x slower than not using CloudHub Persistent Queues. The performance is further significantly
decreased if using the Encryption option. The reason for this is because CloudHub Persistent Queues
uses a network attached store, which has both network latency and physical limitations in terms of
read / write speed. When disabled, VM queues will be stored in-memory, and has almost
instantaneous latency.
● Message retention. CloudHub Persistent Queues will retain messages for 4 days. This retention period
is not customisable. Because of this CloudHub Persistent Queues should not be used for any long term
storage.
● Batch Jobs. Mule Batch Jobs internally make use of queues. When CloudHub Persistent Queues is
enabled for an application with Batch processors, severe performance degradation will occur. Batch
also has its own set of limitations incurred if using CloudHub Persistent Queues. The significant one
being that message reliability is not guaranteed. Message loss can still occur in the case of application
restart (intentional or unintentional restarts). This limitation means there is no benefit from having
CloudHub Persistent Queues enabled for a batch application. 9

10. VM Connector
VM Connector manages the inter or intra communication via persistent or transient asynchronous queue.
● Transient Queue is faster than Persistent Queue but not reliable.
● Persistent Queue is reliable but slower than Transient Queue.
VM Connector- Use Cases
● One of the use cases for using VM connector when you want to pass message between flows.
● Distribute work across a cluster.
● Communicate with different apps that are running in the same Mule domain
● Perform simple queueing that does not justify the use of a full JMS broker
10

11. VM Connector
VM Queue Operations
● Listener
● Publish
● Consume
● Publish Consumer
11

12. Live Demonstration
• Publish and Consumer messages Request Reply
Pattern
• Reliability and Transactions
• Persistent Queue

13. Object Store
Object Store enables you
● Retrieve a list of object stores and keys associated with an application.
● Store and retrieve key-value pairs in an object store.
● Delete key-value pairs from an object store.
Object Store Connector is a Mule component that allows for simple key-value storage. Although it can
serve a wide variety of use cases, it is mainly design for:
● Storing synchronization information, such as watermarks.
● Storing temporary data
● Storing User Information
13

14. Object Store V2
Object Store v2 lets CloudHub applications store data and states across batch processes, Mule
components and applications, from within an application or by using the Object Store REST API.
● On-premises Mule applications can access Object Store v2 only via the Object Store REST API.
● You can configure a Mule app to use the Object Store REST API to store and retrieve values from an
object store in another Mule app.
14

15. Object Store V2 - Features
● Supports both static and rolling TTL (time to live) for data persistence.
● Allows for an unlimited number of entries. There is no limit on the total size of v2 object-stores.
● Stores values up to 10 MB (when Base64 encoded) in size. You can estimate base64 size of a payload
as follows: CEILING(base10size * 1024/3) * 4, where base10size is object size in bytes. + Example: A
base10 payload size of 7.5 MB converts to 10.24 MB base64.
● Is available in all supported regions and availability zones within each region.
● Is co-located in the same region as your workers. For example, workers hosted in Singapore would use
Object Store v2 hosted in Singapore.
● Provides a Mule connector and REST interface for access by external applications.
● Provides end-to-end secure TLS-transport.
● Encrypts persistent storage to FIPS 140-2 compliant standards.
15

16. Object Store V2 - Features
● The Object Store Connector works for both Object Store version 1 and 2.
● Object Store v2 enforces a maximum data persistence of 30 days from time of creation.
● The free Object Store v2 limits usage to 10 transactions per second (TPS) per application.
● Premium add-on customers are allowed up to 100 TPS per app.
● To increase your TPS limit, contact the MuleSoft account teams
16

17. How long can data persist in Object Store v2?
The maximum TTL (time to live) is 2592000 seconds (30 days).
The entryTtl value, specified in the global configuration parameters for the connector, determines when
to evict key-value pairs from the store.
The TTL can be either rolling or static.
Rolling
Accessing the data extends the TTL for 30 days.
The minimum rolling TTL is 1 second. The maximum rolling TTL is unlimited as long as the data continues
to be accessed.
Static
The value of entryTtl determines when the data is evicted.
The minimum static TTL is 1 second. The maximum (and default) static TTL is 2592000 seconds (30 days).
If the entryTtl value is greater than 30 days, TTL is set to the maximum value of 2592000 seconds (30
days) without returning an error.
Updating the data (for example, by overwriting the key) resets the TTL.
17

18. Object Store V1 V/S Object Store V2
18
Object Store V1 Object Store V2
Maximum of 100,000 key:value pairs per application
768 byte key size
1 MB value size
1 GB of data per application
Supported by ObjectStore Connector, but attributes like
entryTTL, Expiration Interval, Max Entries are _not_
configurable.
No API Rate limit.
• Key time to live is 30 days, and extended by 30 days if
accessed.
• Unlimited number of entries, unlimited total size of object
stores.
• Max value size 10 MB.
• Max key length 256 characters.
• Supported in each deployment region, and is co-located with
your CloudHub application
• Secured end-to-end with TLS-transport
• Encrypts persistent storage to FIPS 140-2 compliant
standards
• Supported by Object Store Connector, but attributes
like Expiration Interval, Max Entries are _not_ configurable.
• Standard version provides a maximum API request rate limit
of 10TPS.
• Premium OSv2 provides a maximum API request rate limit
of 100TPS (upgradable through Object Store Premium
subscription purchase).

19. 19

20. Get ready to WIN a Special Gift from MuleSoft Community
Quiz Time

21. Introduce yourself to your neighbor
Networking time

22. Thank you