Connor Fitzgerald: RPA - Building Your First Automation Dipak Patel: Large Payload Processing with MuleSoft Jared Mosley: Handling Bursts of Out of Order Events

1. All contents © MuleSoft, LLC
DENVER MULESOFT MEETUP
Greatest MuleSoft Hits of 2022
Speakers:
Connor Fitzgerald: MuleSoft Lead Consultant
Dipak Patel: MuleSoft Architect
Jared Mosley: MuleSoft Lead Consultant

2. All contents © MuleSoft, LLC
Speaker
2
Brian Statkevicus
MuleSoft Ambassador & MuleSoft Practice Manager
Big Compass
+ Based in the Denver metro area
+ Over 20 years of Integration experience
+ Hobbies - Skiing, Hiking, Participating in 5Ks
(I’m too slow to say I actually ‘run’ in them)
+ Ex-soccer ref, but I still know how to show a
yellow and red card

3. All contents © MuleSoft, LLC 3
Stephanie Lawrence
Denver MuleSoft Meetup Coordinator
Big Compass
Denver Meetup Introductions
Brian Statkevicus
MuleSoft Practice Manager
Big Compass
Aaron Lieberman
Director, Platform Engineering
Science 37

4. All contents © MuleSoft, LLC 4
● First person to call out the correct answer, wins!
● One answer per person per question
○ If you answer more than 1x per question, you are ineligible for that question
● If you’ve already won during today’s Meetup, you cannot win again
● Big Compass makes the final decision on the trivia winners that will be submitted to MuleSoft
(see next slide)
Trivia Questions - “House Rules”
More important Trivia guidelines on next slide

5. All contents © MuleSoft, LLC 5
● Big Compass emails trivia winners a link to a Google Form for the trivia winners to complete
○ Your email MUST match your training account email; otherwise, you won’t receive a voucher
● Once you complete this form, MuleSoft will verify:
○ You haven’t already won this month
○ You’re not trying other means to circumvent the 1x/month rule
● MuleSoft will send you the voucher
○ The voucher is a training voucher, not a standalone exam voucher. If you select an exam on the
Google Form as opposed to selecting a course, you will not receive a voucher.
○ MuleSoft retains the final decision on determining your eligibility to receive a training voucher
● If you have trouble redeeming a voucher or have not received a voucher, open a ticket at
training.mulesoft.com/question. DO NOT CONTACT US!
Trivia Questions - MuleSoft Guidelines

6. All contents © MuleSoft, LLC 6
● You must complete the training request form by December 31, 2022
● You MUST schedule a class by January 20, 2023. The actual class can occur anytime, but
you must schedule by January 20, 2023
Otherwise, your voucher will be null and void
Final note about Vouchers

7. All contents © MuleSoft, LLC
+ Connor Fitzgerald: RPA - Building Your First Automation
+ Dipak Patel: Large Payload Processing with MuleSoft
+ Jared Mosley: Handling Bursts of Out of Order Events
Agenda

8. All contents © MuleSoft, LLC
Connor Fitzgerald
About Me
● MuleSoft Lead Consultant | Big Compass
● Live in Atlanta, Georgia
● Hobbies - DIY Projects and Exercise

9. All contents © MuleSoft, LLC
Robotic Process
Automation
Building Your First Automation

10. All contents © MuleSoft, LLC
Robotic Process Automation
Why? How?
Overview
What?

11. All contents © MuleSoft, LLC
MuleSoft RPA
RPA Recorder
● Record / Learn
● Scaffold
● Model Process
RPA Builder
● Manage RPA Assets
● Build
● Test
Components Overview
RPA Manager
● Manage
● Evaluate
● Design
● Administer

12. All contents © MuleSoft, LLC
Use Case Demo 1
Invoice to Legacy CRM
Use Case Requirements
● Input: Scanned PDF Invoices
● Output: CRM/ERP Database Entry
● Limitation & Hurdles
○ Handwritten values
○ Legacy System
■ Target is not reachable via API

13. All contents © MuleSoft, LLC
API-Led Integration
Demo 1 - Automation Options

14. All contents © MuleSoft, LLC
Robotic Process Automation
Demo 1 - Automation Options

15. All contents © MuleSoft, LLC
Use Case Demo 2
Emails to External CRM
Use Case Requirements
● Input: Emailed Service Tickets
● Output: 3rd Party CRM System
● Limitation & Hurdles
○ No Existing or Planned Integrations with Target
○ Timeline and Budget

16. All contents © MuleSoft, LLC
API-Led Integration
Demo 2 - Automation Options

17. All contents © MuleSoft, LLC
Robotic Process Automation
Demo 2 - Automation Options

18. All contents © MuleSoft, LLC
Trivia Question

19. All contents © MuleSoft, LLC 19
A. RPA can be used a stopgap while robust integrations are in development
B. RPA can work in tandem with other integration tools
C. RPA can be used to replace batch integrations
D. Process logic is defined using RPA Builder
Trivia Question #1 - Which of the following is false?

20. All contents © MuleSoft, LLC
Questions?

21. All contents © MuleSoft, LLC
Dipak Patel
About Me
● MuleSoft Architect | Big Compass
● Live in Denver, Colorado area
● Hobbies: sports (Tennis, Volleyball,
Table Tennis, Cricket, Badminton), Drawing
dipak.patel@bigcompass.com | www.linkedin.com/in/dipakppatel

22. All contents © MuleSoft, LLC
Large Payload Processing
Using MuleSoft

23. All contents © MuleSoft, LLC
Challenges
Large payloads pose many challenges, such as:
● HTTP Request timeout
● API call timeout during processing
● Process running out of memory
● Database connectivity errors
● MQ message size limitations

24. All contents © MuleSoft, LLC
Use Case
● Two system APIs to read large amount of data from HTTP urls and two
system APIs to store that data into a database tables
● Number of records are undefined - could be thousands or even millions of
records
● Dynamic data - being updated at any arbitrary frequency
● Insert retrieved data into database of choice

25. All contents © MuleSoft, LLC
Design Considerations
● Follow API-Led architecture
● The process API should be flexible enough to add additional system APIs
when the need arises
● Non-functional requirements for security, persistence, and performance
● Streaming, batching, throttling, and reliability must be used to deal with large
payload processing
● Mule’s batch processing vs MQ based solution

26. All contents © MuleSoft, LLC
High Level Design
● Process API is invoked by a
scheduler or on-demand.
● Process API calls two system APIs
to bring data as well as two system
APIs to insert data using HTTP
Request.
● System APIs send the response
back to process API using MQ.
● DB bulk insert operation is used.
● Client-id and client-secret along
with secured properties are used.
● DLQs are used for failed batches.

27. All contents © MuleSoft, LLC
Challenges and Solutions
Challenge Solution
HTTP timeout in system APIs
to retrieve due to large data
stream
Use HTTP response streaming.
Set Output MIME Type using
outputMimeType = "application/csv;
streaming=true"
Out of memory in system API
to Transform large data
stream
Use streaming functionality in Transform.
@StreamCapable() annotation
Out of memory in system API
to publish records in MQ
This happens because of MQ message limit
of 10 MB.
Use batch of 100 records using for-each.
HTTP timeout in process API
while calling system APIs
Make system APIs async behavior.
Wrap all processors in Try >> Async scope.
HTTP connectivity error in
process API while calling DB
system APIs
Use throttling in MQ subscribe to limit
number of message processing and
therefore number of open connections.
Use Subscribe type of ‘Polling’ with Fetch
Size of ‘10’ every second.

28. All contents © MuleSoft, LLC
Implementation - Process API
Calls to long running
System APIs are wrapped
in Async to avoid HTTP
timeout error.

29. All contents © MuleSoft, LLC
Implementation - System API
Streaming is enabled for the
HTTP URL Request and
Transformer to avoid Out of
Memory error.
Batching (using for-each) is used
while publishing to MQ to avoid 10
MB message limitation.

30. All contents © MuleSoft, LLC
Implementation - Process and System API
Throttling is used in the
message subscriber to avoid
HTTP connectivity error in the
Database API Request.
Streaming is used
in the transformer
to avoid Out of
Memory error.
Bulk Insert is
used to improve
performace

31. All contents © MuleSoft, LLC
Resources
● Streaming in Dataweave:
https://docs.mulesoft.com/dataweave/2.4/dataweave-streaming
● Streaming in HTTP Listener:
https://docs.mulesoft.com/http-connector/1.6/http-listener-ref
● MQ subscriber flow control: refer to Predictable Message Consumption in
https://docs.mulesoft.com/anypoint-mq-connector/3.x/anypoint-mq-listener

32. All contents © MuleSoft, LLC
Trivia Question

33. All contents © MuleSoft, LLC 33
A. Batch processing may not allow you to have separate system APIs to retrieve
and insert data into a database, and therefore it would break the API-led
design principles.
B. Batch processing increases application complexity while limiting scalability.
C. Both
D. None
Trivia Question #2 - What is true about Mule’s batch
processing option for this use case?

34. All contents © MuleSoft, LLC
Questions?

35. All contents © MuleSoft, LLC
Jared Mosley
About Me
● MuleSoft Lead Consultant | Big Compass
● Live in Seattle, Washington
● Hobbies - Music, Video Games, and Reading
● Passions - Curiosity and Connections

36. All contents © MuleSoft, LLC
Handling Bursts of
Out of Order Events
Shopify to Salesforce

37. All contents © MuleSoft, LLC
The Problem
Shopify Salesforce
eShopSync
Connector

38. All contents © MuleSoft, LLC
Problems - Unordered Events
● Large objects with a top-level
change timestamp
● We only care about changes to
certain fields
● Custom triggers on Shopify cause a
series of updates to an object on
common actions
● Salesforce is receiving an update
anytime the modification timestamp
changes

39. All contents © MuleSoft, LLC
Problems - Time
● Events from Shopify are not guaranteed to be
ordered
● Events on an object come in bursts of 1-10
events in a second
● No way to tell if an event burst has finished
● No “rest” state for an object as it can be modified
at any time
● Only queues are VM queues available
● No intermediary storage is available

40. All contents © MuleSoft, LLC
The Good Things!
● We only care about the latest state of the
object
● A 20 second delay is considered acceptable
● We know which fields we care about ahead
of time
● We don’t need to automatically store or
replay events

41. All contents © MuleSoft, LLC
The Solution
Shopify Salesforce
MuleSoft

42. All contents © MuleSoft, LLC
The Solution
Shopify Salesforce
MuleSoft
latestHash
latestUpdatedAt
updatedAt
eventId
latestHash
latestUpdatedAt

43. All contents © MuleSoft, LLC
Wrap Up
● We can track every event from Shopify to
Salesforce
● We have greatly reduced the number of calls to
Salesforce
● Our events are always propagated in order
● We don’t make unnecessary update calls
● We can manually retrigger specific or latest
updates if needed

44. All contents © MuleSoft, LLC
Trivia Question

45. All contents © MuleSoft, LLC 45
A. All events are sent to Salesforce
B. The event is newer than the one we have
C. The event’s hash doesn’t match the one we have
D. It is the latest event in the burst of events
E. B & C
F. C & D
Trivia Question #3 - What events do we send to Salesforce?

46. All contents © MuleSoft, LLC
Questions?

47. All contents © MuleSoft, LLC
Next Steps

48. All contents © MuleSoft, LLC 48
● Share:
○ Tweet using the hashtag #MuleSoftMeetups
○ Invite your network to join: https://meetups.mulesoft.com/denver
● Feedback:
○ Complete the Meetup survey, provide feedback, and suggest topics for upcoming events
○ Contact MuleSoft at meetups@mulesoft.com for ways to improve the program
What’s next?

49. All contents © MuleSoft, LLC 49
● Have a great Holiday season! See you in 2023
● Future (tentative) Meetup schedule:
○ Mid February
○ Mid April
○ Mid June
○ Mid August
○ End of September
○ Mid November
○ Plus any special virtual roadshows
● We are looking for topic suggestions and speakers!
○ DataWeave is always popular
○ Local customer experience
○ Logging revisited
○ Deeper dive into RPA Builder
○ Tableau + MuleSoft
Denver Meetups 2023

50. All contents © MuleSoft, LLC
Thank you!