Paul McCollum will be showing us some refined practices around using some special features of Platform Events to Supercharge your Flows and process MILLIONS of records. This is an advanced level pattern and not for the faint of heart. Come check it out if you are tired of having to switch to Apex for any automation that affects more than a few hundred records.
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Supercharge Flows for thousands of records with Platform Events
1. The “Event Turnstile” Pattern: A #FlowFirst
approach that will turbocharge your Flows and let
you use Flow for many more challenges without
having to use Apex
Break Record Limits with
Recursive Flows.
(a.k.a. Recursive Flows)
3. You are about to learn how to use Flows in a way that can POSSIBLY create infinite loops.
Flow has been designed VERY SPECIFICALLY to prevent exactly this.
****Use at your own risk.****
If you break your org or your tenant, remember: You DON’T KNOW ME!
This is your only warning.
4. The Problem:
What we did. What stopped us!
Something
starts our Flow
CPU
time
Iteration
#
Memory
Size
Error
Transactions can only run for so many* seconds
before the SF engine kills the process.
Flows have a maximum of 2000 “iterations” before
they error out.
Flows and query data can only contain or save
‘state’ with so much* data or it errors out.
* See governor limits. Vague ‘so’ given here as limits change.
Pause?
Sub-flow?
Transaction
Governor Limits
5. Who needs to know this stuff
I love flows and I hate limits,
tell me the cheat codes!!!
I’m learning Flows but I’ve
never tried to use them for
anything “Big”.
I’m going to show the unique
part of the approach first….
And then go through all the
pieces that make up how the
whole thing works.
7. Update Records
(YOUR PURPOSE)
Another Perspective: New Pattern
Something
starts our Flow
Event Triggered
Flow
Get Records
(non-tagged)
Loop over
Records
Record Records
(for tagging)
Build an Update
Collection
Update Records
(for tagging)
Count Until
“MaxLoop”
Write
ANOTHER
Event
THEN
Transaction
Transaction
The
Magic
Event
8. Developer Concepts used to attack the problem.
It’s as easy as 1, 2 , 3 …
1. Shaping the record size that our flows will process per run.
1. Plan before you build. Know your relative data sizes.
2. Saving our progress on incomplete runs.
3. Creating and Using “Platform Events” to trigger flows.
10. Step 1: Plan Data Sizes:
Limbo under the limits per Flow instance.
Under 50k Records
“Get Records”
Under 2k Iterations
“Flow Elements”
Under (2k ÷ Elements)
“Loops”
11. Steps
• Most parts of a flow, count as iterations.
Flows currently have a 2000 iteration
limit.
CPU time
• If a transaction takes too long to finish,
you will get a CPU time error.
Memory
• To ‘pause’ a process, all the memory
(records, variables, values) must be
saved for later. Too much data = error.
Flow Limits (basics)
12. Flows
Steps, Loops and Transactions
In this Flow:
• There are 3 One-time iterations (actions).
• And 3 one per record (Loop) iterations (actions).
Flows can have a maximum of 2000 iterations per run.
Iterations are Special Steps in a flow.
Once per Flow Once per Flow
Once per RECORD
13. Flows
Steps, Loops and Transactions
To calculate Iteration limits:
• Multiply the number of Loop actions.
• Add the number of 1-time actions
• Then Divide 2000 by that number.
Once per Flow Once per Flow
Once per RECORD
2000 / (“Loop iterations”) - “1-time actions” = Max Loop Size
14. Flows
Steps, Loops and Transactions
Examples::
2 Loop actions = 2000/2 = 1000 records per Flow.
5 Loop actions = 2000/5 = 400 records per Flow
20 Loop actions = 2000/20 = 100 records per Flow
Flows can have a maximum of 2000 iterations per run.
Iterations are Special Steps in a flow.
Once per Flow Once per Flow
Once per RECORD
2000 / (“Loop iterations”) - “1-time actions” = Max Loop Size
Examples:
2 LA + 10 SA = 2000/2-10 = 990 records per Flow.
5 LA + 10 SA = 2000/5 -10 = 390 records per Flow
20 LA + 10 SA = 2000/20 -10 = 90 records per Flow
SA = Single (one time) Action
LA = Loop (repeated) Action
16. Step 2: Counting, Saving and Resuming
Remember: Maximum “Get Records” Size = 50k
Be aware of indexes.
Batch Tracking:
Create a field on your target object that you can ‘mark’ as done for each batch.
If your process runs daily, make it a Date Field.
If your process runs Monthly, make it a Date Field.
If it runs for anything else, give it a name that’s going to be different the next time it runs.
But usually ‘date’ is a good batch label.
THEN
Marking your place:
You need to have several tracking variables in your head AND YOUR FLOW to make this work.
• Max Loops
• Max Runs
• Field in your recordset to mark complete on this “Run”.
- e.g., Date/Event/Time/Reporting Period
Each time you finish a batch run, update the records you’ve processed with the batch label in your new batch field.
19. Step 3: Flow: Creating Events
They don’t seem special, but they really are.
Created Different but otherwise just like an object, field and records.
Except you can’t query them, so testing can be ‘fun’.
21. Update Records
(YOUR PURPOSE)
Assembling the parts
Something
starts our Flow
Event Triggered
Flow
Get Records
(non-tagged)
Loop over
Records
Record Records
(for tagging)
Build Update
Collection
Update Records
(for tagging)
Count Until
“MaxLoop”
Write
Event
Write
ANOTHER
Event
THEN
28. Advanced Concepts
Outer Loops for MOAR records
-Sizing your data to fit under the hurdles
Safety Counters
Managing Multiple Flows with the same Event Object
-”Flow Turnstile Event Object” vs. “Thing that Recursively does this one flow”.
