We are back with another set of interesting topics. Want to learn all about processing high data volumes with an ERP? We got you covered! Or maybe your heart is more at the Front-End and you want to know how to use SASS with OutSystems? We also got you covered.
WSO2CON 2024 - Not Just Microservices: Rightsize Your Services!
Netherlands OSUG | Sep 30
1.
2.
3.
4. 30 SEPTEMBER 2021
Near Real-Time ERP Integration
Processing High Data Volumes
Ozan Çali
OutSystems Developer @ Product League
ocali@product-league.com
6. 200 shelves in each store
400 stores
15000 articles
Multiple prices for different
units / dates / stores
ERP
• Master of data
• Near-static store & article data
• Frequent updates for shelf plans & prices
6
7. 7
Large central price changes each week
~6M total shelf plans,
available every night
~20M total price records,
available immediately
~15000 articles,
rarely change
Store
employee
Backoffice
admin
Reporter
8. Use Cases
Client wants to have shelf labels printed
when any label field is updated in ERP
8
9. Use Cases
Create jobs to print shelf labels automatically
when there is a change to any article-related data
9
13. Read-Through Cache
1. Try to fetch from cache (DB)
2. If not found or outdated, get from ERP and cache it
13
14. Synchronization Patterns
Direct Integration Cold Cache with batch sync Cold Cache with real-time sync
Article_CS
Article_IS
ERP
Article_CS
Article_IS
ERP
Article_Sync Article_CS
Article_IS
ERP
Article_API
14
15. Direct Integration
Always call ERP?
• Suitable if information changes too often
• Data always up-to-date
• Only good for small amounts
• More latency
• More hits to ERP
• More APIs, one per retrieval use case
• Single point of failure
Article_CS
Article_IS
ERP
15
17. Get changes periodically?
Cold Cache with batch sync • Simple to set up IS
• Enables data manipulation in OutSystems
• Less calls to ERP
• Isolated Sync module
-> consumers of CS is not impacted by the Sync code
Data may be outdated
Article_CS
Article_IS
ERP
Article_Sync
17
19. Get changes as they happen?
Cold Cache with real-time sync
• ERP calls OutSystems when there is a change
• API and IS isolated from CS
• Data always up to date
More alignment between ERP and OutSystems teams
Article_CS
Article_IS
ERP
Article_API
19
call OS about
a change
21. A business rule appears!
Cold Cache with real-time sync
• ERP calls back OutSystems when there is a change
• API and IS isolated from CS
• Data always up to date
More alignment between ERP and OutSystems teams
…Might be insufficient for a high load of changes
“Every Wed morning, we
send all price changes for
all articles & stores”
Article_CS
Article_IS
ERP
Article_API
21
22. Extra flavor: Queueing
Queued real-time sync
Synchronization of data is processed in parallel
Quick insert of update request in a queue
BPT reacts to the inserted queue message
Article_CS
Article_IS
ERP
Article_API
22
23. What if the size of incoming change is very big?
23
24. Divide and conquer!
Queued real-time sync with buckets
Each subset starts a BPT, which in turn calls
back ERP to get the details for changes
A timer gets the first unprocessed staging and
divides it into smaller subsets (= buckets)
Article_Sync
ERP calls our API about a change
Each call to our API is quickly saved as JSON
in a Staging entity
Each returning set from ERP is flat-saved in a
Notification entity with StagingId FK.
Article_CS
Article_IS
ERP
Article_API
When the last bucket is finished with
processing, all records from the Notification
table for that StagingId will be upserted into CS.
24
25. Only IDs of the changes
from ERP to our API
Max no. of allowed articles
in a call to ERP: 2 (from SP)
E.g. there are 5 changes
I II
25
26. In Sync module, we create 3
buckets out of the staging
They each start a BPT each, to
get the details from ERP
III IV
26
27. The retrieved details are flat-saved in a Notification table,
to be merged to cache after the staging is fully processed.
V
27
30. A limitation appears!
“But you can only do 10 requests
to ERP concurrently!”
Queued real-time sync with buckets OS default:
• 10 simultaneous BPT threads
• 20 simultaneous LightBPTs threads
per OS FE server!
Article_Sync Article_CS
Article_IS
ERP
Article_API
30
31. Throttling to the rescue!
Intentional slowing down of the requests to ERP
Control the number of active BPTs
31
32. Define max allowed no. of concurrent buckets in a SP
Subtract no. of currently in-progress buckets
Launch a BPT for each bucket with no. of available threads
Throttling to the rescue!
32
36. Isolation
• Integration Service per independent concept
• Separation of concerns
• Each concept has its own life cycle
• Also separate BPTs for each concept
36
37. Bootstrapping Initial Load
• Need readily available data
for go-live?
o Design proactively
o Reusable actions
37
38. Exception Handling
• Handling errors per use-case
• Lets us to decide
o whether to retry or stop the process instance
o to log the message into right location and notify the
right people
38
41. Data cleanup
• A timer for each concept to
delete records that are older than
X days
• Old data can be backed up to
another archive DB for logging
reasons
41
42. References
• Designing the Architecture of OS Applications
• Integration Patterns for Core Services Abstraction
• Processes (BPT)
• Using MERGE in SQL
42
43.
44. TRANSFER SOLUTIONS
A bit of sass never hurt anyone
1
Getting Sassy with
Outsystems
Getting Sassy with
Outsystems
Joris Albeda
joris.albeda@transfer-solutions.com
45. GETTING SASSY WITH OUTSYSTEMS
|1. What is Sass
|2. But How!
|3. Sassy Outsystems in your team
2 Getting Sassy with Outsystems
46. PART 1: WHAT IS IT
Getting Sassy with Outsystems
3
Syntactically Awesome StyleSheets
47. PART 1: WHAT IS IT
Getting Sassy with Outsystems
4
SCSS: Sassy CSS
THE EXPANSION
48. PART 1: WHAT IS IT
Getting Sassy with Outsystems
5
| Sass is a preprocessor.
| New functionality
| Neater code, which is compiled to CSS!
Sass Compiler
49. WHAT IS THIS FUNCTIONALITY YOU SPEAK OF?
6 Getting Sassy with Outsystems
50. 7 Getting Sassy with Outsystems
PART 1: WHAT IS IT
| Variables
| Nesting
| External files
| Extending, mixins, functions, @if en @for, and much more…
In summary
51. PART 2: BUT HOW?
Getting Sassy with Outsystems
8
A setup for Outsystems and Sass
52. 9 Getting Sassy with Outsystems
PART 2: BUT HOW?
| Visual Studio Code with extensions:
| Live Server for hosting the CSS
| Live Sass Compiler for compiling your Sass
| SCSS IntelliSense for checking your Sass
| Super CSS Inject
| For Chrome
| For Firefox
Ingredients
54. 11 Getting Sassy with Outsystems
PART 3: SASSY OUTSYSTEMS IN YOUR TEAM
Two scenarios
| Scenario 1: Only you manage the Theme
| Scenario 2: Managing the Theme together
55. 12 Getting Sassy with Outsystems
PART 3: SASSY OUTSYSTEMS IN YOUR TEAM
Scenario 1: Only you manage the Theme
56. 13 Getting Sassy with Outsystems
PART 3: SASSY OUTSYSTEMS IN YOUR TEAM
| Only you use Sass
| Communicate clearly to your team!
| Repository is recommended (Git, SVN,
…)
| Team members add new CSS at the
bottom of the Theme CSS
Scenario 1: Only you manage the Theme
57. 14 Getting Sassy with Outsystems
PART 3: SASSY OUTSYSTEMS IN YOUR TEAM
| Only you use Sass
| Communicate clearly to your team!
| Repository is recommended (Git, SVN,
…)
| Team members add new CSS at the
bottom of the Theme CSS
Scenario 1: Only you manage the Theme
You process new CSS in the Sass
58. 15 Getting Sassy with Outsystems
PART 3: SASSY OUTSYSTEMS IN YOUR TEAM
| Agree to use Sass together!
| Repository is needed (Git, SVN, …)
Scenario 2: Managing the Theme together
59. 16 Getting Sassy with Outsystems
CONCLUSION
| Sass: make your CSS better and neater
| InjectCSS: Develop more quickly
| Do as I did, and try it out!
60. 17 Getting Sassy with Outsystems
CONCLUSION
| Sass: make your CSS better and neater
| InjectCSS: Develop more quickly
| Do as I did, and try it out!
61. 18 Getting Sassy with Outsystems
LINKS
| Visual Studio Code with extensions:
| Live Server for hosting the CSS
| Live Sass Compiler for compiling the Sass
| SCSS IntelliSense for checking the Sass
| Super CSS Inject
| For Chrome
| For Firefox
| Sass documentation
| Sass 7-1 project structure
| Need help? Ideas? joris.albeda@transfer-solutions.com
For your convenience
