8. What we are working
with:
• Fully serverless
• Large volume of events
• Sub second performance
• Process raw event data
• Data aggregation, filtering
and sorting
• Kotlin/Java Engineering
team
16. Why Kotlin?
• Fast & efficient execution
• Supports lightweight
concurrency
• AWS SDK Support
• Agnostic of AWS runtime
version
• Runs on ARM 64
• Multiplatform:
JVM/JS/Native(GraalVM)
17. Kotlin/JVM vs Kotlin/JS
vs Kotlin/GraalVM
Benchmark
• Generates DynamoDB
materialized view
• DynamoDB parallel scans
• Lightweight concurrency
• Lambda Power Tuner used
to generate executions
• Compared on different
memory size
22. Conclusion
• For sub-second performance
• Snappy results for data aggregation & filtering
• Best used when access patterns are known
DynamoDB Materialized Views
• Go native!
• Reduce footprint
• Improve performance
Kotlin Lambdas
Good evening! My name is Elena van Engelen and I am a Senior Software Engineer at PostNL. I started my software engineering career 22 years ago…. yes I know, I dont look that old ;). Since then I worked in quite a few fun companies, such as Sun Microsystems which since then merged with Oracle, KPN, and bol.com. And now I am having a lot of fun at PostNL with AWS Serverless cloud solutions. So today I’ll be talking about…
MCP – Mission control platform
But before I dive into our solution, let me give of bit a little context to the problem we are trying to solve. The main actor in this use case is a roll contaner – (no its not a container for ..)
Roll containers are used everywhere at Post NL
Is used to pick up customer packages
Is used to transport packages
Is used to group packages per destination to transport the packages to the distribution centers closest to the final destination of the packages
Can be carried by trucks or rolled manually
Can be folded to save storage when not in use
How we keep track of the roll cages?
More than 505 mil updates p/d
The number of updates and movement of rollcontainers leads me to the PostNL problem
One problem: Where are all the roll containers?
Supply and demand
Do we have too many available roll containers at a given location?
Do we need more roll containers at a given location?
Usage over time and predictions
Do we have enough roll containers for peak season?
Can we relocate roll containers in preparation for peak season?
Lost roll containers
Where are roll containers?
Where were roll containers last seen?
At PostNL we have over 300K Roll containers. But where are they and how are they used? Mission Control Platform aims to gain real time as well as historical insights into where roll containers are and how are they used, and make predictions
One problem: Where are all the roll containers?
Supply and demand
Do we have too many available roll containers at a given location?
Do we need more roll containers at a given location?
Usage over time and predictions
Do we have enough roll containers for peak season?
Can we relocate roll containers in preparation for peak season?
Lost roll containers
Where are roll containers?
Where were roll containers last seen?
At PostNL we have over 300K Roll containers. But where are they and how are they used? Mission Control Platform aims to gain real time as well as historical insights into where roll containers are and how are they used, and make predictions
We want to be able tp process the raw event data, analyse it, aggregate it into dashboards that help users on the floor to plan delivery processes better, streamline delivery processes and prevent any problems or delays. So how d you
* Iot events from sensors of roll cages tell us there location (at a PotsNL location, in a truck, at a customer location)
Scan events give us insites into the products in the roll cages, priority with which roll cage should be handled and process stage it is in
QR code to process delivery, Iot is where they are now and scanning is where they are going to
4 mins
* Full serverless
Team Kotlin/Java
With these contraints
Which brings me to the main topics of my presentation, Dynamo DB materialised views & Kotlin Lambda
(6 mins in)
Considering all these requirements , and because in our case the access patterns are known, we chose dnamodb
for its performance
Maintenance overhead, for example in some simple query changes that in some databases like sequel database could just be generating a new query based on user selection without a need in any code changes. For materialised view such changes would offten require business logic changes that create such a materialised view
Before I go into our reasons for choosing Koltin, I want to briefly introduce the language. Someone once asked me, isn’t Kotlin for front end development of android applications? Actually, Kotlin was invented 11 years ago as a cross-platform general purpose language and in 2019 was afficially adopted by Google for android app development. However it is widely used for backend development by a large number of big and small companies. Examples include: JetBrains, Rabobank, ING, Albert Hein, Jumbo, bol.com, NS , Google, Netflix, and Amazon used it to build Amazon Quantum Ledger Database (QLDB). Kotlin is loved for its clear and concise programming constructs, with less lines of code.
Multiplatform: JVM/JS/Native(GraalVM) , these runtimes perform differently so we decided to compare them. Standard Kotlin Native we are waiting for Kotlin AWS SDK from AWS which is WIP, therefore choose GlaalVM in stad of LLVM for the native runtime because AWS SDK v2 supports it. Kotlin AWS SDK for LLVM based native is working in progress.
GralVM uses static analysis of the code and uses Ahead of Time (AoT) compilation
CDK – infrustructure as code. CDK in Kotlin is not generally available – its still WIP, we are using typecript
You can run Kotlin on multiple platforms, on top of JVM on top of JS, but there are some tradeoffs between them and I’d like to focus on this in the next part of my presentation
We see majod diff , on runtime duration and cold start, and then show the slide,
Add bullet points animation, font bigger less text
Add units
Benchmark is lambda generate materialised view, put to the right
6 mins
1000MB JVM = 15 seconds & JS = 5 seconds & GraalVM 1,5 seconds
4000MB JVM = 5 seconds & JS 2 seconds & graal is 0,5 second , as we add memory the gap reduces
1000MB JS = almost 4 seconds & JVM = 1 second & 900 ms
4000MB JS 1 second & JVM = 0,460 seconds & GraalVM 300 ms
https://aws.amazon.com/blogs/developer/aws-sdk-for-kotlin-alpha-release/ announced
Ussed docker image
This how we are using kotlin lambdas and dynamo
2 flows, real time and reporting. Real time shoes when roll cage has left the building in real time
Dynamocaly generating materialised views by pre processing lambda
AppSynch works very well with dynamo DB , we use Velocity templates
1 min
If you want sub second performance, and access patterns are known, then materialised views are a good option. It does not need to be a front end application that consumes the materialised views, it can also be a backend application that needs the data.
Go Native with Kotlin Lambdas to reduce cold start & improve performance
