Navigate Data Service using AWS presentation from our meetup @ Port of Rotterdam

1. Arno Broekhof
David Lopez Penha
Serverless
`on Land, Sea and Cloud`

2. Let’s Meet
Arno Broekhof
David Lopez Penha
Data Engineer
Data Engineer

3. Disclaimer
- This is not a talk about serverless and AWS is great,
but a talk about our experience of going cloud native. -
- Our opinions are our own -

4. product
architecture
event-drivencloud-native

5. !5
Navigate
Find & compare all deep-sea,
short-sea, rail & barge
connections

6. Why Navigate?
• No overview of all possible connections
or transit possibilities
• Insufficient transparency in logistic
chains
• Fragmented offer of service providers
in the port
• 25% of all transported containers are
empty
• Insufficient insight in footprint of
connections
• What are my options via Rotterdam?

7. EXAMPLE: Connect Rotterdam with Shanghai

8. Current Setup Huidige Situatie
Data
Service Backend
Frontend
&
Backend
Route
Calculation
Route calculator
2 HTTP Rest Endpoints:
/schedules
/companies

9. Data Service Back-End

10. Data
Providers
Client
Supervisor Node Spark Cluster
API Gateway Postgres DB
Data Service Back-End (On Premise)

11. Challenges of Scaling Up
On Premise:
• Typically `always-on` architecture
• High idle times
• Upgrades / updates are slow to role out
• High probability of conflict in multi-tenant systems
• Product isolation is difficult to manage

12. Initial Setup (AWS Cloud)
$ 11.129,25 per month

13. Rethinking Architecture - Requirements?
• Serve: 2 HTTP Get Endpoints
• Ingest: External data suppliers
• Security: Encryption at rest & in-transit
• Scheduling: 2x Daily and ON-Demand
• Performant at scale

14. Client Endpoint API Gateway Lambda DynamoDB
API Gateway, Lambda function & DynamoDB

15. Severless Application Model
“AWS Serverless Application Model (AWS SAM) extends AWS CloudFormation
to provide a simplified way of defining the Amazon API Gateway APIs, AWS
Lambda functions, and Amazon DynamoDB tables…”
https://aws.amazon.com/about-aws/whats-new/2016/11/introducing-the-aws-serverless-application-model/
Serve

16. SAM Template Specification

17. Data Ingesting ( ETL )
GLUE EMR DYNAMODB
S3
LAMBDA
EXTERNAL
DATA PROVIDER
FTPS
CRAWLER
CLOUDWATCH
SCHEDULER

19. Can it be simpler?
BATCH
ECS
S3
CONTAINER - INGEST CONTAINER - TRANSFORM
DYNAMODB
CLOUDWATCH
EXTERNAL
DATA PROVIDER

20. Yeah cool, but what about testing?

21. Integration Testing
Spin-up containers before running IT Tests
• S3
• DynamoDB
• Test FTP Server
• SAM local API

22. OK, how do you deploy?

23. Deploy - AWS CloudFormation

24. Current Architecture
Data Providers
Client
DevOps
CloudFormation
Elastic IP
NAT Gateway
Batch
S3
Lambda DynamoDBAPI Gateway
Endpoints
CloudWatch

25. Lessons Learned

26. • Read the docs!
• There is a limit on everything
• Multiple ways to reach you’re goal
• Choose the right service
• YAML Indentations
Lessons Learned

27. Questions?

28. Thanks for your time.
Got feedback? Please share!
all pictures belong
to their
respective owners
@broekhofa arnobroekhof
david-j-lopez-penha