Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to [Heap con19] designing data intensive applications in serverless architecture
Similar to [Heap con19] designing data intensive applications in serverless architecture (20)
Recently uploaded
Recently uploaded (20)
[Heap con19] designing data intensive applications in serverless architecture
- 1. 1@matvi3nko DESIGNING DATA-INTENSIVE APPLICATIONS IN SERVERLESS ARCHITECTURE
- 2. Nikolay Matvienko Senior Software Engineer and Node.js expert at Grid Dynamics Diagnostics, performance improvement consultant. Work in USA and in Russia Designed serverless computing platforms with AWS You can find me at twitter.com/matvi3nko github.com/matvi3nko 2
- 3. 1. SERVERLESS COMPUTING HAS BECOME VERY POPULAR 2. SERVERLESS SOLUTIONS ARE ~60% AND UP TO SEVERAL TIMES CHEAPER 3@matvi3nko DESIGNING SERVERLESS 3. EVERY YEAR AWS ANNOUNCES NEW SERVICES 4. BUT THE LACK OF PATTERNS AND MISUSE CAN MAKE THE SOLUTION SEVERAL TIMES MORE EXPENSIVE FaaS containers 21% 19%
- 4. 4@matvi3nko DATA‑INTENSIVE APPLICATIONS DATA ETL PIPELINE - EXTRACT - TRANSFORM - LOAD DATA WAREHOUSE ANALYSE VISUALISE STEP 1 STEP 2 ETL JOB IS DONE? SCALABILITY
- 5. 5@matvi3nko WHERE DOES THE DATA COME FROM? IoT DEVICES REAL-TIME STREAM PROCESSING ENTERPRISE PLATFORMS BATCH / STREAM PROCESSING DB DBDATA LAKE
- 6. 6@matvi3nko USE EXISTING OR BUILD YOUR OWN PIPELINE DATA DATA WAREHOUSE GLUE JOBs KINESIS STREAMS & ANALYTICS ETL BATCH PROCESSING STREAM PROCESSING AWS LAMBDA
- 7. 7@matvi3nko DYNAMICALLY UPDATED PROCESSING LOGIC AWS LAMBDA DATA WAREHOUSE PROCESSING RULES PROCESSING RULES PROCESSINGRULES PROCESSING RULES PROCESSING RULES PROCESSING RULES PROCESSING RULES REAL-LIFE CASE: 300+ CUSTOMERS EACH ESTABLISHES HIS OWN DATA PROCESSING RULES
- 8. 8@matvi3nko DYNAMICALLY UPDATED PROCESSING LOGIC AWS LAMBDA DATA WAREHOUSE PROCESSING RULES PROCESSING RULES PROCESSINGRULES PROCESSING RULES PROCESSING RULES PROCESSING RULES PROCESSING RULES REAL-LIFE CASE: 300+ CUSTOMERS EACH ESTABLISHES HIS OWN DATA PROCESSING RULES
- 9. 9@matvi3nko SERVERLESS COMPUTE SERVICE AWS LAMBDA SERVERLESS FUNCTION * Symbol in the presentation export const handler = async (event) => { const data = event.Records[0].body; // - TRANSFORM data // - WRITE to DB or // - PUT TO QUEUE/STREAM return 'success'; }; - CHOOSE YOUR CODE LANGUAGE - NO INFRASTRUCTURE TO MANAGE - TRIGGERED BY EVENTS - HIGH SCALABLE - STATELESS - COST-EFFECTIVE
- 10. 10@matvi3nko AWS S3 FOR DATA LAKE NEW DATA S3
- 11. 11@matvi3nko BATCH DATA PROCESSING SNAPSHOT: /yyyy=2019/mm=09/dd=27/ TABLES/FILES FUNCTION MERGED RECORD from tables/files READ FILES MERGE RECORDS FUNCTION PROCESS/ANALYZE MODEL DB DONE! FUNCTION SCAN FOLDERS GET ALL FILES REPLACED ONAWS LAMBDA ICON
- 12. 12@matvi3nko TRANSPORT SERVICES NOTIFICATION TOPIC FAN-OUT - PUB/SUB - NOT WORRIED ABOUT DELIVERY DATA STREAMS KINESIS STREAMS - BATCH 1–10000 RECORDS - NOT AUTOSCALE - 6 MB MESSAGE SIZE LIMIT - EXACTLY ONCE PROCESSING - HOURLY СOST REAL-TIME PROCESSING QUEUE - BATCH 1–10 RECORDS - AUTOSCALE - 256 KB MESSAGE SIZE LIMIT - AT LEAST ONCE PROCESSING BATCH PROCESSING DYNAMODB NoSQL DB DYNAMODB STREAMS DISTRIBUTED TRANSACTIONS - BATCH 1–1000 RECORDS - AUTOSCALE - 6 MB MESSAGE SIZE LIMIT - EXACTLY ONCE PROCESSING - REACT ON EACH CHANGE
- 13. 13@matvi3nko BATCH PROCESSING ARCHITECTURE SNAPSHOT: /yyyy=2019/mm=05/dd=25/ TABLES/FILES PATHS FUNCTION PROCESS/ANALYZE MODEL DBFUNCTION MERGED RECORD from tables/files READ FILES MERGE RECORDS FUNCTION SCAN GET FILES FOR EACH FOLDER QUEUENOTIFICATION TOPIC QUEUE QUEUE
- 14. 14@matvi3nko REAL-TIME PROCESSING ARCHITECTURE FUNCTION PROCESS/ANALYZE MODEL DB MERGED RECORD from tables/files DATA STREAMS SNAPSHOT: /yyyy=2019/mm=05/dd=25/ TABLES/FILES for each folder FUNCTION READ FILES MERGE RECORDS FUNCTION SCAN GET FILES FOR EACH FOLDER QUEUENOTIFICATION TOPIC QUEUE
- 15. 15@matvi3nko DATA EXTRACTION PATTERNS 1. MOVE FROM BIG DATA TO A LARGE NUMBER OF MESSAGES 2. USE THE QUEUES FOR MESSAGES, AND DATA STREAMS TO TRANSFER MODELS / LARGE COLLECTION 3. BUT DO NOT RUSH TO USE STREAMS. CHOOSE TRANSPORT FOR YOUR NEEDS
- 16. 16@matvi3nko DATA TRANSFORMATION DATA ETL PIPELINE - EXTRACT - TRANSFORM - LOAD DATA WAREHOUSE ANALYSE VISUALISE STEP 1 STEP 2 ETL JOB IS DONE? SCALABILITY
- 17. 17@matvi3nko TRANSFORM SECTION SNAPSHOT: /yyyy=2019/mm=05/dd=25/ TABLES/FILES for each factory FUNCTION PROCESS/ANALYZE MODEL DBFUNCTION MERGED RECORD from tables/files READ FILES MERGE RECORDS FUNCTION SCAN GET FILES FOR EACH FACTORY QUEUENOTIFICATION TOPIC QUEUE QUEUE
- 18. 18@matvi3nko SEPARATE FUNCTIONS BY RESPONSIBILITY FUNCTION PROCESS/ANALYZE MODEL MERGED RECORD from tables/files QUEUE MERGER FAN-OUT COPIER SPLITTER WRITER FILTER
- 19. 19@matvi3nko BOUNDARY CONTEXT MERGER SPLITTER WRITER FILTER S3 DB DATA STREAMS KINESIS DATA ANALYTICS QUEUE CONTEXT 1 DATA STREAMS WRITER CONTEXT 2 DATA STREAMS PROCESS CONTEXT 4 CONTEXT 3 DATA STREAMS
- 20. 20@matvi3nko QUERIES & CONNECTIONS PROBLEMS TRANSFORM 1 TRANSFORM 2 TRANSFORM 3 SQL DB SELECT INSERT PROBLEMS: 1. MANY LAMBDAS HAS TO QUERY SQL DB 2. A LOT OF CONNECTIONS 3. A LOT OF DEPENDENCIES UPDATE
- 21. 21@matvi3nko DB QUERY LOGIC ENCAPSULATION TRANSFORM 1 TRANSFORM 2 TRANSFORM 3 GET PUT BENEFITS: 1. QUERIES & LOGIC IS HIDDEN BEHIND THE API 2. LESS CONNECTIONS, CONTROLLED CONNECTIONS POOL 3. LESS DEPENDENCIES UPDATE SQL DB API
- 22. 22@matvi3nko RELIABILITY: RETRY STRATEGY New SNAPSHOT DATE QUEUE QUEUE FILES PATHS SCAN ALL OBJECTS FOR EACH FOLDER DATA LAKE QUEUE MODELS GET ALL RECORDS AND AGGREGATE EACH WITH DETAILS WRITEDISPATCH DATABASE TRANSFORM RETRIES 3 TIMES (BY DEFAULT) logo
- 23. 23@matvi3nko DEAD LETTER QUEUE ... TRANSFORM RETRIES 3 TIMES 1 2 returns Error DLQ STORES FAILED MESSAGES 3 ACTOR REPROCESS PULLS MESSAGES FROM DLQ BY THE REQUEST 5 6 7 TOPIC ... 4 REPROCESSING TOPIC ”reprocess_[type]_errors”
- 24. 24@matvi3nko KINESIS ERROR HANDLING ... ... RETRIES N TIMES 24h - 7d ANALYZE FUNCTIONINFRASTRUCTURE ERRORS reject(error) resolve() SEND RECORD TO SNS TOPIC DLQ STORES FAILED RECORDS BUSINESS ERRORS INTERNAL ERRORS REPROCESS PULLS MESSAGES FROM DLQ BY THE REQUEST Pull by error type
- 25. 25@matvi3nko DLQ FOR THE QUEUE New SNAPSHOT DATE QUEUE QUEUE OBJECTS PATHS SCAN ALL OBJECTS FOR EACH FOLDER DATA LAKE QUEUE MODELS GET ALL RECORDS AND AGGREGATE EACH WITH DETAILS ANALYZEDISPATCH MERGE BY FACTORY DATABASE TRANSFORM ACTOR REPROCESSING TOPIC ”reprocess_messages” REPROCESS PULLS MESSAGES FROM DLQ BY THE REQUEST DLQ TOPIC 1. EASY TO TROUBLESHOOT 2. NO NEED TO REPROCESS GIGABYTES OF DATA AGAIN
- 26. 26@matvi3nko DATA TRANSFORMATION PATTERNS 1. ONE LAMBDA FUNCTION – ONE RESPONSIBILITY 2. DIVIDE THE PIPELINE INTO BOUNDARY CONTEXTS WITH FIXED DATA INTERFACES 3. ENCAPSULATE DB QUERIES BEHIND API FUNCTION 4. USE DEAD LETTER QUEUES FOR RELIABILITY
- 27. 27@matvi3nko DATA LOADING DATA ETL PIPELINE - EXTRACT - TRANSFORM - LOAD DATA WAREHOUSE ANALYSE VISUALISE STEP 1 STEP 2 ETL JOB IS DONE? SCALABILITY
- 28. 28@matvi3nko New SNAPSHOT DATE QUEUE QUEUE FILES PATHS SCAN ALL OBJECTS FOR EACH FOLDER DATA LAKE DATA STREAMS RECORDS GET ALL RECORDS AND AGGREGATE EACH WITH DETAILS ANALYZE & WRITE DISPATCH MERGE BY TYPE DATABASE TRANSFORM ACTOR REPROCESSING TOPIC ”reprocess_messages” REPROCESS PULLS MESSAGES FROM DLQ BY THE REQUEST DLQ TOPIC LOAD SECTION
- 29. 29@matvi3nko WRITING TO A DATABASES DBLAMBDA MICROSERVICE PROCESSES DATA ANR/OR FILTERS DATA WRITES TO DB Points of failure Points of failure LAMBDA UI DASHBOARD ACCOUNTING FURTHER DATA PROCESSING 1 – 10K MESSAGES 1. RETRY THE BATCH? 2. RETRY FAILED RECORD MANUALLY IN LAMBDA? 1 – 10K MESSAGES
- 30. 30@matvi3nko RETRY PROBLEM LAMBDA UI DASHBOARD ACCOUNTING FURTHER DATA PROCESSING 1 – 10K MESSAGES 1. [BAD] REMOVE/ROLLBACK SUCCESSFUL RECORDS DUPLICATE DUPLICATE 2. [BAD] ORCHESTRATE REPEAT IN LAMBDA
- 31. 31@matvi3nko DECOUPLED WRITERS DB 1 PROCESSING ERRORS INFRASTRUCTURE ERRORS DB 2 S3 DATA LAKE QUEUE QUEUE QUEUE PROCESSOR / FILTER WRITER WRITER WRITER INDEPENDENT RETRIES BETTER SCALABILITY1000 INSTANCES 500 INSTANCES 100 INSTANCES 200 INSTANCES
- 32. 32@matvi3nko DISTRIBUTED TRANSACTIONS DB 1 DB 2 S3 DATA LAKE QUEUE QUEUE QUEUE PROCESSOR / FILTER WRITER WRITER WRITER HOW TO ROLLBACK OTHERS?
- 33. 33@matvi3nko USE AWS STEP FUNCTIONS DB 1 DB 2 S3 DATA LAKE PROCESSOR / FILTER WRITER WRITER WRITER AWS STEP FUNCTIONS TRANSACTION INSTRUCTIONS
- 34. 34@matvi3nko DISTRIBUTED TRANSACTIONS DYNAMODB NoSQL DB QUEUEPROCESSOR / FILTER WRITER 1 DYNAMODB STREAMS S3 DATA LAKE AURORA SQL DB DYNAMODB PUSHES EVENT TO STREAM ON INSERT | MODIFY | REMOVE RECORD IN DB WRITER 2 / STREAM SUBSCRIBER 1 WRITER 3 / STREAM SUBSCRIBER 2
- 35. 35@matvi3nko DATA WAREHOUSE S3 DATA LAKE AURORA SQL DB DYNAMODB NoSQL DB DB WRITER NEPTUNE GRAPH DB CONSUMERS DATA ACCESS SPEED? DATA ACCESS PATTERNS? QUERIES? ANTI-FRAUD RISKS FORECAST MACHINE LEARNING
- 36. 36@matvi3nko DATA LOADING PATTERNS 1. DECOUPLE FUNCTIONS BY POINTS OF FAILURE 2. USE INFRASTRUCTURE AS A CODE VS ORCHESTRATIONS IN THE CODE 3. USE AWS STEP FUNCTIONS AND DYNAMODB STREAMS FOR TRANSACTIONS 4. PIPELINE IS THE CORE, THINK ABOUT FUTURE PLATFORMS AROUND IT AND HOW THEY WILL HAVE ACCESS TO DATA
- 37. 37@matvi3nko DATA ETL PIPELINE - EXTRACT - TRANSFORM - LOAD DATA WAREHOUSE ANALYSE VISUALISE STEP 1 STEP 2 ETL JOB IS DONE? SCALABILITY HOW TO UNDERSTAND THAT THE JOB IS COMPLETED?
- 38. 38@matvi3nko THE PROBLEM OF DISTRIBUTED DATA PROCESSING DATA ETL PIPELINE EXTRACT TRANSFORM LOAD DATA WAREHOUSE STREAM PROCESSING PIPELINE DATA WAREHOUSE WRITE IN PARALLEL WRITE IN PARALLEL
- 39. 39@matvi3nko ETL JOB STATE CONTROL New SNAPSHOT DATE QUEUE QUEUE FILES PATHS SCAN ALL OBJECTS FOR EACH FOLDER QUEUE MODELS GET ALL RECORDS AND AGGREGATE EACH WITH DETAILS WRITEDISPATCH DATABASE TRANSFORM DYNAMODB JOB TABLE DYNAMODB PROCESSED RECORD TABLE DYNAMODB STREAMS JOB IS NOT DONE && COUNT OF IS NOT DONE RECORDS ===0 START JOB IS DONE! END
- 40. 40@matvi3nko DATA ETL PIPELINE - EXTRACT - TRANSFORM - LOAD DATA WAREHOUSE ANALYSE VISUALISE STEP 1 STEP 2 ETL JOB IS DONE? SCALABILITY DATA ANALYSIS
- 41. 41@matvi3nko ORCHESTRATE MULTIPLE ETL JOBS USING AWS STEP FUNCTIONS AND AWS LAMBDA ETL JOB STATE DONE / NOT DONE ETL JOB RUNNER JOB STATE CONTROLLER READER AWS STEP FUNCTIONS INSTRUCTIONSSNS TOPIC MESSAGE CLOUD WATCH CHRON EVENT S3 DATA LAKE STRUCTURED DATA GLUE JOB ETL JOB ANALYSER STEP 1 STEP 2 STEP 3 AWS LAMBDA ETL
- 42. 42@matvi3nko DATA ETL PIPELINE - EXTRACT - TRANSFORM - LOAD DATA WAREHOUSE ANALYSE VISUALISE STEP 1 STEP 2 ETL JOB IS DONE? SCALABILITY DATA ANALYSIS
- 43. QUEUE 43@matvi3nko DATE OBJECTS/FILES PATHS ANALYZED RECOFILES DB SNAPSHOT: /yyyy=2019/mm=05/dd=25/ FUNCTION DBFUNCTIONFUNCTION QUEUENOTIFICATION TOPIC QUEUE SCALABILITY ~100 ~1000 ~10
- 44. 44 @matvi3nko BLUEPRINT ETL JOB STATE DONE / NOT DONE ETL JOB RUNNER JOB STATE CONTROLLERAWS STEP FUNCTIONS SNS CLOUD WATCH CHRON EVENT S3 DATA LAKE STRUCTURED DATA GLUE JOB ETL JOB JOB STATE REDIS ETL BATCH PROCESSING AD-HOC STREAM PROCESSING AURORA DYNAMO KINESIS DLQ DLQ READER PROCESSORS DYNAMO JOBS DLQ 5K – 10K FUNCTION IN PARALLEL PROCESSES 10-100 GB/SEC
- 45. 45@matvi3nko CONCLUSIONS 1. DESIGN MISTAKES = SOLUTION PRICE 2. IN SERVERLESS YOU CAN BUILD VERY BUILD A VERY FLEXIBLE ARCHITECTURE YOU CAN SWITCH FROM BATCH TO STREAM PROCESSING 3. YOU CAN RUN THOUSANDS OF LAMBDA FUNCTIONS IN PARALLEL AND PROCESS GBs OF DATA PER SEC
- 46. THANKS! 46 Nikolay Matvienko Grid Dynamics We are now in Belgrade too. You can find me at twitter.com/matvi3nko github.com/matvi3nko