Download as PDF, PPTX
Uploaded byClaudio Pontili
beSharp a serverless approach to big data on aws
Claudio Pontili, a senior cloud solution architect at beSharp, presented on using serverless architectures for big data on AWS. He discussed using Lambda for ETL processes and Glue for managed ETL jobs. He also covered CI/CD for deploying Lambda and Glue code, data warehousing on Aurora Serverless v1, and a fully serverless big data architecture. Some key learnings included using serverless for high availability and scalability with no effort, pausing Aurora Serverless v1 clusters when not in use, and using infrastructure as code to deploy architectures.
More Related Content
Similar to beSharp a serverless approach to big data on aws
![Jumpstart your idea with AWS Serverless [Oct 2020]](https://cdn.slidesharecdn.com/ss_thumbnails/jumpstartyourideawithawsserverlessoct2020-240904053409-07f4072b-thumbnail.jpg?width=640&height=640&fit=bounds)
beSharp a serverless approach to big data on aws
- 1. AWS SummitEMEA Claudio Pontili AWSSenior Cloud Solution Architect Claudio.pontili@besharp.it AServerless approachtoBigDataonAWS
- 3. Claudio Pontili 10+ yearsof experience on AWS Senior Cloud Solution Architect AWS Authorized Instructor Champion Claudio.pontili@besharp.it https://www.linkedin.com/in/claudiopontili/
- 4. Agenda • Using Lambdafor ETLs • Glue ETLs • CI/CD to deploy code inside Lambdas and Glue Jobs • Datawarehousing on Aurora Serverless v1 • A full serverless Big Data Architecture • What we’ve learned
- 5. Using Lambda forETLs 1/2 • You can use Python + Panda library • A lambda can have 10 GB of memory and a lot of CPU power • A lambda can run for 15 minutes • Max deployment package 50 MB (zipped) • Container image code package size 10 GB • /tmp directory storage 512 MB
- 6. Using Lambda forETLs 2/2
- 7. Glue Jobs, DataCatalog and Crawler 1/2 • Fully managed Data Catlog and Extract-Transform-Load (ETL) service • Automates data discovery, conversion, mapping and job scheduling • Glue runs your ETL jobs in an Apache Spark serverless envinronment • Allow to scale your ETLs jobs • Can easily schedule a crawler to to create a catalog of files stored on S3 • Too much code? Try Glue Databrew
- 8.
- 9. Glue Jobs, DataCatalog and Crawler 2/2
- 10.
- 11. RDS Aurora Serverless •MySql and PostgreSQL supported (reuse the experience of your team) • Pay per ACU/hours (2 GB of memory) • Scales from 1 to 256 ACU • You can pause the cluster during the night • Aurora Serverless v2 in preview for MultiAz, Read-Replicas, faster scale
- 12. Glue Jobs, DataCatalog and Crawler 2/2
- 13. What we’ve learned •Serverless gives you High Availability and great scalability with no effort • Pause Aurora Serverless v1 (it will take about 30 seconds to restart) • Use IaC (Cloudformation, Terraform, CDK, etc) to deploy your infrastructure • Tune your lamba memory using https://github.com/alexcasalboni/aws- lambda-power-tuning • S3 is cheap but try not to write tiny (<128KB) files • Serverless can be pretty cheap if it’s used in the right way
- 14.
- 15. www.besharp.it info@besharp.it +39 0382 1692920 beSharpsrl - viale Ludovico il Moro 27 - 27100 Pavia (ITALY) VAT ID IT02415160189 Follow beSharp on