Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Nils Mohr & Jake Pearce - 100 years of flight data at British Airways. Past, present, & future
Similar to Nils Mohr & Jake Pearce - 100 years of flight data at British Airways. Past, present, & future (20)
More from AWSCOMSUM
More from AWSCOMSUM (20)
Recently uploaded
Recently uploaded (20)
Nils Mohr & Jake Pearce - 100 years of flight data at British Airways. Past, present, & future
- 1. 100 years of flight data at British Airways past, present, and the future
- 2. Agenda 1. Our Heritage 2. Engineering data 3. Let’s talk AWS 4. Expanding beyond ai2
- 3. Our Heritage
- 8. Techlog data
- 9. ACARS QX QUKMEBA LONKFBA .LONAXBA 221134 OFF01 BA414 G-NEOV 221134 1134DIA02EGLLEDDT
- 10. Flight data
- 11. Let’s talk AWS
- 12. The architecture in 2016 Database FrontendBackend & Processing Rest API
- 13. Every good story starts in S3 British Airways S3 Storage On prem software/tools Transfer into AWS AWS Cloud
- 14. First steps – Lambda
- 15. Private subnet The first step S3 Storage AWS Cloud Data cleaning and optimization Meta data generation Problem evaluation Amazon Aurora Amazon DynamoDB SQS SQS SQS VPC Amazon API Gateway
- 16. Infrastructure as code AWS CloudFormation
- 18. CI/CD? No AWS toolkit for automatically deploying Lambdas from git We used boto3 script to deploy Lambdas And filled our 75GB of code storage within a week
- 19. Private subnet Nobody likes to manage EC2s S3 Storage Problem evaluation AWS Cloud Amazon Aurora Amazon DynamoDB VPC Amazon API Gateway Data cleaning and optimization Meta data generation SQS SQS SQS
- 21. Docker to the rescue FROM amazonlinux:latest RUN pip install requirements.txt COPY . . CMD [ "python", "./hello_aeroplane.py" ]
- 22. The road to Fargate • CodePipeline and CodeDeploy seamlessly integrate with ECR and Fargate • Scaling between zero and 50 task works easily • No maintenance of the underlying infrastructure ECS Fargate Kubernetes
- 23. The road to Fargate - downsides • Your logs are in CloudWatch – With delays • Fargate is more expensive than Kubernetes - but you don’t need to understand Kubernetes
- 24. Private subnet Containers S3 Storage AWS Cloud Data cleaning and optimization Meta data generation Problem evaluation Amazon Aurora Amazon DynamoDB SQS SQS SQS VPC Amazon API Gateway
- 25. Happy now?! Meta data generation Problem evaluation SQS A320 B787 B747 A380 B787 – monitor 2 B747 – monitor 1 B747 – monitor 2 A380 – monitor 1 A320 – monitor 1 A320 – monitor 2 A320 – monitor 3 B787 – monitor 1
- 26. Solutions? Fargate with autoscaling based on SQS Lambda triggered by SQS
- 27. Private subnet Lambda Layers S3 Storage AWS Cloud Data cleaning and optimization Meta data generation Problem evaluation Amazon Aurora Amazon DynamoDB SQS SQS SQS VPC Amazon API Gateway
- 28. Lambda Layers Without Layers hello_aeroplane.py /BA_ai2 /Numpy …… Lambda.zip With Layers hello_aeroplane.py /BA_ai2 /Numpy …… Lambda.zip Ba_ai2_layer.zip Numpy_layer.zip
- 29. CloudFormation and Lambda Layers Parent Stack Function 1 Function 4 Function 2 Function 3 Output - Layer1 - Layer2 Parent Stack Function 1 Function 4 Function 2 Function 3 Parameter Store Output - Layer1 - Layer2
- 32. Data is in our DNA (70 years of data collection and analysis) Data informs all of our business and operational decisions Digital is an attitude 2019: British Airways is celebrating 100 years of aviation
- 33. We need to integrate cloud and legacy platforms Flight Departing Arriving Dur Frm BA001 LHR LAS 602 747 BA002 LAS LHR 690 747 BA003 LHR CDG 110 A320 BA004 CDG LHR 111 A320 BA005 LGW GLS 92 A320 BA006 GLS LGW 87 A320 BA007 LCY ROM 210 777 BA008 ROM LCY 198 777 Things Analysts Business Users Collect Store Use Systems
- 35. Thank you