Aws architecture problems while being fancy
•1 like•221 views
Download to read offline
Report
Being a pioneer in a field most of the time gives you a boost when you are showing your portfolio. It is enriched with lots of fancy and buzz words and when you talk with your friends and colleagues you sound as you are talking rocket science and you look smart. And that is a good thing. On the other side, you have a lot of hidden traps that you need to overcome when working with new architecture technologies. In this talk I will present the architecture that me and my team created on top of AWS. The architecture currently is serving millions of users on daily basis with 90%+ score in every performance test. It consists of 15+ services that are provided by AWS. The presentation will focus on all of the good things and the problems that we are facing while using AWS services.
Recommended
More Related Content
What's hot
What's hot(20)
Similar to Aws architecture problems while being fancy
Similar to Aws architecture problems while being fancy(20)
Recently uploaded
Recently uploaded(20)
Aws architecture problems while being fancy
- 2. About me Goran Kopevski Tech Lead @ Global Savings Group
- 3. Agenda ▰ Benefits of using AWS Cloud ▰ Fancy selling point ▰ Common design patterns and problems
- 4. What is AWS Marketing eyes: Amazon Web Services (AWS) is a secure cloud services platform, offering compute power, database storage, content delivery and other functionality to help businesses scale and grow. Engineer eyes: ▰ Managed services ▰ Easier way for development and deployment ▰ New architecture horizonts
- 5. But why AWS or any other cloud? Four fundamental principles for cloud: ▰ Fault tolerant systems ▰ Scalability ▰ Elasticity ▰ Cost effective
- 6. What kind of services they are offering
- 7. The good part ▰ Polished services ▻ EC2 ▻ S3 ▻ EB ▻ CF ▻ AWS RDS ▻ …. ▰ If a service gains popularity it gets big investment from AWS
- 8. Challenges ▰ For the sake of having a “service”, let's roll it out ▰ If service is popular -> invest, ▻ if not -> ignore it :) ▰ Stubbornness and simply ignoring requests ▰ Forcing you use their vision about cloud services ▻ Workarounds for other scenarios
- 9. The fancy smart wording ▰ “I am experienced in using Elastic mapReduce for distributed cloud processing of large data sets across clusters of computers using simple programming models” ▰ “I am using DynamoDB which a fast and flexible NoSQL database service for all applications that need consistent, single-digit millisecond latency at any scale”
- 10. The real wording ▰ “I am experienced in using Elastic mapReduce for distributed cloud processing of large data sets across clusters of computers using simple programming models” ▰ In normal (real) wording “I am using Hadoop” ▰ “I am using DynamoDB which a fast and flexible NoSQL database service for all applications that need consistent, single-digit millisecond latency at any scale” ▰ After some experience “I am using simple key value db”
- 11. AWS API Gateway: The good parts ▰ API Caching ▰ API limiter ▻ Example: max 1000 requests to specific endpoint ▰ Support for swagger definition of endpoints ▰ Good security
- 12. AWS API Gateway: Challenges ▰ Multipart requests ▻ Encode image in base64 and send it like that ▰ 10 MB limit payload ▻ Use streaming request ▰ Creation of endpoint ▻ Swagger custom parameters
- 13. Regions problem: The good parts ▰ Regions on every continent ▻ Closer to your clients ▰ Multiple availability zones per region ▰ Main power of the AWS infrastructure ▻ Prerequisite for fault tolerant systems
- 14. Regions problem: The challenges ▰ Some services available but not all ▻ First N.Virginia and Ireland then move it to other regions ▰ Real world scenarios: ▻ DynamoDB caching ▻ DynamoDB backup ▻ CodePipeline ▻ AWS Fargate ▻ … ▰ https://aws.amazon.com/about-aws/global-infrastructure/regional-product-services/
- 15. Dynamo: The good parts ▰ Fast write ▰ Fast read ▻ Under some conditions ▰ Autoscaling is managed by AWS ▰ You pay for throughput ▻ number of request ▻ speed for writing/reading ▻ You can have 100000000…. TB of data
- 16. Dynamo: The challenging part! ▰ For every simple query you need to write a lot of code instead of “1 liner” ▰ SELECT * FROM X WHERE Status=’Published’ AND date>:date:
- 17. Dynamo: Even more challenges ▰ If you want to query by other parameters (not primary key), you need indexes ▻ Dynamo supports up to 5 indexes :) ▰ Versioning does not work with batch write ▻ You need to handle it yourself ▻ https://github.com/bchew/dynamodump ▰ https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Limits.html (racism link)
- 18. Dynamo problems with backup using AWS EMR ▰ Solution provided by AWS was to use new AWS service (AWS EMR, (Hadoop)) ▻ https://aws.amazon.com/blogs/aws/aws-howto-using-amazon-elastic- mapreduce-with-dynamodb/ ▰ The bad part is it was not working consistently ▻ In a test scenario we restored 80% of the data
- 19. Dynamo solution for backup From November 2017 AWS started to support dynamo table backup as a managed service
- 21. Cloudformation templates ▰ Code as infrastructure ▰ Love hate relationship service ▰ If you use it properly and understand you have incredibly good tool ▻ If not then you will hate it ▰ Interesting limitation if you don’t pay much attention ▻ 200 resources max per template
- 22. Lambda λ Pros: ▰ FaaS ▻ Pay per execution ▰ No scaling problems ▰ Operational management ▰ Faster innovation Cons: ▰ No control over environment ▰ Lack of operational tools ▰ Architectural complexity
- 23. AWS SQS ▰ Amazon Simple Queue Service (SQS) is a fully managed message queuing service that makes it easy to decouple and scale microservices, distributed systems, and serverless applications. ▰ Nearly unlimited number of transactions per second ▻ 120,000 inflight messages in a queue ▰ Only 1 bad word => Limits: ▻ Activemq 8Gb ▻ RabitMQ 2Gb ▻ AWS SQS 256KB ▰ https://stackshare.io/stackups/amazon-sqs-vs-kafka-vs-rabbitmq
- 24. Cloudwatch: The good parts ▰ Out of the box integration with AWS ▻ SNS/SQS ▻ Logging ▻ Lambda ▻ ... ▰ Monitoring tool ▰ Supports for multiple type of notifications
- 27. CodePipeline: The good parts ▰ Super easy setup! ▰ Good integration in AWS ecosystem
- 28. CodePipeline: The challenges ▰ Integration with 3rd party goes with custom lambda ▻ Lambda for sonar (community) ▻ Lambda for github (community) ▰ No parameterized builds ▰ Code Pipeline Monitoring
- 29. The custom lambda problem ▰ If you need to tune the system to the way you want to work in AWS system easiest way is with Custom Lambda! ▰ Example: ▻ Integration of Sonar with CodePipeline ▻ Integration of Github builds in CodePipeline ▻ Sending logs from Cloudwatch to ElasticSearch https://forums.aws.amazon.com/thread.jspa?threadID=227681
- 30. AWS ES: The good parts ▰ Managed service ▰ Easy setup ▰ Integration with AWS ecosystem ▻ IAM Roles ▻ Kinesis ▻ EC2 instances
- 31. AWS ES: Challenges ▰ Transport protocol is disabled ▰ Only HTTP requests ▻ https://forums.aws.amazon.com/thread.jspa?messageID=784997 ▰ Sometimes returns 500 :) ▰ Out of the box automatic autoscaling is not supported
- 32. Conclusion ▰ Consult/Research before choosing specific AWS service ▰ Managing whole infrastructure is easy with AWS ▰ If you don’t have very specific requirements go with AWS
- 33. THANKS! Any questions? You can find me at gkopevski@gmail.com
Editor's Notes
- Few words about me Tech lead at Intertec, part of GSG, Responsible for the architecture and well being of the Travel domains
- All of the things i am going to say today are real scenarios that we encountered while using AWS. To mention explicitly this is not hating presentation but rather realistic objective summary about pros/cons of some of the AWS services
- Now what is the good part. The thing I have concluded while working with AWS is
- Big portfolio And if you are not able to find better workaround you are going to use their
- I promised that i will be fancy so lets go: https://aws.amazon.com/emr/
- I promised that i will be fancy so lets go: https://aws.amazon.com/emr/ The point here is some devops/architect comes and start ...
- First lets sepak about the service itself: Good (few good words)
- First lets sepak about the service itself: Good (few good words)
- Cor
- https://www.nordcloud.com/tech-blog/aws-dynamodb-design-considerations
- https://www.nordcloud.com/tech-blog/aws-dynamodb-design-considerations
- https://www.nordcloud.com/tech-blog/aws-dynamodb-design-considerations
- https://cloudonaut.io/cloudformation-vs-terraform/
- https://www.quora.com/What-is-the-advantages-and-disadvantages-of-using-AWS-Lambda-with-and-without-Serverless
- This is really intresting example of not thinking out of the box.
- https://aws.amazon.com/cloudwatch/
- Filebeat is an open source file harvester, mostly used to fetch logs files and feed them into logstash. Logstash is a log pipeline tool that accepts inputs from various sources, executes different transformations, and exports the data to various targets. Elasticsearch is a distributed, RESTful search and analytics engine based on the Lucene search engine. Kibana is a visualization layer that works on top of Elasticsearch
- If you go under the hood and inspect some of the lambdas you will notice https://aws.amazon.com/kinesis/data-firehose/