SlideShare a Scribd company logo

Skyscanner Engineering: Running aiohttp at scale by Pau Freixes

Skyscanner Hotels is built on top of Aiohttp and has been chosen as one of the formally supported frameworks by the company. We recently presented this talk which tackles how Aiohttp is used, what we have learned and how we are trying to generalize some implementations and patterns to be used by all micro services from scratch at Skyscanner. The talk will cover also essential things related with Asyncio, HTTP protocol, and the AWS infrastructure. All of them are essential to understanding some of the points that are covered during this talk.

1 of 42
Download to read offline
Running Aiohttp at
scale
by Pau Freixes
● About me
● Skyscanner and aiohttp, why?
● Tracing incoming requests
● Calling other microservices
● DNS in AWS with Aiohttp
● Misleading timeouts, the reactor saturation side effect.
● Desired plans
● Questions
Running Aiohttp at scale @SkyscannerEng
Pau Freixes @pfreixes
● Senior Software Engineer working at Skyscanner for almost 2 years.
● Member of the Hotels Attachment Squad.
● Also collaborator of Mshell Squad. Helping with the Python stuff.
● Open source committer aioredis, aiohttp, etc.
Running Aiohttp at scale @SkyscannerEng
Skyscanner and aiohttp, why?
Running Aiohttp at scale @SkyscannerEng
Running Aiohttp at scale @SkyscannerEng
● Skyscanner loves microservice pattern architecture.
● Microservices talk to each other using HTTP
● Most used languages used at Skyscanner Java, JavaScript, Python have an
official and supported HTTP framework.
● The adoption of an standard HTTP framework: Share knowledge, Avoid
fragmentation, Implementation of communalities
Running Aiohttp at scale @SkyscannerEng
Why Aiohttp? Or why Asyncio?
● Looking for a framework based on IO bound scenarios
○ AWS API, Dynamodb, S3, etc.
○ Microservices architecture. I.e DDBs abstracted on top of HTTP
Rest services.
● Aiohttp meets the basic requirements:
○ Acceptable performance vs commodity.
○ There is an active community.
● Enough maturity of Asyncio
○ Asyncio API becomes stable since Python 3.5
○ Reputed Python HTTP services have plans to use Asyncio i.e.
Tornado.

Recommended

Skyscanner Engineering: Putting sync async servers to test by Jordi Soucheiron
Skyscanner Engineering: Putting sync async servers to test by Jordi SoucheironSkyscanner Engineering: Putting sync async servers to test by Jordi Soucheiron
Skyscanner Engineering: Putting sync async servers to test by Jordi SoucheironSkyscanner Engineering
 
Skyscanner presents: Getting started with tornado with Sharknado
Skyscanner presents: Getting started with tornado with SharknadoSkyscanner presents: Getting started with tornado with Sharknado
Skyscanner presents: Getting started with tornado with SharknadoSkyscanner Engineering
 
Code First: Girls - A presentation by Emily Stewart, Skyscanner
Code First: Girls - A presentation by Emily Stewart, SkyscannerCode First: Girls - A presentation by Emily Stewart, Skyscanner
Code First: Girls - A presentation by Emily Stewart, SkyscannerSkyscanner Engineering
 
AI Trends in Creative Operations 2024 by Artwork Flow.pdf
AI Trends in Creative Operations 2024 by Artwork Flow.pdfAI Trends in Creative Operations 2024 by Artwork Flow.pdf
AI Trends in Creative Operations 2024 by Artwork Flow.pdfmarketingartwork
 
PEPSICO Presentation to CAGNY Conference Feb 2024
PEPSICO Presentation to CAGNY Conference Feb 2024PEPSICO Presentation to CAGNY Conference Feb 2024
PEPSICO Presentation to CAGNY Conference Feb 2024Neil Kimberley
 
Content Methodology: A Best Practices Report (Webinar)
Content Methodology: A Best Practices Report (Webinar)Content Methodology: A Best Practices Report (Webinar)
Content Methodology: A Best Practices Report (Webinar)contently
 
How to Prepare For a Successful Job Search for 2024
How to Prepare For a Successful Job Search for 2024How to Prepare For a Successful Job Search for 2024
How to Prepare For a Successful Job Search for 2024Albert Qian
 

More Related Content

Recently uploaded

Paper Machine Troubleshooting manual for paper makers
Paper Machine Troubleshooting manual for paper makersPaper Machine Troubleshooting manual for paper makers
Paper Machine Troubleshooting manual for paper makersNoman khan
 
Series of training sessions by our experts for you to provide necessary insig...
Series of training sessions by our experts for you to provide necessary insig...Series of training sessions by our experts for you to provide necessary insig...
Series of training sessions by our experts for you to provide necessary insig...AshishChanchal1
 
CCW332-DIGITAL MARKETING QUESTION BANK WITH ANSWERS
CCW332-DIGITAL MARKETING QUESTION BANK WITH ANSWERSCCW332-DIGITAL MARKETING QUESTION BANK WITH ANSWERS
CCW332-DIGITAL MARKETING QUESTION BANK WITH ANSWERSTamil949112
 
Pointers and Array, pointer and String.pptx
Pointers and Array, pointer and String.pptxPointers and Array, pointer and String.pptx
Pointers and Array, pointer and String.pptxAnanthi Palanisamy
 
Chase Commerce Center History Nordberg manufacturing Rexnord Global power com...
Chase Commerce Center History Nordberg manufacturing Rexnord Global power com...Chase Commerce Center History Nordberg manufacturing Rexnord Global power com...
Chase Commerce Center History Nordberg manufacturing Rexnord Global power com...drezdzond
 
POST HARVEST Threshing equipment PPT 2.pptx
POST HARVEST Threshing equipment PPT 2.pptxPOST HARVEST Threshing equipment PPT 2.pptx
POST HARVEST Threshing equipment PPT 2.pptxARUL S
 
Introduction to Machine Learning Unit-1 Notes for II-II Mechanical Engineerin...
Introduction to Machine Learning Unit-1 Notes for II-II Mechanical Engineerin...Introduction to Machine Learning Unit-1 Notes for II-II Mechanical Engineerin...
Introduction to Machine Learning Unit-1 Notes for II-II Mechanical Engineerin...C Sai Kiran
 
Center Enamel is the leading fire water tanks manufacturer in China.docx
Center Enamel is the leading fire water tanks manufacturer in China.docxCenter Enamel is the leading fire water tanks manufacturer in China.docx
Center Enamel is the leading fire water tanks manufacturer in China.docxsjzzztc
 
INTERACTIVE AQUATIC MUSEUM AT BAGH IBN QASIM CLIFTON KARACHI
INTERACTIVE AQUATIC MUSEUM AT BAGH IBN QASIM CLIFTON KARACHIINTERACTIVE AQUATIC MUSEUM AT BAGH IBN QASIM CLIFTON KARACHI
INTERACTIVE AQUATIC MUSEUM AT BAGH IBN QASIM CLIFTON KARACHIKiranKandhro1
 
Basic Instrumentation Symbols | P&ID | PFD | Gaurav Singh Rajput
Basic Instrumentation Symbols | P&ID | PFD | Gaurav Singh RajputBasic Instrumentation Symbols | P&ID | PFD | Gaurav Singh Rajput
Basic Instrumentation Symbols | P&ID | PFD | Gaurav Singh RajputGaurav Singh Rajput
 
CDE_Sustainability Performance_20240214.pdf
CDE_Sustainability Performance_20240214.pdfCDE_Sustainability Performance_20240214.pdf
CDE_Sustainability Performance_20240214.pdf8-koi
 
Laser And its Application's - Engineering Physics
Laser And its Application's - Engineering PhysicsLaser And its Application's - Engineering Physics
Laser And its Application's - Engineering PhysicsPurva Nikam
 
Chapter 1 - Drilling Fluid Functions GR.ppt
Chapter 1 - Drilling Fluid Functions GR.pptChapter 1 - Drilling Fluid Functions GR.ppt
Chapter 1 - Drilling Fluid Functions GR.pptzeidali3
 
Introduction and replication to DragonflyDB
Introduction and replication to DragonflyDBIntroduction and replication to DragonflyDB
Introduction and replication to DragonflyDBMarian Marinov
 
Microservices: Benefits, drawbacks and are they for me?
Microservices: Benefits, drawbacks and are they for me?Microservices: Benefits, drawbacks and are they for me?
Microservices: Benefits, drawbacks and are they for me?Marian Marinov
 
python presentation lists,strings,operation
python presentation lists,strings,operationpython presentation lists,strings,operation
python presentation lists,strings,operationManjuRaghavan1
 
The Crystal London Arsitektur Hijau Dinda
The Crystal London Arsitektur Hijau DindaThe Crystal London Arsitektur Hijau Dinda
The Crystal London Arsitektur Hijau Dindadindapebriani27
 
Amplitude modulation and Demodulation Techniques
Amplitude modulation and Demodulation TechniquesAmplitude modulation and Demodulation Techniques
Amplitude modulation and Demodulation TechniquesRich171473
 
Integrity Constraints in Database Management System.pptx
Integrity Constraints in Database Management System.pptxIntegrity Constraints in Database Management System.pptx
Integrity Constraints in Database Management System.pptxPallaviPatil905338
 

Recently uploaded (20)

Paper Machine Troubleshooting manual for paper makers
Paper Machine Troubleshooting manual for paper makersPaper Machine Troubleshooting manual for paper makers
Paper Machine Troubleshooting manual for paper makers
 
Series of training sessions by our experts for you to provide necessary insig...
Series of training sessions by our experts for you to provide necessary insig...Series of training sessions by our experts for you to provide necessary insig...
Series of training sessions by our experts for you to provide necessary insig...
 
CCW332-DIGITAL MARKETING QUESTION BANK WITH ANSWERS
CCW332-DIGITAL MARKETING QUESTION BANK WITH ANSWERSCCW332-DIGITAL MARKETING QUESTION BANK WITH ANSWERS
CCW332-DIGITAL MARKETING QUESTION BANK WITH ANSWERS
 
Pointers and Array, pointer and String.pptx
Pointers and Array, pointer and String.pptxPointers and Array, pointer and String.pptx
Pointers and Array, pointer and String.pptx
 
Chase Commerce Center History Nordberg manufacturing Rexnord Global power com...
Chase Commerce Center History Nordberg manufacturing Rexnord Global power com...Chase Commerce Center History Nordberg manufacturing Rexnord Global power com...
Chase Commerce Center History Nordberg manufacturing Rexnord Global power com...
 
POST HARVEST Threshing equipment PPT 2.pptx
POST HARVEST Threshing equipment PPT 2.pptxPOST HARVEST Threshing equipment PPT 2.pptx
POST HARVEST Threshing equipment PPT 2.pptx
 
Introduction to Machine Learning Unit-1 Notes for II-II Mechanical Engineerin...
Introduction to Machine Learning Unit-1 Notes for II-II Mechanical Engineerin...Introduction to Machine Learning Unit-1 Notes for II-II Mechanical Engineerin...
Introduction to Machine Learning Unit-1 Notes for II-II Mechanical Engineerin...
 
Center Enamel is the leading fire water tanks manufacturer in China.docx
Center Enamel is the leading fire water tanks manufacturer in China.docxCenter Enamel is the leading fire water tanks manufacturer in China.docx
Center Enamel is the leading fire water tanks manufacturer in China.docx
 
INTERACTIVE AQUATIC MUSEUM AT BAGH IBN QASIM CLIFTON KARACHI
INTERACTIVE AQUATIC MUSEUM AT BAGH IBN QASIM CLIFTON KARACHIINTERACTIVE AQUATIC MUSEUM AT BAGH IBN QASIM CLIFTON KARACHI
INTERACTIVE AQUATIC MUSEUM AT BAGH IBN QASIM CLIFTON KARACHI
 
Basic Instrumentation Symbols | P&ID | PFD | Gaurav Singh Rajput
Basic Instrumentation Symbols | P&ID | PFD | Gaurav Singh RajputBasic Instrumentation Symbols | P&ID | PFD | Gaurav Singh Rajput
Basic Instrumentation Symbols | P&ID | PFD | Gaurav Singh Rajput
 
CDE_Sustainability Performance_20240214.pdf
CDE_Sustainability Performance_20240214.pdfCDE_Sustainability Performance_20240214.pdf
CDE_Sustainability Performance_20240214.pdf
 
Laser And its Application's - Engineering Physics
Laser And its Application's - Engineering PhysicsLaser And its Application's - Engineering Physics
Laser And its Application's - Engineering Physics
 
Chapter 1 - Drilling Fluid Functions GR.ppt
Chapter 1 - Drilling Fluid Functions GR.pptChapter 1 - Drilling Fluid Functions GR.ppt
Chapter 1 - Drilling Fluid Functions GR.ppt
 
Introduction and replication to DragonflyDB
Introduction and replication to DragonflyDBIntroduction and replication to DragonflyDB
Introduction and replication to DragonflyDB
 
Microservices: Benefits, drawbacks and are they for me?
Microservices: Benefits, drawbacks and are they for me?Microservices: Benefits, drawbacks and are they for me?
Microservices: Benefits, drawbacks and are they for me?
 
python presentation lists,strings,operation
python presentation lists,strings,operationpython presentation lists,strings,operation
python presentation lists,strings,operation
 
The Crystal London Arsitektur Hijau Dinda
The Crystal London Arsitektur Hijau DindaThe Crystal London Arsitektur Hijau Dinda
The Crystal London Arsitektur Hijau Dinda
 
Amplitude modulation and Demodulation Techniques
Amplitude modulation and Demodulation TechniquesAmplitude modulation and Demodulation Techniques
Amplitude modulation and Demodulation Techniques
 
Integrity Constraints in Database Management System.pptx
Integrity Constraints in Database Management System.pptxIntegrity Constraints in Database Management System.pptx
Integrity Constraints in Database Management System.pptx
 
Présentation IIRB 2024 M.Campoverde R.Duval
Présentation IIRB 2024 M.Campoverde R.DuvalPrésentation IIRB 2024 M.Campoverde R.Duval
Présentation IIRB 2024 M.Campoverde R.Duval
 

Featured

Social Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie InsightsSocial Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie InsightsKurio // The Social Media Age(ncy)
 
Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024Search Engine Journal
 
5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summary5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summarySpeakerHub
 
ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd Clark Boyd
 
Getting into the tech field. what next
Getting into the tech field. what next Getting into the tech field. what next
Getting into the tech field. what next Tessa Mero
 
Google's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search IntentGoogle's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search IntentLily Ray
 
Time Management & Productivity - Best Practices
Time Management & Productivity -  Best PracticesTime Management & Productivity -  Best Practices
Time Management & Productivity - Best PracticesVit Horky
 
The six step guide to practical project management
The six step guide to practical project managementThe six step guide to practical project management
The six step guide to practical project managementMindGenius
 
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...RachelPearson36
 
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...Applitools
 
12 Ways to Increase Your Influence at Work
12 Ways to Increase Your Influence at Work12 Ways to Increase Your Influence at Work
12 Ways to Increase Your Influence at WorkGetSmarter
 
Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko (Belka G...
Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko (Belka G...Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko (Belka G...
Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko (Belka G...DevGAMM Conference
 
Barbie - Brand Strategy Presentation
Barbie - Brand Strategy PresentationBarbie - Brand Strategy Presentation
Barbie - Brand Strategy PresentationErica Santiago
 
Good Stuff Happens in 1:1 Meetings: Why you need them and how to do them well
Good Stuff Happens in 1:1 Meetings: Why you need them and how to do them wellGood Stuff Happens in 1:1 Meetings: Why you need them and how to do them well
Good Stuff Happens in 1:1 Meetings: Why you need them and how to do them wellSaba Software
 
Introduction to C Programming Language
Introduction to C Programming LanguageIntroduction to C Programming Language
Introduction to C Programming LanguageSimplilearn
 
The Pixar Way: 37 Quotes on Developing and Maintaining a Creative Company (fr...
The Pixar Way: 37 Quotes on Developing and Maintaining a Creative Company (fr...The Pixar Way: 37 Quotes on Developing and Maintaining a Creative Company (fr...
The Pixar Way: 37 Quotes on Developing and Maintaining a Creative Company (fr...Palo Alto Software
 

Featured (20)

Social Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie InsightsSocial Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie Insights
 
Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024
 
5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summary5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summary
 
ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd
 
Getting into the tech field. what next
Getting into the tech field. what next Getting into the tech field. what next
Getting into the tech field. what next
 
Google's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search IntentGoogle's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search Intent
 
How to have difficult conversations
How to have difficult conversations How to have difficult conversations
How to have difficult conversations
 
Introduction to Data Science
Introduction to Data ScienceIntroduction to Data Science
Introduction to Data Science
 
Time Management & Productivity - Best Practices
Time Management & Productivity -  Best PracticesTime Management & Productivity -  Best Practices
Time Management & Productivity - Best Practices
 
The six step guide to practical project management
The six step guide to practical project managementThe six step guide to practical project management
The six step guide to practical project management
 
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
 
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...
 
12 Ways to Increase Your Influence at Work
12 Ways to Increase Your Influence at Work12 Ways to Increase Your Influence at Work
12 Ways to Increase Your Influence at Work
 
ChatGPT webinar slides
ChatGPT webinar slidesChatGPT webinar slides
ChatGPT webinar slides
 
More than Just Lines on a Map: Best Practices for U.S Bike Routes
More than Just Lines on a Map: Best Practices for U.S Bike RoutesMore than Just Lines on a Map: Best Practices for U.S Bike Routes
More than Just Lines on a Map: Best Practices for U.S Bike Routes
 
Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko (Belka G...
Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko (Belka G...Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko (Belka G...
Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko (Belka G...
 
Barbie - Brand Strategy Presentation
Barbie - Brand Strategy PresentationBarbie - Brand Strategy Presentation
Barbie - Brand Strategy Presentation
 
Good Stuff Happens in 1:1 Meetings: Why you need them and how to do them well
Good Stuff Happens in 1:1 Meetings: Why you need them and how to do them wellGood Stuff Happens in 1:1 Meetings: Why you need them and how to do them well
Good Stuff Happens in 1:1 Meetings: Why you need them and how to do them well
 
Introduction to C Programming Language
Introduction to C Programming LanguageIntroduction to C Programming Language
Introduction to C Programming Language
 
The Pixar Way: 37 Quotes on Developing and Maintaining a Creative Company (fr...
The Pixar Way: 37 Quotes on Developing and Maintaining a Creative Company (fr...The Pixar Way: 37 Quotes on Developing and Maintaining a Creative Company (fr...
The Pixar Way: 37 Quotes on Developing and Maintaining a Creative Company (fr...
 

Skyscanner Engineering: Running aiohttp at scale by Pau Freixes

  • 2. ● About me ● Skyscanner and aiohttp, why? ● Tracing incoming requests ● Calling other microservices ● DNS in AWS with Aiohttp ● Misleading timeouts, the reactor saturation side effect. ● Desired plans ● Questions Running Aiohttp at scale @SkyscannerEng
  • 3. Pau Freixes @pfreixes ● Senior Software Engineer working at Skyscanner for almost 2 years. ● Member of the Hotels Attachment Squad. ● Also collaborator of Mshell Squad. Helping with the Python stuff. ● Open source committer aioredis, aiohttp, etc. Running Aiohttp at scale @SkyscannerEng
  • 4. Skyscanner and aiohttp, why? Running Aiohttp at scale @SkyscannerEng
  • 5. Running Aiohttp at scale @SkyscannerEng ● Skyscanner loves microservice pattern architecture. ● Microservices talk to each other using HTTP ● Most used languages used at Skyscanner Java, JavaScript, Python have an official and supported HTTP framework. ● The adoption of an standard HTTP framework: Share knowledge, Avoid fragmentation, Implementation of communalities
  • 6. Running Aiohttp at scale @SkyscannerEng Why Aiohttp? Or why Asyncio? ● Looking for a framework based on IO bound scenarios ○ AWS API, Dynamodb, S3, etc. ○ Microservices architecture. I.e DDBs abstracted on top of HTTP Rest services. ● Aiohttp meets the basic requirements: ○ Acceptable performance vs commodity. ○ There is an active community. ● Enough maturity of Asyncio ○ Asyncio API becomes stable since Python 3.5 ○ Reputed Python HTTP services have plans to use Asyncio i.e. Tornado.
  • 7. Running Aiohttp at scale @SkyscannerEng But choosing Aiohttp/Asyncio also means face some uncertainties: ● Asynchronous code != synchronous code ○ Different problems ○ Different patterns ○ Less experience ● Some libraries might be not mature at all ○ Not enough time for growing ○ Small community ○ Less feedback ○ etc
  • 8. Tracing incoming requests Running Aiohttp at scale @SkyscannerEng
  • 9. Running Aiohttp at scale @SkyscannerEng We need to know what is happening and what happened in our microservice. ● HTTP endpoints statistics: ○ Time per request ○ Statistics such as avg, p90, p99 ○ Number of requests ○ Status code per request. Errors ● HTTP Access log ○ Historically access. ○ Indexed by fields such as status code, endpoint, ○ Identify each line of log to a specific request.
  • 10. Docker foundations of our microservice that enables the request tracing Running Aiohttp at scale @SkyscannerEng ● Requests goes into through the HAProxy ● AioHttp microservice handles the incoming request ● AioHttp sends metrics to the StatsD container ○ Per each metric (real time) ○ Low latency network ● AioHttp sends logs to the Heka container ○ Per each log ( real time) ○ Low latency network ● StatsD sends aggregations of metrics (OpenTSDB) ○ Avg, p90, p99 ○ Almost not real time, per minute ● Heka parsers and sends the data (ElasticSearch) ○ Almost not real time
  • 11. Running Aiohttp at scale @SkyscannerEng Aiohttp middlewares are the perfect place to instrumentalize incoming requests: async def middleware_timing(self, app, handler): async def timing(request): start = app.loop.time() response = await handler(request) print(app.loop.time() - start) return response return timing app = web.Application(middlewares=[timing])
  • 12. Running Aiohttp at scale @SkyscannerEng Aiohttp microservices at Skyscanner come with the following middlewares for free: ● Metric requests: Produce statistics by request. ● Access Log: Upload access log. ● Correlation id: Identify univocally an incoming request.
  • 13. Running Aiohttp at scale @SkyscannerEng Demo about metrics and access log
  • 14. Running Aiohttp at scale How could we follow the code path executed by an specific request ? async def foo(request): logging.info("Doing some complicate stuff") await asyncio.sleep(1) async def bar(request): start = loop.time() await foo(request) logging.info("Time for foo {}".format(loop.time() - start) async def view(request): logging.info("New request") await bar(request)
  • 15. Running Aiohttp at scale @SkyscannerEng Any request at Skyscanner is identified by an unique ID, this identifier is saved at some place that will be used automatically by any logging call.
  • 16. Running Aiohttp at scale @SkyscannerEng aiotask-context stores information within the current asyncio.task instance. async foo(): aiotask_context.set("key", True) await asyncio.sleep(1) aiotask_context.get("key")
  • 17. Running Aiohttp at scale @SkyscannerEng The request id is stored as a task attribute by a middleware to make it available at any code place. async def correlation_id(self, app, handler): async def save_correlation_id(request): correlation_id = request.headers.get( "Skyscanner-Correlation-Id", request.headers.get( "X-Correlation-Id", str(uuid.uuid4() ) ) context.set("Skyscanner-Correlation-Id", correlation_id) return await handler(request)
  • 18. Running Aiohttp at scale @SkyscannerEng When the aiohttp microservice is started a new logging filter is installed to populate automatically the request id at each logging call. class RequestId(logging.Filter): def filter(self, record): correlation_id = context.get("Skyscanner-Correlation-Id") record.correlationid = correlation_id return True
  • 19. Running Aiohttp at scale @SkyscannerEng Installation of the RequestId filter LOG_SETTINGS = { 'handlers': { 'console': { 'class': 'logging.StreamHandler', 'level': 'INFO', 'formatter': 'default', 'filters': ['correlationid'], } }, 'formatters': { 'default': { 'format': '%(asctime)s %(levelname)s %(correlationid)s | %(message)s', }, }, }
  • 20. Running Aiohttp at scale @SkyscannerEng All logging calls will log the request id as a literal and also as a new field to be indexed by ElasticSearch try: price_per_night = request.price / request.nights except ZeroDivisionError: logging.warning("Invalid `night` value param") raise
  • 21. Running Aiohttp at scale @SkyscannerEng Demo about request id
  • 22. Calling other microservices Running Aiohttp at scale @SkyscannerEng
  • 23. Running Aiohttp at scale @SkyscannerEng async validate_currency(request): session = ClientSession() resp = await session.post("http://currency.eu-west-1.skyscnr.local", data={'currency': request.currency} ) if resp.status_code != 200: raise ValidationError( "Currency {} invalid".format(request.currency)) I.e validate if the currency sent by a request is valid or not. We will use an external microservice service for that:
  • 24. Running Aiohttp at scale @SkyscannerEng We need to know what is happening and what happened with the calls to external microservices. ● Time per request ○ Statistics such as avg, p90, p99 ● Number of requests ● Status code per request. Errors
  • 25. Running Aiohttp at scale @SkyscannerEng Aiohttp does not provide an official a way to trace request events, yet. An ad hoc class called MetricsClientSession is implemented to replace the official ClientSession. class MetricsClient(ClientSession): def _request(self, *args, **kwargs): start = self._loop.time() response = super(MetricsClient, self)._request(*args, **kwargs) elapsed = self._loop.time() - start logging.info("Time spent {}".format(elapsed)) return response
  • 26. Running Aiohttp at scale @SkyscannerEng Demo about calling other services
  • 27. DNS in AWS with Aiohttp Running Aiohttp at scale @SkyscannerEng
  • 28. Running Aiohttp at scale @SkyscannerEng $ dig currency.eu-west-1.skyscnr.local currency.eu-west-1.skyscnr.local. 59 IN A 10.51.106.106 currency.eu-west-1.skyscnr.local. 59 IN A 10.51.165.90 currency.eu-west-1.skyscnr.local. 59 IN A 10.51.35.2 AWS: ● DNS TTL 60 seconds ● IP addresses can change ● Number of IP addresses can grow
  • 29. Running Aiohttp at scale @SkyscannerEng Aiohttp versions < 2 does not support it A DNS cache was implemented based on the AWS requirements. This implementation would become the official one for Aiohttp 2 versions.
  • 30. Running Aiohttp at scale @SkyscannerEng >>> from aiohttp.connector import TCPConnector >>> connector = TCPConnector(dns_ttl=60) >>> >>> async def ip(hostname, port): >>> hosts = await connector._resolve_host(hostname, port) >>> print(next(hosts)['host']) >>> >>> asyncio.get_event_loop().run_until_complete(ip("currency.eu-west-1.skyscnr.local", 8080)) 10.51.165.90 >>> asyncio.get_event_loop().run_until_complete(ip("currency.eu-west-1.skyscnr.local", 8080)) 10.51.35.2 >>> asyncio.get_event_loop().run_until_complete(ip("currency.eu-west-1.skyscnr.local", 8080)) 10.51.106.106 >>> asyncio.get_event_loop().run_until_complete(ip("currency.eu-west-1.skyscnr.local", 8080)) 10.51.165.90
  • 31. Running Aiohttp at scale @SkyscannerEng DNS cache and the dog pile effect. The following code will make 100 DNS queries without using the cache. import asyncio tasks = [validate_currency('EUR') for i in range(100)] loop = asyncio.get_event_loop() loop.run_until_complete(asyncio.gather(*tasks))
  • 32. Running Aiohttp at scale @SkyscannerEng The dog pile effect happens when there is a miss in the cache, all ongoing requests will end up performing a DNS query. To get rid of this side effect, a throttling mechanism was implemented. Available in Aiohttp 2.3.
  • 33. Running Aiohttp at scale @SkyscannerEng # there was a miss in the cache if host in self._throttle_dns_events: yield from self._throttle_dns_events[host].wait() else: self._throttle_dns_events[host] = EventResultOrError(self._loop) addrs = yield from self._resolver.resolve( host, port, family=self._family) self._cached_hosts.add(host, addrs) self._throttle_dns_events[host].set() return self._cached_hosts.next_addrs(host)
  • 34. Misleading timeouts, the reactor saturation side effect. Running Aiohttp at scale @SkyscannerEng
  • 35. Running Aiohttp at scale @SkyscannerEng Calls to third services are protected by timeouts, having the chance to do the proper countermeasures. async validate_currency(request): session = ClientSession() try: resp = await session.post("http://currency.eu-west-1.skyscnr.local", data={'currency': request.currency} timeout=1, ) except asyncio.TimeoutError: raise HttpError(504)
  • 36. Running Aiohttp at scale @SkyscannerEng When the reactor is saturated the timeouts might be triggered. Timeouts are handled internally by asyncio as future callbacks that will cancel a specific Future. def cancel_future(future): future.cancel() async def request(*args, timeout=2): f = asyncio.Future() asyncio.call_later(timeout, cancel_future, f) # some internal stuff that triggers the network # operations return f
  • 37. Running Aiohttp at scale @SkyscannerEng Lets try to monitor the reactor saturation, how ?
  • 38. Running Aiohttp at scale @SkyscannerEng …. with the LAG of a scheduled function. The time lapsed between executions can be used to measure how busy is a reactor.. def lag(): elapsed = before - loop.time() if elapsed > 1: print("Reactor had a delay") loop.call_later(lag, 1)
  • 39. Running Aiohttp at scale @SkyscannerEng Example of the lag metric. You can identify the reactor saturation that happened at some point.
  • 40. Desired Plans Running Aiohttp at scale @SkyscannerEng
  • 41. Running Aiohttp at scale @SkyscannerEng ● Trace queued operations ○ HTTP pool has a connection limit ○ Once the limit is reached the operation is queued ○ When there is a free connection, operation is unqueued ● AWS Xray support ○ Another Middleware ○ Trace calls to third services ● Back pressure at HTTP layer ○ When the reactor is too busy return 504 ○ Scale horizontally when there are a flood of 504
  • 42. Edinburgh • Glasgow • Singapore • Beijing • Miami • Barcelona • Shenzhen • Sofia • Budapest • London • Tokyo Questions? Slides http://bit.ly/runningatscale