Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Pycon Korea 2018-Sanic을 활용하여 Microservice 구축하기-이재면

1,042 views

Published on

These presentation slides are what will be presented at Pycon Korea 2018. They explain what microservices architecture is and Why company 'mymusictaste' moved from monolithic application to microservices architecture. The slides also explains how we built microservices with Python and utilized AsyncIo and Sanic. We built our own framework to add features on Sanic to meet our needs. This is to share our experiences transforming to micro services architecture as a startup company with everyone at Pycon Korea 2018.

Pycon Korea 2018-Sanic을 활용하여 Microservice 구축하기-이재면

  1. 1. Sanic Microservice Microservices architecture
  2. 2. Who am I? mymusictaste.com Person who wants to be crazy about something, like a geek DevOps Catlover Instagram @our_durian
  3. 3. The Takeaway AsyncIO Sanic
  4. 4. 01 Monolithic vs Microservices 02 AsyncIO & Sanic 03 Transformation to Insanic 04 Challenges & Plan Contents
  5. 5. 01 Monolithic vs Microservices . MMT .
  6. 6. Microservices - - - - A collection of loosely coupled services Parallelizes development by small autonomous teams to development scale their respective services independently polyglot developed by Python developed by Go more resilient to architecture erosion easier to understand, develop, test
  7. 7. Microservices Container & orchestration Continous deployment => => &
  8. 8. Microservices Don’t even consider microservices unless you have a system that’s too complex to manage as monolithic : https://www.martinfowler.com/bliki/MicroservicePremium.html
  9. 9. Why did we move to microservices architecture? Django Microservices Acrchitecture
  10. 10. 02 AsyncIO vs Sanic python 3.4 standard library AsyncIO . AsyncIO Web framework Sanic .
  11. 11. This module provides infrastructure for writing single-threaded concurrent code using coroutines, multiplexing I/O access over socket and other resources, running network clients and servers, and other related primitives. import asyncio async def compute(x, y): print("Compute %s + %s ..." % (x, y)) await asyncio.sleep(1.0) return x + y async def print_sum(x, y): result = await compute(x, y) print("%s + %s = %s" % (x, y, result)) loop = asyncio.get_event_loop() loop.run_until_complete(print_sum(1, 2)) loop.close() AsyncIO
  12. 12. import asyncio async def compute(x, y): print("Compute %s + %s ..." % (x, y)) await asyncio.sleep(1.0) return x + y async def print_sum(x, y): result = await compute(x, y) print("%s + %s = %s" % (x, y, result)) loop = asyncio.get_event_loop() loop.run_until_complete(print_sum(1, 2)) loop.close() AsyncIO
  13. 13. - results = await asyncio.gather(*tasks) - await asyncio.wait(futures, return_when=FIRST_COMPLETED) - asyncio.ensure_future(task) - future.add_done_callback(callback) AsyncIO
  14. 14. AsyncIO I/O bound CPU bound A B A single-threaded concurrent code B cooperative multitasking A B A what if B is not cooperative?
  15. 15. “So microservices use a distributed system to improve modularity. But distributed software has a major disadvantage, the fact that it's distributed. The second mitigation is to use asynchrony. If make six asynchronous calls in parallel you're now only as slow as the slowest call instead of the sum of their latencies. This can be a big performance gain, but comes at another cognitive cost. Asynchronous programming is hard: hard to get right, and much harder to debug. But most microservice stories I've heard need asynchrony in order to get acceptable performance.” Martin Fowler
  16. 16. SANIC - Sanic is a Flask-like Python 3.5+ web server https://github.com/channelcat/sanic from sanic import Sanic from sanic.response import json app = Sanic() @app.route('/') async def test(request): return json({'hello': 'world'}) if __name__ == '__main__': app.run(host='0.0.0.0', port=8000)
  17. 17. SANIC sanic/server.py class HttpProtocol(asyncio.Protocol) def connection_made(self, transport): def connection_lost(self, exc): def data_received(self, data): server_coroutine = loop.create_server( server, host, port, ssl=ssl, reuse_port=reuse_port, sock=sock, backlog=backlog ) http_server = loop.run_until_complete(server_coroutine) loop.run_forever() Create TCP server self.transport.write( response.output( self.request.version, keep_alive, self.keep_alive_timeout))
  18. 18. 03 Transformation to Insanic MMT Sanic web framework . MMT web framework , .
  19. 19. Sanic Insanic INSANIC
  20. 20. Settings Settings for Insanic differ a lot from sanic's config pattern. While implementing, I found the need to access the settings from modules where the sanic application was not accessable. Thus the settings object in insanic takes a lot from django's settings configuration where the settings object is a single instantiated config object that can be imported anywhere. In addition to sanic's config object, where it is usually instantiated and attached to sanic's application instance, the settings object in insanic is a lazy loaded with dependencies on the service name. Settings Loading There are several steps/places the settings object loads settings from. 1. Loads the global_settings.py from within insanic. These are the default settings that insanic needs to run. 2. Then with the SERVICE_VARIABLE, tries to load config.py from within the project. 3. common settings are loaded from VAULT 4. service settings are loaded from VAULT 5. Any environment variables are loaded. Must be prefixed with set prefix. (default its INSANIC) INSANICINSANIC
  21. 21. View Authentication and Permission Handling Insanic takes the original views from sanic and modifies them to handle authentication and permission handling. A lot of patterns were taken from Django Rest Framework. The bulk of the updates is through the dispatch_request method. To register authentication and permissions, we must first create or use the general authentication and permission provided by insanic Views # views.py from sanic.response import json from insanic import permissions, authentication from insanic.views import InsanicView class GottaGoFastView(InsanicView): permission_classes = (permissions.AllowAny,) authentication_classes = (authentication.JSONWebTokenAuthentication, ) async def get(self, request, *args, **kwargs): return json({"how fast?": "insanely fast"}) Permissions - AllowAny - IsAuthenticated - IsAdminUser - IsAuthenticatedOrReadOnly - IsOwnerOrAdmin - IsServiceOnly INSANIC
  22. 22. Request Object Apart from the request attributes provided by the sanic request object, insanic creates additional attributes with a lot of inspiration from Django-REST-Framework but with async compatibilities. Request Parsing Authentication .data .query_params .user user = await request.user .service service = await request.service INSANIC
  23. 23. Inter Service Communications One of the core additional features of insanic, that differentiates it from other frameworks, is the Service object. Service Features • Resolve address and endpoint construction • Header preparation and injections for https requests • Error handling • Circuit Breaker patching • Response handling from insanic.loading import get_service ArtistService = get_service('artist') response, status_code = await ArtistService.http_dispatch('GET', '/api/v1/artists/', query_params={"query": "insanic"}, include_status_code=True) INSANIC
  24. 24. Pytest helpers Public Facing API Gateway Registration Tracing with AWS X-Ray service Addition of these features are ongoing - Contract test - OpenAPI 3.0 specification document generation - inter service communication with RPC option INSANIC
  25. 25. 04 Challenges & Plan AsyncIO Sanic Microservices architecture . , .
  26. 26. In… series / wrapper for asyncio aiohttp Asynchronous HTTP client/server for asyncio aioelasticsearch elasticsearch-py wrapper for asyncio asyncpg A fast PostgreSQL database client library aioredis Asyncio Redis support InPynamoDB This transforms PynamoDB’s basic methods working asynchronously used  aiobotocore. Infuse This is heavily based on pybreaker. For full documentation refer to pybreaker. What is different from pybreaker is that it includes asynchronous storage options. async-python-mailchimp-api A straighforward python asynchronous client for v3 of MailChimp API using aiohttp >= 3.0.0. This Project forked from python-mailchimp CHALLENGES
  27. 27. CHALLENGES Where you are
  28. 28. PLAN 1. Insanic (2018. 8. 6. Release) 2. Insanic Open source Main developer for Insanic Kwang Jin Kim https://github.com/MyMusicTaste/recruit We are hiring!

×