Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

How we realized SOA by Python at PyCon JP 2015

7,745 views

Published on

How we realized SOA by Python at PyCon JP 2015

Published in: Technology
  • Be the first to comment

How we realized SOA by Python at PyCon JP 2015

  1. 1. How we realize SOA by Python In PyCon JP 2015
  2. 2. #PyConJP_C
  3. 3. SOA meaning Service Oriented Architecture
  4. 4. What's SOA SOA is an approach to create a system based on small servers separated for small functions.
  5. 5. SOA WebApp Search Auth Other Images
  6. 6. https://www.nginx.com/blog/building-microservices-using-an-api-gateway/
  7. 7. So... is the SOA comfy? • Yes. • But... no silver bullet.
  8. 8. Pros • Separated responsibilities • Rapid integration • Flexible scaling
  9. 9. Cons • Lots of servers ● To create ● To deploy ● To monitor
  10. 10. Can't realize SOA with legacy style
  11. 11. • Direct transmitting • Manual deploying • Manual testing
  12. 12. So how we realize SOA
  13. 13. Today you can learn • Case study of SOA by Python • Practices to manage lots of services
  14. 14. Today you can't lean • Step by step guide • So much about Python's core
  15. 15. Agenda About Our case Summary Architecture Creating MonitoringTesting Deploying
  16. 16. Who I am • Hiroki Kiyohara (hirokiky) • BePROUD Inc • Developer, Consultant, Trainer • Admin of djangoproject.jp
  17. 17. Who we are • News paper company ● Over 400,000 paid user (Web) • Migrating to in-house production
  18. 18. Our case
  19. 19. Architecture
  20. 20. Architecture
  21. 21. Architecture REST API
  22. 22. Architecture Separated from Web
  23. 23. API Gateway • Proxy for back-end servers • Handling permissions
  24. 24. Creating
  25. 25. Creating • Need to create many services • Share docs
  26. 26. Django • Easy to create • Many libraries • Community and knowledge
  27. 27. DjangoRestFramework • Framework on Django to create REST API • Useful modules • Smart Auth/Authz framework
  28. 28. DRF: Serializer • Django's Form for REST API ● Nested ● Array of objects
  29. 29. Serializer is really simple
  30. 30. Validating by serializer >>> from search.serializers import BulkPostSerializer >>> BulkPostSerializer(data={ ... "name": "hiroki", ... "posts": [ ... {"title": "PyCon JP 2015", ... "body": "This is awesome event"}, ... {"title": "How we realize SOA", ... "body": "Python is awesome"}, ... ] ... }) >>> serializer.is_valid() True >>> serializer.validated_data OrderedDict([('name', 'hiroki'), ('pos...
  31. 31. cookiecutter • Template of repository • Share best practices
  32. 32. . ├── README.md ├── project_name │ ├── manage.py │ └── project_name │ ├── settings │ │ ├── __init__.py │ │ └── test.py │ ├── urls.py │ └── wsgi.py ├── .elasticbeanstalk/ ├── .coveragerc ├── .gitignore ├── circle.yml ├── Dockerfile ├── Dockerrun.aws.json ├── requirements.txt ├── setup.cfg └── tox.ini
  33. 33. OK so... created • Many applications • Share docs
  34. 34. django-rest-swagger • API docs from docstring and code • Demo-able API docs
  35. 35. django-rest-swagger
  36. 36. django-rest-swagger
  37. 37. Creating • Django • DjangoRestFramework • cookiecutter • tox
  38. 38. Testing
  39. 39. Testing • Handy unit tests • E2E tests
  40. 40. I know you won't run. MEE TOO • Complex • Slow • Useless
  41. 41. Speed up by django's setting • On memory sqlite • Dummy or local cache • Light hasher Refer Two scoops of Django
  42. 42. Use nice tools • tox • flake8 • ...
  43. 43. tox • Just type `tox` • Run several tests in separated virtualenv
  44. 44. [tox] envlist = py34, flake8 skipsdist = True setupdir = ./myprj/ [testenv:py34] deps = coverage -rrequirements.txt setenv = DJANGO_SETTINGS_MODULE = myprj.settings.test commands = coverage erase coverage run myprj/manage.py test myprj coverage report [testenv:flake8] basepython = python3.4 deps = flake8 commands = flake8 myprj
  45. 45. [tox] envlist = py34, flake8 skipsdist = True setupdir = ./myprj/ [testenv:py34] deps = coverage -rrequirements.txt setenv = DJANGO_SETTINGS_MODULE = myprj.settings.test commands = coverage erase coverage run myprj/manage.py test myprj coverage report [testenv:flake8] basepython = python3.4 deps = flake8 commands = flake8 myprj Installing dependencies
  46. 46. [tox] envlist = py34, flake8 skipsdist = True setupdir = ./myprj/ [testenv:py34] deps = coverage -rrequirements.txt setenv = DJANGO_SETTINGS_MODULE = myprj.settings.test commands = coverage erase coverage run myprj/manage.py test myprj coverage report [testenv:flake8] basepython = python3.4 deps = flake8 commands = flake8 myprj Using settings for test
  47. 47. [tox] envlist = py34, flake8 skipsdist = True setupdir = ./myprj/ [testenv:py34] deps = coverage -rrequirements.txt setenv = DJANGO_SETTINGS_MODULE = myprj.settings.test commands = coverage erase coverage run myprj/manage.py test myprj coverage report [testenv:flake8] basepython = python3.4 deps = flake8 commands = flake8 myprj Running tests with coverage
  48. 48. [tox] envlist = py34, flake8 skipsdist = True setupdir = ./myprj/ [testenv:py34] deps = coverage -rrequirements.txt setenv = DJANGO_SETTINGS_MODULE = myprj.settings.test commands = coverage erase coverage run myprj/manage.py test myprj coverage report [testenv:flake8] basepython = python3.4 deps = flake8 commands = flake8 myprj Static analysis
  49. 49. Just... $ tox
  50. 50. flake8 • Code style check • Static analysis
  51. 51. flake8 • Don't review about syntax • Remove tiny and messy code bugs
  52. 52. And more • responses • testfixtures • factory-boy
  53. 53. Of cause, DO NOT • Accessing outer services • Setting middle wares locally
  54. 54. E2E testing • Using requests • Checking connection of each services
  55. 55. Locust.io • Load tests by Python • Distributed clients but aggregated report
  56. 56. Test • Faster Django setting • tox • flake8 • locust.io
  57. 57. Deploying
  58. 58. Reduce costs to deploy • Automated deploy by ElasticBeanstalk • Master deploying from CI
  59. 59. Auto deploy • Master branch is on development env • Deploying from CI tool
  60. 60. Deploying
  61. 61. Deploying Deploy It
  62. 62. Deploying GREEN
  63. 63. Deploying Push
  64. 64. Deploying Deploy
  65. 65. Deploying OK Pulling
  66. 66. Deploying Deployed
  67. 67. Also from CLI or Web console • Just type `$ eb deploy` • Or through the Web
  68. 68. Deploy • GitHub • CircleCI • Docker • ElasticBeanstalk
  69. 69. Monitoring
  70. 70. A lots of servers to monitor • Many services, lots servers • Immutable infrastructure
  71. 71. Monitoring tools
  72. 72. Sentry • Event (log) aggregation platform • Manage console of each events
  73. 73. Sentry • Many events but one notification • Marked as `Resolved`, it'll come again
  74. 74. Fluentd • Aggregating access log • S3 and BigQuery
  75. 75. NewRelic • Resource monitoring for containers • And APM is awesome
  76. 76. NewRelic APM • Analyzing Python application • Which progress is slow?
  77. 77. Rundeck • Job scheduler • Better crond
  78. 78. Rundeck
  79. 79. Monitor • Sentry • Fluentd • NewRelic • Rundeck
  80. 80. So...
  81. 81. Will SOA be a silver bullet?
  82. 82. No
  83. 83. Pros • Separated responsibilities • Rapid integration • Flexible scaling
  84. 84. But you need to do • Easy creating • Handy testing • Auto deploying • Relief monitoring
  85. 85. What should we do next? • Non-blocking • Nicer container platform
  86. 86. Thank you • For listening • For this nice event

×