Download as PDF, PPTX
Uploaded byOsvaldo Santana Neto
Olist Architecture v2.0
The document discusses the transition from a monolithic architecture to microservices for improved scalability, reliability, and safety in web applications. It covers various aspects of microservices including communication methods, deployment strategies, and the tools used for development, as well as challenges such as data consistency and contract management between services. The author, Osvaldo Santana, emphasizes the importance of modularity and asynchronous operations in building robust microservices systems.
More Related Content
Similar to Olist Architecture v2.0
Olist Architecture v2.0
- 1. Olist Architecture From monotithto microservices
- 2. • Programador amadordesde 1986, profissional desde 1989 e pythonista desde 2000 • Criador e ex-moderador da da python- brasil@yahoo (atual python-brasil@googlegroups) • Criador da primeira versão do www.python.org.br • Sócio fundador e ex-presidente da Associação Python Brasil • Autor do (antigo) livro Python e Django • Atualmente na Olist • osvaldo@olist.com e @osantana (ou @osantanabr) Osvaldo
- 3. O que nósfazemos? Merchants Online | Offline
- 4.
- 5.
- 6.
- 7.
- 8.
- 9. New Platform Let's writea new version...
- 10.
- 11.
- 12.
- 13.
- 14.
- 15.
- 16. Premisse • No matterif a system is internal or external, it eventually… • … goes offline… • … crashes… • … or change their behaviour without notice.
- 17.
- 18.
- 19.
- 20.
- 21. Communication via API •RESTful communication between services • Synchronous Service Calls • Strict dependency between services A P I A P I G a t e w a y A P I A P I HTTP Requests
- 22. Communication via API •RESTful communication between services • Synchronous Service Calls • Strict dependency between services A P I A P I G a t e w a y A P I A P I HTTP Requests
- 23. Communication via API •RESTful communication between services • Synchronous Service Calls • Strict dependency between services A P I A P I G a t e w a y A P I A P I HTTP Requests
- 24. Communication via API •RESTful communication between services • Synchronous Service Calls • Strict dependency between services A P I A P I G a t e w a y A P I A P I HTTP Requests
- 25. Communication via API •RESTful communication between services • Synchronous Service Calls • Strict dependency between services • No resilience • No safety* (eg. data loss on request failures) A P I A P I G a t e w a y A P I A P I HTTP Requests
- 26. Async Task Execution Workers A P I Task#1 Task #2 Task #3 call task #1 call task #2 call task #3
- 27. Async Task Execution •Asynchronous Procedure Calls • Queue based task execution • Client must knows about their dependents (I've to change API every time I need to add a new task) Workers A P I Task #1 Task #2 Task #3 call task #1 call task #2 call task #3
- 28. Async Task Execution •Asynchronous Procedure Calls • Queue based task execution • Client must knows about their dependents (I've to change API every time I need to add a new task) Workers A P I Task #1 Task #2 Task #3 call task #1 call task #2 call task #3
- 29. Async Task Execution •Asynchronous Procedure Calls • Queue based task execution • Client must knows about their dependents (I've to change API every time I need to add a new task) Workers A P I Task #1 Task #2 Task #3 call task #1 call task #2 call task #3
- 30. Async Task Execution •Asynchronous Procedure Calls • Queue based task execution • Client must knows about their dependents (I've to change API every time I need to add a new task) Workers A P I Task #1 Task #2 Task #3 call task #1 call task #2 call task #3
- 31. Async Task Execution •Asynchronous Procedure Calls • Queue based task execution • Client must knows about their dependents (I've to change API every time I need to add a new task) Workers A P I Task #1 Task #2 Task #3 call task #1 call task #2 call task #3 Task #4 call task #4
- 32. Async Task Execution •Asynchronous Procedure Calls • Queue based task execution • Client must knows about their dependents (I've to change API every time I need to add a new task) • No decoupling ➡ No modularity • No safety* (eg. data loss on deployment failures) Workers A P I Task #1 Task #2 Task #3 call task #1 call task #2 call task #3 Task #4 call task #4
- 33. Message Event Triggering Event A P I ServiceService Service
- 34. Message Event Triggering •Action Event triggering • Queue based message event handling • Event as messages Event A P I Service Service Service
- 35. Message Event Triggering •Action Event triggering • Queue based message event handling • Event as messages Event A P I Service Service Service
- 36. Message Event Triggering •Action Event triggering • Queue based message event handling • Event as messages Event A P I Service Service Service queues producer consumers
- 37. Message Event Triggering •Action Event triggering • Queue based message event handling • Event as messages • That's it! Event A P I Service Service Service queues producer consumers
- 38.
- 39. Messages • Also knownas Resource in REST context • Follow a contract (schema) • Can be wrapped with metadata (eg. SNS/SQS metadata)
- 40. Topics (global) • Publisherin PubSub Pattern • Global topics for message publication • Topics belong to the system (or architecture) and not to a (micro)service • AWS SNS
- 41. Service Queues • Subscribersin PubSub Pattern • Queues subscribe topics • One queue belong exclusively to one (micro)service • AWS SQS • SQS can be used as a SNS subscriber
- 42.
- 43. API • Data EntryPoint • Data Validation • Workflow Management (eg. status/state machine) • Data Persistence • Event Triggering • Idempotency handling (eg. discard duplicated requests returning a HTTP 304 Not Modified) • Python, Django, DRF A P I
- 44. API (webhook) • DataEntry Point • No Data Persistence • Proxy HTTP ➡ SNS • Event Triggering • Python, Django, DRF W e b h o o k
- 45. Service (consumer) • EventHandling / Message Processing • Business Logic • Python, Loafer API Get Data...Service
- 46. Service (broker) • EventHandling / Message Processing • Business Logic • Python, Loafer API Broker
- 47. Service (job) • ScheduledJob • No Persistence • Event Triggering • Python 3 Job ⏲
- 48. Client • Web orMobile Applications • No Persistence (or basic persistence) • Web presentation of APIs' resources • Python 3, Django W e b A p p A P I
- 49. Libraries • Common Libraries— common utilities for APIs and Services (eg. event triggering/topic publishing) • Client Libraries — libraries to connect our APIs • Open Source Libraries — useful libraries for community (eg. correios)
- 50. Tools • Data Migrator— tool that connects in all databases and provides a small framework for data migration. It uses Kenneth Reiz's records library. It was initially used to migrate data from the old version of our application. • Toolbelt — tool that provide basic management commands to interact with our APIs, partner APIs and to make ease to manage SQS queues or trigger some events in SNS Topics.
- 51.
- 52.
- 53.
- 54.
- 55.
- 56.
- 57.
- 58.
- 59. Working Sample Webapp Fulfillment API http request topic trigger event queue plp generating service Carrier API topic queue plp-closing-servicesuccess queue generate pdf uploadpdf to s3 set tracking code and pdf url at fulfillment-api Correios' API
- 60. Working Sample Webapp Fulfillment API http request topic trigger event queue plp generating service Carrier API topic queue plp-closing-servicesuccess queue generate pdf uploadpdf to s3 set tracking code and pdf url at fulfillment-api Correios' API
- 61.
- 64. • Hosted onHeroku PaaS (our secret weapon!) • Easy deployment, configuration management, log handling, etc • Heroku PostgreSQL Database • Easy deployment, easy configuration and setup, easy backup, replica and foreign data wrappers • Other services and tools • Logentries, Sentry and New Relic
- 65.
- 66. • It's hardto make evolution of message/resource contracts between services • All sequential process must be splitted over multiple (small) services • Denormalization of data can easily lead to problems of data consistency if we do not take certain precautions • Information needed for one API must be replicated through services and stored locally • Data migration or refactoring in several services requires the development of an specific application
- 67.
- 68.
- 69. Development Tools • Github- code management • CircleCI - continuous integration • vim / PyCharm / SublimeText / etc - development
- 70. Communication Tools • JIRA- project management • Confluence - documentation • Google Apps - Mail, Calendar, Docs • Slack - chat and monitoring integrations • Mumble - voice conference • tmux, ngrok, vim (and mumble) - pair programming
- 71.