Download to read offline
Uploaded byOsvaldo Santana Neto
Plataforma distribuída de Microserviços ou, como a Olist funciona
The document describes Olist's migration from a monolithic to a microservices architecture. Some key points: - The original version (V1) was built with AngularJS in a monolithic structure, which lacked scalability, reliability, and safety. - The new architecture uses microservices with inter-service communication via APIs and asynchronous messaging. Services are independent and communicate through RESTful APIs or by publishing messages to queues. - The architecture provides modularity, scalability, resilience, and safety compared to the monolithic version. Example workflows are described to show how different services interact to complete tasks. - Development is remote using tools like GitHub, JIRA, Slack for collaboration and Her
More Related Content
Similar to Plataforma distribuída de Microserviços ou, como a Olist funciona
![[BDD 2025 - Full-Stack Development] PHP in AI Age: The Laravel Way. (Rizqy Hi...](https://cdn.slidesharecdn.com/ss_thumbnails/fs-phpinaiagethelaravelway-251125012602-ef9d330e-thumbnail.jpg?width=640&height=640&fit=bounds)
![[BDD 2025 - Artificial Intelligence] Building AI Systems That Users (and Comp...](https://cdn.slidesharecdn.com/ss_thumbnails/ai-buildingaisystemsthatusersandcompanieslove-251124030845-038f7732-thumbnail.jpg?width=640&height=640&fit=bounds)
![[BDD 2025 - Artificial Intelligence] AI for the Underdogs: Innovation for Sma...](https://cdn.slidesharecdn.com/ss_thumbnails/ai-aifortheunderdogsinnovationforsmallbusinesses-251124030839-72a599a4-thumbnail.jpg?width=640&height=640&fit=bounds)
Plataforma distribuída de Microserviços ou, como a Olist funciona
- 1.
- 2.
- 3.
- 4.
- 5.
- 6.
- 7.
- 8.
- 9.
- 10.
- 11. LET'S WRITE ANEWVERSION
- 12.
- 13.
- 14.
- 15.
- 16.
- 17.
- 18. PREMISSE • No matterif a system is internal or external, it eventually… • … goes offline… • … crashes… • … or change their behaviour without notice.
- 19.
- 20.
- 21.
- 22.
- 23.
- 24.
- 25. COMMUNICATIONVIA API • RESTfulcommunication between services • Synchronous Service Calls • Strict dependency between services A P I A P I G a t e w a y A P I A P I HTTP Requests
- 26. COMMUNICATIONVIA API • RESTfulcommunication between services • Synchronous Service Calls • Strict dependency between services A P I A P I G a t e w a y A P I A P I HTTP Requests
- 27. COMMUNICATIONVIA API • RESTfulcommunication between services • Synchronous Service Calls • Strict dependency between services A P I A P I G a t e w a y A P I A P I HTTP Requests
- 28. COMMUNICATIONVIA API • RESTfulcommunication between services • Synchronous Service Calls • Strict dependency between services A P I A P I G a t e w a y A P I A P I HTTP Requests
- 29. COMMUNICATIONVIA API • RESTfulcommunication between services • Synchronous Service Calls • Strict dependency between services • No resilience • No safety* (eg. data loss on request failures) A P I A P I G a t e w a y A P I A P I HTTP Requests
- 30. Workers ASYNCTASK EXECUTION A P I Task #1 Task#2 Task #3 call task #1 call task #2 call task #3
- 31. Workers ASYNCTASK EXECUTION • AsynchronousProcedure Calls • Queue based task execution • Client must knows about their dependents (I've to change API every time I need to add a new task) A P I Task #1 Task #2 Task #3 call task #1 call task #2 call task #3
- 32. Workers ASYNCTASK EXECUTION • AsynchronousProcedure Calls • Queue based task execution • Client must knows about their dependents (I've to change API every time I need to add a new task) A P I Task #1 Task #2 Task #3 call task #1 call task #2 call task #3
- 33. Workers ASYNCTASK EXECUTION • AsynchronousProcedure Calls • Queue based task execution • Client must knows about their dependents (I've to change API every time I need to add a new task) A P I Task #1 Task #2 Task #3 call task #1 call task #2 call task #3
- 34. Workers ASYNCTASK EXECUTION • AsynchronousProcedure Calls • Queue based task execution • Client must knows about their dependents (I've to change API every time I need to add a new task) A P I Task #1 Task #2 Task #3 call task #1 call task #2 call task #3
- 35. Workers ASYNCTASK EXECUTION • AsynchronousProcedure Calls • Queue based task execution • Client must knows about their dependents (I've to change API every time I need to add a new task) A P I Task #1 Task #2 Task #3 call task #1 call task #2 call task #3 Task #4 call task #4
- 36. Workers ASYNCTASK EXECUTION • AsynchronousProcedure Calls • Queue based task execution • Client must knows about their dependents (I've to change API every time I need to add a new task) • No decoupling ➡ No modularity • No safety* (eg. data loss on deployment failures) A P I Task #1 Task #2 Task #3 call task #1 call task #2 call task #3 Task #4 call task #4
- 37.
- 38. MESSAGE EVENTTRIGGERING • ActionEvent triggering • Queue based message event handling • Event as messages Event A P I Service Consumer Service Consumer Service Consumer queues
- 39. MESSAGE EVENTTRIGGERING • ActionEvent triggering • Queue based message event handling • Event as messages Event A P I Service Consumer Service Consumer Service Consumer queues
- 40. MESSAGE EVENTTRIGGERING • ActionEvent triggering • Queue based message event handling • Event as messages Event A P I Service Consumer Service Consumer Service Consumer queues
- 41. MESSAGE EVENTTRIGGERING • ActionEvent triggering • Queue based message event handling • Event as messages • It works! Event A P I Service Consumer Service Consumer Service Consumer queues
- 42.
- 43. MESSAGES • Also knownas Resource in REST context • Follow a contract (schema) • Can be enveloped with metadata (eg. SNS/SQS metadata)
- 44. GLOBALTOPICS • Publisher inPubSub Pattern • Global topics for message publication • Topics belong to the system (or architecture) and not to a (micro)service • We use AWS SNS • Kafka was not available on our hosting provider (Heroku) at the time we choose SNS
- 45. SERVICE QUEUES • Subscribersin PubSub Pattern • Queues subscribe topics • One queue belong exclusively to one (micro)service • We use AWS SQS • SQS can be used as a SNS subscriber
- 46. API • Main DataEntry Point • DataValidation • Data Persistence • Workflow Management (eg. status and state machine) • EventTriggering • Idempotency handling (eg. discard duplicated requests returning a HTTP 304 Not Modified) • Python 3, Django REST Framework A P I
- 47. WEBHOOK • Data EntryPoint • No Data Persistence • Proxy HTTP ➡ SNS • EventTriggering • Python 3, Django REST Framework W e b h o o k
- 48. SERVICE • Event Handling/ Message Processing • Business Logic • Some service could trigger some events but it's not so usual • ServiceTypes: • Dequeuer - process messages from one queue • Broker - process messages from multiple queues • Python 3, Loafer / Asyncio API #1 Event #1 Service
- 49. JOB • Scheduled Job •No Persistence • EventTriggering • Python 3, etc. Job ⏲
- 50. CLIENT • Web orMobile Applications • No Persistence (or basic persistence) • Web presentation of APIs' resources • Python 3, Django W e b A p p A P I
- 51. LIBRARIES • Common Libraries— common utilities for APIs and Services (eg. event triggering/topic publishing) • Client Libraries — libraries to connect our APIs • Auth Library — library to handle authentication basics • Open Source Libraries — useful libraries for community (eg. correios)
- 52. TOOLS • Legacy DataMigrator — tool that connects in all databases and provides a small framework for data migration. It uses Kenneth Reiz's records library. It was initially used to migrate data from the old version of our application. • Toolbelt — tool that provide basic management commands to interact with our APIs, partner APIs and to make ease to manage SQS queues or trigger some events in SNSTopics.
- 53. WORKING SAMPLE posting listand shipping label generation Webapp Fulfillment API http request topic trigger event queue plp generating service Internal Carrier API topic queue plp-closing-service External Service success queue • generate pdf • upload pdf to s3 • set tracking code and pdf url at fulfillment-api
- 54. WORKING SAMPLE posting listand shipping label generation Webapp Fulfillment API http request topic trigger event queue plp generating service Internal Carrier API topic queue plp-closing-service External Service success queue • generate pdf • upload pdf to s3 • set tracking code and pdf url at fulfillment-api
- 55. WORKING SAMPLE posting listand shipping label generation Webapp Fulfillment API http request topic trigger event queue plp generating service Internal Carrier API topic queue plp-closing-service External Service success queue • generate pdf • upload pdf to s3 • set tracking code and pdf url at fulfillment-api
- 56. WORKING SAMPLE posting listand shipping label generation Webapp Fulfillment API http request topic trigger event queue plp generating service Internal Carrier API topic queue plp-closing-service External Service success queue • generate pdf • upload pdf to s3 • set tracking code and pdf url at fulfillment-api
- 57. WORKING SAMPLE posting listand shipping label generation Webapp Fulfillment API http request topic trigger event queue plp generating service Internal Carrier API topic queue plp-closing-service External Service success queue • generate pdf • upload pdf to s3 • set tracking code and pdf url at fulfillment-api
- 58. WORKING SAMPLE posting listand shipping label generation Webapp Fulfillment API http request topic trigger event queue plp generating service Internal Carrier API topic queue plp-closing-service External Service success queue • generate pdf • upload pdf to s3 • set tracking code and pdf url at fulfillment-api
- 59. WORKING SAMPLE posting listand shipping label generation Webapp Fulfillment API http request topic trigger event queue plp generating service Internal Carrier API topic queue plp-closing-service External Service success queue • generate pdf • upload pdf to s3 • set tracking code and pdf url at fulfillment-api
- 60.
- 61.
- 62. DEVELOPMENTTOOLS • Github -code management • CircleCI - continuous integration • vim / PyCharm / SublimeText / etc - development
- 63. COMMUNICATIONTOOLS • JIRA -project management • Confluence - documentation • Google Apps - Mail, Calendar, Docs • Slack - chat and monitoring integrations • Mumble - voice conference • tmux, ngrok, vim (and mumble) - pair programming
- 64.
- 66. • Hosted onHeroku PaaS (our secret weapon!) • Easy deployment, configuration management, log handling, etc • Heroku PostgreSQL Database • Easy deployment, easy configuration and setup, easy backup, replica and foreign data wrappers • Other services and tools • Logentries, Sentry and New Relic
- 67.
- 68. • It's hardto make evolution of message/resource contracts between services • All sequential process must be splitted over multiple (small) services • Denormalization of data can easily lead to problems of data consistency if we do not take certain precautions • Information needed for one API must be replicated through services and stored locally • Data migration or refactoring in several services requires the development of an specific application