Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
OLIST ARCHITECTURE
Merchants
Online & Offline
BEFORE…
OLISTVERSION ONE (AKAV1)
AngularJS
OLISTVERSION ONE (AKAV1)
AngularJS
COMPLEX!
MONOLITHIC
MONOLITHIC
No Scalability
MONOLITHIC
No Scalability
No Reliability
MONOLITHIC
No Scalability
No Reliability
No Safety
THEN…
LET'S WRITE A NEWVERSION
REQUIREMENTS
SIMPLICITY
SCALABILITY
RESILIENCE
MODULARITY
SAFETY
PREMISSE
• No matter if a system is internal or external, it eventually…
• … goes offline…
• … crashes…
• … or change their...
A NEW ARCHITECTURE
MICROSERVICES
MICROSERVICES or SOA?
MICROSERVICES or SOA?
IMPLEMENTATION MODELS
COMMUNICATIONVIA API
A
P
I
A
P
I
G
a
t
e
w
a
y
A
P
I
A
P
I
HTTP Requests
COMMUNICATIONVIA API
• RESTful communication between
services
• Synchronous Service Calls
• Strict dependency between serv...
COMMUNICATIONVIA API
• RESTful communication between
services
• Synchronous Service Calls
• Strict dependency between serv...
COMMUNICATIONVIA API
• RESTful communication between
services
• Synchronous Service Calls
• Strict dependency between serv...
COMMUNICATIONVIA API
• RESTful communication between
services
• Synchronous Service Calls
• Strict dependency between serv...
COMMUNICATIONVIA API
• RESTful communication between
services
• Synchronous Service Calls
• Strict dependency between serv...
Workers
ASYNCTASK EXECUTION
A
P
I
Task #1
Task #2
Task #3
call task #1
call task #2
call task #3
Workers
ASYNCTASK EXECUTION
• Asynchronous Procedure Calls
• Queue based task execution
• Client must knows about their
de...
Workers
ASYNCTASK EXECUTION
• Asynchronous Procedure Calls
• Queue based task execution
• Client must knows about their
de...
Workers
ASYNCTASK EXECUTION
• Asynchronous Procedure Calls
• Queue based task execution
• Client must knows about their
de...
Workers
ASYNCTASK EXECUTION
• Asynchronous Procedure Calls
• Queue based task execution
• Client must knows about their
de...
Workers
ASYNCTASK EXECUTION
• Asynchronous Procedure Calls
• Queue based task execution
• Client must knows about their
de...
Workers
ASYNCTASK EXECUTION
• Asynchronous Procedure Calls
• Queue based task execution
• Client must knows about their
de...
MESSAGE EVENTTRIGGERING
Event
A
P
I
Service
Consumer
Service
Consumer
Service
Consumer
queues
MESSAGE EVENTTRIGGERING
• Action Event triggering
• Queue based message event handling
• Event as messages
Event
A
P
I
Ser...
MESSAGE EVENTTRIGGERING
• Action Event triggering
• Queue based message event handling
• Event as messages
Event
A
P
I
Ser...
MESSAGE EVENTTRIGGERING
• Action Event triggering
• Queue based message event handling
• Event as messages
Event
A
P
I
Ser...
MESSAGE EVENTTRIGGERING
• Action Event triggering
• Queue based message event handling
• Event as messages
• It works!
Eve...
BASIC BUILDING BLOCKS
MESSAGES
• Also known as Resource in REST
context
• Follow a contract (schema)
• Can be enveloped with metadata (eg.
SNS/S...
GLOBALTOPICS
• Publisher in PubSub Pattern
• Global topics for message publication
• Topics belong to the system (or
archi...
SERVICE QUEUES
• Subscribers in PubSub Pattern
• Queues subscribe topics
• One queue belong exclusively to one
(micro)serv...
API
• Main Data Entry Point
• DataValidation
• Data Persistence
• Workflow Management (eg. status and state
machine)
• Even...
WEBHOOK
• Data Entry Point
• No Data Persistence
• Proxy HTTP ➡ SNS
• EventTriggering
• Python 3, Django REST Framework
W
...
SERVICE
• Event Handling / Message Processing
• Business Logic
• Some service could trigger some events but it's
not so us...
JOB
• Scheduled Job
• No Persistence
• EventTriggering
• Python 3, etc.
Job
⏲
CLIENT
• Web or Mobile Applications
• No Persistence (or basic persistence)
• Web presentation of APIs' resources
• Python...
LIBRARIES
• Common Libraries — common utilities for APIs and Services (eg.
event triggering/topic publishing)
• Client Lib...
TOOLS
• Legacy Data Migrator — tool that connects in all databases and
provides a small framework for data migration. It u...
WORKING SAMPLE

posting list and shipping label generation
Webapp
Fulfillment

API
http

request
topic
trigger

event
queue...
WORKING SAMPLE

posting list and shipping label generation
Webapp
Fulfillment

API
http

request
topic
trigger

event
queue...
WORKING SAMPLE

posting list and shipping label generation
Webapp
Fulfillment

API
http

request
topic
trigger

event
queue...
WORKING SAMPLE

posting list and shipping label generation
Webapp
Fulfillment

API
http

request
topic
trigger

event
queue...
WORKING SAMPLE

posting list and shipping label generation
Webapp
Fulfillment

API
http

request
topic
trigger

event
queue...
WORKING SAMPLE

posting list and shipping label generation
Webapp
Fulfillment

API
http

request
topic
trigger

event
queue...
WORKING SAMPLE

posting list and shipping label generation
Webapp
Fulfillment

API
http

request
topic
trigger

event
queue...
DEVELOPMENT
REMOTETEAM
DEVELOPMENTTOOLS
• Github - code management
• CircleCI - continuous integration
• vim / PyCharm / SublimeText / etc - deve...
COMMUNICATIONTOOLS
• JIRA - project management
• Confluence - documentation
• Google Apps - Mail, Calendar, Docs
• Slack - ...
DEPLOYMENT
• Hosted on Heroku PaaS (our secret weapon!)
• Easy deployment, configuration management, log handling, etc
• Heroku Postgr...
CHALLENGES
• It's hard to make evolution of message/resource contracts between
services
• All sequential process must be splitted ove...
Plataforma distribuída de Microserviços ou, como a Olist funciona
You’ve finished this document.
Download and read it offline.
Upcoming SlideShare
What to Upload to SlideShare
Next

of

YouTube videos are no longer supported on SlideShare

View original on YouTube

Plataforma distribuída de Microserviços ou, como a Olist funciona Slide 2 Plataforma distribuída de Microserviços ou, como a Olist funciona Slide 3 Plataforma distribuída de Microserviços ou, como a Olist funciona Slide 4 Plataforma distribuída de Microserviços ou, como a Olist funciona Slide 5 Plataforma distribuída de Microserviços ou, como a Olist funciona Slide 6 Plataforma distribuída de Microserviços ou, como a Olist funciona Slide 7 Plataforma distribuída de Microserviços ou, como a Olist funciona Slide 8 Plataforma distribuída de Microserviços ou, como a Olist funciona Slide 9 Plataforma distribuída de Microserviços ou, como a Olist funciona Slide 10 Plataforma distribuída de Microserviços ou, como a Olist funciona Slide 11 Plataforma distribuída de Microserviços ou, como a Olist funciona Slide 12 Plataforma distribuída de Microserviços ou, como a Olist funciona Slide 13 Plataforma distribuída de Microserviços ou, como a Olist funciona Slide 14 Plataforma distribuída de Microserviços ou, como a Olist funciona Slide 15 Plataforma distribuída de Microserviços ou, como a Olist funciona Slide 16 Plataforma distribuída de Microserviços ou, como a Olist funciona Slide 17 Plataforma distribuída de Microserviços ou, como a Olist funciona Slide 18 Plataforma distribuída de Microserviços ou, como a Olist funciona Slide 19 Plataforma distribuída de Microserviços ou, como a Olist funciona Slide 20 Plataforma distribuída de Microserviços ou, como a Olist funciona Slide 21 Plataforma distribuída de Microserviços ou, como a Olist funciona Slide 22 Plataforma distribuída de Microserviços ou, como a Olist funciona Slide 23 Plataforma distribuída de Microserviços ou, como a Olist funciona Slide 24 Plataforma distribuída de Microserviços ou, como a Olist funciona Slide 25 Plataforma distribuída de Microserviços ou, como a Olist funciona Slide 26 Plataforma distribuída de Microserviços ou, como a Olist funciona Slide 27 Plataforma distribuída de Microserviços ou, como a Olist funciona Slide 28 Plataforma distribuída de Microserviços ou, como a Olist funciona Slide 29 Plataforma distribuída de Microserviços ou, como a Olist funciona Slide 30 Plataforma distribuída de Microserviços ou, como a Olist funciona Slide 31 Plataforma distribuída de Microserviços ou, como a Olist funciona Slide 32 Plataforma distribuída de Microserviços ou, como a Olist funciona Slide 33 Plataforma distribuída de Microserviços ou, como a Olist funciona Slide 34 Plataforma distribuída de Microserviços ou, como a Olist funciona Slide 35 Plataforma distribuída de Microserviços ou, como a Olist funciona Slide 36 Plataforma distribuída de Microserviços ou, como a Olist funciona Slide 37 Plataforma distribuída de Microserviços ou, como a Olist funciona Slide 38 Plataforma distribuída de Microserviços ou, como a Olist funciona Slide 39 Plataforma distribuída de Microserviços ou, como a Olist funciona Slide 40 Plataforma distribuída de Microserviços ou, como a Olist funciona Slide 41 Plataforma distribuída de Microserviços ou, como a Olist funciona Slide 42 Plataforma distribuída de Microserviços ou, como a Olist funciona Slide 43 Plataforma distribuída de Microserviços ou, como a Olist funciona Slide 44 Plataforma distribuída de Microserviços ou, como a Olist funciona Slide 45 Plataforma distribuída de Microserviços ou, como a Olist funciona Slide 46 Plataforma distribuída de Microserviços ou, como a Olist funciona Slide 47 Plataforma distribuída de Microserviços ou, como a Olist funciona Slide 48 Plataforma distribuída de Microserviços ou, como a Olist funciona Slide 49 Plataforma distribuída de Microserviços ou, como a Olist funciona Slide 50 Plataforma distribuída de Microserviços ou, como a Olist funciona Slide 51 Plataforma distribuída de Microserviços ou, como a Olist funciona Slide 52 Plataforma distribuída de Microserviços ou, como a Olist funciona Slide 53 Plataforma distribuída de Microserviços ou, como a Olist funciona Slide 54 Plataforma distribuída de Microserviços ou, como a Olist funciona Slide 55 Plataforma distribuída de Microserviços ou, como a Olist funciona Slide 56 Plataforma distribuída de Microserviços ou, como a Olist funciona Slide 57 Plataforma distribuída de Microserviços ou, como a Olist funciona Slide 58 Plataforma distribuída de Microserviços ou, como a Olist funciona Slide 59 Plataforma distribuída de Microserviços ou, como a Olist funciona Slide 60 Plataforma distribuída de Microserviços ou, como a Olist funciona Slide 61 Plataforma distribuída de Microserviços ou, como a Olist funciona Slide 62 Plataforma distribuída de Microserviços ou, como a Olist funciona Slide 63 Plataforma distribuída de Microserviços ou, como a Olist funciona Slide 64 Plataforma distribuída de Microserviços ou, como a Olist funciona Slide 65 Plataforma distribuída de Microserviços ou, como a Olist funciona Slide 66 Plataforma distribuída de Microserviços ou, como a Olist funciona Slide 67 Plataforma distribuída de Microserviços ou, como a Olist funciona Slide 68 Plataforma distribuída de Microserviços ou, como a Olist funciona Slide 69
Upcoming SlideShare
What to Upload to SlideShare
Next
Download to read offline and view in fullscreen.

0

Share

Plataforma distribuída de Microserviços ou, como a Olist funciona

Download to read offline

Nessa apresentação eu demonstro com detalhes como a arquitetura da Olist foi implementada usando microserviços distribuídos que se comunicam através de um sistema de mensageria baseado no pattern Publisher/Subscribers (PubSub).

A plataforma Olist usa Python 3.6, Django, Django REST Framework, Loafer (asyncio), AWS SNS/SQS e PostgreSQL. Fazemos deployment no Heroku e desenvolvemos tudo isso com uma equipe distribuída por quase todas as regiões do Brasil (ainda falta um representante do Norte!).

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all
  • Be the first to like this

Plataforma distribuída de Microserviços ou, como a Olist funciona

  1. 1. OLIST ARCHITECTURE
  2. 2. Merchants Online & Offline
  3. 3. BEFORE…
  4. 4. OLISTVERSION ONE (AKAV1) AngularJS
  5. 5. OLISTVERSION ONE (AKAV1) AngularJS COMPLEX!
  6. 6. MONOLITHIC
  7. 7. MONOLITHIC No Scalability
  8. 8. MONOLITHIC No Scalability No Reliability
  9. 9. MONOLITHIC No Scalability No Reliability No Safety
  10. 10. THEN…
  11. 11. LET'S WRITE A NEWVERSION
  12. 12. REQUIREMENTS
  13. 13. SIMPLICITY
  14. 14. SCALABILITY
  15. 15. RESILIENCE
  16. 16. MODULARITY
  17. 17. SAFETY
  18. 18. PREMISSE • No matter if a system is internal or external, it eventually… • … goes offline… • … crashes… • … or change their behaviour without notice.
  19. 19. A NEW ARCHITECTURE
  20. 20. MICROSERVICES
  21. 21. MICROSERVICES or SOA?
  22. 22. MICROSERVICES or SOA?
  23. 23. IMPLEMENTATION MODELS
  24. 24. COMMUNICATIONVIA API A P I A P I G a t e w a y A P I A P I HTTP Requests
  25. 25. COMMUNICATIONVIA API • RESTful communication between services • Synchronous Service Calls • Strict dependency between services A P I A P I G a t e w a y A P I A P I HTTP Requests
  26. 26. COMMUNICATIONVIA API • RESTful communication between services • Synchronous Service Calls • Strict dependency between services A P I A P I G a t e w a y A P I A P I HTTP Requests
  27. 27. COMMUNICATIONVIA API • RESTful communication between services • Synchronous Service Calls • Strict dependency between services A P I A P I G a t e w a y A P I A P I HTTP Requests
  28. 28. COMMUNICATIONVIA API • RESTful communication between services • Synchronous Service Calls • Strict dependency between services A P I A P I G a t e w a y A P I A P I HTTP Requests
  29. 29. COMMUNICATIONVIA API • RESTful communication between services • Synchronous Service Calls • Strict dependency between services • No resilience • No safety* (eg. data loss on request failures) A P I A P I G a t e w a y A P I A P I HTTP Requests
  30. 30. Workers ASYNCTASK EXECUTION A P I Task #1 Task #2 Task #3 call task #1 call task #2 call task #3
  31. 31. Workers ASYNCTASK EXECUTION • Asynchronous Procedure Calls • Queue based task execution • Client must knows about their dependents (I've to change API every time I need to add a new task) A P I Task #1 Task #2 Task #3 call task #1 call task #2 call task #3
  32. 32. Workers ASYNCTASK EXECUTION • Asynchronous Procedure Calls • Queue based task execution • Client must knows about their dependents (I've to change API every time I need to add a new task) A P I Task #1 Task #2 Task #3 call task #1 call task #2 call task #3
  33. 33. Workers ASYNCTASK EXECUTION • Asynchronous Procedure Calls • Queue based task execution • Client must knows about their dependents (I've to change API every time I need to add a new task) A P I Task #1 Task #2 Task #3 call task #1 call task #2 call task #3
  34. 34. Workers ASYNCTASK EXECUTION • Asynchronous Procedure Calls • Queue based task execution • Client must knows about their dependents (I've to change API every time I need to add a new task) A P I Task #1 Task #2 Task #3 call task #1 call task #2 call task #3
  35. 35. Workers ASYNCTASK EXECUTION • Asynchronous Procedure Calls • Queue based task execution • Client must knows about their dependents (I've to change API every time I need to add a new task) A P I Task #1 Task #2 Task #3 call task #1 call task #2 call task #3 Task #4 call task #4
  36. 36. Workers ASYNCTASK EXECUTION • Asynchronous Procedure Calls • Queue based task execution • Client must knows about their dependents (I've to change API every time I need to add a new task) • No decoupling ➡ No modularity • No safety* (eg. data loss on deployment failures) A P I Task #1 Task #2 Task #3 call task #1 call task #2 call task #3 Task #4 call task #4
  37. 37. MESSAGE EVENTTRIGGERING Event A P I Service Consumer Service Consumer Service Consumer queues
  38. 38. MESSAGE EVENTTRIGGERING • Action Event triggering • Queue based message event handling • Event as messages Event A P I Service Consumer Service Consumer Service Consumer queues
  39. 39. MESSAGE EVENTTRIGGERING • Action Event triggering • Queue based message event handling • Event as messages Event A P I Service Consumer Service Consumer Service Consumer queues
  40. 40. MESSAGE EVENTTRIGGERING • Action Event triggering • Queue based message event handling • Event as messages Event A P I Service Consumer Service Consumer Service Consumer queues
  41. 41. MESSAGE EVENTTRIGGERING • Action Event triggering • Queue based message event handling • Event as messages • It works! Event A P I Service Consumer Service Consumer Service Consumer queues
  42. 42. BASIC BUILDING BLOCKS
  43. 43. MESSAGES • Also known as Resource in REST context • Follow a contract (schema) • Can be enveloped with metadata (eg. SNS/SQS metadata)
  44. 44. GLOBALTOPICS • Publisher in PubSub Pattern • Global topics for message publication • Topics belong to the system (or architecture) and not to a (micro)service • We use AWS SNS • Kafka was not available on our hosting provider (Heroku) at the time we choose SNS
  45. 45. SERVICE QUEUES • Subscribers in PubSub Pattern • Queues subscribe topics • One queue belong exclusively to one (micro)service • We use AWS SQS • SQS can be used as a SNS subscriber
  46. 46. API • Main Data Entry Point • DataValidation • Data Persistence • Workflow Management (eg. status and state machine) • EventTriggering • Idempotency handling (eg. discard duplicated requests returning a HTTP 304 Not Modified) • Python 3, Django REST Framework A P I
  47. 47. WEBHOOK • Data Entry Point • No Data Persistence • Proxy HTTP ➡ SNS • EventTriggering • Python 3, Django REST Framework W e b h o o k
  48. 48. SERVICE • Event Handling / Message Processing • Business Logic • Some service could trigger some events but it's not so usual • ServiceTypes: • Dequeuer - process messages from one queue • Broker - process messages from multiple queues • Python 3, Loafer / Asyncio API #1 Event #1 Service
  49. 49. JOB • Scheduled Job • No Persistence • EventTriggering • Python 3, etc. Job ⏲
  50. 50. CLIENT • Web or Mobile Applications • No Persistence (or basic persistence) • Web presentation of APIs' resources • Python 3, Django W e b A p p A P I
  51. 51. LIBRARIES • Common Libraries — common utilities for APIs and Services (eg. event triggering/topic publishing) • Client Libraries — libraries to connect our APIs • Auth Library — library to handle authentication basics • Open Source Libraries — useful libraries for community (eg. correios)
  52. 52. TOOLS • Legacy Data Migrator — tool that connects in all databases and provides a small framework for data migration. It uses Kenneth Reiz's records library. It was initially used to migrate data from the old version of our application. • Toolbelt — tool that provide basic management commands to interact with our APIs, partner APIs and to make ease to manage SQS queues or trigger some events in SNSTopics.
  53. 53. WORKING SAMPLE
 posting list and shipping label generation Webapp Fulfillment
 API http
 request topic trigger
 event queue plp generating service Internal Carrier
 API topic queue plp-closing-service External Service success queue • generate pdf • upload pdf to s3 • set tracking code and pdf url at fulfillment-api
  54. 54. WORKING SAMPLE
 posting list and shipping label generation Webapp Fulfillment
 API http
 request topic trigger
 event queue plp generating service Internal Carrier
 API topic queue plp-closing-service External Service success queue • generate pdf • upload pdf to s3 • set tracking code and pdf url at fulfillment-api
  55. 55. WORKING SAMPLE
 posting list and shipping label generation Webapp Fulfillment
 API http
 request topic trigger
 event queue plp generating service Internal Carrier
 API topic queue plp-closing-service External Service success queue • generate pdf • upload pdf to s3 • set tracking code and pdf url at fulfillment-api
  56. 56. WORKING SAMPLE
 posting list and shipping label generation Webapp Fulfillment
 API http
 request topic trigger
 event queue plp generating service Internal Carrier
 API topic queue plp-closing-service External Service success queue • generate pdf • upload pdf to s3 • set tracking code and pdf url at fulfillment-api
  57. 57. WORKING SAMPLE
 posting list and shipping label generation Webapp Fulfillment
 API http
 request topic trigger
 event queue plp generating service Internal Carrier
 API topic queue plp-closing-service External Service success queue • generate pdf • upload pdf to s3 • set tracking code and pdf url at fulfillment-api
  58. 58. WORKING SAMPLE
 posting list and shipping label generation Webapp Fulfillment
 API http
 request topic trigger
 event queue plp generating service Internal Carrier
 API topic queue plp-closing-service External Service success queue • generate pdf • upload pdf to s3 • set tracking code and pdf url at fulfillment-api
  59. 59. WORKING SAMPLE
 posting list and shipping label generation Webapp Fulfillment
 API http
 request topic trigger
 event queue plp generating service Internal Carrier
 API topic queue plp-closing-service External Service success queue • generate pdf • upload pdf to s3 • set tracking code and pdf url at fulfillment-api
  60. 60. DEVELOPMENT
  61. 61. REMOTETEAM
  62. 62. DEVELOPMENTTOOLS • Github - code management • CircleCI - continuous integration • vim / PyCharm / SublimeText / etc - development
  63. 63. COMMUNICATIONTOOLS • JIRA - project management • Confluence - documentation • Google Apps - Mail, Calendar, Docs • Slack - chat and monitoring integrations • Mumble - voice conference • tmux, ngrok, vim (and mumble) - pair programming
  64. 64. DEPLOYMENT
  65. 65. • Hosted on Heroku PaaS (our secret weapon!) • Easy deployment, configuration management, log handling, etc • Heroku PostgreSQL Database • Easy deployment, easy configuration and setup, easy backup, replica and foreign data wrappers • Other services and tools • Logentries, Sentry and New Relic
  66. 66. CHALLENGES
  67. 67. • It's hard to make evolution of message/resource contracts between services • All sequential process must be splitted over multiple (small) services • Denormalization of data can easily lead to problems of data consistency if we do not take certain precautions • Information needed for one API must be replicated through services and stored locally • Data migration or refactoring in several services requires the development of an specific application

Nessa apresentação eu demonstro com detalhes como a arquitetura da Olist foi implementada usando microserviços distribuídos que se comunicam através de um sistema de mensageria baseado no pattern Publisher/Subscribers (PubSub). A plataforma Olist usa Python 3.6, Django, Django REST Framework, Loafer (asyncio), AWS SNS/SQS e PostgreSQL. Fazemos deployment no Heroku e desenvolvemos tudo isso com uma equipe distribuída por quase todas as regiões do Brasil (ainda falta um representante do Norte!).

Views

Total views

3,778

On Slideshare

0

From embeds

0

Number of embeds

2,444

Actions

Downloads

8

Shares

0

Comments

0

Likes

0

×