Microservices with Zoran Majstorović discusses moving from monolithic applications to microservices architectures using RabbitMQ/AMQP for messaging. It covers refactoring Ruby code using callbacks to use a consumer microservice. It also demonstrates a simple way to deploy a microservice in a Docker container to AWS ECS. Key topics include decoupling services, different integration options like messaging, AMQP concepts, and using RabbitMQ with Ruby.

1.
Microservices
with
Zoran Majstorović
github.com/zmajstor
twitter.com/z0maj

2. Content
• Intro
from monolith decoupling to microservices integration
• The Main Part
microservices messaging with RabbitMQ/AMQP
• Ruby Code Refactoring
from ActiveRecord Callback to Consumer as a Microservice
• A Simple Microservice Deployment
in docker container to AWS ECS

3. Let's start with a bad
example

4. A better example
... or can we just publish the event and let subscribed consumers do the work?
a complete to-do list for this event

5. Decoupling
Publisher / Producer
Subscriber /
Consumer
Subscriber /
Consumer
Subscriber /
Consumer

6. Monolith vs Microservices
vs
Complexity in
Interactions
Code
Complexity

7. Microservice
• a small program that handle one task
• independently deployable
• work together by communicating so that can accomplish
the larger task
• integrated system on the whole provides value
https://martinfowler.com/
articles/microservices.html
https://youtu.be/wgdBVIX9ifA

8. App Integration Options
• File Transfer
• Shared Database
• Remote Procedure Invocation
• Messaging
While all four approaches solve essentially the same problem,
each style has its distinct advantages and disadvantages.
http://www.enterpriseintegrationpatterns.com/patterns/messaging/IntegrationStylesIntro.html

9. File Transfer
Consumer A
Consumer B
Consumer C
DATA
FILE
Producer
Export Import

10. Shared Database
Consumer A
Consumer B
Consumer C
Producer DB

11. Remote Procedure
Invocation
Consumer A
Consumer B
Consumer C
Producer
Call
Result

12. Messaging
Exchanges Bindings Queues
Consumer A
Consumer B
Consumer C
Producer
AMQP 0-9-1 Message Broker

13. • a binary, networking protocol with minimal overhead
• originated in 2003 by John O'Hara at JPMorgan Chase
• version 0.9.1 was specified in 2008
• in 2011 OASIS standards group change it to a
significantly different beast

14. AMQP 0-9-1
• virtual host is a logical partition within the server
(a multi-tenant system similar to virtual hosts in Apache/Nginx)
• each of virtual hosts can have many connections,
channels, queues, exchanges and some other things
• bindings binds an exchange to a queue or many queues
• exchange can be: direct, fanout, with topic or headers
• queue is a buffer that holds messages on behalf of a
consumer (or consumers)

15. The Producer
• external application which creates messages and
decides:
• how the attributes should be configured for routing
• from which exchange the messages should start from
• what is the actual payload that is being sent

16. The Message
• an atomic unit of processing
• consists of:
• content (body, bare message or payload)
• attributes - metadata about the message like content type,
encoding, routing key, whether the message will be persistent or
not, and whether it has a priority level, etc.
• is created (published) by the producer
• may go through more than one exchange before landing in
the right queue

17. The Exchange
• Direct exchange route messages based on an exact match with the
specified routing key
• Fanout exchange automatically route the message to all the queues
known to them (ignores the routing key)
• Topic exchange pattern match on the routing key (topic) to route the
messages
• Header exchange use the message header attributes for matching the
queue
• Default (anonymous) exchange is a direct exchange (created
automatically) which routes messages with empty exchange name - it
compares routing key with the queue name

18. Multiple Queues and
Consumers
Consumer A
Queue 1
Queue 2
Queue 3
Consumer B
Consumer C

19. The Consumer
• external application which is subscribed to one or more queues
• alerted whenever a message shows up in the subscribed queue
• can poll the queue at regular intervals to see which messages
were added in the queue since the last time it made the request
• can send acknowledgments back to RabbitMQ that the
message has been:
• received (known as ack), or
• rejected (nack for negative acknowledgments)

20. Bindings
• Routing Key
• Publisher: exchange.publish(payload, routing_key: 'foo')
• Queue Binding: queue.bind(exchange, routing_key: 'foo')
• Headers
• Publisher: exchange.publish(payload, headers: { ... })
• Queue Binding: queue.bind(exchange, arguments: { ... })

21. • an message broker implemented with Erlang/OTP
• implements all the AMQP 0.9.1 concepts:
messages, queues, exchanges, bindings, virtual hosts ...
• ... and with plugins for other messaging protocols such as
STOMP, XMPP, and more
• has fantastic client support in a variety of popular
languages: Ruby, Python, Java, PHP, C#, JavaScript, Go, Elixir,
Objective-C, Swift, ...

22. Gems
• Bunny - AMPQ client for Ruby (MRI)
• March Hare - AMPQ client for JRuby
• Sneakers - a background consumer/worker
• Hutch - asynchronous inter-service communication
• RabbitMQ HTTP API client (various management features)
• Ruby RabbitMQ Clients Blog: http://
blog.rubyrabbitmq.info/

23. Features
• No message loss
• Persistent Messages
• Publisher Confirms
• Message Acknowledgment
• Mirrored Queues
• Message Ordering
• Dead Letter Exchanges
• Alternate Exchanges
• Message and Queues TTLs
• Consumer Priorities
• Federation
... and many more

24. Back to Refactoring
... or can we just publish the event and let subscribed consumers do the work?

25. Introducing Publisher

26. Publish Message with Topic

27. Message Consumer

28. Publish Message with
Headers

29. Message Consumer

30. Better Ruby Code
https://github.com/jondot/sneakers
https://github.com/gocardless/hutch

31. Patterns
=
proven solutions
to recurring
problems
www.enterpriseintegrationpatterns.com

32. Messaging Patterns (65)
Creative Commons Attribution 4.0 license.http://www.enterpriseintegrationpatterns.com/patterns/messaging/index.html

33. Microservice Messaging
Benefits
• language-agnostic
• amplifies loose-coupling
• makes scaling and rearchitecting simpler
• better reliability and availability - messages will be waiting
until the consumer is ready to process them
• multiple communication patterns: events, message/reply,
notification, async request-response, pub/sub, etc.

34. Microservice Messaging
Drawbacks
• introducing new complexity into the system
(the message broker)
• a failure in the message broker can cause severe effects
on the system (highly-available message broker)
• big messages can cause network congestion
• "eventual data consistency" (instead of immediate
consistency across different microservices)

35. Message Consumer Microservice
Deployment
with Docker
to AWS ECS

36. Installing Docker CE on
Workstation
• macOS: https://store.docker.com/editions/community/docker-ce-
desktop-mac
• Linux Ubuntu: https://store.docker.com/editions/community/docker-ce-
server-ubuntu
• or any other Linux distro: https://store.docker.com/search?
offering=community&operating_system=linux&type=edition

37. My Ruby Project Structure

38. Dockerfile

39. docker-compose.yml
docker-compose build my_consumer:1.0.0

40. Elastic Container Registry
export repositoryName=my_consumer
export repositoryURI=123456789012.dkr.ecr.us-east-1.amazonaws.com
export containerName=my_consumer
export ver=1.0.0
export imageURI=$repositoryURI/$repositoryName:$ver
aws ecr create-repository --repository-name $repositoryName
eval $(aws ecr get-login --no-include-email)
docker tag $containerName:$ver $imageURI
docker push $imageURI

41. Elastic Container
Service (ECS)
• logical way to group resources (Tasks and Services)
• currently provides two launch types:
• EC2
• Fargate (abstract away EC2 instances)
collection of ECS resources: https://github.com/nathanpeck/awesome-ecs
export clusterName=staging
aws ecs create-cluster --cluster-name $clusterName

42. ECS Cluster
Building Blocks
Task Definition
• specify the resources for a Docker container or group of containers, such as:
docker image (registry), ports, CPU/RAM, logs, any volumes, environment
variables, etc.
Task
• running containers according to the Task Definition (one-off or long-running)
• TaskRole allow access to S3, DynamoDB, etc.
Service
• manage long-running tasks (defines desired count and replace failed containers)
• integrates with Elastic Load Balancer

43. ECS Task Definition
{
"containerDefinitions": [
{
"command": [
"bin/worker"
],
"workingDirectory": "/app",
"essential": true,
"image": $imageURI,
"logConfiguration": {
"logDriver": "awslogs"
},
"name": $containerName
}
],
"cpu": "256",
"memory": "512",
"executionRoleArn": "arn:aws:iam::123456789012:role/ecsTaskExecutionRole",
"family": "my-consumer-fargate",
"networkMode": "awsvpc",
"requiresCompatibilities": [
"FARGATE"
]
}

44. ECS Task
export taskDefinition=my-consumer-fargate
aws ecs register-task-definition
--cli-input-json file://my-consumer-fargate-task-definition.json
aws ecs run-task --launch-type FARGATE
--cluster $clusterName
--task-definition $taskDefinition
--network-configuration
"awsvpcConfiguration={subnets=[subnet-abc], securityGroups=[sg-01]}"

45. ECS Service
• runs and maintains the required number of tasks
associated with the elastic load balancer
aws ecs create-service --cluster $clusterName
--service-name my-consumer-service
--task-definition $taskDefinition
--desired-count 2
--launch-type FARGATE
--network-configuration
"awsvpcConfiguration={subnets=[subnet-abc], securityGroups=[sg-01]}"

46. Deploying Updates
1. Build a new image and push it to the repository
2. Create a new revision of the Task Definition
(revision numbers increment automatically)
3. Update Service to use new Task Definition revision
or simply use: https://github.com/silinternational/ecs-deploy
ecs-deploy -c $clusterName -n my-consumer-service -i $imageURI

47. Advanced Example of AWS
ECS Deployment with Terraform
VPC with 2 subnets
(1 public and 1 private)
in each Availability Zone
https://github.com/duduribeiro/terraform_ecs_fargate_example
https://thecode.pub/easy-deploy-your-docker-applications-to-aws-using-ecs-and-fargate-a988a1cc842f

48. AWS
CodeBuild
AWS
CodePipeline
screenshots from: https://thecode.pub/easy-deploy-your-docker-applications-to-aws-using-ecs-and-fargate-a988a1cc842f

49. AWS ECS
vs Kubernetes
1. AWS ECS can not be run on-premise
2. AWS ECS lacks advanced features
These two differences can either be seen as weakness or as
strengths.

50. That's All!
Thank You