This document summarizes Hoiio's microservices architecture using Mesos, Marathon, and various Netflix OSS tools. Key points: 1) Hoiio uses Mesos and Marathon for container orchestration, deploying microservices as Docker containers across a cluster of machines. Marathon manages tasks and performs health checks. 2) Service discovery is handled by Consul or Eureka. Eureka tracks which services are running on which hosts, and Ribbon clients can load balance requests using this data. 3) The Zuul API gateway acts as a reverse proxy, routing requests to microservices while handling authentication, retries, and routing rules defined in its configuration. It integrates with Eureka/

1. Microservices on Mesos & Netflix
OSS
By hieu@hoiio.com

2. Before we start
(or ground rules)
● Forgive my shaking voice
● Feel free to tell me I’m stupid
● Feel free to interrupt me for
question

3. Agenda
● About Hoiio Stack
● Phoenix Deployment
○ Mesos/Marathon
● Netflix OSS @ Hoiio
○ Service Discovery
■ Consul
■ Eureka
○ API Gateway
● (Monitoring)

4. About Hoiio

5. ● VoIP/SIP
● SMS/Email
● Connected Apps
● HR suite
Services

6. ● Distributed multi-service
Micro-services
● RabbitMQ as message broker
Peer-to-peer via HTTP
● Kafka as event-store
● JAVA/Python/... modules
● Entirely on AWS
Platform
zuul
auth
billingsip
HTTP

7. Phoenix Deployment
Mesos/Marathon

8. ● Abstract cluster of machines to
a single “black-box” machine
● Master nodes, Slave/Agent
nodes
● Tasks are submitted to master
● Master schedules job to one of
the slaves
Mesos

9. http://www.slideshare.net/spodila/prezo-tovmware-june2015

10. ● Framework running on top of
Mesos
● Manage tasks config, number of
instance,...
● Healthcheck
● REST interface
● Mesos as OS, Marathon as Task
Manager
Marathon

11. ● Framework running on top of
Mesos
● Manage tasks config, number of
instance,...
● Healthcheck
● REST interface
● Mesos as OS, Marathon as Task
Manager
Marathon
Mesos
Slave
Mesos
Master
Marathon
CPU/M
emory
Kernel
Schedu
ler
Task
Manager

13. ● Docker as container
○ Supported by Mesos
○ Use AWS ECR as private repo/
Private repo running on Marathon
● Marathon performs healthcheck
and replaces unhealthy
instances
● Replacement takes seconds!
Phoenix?

14. {
"id": "ms-uat-xxx",
"mem": 384,
“cpu”: 0.5,
...

16. Service Discovery
Netflix Eureka

17. ● Eureka Server & Client
● Server route are replicated
● Each Client hold a copy of route
table
● Route table are updated in
background
https://github.com/Netflix/eureka/wiki/
Eureka-at-a-glance

18. ● Eureka
○ Eureka server tracks which service
is running where (which ip and
port?)
○ All records are replicated to all
eureka-clients
● Ribbon
○ Pick a server from records replica
on local eureka-client
○ Make request to picked server
○ Retry if configured
10.0.12.16:1234 10.0.140.21:4321
10.0.140.26:6789
Eureka
10.0.12.16:1234
10.0.140.21:4321,
10.0.140.26:6789
10.0.12.16:1234
10.0.140.21:4321,
10.0.140.26:6789
10.0.12.16:1234
10.0.140.21:4321,
10.0.140.26:6789
10.0.12.16:1234
10.0.12.16:1234
10.0.140.21:4321,
10.0.140.26:6789
R
10.0.140.21:4321
10.0.140.21:4321

19. Auth1
Routes
● Single-point-of-failure? Not
really
○ Route table are replicated
○ Each client has a copy
○ Routes are queried from local copy
● When Eureka is down
○ New servers are not updated
○ Might call to a dead server ->
retry on local server list with
Ribbon
SIP
Auth2
HTTP
Routes
Eureka
Server
Routes
Routes

20. String moduleVipAddress = "call.hoiio.info"
Observable<HttpResponse> response = HoiioRibbonRequest.getInstance().makeRequest(
moduleVipAddress,
UUID.randomUUID().toString(),
httpRequest);

21. ● Timeout and Retry
○ Defined in HoiioRibbonRequest
○ Default:
■ Timeout: 10s
■ Retry:
● Same server: 0
● Next server: 3
○ Can be re-configured
10.0.12.16:1234
10.0.12.16:1234
10.0.140.21:4321,
10.0.140.26:6789
R
10.0.140.21:4321
10.0.140.21:4321

22. HttpClient httpClient;
RetryPolicy retryPolicy;
String moduleVipAddress = "call.hoiio.info"
Integer sameServerRetry = 1
Integer nextServerRetry = 1
retryPolicy = new RetryPolicy(
new DefaultLoadBalancerRetryHandler(
sameServerRetry,
nextServerRetry,
true
)
)
}
Integer timeout = 60;
httpClient = new HttpClient(500, 50, timeout*1000);
Configuration config = new Configuration(moduleVipAddress, httpClient, retryPolicy);
Observable<HttpResponse> httpResponse = HoiioRibbonRequest.getInstance().makeRequest(
config,
correlationId,
httpRequest);

23. Service Discovery
Consul

24. ● Clustering with agent on each
instance
● Service info is shared in cluster
● Agent has REST interface to
register/deregister/checks/quer
y/…
● Zuul-pronted as primary
reversed proxy
Implementation
service.json
service.json
Zuul

26. HoiioConsulLoadBalancer lb = new HoiioConsulLoadBalancer(appName, ConsulService.Info.environment(), tag);
HttpResponse httpResponse;
try {
httpResponse = lb.execute(new HttpCmd(httpRequest))
} catch (NoServerException ignored) {
ZuulLogger.logger.error("No server for " + appName)
httpResponse = responseFactory.get().newHttpResponse(
new BasicStatusLine(HttpVersion.HTTP_1_1, 503, "Service not available"),
null);
}

27. API Gateway
with Netflix Zuul and Archaius

28. ● Single gateway for API
● API mapping for easy
understanding
● Optimize number of request
called
● Reject malformed request
Problems
sms
auth
billingsip
HTTP

29. ● Why Zuul?
○ Apps does not have Eureka Client
○ Cron jobs
○ Exposing API
● What Zuul does
○ Represent API caller (Apps,
Cronjob, Partner,...) to talk to
modules (act as a proxy)
■ Relay request
■ Retry
○ Authenticate request
10.0.12.16:1234
10.0.140.21:4321
10.0.140.26:6789
Eureka
10.0.12.16:1234
10.0.140.21:4321,
10.0.140.26:6789
10.0.12.16:1234
10.0.140.21:4321,
10.0.140.26:6789
10.0.12.16:1234
10.0.140.21:4321,
10.0.140.26:6789
10.0.12.16:1234
10.0.140.21:4321,
10.0.140.26:6789
10.0.12.16:1234
10.0.140.21:4321,
10.0.140.26:6789
Z
Z
/a/b/c
10.0.140.26:6789
10.0.12.16:1234
/a/b/c -> /a/c
/a/c
Microservice

30. ● Pre, Route, Post Filter
○ Groovy filter
○ Has priority
● Integrate with Archaius for
Dynamic configuration
● Integrate with Eureka/Consul
for service discovery
Netflix Zuul
Reject
malformed
Authenticate
Route using
Eureka
Ribbon/
Eureka
Add header
pre
route
post
Archaius
Route mapping
/sms/send /sms/send ->
{“module”:”sms”,
“uri”:”sendOneSms”}
/sendOneSms

31. ● Timeout and Retry
○ Zuul represents API callers to talk
to modules -> must tell Zuul
timeout and retry for each API
○ Default values
■ Timeout: 10s
■ Retry:
● Same server: 0
● Next server: 3
{
"vipAddress": "auth.hoiio.info",
"module": "auth",
“apis”: [
{
"from":"/v1/otp",
"to": "/private/v1/otp",
"type": "private",
"timeout": 60,
"retry": {
"same": 1,
"next": 2
}
}
]
}

34. Monitoring

35. ● Remember Consul?
● Consul watch
○ Trigger action when a service
status changes
Service status
service.json
{
"service": {
"name": "MS-Apps-1-46",
"tags": ["prod"],
"address": "10.0.14.10",
"port": 8080,
"checks": [
{
"script":
"/opt/consul/bin/MS-Apps-1-46-healthcheck.sh",
"interval": "60s"
}
]
}
}

36. ● Metric sources:
○ CollectD/cAdvisor
○ Cloudwatch
● Metric storage:
○ InfluxDB
● Visualization:
○ Grafana
Instance stats
Kapacitor
Cloudwatch
CAS
slack
/sms

38. Thank you!

39. We are hiring!
● Fresh web engineer
● Senior web engineer
● Internship