You can find the first part of this presentation here: https://www.slideshare.net/secret/pAvK8Qd9f07oa This presentation takes a deep dive into how the Million Song Library, a microservices-based application, was built using the Netflix Stack, Cassandra and Datastax. To learn more about Million Song Library and its components visit the project on GitHub: https://github.com/kenzanlabs/million-song-library Lea

1. Architecting for
Microservices Part 2
Denver Code Club Meetup Bill Schwanitz, Technical Architect
December 15, 2016 Craig Martin, Vice President of Engineering

2. Agenda
• Microservices (isn’t that what we are here
for?)
• What is a microservice?
• NETFLIX OSS
• Zuul
• Eureka
• Hystrix
• Archaius
• Million Song Library code walkthrough
• Build Million Song Library

3. An overview of Microservices
• N number of modular components joined
together via the network.
• Decomposed by units of business or functionality
• Combined makes up an entire backend system
• Single Responsibility Principle (SRP)
• Code with a singleness of purpose
• Interface Segregation Principle (ISP)
• No client should be forced to depend on
methods it does not use
• Service grouped by business domain area or
capability
• Many standalone “vertical” stack focused on a
single domain

5. • Netflix OSS
• “Gatekeeper”
Routing
Monitoring
Security
• Makes endpoints available
• Main use cases
Auth-N (not Auth-z)
DDOS
Transformation
• Groovy and Java
Filters in Groovy
Zuul
Challenges
• Performance
• Forgetting to open endpoints
• Logging

6. • “Pre” filter - First line of defence
Check URL (Sitemap)
Auth-N
Structure
Data Transformation
• “routing” filters
Endpoint Mapping
Service Discovery
• “post” filters
Data transformation
logging
• “static” filters
Healthcheck
References:
MSL server/msl-zuul:
github.com/kenzanlabs/msl-
zuul/tree/master/src/main/groovy/filters
Zuul - How does it
work?
Lessons Learned
• Keep it lightweight
• Can be used for
soaking
Other Options
• NginX
• ELBs
• API Gateway

7. • Netflix OSS Service Registry
• Used for “discovery”
• Heartbeat of the application
• Why not use ELBs?
Fewer IPs
Compromise Security
Groups
Stateless
• Replicated across regions
• Load balancing
• “Push” heartbeat monitoring
Eureka
Challenges
• Only works with
AWS
• Performance
• Scalability
Other Options
• Containerization
(Kubernetes)
• Consul
• ELBs
What is it?

8. • Eureka Client
Registers with Eureka Server
Part of the bootstrap process
Routinely updates
• Eureka Server
Maintains mapping of VIPs to IPs
Redundancy
Round Robin Load Balancing
Netflix OSS Service Registry
• Typically abstract discovery via
Ribbon Client
Best Practices
• Discovery shouldn’t be a dev
responsibility
• Single point of failure
• Scale “correctly”
Eureka
How it works.

9. Hystrix - What is it?
Controls the interactions between
distributed services, by:
• Adding latency tolerance logic
• Adding fault tolerance logic
• Isolating points of access between the
services
• Stopping cascading failures across
them
• Providing fallback options
Hystrix - Why?
Compared to Monolithic applications,
Microservices strongly rely upon networks.
So failure detection and manipulation logic
is essential.
Normal function (Closed)
When a system is functioning smoothly
Failure state (Open)
At this juncture, every call to the dependency is short-circuited with a
HystrixRuntimeException, giving clear indication of its cause.
Half-open state
Once the sleep Interval passes, Hystrix checks system availability, letting other
requests fail-fast until the response is obtained. If the call is successful, go to
Closed; in case of failure, go back to Open

10. Hystrix - How?
● Wrapping all external calls in a HystrixCommand or HystrixObservableCommand
● Timing-out calls when needed.
● Maintaining a small thread-pool for each dependency (monitor load)
● Measuring successes, failures , timeouts, and thread rejections.
● Tripping a circuit-breaker to stop all requests to a particular service for a period of time
● Performing fallback logic when a request fails, is rejected, times-out, or short-circuits.
● Monitoring metrics and configuration changes in near real-time.
For Additional information:
https://github.com/Netflix/Hystrix/wiki/How-it-Works

11. Archaius
Java library that provides APIs
to access and utilize properties
that can change dynamically at
runtime. It includes the following
features:
● Dynamic, Typed Properties
● High throughput and Thread Safe
Configuration operations
● A polling framework that allows users to
obtain property changes to a
Configuration Source
● Allows retrieval of properties from local
properties files or a properties server
○ Darchaius.configurationSource.additi
onalUrls=file:///apps/myapp/applicatio
n.properties
○ Darchaius.configurationSource.additi
onalUrls=http://myserver/properties
● Automatically updates to all servers

12. Million Song
Library

13. • Java 8
• Netflix OSS (Eureka, Hystrix, Zuul,
Archaius)
• Datastax (Cassandra)
• Swagger
Backend Tech
Platform Tech
• Maven
• Docker
• Spinnaker
Documentation Tech
• Swagger
• Ascii Docs

14. Time to Look at Some Code
Netflix OSS
• Zuul
• Eureka
• Hystrix
• Archaius

15. Zuul - Example of Zuul
Configuration -
pom.xml
• In MSL, we configure Zuul via
the pom.xml and Archaius
properties
• So here you see the
pom.xml:
Zuul runs in jetty
We can supply additional
configuration via
Archaius
Zuul will listen on port
9000
Example:
<plugin>
<groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-maven-plugin</artifactId>
<version>${jetty-version}</version>
<configuration>
<systemProperties>
<systemProperty>
<name>archaius.deployment.applicationId</name>
<value>zuul</value>
</systemProperty>
</systemProperties>
.
.
.
<httpConnector>
<port>9000</port>
</httpConnector>
</configuration>
</plugin>
References:
github.com/kenzanlabs/msl-zuul

16. Zuul - Example of Zuul
Configuration - properties
• In MSL, we configure Zuul via
the pom.xml and Archaius
properties:
Locations of the pre, routing
and post filters
Allowable and default
clients
Configuration for one of
those clients
• And an example of a URL
Notice it goes to port 9000
Has the client name
loginedge
Then the rest of the URL is
the URI to the client
• So Zuul redirects this request to
login-edge:9001
Properties:
zuul.filter.pre.path=src/main/groovy/filters/pre
zuul.filter.routing.path=src/main/groovy/filters/route
zuul.filter.post.path=src/main/groovy/filters/post
zuul.niws.clientlist=loginedge|accountedge|catalogedge|ratingsedge
loginedge.zuul.client.DeploymentContextBasedVipAddresses=msl.log
in.edge
loginedge.zuul.client.Port=9001
Example URL:
https://msl.kenzanlabs.com:9000/loginedge/login
References:
github.com/kenzanlabs/msl-zuul/tree/master/src/main/resources

17. Eureka Example
• Registration
Tell Eureka Server that a
new instance of a
microservice has
started
Needs eureka properties
Needs Karyon to be
instantiated
Handles routine
heartbeat from client
to server
• Healthcheck
Allows Eureka Server to
check if instance is still
up and running
Eureka Server will
remove any down
instance
Properties:
eureka.name=loginedge
eureka.vipAddress=msl.login.edge
eureka.port=9001
eureka.serviceUrl.default=http://localhost:8080/eureka/v2/
eureka.region=default
eureka.preferSameZone=true
eureka.registration.enabled=true
Karyon Startup:
KaryonServer server = new KaryonServer();
server.start();
Healthcheck ReST Endpoint:
@Path("/healthcheck")
public class HealthCheckResource {
@GET
@Produces(MediaType.APPLICATION_JSON)
public Response healthcheck() {
return Response.status(Response.Status.OK).build();
}
}

18. Hystrix example
Setup
• Create a subclass of
HystrixCommand
• The command group key
groups commands for
configuration, thread pooling,
etc.
• Override the run() method
• Optionally override the
getFallback() method
Calling
• Instantiate your command
class
• Call its execute() method
Code:
class ValidateAccountCommand extends HystrixCommand<Boolean>
{
private Account account;
protected ValidateAccountCommand(Account account) {
super(Setter.withGroupKey(HystrixCommandGroupKey.Factory
.asKey("validateAccount")));
this.account = account;
}
@Override
protected Boolean run() throws Exception {
return AccountValidator.isValid(account);
}
@Override
protected Boolean getFallback() {
return false;
}
}
public Boolean save(Account account) throws Exception {
return (new ValidateAccountCommand(account).execute());
}

19. Archaius
Example• Configuration
Declare the properties
source(s)
Can be a local file or a
properties server
• Use
Properties are typed
Boolean
Int
Long
String
etc.
Only need to declare the
DynamicXProperty
once
Then just use x.get()
whenever you need to
use the value
Calling x.get() once and
storing the result is an
anti-pattern - doesn’t
allow for dynamic
reconfiguration
Properties:
archaius.configurationSource.additionalUrls=
file://msl-login-edge-config/edge-config.properties
or
archaius.configurationSource.additionalUrls=http://myserver/properties
Code:
static final DynamicLongProperty timeToWait = DynamicPropertyFactory
.getInstance().getLongProperty("msl.sleep", 100);
void foo() {
long t = timeToWait.get();
sleep(t)
}
Anti-pattern:
static final long t = DynamicPropertyFactory
.getInstance().getLongProperty("msl.sleep", 100).get();
void foo() {
sleep(t)
}

20. Why?
• The API is described in one place, a
yaml formatted file (can also be
described using annotations in an
existing Java file)
• From the yaml file, swagger tools
generate client and server code stubs,
and documentation
• Since code and documentation are
created from the same yaml file, the
code and the documentation will never
diverge
Swagger - What?
A specification and associated tools for
describing, producing, consuming, and
visualizing a ReST API
YAML File:
.
.
.
/catalog-edge/browse/album:
x-swagger-router-controller: catalog_controller
get:
description: "Get browsing data for albums in the catalog"
tags:
- Catalog
operationId: browse_albums
parameters:
-
$ref: "#/parameters/PagingState"
-
$ref: "#/parameters/Items"
-
$ref: "#/parameters/Facets"
responses:
"200":
description: Success
schema:
$ref: "#/definitions/AlbumList"
"400":
description: “Invalid pagingState or facet”
.
.
.
Yaml Details
• First line defines the URI path to the
endpoint
• tags: grouping endpoints for docs
• x-swagger-router-controller: the name
of the Node controller file
• operationId: name of the function in
server stub
• parameters: optional path and query
parameters

21. Swagger - Generated Server Stub
CatalogEdgeApi.java
@Path("/catalog-edge")
@Consumes({ "application/json" })
@Produces({ "application/json" })
public class CatalogEdgeApi {
private final CatalogEdgeApiService delegate = CatalogEdgeApiServiceFactory.getCatalogEdgeApi();
@GET
@Path("/browse/album")
@Consumes({ "application/json" })
@Produces({ "application/json" })
public Response browseAlbums(
@QueryParam("items") Integer items,
@QueryParam("pagingState") String pagingState,
@QueryParam("facets") String facets)
) throws NotFoundException {
return delegate.browseAlbums(items,pagingState,facets);
}
...
}
CatalogEdgeApiService.java
public abstract class CatalogEdgeApiService {
public abstract Response browseAlbums(Integer items,String pagingState,String facets)
throws NotFoundException;
}

22. MSL Local Installation
Prerequisites
● git installed
● personal git account created
CD to where you want to install it
cd ~
Retrieve the main MSL repository from github
git clone https://github.com/kenzanmedia/million-song-library
CD into the new directory
cd million-song-library/bin
Be sure the setup script is executable
chmod +x setup.sh
Run the setup script
./setup.sh

23. Want to learn more?
Follow us!
@kenzanmedia
www.linkedin.com/company/kenzan-media
techblog.kenzan.com
www.facebook.com/kenzanmedia/