Stability patterns devoxx_pl_2017

@DanLebrero#DevoxxPL
Platinum Sponsor:
Applying Stability Patterns:
a case study
Dan Lebrero
IG

* 17 years JVM
* Monolithic webapps, embedded systems,
low latency, micro-services, big data
* Technical Architect
* World leader CFD and Spread bet
* FTSE 250
* London, Krakow and Bangalore

Stability patterns
• Timeouts
• Circuit Breakers
• Bulkheads
• Steady State
• Fail Fast
• Handshaking
• Test Harness
• Decoupling Middleware

FAILURE IS
INEVITABLE

Stability patterns

Market Analysis (MA)

@DanLebrero#DevoxxPL @DanLebrero#DevoxxPL

Market Analysis Dependencies

• Single Sign On

• Single Sign On
• User Preferences

• Single Sign On
• CMS

• Single Sign On
• CMS
• Market Taxonomy

• Single Sign On
• CMS
• Market Taxonomy
• Client Information

• Single Sign On
• CMS
• Market Taxonomy
• Dealing API

• Single Sign On
• CMS
• Market Taxonomy
• Dealing API
• View Permissions + Market Pricing
• Chart Patterns
• News
• Client Sentiment
• Company Fundamentals
• Company Calendar

• Single Sign On
• CMS
• Market Taxonomy
• Dealing API
• Chart Patterns
• News
Other services: Search, Correlations, Economic
Calendar, Videos, Market Screener

“Integration points are the number-
one killer of systems” – Release It!,
Michael Nygard

Key architecture principles:
• All data on load (SEO/UX)

Key architecture principles:
• All data on load (SEO/UX)
• No service is essential

• Single Sign On
• CMS
• Market Taxonomy
• Dealing API
• Chart Patterns
• News

F5LB
F5LB
F5LB
F5Charts
F5News
F5Prices
F5MA
Market Analysis architecture
F5LB
F53rd Party
F53rd Party

Charts +
PermsCS + Perms + Prices
Fundamentals
Market Analysis request / response
News
response
collect
request
parallelise

TIMEOUTS
Never wait forever

FAILURE IS
INEVITABLE
(I)

FAILURE IS
INEVITABLE
(II)

Timeouts (I) - Parallelised work
collect
parallelise

parallelise
Timed wait
collect

Cancelunfinished
andcollect
parallelise
Timed wait

Timeouts (II) - Apache HTTP Client
Connection
pool timeout
Connection
timeout
Socket
timeout

Connection
pool timeout
Connection
timeout
Socket
timeout
(1
sec.)
(1
sec.)
(1
sec.)

protected void doGet(HttpServletRequest req, HttpServletResp
byte[] response = "abcdefghijklmnopqrstuvwxyz".getBytes("UTF-8");
for (byte b : response) {
resp.getOutputStream().print(b);
resp.getOutputStream().flush();
Thread.sleep(500);
}
}

Connection
timeout
Socket
timeout
(1
sec.)
(1
sec.)
(1
sec.)
Socket
timeout
Socket
timeout
(1
sec.)

Timeouts (III) - AsyncHttpClient
Connection
timeout
Socket
timeout
Connection
pool timeoutX

Connection
timeout
Socket
timeout
(1
sec.)
(1
sec.)

Connection
timeout
Socket
timeout
Socket
timeout
Socket
timeout
(1
sec.)
(1
sec.)
(1
sec.)
(1
sec.)

Connection
timeout
Socket
timeout
Socket
timeout
Socket
timeout
Request
timeout

Connection
timeout
Socket
timeout
Socket
timeout
Socket
timeout
Request
timeout
(3
sec.)
(1
sec.)
(1
sec.)
(1
sec.)
(1
sec.)

Timeouts (IV) - Very famous commercial DB
Connection
pool timeout
Connection
timeout
Socket
timeout

Timeouts (V) - Locks and queues
new Object().wait(10);
new ReentrantLock().tryLock(10, MILLISECONDS);
new Semaphore(1).tryAcquire(10, MILLISECONDS);
new CountDownLatch(1).await(10, MILLISECONDS);
new LinkedBlockingQueue(1).offer(o, 10, MILLISECONDS);
new LinkedBlockingQueue(1).poll(10, MILLISECONDS);

CIRCUIT BREAKERS
Don't do it if it hurts

Circuit Breakers
CLOSED
(pass though)

Circuit Breakers
CLOSED
(pass though)
OPEN
(don’t bother)
Too many
errors

Circuit Breakers
CLOSED
(pass though)
OPEN
(don’t bother)
HALF-OPEN
(give it a try)
Too many
errors
Every X
minutes

Circuit Breakers
• Around all HTTP calls.
• 1 circuit breaker per service: too coarse-grained or
services not micro enough.
• Raise alerts when circuit is open for more than 15 mins.
• Netflix Hystrix

BULKHEADS
Contain damage

Bulkheads - Mandatory image of a ship

Bulkheads (I) - In the small
Market Analysis
HTTP Client HTTP Client HTTP Client HTTP Client
News
Fundamentals
Charts
Prices

Bulkheads (II) - In the large
Live DC Demo DC

Bulkheads (III) - In the medium
Linux
Tomcat
app1
Tomcat
app1
Tomcat
app1
app2
app3
app2 app2
app3app3

Bulkheads (III) - In the medium
Linux
Tomcat
app
Tomcat
app
Tomcat
app

STEADY STATE
Nothing is infinite

Steady State
• Log rotation
• Cache size limit
• DB/HTTP connection pools
• Data archiving and purging
• Pagination in APIs

Steady State (I) - Thread pools
Thread Pool

Thread Pool
Work Queue
TaskTaskTaskTask

Thread Pool
Work Queue
TaskTaskTaskTask
Rejection Policy

Rejection Policy
• Abort
• Caller Runs
• Discard
• Discard Oldest

FAIL FAST
Don't waste your time

Fail Fast
• Check SLA before starting
• Check circuit breakers needed
• Reserve resources
• Validate user input

Fail Fast
• Check SLA before starting
• Check circuit breakers needed
• Reserve resources
• Validate user input
X
X
X

HANDSHAKING
Agree before doing

Handshaking
• DB connections pool (“select 1 from dual”)
• Heartbeats: Apache HTTPD -> Tomcat AJP

Market
Analysis
Varnish
Handshaking (I) - Varnish
Check

Client
Market
Analysis
Varnish
MA
Ready?
Check

Client
Market
Analysis
Varnish
MA
Ready?
Check
Yes

Client
Market
Analysis
Varnish
MA
Ready?
Check
Yes
No
Return cached

TEST HARNESS
Be evil

Test Harness
• Accept connection but never send data
• Refuse all connections
• Read request at 1 byte / second
• Write response at 1 byte / second
• Response is garbage
• Response is huge/infinite
• TCP packet loss

Test Harness (I) - Maven Cargo Plugin
Maven
Tomcat
Market
Analysis

Maven
Tomcat
Market
Analysis
Test
Jetty
Evil Servlet

Maven
Tomcat
Market
Analysis
Test
Jetty
Evil Servlet
@Test
HTTP

protected void doGet(HttpServletRequest req, HttpServletResp
byte[] response = "abcdefghijklmnopqrstuvwxyz".getBytes("UTF-8");
for (byte b : response) {
resp.getOutputStream().print(b);
resp.getOutputStream().flush();
Thread.sleep(500);
}
}

Test Harness
• TCP packet loss
X
X
X

Docker
Test Harness (II) - Docker
Maven
Test

Docker
Fake
dependency
container
ToxiProxy
container
Market
Analysis
container
Maven
Test
1. Start

Docker
Fake
dependency
container
ToxiProxy
container
Market
Analysis
container
Maven
Test
1. Start
2. Config fault

Docker
Fake
dependency
container
ToxiProxy
container
Market
Analysis
container
Maven
Test
1. Start
2. Config fault
3. @Test

• TCP packet loss
• Process crashing
• Slow disks
• Contented CPU
Test Harness (II) - Docker - Arquillian Cube Q

DECOUPLING
MIDDLEWARE
Avoid waiting

News 3rd
Party
MA
Decoupling Middleware (I) - Read through cache

News 3rd
Party
Cache
with
TTL
MA
Decoupling Middleware (I) - Read through cache

3rd
Party
NewsMA
Decoupling Middleware (II) - Background fetch
Local
Storage

DB
CMSMA
Decoupling Middleware (II) - Background fetch
Reference Data
Local Copy
MT

3rd
Party
ChartsMA
Decoupling Middleware (III) - Come back latter
Cache

3rd
Party
ChartsMA
Cache
COME BACK!

3rd
Party
ChartsMA
Cache
DATA
COME BACK!

Searc
h
Market Analysis
Decoupling Middleware (IV) - Async IO
Tomcat Pool

Searc
h
Market Analysis
Decoupling Middleware (IV) - Async IO
Tomcat Pool
AsyncHttp Pool

Linux
Tomcat
MA
Tomcat
News
Tomcat
xxx

F5LB
F5LB
F5LB
F5Charts
F5News
F5Prices
F5MAF5LB
F53rd Party
F53rd Party

MA LB
CS
CS
CS

MA LB
CS
CS
CS
A
B
A
B

References
https://pragprog.com/book/mnee2/release-it-second-editionRelease It!
by Michael T. Nygard
https://github.com/Netflix/Hystrix
http://wiremock.org
https://github.com/arquillian/arquillian-cube-q
https://m.youtube.com/watch?v=vt5p1qflM3U
Hystrix
WireMock
Arquillian
Cube Q
Automating resilience testing with Docker and Property Based Tests
by Daniel Lebrero
https://hc.apache.org/httpcomponents-client-4.5.x/tutorial/html/advanced.html
FutureRequestExecutionService
https://github.com/AsyncHttpClient/async-http-clientAsyncHttpClient

FAILURE IS
INEVITABLE
Summary

Summary
Timeouts - Never wait for ever
Circuit breakers - Don’t do it if it hurts
Bulkheads - Contain damage
Steady state - Nothing is infinite
Fail fast - Don’t waste your time
Handshake - Agree before doing
Decoupling middleware - Avoid waiting
Test harness - Be evil

@DanLebrero
dlebrero@gmail.com
danlebrero.com
@IGLabs @LifeAtIG
labs.ig.com/blog
www.iggroup.com/careers/teams/technology
???

Stability patterns devoxx_pl_2017

Recommended

Recommended

More Related Content

Similar to Stability patterns devoxx_pl_2017

Similar to Stability patterns devoxx_pl_2017 (20)

Recently uploaded

Recently uploaded (20)

Stability patterns devoxx_pl_2017