LJC: Microservices in the real world

1. @chbatey
Microservices in the real world
Christopher Batey
Freelancer / The Last Pickle
@chbatey

2. @chbatey
S!*t happens when you put a
network between your classes
Christopher Batey
Freelancer / The Last Pickle
@chbatey

3. @chbatey
Who am I?
• Freelance consultant: Docker, Kubernetes,
Cassandra, JVM
• Work on a OS testing library for Cassandra
• Twitter: @chbatey
Heavily recruiting!!

4. @chbatey
Background

5. @chbatey
Background

6. @chbatey
Background

7. @chbatey
Background

8. @chbatey
Background

9. @chbatey
But why?????

10. @chbatey
Small horizontal scalable services
• Move to small services independently deployed
- Login service
- Device service
- etc
• Move to a horizontally scalable Database that can run active
active in multiple data centres

11. @chbatey
Half way point

12. @chbatey
So... what do systems look like now?

13. @chbatey
Tech used
• All examples are on github
• Technologies used:
- Dropwizard
- Wiremock
- Hystrix
- Saboteur

14. @chbatey
Java microservice definition?
A microservice in Java == no
dependency injection
framework

15. @chbatey
Testing microservices
• You don’t know a service is fault tolerant if you don’t test faults

16. @chbatey
Isolated service tests
Prime
Movie service
Play Movie
Test
Real HTTP/TCP
Test

17. @chbatey
Fault tolerance
1.Don’t take forever - Timeouts
2.Don’t try if you can’t succeed
3.Don’t whack a dead horse
4.Turn broken stuff off

18. @chbatey
1 - Don’t take forever
• If at first you don’t succeed, don’t take forever to tell
someone
• Timeout and fail fast

19. @chbatey
Which timeouts?
• Socket connection timeout
• Socket read timeout

20. @chbatey
Your service hung for 30 seconds :(
Customer
You :(

21. @chbatey
Which timeouts?
• Socket connection timeout
• Socket read timeout
• Resource acquisition

22. @chbatey
Your service hung for 10 minutes :(

23. @chbatey
Let’s think about this

24. @chbatey
Adding an automated test
• Vagrant - launches + provisions localVMs
• Saboteur - uses tc, iptables to simulate network issues
• Wiremock - used to mock HTTP dependencies

25. @chbatey
I can write an automated test for that?
Wiremock:
•User Service
•Device Service
•Pin Service
S
a
b
o
t
e
u
r
Vagrant + Virtual box
Test meAcceptance
prime to drop traffic
reset

26. @chbatey
Dependency VM
Wiremock
Stubbed
Cassandra
192.168.2.2

27. @chbatey
Wiremock
192.168.2.2
App under test
localhost

28. @chbatey
HITTING THE
SERVICE
PRIMING

29. @chbatey
DEMO TIME

30. @chbatey
A little more detail

31. @chbatey
Wiremock
192.168.2.2
App under test
localhost
192.168.2.1
Delay is here

32. @chbatey
Wiremock
192.168.2.2
App under test
localhost
192.168.2.1
Delay is here

33. @chbatey
Wiremock
192.168.2.2
App under test
localhost
192.168.2.1
Delay is here

34. @chbatey
Wiremock
192.168.2.2
App under test
localhost
192.168.2.1
Delay is here

35. @chbatey
Implementing reliable timeouts

36. @chbatey
Implementing reliable timeouts
• Protect the container thread!
• Homemade:Worker Queue + Thread pool (executor)

37. @chbatey
Implementing reliable timeouts
• Protect the container thread!
• Homemade:Worker Queue + Thread pool (executor)
• Hystrix
• Spring cloud Netflix

38. @chbatey
A simple Spring RestController
@RestController
public class Resource {
private static final Logger LOGGER = LoggerFactory.getLogger(Resource.class);
@Autowired
private ScaryDependency scaryDependency;
@RequestMapping("/scary")
public String callTheScaryDependency() {
LOGGER.info("Resource later: I wonder which thread I am on!");
return scaryDependency.getScaryString();
}
}

39. @chbatey
Scary dependency
@Component
public class ScaryDependency {
private static final Logger LOGGER = LoggerFactory.getLogger(ScaryDependency.class);
public String getScaryString() {
LOGGER.info("Scary Dependency: I wonder which thread I am on! Tomcats?”);
if (System.currentTimeMillis() % 2 == 0) {
return "Scary String";
} else {
Thread.sleep(5000)
return “Slow Scary String";
}
}
}

40. @chbatey
All on the tomcat thread
13:47:20.200 [http-8080-exec-1] INFO info.batey.examples.Resource -
Resource later: I wonder which thread I am on!
13:47:20.200 [http-8080-exec-1] INFO
info.batey.examples.ScaryDependency - Scary Dependency: I wonder which
thread I am on! Tomcats?

41. @chbatey
Scary dependency
@Component
public class ScaryDependency {
private static final Logger LOGGER = LoggerFactory.getLogger(ScaryDependency.class);
@HystrixCommand()
public String getScaryString() {
LOGGER.info("Scary Dependency: I wonder which thread I am on! Tomcats?”);
if (System.currentTimeMillis() % 2 == 0) {
return "Scary String";
} else {
Thread.sleep(5000)
return “Slow Scary String";
}
}
}

42. @chbatey
What an annotation can do...
13:51:21.513 [http-8080-exec-1] INFO info.batey.examples.Resource
- Resource later: I wonder which thread I am on!
13:51:21.614 [hystrix-ScaryDependency-1] INFO
info.batey.examples.ScaryDependency - Scary Dependency: I wonder
which thread I am on! Tomcats? :P

43. @chbatey
Async libraries are your friend
• DataStax Java Driver
- Guava ListenableFuture

44. @chbatey
Drive this via requirements
• Reliable timeout
• Throttling
• Monitoring of failures / successes

45. @chbatey
Timeouts take home
• You can’t use network level timeouts for SLAs
• Test your SLAs - if someone says you can’t, hit them with a
stick
• Scary things happen without network issues

46. @chbatey
Fault tolerance
1.Don’t take forever - Timeouts
2.Don’t try if you can’t succeed
3.Don’t whack a dead horse
4.Turn broken stuff off

47. @chbatey
2 - Don’t try if you can’t succeed

48. @chbatey
Complexity
“When an application grows
in complexity it will
eventually start sending
emails”

49. @chbatey
Complexity
“When an application grows
in complexity it will
eventually start using
queues and thread pools”

50. @chbatey
Don’t try if you can’t succeed

51. @chbatey
Don’t try if you can’t succeed
• Executor Unbounded queues :(
- newFixedThreadPool
- newSingleThreadExecutor
- newThreadCachedThreadPool
• Bound your queues and threads
• Fail quickly when the queue / maxPoolSize is met
• Know your drivers

52. @chbatey
This is a functional requirement
• Set the timeout very high
• Use Wiremock to add a large delay to the requests
• Set queue size and thread pool size to 1
• Send in 2 requests to use the thread and fill the queue
• What happens on the 3rd request?

53. @chbatey
Fault tolerance
1.Don’t take forever - Timeouts
2.Don’t try if you can’t succeed
3.Don’t whack a dead horse
4.Turn broken stuff off

54. @chbatey
5 - Don’t whack a dead horse
Movie Player
User
Service
Device
Service
Pin
Service

55. @chbatey
What to do…
• Yes this will happen…
• Mandatory dependency - fail *really* fast
• Throttling
• Fallbacks

56. @chbatey
Circuit breaker pattern
(Picture from Akka docs)

57. @chbatey
Fault tolerance
1.Don’t take forever - Timeouts
2.Don’t try if you can’t succeed
3.Don’t whack a dead horse
4.Turn broken stuff off

58. @chbatey
6 - Turn off broken stuff
• The kill switch

59. @chbatey
To recap
1.Don’t take forever - Timeouts
2.Don’t try if you can’t succeed
3.Don’t whack a dead horse
4.Turn broken stuff off

60. @chbatey
Links
• Examples:
- https://github.com/chbatey/spring-cloud-example
- https://github.com/chbatey/dropwizard-hystrix
- https://github.com/chbatey/vagrant-wiremock-saboteur
• Tech:
- https://github.com/Netflix/Hystrix
- https://www.vagrantup.com/
- http://wiremock.org/
- https://github.com/tomakehurst/saboteur

61. @chbatey
Questions?
Thanks for listening!
Questions later? @chbatey
christopher.batey@gmail.com
http://christopher-batey.blogspot.co.uk/