Devnexus 2017 As we architect our systems for greater demands, scale, uptime, and performance, the hardest thing to control becomes the environment in which we deploy and the subtle but crucial interactions between complicated systems. And microservices obviously are the way to go forward with those complicated systems. But what makes it so hard to build them? And why should you embrace failure instead of doing what we can do best: Preventing failure. This talk introduces you to the problem domain of a distributed system which consists of a couple of microservices. It shows how to build, deploy and orchestrate the chaos and introduces you to a couple of patterns to prevent and compensate failure.

1. Architecting for Failure
Why are distributed systems so hard?
Markus Eisele

2. @myfear

3. Evolution

4. Extreme Uptime
(99.999)
Vertical Scaling
Custom
Hardware
Hardware High
Availability
Centralized
Designed for
availability (99.9)
Commodity
Hardware
Replicated
Designed for
failure (99.999)
Horizontal
Scaling
Virtualized /
Cloud
Software High
Availability
Distributed
Centralized Shared Self Service
“Big Iron” “Enterprise” “Cloud”

5. 60s 80s 90s 2000 2014 2016 2020 2030
NumberofEnterpriseProjects
Mainframe Enterprise Cloud
Distribution of Projects over time.
Disclaimer:
My personal prediction!

6. Today’s biggest problem?

7. High Infrastructure
Cost
11%
Awful Downtime
9%
Meeting Demand
21%
Release Frquency
20%
Developer Velocity
39%

8. Meeting demands.
http://www.internetlivestats.com/internet-users/
J2EE
Spring
RoR
Akka
Reactive Manifesto
Microservices

9. What the hell is “Developer Velocity“ anyway?

10. Release frequency!!
bit.ly/helloworldmsa

11. And this is why we have
Microservices..

12. Scale
Deploy
Develop
Independently

14. REQ: Building and Scaling Microservices
• Lightweight runtime
• Cross – Service Security
• Transaction Management
• Service Scaling
• Load Balancing
• SLA’s
• Flexible Deployment
• Configuration
• Service Discovery
• Service Versions
• Monitoring
• Governance
• Asynchronous communication
• Non-blocking I/O
• Streaming Data
• Polyglot Services
• Modularity (Service definition)
• High performance persistence (CQRS)
• Event handling / messaging (ES)
• Eventual consistency
• API Management
• Health check and recovery

15. If the components do not compose
cleanly, then all you are doing is
shifting complexity from inside a
component to the connections
between components. Not just does
this just move complexity around, it
moves it to a place that's less explicit
and harder to control.
Martin Fowler
https://martinfowler.com/articles/microservices.html
“

16. How do we handle “failures” in
centralized or shared infrastructures?

18. Why did Application Server become a thing?
• Network and Threading
• Two Phase Commit (2PC)
• Shared resources
• Manageability
• Clustering supports scalability,
performance, and availability.
• Programing models
• Standardization
https://antoniogoncalves.org/2013/07/03/monster-component-in-java-ee-7/

19. Checked vs. Unchecked Exceptions
If a client can reasonably be expected to recover
from an exception, make it a checked exception.
If a client cannot do anything to recover from the
exception, make it an unchecked exception.
“
https://docs.oracle.com/javase/tutorial/essential/exceptions/runtime.html

20. It wasn’t easy – but manageable.
https://docs.oracle.com/javase/tutorial/essential/exceptions/runtime.html
• MVC handles checked
• Global exception handlers handle unchecked
• Centralized log files

22. 'If it ain't broke, don't fix it!'
Bert Lance 1977.
“

23. What is different for Microservices?

24. Microservices are Distributed Systems.

27. • Reactive Microservices Framework for the JVM
• Focused on right sized services
• Asynchronous I/O and communication as first class
priorities
• Highly productive development environment
• Takes you all the way to production
• https://github.com/lagom/online-auction-java
What is Lagom?

28. Protect Yourself
with Circuit
Breakers

29. Circuit Breakers

30. Circuit Breakers

31. Circuit Breakers

32. Circuit Breakers

33. Circuit Breakers
default Descriptor descriptor() {
return named("item").withCalls(
pathCall("/api/item", this::createItem),
restCall(Method.POST, "/api/item/:id/start", this::startAuction),
pathCall("/api/item/:id", this::getItem),
restCall(Method.PUT, "/api/item/:id", this::updateItem),
pathCall("/api/item?userId&status", this::getItemsForUser)
)
.withCircuitBreaker(CircuitBreaker.identifiedBy("item"))

34. Degraded beats
Unavailable

35. Degraded > Unavailable
Search
Bid
Item

36. Degraded > Unavailable
Search
Bid
Item

37. CompletionStage<PSequence<Bid>> bidHistoryFuture =
bidService.getBids(itemUuid)
.invoke().exceptionally(error -> {
log.warn("Bidding service failed to load", error);
return TreePVector.empty()
});
https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/CompletionStage.html#exceptionally-java.util.function.Function-

38. Bulkheading
(Kind of Important)

40. Duplication
isn’t a bad
thing

41. Degraded > Unavailable
Search
Bid
Item

42. Publish/Subscribe
Topic<BidEvent> bidEvents();
default Descriptor descriptor() {
return named("bidding").withCalls(
pathCall("/api/item/:id/bids", this::placeBid),
pathCall("/api/item/:id/bids", this::getBids)
).publishing(
topic("bidding-BidEvent", this::bidEvents)
)

43. Publish/Subscribe
Topic<BidEvent> bidEventTopic = biddingService.bidEvents();
bidEventTopic.subscribe()
.atLeastOnce(Flow.<BidEvent>create().map(this::toDocument)
.mapAsync(1, indexedStore::store));

44. Always have a plan B.

45. •Fallback pattern (cache instead of dB)
•The cost of resilience should be accuracy or
latency.
•CAP Theorem: Your choice: sacrifice availability
or consistency. You can't have all three.
What you can do..
https://codahale.com/you-cant-sacrifice-partition-tolerance/

46. Do you remember?

47. 8 fallacies of distributed computing
1. The network is reliable
2. Latency is zero
3. Bandwidth is infinite
4. The network is secure
5. Topology doesn't change
6. There is one administrator
7. Transport cost is zero
8. The network is homogeneous

48. Lessons learned.

49. Some things to remember.
• Distributed systems are different because they fail often.
• Writing robust distributed systems costs more than writing
robust single-machine systems.
• Robust, open source distributed systems are much less common
than robust, single-machine systems.
• Coordination is very hard.
• “It’s slow” is the hardest problem you’ll ever debug.
• Find ways to be partially available.
https://www.somethingsimilar.com/2013/01/14/notes-on-distributed-systems-for-young-bloods/

50. Where do we go from here?

51. http://www.ofbizian.com/2016/07/from-fragile-to-antifragile-software.html

53. Next Steps! Download and try Lagom!
Project Site:
http://www.lightbend.com/lagom
GitHub Repo:
https://github.com/lagom
Documentation:
http://www.lagomframework.com/documentation/1.3.x/java/Home.html
Example:
https://github.com/lagom/online-auction-java

54. Written for architects and developers that must
quickly gain a fundamental understanding of
microservice-based architectures, this free O’Reilly
report explores the journey from SOA to
microservices, discusses approaches to dismantling
your monolith, and reviews the key tenets of a
Reactive microservice:
• Isolate all the Things
• Act Autonomously
• Do One Thing, and Do It Well
• Own Your State, Exclusively
• Embrace Asynchronous Message-Passing
• Stay Mobile, but Addressable
• Collaborate as Systems to Solve Problems
http://bit.ly/ReactiveMicroservice

55. The detailed example in this report is based on
Lagom, a new framework that helps you follow the
requirements for building distributed, reactive
systems.
• Get an overview of the Reactive Programming
model and basic requirements for developing
reactive microservices
• Learn how to create base services, expose
endpoints, and then connect them with a
simple, web-based user interface
• Understand how to deal with persistence, state,
and clients
• Use integration technologies to start a
successful migration away from legacy systems
http://bit.ly/DevelopReactiveMicroservice