Advertisement
Microservices with Apache Camel, Docker and Fabric8 v2
Jun. 25, 2015•0 likes•8,395 views
19 likes
Be the first to like this
Show More
views
Total views
0
On Slideshare
0
From embeds
0
Number of embeds
0
Download to read offline
Report
Software
My talk from Red Hat Summit 2015 about the pros/cons of microservices, how integration is a strong requirement for doing distributed systems designs, and how open source projects like Apache Camel, Docker, Kubernetes, OpenShift and Fabric8 can help simplify and manage microservice environments
Christian PostaFollow
Global Field CTO at Solo.ioAdvertisement
Advertisement
Advertisement
Recommended
Chicago Microservices Integration TalkChristian Posta
Essential Camel ComponentsChristian Posta
Solving Enterprise Integration with Apache CamelChristian Posta
Camel oneactivemq posta-finalChristian Posta
Simplify integrations-final-pdfChristian Posta
Why real integration developers ride CamelsChristian Posta
Microservices with Apache Camel, Docker and Fabric8 v2
- #microservices with Apache Camel, Docker and Fabric8v2
- Christian Posta Principal Middleware Specialist/Architect Blog: http://blog.christianposta.com Twitter: @christianposta Email: christian@redhat.com • Committer on Apache Camel, Apache ActiveMQ, Fabric8 • Recovering consultant • Spent lot of time working with one of the largest Microservices, web- scale, unicorn companies • Frequent blogger and speaker about open-source, cloud, and microservices
- • Microservices: the good, the bad • Do you even need integration? • Real developers ride Camels • Microservices the Docker way with Fabric8 • Questions / Demo Agenda
- Microservices: the good, the bad
- • Agile • SOA principles • Domain Driven Design • Hexagonal Architectures • Pipes and Filters • Actor Model • SEDA Microservices not really new, per-se
- A way to organize teams that mimic the structure of an organization for greater autonomy and reduced cross-team synchronization for the purposes of scalability, faster solution delivery, and ability to manage complexity at the expense of known tradeoffs. So what is “Microservices”?
- • Faster software delivery • Potential for faster innovation • Scale easier • Right technology stack for the problem • Grok services easier • Test individual services easier • Individual deployments
- • System complexity • Operational complexity • Testing is harder across services • Security • It’s freakin hard • Resource overhead • Network overhead • Lack of tooling Drawbacks
- 1. Do you have a scaling problem? 2. You need to deliver software much faster? 3. Do you have organizational complexity? When do you do microservices:
- • Understanding of modularity, API design • Small, cross-functional teams (pizza box) • Self service access to infrastructure • Strong automation capabilities • Mature CI/CD practices Foundations of Microservices
- Do YOU want to do Microservices?
- Microservices Envy: http://www.thoughtworks.com/radar/techniques/microservice-envy You’re not going to do microservices http://christianposta.com/blog/not-gonna-happen
- Do you even need integration?
- Yes.
- 1 Enterprise Service Bus?
- Hexagonal architecture
- Microservices
- Real developers ride Camels
- Real developers ride Camels!
- Enterprise Integration Patterns http://camel.apache.org/eip
- Heavy Lifting: Camel for Microservices
- • Dynamic routing options • REST DSL • Backpressure mechanisms • Circuit Breaker Heavy Lifting: Camel for Microservices
- • “Smart endpoints, dumb pipes” • Endpoint does one thing well • Metadata used for further routing • Really “dynamic” with rules engine (eg, Drools/BRMS) Dynamic Routing
- REST DSL public class OrderProcessorRouteBuilder extends RouteBuilder { @Override public void configure() throws Exception { rest().post(“/order/socks”) .description(“New Order for pair of socks”) .consumes(“application/json”) .route() .to(“activemq:topic:newOrder”) .log(“received new order ${body.orderId}”) .to(“ibatis:storeOrder?statementType=Insert”); } }
- • Throttle EIP • http://camel.apache.org/throttler.html • Blocking SEDA Queue • from(“seda:name?size=100&blockWhenFull=true) • Configure jetty/netty to use blocking acceptor queues • https://wiki.eclipse.org/Jetty/Howto/High_Load • Using Exception handling/retry and DLQ logic when getting flow controlled • http://camel.apache.org/error-handling-in-camel.html Backpressure with Camel
- Circuit breaker public class OrderProcessorRouteBuilder extends RouteBuilder { @Override public void configure() throws Exception { from(“direct:someinterface”) .loadbalance() .circuitBreaker(3, 20000L, MyException.class) .to(“ibatis:storeOrder?statementType=Insert”); } }
- More Information
- Microservices the Docker way with Fabric8
- • Implemented with Zookeeper and git • Intended to be used with a dynamic JVM/Apache Karaf • Profiles store configuration, metadata, end- state deployments • Networking, JVM isolation, orchestration, auto-scaling, health checks, cloud deployments: all up to you Fabric8 V1.x
- • Implemented with Docker and Kubernetes • Use any JVM (or any technology) • Docker images, encourage immutable, well- defined, well-tested deployments • Provides networking, JVM isolation, orchestration, auto-scaling, health checks, cloud deployments Fabric8 V2
- Kubernetes Master RHEL Atomic Minion Minion Minion Minion Minion RHEL Atomic Minion Master Master etcd etcd etcd Jube Kubernetes REST API v1 Webconsole Pre-packaged Apps Developer tools Java Libraries
- • Docker for packaging • Docker/linux containers for process isolation • Kubernetes for container orchestration (start, stop, health check, replicas, service discovery) • Set of microservices/apps for • Management • Continuous Delivery • Integration (iPaaS) • Tools for Java Developers Fabric8 V2
- CI/CD apps API management iPaaS Manage/Monitor Some Hawt Consoles
- • Console • Docker Maven Plugin • Fabric8 Maven Plugin • Dependency Injection (of Kube services too!) • Spring • CDI • Java Libraries • Kubernetes fluent API • Arquillian plugins • Kubernetes Jolokia Fabric8 V2
- Questions? Blog: http://blog.christianposta.com Twitter: @christianposta Email: christian@redhat.com
Editor's Notes
- For many years now we have been finding better ways to build systems. We have been learning from what has come before, adopting new technology, and observing how a new wave of technology companies operate in different ways to create IT systems that help make both their customers and own developers happier. Eric Evans’ Domain Driven Design helped us understand the importance of representing the real world in our code, and showed us a path towards better ways to model our systems. Continuous Delivery showed how we can more effectively and efficiently get our software into production, instilling in us the idea that we should treat every check in as a release candidate. Our understanding of how the Web works has led us to develop better ways of having machines talk to machines. Alistar Cockburn’s Hexagonal Architecture guided us away from layered architectures where business logic could hide. Virtualisation platforms allowed us to provision and resize our machines at will, with Infrastructure Automation giving us a way to handle these machines at scale. Some large, successful organisations like Amazon and Google espoused the view of small teams owning the full life-cycle of their services. And more recently Netflix has shared with us ways of building anti-fragile systems at a scale that would be hard to comprehend just ten years before. Domain Driven Design. Continuous Delivery. On-demand virtualisation. Infrastructure automation. Small autonomous teams. Systems at Scale. Microservices have emerged from this world. They weren’t invented or described before the fact, they emerged as a trend, or a pattern, from real-world use. But they only exist because of all that has gone before. I will pull strands out of this prior work to help paint a picture of how to build, manage and evolve Microservices throughout the rest of this book. Many organisations have found that by embracing fine-grained, microservice architectures, they can deliver software faster and embrace newer technologies. Microservices give us significantly more freedom to allow us to react and make different decisions, allowing us to react faster to the inevitable change that impacts all of us. Service-Oriented Architecture (SOA) is a design approach where multiple services collaborate together to provide some end set of capabilities. A service here typically means a completely separate operating system process. Communication between these services is done via calls across a network rather than method calls within a process boundary. SOA emerged as an approach to combat the challenges of the large monolithic applications. It is an approach which aims to promote the re-usability of software - two or more end-user applications for example could both use the same services. It is an approach which aims to make it easier to maintain or rewrite software, as theoretically we can replace one service with another without anyone knowing, as long as the semantics of the service don’t change too much. SOA at its heart is a very sensible idea. However, despite many efforts to the contrary, there is a lack of good consensus as how to do SOA well. In my opinion what much of the industry has failed to do is look holistically enough at the problem and present a compelling alternative to the narrative set out by various vendors in this space. Many of the problems lain at the door of SOA are actually problems with things like communication protocols (SOAP), vendor middle-ware, a lack of guidance about service granularity, or guidance on picking the wrong places to split your system. We’ll tackle each of these in turn throughout the rest of the book. A cynic might suggests that vendors co-opted (and in some cases drove) the SOA movement in a way to sell more products, and those self-same products in the end undermined the goal of SOA. Much of the conventional wisdom around SOA doesn’t help you understand how to split something big into something small. It doesn’t talk about how big is too big. It doesn’t talk enough about real-world practical ways to ensure that services do not become overly coupled. The number of things that go unsaid is where many of the pitfalls associated with SOA come from. The microservice approach has emerged from real-world use, taking our better understanding of systems and architecture on how to do SOA well. So you should instead think of Microservices as a specific approach for SOA in the same way that XP or Scrum are specific approaches for Agile software development.
- Today’s silver bullet… “If we just do X, we’ll be doing it right” If we just use X, we’ll be doing microservices, and that’s what it’s all about!” no, dumbass, it’s all about delivering business value.. Quickly… Everyone wants tightly integrated business systems However, a business’s model is usually far too complex to understand as a single unit Gotta break things down and modularize and reason about things in smaller units Eric Evans: The goal of the most ambitious enterprise system is a tightly integrated system spanning the entire business. Yet the entire business model for almost any such organization is too large and complex to manage or even understand as a single unit. The system must be broken into smaller parts, in both concept and implementation. The challenge is to accomplish this modularity without losing the benefits of integration, allowing different parts of the system to interoperate to support the coordination of various business operations. A monolithic, all-encompassing domain model will be unwieldy and loaded with subtle duplications and contradictions. A set of small, distinct subsystems glued together with ad hoc interfaces will lack the power to solve enterprise-wide problems and allows consistency problems to arise at every integration point. The pitfalls of both extremes can be avoided with a systematic, evolving design strategy.
- For many years now we have been finding better ways to build systems. We have been learning from what has come before, adopting new technology, and observing how a new wave of technology companies operate in different ways to create IT systems that help make both their customers and own developers happier. Eric Evans’ Domain Driven Design helped us understand the importance of representing the real world in our code, and showed us a path towards better ways to model our systems. Continuous Delivery showed how we can more effectively and efficiently get our software into production, instilling in us the idea that we should treat every check in as a release candidate. Our understanding of how the Web works has led us to develop better ways of having machines talk to machines. Alistar Cockburn’s Hexagonal Architecture guided us away from layered architectures where business logic could hide. Virtualisation platforms allowed us to provision and resize our machines at will, with Infrastructure Automation giving us a way to handle these machines at scale. Some large, successful organisations like Amazon and Google espoused the view of small teams owning the full life-cycle of their services. And more recently Netflix has shared with us ways of building anti-fragile systems at a scale that would be hard to comprehend just ten years before. Domain Driven Design. Continuous Delivery. On-demand virtualisation. Infrastructure automation. Small autonomous teams. Systems at Scale. Microservices have emerged from this world. They weren’t invented or described before the fact, they emerged as a trend, or a pattern, from real-world use. But they only exist because of all that has gone before. I will pull strands out of this prior work to help paint a picture of how to build, manage and evolve Microservices throughout the rest of this book. Many organisations have found that by embracing fine-grained, microservice architectures, they can deliver software faster and embrace newer technologies. Microservices give us significantly more freedom to allow us to react and make different decisions, allowing us to react faster to the inevitable change that impacts all of us.
- Well, isn’t this something we’ve seen before?
- Intended for highly realiable, scalable distributed systems. Allow for planned/unplanned failures without compromising the entire system. You can know when a process dies and why it did (fabric?) You can force processes that depend on each other to die together if one of them goes wrong (co-locate systems in karaf) You can run a logger that automatically logs every uncaught exception for you, and even define your own Nodes can be monitored so you know when they went down (or got disconnected) You can restart failed processes (or groups of failed processes) Have whole applications restarting on different nodes if one fails And a lot more more stuff with the OTP framework
- For many years now we have been finding better ways to build systems. We have been learning from what has come before, adopting new technology, and observing how a new wave of technology companies operate in different ways to create IT systems that help make both their customers and own developers happier. Eric Evans’ Domain Driven Design helped us understand the importance of representing the real world in our code, and showed us a path towards better ways to model our systems. Continuous Delivery showed how we can more effectively and efficiently get our software into production, instilling in us the idea that we should treat every check in as a release candidate. Our understanding of how the Web works has led us to develop better ways of having machines talk to machines. Alistar Cockburn’s Hexagonal Architecture guided us away from layered architectures where business logic could hide. Virtualisation platforms allowed us to provision and resize our machines at will, with Infrastructure Automation giving us a way to handle these machines at scale. Some large, successful organisations like Amazon and Google espoused the view of small teams owning the full life-cycle of their services. And more recently Netflix has shared with us ways of building anti-fragile systems at a scale that would be hard to comprehend just ten years before. Domain Driven Design. Continuous Delivery. On-demand virtualisation. Infrastructure automation. Small autonomous teams. Systems at Scale. Microservices have emerged from this world. They weren’t invented or described before the fact, they emerged as a trend, or a pattern, from real-world use. But they only exist because of all that has gone before. I will pull strands out of this prior work to help paint a picture of how to build, manage and evolve Microservices throughout the rest of this book. Many organisations have found that by embracing fine-grained, microservice architectures, they can deliver software faster and embrace newer technologies. Microservices give us significantly more freedom to allow us to react and make different decisions, allowing us to react faster to the inevitable change that impacts all of us.
- Team communi – conways law…
- Rectangle! Gartner… CPU, memory controller, front side bus, disk IO,.. Etc… like that right?? Not really… still a rectangle box….
- http://alistair.cockburn.us/Hexagonal+architecture Intent Allow an application to equally be driven by users, programs, automated test or batch scripts, and to be developed and tested in isolation from its eventual run-time devices and databases. As events arrive from the outside world at a port, a technology-specific adapter converts it into a usable procedure call or message and passes it to the application. The application is blissfully ignorant of the nature of the input device. When the application has something to send out, it sends it out through a port to an adapter, which creates the appropriate signals needed by the receiving technology (human or automated). The application has a semantically sound interaction with the adapters on all sides of it, without actually knowing the nature of the things on the other side of the adapters. Motivation One of the great bugaboos of software applications over the years has been infiltration of business logic into the user interface code. The problem this causes is threefold: First, the system can’t neatly be tested with automated test suites because part of the logic needing to be tested is dependent on oft-changing visual details such as field size and button placement; For the exact same reason, it becomes impossible to shift from a human-driven use of the system to a batch-run system; For still the same reason, it becomes difficult or impossible to allow the program to be driven by another program when that becomes attractive. The attempted solution, repeated in many organizations, is to create a new layer in the architecture, with the promise that this time, really and truly, no business logic will be put into the new layer. However, having no mechanism to detect when a violation of that promise occurs, the organization finds a few years later that the new layer is cluttered with business logic and the old problem has reappeared. Imagine now that ‘’every’’ piece of functionality the application offers were available through an API (application programmed interface) or function call. In this situation, the test or QA department can run automated test scripts against the application to detect when any new coding breaks a previously working function. The business experts can create automated test cases, before the GUI details are finalized, that tells the programmers when they have done their work correctly (and these tests become the ones run by the test department). The application can be deployed in ‘’headless’’ mode, so only the API is available, and other programs can make use of its functionality — this simplifies the overall design of complex application suites and also permits business-to-business service applications to use each other without human intervention over the web. Finally, the automated function regression tests detect any violation of the promise to keep business logic out of the presentation layer. The organization can detect, and then correct, the logic leak.
- Team communi – conways law…
- Communication is paramount!
- http://martinfowler.com/bliki/CircuitBreaker.html
Advertisement