This document provides an overview and agenda for a presentation on Apache ActiveMQ 5.9.x and Apache Apollo. The presentation will cover new features in ActiveMQ 5.9.x including AMQP 1.0 support, REST management, a new default file-based store using LevelDB, and high availability replication of the store. It will also introduce Apache Apollo and allow for a question and discussion period.
Messaging is the backbone of many top enterprises. It affords reliable, asynchronous data passing to achieve loosely coupled, highly scalable distributed systems. As enterprises large and small become more interconnected, demand for remote and limited devices to be integrated with enterprise systems is surging. Come see how the most widely used, open-source messaging broker, Apache ActiveMQ, fits nicely and how it supports polyglot messaging.
Microservices with Apache Camel, Docker and Fabric8 v2Christian Posta
My talk from Red Hat Summit 2015 about the pros/cons of microservices, how integration is a strong requirement for doing distributed systems designs, and how open source projects like Apache Camel, Docker, Kubernetes, OpenShift and Fabric8 can help simplify and manage microservice environments
Messaging is the backbone of many top enterprises. It affords reliable, asynchronous data passing to achieve loosely coupled, highly scalable distributed systems. As enterprises large and small become more interconnected, demand for remote and limited devices to be integrated with enterprise systems is surging. Come see how the most widely used, open-source messaging broker, Apache ActiveMQ, fits nicely and how it supports polyglot messaging.
Microservices with Apache Camel, Docker and Fabric8 v2Christian Posta
My talk from Red Hat Summit 2015 about the pros/cons of microservices, how integration is a strong requirement for doing distributed systems designs, and how open source projects like Apache Camel, Docker, Kubernetes, OpenShift and Fabric8 can help simplify and manage microservice environments
Moving Gigantic Files Into and Out of the Alfresco RepositoryJeff Potts
This talk is a technical case study showing show Metaversant solved a problem for one of their clients, Noble Research Institute. Researchers at Noble deal with very large files which are often difficult to move into and out of the Alfresco repository.
The Proxy Wars - MySQL Router, ProxySQL, MariaDB MaxScaleColin Charles
As proxies (and database routers) go, the first one I ever used was the now deprecated MySQL Proxy. Since then, I've managed to use MariaDB MaxScale quite a bit (including its fork AirBnB MaxScale), played around with ProxySQL in recent time, and also started taking a look at MySQL Router. In this quick 20-minute overview, we'll discuss why these three exist, a feature comparison, and reasons when to use the right tool for the job.
Cloud Development with Camel and Amazon Web ServicesRobin Howlett
This presentation will demonstrate how to rapidly prototype and develop distributed, scalable applications with Apache Camel, its AWS Components and the AWS Java SDK.
Robin Howlett is Senior Architect at Silver Chalice, a Chicago White Sox affiliated start-up, based in Boulder, CO, with a portfolio of high-value digital-based businesses in the fields of sports, media and entertainment. In 2011, he built the Advanced Media Platform, a proprietary cloud-based platform that services millions of requests per day across dozens of mobile application products, heavily utilizing the Apache Camel framework.
Messaging For the Cloud and MicroservicesRob Davies
Utilising messaging in cloud deployments isn't straightforward, particularly if you want to take advantage of auto scaling. This talk covers the general problems of scaling for cloud deployments, and messaging for faster inter-service communication for Microservices
My talk at ScaleConf 2017 in Cape Town on some tips and tactics for scaling WordPress, with reference to WordPress.com and the container-based VIP Go platform.
Video of my talk is here: https://www.youtube.com/watch?v=cs0DcY80spw
Microservices architecture has many benefits. But it comes at a cost. Running microservices and monitoring what’s going on is tedious. That’s why MicroProfile adopts monitoring as a first-class concept. In this session, learn how MicroProfile runtimes collect metrics and how to seamlessly collect them with tools like Prometheus and Grafana. Learn how MicroProfile makes it easy to connect information about interrelated service calls, how to gather the information and analyze system bottlenecks, how to deploy and scale MicroProfile applications with Kubernetes and how to react to their health status to detect and automatically recover from failures.
Meet MariaDB 10.1 at the Bulgaria Web SummitColin Charles
Meet MariaDB 10.1 at the Bulgaria Web Summit, held in Sofia in February 2016. Learn all about MariaDB Server, and the new features like encryption, audit plugins, and more.
Moving Gigantic Files Into and Out of the Alfresco RepositoryJeff Potts
This talk is a technical case study showing show Metaversant solved a problem for one of their clients, Noble Research Institute. Researchers at Noble deal with very large files which are often difficult to move into and out of the Alfresco repository.
The Proxy Wars - MySQL Router, ProxySQL, MariaDB MaxScaleColin Charles
As proxies (and database routers) go, the first one I ever used was the now deprecated MySQL Proxy. Since then, I've managed to use MariaDB MaxScale quite a bit (including its fork AirBnB MaxScale), played around with ProxySQL in recent time, and also started taking a look at MySQL Router. In this quick 20-minute overview, we'll discuss why these three exist, a feature comparison, and reasons when to use the right tool for the job.
Cloud Development with Camel and Amazon Web ServicesRobin Howlett
This presentation will demonstrate how to rapidly prototype and develop distributed, scalable applications with Apache Camel, its AWS Components and the AWS Java SDK.
Robin Howlett is Senior Architect at Silver Chalice, a Chicago White Sox affiliated start-up, based in Boulder, CO, with a portfolio of high-value digital-based businesses in the fields of sports, media and entertainment. In 2011, he built the Advanced Media Platform, a proprietary cloud-based platform that services millions of requests per day across dozens of mobile application products, heavily utilizing the Apache Camel framework.
Messaging For the Cloud and MicroservicesRob Davies
Utilising messaging in cloud deployments isn't straightforward, particularly if you want to take advantage of auto scaling. This talk covers the general problems of scaling for cloud deployments, and messaging for faster inter-service communication for Microservices
My talk at ScaleConf 2017 in Cape Town on some tips and tactics for scaling WordPress, with reference to WordPress.com and the container-based VIP Go platform.
Video of my talk is here: https://www.youtube.com/watch?v=cs0DcY80spw
Microservices architecture has many benefits. But it comes at a cost. Running microservices and monitoring what’s going on is tedious. That’s why MicroProfile adopts monitoring as a first-class concept. In this session, learn how MicroProfile runtimes collect metrics and how to seamlessly collect them with tools like Prometheus and Grafana. Learn how MicroProfile makes it easy to connect information about interrelated service calls, how to gather the information and analyze system bottlenecks, how to deploy and scale MicroProfile applications with Kubernetes and how to react to their health status to detect and automatically recover from failures.
Meet MariaDB 10.1 at the Bulgaria Web SummitColin Charles
Meet MariaDB 10.1 at the Bulgaria Web Summit, held in Sofia in February 2016. Learn all about MariaDB Server, and the new features like encryption, audit plugins, and more.
Microservices architecture is a very powerful way to build scalable systems optimized for speed of change. To do this, we need to build independent, autonomous services which by definition tend to minimize dependencies on other systems. One of the tenants of microservices, and a way to minimize dependencies, is “a service should own its own database”. Unfortunately this is a lot easier said than done. Why? Because: your data.
We’ve been dealing with data in information systems for 5 decades so isn’t this a solved problem? Yes and no. A lot of the lessons learned are still very relevant. Traditionally, we application developers have accepted the practice of using relational databases and relying on all of their safety guarantees without question. But as we build services architectures that span more than one database (by design, as with microservices), things get harder. If data about a customer changes in one database, how do we reconcile that with other databases (especially where the data storage may be heterogenous?).
For developers focused on the traditional enterprise, not only do we have to try to build fast-changing systems that are surrounded by legacy systems, the domains (finance, insurance, retail, etc) are incredibly complicated. Just copying with Netflix does for microservices may or may not be useful. So how do we develop and reason about the boundaries in our system to reduce complexity in the domain?
In this talk, we’ll explore these problems and see how Domain Driven Design helps grapple with the domain complexity. We’ll see how DDD concepts like Entities and Aggregates help reason about boundaries based on use cases and how transactions are affected. Once we can identify our transactional boundaries we can more carefully adjust our needs from the CAP theorem to scale out and achieve truly autonomous systems with strictly ordered eventual consistency. We’ll see how technologies like Apache Kafka, Apache Camel and Debezium.io can help build the backbone for these types of systems. We’ll even explore the details of a working example that brings all of this together.
Using apache camel for microservices and integration then deploying and managing on Docker and Kubernetes. When we need to make changes to our app, we can use Fabric8 continuous delivery built on top of Kubernetes and OpenShift.
Real-world #microservices with Apache Camel, Fabric8, and OpenShiftChristian Posta
What are and aren't microservices?
Microservices is a validation of the open-source approach to integration and service implementation and a rebuff of the committee-driven SOA approach. In this
Microservices with Apache Camel, DDD, and KubernetesChristian Posta
Building microservices requires more than just infrastructure, but infrastructure does have a role. In this talk we look at microservices from an enterprise perspective and talk about DDD, Docker, Kubernetes and how established open-source projects in the integration space fits a microservices architecture
Atlanta JUG - Integrating Spring Batch and Spring IntegrationGunnar Hillert
This talk is for everyone who wants to efficiently use Spring Batch and Spring Integration together. Users of Spring Batch often have the requirements to interact with other systems, to schedule the periodic execution Batch jobs and to monitor the execution of Batch jobs. Conversely, Spring Integration users periodically have Big Data processing requirements, be it for example the handling of large traditional batch files or the execution of Apache Hadoop jobs. For these scenarios, Spring Batch is the ideal solution. This session will introduce Spring Batch Integration, a project that provides support to easily tie Spring Batch and Spring Integration together. We will cover the following scenarios:
Launch Batch Jobs through Spring Integration Messages
Generate Informational Messages
Externalize Batch Process Execution using Spring Integration
Create Big Data Pipelines with Spring Batch and Spring Integration
How does Apache Pegasus (incubating) community develop at SensorsDataacelyc1112009
A presentation in ApacheCon Asia 2022 from Dan Wang and Yingchun Lai.
Apache Pegasus is a horizontally scalable, strongly consistent and high-performance key-value store.
Know more about Pegasus https://pegasus.apache.org, https://github.com/apache/incubator-pegasus
MariaDB 10.1 what's new and what's coming in 10.2 - Tokyo MariaDB MeetupColin Charles
Presented at the Tokyo MariaDB Server meetup in July 2016, this is an overview of what you can see and use in MariaDB Server 10.1, but more importantly what is planned to arrive in 10.2
MariaDB started life as a database to host the Maria storage engine in 2009. Not long after its inception, the MySQL community went through yet another change in ownership, and it was deemed that MariaDB will be a complete database branch developed to extend MySQL, but with constant merging of upstream changes.
The goal of the MariaDB project is to ensure that everyone is part of the community, including employees of the major steering companies. MariaDB also features enhanced features, some of which are common with the Percona Performance Server. Most importantly, MariaDB is a drop-in replacement and is completely backward compatible with MySQL. In 2010, MariaDB released 5.1 in February, and 5.2 in November – two major releases in a span of one calendar year is a feat that was achieved!
DBAs and developers alike will gain an introduction to MariaDB, what is different with MySQL, how to make use of the feature enhancements, and more.
OSDC 2018 | Scaling & High Availability MySQL learnings from the past decade+...NETWAYS
The MySQL world is full of tradeoffs and choosing a High Availability (HA) solution is no exception. This session aims to look at all of the alternatives in an unbiased nature. While the landscape will be covered, including but not limited to MySQL replication, MHA, DRBD, Galera Cluster, etc. the focus of the talk will be what is recommended for today, and what to look out for. Thus, this will include extensive deep-dive coverage of ProxySQL, semi-sync replication, Orchestrator, MySQL Router, and Galera Cluster variants like Percona XtraDB Cluster and MariaDB Galera Cluster. I will also touch on group replication.
Learn how we do this for our nearly 4000+ customers!
Sina Weibo is the most popular Microblogging platform in China. It has more than 100 million user and tens of millions of daily updates. This slide explains the performance challenges in Weibo platform.
Performance Tuning RocksDB for Kafka Streams’ State Storesconfluent
Performance Tuning RocksDB for Kafka Streams’ State Stores, Bruno Cadonna, Contributor to Apache Kafka & Software Developer at Confluent and Dhruba Borthakur, CTO & Co-founder Rockset
Meetup link: https://www.meetup.com/Berlin-Apache-Kafka-Meetup-by-Confluent/events/273823025/
Technical overview of three of the most representative KeyValue Stores: Cassandra, Redis and CouchDB. Focused on Ruby and Ruby on Rails developement, this talk shows how to solve common problems, the most popular libraries, benchmarking and the best use case for each one of them.
This talk was part of the Conferencia Rails 2009, Madrid, Spain.
http://app.conferenciarails.org/talks/43-key-value-stores-conviertete-en-un-jedi-master
Performance Tuning RocksDB for Kafka Streams' State Stores (Dhruba Borthakur,...confluent
RocksDB is the default state store for Kafka Streams. In this talk, we will discuss how to improve single node performance of the state store by tuning RocksDB and how to efficiently identify issues in the setup. We start with a short description of the RocksDB architecture. We discuss how Kafka Streams restores the state stores from Kafka by leveraging RocksDB features for bulk loading of data. We give examples of hand-tuning the RocksDB state stores based on Kafka Streams metrics and RocksDB’s metrics. At the end, we dive into a few RocksDB command line utilities that allow you to debug your setup and dump data from a state store. We illustrate the usage of the utilities with a few real-life use cases. The key takeaway from the session is the ability to understand the internal details of the default state store in Kafka Streams so that engineers can fine-tune their performance for different varieties of workloads and operate the state stores in a more robust manner.
Companies with batch and stream processing pipelines need to serve the insights they glean back to their users, an often-overlooked problem that can be hard to achieve reliably and at scale. Felix GV and Yan Yan offer an overview of Venice, a new data store capable of ingesting data from Hadoop and Kafka, merging it together, replicating it globally, and serving it online at low latency.
Venice was designed to be the next-generation replacement of the Voldemort Read-Only system, with the intent to provide a broader feature set, better availability characteristics, and a more efficient architecture. Venice is designed for high-throughput ingestion from Hadoop and Kafka, and these data sources can be merged at ingestion time in order to provide semantics similar to those of a lambda architecture but with a simpler, faster, and more available read path. Robustness is a primary architectural concern and, as such, Venice provides highly available reads and writes, self-healing, stringent data validation guarantees, and the ability to roll back entire datasets in cases where bad data is pushed.
Today you can use hosted MySQL/MariaDB/Percona Server in several "cloud providers" in what is considered using it as a service, a database as a service (DBaaS). You can also use hosted PostgreSQL and MongoDB thru various service providers. Learn the differences, the access methods, and the level of control you have for the various public cloud offerings:
- Amazon RDS for MySQL and PostgreSQL
- Google Cloud SQL
- Rackspace OpenStack DBaaS
- The likes of compose.io, MongoLab and Rackspace's offerings around MongoDB
The administration tools and ideologies behind it are completely different, and you are in a "locked-down" environment. Some considerations include:
* Different backup strategies
* Planning for multiple data centres for availability
* Where do you host your application?
* How do you get the most performance out of the solution?
* What does this all cost?
Growth topics include:
* How do you move from one DBaaS to another?
* How do you move all this from DBaaS to your own hosted platform?
Questions like this will be demystified in the talk. This talk will benefit experienced database administrators (DBAs) who now also have to deal with cloud deployments as well as application developers in startups that have to rely on "managed services" without the ability of a DBA.
Presented at Percona Live Amsterdam 2016, this is an in-depth look at MariaDB Server right up to MariaDB Server 10.1. Learn the differences. See what's already in MySQL. And so on.
OSDC 2016 - Tuning Linux for your Database by Colin CharlesNETWAYS
Many operations folk know that performance varies depending on using one of the many Linux filesystems like EXT4 or XFS. They also know of the schedulers available, they see the OOM killer coming and more. However, appropriate configuration is necessary when you're running your databases at scale.
Learn best practices for Linux performance tuning for MariaDB/MySQL (where MyISAM uses the operating system cache, and InnoDB maintains its own aggressive buffer pool), as well as PostgreSQL and MongoDB (more dependent on the operating system). Topics that will be covered include: filesystems, swap and memory management, I/O scheduler settings, using and understanding the tools available (like iostat/vmstat/etc), practical kernel configuration, profiling your database, and using RAID and LVM.
There is a focus on bare metal as well as configuring your cloud instances in.
Learn from practical examples from the trenches.
Cost Effectively Run Multiple Oracle Database Copies at Scale NetApp
Scaling multiple databases with a single legacy storage system works well from a cost perspective, but workload conflicts and hardware contention make these solutions an unattractive choice for anything but low-performance applications.
Comparing Sidecar-less Service Mesh from Cilium and IstioChristian Posta
Service mesh is a powerful pattern for implementing strong zero-trust networking practices, introducing better network observability, and allowing for more fine-grained traffic control. Up until now, the sidecar pattern was used to implement service-mesh capability but as the technology matures, a new pattern has emerged: sidecarless service mesh. Two prominent open-source networking projects, Cilium and Istio, have implemented a sidecar-free approach to service mesh but they both make interesting design decisions and tradeoffs. In this talk we review the architecture of both, focusing on the pros and cons of implementations such as mutual authentication, ingress, and observability.
Understanding Wireguard, TLS and Workload IdentityChristian Posta
Zero Trust Networking has become a standard marketing buzzword but the underlying principles are critical for modern microservice-style architectures. Authentication, authorizations, policy, etc. can be difficult to implement between services and do so in a maintainable way. Google invented their own transparent encryption and authorization protocol called "ALTS" back in 2007 to serve the application layer of Google's Borg workload scheduler, but we don't see others using it outside Google.
In this webinar we look at existing technology like TLS and newcomer Wireguard and see how these technologies come together to provide a secure foundation for workload identity and modern service-to-service networking.
Istio ambient mesh uses a sidecar-less data plane that focuses on ease of operations, incremental adoption, and separation of security boundaries for applications and mesh infrastructure.
In this webinar, we'll explore:
- The forces of modernization and compliance pressures,
- How Zero Trust Architecture (ZTA) can help, and
- How Istio ambient mesh lowers the barrier for establishing the properties necessary to achieve Zero Trust and compliance
Multicluster Kubernetes and Service Mesh PatternsChristian Posta
Building applications for cloud-native infrastructure that are resilient, scalable, secure, and meet compliance and IT objectives gets complicated. Another wrinkle for the organizations with which we work is the fact they need to run across a hybrid deployment footprint, not just Kubernetes. At Solo.io, we build application networking technology on Envoy Proxy that helps solve difficult multi-deployment, multi-cluster, and even multi-mesh problems.
In this webinar, we’re going to explore different options and patterns for building secure, scalable, resilient applications using technology like Kubernetes and Service Mesh without leaving behind existing IT investments. We’ll see why and when to use multi-cluster topologies, how to build for high availability and team autonomy, and solve for things like service discovery, identity federation, traffic routing, and access control.
Cloud-Native Application Debugging with Envoy and Service MeshChristian Posta
Microservices have been great for accelerating the software innovation and delivery, but they also present new challenges, especially as abstractions and automated orchestration at every layer make pinpointing the issue seem like walking around a maze with a blindfold. Existing tools weren’t designed for distributed environments, and the new tools need to consider how to leverage these abstraction layers to better observe, test, and troubleshoot issues.
Christian Posta walks you through Envoy Proxy and service mesh architecture for L7 data plane, the key features in Envoy that can help in debugging and troubleshooting, chaos engineering as a testing methodology for microservices, how to approach a testing and debugging framework for microservices, and new open source tools that address these areas. You’ll explore a workflow to discover and resolve microservices issues, including injecting experiments for stress testing the applications, gathering requests in flight, recording and replaying them, and debugging them step by step without affecting production traffic.
Kubernetes Ingress to Service Mesh (and beyond!)Christian Posta
Kubernetes users need to allow traffic to flow into and within the cluster. Treating the application traffic separately from the business logic allows presents new possibilities in how service to service traffic is served, controlled and observed — and provides a transition to intra cluster networking like Service Mesh. With microservices, there is a concept of both North / South traffic (incoming requests from end users to the cluster) and East / West (intra cluster) communication between the services. In this talk we will explain how Envoy Proxy works in Kubernetes as a proxy for both of these traffic directions and how it can be leveraged to do things like traffic shaping, security, and integrate the north/south to east/west behavior.
Christian Posta (@christianposta) is Global Field CTO at Solo.io, former Chief Architect at Red Hat, and well known in the community for being an author (Istio in Action, Manning, Istio Service Mesh, O'Reilly 2018, Microservices for Java Developers, O’Reilly 2016), frequent blogger, speaker, open-source enthusiast and committer on various open-source projects including Istio, Kubernetes, and many others. Christian has spent time at both enterprises as well as web-scale companies and now helps companies create and deploy large-scale, cloud-native resilient, distributed architectures. He enjoys mentoring, training and leading teams to be successful with distributed systems concepts, microservices, devops, and cloud-native application design.
The exploration of service mesh for any organization comes with some serious questions. What data plane should I use? How does this tie in with my existing API infrastructure? What kind of overhead do sidecar proxies demand? As I've seen in my work with various organizations over the years "if you have a successful microservices deployment, then you have a service mesh whether it’s explicitly optimized as one or not."
In this talk, we seek to understand the role of the data plane and how to pick the right component for the problem context. We start off by establishing the spectrum of data-plane components from shared gateways to in-code libraries with service proxies being along that spectrum. We clearly identify which scenarios would benefit from which part of the data-plane spectrum and show how modern service meshes including Istio, Linkerd, and Consul enable these optimizations.
Deep Dive: Building external auth plugins for Gloo EnterpriseChristian Posta
Using the plugin framework for Ext. Auth Service in Gloo Enterprise, we can build any custom AuthN/AuthZ plugins to handle security requirements not provided out of the box.
Role of edge gateways in relation to service mesh adoptionChristian Posta
API Gateways provide functionality like rate limiting, authentication, request routing, reporting, and more. If you’ve been following the rise in service-mesh technologies, you’ll notice there is a lot of overlap with API Gateways when solving some of the challenges of microservices. If service mesh can solve these same problems, you may wonder whether you really need a dedicated API Gateway solution?
The reality is there is some nuance in the problems solved at the edge (API Gateway) compared to service-to-service communication (service mesh) within a cluster. But with the evolution of cluster-deployment patterns, these nuances are becoming less important. What’s more important is that the API Gateway is evolving to live at a layer above service mesh and not directly overlapping with it. In other words, API Gateways are evolving to solve application-level concerns like aggregation, transformation, and deeper context and content-based routing as well as fitting into a more self-service, GitOps style workflow.
In this talk we put aside the “API Gateway” infrastructure as we know it today and go back to first principles with the “API Gateway pattern” and revisit the real problems we’re trying to solve. Then we’ll discuss pros and cons of alternative ways to implement the API Gateway pattern and finally look at open source projects like Envoy, Kubernetes, and GraphQL to see how the “API Gateway pattern” actually becomes the API for our applications while coexisting nicely with a service mesh (if you adopt a service mesh).
Navigating the service mesh landscape with Istio, Consul Connect, and LinkerdChristian Posta
Service mesh has hit the cloud native computing community like a storm, and we’re starting to see gradual adoption across the enterprise. There are a handful of open source service mesh implementations to choose from, including Istio, Consul Connect, and Linkerd.
Christian Posta details why and when you may want to use a service mesh versus when you may want to just stick with a library, Netflix OSS, or application approach. He digs into three popular open source service mesh implementations and explores their goals, strengths, and weaknesses. You’ll come away with a good foundation from which to explore service mesh technology and ask the right questions to get to the right answer for them.
Distributed microservices introduce new challenges: failure modes are harder to anticipate and resolve. In this session, we present a “Chaos Debugging” framework enabled by three open source projects: Gloo Shot, Squash, and Loop to help you increase your microservices’ “immunity” to issues.
Gloo Shot integrates with any service mesh to implement advanced, realistic chaos experiments. Squash connects powerful and mature debuggers (gdb, dlv, java debugging) to your microservices while they run in Kubernetes. Loop extends the capability of your service mesh to observe your application and record full transactions for sandboxed replay and debugging.
Come to this demo-heavy talk to see how together, Squash, Gloo Shot, and Loop allow you to trigger, replay, and investigate failure modes of your microservices in a language agnostic and efficient manner without requiring any changes to your code.
Leveraging Envoy Proxy and GraphQL to Lower the Risk of Monolith to Microserv...Christian Posta
If you have an existing Java monolith, you know you must take care making changes to it or altering it in any negative way. Often times these monoliths are very valuable to the business and generate a lot of revenue. At the same time, since it’s difficult to make changes to the monolith it’s desirable to move to a microservices architecture. Unfortunately you cannot just do a big-bang migration to a greenfield architecture and will have to incrementally adopt microservices. In this talk, we’ll look at using Gloo proxy which is based on Envoy Proxy and GraphQL to do surgical, function-level traffic control and API aggregation to safely migrate your monolith to microservices and serverless functions.
Service-mesh options with Linkerd, Consul, Istio and AWS AppMeshChristian Posta
Service mesh abstracts the network from developers to solve three main pain points:
How do services communicate securely with one another
How can services implement network resilience
When things go wrong, can we identify what and why
Service mesh implementations usually follow a similar architecture: traffic flows through control points between services (usually service proxies deployed as sidecar processes) while an out-of-band set of nodes is responsible for defining the behavior and management of the control points. This loosely breaks out into an architecture of a "data plane" through which requests flow and a "control plane" for managing a service mesh.
Different service mesh implementations use different data planes depending on their use cases and familiarity with particular technology. The control plane implementations vary between service-mesh implementations as well. In this talk, we'll take a look at three different control plane implementations with Istio, Linkerd and Consul, their strengths, and their specific tradeoffs to see how they chose to solve each of the three pain points from above. We can use this information to make choices about a service mesh or to inform our journey if we choose to build a control plane ourselves.
API Gateways are going through an identity crisisChristian Posta
API Gateways provide functionality like rate limiting, authentication, request routing, reporting, and more. If you've been following the rise in service-mesh technologies, you'll notice there is a lot of overlap with API Gateways when solving some of the challenges of microservices. If service mesh can solve these same problems, you may wonder whether you really need a dedicated API Gateway solution?
The reality is there is some nuance in the problems solved at the edge (API Gateway) compared to service-to-service communication (service mesh) within a cluster. But with the evolution of cluster-deployment patterns, these nuances are becoming less important. What's more important is that the API Gateway is evolving to live at a layer above service mesh and not directly overlapping with it. In other words, API Gateways are evolving to solve application-level concerns like aggregation, transformation, and deeper context and content-based routing as well as fitting into a more self-service, GitOps style workflow.
In this talk we put aside the "API Gateway" infrastructure as we know it today and go back to first principles with the "API Gateway pattern" and revisit the real problems we're trying to solve. Then we'll discuss pros and cons of alternative ways to implement the API Gateway pattern and finally look at open source projects like Envoy, Kubernetes, and GraphQL to see how the "API Gateway pattern" actually becomes the API for our applications while coexisting nicely with a service mesh (if you adopt a service mesh).
KubeCon NA 2018: Evolution of Integration and Microservices with Service Mesh...Christian Posta
Cloud-native describes a way of building applications on a cloud platform to iteratively discover and deliver business value. We now have access to a lot of similar technology that the large internet companies pioneered and used to their advantage to dominate their respective markets. What challenges arise when we start building applications to take advantage of this new technology?
In this talk we'll explore the role of service meshes when building distributed systems, why they make sense, and where they don't make sense. We will look at a class of problem that crops up that service mesh cannot solve, but that frameworks and even new programming languages like Ballerina are aiming to solve
Service-mesh technology promises to deliver a lot of value to a cloud-native application, but it doesn't come without some hype. In this talk, we'll look at what is a "service mesh", how it compares to similar technology (Netflix OSS, API Management, ESBs, etc) and what options for service mesh exist today.
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...UiPathCommunity
💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™:
See how to accelerate model training and optimize model performance with active learning
Learn about the latest enhancements to out-of-the-box document processing – with little to no training required
Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages
This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath.
Speakers:
👨🏫 Andras Palfi, Senior Product Manager, UiPath
👩🏫 Lenka Dulovicova, Product Program Manager, UiPath
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Tobias Schneck
As AI technology is pushing into IT I was wondering myself, as an “infrastructure container kubernetes guy”, how get this fancy AI technology get managed from an infrastructure operational view? Is it possible to apply our lovely cloud native principals as well? What benefit’s both technologies could bring to each other?
Let me take this questions and provide you a short journey through existing deployment models and use cases for AI software. On practical examples, we discuss what cloud/on-premise strategy we may need for applying it to our own infrastructure to get it to work from an enterprise perspective. I want to give an overview about infrastructure requirements and technologies, what could be beneficial or limiting your AI use cases in an enterprise environment. An interactive Demo will give you some insides, what approaches I got already working for real.
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Albert Hoitingh
In this session I delve into the encryption technology used in Microsoft 365 and Microsoft Purview. Including the concepts of Customer Key and Double Key Encryption.
Elevating Tactical DDD Patterns Through Object CalisthenicsDorra BARTAGUIZ
After immersing yourself in the blue book and its red counterpart, attending DDD-focused conferences, and applying tactical patterns, you're left with a crucial question: How do I ensure my design is effective? Tactical patterns within Domain-Driven Design (DDD) serve as guiding principles for creating clear and manageable domain models. However, achieving success with these patterns requires additional guidance. Interestingly, we've observed that a set of constraints initially designed for training purposes remarkably aligns with effective pattern implementation, offering a more ‘mechanical’ approach. Let's explore together how Object Calisthenics can elevate the design of your tactical DDD patterns, offering concrete help for those venturing into DDD for the first time!
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
2. 2
• Apache ActiveMQ
• New features!
• Demos
• Apache ActiveMQ Apollo
Agenda for the night
3. 3
Your speaker
Christian Posta
Blog: http://christianposta.com/blog
Email: ceposta@apache.org
Twitter: @christianposta
• Senior Consultant and Architect at Red Hat (formerly FuseSource)
• Committer at Apache Software Foundation: ActiveMQ, Apollo
• PMC at ActiveMQ
• Author: Essential Camel Components DZone Refcard
• Contributor to Apache Camel
5. 5
• The most widely used open-source
messaging broker
• Highly configurable
• Friendly license (no license fees!)
• Vibrant community (TLP)
• Backbone of top enterprises in retail, e-retail,
financial services, shipping, many others!
Apache ActiveMQ
6. 6
• High performance
• High availability
• Light-weight
• Multi-protocol
• JMS compliant
• Backed by Red Hat!
ActiveMQ Features
7. 7
• TCP, NIO
• UDP
• SSL, SSL+NIO
• VM
• HTTP
• WebSockets
Breadth of connectivity
11. 11
• AMQP 1.0 protocol
• REST management with Jolokia
• Pure master slave deprecated and removed
• Java 7 support
• Split up client libs, modularize core
packages
Quick recap of 5.8.0 release
12. 12
• Fall 2013
• New, faster, default file-based store
• Persistence store HA replication
• New management console, HawtIO
• New “broker:” Camel component
• Other new features
ActiveMQ 5.9 on its way!
14. 14
• LevelDB
• Hardened
• JNI (native) and Java versions
• Java version packaged by default
https://github.com/dain/leveldb
• Can download
http://code.google.com/p/leveldb/downloads/list
Default in 5.9.0 – LevelDB
15. 15
• AMQ Message Store (deprecated!!!)
Don’t use this one!
• KahaDB
• LevelDB
File-based stores
16. 16
• Journal / TX Log
• Indexes
• Recovery Logs
File-based stores
Index Journal
Redo Log
X X X X
17. 17
• Homegrown
• Optimized for messaging
• TX log, WAL log, Indexes
• B-Tree based indexes
• Known bottlenecks
KahaDB
18. 18
• Google NoSQL key-value DB
http://code.google.com/p/leveldb/
• Based on BigTable
• Chrome, Riak, IndexedDB
• No relational model, queries, indexes
• Store keys sorted
LevelDB
19. 19
• Underlying data structures are optimized for
sequential access and lots of writes
http://en.wikipedia.org/wiki/Log-structured_merge-tree
• Concurrent reads
• Pause-less log cleanup
• Built-in compression
• JMX
LevelDB cont’d
21. 21
• Makes for very fast store index!
• Fewer entries in index
• Composite sends only store message once
• HDFS support!
• Replication!?...
LevelDB Store
28. 28
• Extreme reliability – but not as fast
• Recommended if already using an
enterprise database
• No restriction on number of slaves
• Simple configuration
JDBC Master-Slave
30. 30
• Recommended if you have a SAN, or DRBD
• Ensure file locking works – and times out – NFSv4
good!
https://issues.apache.org/jira/browse/AMQ-4378
• No restriction on number of slaves
• Simple configuration
• Best performance
Shared File System M/S
31. 31
Local File System Local File SystemLocal File System
Broker
Slave
Broker
Master
Broker
Slave
ZooKeeper
Cluster
Replicated LevelDB
Master Slave – NEW!
32. 32
ZooKeeper
Cluster
Local File System Local File SystemLocal File System
Broker
Slave
Broker
Master
Client
Broker
Slave
Larry’sRemovals
Replicated LevelDB
Master Slave
33. 33
• Requires a HA ZooKeeper Cluster
• No Single Point of Failure
• Dynamic number of slaves
• Sync Replication
• Local Mem/Disk
• Remote Mem/Disk
• Quorum Mem/Disk
Replicated LevelDB
Master Slave
38. 38
• Manage integration infrastructure from one dashboard
• Customizable
• Plugins
• Camel
• ActiveMQ
• Fabric
• Infinispan
• Tomcat
• Many others!
• Visualizations
• One dashboard to rule them all
HawtIO – http://hawt.io
39. 39
• Default ActiveMQ dashboard
• Visualization of health
• Access to operations to make changes
• Move messages from DLQ to original destinations
• Visualize Camel routes deployed along with broker
• Send messages
• Real-time metrics
HawtIO – http://hawt.io
46. 46
• Use Apache Camel routes
• Creates destination interceptor at runtime
• Embed in camel.xml and deploy with ActiveMQ
• More powerful than existing interceptors (use when
needed)
• http://rajdavies.blogspot.com/2013/09/apache-camel-broker-component-for.html?
tw
Broker component
Say some things about FuseSource + Red Hat here… and what I do… So I started out working for FuseSource about a year and a half ago. Fuse Source was a open-source subscription company built around the integration projects at Apache, specifically Apache ActiveMQ, Camel, ServiceMix, and CXF. Basically these projects are best-of-breed and highly adopted freely by community users and used for mission critical infrastructures when building out SOA and other distributed integrations. The thing is, big companies who invest millions of dollars into their businesses aren’t willing to accept using a mailing list and irc for production support, aka when shit hits the fan, they need to be able to rely on some strong partners who would be able to help them out. That’s were fuse source, fit into that picture. It was started by the guys that co-founded the projects, and they were able to build up an amazing set of support engineers and consultancy teams. Along the way, they hired up a lot of the committers on each of the respective projects, and put together professional documentation, on-site and virtual training, an annual conference devoted specifically to these technologies, as well as and most importantly support subscriptions for both production and dev support. We were officially welcomed into the Jboss Redhat family almost exactly a year ago and the spirit of open-source itnegration and SOA lives on under the RedHat umbrella and in complement of the existing Jboss offerings including EAP, Drools/BRMS, and jBPM, etc.
What is activemq… how would I explain what that is?? A message broker… it moves messages from a producer to a consumer.. It takes the responsibility of delivering a self-contained piece of data between two clients… the producing client doesn’t have to know key details usually found in traditional RPC.. Where is the client? Is the client ready to handle a message? What communication protocol can the client support? A messaging broker helps to abstract away all of those details in an effort to make the programming model conceptually easy… a producer is responsible for producing data.. But how it gets to where it needs to go is delegated to another entity. This is beneficial because it keeps the producer simpler, keeps the interaction simple (from A to B), decouples producer and consumer, and many other things. High level: alleviates responsibility from producer to deliver messages. Low Level: takes a piece of data and tries to deliver it as fast and reliably as possible Highly configurable – swiss army knife of messaging, because messaging use cases vary so widely. What type of messaging, how many consumers/producers, what to do in failure scenarios, what level of througput or latency do you need, what type of hardware and OS are you running on, clustering, HA, reliable/unreliable network, etc… there is no one solution that fits all… Vibrant community – very active mailing list (usually within 15 mins response), JIRAs, bug fixes, new features, books, Zappos, UPS, Home Depot, Walmart, Harris Corp, Yahoo, FAA, SAA, Ticketmaster, GM, IHG, Sabre, CERN, Mars rover projects, Wells Fargo,
Some of the main features from ActiveMQ, and reasons why so many people use it, include High performance… so moving messages asynchronously from a producer to eligible consumers as quickly and as efficiently as possible. High availability… so getting messages from A to B is easy when everything is working correctly, but as you all probably already know, distributed systems have to be able to cope with failures, because failures and unplanned disruptions are basically the “norm”.. The larger the network of your distributed system, the higher chance something will go wrong Light-weight… of course these all sound like buzzwords because they’re so overused and abused by those people interested in selling you stuf… but activemq is truly light-weight in that it can be deployed stand alone, in any J2EE container if desired, or even embedded within your own java application. Other messaging brokers, like MQ Series or even other open-source alternatives like Rabbit, cannot be embeded into your Java app … but why would you do that? Well, just like you can build highly performant, scalable distributed systems, so too can you architect your app internally to use asynchronous message passing, but with reliability built in, and options for extensibilty when needed by using broker networks, and routing with camel, etc.
Some of the main features from ActiveMQ, and reasons why so many people use it, include High performance… so moving messages asynchronously from a producer to eligible consumers as quickly and as efficiently as possible. High availability… so getting messages from A to B is easy when everything is working correctly, but as you all probably already know, distributed systems have to be able to cope with failures, because failures and unplanned disruptions are basically the “norm”.. The larger the network of your distributed system, the higher chance something will go wrong Light-weight… of course these all sound like buzzwords because they’re so overused and abused by those people interested in selling you stuf… but activemq is truly light-weight in that it can be deployed stand alone, in any J2EE container if desired, or even embedded within your own java application. Other messaging brokers, like MQ Series or even other open-source alternatives like Rabbit, cannot be embeded into your Java app … but why would you do that? Well, just like you can build highly performant, scalable distributed systems, so too can you architect your app internally to use asynchronous message passing, but with reliability built in, and options for extensibilty when needed by using broker networks, and routing with camel, etc.
AMQP (1.0) A binary format originally from JPMorgan Chase, Originally targeted toward wire-level and messaging semantic interoperability between messaging vendors, but with version 1.0, it has removed some of the semantics of exchanges and queing and is essentially a wire-level protocol. ActiveMQ uses the popular Proton project, which is the same underpinnings of Apache Qpid. MQTT (3.1) A super-compact bianry protocol developed by IBM and others intended to be used exclusively for reliable pub-sub for limited devices on highly unreliable networks. think small monitoring devices like gas meters, pacemakers, medical sensors, car sensors, etc. The idea is to allow all of these sensors out in the field collecting data, known as the "internet of things" to be able to reliably transmit their data through messaging. OpenWire (v1-10) Is the original binary wire-level protocol intended to be an open standard. It's used to implement the JMS API in a highly-performant and feature-rich way. Used by default unless you explicitly specify another proptocol. Backward compatible (as discussed above) with previous versions. STOMP (1.0, 1.1, 1.2) Simple Text Oriented Messaging Protocol who's main goals are simplicity and basic connectivity: get a client talking to the messaging broker regardless of what platform, or programming language that's used. As a result, clients exist in python, ruby, scala, perl, c/c++, .Net, Delphi, JavaScript, Objective-C, PHP, Erlang, Go, etc, etc.. point is, it's a text-protocol that's easy to implement so many languages already have and are using it.
Async integration – allowing apps to do what they are focused on doing without being tied to the dependencies of other systems just because they need to integrate. For example, App A shouldn’t have to tie up threads and resources trying to communicate with App B synchronously or when App B is not available. The idea is to let App A send its message (data) off knowing it will get to App B when App B is ready to handle it, but not take responsibility for that data xfer. That way App A can continue to do the processing it needs on its side, and if it needs to be aware of a response from App B, that response will show up asynchronously and handled when App A is ready to. Using asynchronous processing like this is one important way to get higher throughput and utilization of your systems. When App A needs to send a piece of data to B, not only does it want to do so fast and efficienlty, but it wants to have some level of guarantee that the data didn’t get lost. ActiveMQ plays the part of mediator and broker here by establishing protocols with the producers and consumers to guarantee message delivery even in the face of unplanned failures. Loose coupling – coupling really should be thought of In terms of levels or degress of coupling. Or better yet, the assumptions made by one system on the other system. Asumptions like availability, programming language, data types, network interruptions, etc, are dealt with and aleviated in some cases with messaging. Heterogeneous architectures – be able to reduce the spider web of point to point connections and allow apps written in different languages on different platforms, owned by different groups with different upgrade paths and different lifecycles to be able to reliably communicate with each other. http://www.openamq.org/doc:amqp-background
AMQP 1.0 protocol was first introduced. AMQP was introduced in 2004-2005 JP Morgan Chase as a way to “commoditize” messaging platforms and achieve interoperability between clients and servers. The original spec and still the 0.8 and 0.9 specs forced certain implementation details onto the different brokers and that severely limited adoption. For 1.0 release, a lot of the implemention detail was removed and it’s now just a binary wire-level protocol. It still remains to see what level of “interoperability” can be achieved. But we do support it. ActiveMQ’s impl is powered by the Apache Qpid proton project which is a protocol engine that third parties can use to easily build out AMQP clients and servers. Jolokia is an awesome opensource project that basically provides an alternative to JSR-160 which specifies the connectors that one typically uses for JMX connections. Usually, it’s done by setting up an mbean server which registeres some JNDI and RMI endpoints which you connect to. Jolokia on the other hand is implemented as a java agent which allows HTTP/REST JSON access to the JMX mbeans. It also provides features like bulk requests, fine-grained security and authorizations, etc. Pure master slave deprecated – and now removed and not available… this was the original “shared nothing” master slave, where the master would dispatch to the slave before ack’ing the producer. In this scenario, if a master went down, the slave would be live and be able to take over. Some of the drawbacks of this approach is there would need to be manual intervention to restore the original master as a slave. So you would have to stop the slave, re-sync the master/slave manually, then restart the pair again. Started testing and certifying for Java7 Refactored a lot of the packages so that are more cleanly organized and can be included as dependencies more finely… for example, if you wanted to strip down the distro to its most bare and required components. Or if you wanted to split out the client libs so that you don’ t need to icnlude the broker code too
Expected out this fall.. The last few bug fixes are being committed as we speak, most large features are already baked and ready for release, but since this is still an open-source and community driven project, we do it in our spare time and are working as best we can to get it out We can expect the default store, to change from KahaDB which has served us very well the past couple years, to a newer faster store. We have some new options for HA replication. New management console which has been long over due. Other features like MQTT over websockets, and others I will briefly mention And of course, the community as well as our commercial customers have unearthed some obscure use cases that bring to life some bugs which we quickly squash.
So LevelDB will become the new default store in the 5.9.0 release. Curious if any in this room have played with LevelDB with or without ActiveMQ? We’ve already had a lot of adoption of it in the community and in commercial customers, and we’ve received a lot of feedback. It’s been hardened and now be ready for prime time. There are two different implementations: a native version and a pure java version. The Java version was ported from the native C++ implementation by Dain Sundstrum (a brilliant dude who works at Facbook now). The java version is what’s shipped and enabled by default in the community distro of 5.. There is also the native version available directly from and maintained by google at the link above. This is written in C++ and takes advantage of some of the specific OS features such as AIO and can be faster in some use cases.
There are effectively two file-based stores currently in ActiveMQ. KahaDB and LevelDB. There is also the original AMQ store which still lingers in previous versions. This version is not supported anymore and should not be used. It’s slower, a little more complex that it needs to be, and is not even available in the newer releases. The other two provide much better performance.
So KahaDB is an awesome file-based messaging DB… it has served the community very well for the past years, including helping us beat out IBM MQ series at UPS It’s very fast, good recovery time on failover, well understood, highly tunable. It was homegrown written by the original project co-founders to better fit the messaging paradigm by keeping as much of the index in memory as possible, batching writes to the disk, and allowing to tune page size, buffers, checkpoints, recovery mechanisms, and others. Its implemented using some of the traditional db components like B-tree indexes, write-ahead-logs, and transaction logs, however, it’s highly optimized for writes and deletes, whereas a traditional RDMS is highly optimized for reads and complex queries. There are known limitations in the implementations and bottlenecks when approaching its limits and they usually show up in the indexes. There are multiple btree indexes to help preserve message order, message priority, durable subscriptions and others. Having to update all these data structures, no matter how creatively, using btress becomes expensive at the upper limits of performance. So, actually, when proving out theories in Apache Apollo, which I’ll talk about next, we found LevelDB to fit our usecases very well.
So KahaDB is an awesome file-based messaging DB… it has served the community very well for the past years, including helping us beat out IBM MQ series at UPS It’s very fast, good recovery time on failover, well understood, highly tunable. It was homegrown written by the original project co-founders to better fit the messaging paradigm by keeping as much of the index in memory as possible, batching writes to the disk, and allowing to tune page size, buffers, checkpoints, recovery mechanisms, and others. Its implemented using some of the traditional db components like B-tree indexes, write-ahead-logs, and transaction logs, however, it’s highly optimized for writes and deletes, whereas a traditional RDMS is highly optimized for reads and complex queries. There are known limitations in the implementations and bottlenecks when approaching its limits and they usually show up in the indexes. There are multiple btree indexes to help preserve message order, message priority, durable subscriptions and others. Having to update all these data structures, no matter how creatively, using btress becomes expensive at the upper limits of performance. So, actually, when proving out theories in Apache Apollo, which I’ll talk about next, we found LevelDB to fit our usecases very well.
LevelDB, as mentioned, is a nosql key-value database from google inspired by their work on map reduce, bigtable, and big data storage algorithms. It’s currently used in Chrome, Riak,
So the LevelDB based store still uses WAL and TX logs, but we now use the LevelDB engine to index the tx log. So now we rely on log-structured merge trees, not btrees, and google’s algoriitm for sorting and compressing things. The structures since they are sorted are very friendly for sequential access type systems, like messaging. They also allow very good concurrency features including reads, wheras kahadb had lots of locks and contention on reads and writes. We also use fewer entries per message and destination, which itself also reduces contention at the index… It also comes with throw-in features like JMX out of the box
So what does this mean? "enqueue" "drained enqueue" and "loaded enqueue"? enqueues while there are no consumers (queue starts empty) enqueues while the queue is being drained (it has consumers) enqueues when queue is very large (millions of messages) in benchmarks against kahadb, we see that it outperforms quite a bit. This is using 20 byte structures and async sends: Questions: What are “loaded enqueues” vs loaded dequeues, vs drained enqueues, etc. We use 20 byte bodies here to really push the storage engine. We don’t use larger bodies because then the data is natrually batched, and would give better performance. So we choose usecases and payloads that would inherently require the store to figure out how to properly batch and deal with the payload.
There is also support for HDFS! So basically the store will keep a local copy as well as upload to HDFS and sync anything missed at regular checkpoints. When a master fails, and a slave takes over, it will d/l the last known sst files with latest indexes. Then normal recovery kicks in. Note, for this set up, Master/Slave elections are started manually or coordinated externally with something like zookeeper. Pauseless log clean up – kahadb had period cleanup intervals during which log operations were suspended.. This isn’t the case with the leveldb indexes and cleanup Composite sends, like Virtual Topics, or the new JMS 2.0 durable subscribers, or fanout-type sends to multiple destinations
What is HA? High Availability… but people have different notions of what HA is… You can have two basic types with ActiveMQ: #1 where messages get high-availability in this case, you have producers and consumers to a broker that require their messages to be delivered even in the case of faults. So if a producer sends to broker A, then broker A guarantees deliver to consumers. #2 where clients must be able to connect and send to a broker.. Regardless of past messages, and with the most minimal of delay… We will be talking about the first kind, where clients must be able to eventually connect to a broker and messages guaranteed to be delivered…
Note these are the out of the box configs for HA,… but consider network of brokers to play a role in HA also…
The high-availability features in activemq depend on the client participating and able to failvoer to the slave brokers when connectivity to the master has been interrupted. To do that, you can use the failover transport which takes URIs for the brokers participating in the HA cluster. The failover is supported out of the box for the Java, .NET and C++ clients. In this case if the master goes away, the client will attempt to connect to the slave. There are parameters for prioritizing the backup (so that the client goes back to the master if the msater becomes available) or for delayed or exponential back off, or randomizign the attempts to reconnect, etc. You can also rely on different discovery mechanisms for locating the brokers in the cluster. You can use Fuse Fabric which under the covers uses zookeeper to locate master/slaves
The general pattern is for Slaves to be in warm-standby mode so that if a master gets hit by a truck, a slave will be elecftd master and the client will rely on its failover protocol to be able to detect which broker is the slave and connect to that one Transactions that have not been committed and are in progress are replayed upon reconnection. These HA scenarios shown above currently rely on some kind of shared backing store like NFS or SAN or RDBMS.
Note these are the out of the box configs for HA,… but consider network of brokers to play a role in HA also…
Simple to configure, easy to understand, uses DB table locks, or lease locks. So the master will first grab the exclusive lock, and all slaves will sit idle until the msater has relinquished the lock. So you can have as many idle broker slaves listening for the lock. The first to get the lock when master goes down becomes elected master. We’ve seen this in enterprises where there is good in-house expertise in RDBMs and they are comfortable administering the DB as SPOF and creating redundant solutions for the DB. Also, one thing to keep in mind, RDBMs are not optimized for messaging based use-cases, and will not be nearly as performant as the file-based solutions. On the other hand, all of your RDBMs tools are relevant, messages can be queried, etc…
Ensure your shared file locks work This requires distributed exclusive locks, thus file systems that support this. Because the election depends on the master getting the lock while slaves line up behind waiting for the lock. Basically boils down to being able to take take locks using the way the JVM creates locks (using posix locks, lockf and fcntl). flock is not sufficient… for eg, ocfs2 uses flock but w. newer linux kernel can be run in “ userspace cluster ” that uses lockf and AMQ can use that. You can add as many slaves as you’d like. Highly tunable for best performance.
Coordination and master election are handled by Zookeeper Apache Zookeeper is a distributed coordination and configuraiton service that itself is HA. It is a TLP from Apache that originated as the cluster coordination service used by Hadoop in a Hadoop cluster. You can use zookeeper to build out distributed data structures (queues), locks, synchronizations,, or use it for central location for configuration, or master election. Zookeeper behaves properly when nodes go down, and even during network partiitions. Zookeeper ensemble should always be odd, 1, 3, 5, etc. because a majority is always needed for voting This is what Fuse Fabric uses. Think of it kinda like an LDAP with directory structure with znodes Locks, Barriers, Queues, Master Election, Rendezvous, Group Membership
Management of your middleware infrastructure is critical. ActiveMQ had two main avenues for management and monitoring: #1 JMX through Jconsole, VisualVM, or using a tool like nagios or any of the commercial options for monitoring. #2 was a web-based console that shipped out of the box. The webconsole does give basic overview so you can see queues, basic destination stats, connections, network connectors, etc. Bottom line it was fairly limited, had no visualizations, and could not perform some very common and needed operations on the broker. For example, it was clumsy to move a message from a DLQ back to an original destination. However, one of the biggest motivations for moving away from this webconsole is: Every product comes with its own console! ActiveMQ’s was “just another console” .. So when you had FuseESB which contained ActiveMQ, Karaf, Felix, Camel, and potentially other JVM based apps with their own consoles, you had n consoles which becomes a huge pain.
So HawtIO is a single page web app written with angular-js and written with plugins and customization in mind. It’s intended to be the one and only one console that your JVM apps need for monitring and visualiztions. HawtIO will automatically discover your plugins or services that it knows how to manage and will enable them on your dashboard. It can mange and visualize your Camel, ActiveMQ, Fabric, FuseESB, Jboss, Infinispan, tomcat, others….
So HawtIO is a single page web app written with angular-js and written with plugins and customization in mind. It’s intended to be the one and only one console that your JVM apps need for monitring and visualiztions. HawtIO will automatically discover your plugins or services that it knows how to manage and will enable them on your dashboard. It can mange and visualize your Camel, ActiveMQ, Fabric, FuseESB, Jboss, Infinispan, tomcat, others….
Management of your middleware infrastructure is critical. ActiveMQ had two main avenues for management and monitoring: #1 JMX through Jconsole, VisualVM, or using a tool like nagios or any of the commercial options for monitoring. #2 was a web-based console that shipped out of the box. The webconsole does give basic overview so you can see queues, basic destination stats, connections, network connectors, etc. Bottom line it was fairly limited, had no visualizations, and could not perform some very common and needed operations on the broker. For example, it was clumsy to move a message from a DLQ back to an original destination. However, one of the biggest motivations for moving away from this webconsole is: Every product comes with its own console! ActiveMQ’s was “just another console” .. So when you had FuseESB which contained ActiveMQ, Karaf, Felix, Camel, and potentially other JVM based apps with their own consoles, you had n consoles which becomes a huge pain.
Management of your middleware infrastructure is critical. ActiveMQ had two main avenues for management and monitoring: #1 JMX through Jconsole, VisualVM, or using a tool like nagios or any of the commercial options for monitoring. #2 was a web-based console that shipped out of the box. The webconsole does give basic overview so you can see queues, basic destination stats, connections, network connectors, etc. Bottom line it was fairly limited, had no visualizations, and could not perform some very common and needed operations on the broker. For example, it was clumsy to move a message from a DLQ back to an original destination. However, one of the biggest motivations for moving away from this webconsole is: Every product comes with its own console! ActiveMQ’s was “just another console” .. So when you had FuseESB which contained ActiveMQ, Karaf, Felix, Camel, and potentially other JVM based apps with their own consoles, you had n consoles which becomes a huge pain.