All Things Open 2014 - Day 1
Wednesday, October 22nd, 2014
Gavin Roy
VP of Architecture for AWeber Communications
Back Dev
Messaging Standards and Systems - AMQP & RabbitMQ
Find more by Gavin here: https://speakerdeck.com/gmr
RabbitMQ is an open source message-broker software that originally implemented the Advanced Message Queuing Protocol (AMQP).it accepts and forwards messages.
PPT concetrates on :
1. Introduction to RabbitMQ .
2. Difference between AMQP and JMS
3. Different types of exchanges
4. Routing Key and Binding concept
The session presents in the details the RabbitMQ message broker along with demonstrations using the Java client, the Spring integration for RabbitMQ and the administration tools provided as part of the RabbitMQ installation.
Let's learn about :
What is RabbitMQ ?
Why RabbitMQ ?
RabbitMQ vs Kafka ? Which one is suitable for your purpose.
Advantages of RabbitMQ
Support By RabbitMQ
RabbitMQ is one part of Message Broker that implemented Advance Message Queue Protocol (AMQP), that help your application to communicate each other, when you extends your application scale.
A presentation from internal meeting on Message Broker System and RabbitMQ. RabbitMQ is open source message broker software that implements the Advanced Message Queuing Protocol (AMQP).
RabbitMQ is an open source message-broker software that originally implemented the Advanced Message Queuing Protocol (AMQP).it accepts and forwards messages.
PPT concetrates on :
1. Introduction to RabbitMQ .
2. Difference between AMQP and JMS
3. Different types of exchanges
4. Routing Key and Binding concept
The session presents in the details the RabbitMQ message broker along with demonstrations using the Java client, the Spring integration for RabbitMQ and the administration tools provided as part of the RabbitMQ installation.
Let's learn about :
What is RabbitMQ ?
Why RabbitMQ ?
RabbitMQ vs Kafka ? Which one is suitable for your purpose.
Advantages of RabbitMQ
Support By RabbitMQ
RabbitMQ is one part of Message Broker that implemented Advance Message Queue Protocol (AMQP), that help your application to communicate each other, when you extends your application scale.
A presentation from internal meeting on Message Broker System and RabbitMQ. RabbitMQ is open source message broker software that implements the Advanced Message Queuing Protocol (AMQP).
SpringOne Platform 2016
Speakers: Kevin Hoffman; Advisory Solutions Architect, Pivotal & Chris Umbel; Advisory Architect, Pivotal
With the advent of ASP.NET Core, developers can now build cross-platform microservices in .NET. We can build services on the Mac, Windows, or Linux and deploy anywhere--most importantly to the cloud.
In this session we'll talk about Cloud Native .NET, building .NET microservices, and deploying them to the cloud. We'll build services that participate in a robust ecosystem by consuming OSS servers such as Spring Cloud Configuration Server and Eureka. We'll also show how these .NET microservices can take advantage of circuit breakers and be automatically deployed to the cloud via CI/CD pipelines.
Apache Kafka vs RabbitMQ: Fit For Purpose / Decision TreeSlim Baltagi
Kafka as a streaming data platform is becoming the successor to traditional messaging systems such as RabbitMQ. Nevertheless, there are still some use cases where they could be a good fit. This one single slide tries to answer in a concise and unbiased way where to use Apache Kafka and where to use RabbitMQ. Your comments and feedback are much appreciated.
An introduction to KrakenD, the ultra-high performance API Gateway with middlewares. An opensource tool built using go that is currently serving traffic in major european sites.
Easy enterprise application integration with RabbitMQ and AMQPRabbit MQ
VMware vFabric RabbitMQ Technical Webinar December 2010 by VMware engineer Emile Joubert. Covers common integration patterns, and how RabbitMQ makes these easily implemented, using AMQP as a communications mechanism.
You can view a recording of this presentation on YouTube: http://www.youtube.com/user/SpringSourceDev#p/c/5956C6D9EC319817/0/ABGMjX4K0D8
Architecture | The Future of Messaging: RabbitMQ and AMQP | Eberhard WolffJAX London
2011-11-02 | 05:45 PM - 06:35 PM
The JMS standard is 9 years old - but outside the Java community innovation is happening. The AMQP standard with implementations like RabbitMQ is gaining more and more traction. This session explains the standard and its advantages. It will also show how an AMQP application can be implemented using Java.
Kafka Streams is a new stream processing library natively integrated with Kafka. It has a very low barrier to entry, easy operationalization, and a natural DSL for writing stream processing applications. As such it is the most convenient yet scalable option to analyze, transform, or otherwise process data that is backed by Kafka. We will provide the audience with an overview of Kafka Streams including its design and API, typical use cases, code examples, and an outlook of its upcoming roadmap. We will also compare Kafka Streams' light-weight library approach with heavier, framework-based tools such as Spark Streaming or Storm, which require you to understand and operate a whole different infrastructure for processing real-time data in Kafka.
We start with an introduction to what Apache Camel is, and how you can use Camel to make integration much easier. Allowing you to focus on your business logic, rather than low level messaging protocols, and transports. You will also hear what other features Camel provides out of the box, which can make integration much easier for you.
We look into web console tooling that allows you to get insight into your running Apache Camel applications, which has among others visual route diagrams with tracing/debugging and profiling capabilities. In addition to the web tooling we will also show you other tools in the making.
Overview of kubernetes network functionsHungWei Chiu
In this slides, I briefly introduce the network function in the kubernetes and explain how kubernetes implement them.
Those function includes the container network interface (CNI) and kubernetes service.
In the last, I introduce the multus CNI which is designed for multiple networks in the container and it's necessary in some use case, such as SDN/NFV/5G
SpringOne Platform 2016
Speakers: Kevin Hoffman; Advisory Solutions Architect, Pivotal & Chris Umbel; Advisory Architect, Pivotal
With the advent of ASP.NET Core, developers can now build cross-platform microservices in .NET. We can build services on the Mac, Windows, or Linux and deploy anywhere--most importantly to the cloud.
In this session we'll talk about Cloud Native .NET, building .NET microservices, and deploying them to the cloud. We'll build services that participate in a robust ecosystem by consuming OSS servers such as Spring Cloud Configuration Server and Eureka. We'll also show how these .NET microservices can take advantage of circuit breakers and be automatically deployed to the cloud via CI/CD pipelines.
Apache Kafka vs RabbitMQ: Fit For Purpose / Decision TreeSlim Baltagi
Kafka as a streaming data platform is becoming the successor to traditional messaging systems such as RabbitMQ. Nevertheless, there are still some use cases where they could be a good fit. This one single slide tries to answer in a concise and unbiased way where to use Apache Kafka and where to use RabbitMQ. Your comments and feedback are much appreciated.
An introduction to KrakenD, the ultra-high performance API Gateway with middlewares. An opensource tool built using go that is currently serving traffic in major european sites.
Easy enterprise application integration with RabbitMQ and AMQPRabbit MQ
VMware vFabric RabbitMQ Technical Webinar December 2010 by VMware engineer Emile Joubert. Covers common integration patterns, and how RabbitMQ makes these easily implemented, using AMQP as a communications mechanism.
You can view a recording of this presentation on YouTube: http://www.youtube.com/user/SpringSourceDev#p/c/5956C6D9EC319817/0/ABGMjX4K0D8
Architecture | The Future of Messaging: RabbitMQ and AMQP | Eberhard WolffJAX London
2011-11-02 | 05:45 PM - 06:35 PM
The JMS standard is 9 years old - but outside the Java community innovation is happening. The AMQP standard with implementations like RabbitMQ is gaining more and more traction. This session explains the standard and its advantages. It will also show how an AMQP application can be implemented using Java.
Kafka Streams is a new stream processing library natively integrated with Kafka. It has a very low barrier to entry, easy operationalization, and a natural DSL for writing stream processing applications. As such it is the most convenient yet scalable option to analyze, transform, or otherwise process data that is backed by Kafka. We will provide the audience with an overview of Kafka Streams including its design and API, typical use cases, code examples, and an outlook of its upcoming roadmap. We will also compare Kafka Streams' light-weight library approach with heavier, framework-based tools such as Spark Streaming or Storm, which require you to understand and operate a whole different infrastructure for processing real-time data in Kafka.
We start with an introduction to what Apache Camel is, and how you can use Camel to make integration much easier. Allowing you to focus on your business logic, rather than low level messaging protocols, and transports. You will also hear what other features Camel provides out of the box, which can make integration much easier for you.
We look into web console tooling that allows you to get insight into your running Apache Camel applications, which has among others visual route diagrams with tracing/debugging and profiling capabilities. In addition to the web tooling we will also show you other tools in the making.
Overview of kubernetes network functionsHungWei Chiu
In this slides, I briefly introduce the network function in the kubernetes and explain how kubernetes implement them.
Those function includes the container network interface (CNI) and kubernetes service.
In the last, I introduce the multus CNI which is designed for multiple networks in the container and it's necessary in some use case, such as SDN/NFV/5G
Practical Message Queuing Using RabbitMQ (PHPem, 3rd July 2014)James Titcumb
RabbitMQ is a message broker - an application that allows communication between applications by way of a message queuing system. In this talk, we’ll set up an RabbitMQ instance, take an intermediate-level look into the technical features it provides and also how you can apply RabbitMQ in your in applications to scale them efficiently.
Introduction to RabbitMQ, Amqp and some messaging patterns.
Easy to follow code examples step by step provided.
Sample code and escenarios can be found at: https://gist.github.com/javierarilos/9348168
Following up from AMQP presentation, this is a more in-depth coverage of RabbitMQ with workshop-style walkthrough, covering various aspects of the system.
LVIV IT Arena - SmartHome using low cost componentsOriol Rius
Summary of automated systems that I have at home and demonstration about how to use MIIMETIQ LITE (Nexiona) with Mosquitto, NodeRED and Homie. With devices like C.H.I.P. and ESP8266.
"Enterprise Messaging with RabbitMQ" by Luis Majano is a comprehensive presentation that explores the concept of enterprise messaging and demonstrates how RabbitMQ, a widely used message broker, can facilitate effective communication within complex software systems. This presentation aims to provide a deep understanding of messaging patterns, architectures, and the benefits of adopting RabbitMQ in enterprise-level applications.
The presentation begins by introducing the fundamental principles of enterprise messaging and its significance in modern software development. It emphasizes the need for scalable, reliable, and decoupled communication between components and systems to ensure seamless operation and flexibility.
Luis Majano, an expert in enterprise software development, then delves into the key features and capabilities of RabbitMQ. He explains how RabbitMQ enables the implementation of various messaging patterns such as publish/subscribe, request/reply, and message routing. The presentation highlights the versatility of RabbitMQ in handling different messaging scenarios and discusses its integration possibilities with different programming languages and frameworks.
Moreover, the presentation addresses crucial concepts related to RabbitMQ, including exchanges, queues, bindings, and message acknowledgment. It explains how these components work together to enable message routing, filtering, and delivery guarantees in complex distributed systems.
Luis Majano also demonstrates the practical implementation of RabbitMQ through code examples and architectural diagrams. He illustrates how to set up RabbitMQ clusters for high availability and scalability, and how to handle common messaging challenges such as message persistence, fault tolerance, and load balancing.
Throughout the presentation, Luis Majano shares best practices, tips, and strategies for designing and implementing robust messaging solutions using RabbitMQ. He discusses message serialization, error handling, monitoring, and performance optimization techniques to ensure optimal message throughput and system responsiveness.
Beyond REST and RPC: Asynchronous Eventing and Messaging PatternsClemens Vasters
In this session you will learn about when and why to use asynchronous communication with and between services, what kind of eventing/messaging infrastructure you can use in the cloud and on the edge, and how to make it all work together.
Internet companies with huge traffic and millions of users have tasks involved that cannot be served in a request. RabbitMQ can process tasks or communication between different app components asynchronously but close to real time.
ITB2019 Multi-language / multi-OS communication using RabbitMQ - Wil de BruinOrtus Solutions, Corp
Our web hosting company is using many different services for managing e-mail, spam-filters, DNS, domain registrations, SSL registrations, ticket systems and more. Some of these services have well defined Web API’s, others can only be managed by simple command line scripts.
In this session we will explain how we tried to automate the various workflows by using a messaging system such as RabbitMQ for communication between our cfml based customer control panel and these services.
Overview for RabbitMQ with AMQP protocol and a discussion for the AMQP model with message properties and publish and consume message
types of exchange
main properties of the queue
Cloud-native approach of implementing message broker using KubeMQ in k8s cluster. This approach is fast from the traditional implementation (using Apache kafka, RabbitMq). This presentation was given in OSCONF 2020 supported by Collanix and Docker Bangalore Community.
To introduce attendees to the basics of messaging queues, their goals and applications from CFML. Messaging enables software applications to connect and scale. Thus, providing applications to connect to each other as components of a larger application, or to user devices and data. Messaging is asynchronous, and can decouple your software concerns with ease. However, messaging is much more than the traditional publish/subscribe patterns but also the ability to create work queues, routing and much more.
Message queuing is becoming an essential part of modern architectures and essential for asynchronous architectures and microservices. In this session will be described the benefits of messaging systems, the software solutions that are available and typical messaging architectures. Examples will be made using Azure Storage Queues, Azure Service Bus and RabbitMQ. This presentation is primarily about messaging, however as this session is for tech hipsters, the demos will be done giving an extensive introduction to Azure functions, Azure Resource Manager Templates, .Net core and Docker.
Velocity Conference '13: Asynchronous messaging for performance optimization,...Al Sargent
How do Google, Twitter, and Instagram ensure fast application performance at scale? One technique is asynchronous messaging using RabbitMQ to prevent application bottlenecks. In this session, we’ll cover common asynchronous messaging patterns and how to implement them in RabbitMQ, common pitfalls to avoid, and how to cluster RabbitMQ for increased scalability and reliability.
Building Reliability - The Realities of ObservabilityAll Things Open
Presented at the ATO RTP Meetup
Presented by Jeremy Proffit, Director of DevSecOps & SRE for Customer Care and Communications, Ally
Title: Building Reliability - The Realities of Observability
Abstract: Join me as we discuss true observability, learn what works and what doesn't. We'll not only discuss dashboards, monitoring and alerting, but how these can be built by automation or included in your IAC modules. We'll talk about how to properly alert staff based on priority to keep your staff and yourself sane. And even discuss architecture and how it impacts reliably and why serverless isn't always the best at being reliable.
Presented at the ATO RTP Meetup
Presented by Peter Zaitsev, Founder of Percona
Title: Modern Database Best Practices
Abstract: There are now more Database choices available for developers than ever before - there are general purpose databases and specialized databases, single node and distributed databases, Open Source, Proprietary databases and databases available exclusively in the cloud. In this presentation we will cover the best practices of choosing database(s) for your applications, best practices as it comes to application development as well as managing those databases to achieve best possible performance, security, availability at the lowest cost.
All Things Open 2023
Presented at All Things Open 2023
Presented by Deb Bryant - Open Source Initiative, Patrick Masson - Apereo Foundation, Stephen Jacobs - Rochester Institute of Technology, Ruth Suehle - SAS, & Greg Wallace - FreeBSD Foundation
Title: Open Source and Public Policy
Abstract: New regulations in the software industry and adjacent areas such as AI, open science, open data, and open education are on the rise around the world. Cyber Security, societal impact of AI, data and privacy are paramount issues for legislators globally. At the same time, the COVID-19 pandemic drove collaborative development to unprecedented levels and took Open Source software, open research, open content and data from mainstream to main stage, creating tension between public benefit and citizen safety and security as legislators struggle to find a balance between open collaboration and protecting citizens.
Historically, the open source software community and foundations supporting its work have not engaged in policy discussions. Moving forward, thoughtful development of these important public policies whilst not harming our complex ecosystems requires an understanding of how our ecosystem operates. Ensuring stakeholders without historic benefit of representation in those discussions becomes paramount to that end.
Please join our open discussion with open policy stakeholders working constructively on current open policy topics. Our panelists will provide a view into how oss foundations and other open domain allies are now rising to this new challenge as well as seizing the opportunity to influence positive changes to the public’s benefit.
Topics: Public Policy, Open Science, Open Education, current legislation in the US and EU, US interest in OSS sustainability, intro to the Open Policy Alliance
Find more info about All Things Open:
On the web: https://www.allthingsopen.org/
Twitter: https://twitter.com/AllThingsOpen
LinkedIn: https://www.linkedin.com/company/all-things-open/
Instagram: https://www.instagram.com/allthingsopen/
Facebook: https://www.facebook.com/AllThingsOpen
Mastodon: https://mastodon.social/@allthingsopen
Threads: https://www.threads.net/@allthingsopen
2023 conference: https://2023.allthingsopen.org/
Weaving Microservices into a Unified GraphQL Schema with graph-quilt - Ashpak...All Things Open
Presented at All Things Open 2023
Presented by Ashpak Shaikh & Lucy Shen - Intuit
Title: Weaving Microservices into a Unified GraphQL Schema with graph-quilt
Abstract: The magic of GraphQL is that it provides data access through a single endpoint—clean and easy. But as the number of GraphQL microservices your tech stack depends on starts to grow, that single-endpoint purpose becomes a new multi-endpoint problem. Ideally, we would have an orchestrator that could aggregate schemas from multiple microservices into a unified GraphQL schema and route the requests to the appropriate microservice.
Enter graph-quilt, an open source Java library that provides recursive schema stitching and Apollo Federation style schema composition. In this talk, we’ll walk through our GraphQL journey and show you how to use graph-quilt to simplify your data orchestration needs. We will also share our open sourced reference implementation of a highly performant graph-quilt gateway currently being used in production here at Intuit, where we’ve had incredible success in scaling the gateway with 50+ microservices and 150+ clients.
Find more info about All Things Open:
On the web: https://www.allthingsopen.org/
Twitter: https://twitter.com/AllThingsOpen
LinkedIn: https://www.linkedin.com/company/all-things-open/
Instagram: https://www.instagram.com/allthingsopen/
Facebook: https://www.facebook.com/AllThingsOpen
Mastodon: https://mastodon.social/@allthingsopen
Threads: https://www.threads.net/@allthingsopen
2023 conference: https://2023.allthingsopen.org/
The State of Passwordless Auth on the Web - Phil NashAll Things Open
Presented at All Things Open 2023
Presented by Phil Nash - Sonar
Title: The State of Passwordless Auth on the Web
Abstract: Can we get rid of passwords yet? They make for a poor user experience and users are notoriously bad with them. The advent of WebAuthn has brought a passwordless world closer, but where do we really stand?
In this talk we'll explore the current user experience of WebAuthn and the requirements a user has to fulfil to authenticate without a password. We'll also explore the fallbacks and safeguards we can use to make the password experience better and more secure. By the end of the session you'll have a vision of how authentication could look in the future and a blueprint for how to build the best auth experience today.
Find more info about All Things Open:
On the web: https://www.allthingsopen.org/
Twitter: https://twitter.com/AllThingsOpen
LinkedIn: https://www.linkedin.com/company/all-things-open/
Instagram: https://www.instagram.com/allthingsopen/
Facebook: https://www.facebook.com/AllThingsOpen
Mastodon: https://mastodon.social/@allthingsopen
Threads: https://www.threads.net/@allthingsopen
2023 conference: https://2023.allthingsopen.org/
Total ReDoS: The dangers of regex in JavaScriptAll Things Open
Presented at All Things Open 2023
Presented by Phil Nash - Sonar
Title: Total ReDoS: The dangers of regex in JavaScript
Abstract: Regular expressions are complicated and can be hard to learn. On top of that, they can also be a security risk; writing the wrong pattern can open your application up to denial of service attacks. One token out of place and you invite in the dreaded ReDoS.
But how can a regular expression cause this? In this talk we’ll track down the patterns that can cause this trouble, explain why they are an issue and propose ways to fix them now and avoid them in the future. Together we’ll demystify these powerful search patterns and keep your application safe from expressions that behave in a way that is anything but regular.
Find more info about All Things Open:
On the web: https://www.allthingsopen.org/
Twitter: https://twitter.com/AllThingsOpen
LinkedIn: https://www.linkedin.com/company/all-things-open/
Instagram: https://www.instagram.com/allthingsopen/
Facebook: https://www.facebook.com/AllThingsOpen
Mastodon: https://mastodon.social/@allthingsopen
Threads: https://www.threads.net/@allthingsopen
2023 conference: https://2023.allthingsopen.org/
What Does Real World Mass Adoption of Decentralized Tech Look Like?All Things Open
Presented at All Things Open 2023
Presented by Karl Mozurkewich - Storj
Title: What Does Real World Mass Adoption of Decentralized Tech Look Like?
Abstract: We delve into the transformative potential of decentralized technology. Beginning with a brief overview of the rise of centralization with the advent of the internet and the counter-shift marked by blockchain we explore the intrinsic characteristics of decentralized and distributed systems, such as trustless operations, peer-to-peer networks, and enterprise application scalability. Various sectors, including finance, supply chains, media and entertainment, data science and cloud infrastructure are on the brink of disruption. The societal implications are vast, with the potential for greater individual empowerment, a greener planet and more viable resource utilization, but concerns about data security persist.
Find more info about All Things Open:
On the web: https://www.allthingsopen.org/
Twitter: https://twitter.com/AllThingsOpen
LinkedIn: https://www.linkedin.com/company/all-things-open/
Instagram: https://www.instagram.com/allthingsopen/
Facebook: https://www.facebook.com/AllThingsOpen
Mastodon: https://mastodon.social/@allthingsopen
Threads: https://www.threads.net/@allthingsopen
2023 conference: https://2023.allthingsopen.org/
Presented at All Things Open 2023
Presented by Anastasia Lalamentik - Kaleido
Title: How to Write & Deploy a Smart Contract
Abstract: In this talk, Anastasia Lalamentik, Full Stack Engineer at Kaleido, will walk through how Ethereum smart contracts work and go over related concepts like gas fees, the Ethereum Virtual Machine (EVM), the block explorer, and the Solidity programming language. This is vital to anyone who wants to build a blockchain app and is a great introduction to blockchain technology for newcomers to the space.
By the end of the talk, attendees will better understand how to:
- Write a simple smart contract
- Deploy their smart contract to an Ethereum test network through the latest tools like Hardhat and the MetaMask wallet
- Test interactions with their deployed smart contract and ensure that everything is working properly
Additionally, participants will get to interact with Anastasia's deployed smart contract at the end of the talk. Anastasia’s past talks have attracted and have been attended by a diverse group of participants with a range of experience in the space.
Find more info about All Things Open:
On the web: https://www.allthingsopen.org/
Twitter: https://twitter.com/AllThingsOpen
LinkedIn: https://www.linkedin.com/company/all-things-open/
Instagram: https://www.instagram.com/allthingsopen/
Facebook: https://www.facebook.com/AllThingsOpen
Mastodon: https://mastodon.social/@allthingsopen
Threads: https://www.threads.net/@allthingsopen
2023 conference: https://2023.allthingsopen.org/
Spinning Your Drones with Cadence Workflows, Apache Kafka and TensorFlowAll Things Open
Presented at All Things Open 2023
Presented by Paul Brebner - Instaclustr (by Spot by NetApp)
Title: Spinning Your Drones with Cadence Workflows, Apache Kafka and TensorFlow
Abstract: In this talk we’ll build a Drone delivery application, and then use it to do some Machine Learning “on the fly”.
In the 1st part of the talk, we'll build a real-time Drone Delivery demonstration application using a combination of two open-source technologies: Uber’s Cadence (for stateful, scheduled, long-running workflows), and Apache Kafka (for fast streaming data).
With up to 2,000 (simulated) drones and deliveries in progress at once this application generates a vast flow of spatio-temporal data.
In the 2nd part of the talk, we'll use this platform to explore Machine Learning (ML) over streaming and drifting Kafka data with TensorFlow to try and predict which shops will be busy in advance.
Find more info about All Things Open:
On the web: https://www.allthingsopen.org/
Twitter: https://twitter.com/AllThingsOpen
LinkedIn: https://www.linkedin.com/company/all-things-open/
Instagram: https://www.instagram.com/allthingsopen/
Facebook: https://www.facebook.com/AllThingsOpen
Mastodon: https://mastodon.social/@allthingsopen
Threads: https://www.threads.net/@allthingsopen
2023 conference: https://2023.allthingsopen.org/
Presented at the All Things Open 2023 Inclusion and Diversity in Open Source Event
Presented by Efraim Marquez-Arreaza - Red Hat
Title: DEI Challenges and Success
Abstract: In today's world, many companies and organizations have Diversity, Equity and Inclusion (DEI) communities. Red Hat Unidos is a DEI community focused on advocating for the Hispanic/Latine community. In this talk, we would like to share our challenges and success during the past 4-years and plans for the future.
Find more info about All Things Open:
On the web: https://www.allthingsopen.org/
Twitter: https://twitter.com/AllThingsOpen
LinkedIn: https://www.linkedin.com/company/all-things-open/
Instagram: https://www.instagram.com/allthingsopen/
Facebook: https://www.facebook.com/AllThingsOpen
Mastodon: https://mastodon.social/@allthingsopen
Threads: https://www.threads.net/@allthingsopen
2023 conference: https://2023.allthingsopen.org/
Presented at All Things Open 2023
Presented by Lydia Cupery - HubSpot
Title: Scaling Web Applications with Background Jobs: Takeaways from Generating a Huge PDF
Abstract: Do you need to perform time-consuming or CPU-intensive processes in your web application but are concerned about performance? That’s where background jobs come in. By offloading resource-intensive tasks to separate worker processes, you can improve the scalability of your web application.
In this talk, I'll share my experience of using background jobs to scale our web application. I'll discuss the challenges my team faced that led us to adopt background jobs. Then, I'll share practical tips on how to design background jobs for CPU-intensive or time-consuming processes, such as generating huge PDFs and batch emailing. I'll wrap up by going over the performance and cost tradeoffs of background jobs.
I'll use Typescript, Express, and Heroku as examples in this talk, but the concepts and best practices that I'll share are applicable to other languages and tools.
Find more info about All Things Open:
On the web: https://www.allthingsopen.org/
Twitter: https://twitter.com/AllThingsOpen
LinkedIn: https://www.linkedin.com/company/all-things-open/
Instagram: https://www.instagram.com/allthingsopen/
Facebook: https://www.facebook.com/AllThingsOpen
Mastodon: https://mastodon.social/@allthingsopen
Threads: https://www.threads.net/@allthingsopen
2023 conference: https://2023.allthingsopen.org/
Presented at All Things Open 2023
Presented by Robert Aboukhalil - CZI
Title: Supercharging tutorials with WebAssembly
Abstract: sandbox.bio is a free platform that features interactive command-line tutorials for bioinformatics. This talk is a deep-dive into how sandbox.bio was built, with a focus on how WebAssembly enabled bringing command-line tools like awk and grep to the web. Although these tools were originally written in C/C++, they all run directly in the browser, thanks to WebAssembly! And since the computations run on each user's computer, this makes the application highly scalable and cost-effective.
Along the way, I'll discuss how WebAssembly works and how to get started using it in your own applications. The talk will also cover more advanced WebAssembly features such as threads and SIMD, and will end with a discussion of WebAssembly's benefits and pitfalls (it's a powerful technology, but it's not always the right tool!).
Find more info about All Things Open:
On the web: https://www.allthingsopen.org/
Twitter: https://twitter.com/AllThingsOpen
LinkedIn: https://www.linkedin.com/company/all-things-open/
Instagram: https://www.instagram.com/allthingsopen/
Facebook: https://www.facebook.com/AllThingsOpen
Mastodon: https://mastodon.social/@allthingsopen
Threads: https://www.threads.net/@allthingsopen
2023 conference: https://2023.allthingsopen.org/
Presented at All Things Open 2023
Presented by K.S. Bhaskar - YottaDB LLC
Title: Using SQL to Find Needles in Haystacks
Abstract: Database journal files capture every update to a database. A database of a few hundred GB can generate GBs worth of journal files every minute at busy times. Troubleshooting and forensices, especially of rare and intermittent problems, such as which process made what update and when, is an exercise of finding needles in haystacks. A similar problem exists with syslogs. A solution is to load the journal files and syslogs into a database, and use SQL to query the database. Bhaskar will present and demonstrate this with a 100% FOSS stack.
Find more info about All Things Open:
On the web: https://www.allthingsopen.org/
Twitter: https://twitter.com/AllThingsOpen
LinkedIn: https://www.linkedin.com/company/all-things-open/
Instagram: https://www.instagram.com/allthingsopen/
Facebook: https://www.facebook.com/AllThingsOpen
Mastodon: https://mastodon.social/@allthingsopen
Threads: https://www.threads.net/@allthingsopen
2023 conference: https://2023.allthingsopen.org/
Configuration Security as a Game of Pursuit InterceptAll Things Open
Presented at All Things Open 2023
Presented by Wes Widner - Automox
Title: Configuration Security as a Game of Pursuit Intercept
Abstract: In this session we will take a look at the emerging field of cloud security posture management and how we can approach the problem space using a class of board games known as pursuit/intercept. Using the game Scotland Yard as a visual illustration we'll explore the cognitive and technical limitations that all CSPM systems face and what you should look for when evaluating the strengths and weakness of CSPM vendors and approaches.
Find more info about All Things Open:
On the web: https://www.allthingsopen.org/
Twitter: https://twitter.com/AllThingsOpen
LinkedIn: https://www.linkedin.com/company/all-things-open/
Instagram: https://www.instagram.com/allthingsopen/
Facebook: https://www.facebook.com/AllThingsOpen
Mastodon: https://mastodon.social/@allthingsopen
Threads: https://www.threads.net/@allthingsopen
2023 conference: https://2023.allthingsopen.org/
Presented at All Things Open 2023
Presented by Carol Huang & Mike Fix - Stripe
Title: Scaling an Open Source Sponsorship Program
Abstract: We already know this: the open-source ecosystem needs further monetary investment from the companies that benefit most from it. Likewise, companies say they want to participate in these initiatives, but find it hard to dedicate resources to open source funding when there isn’t a clear ROI.
This talk discusses how the Open Source Program Office at Stripe built a scalable, sustainable open source sponsorship model that aligns internal company incentives with those of open source maintainers and the community at large. We go over the unique “platformization” of our OSPO that allowed us to create multiple funding models, such as BYOB (Bring Your Own Budget), and share lessons learned from this experience as well as other OSPOs.
Find more info about All Things Open:
On the web: https://www.allthingsopen.org/
Twitter: https://twitter.com/AllThingsOpen
LinkedIn: https://www.linkedin.com/company/all-things-open/
Instagram: https://www.instagram.com/allthingsopen/
Facebook: https://www.facebook.com/AllThingsOpen
Mastodon: https://mastodon.social/@allthingsopen
Threads: https://www.threads.net/@allthingsopen
2023 conference: https://2023.allthingsopen.org/
Build Developer Experience Teams for Open SourceAll Things Open
Presented at All Things Open 2023
Presented by Arundeep Nagaraj - Amazon Web Services (AWS)
Title: Build Developer Experience Teams for Open Source
Abstract: Open Source has become the default strategy for many IT organizations and Enterprises. However, the constant challenge with Open Source leaders of these organizations has been -
How is my product's developer experience?
Is this the right metric to track?
How can I scale my team to support our products better?
How can I add automation to scale redundant workflows?
If my product involves working with developers, how can I scale to the complexity of the requests and reduce Engineering bandwidth?
The challenges within support of open source products continues to magnify depending on the end user persona whether they are consumers or contributors to your product. Consumers utilize your product, SDK's and API's and are blocked with using it or run into issues, whereas contributors are advanced users of your software that understands the codebase to provide a meaningful contribution back to the product.
The answer to the above is to look at Open Source support as a first-class citizen of your corporate support strategy. To employ the right level of developer focused support as opposed to traditional infrastructure based support is key to scale to the amount of developers using your product. Supporting customers in the open involves more than pure support - building customer / developer experiences (DX) in the open (across platforms and communities) that pivots over the ability of your product's users or developers to be focused on the end-to-end value add. This helps with your active developer growth and retention of users.
Key Takeaways:
- IT leaders of Open Source will learn to employ strategies to build a DX team that engages on multiple platforms
- Work on identifying accurate metrics for product and organization
- Innovate on platforms such as Discord to build a bot and a dashboard
- Ability to leverage customer feedback and iterate over the customer success flywheel
- Distinguish between DX and Developer Advocacy (DA)
Find more info about All Things Open:
On the web: https://www.allthingsopen.org/
Twitter: https://twitter.com/AllThingsOpen
LinkedIn: https://www.linkedin.com/company/all-things-open/
Instagram: https://www.instagram.com/allthingsopen/
Facebook: https://www.facebook.com/AllThingsOpen
Mastodon: https://mastodon.social/@allthingsopen
Threads: https://www.threads.net/@allthingsopen
2023 conference: https://2023.allthingsopen.org/
Presented at All Things Open 2023
Presented by Danny McCormick - Google
Title: Deploying Models at Scale with Apache Beam
Abstract: Apache Beam is an open source tool for building distributed scalable data pipelines. This talk will explore how Beam can be used to perform common machine learning tasks, with a heavy focus on running inference at scale. The talk will include a demo component showing how Beam can be used to deploy and update models efficiently on both CPUs and GPUs for inference workloads.
An attendee can expect to leave this talk with a high level understanding of Beam, the challenges of deploying models at scale, and the ability to use Beam to easily parallelize their inference workloads.
Find more info about All Things Open:
On the web: https://www.allthingsopen.org/
Twitter: https://twitter.com/AllThingsOpen
LinkedIn: https://www.linkedin.com/company/all-things-open/
Instagram: https://www.instagram.com/allthingsopen/
Facebook: https://www.facebook.com/AllThingsOpen
Mastodon: https://mastodon.social/@allthingsopen
Threads: https://www.threads.net/@allthingsopen
2023 conference: https://2023.allthingsopen.org/
Sudo – Giving access while staying in controlAll Things Open
Presented at All Things Open 2023
Presented by Peter Czanik - One Identity
Title: Sudo – Giving access while staying in control
Abstract: Sudo is used by millions to control and log administrator access to systems, but using the default configuration only, there are plenty of blind spots. Using the latest features in sudo let you watch some previously blind spots and control access to them. Here are four major new features, which arrived since the 1.9.0 release, allowing you see your blind spots:
- configuring a working directory or chroot within sudo often makes full shell access redundant
- JSON-formatted logs give you more details on events and are easier to act on
- relays in sudo_logsrvd make session recording collection more secure and reliable
- you can log and control sub-commands executed by the command run through sudo
Let us take a closer look at each of these.
Previously, there were quite a few situations where you had to give users full shell access through sudo. Typical examples include when you need to run a command from a given directory, or running commands in a chroot environment. You can now configure the working directory or the chroot directory and give access only to the command the user really needs.
Logging is a central role of sudo, to see who did what on the system. Using JSON-formatted log messages gives you even more information about events. What is even more: structured logs are easier to act on. Setting up alerting for suspicious events is much easier when you have a single parser to configure for any kind of sudo logs. You can collect sudo logs not only by local syslog, but also by using sudo_logsrvd, the same application used to collect session recordings.
Speaking of session recordings: instead of using a single central server, you can now have multiple levels of sudo_logsrvd relays between the client and the final destination. This allows session collection even if the central server is unavailable, providing you with additional security. It also makes your network configuration simpler.
Finally, you can log sub-commands executed from the command started through sudo. You can see commands started from a shell. No more unnoticed shell access from text editors. Best of all: you can also intercept sub-commands.
These are just a few of the most prominent features helping you to watch and control previous blind spots on your systems. See these and other possibilities in action in some live demos during our presentation.
Find more info about All Things Open:
On the web: https://www.allthingsopen.org/
Twitter: https://twitter.com/AllThingsOpen
LinkedIn: https://www.linkedin.com/company/all-things-open/
Instagram: https://www.instagram.com/allthingsopen/
Facebook: https://www.facebook.com/AllThingsOpen
Mastodon: https://mastodon.social/@allthingsopen
Threads: https://www.threads.net/@allthingsopen
2023 conference: https://2023.allthingsopen.org/
Fortifying the Future: Tackling Security Challenges in AI/ML ApplicationsAll Things Open
Presented at All Things Open 2023
Presented by Christine Abernathy - F5, Inc.
Title: Fortifying the Future: Tackling Security Challenges in AI/ML Applications
Abstract: As Artificial Intelligence (AI) and Machine Learning (ML) applications continue to surge, it is crucial to be aware of and address the security risks associated with these technologies. In this talk, Christine will explore AI/ML failure modes, threats, and mitigation strategies. She will guide you through the fundamentals of ML models then introduce you to key security challenges such as adversarial attacks, data poisoning, model inversion, model stealing, and membership inference attacks, using real-world examples to demonstrate their potential impact.
Christine will also discuss privacy and ethical considerations in ML, touching upon techniques like federated learning and shedding light on the current regulatory landscape surrounding security risks. If you are developing AI/ML applications or incorporating AI/ML components into your technology stack, check out this talk. You will walk away with a deeper understanding of the current AI/ML security landscape and a toolkit to help you address these risks, enabling you to build safer, more secure, and privacy-aware applications.
Find more info about All Things Open:
On the web: https://www.allthingsopen.org/
Twitter: https://twitter.com/AllThingsOpen
LinkedIn: https://www.linkedin.com/company/all-things-open/
Instagram: https://www.instagram.com/allthingsopen/
Facebook: https://www.facebook.com/AllThingsOpen
Mastodon: https://mastodon.social/@allthingsopen
Threads: https://www.threads.net/@allthingsopen
2023 conference: https://2023.allthingsopen.org/
Securing Cloud Resources Deployed with Control Planes on Kubernetes using Gov...All Things Open
Presented at All Things Open 2023
Presented by Carlos Santana - AWS
Title: Securing Cloud Resources Deployed with Control Planes on Kubernetes using Governance and Policy as Code
Abstract: Are you concerned about the security of your cloud resources deployed on Kubernetes? Are you struggling to ensure compliance with regulatory requirements while managing your cloud infrastructure? If yes, then this talk is for you!
We will discuss how to secure cloud resources deployed with Crossplane on Kubernetes using Governance and Policy as Code. We will explore how to leverage Governance and Policy as Code tools like Rego, Kyverno, and OPA to ensure security and compliance.
By the end of this talk, you will have a better understanding of the challenges associated with securing cloud resources deployed with Crossplane or ACK on Kubernetes, the importance of Governance and Policy as Code in ensuring security and compliance, and why it is critical to use open source and open standards in these technologies.
Find more info about All Things Open:
On the web: https://www.allthingsopen.org/
Twitter: https://twitter.com/AllThingsOpen
LinkedIn: https://www.linkedin.com/company/all-things-open/
Instagram: https://www.instagram.com/allthingsopen/
Facebook: https://www.facebook.com/AllThingsOpen
Mastodon: https://mastodon.social/@allthingsopen
Threads: https://www.threads.net/@allthingsopen
2023 conference: https://2023.allthingsopen.org/
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...UiPathCommunity
💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™:
See how to accelerate model training and optimize model performance with active learning
Learn about the latest enhancements to out-of-the-box document processing – with little to no training required
Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages
This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath.
Speakers:
👨🏫 Andras Palfi, Senior Product Manager, UiPath
👩🏫 Lenka Dulovicova, Product Program Manager, UiPath
Securing your Kubernetes cluster_ a step-by-step guide to success !KatiaHIMEUR1
Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster.
However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks.
In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.
SAP Sapphire 2024 - ASUG301 building better apps with SAP Fiori.pdfPeter Spielvogel
Building better applications for business users with SAP Fiori.
• What is SAP Fiori and why it matters to you
• How a better user experience drives measurable business benefits
• How to get started with SAP Fiori today
• How SAP Fiori elements accelerates application development
• How SAP Build Code includes SAP Fiori tools and other generative artificial intelligence capabilities
• How SAP Fiori paves the way for using AI in SAP apps
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdfPaige Cruz
Monitoring and observability aren’t traditionally found in software curriculums and many of us cobble this knowledge together from whatever vendor or ecosystem we were first introduced to and whatever is a part of your current company’s observability stack.
While the dev and ops silo continues to crumble….many organizations still relegate monitoring & observability as the purview of ops, infra and SRE teams. This is a mistake - achieving a highly observable system requires collaboration up and down the stack.
I, a former op, would like to extend an invitation to all application developers to join the observability party will share these foundational concepts to build on:
PHP Frameworks: I want to break free (IPC Berlin 2024)Ralf Eggert
In this presentation, we examine the challenges and limitations of relying too heavily on PHP frameworks in web development. We discuss the history of PHP and its frameworks to understand how this dependence has evolved. The focus will be on providing concrete tips and strategies to reduce reliance on these frameworks, based on real-world examples and practical considerations. The goal is to equip developers with the skills and knowledge to create more flexible and future-proof web applications. We'll explore the importance of maintaining autonomy in a rapidly changing tech landscape and how to make informed decisions in PHP development.
This talk is aimed at encouraging a more independent approach to using PHP frameworks, moving towards a more flexible and future-proof approach to PHP development.
Generative AI Deep Dive: Advancing from Proof of Concept to ProductionAggregage
Join Maher Hanafi, VP of Engineering at Betterworks, in this new session where he'll share a practical framework to transform Gen AI prototypes into impactful products! He'll delve into the complexities of data collection and management, model selection and optimization, and ensuring security, scalability, and responsible use.
Essentials of Automations: Optimizing FME Workflows with ParametersSafe Software
Are you looking to streamline your workflows and boost your projects’ efficiency? Do you find yourself searching for ways to add flexibility and control over your FME workflows? If so, you’re in the right place.
Join us for an insightful dive into the world of FME parameters, a critical element in optimizing workflow efficiency. This webinar marks the beginning of our three-part “Essentials of Automation” series. This first webinar is designed to equip you with the knowledge and skills to utilize parameters effectively: enhancing the flexibility, maintainability, and user control of your FME projects.
Here’s what you’ll gain:
- Essentials of FME Parameters: Understand the pivotal role of parameters, including Reader/Writer, Transformer, User, and FME Flow categories. Discover how they are the key to unlocking automation and optimization within your workflows.
- Practical Applications in FME Form: Delve into key user parameter types including choice, connections, and file URLs. Allow users to control how a workflow runs, making your workflows more reusable. Learn to import values and deliver the best user experience for your workflows while enhancing accuracy.
- Optimization Strategies in FME Flow: Explore the creation and strategic deployment of parameters in FME Flow, including the use of deployment and geometry parameters, to maximize workflow efficiency.
- Pro Tips for Success: Gain insights on parameterizing connections and leveraging new features like Conditional Visibility for clarity and simplicity.
We’ll wrap up with a glimpse into future webinars, followed by a Q&A session to address your specific questions surrounding this topic.
Don’t miss this opportunity to elevate your FME expertise and drive your projects to new heights of efficiency.
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
The Art of the Pitch: WordPress Relationships and Sales
Messaging Standards and Systems - AMQP & RabbitMQ
1. Messaging Standards and Systems
AMQP & RabbitMQ
Gavin M. Roy
VP of Architecture
AWeber Communications
Twitter: @Crad
All Things Open
October 22, 2014
2. About Me
VP of Architecture
AWeber Communications
Blame me for pika, rabbitpy,
pamqp, and a handful of
RabbitMQ plugins
Blog: https://gavinroy.com
Github: https://github.com/gmr
Book: http://manning.com/roy
7. AMQP 0-9-1
❖ Currently has wider support than AMQP 1.0
❖ Multiple Broker Implementations:
RabbitMQ, Apache Qpid, and SwiftMQ to name a few
❖ Specifies a Model and Protocol
8. Native AMQP 0-8/0-9-1 Clients
C
Clojure
Cobol
Common Lisp
Delphi
Erlang
Go
Groovy
Haskell
Java
JavaScript
.NET
OCaml
Perl
PHP
Python
Ruby
Scala
9. ❖ Exchange
Receives and route messages
❖ Message Queue
Stores messages until they can be
consumed
❖ Binding
Defines the relationship between an Exchange and Queue and provides
routing criteria
Advanced Message Queueing Model
X
Exchange
Binding
Message Queue
10. Routing Keys
❖ Provided when publishing a message
❖ Compared against binding keys by exchanges
❖ Ideally provide context to the message:
❖ Connote the type of the message
❖ Categorize the content in the message
❖ Specify the type of consumer that should receive it
11. Advanced Message Queueing Protocol
❖ Compact, binary frame format wire protocol
❖ Bi-directional RPC
❖ Commands consist of Classes and Methods:
❖ Example Request:
Queue.Declare(name=“foo”)
❖ Response:
Queue.DeclareOk(messages=0, consumers=0)
12. Common AMQP Terms
❖ Broker
A server that implements AMQP
❖ Producer or Publisher
A client application that sends messages to a broker
❖ Consumer
A client application that reads messages from a queue
16. AMQP Messages
❖ Comprised of 3 or more frames:
❖ Method Frame
Basic.Publish, Basic.Deliver, etc
❖ Content Header with body size & message properties
timestamp, message-id, app-id, etc
❖ n Body Frames with the opaque message payload
18. RabbitMQ
❖ Open Source (MPL)
❖ Written in Erlang/OTP
❖ Developed/Maintained by Pivotal
❖ Multi-Protocol
AMQP 0-9-1 & 1.0, MQTT, STOMP, XMPP,
HTTP, Web-STOMP & More
❖ Roots in AMQP 0-8/0-9-1
19. Who Uses It?
Agora Games
Chef
Google AdMob
Instagram
MeetMe
Mercado Libre
Mozilla
NASA
New York Times
National Science Foundation
Openstack
Rapportive
Reddit
Soundcloud
(and many more)
20. Why Use RabbitMQ?
❖ Create loosely-coupled applications
❖ Communicate across applications or platforms
❖ Tap into pre-existing message flows for new purposes
❖ Scale-out clustering for growth, throughput, and HA
❖ Federation for WAN latencies and network partitions
❖ Extensible plugin-in architecture
21. RabbitMQ Extensions to AMQP
Authentication Failure
Exchange to Exchange Bindings
Delivery Confirmations
Basic.Nack
Consumer cancellations
Consumer priorities
Dead Letter Exchanges
Alternate Exchanges
Connection blocking
Message CC & BCC Routing
Queue Length Limits
Per Queue Message TTL
Per Message TTL
Queue TTL
Message User ID Validation
Auto-delete exchanges
22. RabbitMQ Clustering
❖ LAN Only
❖ Adds highly-available queues
❖ Is cohesive, publish and consume
from any node
❖ Leverages native Erlang clustering
and communication
❖ Has multiple strategies for dealing
with network partitions
❖ Manually configured via
configuration or command line*
23. RabbitMQ Plugins
auth-backend-amqp
Use AMQP to respond to RabbitMQ
auth requests
auth-backend-http
Use a custom web service to provide
authentication
autocluster-consul
Automatically create clusters using
Consul
influxdb-storage-exchange
Store JSON messages as events in
InfluxDB
presence-exchange
Publishes messages upon binding
changes
rabbitmq-toke
Tokyo Cabinet backing store for queue
messages
rabbitmq-top
Top like view of RabbitMQ processes
in the management UI
sharding
Scale out RabbitMQ with automatic
queue sharding
(and many more)
25. Built-In Exchange Types
❖ Direct
String matching on the routing key
❖ Fanout
No routing key, messages delivered to all bound queues
❖ Topic
Pattern matching in the routing key
❖ Headers
No routing key, value matching in the headers property
26. Topic Exchange Binding Keys
Routing Key: namespace.delimited.keys
#
Receive all messages
namespace.#
Receive all messages in namespace
namespace.delimited.*
Receive all namespace.delimited messages
namespace.*.keys
Receive all namespace messages ending with keys
27. Exchange to Exchange Binding
X
X
X
Queue
Queue
Queue
Queue
Queue
Queue
Messages published into a topic exchange are
routed to two other exchanges and a queue
Messages published into the consistent-hashing
exchange are distributed amongst three queues
Messages published into a direct
exchange routed to two queues
P
29. Dead Letter Exchanges
P X
Reject Queue
Queue C
X
Consumer rejects the
message without requeue
Dead-Letter Exchange
routes the message
30. Exchange Plugins
Consistent Hashing
Distribute messages via hashed value
of routing key
Event
Publishes messages on AMQP events
such as queue creation
Random
Distribute messages across all bound
queues randomly
PostgreSQL LISTEN
Subscribes to and publishes
PostgreSQL notifications
Recent History
Sends the last n messages to any
newly bound queue
Reverse Topic
Allows for routing patterns at publish
time and not via binding
Riak Storage
Stores messages published through
the exchange into Riak
Script Exchange
Calls out to external scripts for
message routing
33. Persisted Messages IO Model
Disk
Queue
Message Queue
Queue
Messages with delivery-mode 2 sent
into RabbitMQ are persisted to disk
Pointers to the message are placed
in the queue data structures
When the message is no longer in
a queue, it is removed from disk
34. HA Queues & Performance
Queue
Queue
Queue
P
RabbitMQ Cluster
A publisher sends to any
node in a RabbitMQ cluster
The RabbitMQ servers in the
cluster synchronize the state
of the message in the queue
The message that was published is put in
the queue and is stored on each server
37. Trending & Monitoring
❖ RabbitMQ Management Plugin provides internal stats
❖ Queue depths, connection counts, throughput, memory
usage, etc
❖ Monitor with common tools such as Nagios or Sensu to
services such as Boundary and NewRelic
❖ Stream based monitoring with Riemann for anomaly
detection
38. Thoughts on Configuration
❖ Use configuration management!
❖ Even for exchanges, queues, and bindings if possible
❖ Helpful for disaster recovery
❖ Use RabbitMQ’s Policies when possible
❖ Exchanges and queues are immutable
❖ Deleting and redeclaring for changes can be
disruptive
44. RPC
C / PC / P
P / C
C / PResponse Queue
X
X
Worker
Queue
Front-end web app
receives a request
Internet
App publishes RPC request
One of the subscribed
worker consumers receives
the RPC request
After processing the request,
the worker publishes the replyApp receives the reply
from the response queue
App replies to client
45. Delayed Messages
WARNING: This pattern is a hack and can cause undesirable situations should the wait queue overwhelm the system!
P X
X
Wait Queue
Ready Queue C
The wait queue is declared with a
message TTL and a dead-letter exchangeMessages that will be delivered
later are routed to a wait queue
When a message times out due to
the TTL, it is published to the DLX
The internally re-published message is
routed to the queue with active consumers