ØMQ is a messaging library that provides asynchronous message passing between applications using in-memory queues. It works on top of several transports including TCP, IPC, in-memory, and multicast protocols. ØMQ aims for horizontal scalability, network topology awareness, and simplicity in communication patterns. It can be installed from source or packages and has bindings for many languages including Node.js. ØMQ supports common socket types for request/reply, publish/subscribe, and asynchronous messaging patterns.
Choosing a communication platform is an important decision. From simple two-way communication to complex multi-node architectures, ZeroMQ, the embeddable networking library, helps provide a safe, fast and reliable communication medium.
This webinar will give you an overview of the ZeroMQ architecture, explaining the advantages and exploring usage patterns and cross-platform capabilities. We'll also go through examples of the patterns using different languages, including C++, Swift, Python and C.
It describes the MMC storage device driver functionality in Linux Kernel and it's role. It explains different type of storage devices available and how they are handled from MMC driver point of view. It describes eMMC (internal storage) device and SD (external storage) devices in details and SD protocol used for communicating with these devices in Linux.
This presentation features a walk through the Linux kernel networking stack covering the essentials and recent developments a developer needs to know. Our starting point is the network card driver as it feeds a packet into the stack. We will follow the packet as it traverses through various subsystems such as packet filtering, routing, protocol stacks, and the socket layer. We will pause here and there to look into concepts such as segmentation offloading, TCP small queues, and low latency polling. We will cover APIs exposed by the kernel that go beyond use of write()/read() on sockets and will look into how they are implemented on the kernel side.
High-Performance Networking Using eBPF, XDP, and io_uringScyllaDB
In the networking world there are a number of ways to increase performance over naive use of basic Berkeley sockets. These techniques have ranged from polling blocking sockets, non-blocking sockets controlled by Epoll, all the way through completely bypassing the Linux kernel for maximum network performance where you talk directly to the network interface card by using something like DPDK or Netmap. All these tools have their place, and generally occupy a space from convenience to performance. But in recent years, that landscape has changed massively.. The tools available to the average Linux systems developer have improved from the creation of io_uring, to the expansion of bpf from a simple filtering language to a full-on programming environment embedded directly in the kernel. Along with that came something called XDP (express datapath). This was Linux kernel's answer to kernel-bypass networking. AF_XDP is the new socket type created by this feature, and generally works very similarly to something like DPDK. History lessons out of the way, this talk will look into, and discuss the merits of this technology, it's place in the broader ecosystem and how it can be used to attain the highest level of performance possible. This talk will dive into crucial details, such as how AF_XDP works, how it can be integrated into a larger system and finally more advanced topics such as request sharding/load balancing. There will be detailed look at the design of AF_XDP, the eBpf code used, as well as the userspace code required to drive it all. It will also include performance numbers from this setup compared to regular kernel networking. And most importantly how to put all this together to handle as much data as possible on a single modern multi-core system.
The landscape for storing your big data is quite complex, with several competing formats and different implementations of each format. Understanding your use of the data is critical for picking the format. Depending on your use case, the different formats perform very differently. Although you can use a hammer to drive a screw, it isn’t fast or easy to do so.
The use cases that we’ve examined are:
* reading all of the columns
* reading a few of the columns
* filtering using a filter predicate
* writing the data
Furthermore, different kinds of data have distinct properties. We've used three real schemas:
* the NYC taxi data http://tinyurl.com/nyc-taxi-analysis
* the Github access logs http://githubarchive.org
* a typical sales fact table with generated data
Finally, the value of having open source benchmarks that are available to all interested parties is hugely important and all of the code is available from Apache.
Sphinx autodoc - automated api documentation - PyCon.KR 2015Takayuki Shimizukawa
Using the automated documentation feature of Sphinx, you can make with ease the extensive documentation of Python program.
You just write python function documents (docstrings), Sphinx organizes them into the document, can be converted to a variety of formats.
In this session, I'll explain a documentation procedure that uses with sphinx autodoc and autosummary extensions.
Introduction to Apache Spark Developer TrainingCloudera, Inc.
Apache Spark is a next-generation processing engine optimized for speed, ease of use, and advanced analytics well beyond batch. The Spark framework supports streaming data and complex, iterative algorithms, enabling applications to run 100x faster than traditional MapReduce programs. With Spark, developers can write sophisticated parallel applications for faster business decisions and better user outcomes, applied to a wide variety of architectures and industries.
Learn What Apache Spark is and how it compares to Hadoop MapReduce, How to filter, map, reduce, and save Resilient Distributed Datasets (RDDs), Who is best suited to attend the course and what prior knowledge you should have, and the benefits of building Spark applications as part of an enterprise data hub.
Cgroups, namespaces, and beyond: what are containers made from? (DockerCon Eu...Jérôme Petazzoni
Linux containers are different from Solaris Zones or BSD Jails: they use discrete kernel features like cgroups, namespaces, SELinux, and more. We will describe those mechanisms in depth, as well as demo how to put them together to produce a container. We will also highlight how different container runtimes compare to each other.
This talk was delivered at DockerCon Europe 2015 in Barcelona.
Linux offers an extensive selection of programmable and configurable networking components from traditional bridges, encryption, to container optimized layer 2/3 devices, link aggregation, tunneling, several classification and filtering languages all the way up to full SDN components. This talk will provide an overview of many Linux networking components covering the Linux bridge, IPVLAN, MACVLAN, MACVTAP, Bonding/Team, OVS, classification & queueing, tunnel types, hidden routing tricks, IPSec, VTI, VRF and many others.
Ever tried to get get clarity on what kinds of memory there are and how to tune each of them ? If not, very likely your jobs are configured incorrectly. As we found out, its is not straightforward and it is not well documented either. This session will provide information on the types of memory to be aware of, the calculations involved in determining how much is allocated to each type of memory and how to tune it depending on the use case.
Building a redundant CloudStack management cluster - Vladimir MelnikShapeBlue
Building a redundant CloudStack management cluster. Building and maintaining an open-source-driven clustered environment for Apache CloudStack management server with GNU Linux, HAProxy, HeartBeat, Bind, OpenLDAP and other tools.
LinuxCon 2015 Linux Kernel Networking WalkthroughThomas Graf
This presentation features a walk through the Linux kernel networking stack for users and developers. It will cover insights into both, existing essential networking features and recent developments and will show how to use them properly. Our starting point is the network card driver as it feeds a packet into the stack. We will follow the packet as it traverses through various subsystems such as packet filtering, routing, protocol stacks, and the socket layer. We will pause here and there to look into concepts such as networking namespaces, segmentation offloading, TCP small queues, and low latency polling and will discuss how to configure them.
Choosing a communication platform is an important decision. From simple two-way communication to complex multi-node architectures, ZeroMQ, the embeddable networking library, helps provide a safe, fast and reliable communication medium.
This webinar will give you an overview of the ZeroMQ architecture, explaining the advantages and exploring usage patterns and cross-platform capabilities. We'll also go through examples of the patterns using different languages, including C++, Swift, Python and C.
It describes the MMC storage device driver functionality in Linux Kernel and it's role. It explains different type of storage devices available and how they are handled from MMC driver point of view. It describes eMMC (internal storage) device and SD (external storage) devices in details and SD protocol used for communicating with these devices in Linux.
This presentation features a walk through the Linux kernel networking stack covering the essentials and recent developments a developer needs to know. Our starting point is the network card driver as it feeds a packet into the stack. We will follow the packet as it traverses through various subsystems such as packet filtering, routing, protocol stacks, and the socket layer. We will pause here and there to look into concepts such as segmentation offloading, TCP small queues, and low latency polling. We will cover APIs exposed by the kernel that go beyond use of write()/read() on sockets and will look into how they are implemented on the kernel side.
High-Performance Networking Using eBPF, XDP, and io_uringScyllaDB
In the networking world there are a number of ways to increase performance over naive use of basic Berkeley sockets. These techniques have ranged from polling blocking sockets, non-blocking sockets controlled by Epoll, all the way through completely bypassing the Linux kernel for maximum network performance where you talk directly to the network interface card by using something like DPDK or Netmap. All these tools have their place, and generally occupy a space from convenience to performance. But in recent years, that landscape has changed massively.. The tools available to the average Linux systems developer have improved from the creation of io_uring, to the expansion of bpf from a simple filtering language to a full-on programming environment embedded directly in the kernel. Along with that came something called XDP (express datapath). This was Linux kernel's answer to kernel-bypass networking. AF_XDP is the new socket type created by this feature, and generally works very similarly to something like DPDK. History lessons out of the way, this talk will look into, and discuss the merits of this technology, it's place in the broader ecosystem and how it can be used to attain the highest level of performance possible. This talk will dive into crucial details, such as how AF_XDP works, how it can be integrated into a larger system and finally more advanced topics such as request sharding/load balancing. There will be detailed look at the design of AF_XDP, the eBpf code used, as well as the userspace code required to drive it all. It will also include performance numbers from this setup compared to regular kernel networking. And most importantly how to put all this together to handle as much data as possible on a single modern multi-core system.
The landscape for storing your big data is quite complex, with several competing formats and different implementations of each format. Understanding your use of the data is critical for picking the format. Depending on your use case, the different formats perform very differently. Although you can use a hammer to drive a screw, it isn’t fast or easy to do so.
The use cases that we’ve examined are:
* reading all of the columns
* reading a few of the columns
* filtering using a filter predicate
* writing the data
Furthermore, different kinds of data have distinct properties. We've used three real schemas:
* the NYC taxi data http://tinyurl.com/nyc-taxi-analysis
* the Github access logs http://githubarchive.org
* a typical sales fact table with generated data
Finally, the value of having open source benchmarks that are available to all interested parties is hugely important and all of the code is available from Apache.
Sphinx autodoc - automated api documentation - PyCon.KR 2015Takayuki Shimizukawa
Using the automated documentation feature of Sphinx, you can make with ease the extensive documentation of Python program.
You just write python function documents (docstrings), Sphinx organizes them into the document, can be converted to a variety of formats.
In this session, I'll explain a documentation procedure that uses with sphinx autodoc and autosummary extensions.
Introduction to Apache Spark Developer TrainingCloudera, Inc.
Apache Spark is a next-generation processing engine optimized for speed, ease of use, and advanced analytics well beyond batch. The Spark framework supports streaming data and complex, iterative algorithms, enabling applications to run 100x faster than traditional MapReduce programs. With Spark, developers can write sophisticated parallel applications for faster business decisions and better user outcomes, applied to a wide variety of architectures and industries.
Learn What Apache Spark is and how it compares to Hadoop MapReduce, How to filter, map, reduce, and save Resilient Distributed Datasets (RDDs), Who is best suited to attend the course and what prior knowledge you should have, and the benefits of building Spark applications as part of an enterprise data hub.
Cgroups, namespaces, and beyond: what are containers made from? (DockerCon Eu...Jérôme Petazzoni
Linux containers are different from Solaris Zones or BSD Jails: they use discrete kernel features like cgroups, namespaces, SELinux, and more. We will describe those mechanisms in depth, as well as demo how to put them together to produce a container. We will also highlight how different container runtimes compare to each other.
This talk was delivered at DockerCon Europe 2015 in Barcelona.
Linux offers an extensive selection of programmable and configurable networking components from traditional bridges, encryption, to container optimized layer 2/3 devices, link aggregation, tunneling, several classification and filtering languages all the way up to full SDN components. This talk will provide an overview of many Linux networking components covering the Linux bridge, IPVLAN, MACVLAN, MACVTAP, Bonding/Team, OVS, classification & queueing, tunnel types, hidden routing tricks, IPSec, VTI, VRF and many others.
Ever tried to get get clarity on what kinds of memory there are and how to tune each of them ? If not, very likely your jobs are configured incorrectly. As we found out, its is not straightforward and it is not well documented either. This session will provide information on the types of memory to be aware of, the calculations involved in determining how much is allocated to each type of memory and how to tune it depending on the use case.
Building a redundant CloudStack management cluster - Vladimir MelnikShapeBlue
Building a redundant CloudStack management cluster. Building and maintaining an open-source-driven clustered environment for Apache CloudStack management server with GNU Linux, HAProxy, HeartBeat, Bind, OpenLDAP and other tools.
LinuxCon 2015 Linux Kernel Networking WalkthroughThomas Graf
This presentation features a walk through the Linux kernel networking stack for users and developers. It will cover insights into both, existing essential networking features and recent developments and will show how to use them properly. Our starting point is the network card driver as it feeds a packet into the stack. We will follow the packet as it traverses through various subsystems such as packet filtering, routing, protocol stacks, and the socket layer. We will pause here and there to look into concepts such as networking namespaces, segmentation offloading, TCP small queues, and low latency polling and will discuss how to configure them.
The Department of Environment has approved this faulty EIA submitted by the Power Development Board. The project would be implemented by the governments of Bangladesh and India.
Anatomy of Brain by MRI
In this presentation we will discuss the cross sectional anatomy of brain. Then we will discuss the Most common diseases to be evaluated by brain imaging.
In my opinion this presentation is a road map for beginars.
Overview on the Toyota production system principles, techniques and theories.
The presentation include:
-Lean Manufacturing Principles
-Productivity Measurement, Analysis and Improvement
-Effect & Elimination of the Manufacturing 7-Wastes
-Lean Improvement Techniques
-Lean Management for Making Improvement & Gaining Sustainability
Principles and Practices in Continuous Deployment at EtsyMike Brittain
Presented at ALM Forum 2014.
Like what you've read? We're frequently hiring for a variety of engineering roles at Etsy. If you're interested, drop me a line or send me your resume: mike@etsy.com.
http://www.etsy.com/careers
ZMQ socket based networking library, which can also be molded as broker-less messaging framework. The slides talks about how zmq been woven to work with openstack services.
Cotopaxi - IoT testing toolkit (Black Hat Asia 2019 Arsenal)Jakub Botwicz
Presentation about Cotopaxi toolkit from Black Hat Asia 2019 Arsenal session. Author: Jakub Botwicz
https://www.blackhat.com/asia-19/arsenal/schedule/index.html#cotopaxi-iot-protocols-security-testing-toolkit-14325
IBM MQ systems route billions of messages around the world each day. This presentation looks at the tools available in MQ for z/OS to allow you to understand where your messages are flowing, and things you can use if the messages aren't going where you expect.
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
Welcome to the first live UiPath Community Day Dubai! Join us for this unique occasion to meet our local and global UiPath Community and leaders. You will get a full view of the MEA region's automation landscape and the AI Powered automation technology capabilities of UiPath. Also, hosted by our local partners Marc Ellis, you will enjoy a half-day packed with industry insights and automation peers networking.
📕 Curious on our agenda? Wait no more!
10:00 Welcome note - UiPath Community in Dubai
Lovely Sinha, UiPath Community Chapter Leader, UiPath MVPx3, Hyper-automation Consultant, First Abu Dhabi Bank
10:20 A UiPath cross-region MEA overview
Ashraf El Zarka, VP and Managing Director MEA, UiPath
10:35: Customer Success Journey
Deepthi Deepak, Head of Intelligent Automation CoE, First Abu Dhabi Bank
11:15 The UiPath approach to GenAI with our three principles: improve accuracy, supercharge productivity, and automate more
Boris Krumrey, Global VP, Automation Innovation, UiPath
12:15 To discover how Marc Ellis leverages tech-driven solutions in recruitment and managed services.
Brendan Lingam, Director of Sales and Business Development, Marc Ellis
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...UiPathCommunity
💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™:
See how to accelerate model training and optimize model performance with active learning
Learn about the latest enhancements to out-of-the-box document processing – with little to no training required
Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages
This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath.
Speakers:
👨🏫 Andras Palfi, Senior Product Manager, UiPath
👩🏫 Lenka Dulovicova, Product Program Manager, UiPath
Essentials of Automations: The Art of Triggers and Actions in FMESafe Software
In this second installment of our Essentials of Automations webinar series, we’ll explore the landscape of triggers and actions, guiding you through the nuances of authoring and adapting workspaces for seamless automations. Gain an understanding of the full spectrum of triggers and actions available in FME, empowering you to enhance your workspaces for efficient automation.
We’ll kick things off by showcasing the most commonly used event-based triggers, introducing you to various automation workflows like manual triggers, schedules, directory watchers, and more. Plus, see how these elements play out in real scenarios.
Whether you’re tweaking your current setup or building from the ground up, this session will arm you with the tools and insights needed to transform your FME usage into a powerhouse of productivity. Join us to discover effective strategies that simplify complex processes, enhancing your productivity and transforming your data management practices with FME. Let’s turn complexity into clarity and make your workspaces work wonders!
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
zkStudyClub - Reef: Fast Succinct Non-Interactive Zero-Knowledge Regex ProofsAlex Pruden
This paper presents Reef, a system for generating publicly verifiable succinct non-interactive zero-knowledge proofs that a committed document matches or does not match a regular expression. We describe applications such as proving the strength of passwords, the provenance of email despite redactions, the validity of oblivious DNS queries, and the existence of mutations in DNA. Reef supports the Perl Compatible Regular Expression syntax, including wildcards, alternation, ranges, capture groups, Kleene star, negations, and lookarounds. Reef introduces a new type of automata, Skipping Alternating Finite Automata (SAFA), that skips irrelevant parts of a document when producing proofs without undermining soundness, and instantiates SAFA with a lookup argument. Our experimental evaluation confirms that Reef can generate proofs for documents with 32M characters; the proofs are small and cheap to verify (under a second).
Paper: https://eprint.iacr.org/2023/1886
Enhancing Performance with Globus and the Science DMZGlobus
ESnet has led the way in helping national facilities—and many other institutions in the research community—configure Science DMZs and troubleshoot network issues to maximize data transfer performance. In this talk we will present a summary of approaches and tips for getting the most out of your network infrastructure using Globus Connect Server.
Elevating Tactical DDD Patterns Through Object CalisthenicsDorra BARTAGUIZ
After immersing yourself in the blue book and its red counterpart, attending DDD-focused conferences, and applying tactical patterns, you're left with a crucial question: How do I ensure my design is effective? Tactical patterns within Domain-Driven Design (DDD) serve as guiding principles for creating clear and manageable domain models. However, achieving success with these patterns requires additional guidance. Interestingly, we've observed that a set of constraints initially designed for training purposes remarkably aligns with effective pattern implementation, offering a more ‘mechanical’ approach. Let's explore together how Object Calisthenics can elevate the design of your tactical DDD patterns, offering concrete help for those venturing into DDD for the first time!
In his public lecture, Christian Timmerer provides insights into the fascinating history of video streaming, starting from its humble beginnings before YouTube to the groundbreaking technologies that now dominate platforms like Netflix and ORF ON. Timmerer also presents provocative contributions of his own that have significantly influenced the industry. He concludes by looking at future challenges and invites the audience to join in a discussion.
PHP Frameworks: I want to break free (IPC Berlin 2024)Ralf Eggert
In this presentation, we examine the challenges and limitations of relying too heavily on PHP frameworks in web development. We discuss the history of PHP and its frameworks to understand how this dependence has evolved. The focus will be on providing concrete tips and strategies to reduce reliance on these frameworks, based on real-world examples and practical considerations. The goal is to equip developers with the skills and knowledge to create more flexible and future-proof web applications. We'll explore the importance of maintaining autonomy in a rapidly changing tech landscape and how to make informed decisions in PHP development.
This talk is aimed at encouraging a more independent approach to using PHP frameworks, moving towards a more flexible and future-proof approach to PHP development.
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...SOFTTECHHUB
The choice of an operating system plays a pivotal role in shaping our computing experience. For decades, Microsoft's Windows has dominated the market, offering a familiar and widely adopted platform for personal and professional use. However, as technological advancements continue to push the boundaries of innovation, alternative operating systems have emerged, challenging the status quo and offering users a fresh perspective on computing.
One such alternative that has garnered significant attention and acclaim is Nitrux Linux 3.5.0, a sleek, powerful, and user-friendly Linux distribution that promises to redefine the way we interact with our devices. With its focus on performance, security, and customization, Nitrux Linux presents a compelling case for those seeking to break free from the constraints of proprietary software and embrace the freedom and flexibility of open-source computing.
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
3. Facts about ØMQ
Low level messaging library
○ It is written in C++.
○ A thin layer between application and transport
layers.
○ Message-passing using in-memory queues.
○ Concise API which talks over the ZMTP protocol.
○ Works on top of TCP, IPC, in-memory, and
PGM/EPGM communication protocols.
4. ○ Philosophy: horizontal scalability to reduce points
of failure.
○ Network topology aware. Point to point
communication reduce latencies.
○ Communication is reduced to a few patterns, which
combined are powerful.
○ No need to have a multitenant service.
Facts about ØMQ
A broker is not mandatory
5. Facts about ØMQ
Messaged oriented communication
○ Messages as first-class citizen. No need to fight
with framing or buffering.
○ You can send multipart messages.
○ Messages are atomic.
○ Asynchronous messaging. A single background I/O
thread does the job for ØMQ sockets.
"A ØMQ message is a discrete unit of data passed between
applications or components of the same application. ØMQ messages
have no internal structure and from the point of view of ØMQ itself
they are considered to be opaque binary data."
6. Facts about ØMQ
Fast for development
○ Interchangeable transports: e.g, scaling from one
IPC server to a bunch of TCP.
○ Automatic reconnections.
○ Multicore made easy.
○ Provides zmq_poll to read BSD sockets.
7. ○ It doesn't provide message persistence.
○ It doesn't provide data serialization.
○ It doesn't provide data compression.
○ It doesn't provide message encryption.
○ It doesn't provide security mechanisms. Not in the
next version of the ZMTP protocol !
(*) ZTMP 3.0 draft protocol: http://hintjens.com/blog:39
Facts about ØMQ
Aims for simplicity
8. Facts about ØMQ
Platforms & Languages
○ Bindings for 30+ languages including NodeJS, Ruby,
Python, Java, C, C++, Scala, Erlang, Perl, PHP, .NET.
○ Linux & Windows.
○ Raspberry PI.
○ Android OS.
10. Install
On Linux from source
sudo apt-get install build-essential libtool autoconf automake uuid-dev
wget http://download.zeromq.org/zeromq-3.2.3.tar.gz
tar xvzf zeromq-3.2.3.tar.gz
./configure
make
sudo make install
sudo ldconfig
NOTES:
(*) Check http://download.zeromq.org/ for more versions.
(**) Avoid 3.0.x, 3.1.x, 3.2.0, 3.2.1 versions. They lack backwards
compatibility.
(***) Check ./configure --help for more options on build.
(****) ./configure --with-pgm
20. Sockets
Transports
Inter-Process Communication (IPC)
● usage: ipc://<address>
● Requires RW permissions on the specified path.
(*) http://api.zeromq.org/3-2:zmq-ipc
sock.bind("ipc:///tmp/mysocket");
ØMQ C API
zmq_bind(socket, "ipc:///tmp/mysocket");
NodeJS ZMQ
21. Sockets
Transports
In-process Communication (in-memory)
● usage: inproc://<address>
● Requires to bind before connect.
● Max. address name length is 256.
(*) http://api.zeromq.org/3-2:zmq-inproc
(**) Buggy on node-zmq package.
sock.bind("inproc://queue");
ØMQ C API
zmq_bind(socket, "ipc://queue");
NodeJS ZMQ
22. Sockets
Transports
PGM (multicast)
● usage: pgm://<address>;<multicast_address>:<port>
● Requires sudo privileges to access raw IP sockets.
● Needs ØMQ built with PGM extension (./configure --with-pgm).
(*) http://api.zeromq.org/3-2:zmq-pgm
sock.bind(
"pgm://192.168.1.100;239.192.1.1:3055"
);
ØMQ C API
zmq_bind(socket, "pgm://192.168.1.100;
239.192.1.1:3055");
NodeJS ZMQ
23. Sockets
Transports
Encapsulated PGM (multicast)
● usage: epgm://<address>;<multicast_address>:<port>
● Needs ØMQ built with PGM extension (./configure --with-pgm).
(*) http://api.zeromq.org/3-2:zmq-pgm
sock.bind(
"epgm://192.168.1.100;239.192.1.1:3055"
);
ØMQ C API
zmq_bind(socket, "epgm://192.168.1.100;
239.192.1.1:3055");
NodeJS ZMQ
25. Sockets
Multipart messages
ØMQ Queue
zmq_send(socket, "Como", 4, ZMQ_SNDMORE);
Messages are atomic
A ØMQ message is composed of 1 or more
message parts.
zmq_send(socket, "andas", 5, 0);
4 ComoPart 1
5 andasPart 2
ØMQ ensures that peers receive either all message parts of a message or
none at all.
The total number of message parts is unlimited except by available memory.
28. Sockets
Batched messaging
ØMQ Queue
3 Msg
"ØMQ batches messages in opportunistic manner. It sends all the
messages available at the moment in one go. Latency of subsequent
messages will be improved because sending single batch to the card
is faster then sending lot of small messages."
sock.send("Part A",
zmq.ZMQ_SNDMORE);
sock.send("Msg");
sock.send("Part
B");
6 Part A
6 Part B
ØMQ Socket
send
send
send
<write busy>
<write ready>
Client Thread ØMQ I/O Thread
send
batch
<write busy>
29. Patterns
Plugging multiple transports
Example
var zmq = require('zmq'),
pub = zmq.socket('pub');
pub.bindSync('tcp://127.0.0.1:5555');
pub.bindSync('ipc:///tmp/zmq.sock');
setInterval(function() {
pub.send("I am polyglot!");
}, 1000);
sub = zmq.socket('sub');
sub.connect('ipc:///tmp/zmq.sock');
sub.subscribe('');
sub.on('message', function(msg) {
console.log("Received: " + msg);
});
var zmq = require('zmq'),
sub = zmq.socket(sub');
sub.connect('tcp://127.0.0.1:5555');
sub.subscribe('');
sub.on('message', function(msg) {
console.log(Received: ' + msg);
});
30. Sockets
Setting Socket Options
(*) http://api.zeromq.org/3-2:zmq-setsockopt
Setsockopt
● usage (NodeJS ZMQ API): sock.setsockopt(<option>, <value>);
or sock.<option> = <value>;
sock.identity = 'monitor-1';
sock.setsockopt('identity', 'monitor-1');
Identity:
// value can be any string up to 255 length.
// identity is required on sockets
// connecting to a Router.
33. Patterns
Request / Reply
Synchronous task distribution
Ø REP #1
Ø REP # 2
Ø REQ
< round robin >
...
< synchronous >
Used when each message needs to be matched with a response. Handles
only one message at time. Strict send-recv cycle.
37. Patterns
Request / Reply
Dealer socket message delivery
Messages have to be multipart, consisting on: an empty delimiter header,
followed by the content body parts.
sock.send("Body");
Outgoing Queue
0
6 Body
sock.send(new Buffer([]), zmq.
ZMQ_SNDMORE);
send
send
38. Patterns
Request / Reply
Dealer socket message delivery
Outgoing messages are round-robined among all connected peers.
However, sending depends on the availability of the receiver.
sock.send(
["", "Msg"]);
Outgoing Queue
3 Msg
6 Part A
Peer Socket A Peer Socket B
sock.send(
["", "Bla");
sock.send(
["", "Msg2"]);
send
send
send
To A
To B
To A
send
<write
ready>
<write
busy>
<write
ready>
0
0
4 Msg2
0
39. Patterns
Request / Reply
Dealer socket message delivery
Reconnections are handled automatically: messages are asynchronously
delivered. Response time does not block sending more messages.
Outgoing Queue Peer Socket A Peer Socket B
send
-> Up again
sock.send(
["", "Msg"]);
3 Msg
sock.send(
["", "Msg2"]);
send
send
To A
To A
0
4 Msg2
0
40. Patterns
Request / Reply
Ø REP #1 Ø REP # 2Ø DEALER
enqueue < Msg A >
send < Msg B >
-> REP #1 is down
enqueue < Msg C >
send < Msg A, Msg C >
send < Msg D >
Dealer socket handles peer reconnection automatically. It will send
messages that queued for that peer once it has established connection.
42. Patterns
Request / Reply
Router socket messaging
Messages have to be multipart, consisting on: an identity frame, an empty
delimiter frame, followed by the content body parts. This applies to both
incoming and outgoing messages.
sock.send("Body");
Outgoing Queue
0
6 Body
sock.send("", zmq.
ZMQ_SNDMORE);
send
send
sock.send(
"worker-1", zmq.
ZMQ_SNDMORE); 8 worker-1
send
43. Patterns
Request / Reply
Router is asynchronous
Incoming messages are fair-queued among all connected and available
peers.
Peer Socket A Peer Socket B
ØMQ Router
Incoming Queue
Socketsend
send
send
44. Patterns
Request / Reply
Router is asynchronous
Receiving depends on the availability of the sender.
Peer Socket A Peer Socket B
ØMQ Router
Incoming Queue
Socketsend
send
45. Patterns
Push / Pull
Unidirectional data distribution
The Push / Pull pattern fits well on pipelined communication, i.e., when no
response is required for a given message.
Ø
Ø
...
ØØ
46. Patterns
Push / Pull
Binding the Push socket
Outgoing messages are round-robined among all connected and available
peers.
push.send("Msg");
push.send("Part
B");
Outgoing Queue Peer Socket A Peer Socket B
push.send("Part A",
zmq.ZMQ_SNDMORE);
push.send("Msg2");
3 Msg
6 Part A
6 Part B
4 Msg2
send
send
send
send
47. Patterns
Push / Pull
Binding the Push socket
Outgoing messages are round-robined among all connected and available
peers.
Outgoing Queue Peer Socket A Peer Socket B
-> Down3 Msg
6 Part A
6 Part B
4 Msg2
send
send
send
push.send("Msg");
push.send("Part
B");
push.send("Part A",
zmq.ZMQ_SNDMORE);
push.send("Msg2");
send
send
send
send
48. Patterns
Push / Pull
Binding the Push socket
Disconnections and reconnections are handled automatically.
Outgoing Queue Peer Socket A Peer Socket B
4 More
6 Part 1
6 Part 2
5 Other
send
send
send
push.send("More"); send
push.send("Part
2");
push.send("Part 1",
zmq.ZMQ_SNDMORE);
send
send
push.send("Other");
send
51. Patterns
Push / Pull
Binding the Pull socket
Push's outgoing messages are round-robined among all connected peers.
push.send("Msg");
push.send("Part
B");
Outgoing Queue Peer Socket A Peer Socket B
push.send("Part A",
zmq.ZMQ_SNDMORE);
push.send("Msg2");
send
send
send
send
3 Msg
6 Part A
6 Part B
4 Msg2
To A
To B
To A
52. Patterns
Push / Pull
Binding the Pull socket
Push's outgoing messages are round-robined among all connected peers.
push.send("Msg");
push.send("Part
B");
Outgoing Queue Peer Socket A Peer Socket B
push.send("Part A",
zmq.ZMQ_SNDMORE);
push.send("Msg2");
send
send
send
send
send
send
send
3 Msg
6 Part A
6 Part B
4 Msg2
To A
To B
To A
53. Patterns
Push / Pull
Binding the Pull socket
Disconnections and reconnections are handled automatically.
Outgoing Queue Peer Socket A Peer Socket B
send
send
-> Down4 More
6 Part 1
6 Part 2
5 Other
To A
To A
To B
push.send("More"); send
push.send("Part
2");
push.send("Part 1",
zmq.ZMQ_SNDMORE);
send
send
push.send("Other");
send
54. Patterns
Push / Pull
Binding the Pull socket
Sending is asynchronous, it depends on the availability of the receiver.
Outgoing Queue Peer Socket A Peer Socket B
send
-> Up again
5 OtherTo B
push.send("More"); send
push.send("Part
2");
push.send("Part 1",
zmq.ZMQ_SNDMORE);
send
send
push.send("Other");
send
57. Message distribution
Patterns
Publish / Subscribe
sock.send("Msg");
Outgoing Queue
3 Msg
Subscriber A
Subscriber B
send
send
Outgoing messages are send to each connected and available peers.
Subscriber C
58. Message distribution
Patterns
Publish / Subscribe
sock.send("Msg");
Outgoing Queue
3 Msg
Subscriber A
Subscriber B
send
send
Outgoing messages are send to each connected and available peers.
Subscriber C
5 Othersock.send("Other");
send
59. Subscribers can register to a specific topic.
Data filtering
Patterns
Publish / Subscribe
ØMQ Publisher
Subscriber A
Subscriber B
Subscriber C
Outgoing Queue
subscribe AR
subscribe VE
Socket
subscribe VE
60. Subscribers can register to a specific topic.
Data filtering
Patterns
Publish / Subscribe
ØMQ Publisher
Subscriber A
Subscriber B
Subscriber C
Outgoing Queue
subscribe AR
subscribe VE
Socket
subscribe VE
sock.send("AR
news");
send
7 AR news
send
61. Subscribers can register to a specific topic.
Data filtering
Patterns
Publish / Subscribe
ØMQ Publisher
Subscriber A
Subscriber B
Subscriber C
Outgoing Queue
subscribe AR
subscribe VE
Socket
subscribe VE
sock.send("VE
news");
send
7 VE news
sendsend
62. Patterns
Publish / Subscribe
Data filtering example
pub = zmq.socket('pub');
pub.bindSync("tcp://10.0.0.12:3055");
count = 0
setInterval(function() {
pub.send("TEST " + count++);
}, 1000);
sub = zmq.socket('sub');
sub.connect("tcp://10.0.0.12:3055");
sub.subscribe("TEST");
sub.on("message", function(msg) {
console.log("Received: " + msg);
});
// older messages won't be
// received
> TEST 6
> TEST 7
> TEST 8
> TEST 9
> TEST 10
running this
'might'
output ->
63. Patterns
Publish / Subscribe
Multicast example
var zmq = require('zmq'),
pub = zmq.socket('pub');
pub.bind(
"epgm://10.0.0.12;239.192.1.1:3055",
function(err) {
if(err) throw err;
}
)
setInterval(function() {
pub.send("From pid: " + process.pid);
}, 1000);
var zmq = require('zmq'),
sub = zmq.socket('sub');
sub.connect(
"epgm://10.0.0.13;239.192.1.1:3055"
);
sub.subscribe("");
sub.on('message', function(msg) {
console.log("Received " + msg);
});
(*) See running demo at http://youtu.be/NQrH0SATPk0
64. "XSUB and XPUB are exactly like SUB and PUB except they expose
subscriptions as special messages."
Data distribution proxy
Patterns
XSUB / XPUB
XSUB/XPUB Proxy
Subscriber A
Subscriber B
Subscriber C
XPUB Socket
XSUB Socket
Publisher A
Publisher B
Publisher C
subscribe
PY
forward
subscription
65. Messages are forwarded by the proxy to the registered subscribers.
Data distribution proxy
Patterns
XSUB / XPUB
XSUB/XPUB Proxy
Subscriber A
Subscriber B
Subscriber C
XPUB Socket
XSUB Socket
Publisher A
Publisher B
Publisher C
forward
message
send "PY<msg>"
send