This document provides an overview of lightweight messaging and remote procedure call (RPC) systems in distributed systems. It discusses messaging systems, typical peer-to-peer and broker-based messaging topologies, characteristics and features of messaging systems, main classes of messaging systems including enterprise service buses (ESBs), JMS implementations, AMQP implementations, and lightweight modern systems. It also covers RPC, serialization libraries, differences between messaging and RPC, examples of ZeroMQ for peer-to-peer messaging, Apache Kafka for broker-based messaging, and Twitter Finagle for scalable RPC.
October 2016 HUG: Pulsar, a highly scalable, low latency pub-sub messaging s...Yahoo Developer Network
Yahoo recently open-sourced Pulsar, a highly scalable, low latency pub-sub messaging system running on commodity hardware. It provides simple pub-sub messaging semantics over topics, guaranteed at-least-once delivery of messages, automatic cursor management for subscribers, and cross-datacenter replication. Pulsar is used across various Yahoo applications for large scale data pipelines. Learn more about Pulsar architecture and use-cases in this talk.
Speakers:
Matteo Merli from Pulsar team at Yahoo
High performance messaging with Apache PulsarMatteo Merli
Apache Pulsar is being used for an increasingly broad array of data ingestion tasks. When operating at scale, it's very important to ensure that the system can make use of all the available resources. Karthik Ramasamy and Matteo Merli share insights into the design decisions and the implementation techniques that allow Pulsar to achieve high performance with strong durability guarantees.
October 2016 HUG: Pulsar, a highly scalable, low latency pub-sub messaging s...Yahoo Developer Network
Yahoo recently open-sourced Pulsar, a highly scalable, low latency pub-sub messaging system running on commodity hardware. It provides simple pub-sub messaging semantics over topics, guaranteed at-least-once delivery of messages, automatic cursor management for subscribers, and cross-datacenter replication. Pulsar is used across various Yahoo applications for large scale data pipelines. Learn more about Pulsar architecture and use-cases in this talk.
Speakers:
Matteo Merli from Pulsar team at Yahoo
High performance messaging with Apache PulsarMatteo Merli
Apache Pulsar is being used for an increasingly broad array of data ingestion tasks. When operating at scale, it's very important to ensure that the system can make use of all the available resources. Karthik Ramasamy and Matteo Merli share insights into the design decisions and the implementation techniques that allow Pulsar to achieve high performance with strong durability guarantees.
Introduction and Overview of Apache Kafka, TriHUG July 23, 2013mumrah
Apache Kafka is a new breed of messaging system built for the "big data" world. Coming out of LinkedIn (and donated to Apache), it is a distributed pub/sub system built in Scala. It has been an Apache TLP now for several months with the first Apache release imminent. Built for speed, scalability, and robustness, Kafka should definitely be one of the data tools you consider when designing distributed data-oriented applications.
The talk will cover a general overview of the project and technology, with some use cases, and a demo.
Effectively-once semantics in Apache PulsarMatteo Merli
“Exactly-once” is a controversial term in the messaging landscape. In this presentation we offer a detailed look at effectively-once delivery semantics in Apache Pulsar and how this is achieved without sacrificing performance.
Using Apache Pulsar as a Modern, Scalable, High Performing JMS Platform - Pus...StreamNative
JMS, as the first widely-supported enterprise messaging API, has been in the market for close to 20 years and still plays critical roles in many enterprises nowadays. Many mission-critical business applications are still running in production that follows JMS (2.0) specification on various JMS platforms like ActiveMQ, TibcoEMS, and etc.
However, modern business activities have raised new challenges that JMS can't answer very well such as cross-region message replication, real-time complex event processing, seamless horizontal scalability, and etc. In order to address these challenges, newer enterprise messaging/streaming technologies like Apache Pulsar is needed.
In this presentation, I will do a deep dive investigation on how Apache Pulsar can be used as the next generation unified enterprise messaging/streaming platform that can serve existing JMS applications with very minimum code changes. I will also demonstrate JMS to Pulsar migration with several concrete use cases and examples.
Building a Messaging Solutions for OVHcloud with Apache Pulsar_Pierre ZembStreamNative
OVHcloud is the biggest European cloud provider. From dedicated servers to Managed Kubernetes, from VMware® based Hosted Private Cloud to OpenStack-based Public Cloud, we have over 1.4 million customers worldwide.
Internally, we have been running Apache Kafka for years, and despite all the skills obtained operating multiples clusters with millions of messages per second, we decided to shift and build the foundation of our 'topic-as-a-service' product called ioStream on Apache Pulsar.
In this talk, you will have the insights of why we decided to use Apache Pulsar instead of Apache Kafka as the core of ioStream. We will tell you our journey to use Apache Pulsar, from our deployments to the management, what did work and what did not.
Pulsar - flexible pub-sub for internet scaleMatteo Merli
Pub-Sub messaging is a very convenient abstraction that allows system and application developers to decouple components and let them communicate, by acting as durable buffer for transient data, or as a persistent log from where to recover after crashes. This talk will present an overview of Apache Pulsar, the reasons that led to its development and how it enabled many teams at Yahoo and to build scalable and reliable applications. Apache Pulsar has become the defacto pub-sub messaging at Yahoo serving 100+ applications and processing 100’s of billions of messages for over 3+ years.
In this talk, we will explore in detail different categories of use cases that highlight how Pulsar can be applied to solve a broad range of problems thanks to its flexible messaging model that supports both queuing and streaming semantics with a focus on durability and transaction guarantees.
Micro on NATS - Microservices with MessagingApcera
This is a talk given by Asim Aslam at the NATS London Meetup on May 10th, 2016. It explains what Micro is (Microservices toolkit), and how it uses NATS - a lightweight high performance open source messaging system for microservices, cloud native, and IoT networks written in Golang.
You can learn more about NATS at http://www.nats.io
You can learn more about Micro at https://micro.mu/
Apache Kafka is an open-source message broker project developed by the Apache Software Foundation written in Scala. The project aims to provide a unified, high-throughput, low-latency platform for handling real-time data feeds.
Kafka is a real-time, fault-tolerant, scalable messaging system.
It is a publish-subscribe system that connects various applications with the help of messages - producers and consumers of information.
Pulsar Storage on BookKeeper _Seamless EvolutionStreamNative
Apache Pulsar has a distinct architecture from other messaging systems. There is a clear separation of the compute layer that does message processing and dispatching, from the storage layer that handles persistent message storage, using Apache Bookkeeper. This separation of concerns leads to a very efficient design, in terms of performance and cost.
Messaging systems that provide guaranteed delivery, when used in production use cases, impose on the underlying storage, demands that are very different from simple benchmark scenarios that test write throughput. Pulsar, with both I/O isolation and separation of concerns, performs better than other messaging systems in production use cases. The strategy of I/O isolation provides better performance from each storage node at less cost, and the separation between computing and storage means that compute nodes can be scaled independently from storage. Irrespective of the choice of storage, Pulsar can be configured to get the best performance for any of those storage configurations.
This paper also discusses how some of the latest technologies like NVMe and Persistent Memory can be leveraged at a very low cost overhead, by Pulsar, without any architectural or design changes, with some data from real use cases. The fundamental choice of using Bookkeeper as the storage layer for Pulsar is validated from our experience.
At Hootsuite, we've been transitioning from a single monolithic PHP application to a set of scalable Scala-based microservices. To avoid excessive coupling between services, we've implemented an event system using Apache Kafka that allows events to be reliably produced + consumed asynchronously from services as well as data stores.
In this presentation, I talk about:
- Why we chose Kafka
- How we set up our Kafka clusters to be scalable, highly available, and multi-data-center aware.
- How we produce + consume events
- How we ensure that events can be understood by all parts of our system (Some that are implemented in other programming languages like PHP and Python) and how we handle evolving event payload data.
Introduction and Overview of Apache Kafka, TriHUG July 23, 2013mumrah
Apache Kafka is a new breed of messaging system built for the "big data" world. Coming out of LinkedIn (and donated to Apache), it is a distributed pub/sub system built in Scala. It has been an Apache TLP now for several months with the first Apache release imminent. Built for speed, scalability, and robustness, Kafka should definitely be one of the data tools you consider when designing distributed data-oriented applications.
The talk will cover a general overview of the project and technology, with some use cases, and a demo.
Effectively-once semantics in Apache PulsarMatteo Merli
“Exactly-once” is a controversial term in the messaging landscape. In this presentation we offer a detailed look at effectively-once delivery semantics in Apache Pulsar and how this is achieved without sacrificing performance.
Using Apache Pulsar as a Modern, Scalable, High Performing JMS Platform - Pus...StreamNative
JMS, as the first widely-supported enterprise messaging API, has been in the market for close to 20 years and still plays critical roles in many enterprises nowadays. Many mission-critical business applications are still running in production that follows JMS (2.0) specification on various JMS platforms like ActiveMQ, TibcoEMS, and etc.
However, modern business activities have raised new challenges that JMS can't answer very well such as cross-region message replication, real-time complex event processing, seamless horizontal scalability, and etc. In order to address these challenges, newer enterprise messaging/streaming technologies like Apache Pulsar is needed.
In this presentation, I will do a deep dive investigation on how Apache Pulsar can be used as the next generation unified enterprise messaging/streaming platform that can serve existing JMS applications with very minimum code changes. I will also demonstrate JMS to Pulsar migration with several concrete use cases and examples.
Building a Messaging Solutions for OVHcloud with Apache Pulsar_Pierre ZembStreamNative
OVHcloud is the biggest European cloud provider. From dedicated servers to Managed Kubernetes, from VMware® based Hosted Private Cloud to OpenStack-based Public Cloud, we have over 1.4 million customers worldwide.
Internally, we have been running Apache Kafka for years, and despite all the skills obtained operating multiples clusters with millions of messages per second, we decided to shift and build the foundation of our 'topic-as-a-service' product called ioStream on Apache Pulsar.
In this talk, you will have the insights of why we decided to use Apache Pulsar instead of Apache Kafka as the core of ioStream. We will tell you our journey to use Apache Pulsar, from our deployments to the management, what did work and what did not.
Pulsar - flexible pub-sub for internet scaleMatteo Merli
Pub-Sub messaging is a very convenient abstraction that allows system and application developers to decouple components and let them communicate, by acting as durable buffer for transient data, or as a persistent log from where to recover after crashes. This talk will present an overview of Apache Pulsar, the reasons that led to its development and how it enabled many teams at Yahoo and to build scalable and reliable applications. Apache Pulsar has become the defacto pub-sub messaging at Yahoo serving 100+ applications and processing 100’s of billions of messages for over 3+ years.
In this talk, we will explore in detail different categories of use cases that highlight how Pulsar can be applied to solve a broad range of problems thanks to its flexible messaging model that supports both queuing and streaming semantics with a focus on durability and transaction guarantees.
Micro on NATS - Microservices with MessagingApcera
This is a talk given by Asim Aslam at the NATS London Meetup on May 10th, 2016. It explains what Micro is (Microservices toolkit), and how it uses NATS - a lightweight high performance open source messaging system for microservices, cloud native, and IoT networks written in Golang.
You can learn more about NATS at http://www.nats.io
You can learn more about Micro at https://micro.mu/
Apache Kafka is an open-source message broker project developed by the Apache Software Foundation written in Scala. The project aims to provide a unified, high-throughput, low-latency platform for handling real-time data feeds.
Kafka is a real-time, fault-tolerant, scalable messaging system.
It is a publish-subscribe system that connects various applications with the help of messages - producers and consumers of information.
Pulsar Storage on BookKeeper _Seamless EvolutionStreamNative
Apache Pulsar has a distinct architecture from other messaging systems. There is a clear separation of the compute layer that does message processing and dispatching, from the storage layer that handles persistent message storage, using Apache Bookkeeper. This separation of concerns leads to a very efficient design, in terms of performance and cost.
Messaging systems that provide guaranteed delivery, when used in production use cases, impose on the underlying storage, demands that are very different from simple benchmark scenarios that test write throughput. Pulsar, with both I/O isolation and separation of concerns, performs better than other messaging systems in production use cases. The strategy of I/O isolation provides better performance from each storage node at less cost, and the separation between computing and storage means that compute nodes can be scaled independently from storage. Irrespective of the choice of storage, Pulsar can be configured to get the best performance for any of those storage configurations.
This paper also discusses how some of the latest technologies like NVMe and Persistent Memory can be leveraged at a very low cost overhead, by Pulsar, without any architectural or design changes, with some data from real use cases. The fundamental choice of using Bookkeeper as the storage layer for Pulsar is validated from our experience.
At Hootsuite, we've been transitioning from a single monolithic PHP application to a set of scalable Scala-based microservices. To avoid excessive coupling between services, we've implemented an event system using Apache Kafka that allows events to be reliably produced + consumed asynchronously from services as well as data stores.
In this presentation, I talk about:
- Why we chose Kafka
- How we set up our Kafka clusters to be scalable, highly available, and multi-data-center aware.
- How we produce + consume events
- How we ensure that events can be understood by all parts of our system (Some that are implemented in other programming languages like PHP and Python) and how we handle evolving event payload data.
bigdata 2022_ FLiP Into Pulsar Apps
In this session, Timothy will introduce you to the world of Apache Pulsar and how to build real-time messaging and streaming applications with a variety of OSS libraries, schemas, languages, frameworks, and tools.
FLiP Into Pulsar Apps
08:30 – 09:15
•
23 Nov, 2022
In this session, Timothy will introduce you to the world of Apache Pulsar and how to build real-time messaging and streaming applications with a variety of OSS libraries, schemas, languages, frameworks, and tools.
Capital One Delivers Risk Insights in Real Time with Stream Processingconfluent
Speakers: Ravi Dubey, Senior Manager, Software Engineering, Capital One + Jeff Sharpe, Software Engineer, Capital One
Capital One supports interactions with real-time streaming transactional data using Apache Kafka®. Kafka helps deliver information to internal operation teams and bank tellers to assist with assessing risk and protect customers in a myriad of ways.
Inside the bank, Kafka allows Capital One to build a real-time system that takes advantage of modern data and cloud technologies without exposing customers to unnecessary data breaches, or violating privacy regulations. These examples demonstrate how a streaming platform enables Capital One to act on their visions faster and in a more scalable way through the Kafka solution, helping establish Capital One as an innovator in the banking space.
Join us for this online talk on lessons learned, best practices and technical patterns of Capital One’s deployment of Apache Kafka.
-Find out how Kafka delivers on a 5-second service-level agreement (SLA) for inside branch tellers.
-Learn how to combine and host data in-memory and prevent personally identifiable information (PII) violations of in-flight transactions.
-Understand how Capital One manages Kafka Docker containers using Kubernetes.
Watch the recording: https://videos.confluent.io/watch/6e6ukQNnmASwkf9Gkdhh69?.
AMIS SIG - Introducing Apache Kafka - Scalable, reliable Event Bus & Message ...Lucas Jellema
Introduction of Apache Kafka - the open source platform for real time message queuing and reliable, scalable, distributed event handling and high volume pub/sub implementation.
see GitHub https://github.com/MaartenSmeets/kafka-workshop for the workshop resources.
Linked In Stream Processing Meetup - Apache PulsarKarthik Ramasamy
Apache Pulsar is the next generation messaging system that uses a fundamentally different architecture to achieve durability, performance, scalability, efficiency, multi-tenancy and geo replication.
Big Data Streams Architectures. Why? What? How?Anton Nazaruk
With a current zoo of technologies and different ways of their interaction it's a big challenge to architect a system (or adopt existed one) that will conform to low-latency BigData analysis requirements. Apache Kafka and Kappa Architecture in particular take more and more attention over classic Hadoop-centric technologies stack. New Consumer API put significant boost in this direction. Microservices-based streaming processing and new Kafka Streams tend to be a synergy in BigData world.
Matteo Merli and Sijie Guo from Streamlio gave a hands-on workshop on Apache Pulsar. #fast #durable #pubsub #messaging system. A low latency alternative to #kafka.
Let’s Monitor Conditions at the Conference With Timothy Spann & David Kjerrum...HostedbyConfluent
Let’s Monitor Conditions at the Conference With Timothy Spann & David Kjerrumgaard | Current 2022
At home, I monitor the temperature, humidity, gas levels, ozone, air quality, and other features around my desk.
Let's bring this to the different spots around the conference including lunch tables, vendor booths, hotel rooms, and more. I need to know about these readings now, not when I get back home from the conference. We need to get these sensor readings immediately in case we need to turn on a fan or move to another area. We will also see if my talk produces a lot of hot air!?!??
My setup is pretty simple, a raspberry pi, a breakout garden sensor mount, and as many sensors as I am willing to fly to Austin. The software stack is Python and Java, Apache Pulsar, MQTT, HTML, JQuery, and Apache Kafka.
https://dzone.com/articles/five-sensors-real-time-with-pulsar-and-python-on-a
https://www.datainmotion.dev/2022/04/flip-py-pi-enviroplus-using-apache.html
https://dzone.com/articles/pulsar-in-python-on-pi
(Current22) Let's Monitor The Conditions at the ConferenceTimothy Spann
(Current22) Let's Monitor The Conditions at the Conference
Let's Monitor The Conditions at the Conference
Session Time11:15 am - 12:00 pm Session DateWednesday, 5 October 2022 Session Type:In-Person Location:Ballroom G
Session Description:
At home, I monitor the temperature, humidity, gas levels, ozone, air quality, and other features around my desk. Let's bring this to the different spots around the conference including lunch tables, vendor booths, hotel rooms, and more. I need to know about these readings now, not when I get back home from the conference. We need to get these sensor readings immediately in case we need to turn on a fan or move to another area. We will also see if my talk produces a lot of hot air!? My setup is pretty simple, a raspberry pi, a breakout garden sensor mount, and as many sensors as I am willing to fly to Austin. The software stack is Python and Java, Apache Pulsar, MQTT, HTML, JQuery, and Apache Kafka.
Timothy Spann, StreamNative
Developer Advocate
Tim Spann is a Developer Advocate @ StreamNative where he works with Apache Pulsar, Apache Flink, Apache NiFi, Apache MXNet, TensorFlow, Apache Spark, big data, the IoT, machine learning, and deep learning. Tim has over a decade of experience with the IoT, big data, distributed computing, streaming technologies, and Java programming. Previously, he was a Principal Field Engineer at Cloudera, a Senior Solutions Architect at AirisData and a senior field engineer at Pivotal. He blogs for DZone, where he is the Big Data Zone leader, and runs a popular meetup in Princeton on big data, the IoT, deep learning, streaming, NiFi, the blockchain, and Spark. Tim is a frequent speaker at conferences such as IoT Fusion, Strata, ApacheCon, Data Works Summit Berlin, DataWorks Summit Sydney, and Oracle Code NYC.
JConf.dev 2022 - Apache Pulsar Development 101 with JavaTimothy Spann
JConf.dev 2022 - Apache Pulsar Development 101 with Java
https://2022.jconf.dev/
In this session I will get you started with real-time cloud native streaming programming with Java. We will start off with a gentle introduction to Apache Pulsar and setting up your first easy standalone cluster. We will then l show you how to produce and consume message to Pulsar using several different Java libraries including native Java client, AMQP/RabbitMQ, MQTT and even Kafka. After this session you will building real-time streaming and messaging applications with Java. We will also touch on Apache Spark and Apache Flink.
Timothy Spann
Tim Spann is a Developer Advocate @ StreamNative where he works with Apache Pulsar, Apache Flink, Apache NiFi, Apache MXNet, TensorFlow, Apache Spark, big data, the IoT, machine learning, and deep learning. Tim has over a decade of experience with the IoT, big data, distributed computing, streaming technologies, and Java programming. Previously, he was a Principal Field Engineer at Cloudera, a Senior Solutions Architect at AirisData and a senior field engineer at Pivotal. He blogs for DZone, where he is the Big Data Zone leader, and runs a popular meetup in Princeton on big data, the IoT, deep learning, streaming, NiFi, the blockchain, and Spark. Tim is a frequent speaker at conferences such as IoT Fusion, Strata, ApacheCon, Data Works Summit Berlin, DataWorks Summit Sydney, and Oracle Code NYC. He holds a BS and MS in computer science. https://www.datainmotion.dev/p/about-me.html https://dzone.com/users/297029/bunkertor.html https://conferences.oreilly.com/strata/strata-ny-2018/public/schedule/speaker/185963
[March sn meetup] apache pulsar + apache nifi for cloud data lakeTimothy Spann
https://www.meetup.com/new-york-city-apache-pulsar-meetup/events/283837865/
Learn how to use Apache Pulsar and Apache NiFi to Stream to your Data Lake
Discover how to stream data to and from your data lake or data mart using Apache Pulsar™ and Apache NiFi®. Learn how these cloud-native, scalable open-source projects built for streaming data pipelines work together to enable you to quickly build applications with minimal coding.
|WHAT THE SESSION WILL COVER|
Best Practices for using Pulsar and NiFi
A deep dive on Apache NiFi's Pulsar connector and demos
Building an End-to-End Application in the Hybrid Cloud
Attend for a chance to win a We <3 Pulsar t-shirt! The first 50 registrants who register through here [https://hubs.ly/Q013LTpn0] will be entered in a drawing!
—------------------------
|AGENDA|
6:00 - 7:00 PM EST: Presentation - Tim Spann, StreamNative Developer Advocate
7:00 - 8:00 PM EST: Presentation - John Kuchmek, Cloudera Principal Solutions Engineer
8:00 - 8:30 PM EST: Q&A + Networking
—------------------------
|ABOUT THE SPEAKERS|
John Kuchmek is a Principal Solutions Engineer for Cloudera. Before joining Cloudera, John transitioned to the Autonomous Intelligence team where he was in charge of integrating the platforms to allow data scientists to work with various types of data.
Tim Spann is a Developer Advocate for StreamNative. He works with StreamNative Cloud, Apache Pulsar™, Apache Flink®, Flink® SQL, Big Data, the IoT, machine learning, and deep learning. Tim has over a decade of experience with the IoT, big data, distributed computing, messaging, streaming technologies, and Java programming. Previously, he was a Principal DataFlow Field Engineer at Cloudera, a Senior Solutions Engineer at Hortonworks, a Senior Solutions Architect at AirisData, a Senior Field Engineer at Pivotal and a Team Leader at HPE. He blogs for DZone, where he is the Big Data Zone leader, and runs a popular meetup in Princeton on Big Data, Cloud, IoT, deep learning, streaming, NiFi, the blockchain, and Spark. Tim is a frequent speaker at conferences such as ApacheCon, DeveloperWeek, Pulsar Summit and many more. He holds a BS and MS in computer science. He is currently working on a book about the FLiP Stack.
Let's dive deeper into the world of ODC! Ricardo Alves (OutSystems) will join us to tell all about the new Data Fabric. After that, Sezen de Bruijn (OutSystems) will get into the details on how to best design a sturdy architecture within ODC.
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
Essentials of Automations: Optimizing FME Workflows with ParametersSafe Software
Are you looking to streamline your workflows and boost your projects’ efficiency? Do you find yourself searching for ways to add flexibility and control over your FME workflows? If so, you’re in the right place.
Join us for an insightful dive into the world of FME parameters, a critical element in optimizing workflow efficiency. This webinar marks the beginning of our three-part “Essentials of Automation” series. This first webinar is designed to equip you with the knowledge and skills to utilize parameters effectively: enhancing the flexibility, maintainability, and user control of your FME projects.
Here’s what you’ll gain:
- Essentials of FME Parameters: Understand the pivotal role of parameters, including Reader/Writer, Transformer, User, and FME Flow categories. Discover how they are the key to unlocking automation and optimization within your workflows.
- Practical Applications in FME Form: Delve into key user parameter types including choice, connections, and file URLs. Allow users to control how a workflow runs, making your workflows more reusable. Learn to import values and deliver the best user experience for your workflows while enhancing accuracy.
- Optimization Strategies in FME Flow: Explore the creation and strategic deployment of parameters in FME Flow, including the use of deployment and geometry parameters, to maximize workflow efficiency.
- Pro Tips for Success: Gain insights on parameterizing connections and leveraging new features like Conditional Visibility for clarity and simplicity.
We’ll wrap up with a glimpse into future webinars, followed by a Q&A session to address your specific questions surrounding this topic.
Don’t miss this opportunity to elevate your FME expertise and drive your projects to new heights of efficiency.
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
JMeter webinar - integration with InfluxDB and GrafanaRTTS
Watch this recorded webinar about real-time monitoring of application performance. See how to integrate Apache JMeter, the open-source leader in performance testing, with InfluxDB, the open-source time-series database, and Grafana, the open-source analytics and visualization application.
In this webinar, we will review the benefits of leveraging InfluxDB and Grafana when executing load tests and demonstrate how these tools are used to visualize performance metrics.
Length: 30 minutes
Session Overview
-------------------------------------------
During this webinar, we will cover the following topics while demonstrating the integrations of JMeter, InfluxDB and Grafana:
- What out-of-the-box solutions are available for real-time monitoring JMeter tests?
- What are the benefits of integrating InfluxDB and Grafana into the load testing stack?
- Which features are provided by Grafana?
- Demonstration of InfluxDB and Grafana using a practice web application
To view the webinar recording, go to:
https://www.rttsweb.com/jmeter-integration-webinar
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualityInflectra
In this insightful webinar, Inflectra explores how artificial intelligence (AI) is transforming software development and testing. Discover how AI-powered tools are revolutionizing every stage of the software development lifecycle (SDLC), from design and prototyping to testing, deployment, and monitoring.
Learn about:
• The Future of Testing: How AI is shifting testing towards verification, analysis, and higher-level skills, while reducing repetitive tasks.
• Test Automation: How AI-powered test case generation, optimization, and self-healing tests are making testing more efficient and effective.
• Visual Testing: Explore the emerging capabilities of AI in visual testing and how it's set to revolutionize UI verification.
• Inflectra's AI Solutions: See demonstrations of Inflectra's cutting-edge AI tools like the ChatGPT plugin and Azure Open AI platform, designed to streamline your testing process.
Whether you're a developer, tester, or QA professional, this webinar will give you valuable insights into how AI is shaping the future of software delivery.
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...UiPathCommunity
💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™:
See how to accelerate model training and optimize model performance with active learning
Learn about the latest enhancements to out-of-the-box document processing – with little to no training required
Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages
This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath.
Speakers:
👨🏫 Andras Palfi, Senior Product Manager, UiPath
👩🏫 Lenka Dulovicova, Product Program Manager, UiPath
Key Trends Shaping the Future of Infrastructure.pdfCheryl Hung
Keynote at DIGIT West Expo, Glasgow on 29 May 2024.
Cheryl Hung, ochery.com
Sr Director, Infrastructure Ecosystem, Arm.
The key trends across hardware, cloud and open-source; exploring how these areas are likely to mature and develop over the short and long-term, and then considering how organisations can position themselves to adapt and thrive.
Search and Society: Reimagining Information Access for Radical FuturesBhaskar Mitra
The field of Information retrieval (IR) is currently undergoing a transformative shift, at least partly due to the emerging applications of generative AI to information access. In this talk, we will deliberate on the sociotechnical implications of generative AI for information access. We will argue that there is both a critical necessity and an exciting opportunity for the IR community to re-center our research agendas on societal needs while dismantling the artificial separation between the work on fairness, accountability, transparency, and ethics in IR and the rest of IR research. Instead of adopting a reactionary strategy of trying to mitigate potential social harms from emerging technologies, the community should aim to proactively set the research agenda for the kinds of systems we should build inspired by diverse explicitly stated sociotechnical imaginaries. The sociotechnical imaginaries that underpin the design and development of information access technologies needs to be explicitly articulated, and we need to develop theories of change in context of these diverse perspectives. Our guiding future imaginaries must be informed by other academic fields, such as democratic theory and critical theory, and should be co-developed with social science scholars, legal scholars, civil rights and social justice activists, and artists, among others.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
"Impact of front-end architecture on development cost", Viktor TurskyiFwdays
I have heard many times that architecture is not important for the front-end. Also, many times I have seen how developers implement features on the front-end just following the standard rules for a framework and think that this is enough to successfully launch the project, and then the project fails. How to prevent this and what approach to choose? I have launched dozens of complex projects and during the talk we will analyze which approaches have worked for me and which have not.
3. Messaging System
Message (not packet/byte/…) as a minimal
transmission unit.
The whole system unifies
• Underlying protocol (TCP, UDP)
• UNICAST or MULTICAST
• Data format (message types & structure)
Tied with
• Serialization format (text or binary)
5. Typical broker-based messaging
Producer
[bhost, bport]
Broker Consumer
[bhost, bport]
• Broker is an indirection layer between
producer and consumer.
• Producer PUSHes messages to broker.
• Consumer PULLs messages from broker.
6. The trick is…
Producer
[bhost, bport]
Broker Consumer
[bhost, bport]
• Producers and consumers are logical units.
• Both P and C may be launched in multiple
instances.
• p2p and pubsub terms are expressed in terms of
these logical (!) units.
• Even broker may be distributed or replicated
entity.
8. In a generic case
• A service may be both a consumer for many
producers and a producer to many
consumers
9. Characteristics and Features
• Topology (1-1, 1-N, N-N)
• Retries
• Service discovery
• Guaranteed delivery (in case yes – at-least-once or exactly-once)
• Ordering
• Acknowledge
• Disconnect detection
• Transactions support (can participate in distributed transactions)
• Persistence
• Portability (one or many languages and platforms)
• Distributed or not
• Highly available or not
• Type (p2p or broker-based)
• Load balancing strategy for consumers
• Client backoff strategy for producers
• Tracing support
• Library or standalone software
10. Main classes
• ESBs (Enterprise service buses)
– Slow, but most feature-rich. MuleESB, JbossESB,
Apache Camel, many commercial.
• JMS implementations
– ActiveMQ, JBOSS Messaging, Glassfish, etc.
• AMQP implementations
– RabbitMQ, Qpid, HornetQ, etc.
• Lightweight modern stuff - unstandardized
– ZeroMQ, Finagle, Kafka, Beanstalkd, etc.
11. Messaging Performance
As usual, its about throughput and latency…
Major throughput factors:
– Network hardware used
– UNICAST vs MULTICAST (for fan-out)
Major latency factors:
– Persistence (batched or single-message persistence
involves sequential or random disk writes)
– Transactions
– Broker replication
– Delivery guarantees (at-least-once & exactly-once)
12. Guaranteed delivery
Involves additional logic both on Producer,
Consumer and Broker (if any)!
This is at-least-once delivery:
• Producer needs to get ack’ed by Broker
• Consumer needs to track high-watermark of
messages received from Broker
Exact-once delivery requires more work and even
more expensive. Typically implemented as 2-phase
commit.
13. Ordering (distributed broker scenario)
• Producers receive messages in any order. Very cheap.
No Ordering
• Messages are ordered within single data partition. Such
as: stock symbol, account number, etc. Possible to create
well-performing implementation of distributed broker.
Partitioned Ordering
• All incoming messages are fairly ordered. Scalability and
performance is limited.
Global (fair) ordering
14. Remote procedure calls
Inherently builds on top of some messaging.
Method call as a minimal unit (3 states: may
succeed returning optional value, throw exception,
or time out).
Adds some RPC-specific characteristics & features:
• Sync or async
• Distributed stack traces for exceptions
• Interfaces and structs declaration (possibly, via
some DSL) – often come with serialization library
• May support schema evolution
15. Serialization libraries
Currently, there are 4 clear winners:
1. Google Protocol buffers (with ProtoStuff)
2. Apache Thrift
3. Avro
4. MessagePack
All provide DSLs and schema evolution.
Difference is in wire format and DSL compiler
form (program in C, in Java, or does not require
compilation).
16. Messaging vs RPC
Messaging
• In Broker-enabled case:
Producers are decoupled
from Consumers. Just
push message and don’t
care who pulls it.
• Natively matches
messages to events in
event-sourcing
architectures.
RPC
• Need to know destination
(i.e., service A must know
service B and call
signature).
Messaging and RPC dictate different programming
models. RPC requires higher coupling between
interacting services.
19. ZeroMQ
“It's sockets on steroids. It's like mailboxes with
routing. It's fast!
Things just become simpler. Complexity goes
away. It opens the mind. Others try to explain
by comparison. It's smaller, simpler, but still
looks familiar.”
@ ZeroMQ 2.2 Guide
20. ZeroMQ - features
• Topology – all, very flexible.
• Retries – no.
• Service discovery – no.
• Guaranteed delivery – no.
• Acknowledge – no.
• Disconnect detection – no.
• Transactions support (can participate in distributed transactions) – no.
• Persistence – kind of.
• Portability (one or many languages and platforms) – yes, there are many bindings. However,
library itself is written in C, so there’s only one “native” binding.
• Distributed – yes.
• Highly available or not – no.
• Type (p2p or broker-based) – mostly p2p. In case of N-N topology, a broker needed in form of
ZMQ “Device” with ROUTER/DEALER type sockets.
• Load balancing strategy for consumers – yes (???).
• Client backoff strategy for producers – no.
• Tracing support – no.
• Library or standalone software – platform-native library + language bindings.
21. ZeroMQ – features explained
Isn’t there too much “no”s ?
Yes and no. Most of the features are not
provided out of the box, but may be
implemented manually in client andor server.
Some features are easy to implement
(heartbeats, ack’s, retries, …) some are very
complex (guaranteed delivery, persistence, high
availability).
22. ZeroMQ – what’s bad about it
• First of all – name.
Think of ZMQ as a sockets library and u’re happy.
Consider it messaging middleware and u got frustrated
just while reading guide.
• Complex implementation for multithreaded
clients and servers.
• There were issues with services going down due
to corrupted packets (so, may not be suitable for
WAN).
• Some mess with development process. Initial
ZMQ developers forked ZMQ as Crossroads.io
23. ZeroMQ – what’s good
• Huge list of supported platforms.
• MULTICAST support for fan-out (1-N)
topology.
• High raw performance.
• Fluent connect/disconnect/reconnect
behavior – really feels how it should be.
• Wants to be part of Linux kernel.
24. ZeroMQ – verdict
• Good for non-reliable high performance
communication, when delivery semantics is
not strict. Example - ngx-zeromq module for
NGINX.
• Good if you can invest sufficient effort in
building custom messaging platform on top
of ZMQ as a network library. Example –
ZeroRPC lib by DotCloud.
• Bad for any other purpose.
25. Apache Kafka
“We have built a novel messaging system for log
processing called Kafka that combines the benefits
of traditional log aggregators and messaging
systems. On the one hand, Kafka is distributed and
scalable, and offers high throughput. On the other
hand, Kafka provides an API similar to a messaging
system and allows applications to consume log
events in real time.”
@ Kafka: a Distributed Messaging System for Log Processing,
LinkedIn
26. Kafka - features
• Topology – all.
• Retries – no.
• Service discovery – yes (Zookeeper).
• Guaranteed delivery – no (at-least-once in normal case).
• Acknowledge – no.
• Disconnect detection – yes (Zookeeper).
• Transactions support (can participate in distributed transactions) – no.
• Persistence – yes.
• Portability (one or many languages and platforms) – no.
• Distributed – yes.
• Highly available or not – no (work in progress).
• Type (p2p or broker-based) – broker-enabled with distributed broker.
• Load balancing strategy for consumers – yes.
• Client backoff strategy for producers – yes .
• Tracing support – no.
• Library or standalone software – standalone + client libraries in Java.
28. Kafka - Internals
• Fast writes
– Configurable batching
– All writes are continuous, no need for random disk access
(i.e., works well on commodity SATA/SAS disks in RAID
arrays)
• Fast reads
– O(1) disk search
– Extensive use of sendfile()
– No in-memory data caching inside Kafka – fully relies on
OS file system’s page cache
• Elastic horizontal scalability
– Zookeeper is used for brokers and consumers discovery
– Pubsub topics are distributed among brokers
29. Kafka - conclusion
• Good for event-sourcing architectures
(especially when they add HA support for
brokers).
• Good to decouple incoming stream and
processing to withstand request spikes.
• Very good for logs aggregation and
monitoring data collection.
• Bad for transactional messaging with rich
delivery semantics (exact once etc).
30. Twitter Finagle
“Finagle is a protocol-agnostic, asynchronous
RPC system for the JVM that makes it easy to
build robust clients and servers in Java, Scala,
or any JVM-hosted language.
Finagle supports a wide variety of
request/response- oriented RPC protocols and
many classes of streaming protocols.”
@ Twitter Engineering Blog
31. Finagle - features
• Topology – all, very flexible.
• Retries – yes.
• Service discovery – yes (Zookeper).
• Guaranteed delivery – no.
• Acknowledge – no.
• Disconnect detection – yes.
• Transactions support (can participate in distributed transactions) – no.
• Persistence – no.
• Portability (one or many languages and platforms) – JVM only.
• Distributed – yes.
• Highly available – yes.
• Type (p2p or broker-based) – p2p.
• Load balancing strategy for consumers – yes (least connections etc).
• Client backoff strategy for producers – yes (limited exponential).
• Tracing support – yes (Zipkin ).
• Library or standalone software – Scala library.
32. Finagle – from authors
Finagle provides a robust implementation of:
• connection pools, with throttling to avoid TCP connection churn;
• failure detectors, to identify slow or crashed hosts;
• failover strategies, to direct traffic away from unhealthy hosts;
• load-balancers, including “least-connections” and other strategies;
• back-pressure techniques, to defend servers against abusive
clients and dogpiling.
Additionally, Finagle makes it easier to build and deploy a service that
• publishes standard statistics, logs, and exception reports;
• supports distributed tracing (a la Dapper) across protocols;
• optionally uses ZooKeeper for cluster management; and
• supports common sharding strategies.
37. Finagle - conclusion
• Good for complex JVM-based RPC
architectures.
• Very good for Scala, worse experience with
Java (but yes, they have some utility classes).
• Works well with Thrift and HTTP (plus trivial
protocols), but lacks support for Protobuf and
other popular stuff.
• Active developers community (Google
group), but project infrastructure (maven
repo, versioning, etc) still being improved.
38. Resources
• Moscow Big Systems / Big Data group
http://www.meetup.com/bigmoscow/
• http://www.zeromq.org
• http://zerorpc.dotcloud.com
• http://kafka.apache.org
• http://twitter.github.io/finagle/
This template can be used as a starter file for presenting training materials in a group setting.SectionsSections can help to organize your slides or facilitate collaboration between multiple authors. On the Home tab under Slides, click Section, and then click Add Section.NotesUse the Notes pane for delivery notes or to provide additional details for the audience. You can see these notes in Presenter View during your presentation. Keep in mind the font size (important for accessibility, visibility, videotaping, and online production)Coordinated colors Pay particular attention to the graphs, charts, and text boxes.Consider that attendees will print in black and white or grayscale. Run a test print to make sure your colors work when printed in pure black and white and grayscale.Graphics, tables, and graphsKeep it simple: If possible, use consistent, non-distracting styles and colors.Label all graphs and tables.
Give a brief overview of the presentation. Describe the major focus of the presentation and why it is important.Introduce each of the major topics.To provide a road map for the audience, you can repeat this Overview slide throughout the presentation, highlighting the particular topic you will discuss next.
Summarize presentation content by restating the important points from the lessons.What do you want the audience to remember when they leave your presentation?
Give a brief overview of the presentation. Describe the major focus of the presentation and why it is important.Introduce each of the major topics.To provide a road map for the audience, you can repeat this Overview slide throughout the presentation, highlighting the particular topic you will discuss next.
Give a brief overview of the presentation. Describe the major focus of the presentation and why it is important.Introduce each of the major topics.To provide a road map for the audience, you can repeat this Overview slide throughout the presentation, highlighting the particular topic you will discuss next.
Give a brief overview of the presentation. Describe the major focus of the presentation and why it is important.Introduce each of the major topics.To provide a road map for the audience, you can repeat this Overview slide throughout the presentation, highlighting the particular topic you will discuss next.