Distributed consensus is impossible when at least one process might fail according to the Fischer, Lynch and Paterson (FLP) 1985 paper. The paper proves that no deterministic algorithm can solve consensus under the conditions of the system model used, which allows processes to fail by stopping (crash failures). Specifically, it is impossible to guarantee termination, validity and agreement simultaneously in an asynchronous distributed system where even one process may fail. This seminal result established fundamental limits in distributed computing and spurred further research on weaker models and failure detectors.
Keptn- A Cloud-native application life-cycle orchestration.pdfKnoldus Inc.
In this session we will know about keptn which is a growing an open source enterprise-grade control plane for cloud-native continuous delivery and automated operations.
This slides explains why Paxos is the only correctly way to problems about consensus in a distributed system.
This slides uses several diagram to show how paxos is derived from a naive replication algorithm to a immediate consistent replication algorithm.
It starts with master-slave replication.
Then we refine it to quorum-rw by adding consistency constrain.
And then we refine quorum-rw to paxos by adding atomicity constrain.
It covers a brief introduction to Apache Kafka Connect, giving insights about its benefits,use cases, motivation behind building Kafka Connect.And also a short discussion on its architecture.
Running Apache Kafka in production is only the first step in the Kafka operations journey. Professional Kafka users are ready to handle all possible disasters - because for most businesses having a disaster recovery plan is not optional.
In this session, we’ll discuss disaster scenarios that can take down entire Kafka clusters and share advice on how to plan, prepare and handle these events. This is a technical session full of best practices - we want to make sure you are ready to handle the worst mayhem that nature and auditors can cause.
Visit www.confluent.io for more information.
Keptn- A Cloud-native application life-cycle orchestration.pdfKnoldus Inc.
In this session we will know about keptn which is a growing an open source enterprise-grade control plane for cloud-native continuous delivery and automated operations.
This slides explains why Paxos is the only correctly way to problems about consensus in a distributed system.
This slides uses several diagram to show how paxos is derived from a naive replication algorithm to a immediate consistent replication algorithm.
It starts with master-slave replication.
Then we refine it to quorum-rw by adding consistency constrain.
And then we refine quorum-rw to paxos by adding atomicity constrain.
It covers a brief introduction to Apache Kafka Connect, giving insights about its benefits,use cases, motivation behind building Kafka Connect.And also a short discussion on its architecture.
Running Apache Kafka in production is only the first step in the Kafka operations journey. Professional Kafka users are ready to handle all possible disasters - because for most businesses having a disaster recovery plan is not optional.
In this session, we’ll discuss disaster scenarios that can take down entire Kafka clusters and share advice on how to plan, prepare and handle these events. This is a technical session full of best practices - we want to make sure you are ready to handle the worst mayhem that nature and auditors can cause.
Visit www.confluent.io for more information.
Kafka Tutorial - Introduction to Apache Kafka (Part 1)Jean-Paul Azar
Why is Kafka so fast? Why is Kafka so popular? Why Kafka? This slide deck is a tutorial for the Kafka streaming platform. This slide deck covers Kafka Architecture with some small examples from the command line. Then we expand on this with a multi-server example to demonstrate failover of brokers as well as consumers. Then it goes through some simple Java client examples for a Kafka Producer and a Kafka Consumer. We have also expanded on the Kafka design section and added references. The tutorial covers Avro and the Schema Registry as well as advance Kafka Producers.
This workshop presentation by Ticketmaster discussed Prometheus and Thanos. it focused on where they fit in in the Cloud Native lanscape and how they're being used.
Kafka Connect: Real-time Data Integration at Scale with Apache Kafka, Ewen Ch...confluent
Many companies are adopting Apache Kafka to power their data pipelines, including LinkedIn, Netflix, and Airbnb. Kafka’s ability to handle high throughput real-time data makes it a perfect fit for solving the data integration problem, acting as the common buffer for all your data and bridging the gap between streaming and batch systems.
However, building a data pipeline around Kafka today can be challenging because it requires combining a wide variety of tools to collect data from disparate data systems. One tool streams updates from your database to Kafka, another imports logs, and yet another exports to HDFS. As a result, building a data pipeline can take significant engineering effort and has high operational overhead because all these different tools require ongoing monitoring and maintenance. Additionally, some of the tools are simply a poor fit for the job: the fragmented nature of the data integration tools ecosystem lead to creative but misguided solutions such as misusing stream processing frameworks for data integration purposes.
We describe the design and implementation of Kafka Connect, Kafka’s new tool for scalable, fault-tolerant data import and export. First we’ll discuss some existing tools in the space and why they fall short when applied to data integration at large scale. Next, we will explore Kafka Connect’s design and how it compares to systems with similar goals, discussing key design decisions that trade off between ease of use for connector developers, operational complexity, and reuse of existing connectors. Finally, we’ll discuss how standardizing on Kafka Connect can ultimately lead to simplifying your entire data pipeline, making ETL into your data warehouse and enabling stream processing applications as simple as adding another Kafka connector.
eventbrite_kafka_summit_event_logo_v3-035858-edited.png
Build and Deploy Cloud Native Camel Quarkus routes with Tekton and KnativeOmar Al-Safi
In this talk, we will leverage all cloud native stacks and tools to build Camel Quarkus routes natively using GraalVM native-image on Tekton pipeline and deploy these routes to Kubernetes cluster with Knative installed. We will dive into the following topics in the talk: - Introduction to Camel - Introduction to Camel Quarkus - Introduction to GraalVM Native Image - Introduction to Tekon - Introduction to Knative - Demo shows how to deploy end to end a Camel Quarkus route which include the following steps: - Look at whole deployment pipeline for Cloud Native Camel Quarkus routes - Build Camel Quarkus routes with GraalVM native-image on Tekton pipeline. - Deploy Camel Quarkus routes to Kubernetes cluster with Knative Targeted Audience: Users with basic Camel knowledge
This presentation will recount the story of Macys.com (and Bloomingdales.com)'s selection and migration from legacy RDBMS to NoSQL Cassandra in partnership with DataStax.
We'll start with a mercifully brief backgrounder on our website and our business. Then we will go over the various technologies that we considered, as well as our use case-based performance benchmarks that led to the decision to go with Cassandra.
We'll cover the various schema options that we tried and how we settled on the current one. We'll show you a selection of some of our extensive performance tuning benchmarks.
One thing that differentiates this talk from others on Cassandra is Macy's philosophy of "doing more with less." You will see why we emphasize the performance tuning aspects of iterative development when you see how much processing we can support on relatively small configurations.
And, finally, we will wrap up with our "lessons learned" and a brief look at our future plans.
클라우드 네이티브로의 전환이 확산되면서 애플리케이션을 상호 독립적인 최소 구성 요소로 쪼개는 마이크로서비스(microservices) 아키텍쳐가 각광받고 있는데요.
MSA는 애플리케이션의 확장이 쉽고 새로운 기능의 출시 기간을 단축시킬 수 있다는 장점이 있지만,
반면에 애플리케이션이 커지고 동일한 서비스의 여러 인스턴스가 동시에 실행되면 MSA간 통신이 복잡해 진다는 단점이 있습니다.
서비스 메쉬(Service Mesh)는 이러한 MSA의 트래픽 문제를 보완하기 위해 탄생한 기술로,
서비스 간의 네트워크 트래픽 관리에 초점을 맞춘 네트워킹 모델입니다.
서로 다른 애플리케이션이 얼마나 원활하게 상호작용하는지를 기록함으로써 커뮤니케이션을 최적화하고 애플리케이션 확장에 따른 다운 타임을 방지할 수 있습니다.
서비스 메쉬의 탄생 배경과 기능, 그리고 현재 오픈소스로 배포되어 있는 서비스 메쉬 솔루션에 대해 소개합니다.
Step1. Cloud Native Trail Map
Step2. Service Proxy, Discover, & Mesh
Step3. Service Mesh 솔루션
Step4. Service Mesh 구현화면 - Istio / linkerd
Step5. Multi-cluster (linkerd)
From Zero to Hero with Kafka Connect (Robin Moffat, Confluent) Kafka Summit L...confluent
Integrating Apache Kafka with other systems in a reliable and scalable way is often a key part of a streaming platform. Fortunately, Apache Kafka includes the Connect API that enables streaming integration both in and out of Kafka. Like any technology, understanding its architecture and deployment patterns is key to successful use, as is knowing where to go looking when things aren’t working. This talk will discuss the key design concepts within Kafka Connect and the pros and cons of standalone vs distributed deployment modes. We’ll do a live demo of building pipelines with Kafka Connect for streaming data in from databases, and out to targets including Elasticsearch. With some gremlins along the way, we’ll go hands-on in methodically diagnosing and resolving common issues encountered with Kafka Connect. The talk will finish off by discussing more advanced topics including Single Message Transforms, and deployment of Kafka Connect in containers.
History and Basics of containers, LXC, Docker and Kubernetes. This presentation is given to Engineering colleage students at VIT DevFest 2018. Beginner to Intermediate level.
Building zero data loss pipelines with apache kafkaAvinash Ramineni
Kafka is playing an increasingly important role in messaging and streaming systems and is becoming the defacto messaging platform in many enterprises. Managing and maintaining Kafka deployments and tuning the data pipelines for high-performance and scalability can become a challenging task.
In this session, we will discuss the lessons learned and the best practices for achieving zero data loss pipelines.
How netflix manages petabyte scale apache cassandra in the cloudVinay Kumar Chella
At Netflix, we manage petabytes of data in Apache Cassandra which must be reliably accessible to users in mere milliseconds. To achieve this, we have built sophisticated control planes that turn our persistence layer based on Apache Cassandra into a truly self-driving system. We will start with the user interface that Netflix developers use to interact with their Cassandra databases and dive deep into the automation that powers it all. From cluster creation, through scaling up, to cluster death, complex automation drives large fleets of virtual machines hosted on the AWS cloud. First, we will cover the basics of how Netflix deploys Apache Cassandra. In particular, this begins with how we mold Apache Cassandra to the Netflix philosophy of immutable infrastructure, including managing software and hardware upgrades in the face of ever-failing hardware. Then we will explore the concrete techniques needed for such a massive deployment, specifically pull-based control planes and auto-healing strategies. Next, we will cover how Netflix has automated complex but critical Apache Cassandra maintenance tasks such as continuous snapshot backups and always-on anti-entropy repair for keeping our datasets safe and consistent. Both of these systems have gone through multiple architectural evolutions, and we have learned many lessons along the way. Lastly, we will share some of the ways this has gone wrong, and what you can do to avoid them. We will cover a few case studies of major Cassandra outages at Netflix, their root cause, and what we learned from those incidents. At the end of this talk, we hope that participants leave with concrete understanding of the challenges in running massive scale Apache Cassandra as well as solid advice and techniques for building their own self-driving data persistence layer.
Building Cloud-Native App Series - Part 7 of 11
Microservices Architecture Series
Containers Docker Kind Kubernetes Istio
- Pods
- ReplicaSet
- Deployment (Canary, Blue-Green)
- Ingress
- Service
Building Cloud-Native App Series - Part 2 of 11
Microservices Architecture Series
Event Sourcing & CQRS,
Kafka, Rabbit MQ
Case Studies (E-Commerce App, Movie Streaming, Ticket Booking, Restaurant, Hospital Management)
to transfer data in network from one device to another with acceptable accuracy, so the system must guarantee the transmitted data should be identical to received data.
there should be no errors if any error occurs in how many ways it can be detected and corrected
Kafka Tutorial - Introduction to Apache Kafka (Part 1)Jean-Paul Azar
Why is Kafka so fast? Why is Kafka so popular? Why Kafka? This slide deck is a tutorial for the Kafka streaming platform. This slide deck covers Kafka Architecture with some small examples from the command line. Then we expand on this with a multi-server example to demonstrate failover of brokers as well as consumers. Then it goes through some simple Java client examples for a Kafka Producer and a Kafka Consumer. We have also expanded on the Kafka design section and added references. The tutorial covers Avro and the Schema Registry as well as advance Kafka Producers.
This workshop presentation by Ticketmaster discussed Prometheus and Thanos. it focused on where they fit in in the Cloud Native lanscape and how they're being used.
Kafka Connect: Real-time Data Integration at Scale with Apache Kafka, Ewen Ch...confluent
Many companies are adopting Apache Kafka to power their data pipelines, including LinkedIn, Netflix, and Airbnb. Kafka’s ability to handle high throughput real-time data makes it a perfect fit for solving the data integration problem, acting as the common buffer for all your data and bridging the gap between streaming and batch systems.
However, building a data pipeline around Kafka today can be challenging because it requires combining a wide variety of tools to collect data from disparate data systems. One tool streams updates from your database to Kafka, another imports logs, and yet another exports to HDFS. As a result, building a data pipeline can take significant engineering effort and has high operational overhead because all these different tools require ongoing monitoring and maintenance. Additionally, some of the tools are simply a poor fit for the job: the fragmented nature of the data integration tools ecosystem lead to creative but misguided solutions such as misusing stream processing frameworks for data integration purposes.
We describe the design and implementation of Kafka Connect, Kafka’s new tool for scalable, fault-tolerant data import and export. First we’ll discuss some existing tools in the space and why they fall short when applied to data integration at large scale. Next, we will explore Kafka Connect’s design and how it compares to systems with similar goals, discussing key design decisions that trade off between ease of use for connector developers, operational complexity, and reuse of existing connectors. Finally, we’ll discuss how standardizing on Kafka Connect can ultimately lead to simplifying your entire data pipeline, making ETL into your data warehouse and enabling stream processing applications as simple as adding another Kafka connector.
eventbrite_kafka_summit_event_logo_v3-035858-edited.png
Build and Deploy Cloud Native Camel Quarkus routes with Tekton and KnativeOmar Al-Safi
In this talk, we will leverage all cloud native stacks and tools to build Camel Quarkus routes natively using GraalVM native-image on Tekton pipeline and deploy these routes to Kubernetes cluster with Knative installed. We will dive into the following topics in the talk: - Introduction to Camel - Introduction to Camel Quarkus - Introduction to GraalVM Native Image - Introduction to Tekon - Introduction to Knative - Demo shows how to deploy end to end a Camel Quarkus route which include the following steps: - Look at whole deployment pipeline for Cloud Native Camel Quarkus routes - Build Camel Quarkus routes with GraalVM native-image on Tekton pipeline. - Deploy Camel Quarkus routes to Kubernetes cluster with Knative Targeted Audience: Users with basic Camel knowledge
This presentation will recount the story of Macys.com (and Bloomingdales.com)'s selection and migration from legacy RDBMS to NoSQL Cassandra in partnership with DataStax.
We'll start with a mercifully brief backgrounder on our website and our business. Then we will go over the various technologies that we considered, as well as our use case-based performance benchmarks that led to the decision to go with Cassandra.
We'll cover the various schema options that we tried and how we settled on the current one. We'll show you a selection of some of our extensive performance tuning benchmarks.
One thing that differentiates this talk from others on Cassandra is Macy's philosophy of "doing more with less." You will see why we emphasize the performance tuning aspects of iterative development when you see how much processing we can support on relatively small configurations.
And, finally, we will wrap up with our "lessons learned" and a brief look at our future plans.
클라우드 네이티브로의 전환이 확산되면서 애플리케이션을 상호 독립적인 최소 구성 요소로 쪼개는 마이크로서비스(microservices) 아키텍쳐가 각광받고 있는데요.
MSA는 애플리케이션의 확장이 쉽고 새로운 기능의 출시 기간을 단축시킬 수 있다는 장점이 있지만,
반면에 애플리케이션이 커지고 동일한 서비스의 여러 인스턴스가 동시에 실행되면 MSA간 통신이 복잡해 진다는 단점이 있습니다.
서비스 메쉬(Service Mesh)는 이러한 MSA의 트래픽 문제를 보완하기 위해 탄생한 기술로,
서비스 간의 네트워크 트래픽 관리에 초점을 맞춘 네트워킹 모델입니다.
서로 다른 애플리케이션이 얼마나 원활하게 상호작용하는지를 기록함으로써 커뮤니케이션을 최적화하고 애플리케이션 확장에 따른 다운 타임을 방지할 수 있습니다.
서비스 메쉬의 탄생 배경과 기능, 그리고 현재 오픈소스로 배포되어 있는 서비스 메쉬 솔루션에 대해 소개합니다.
Step1. Cloud Native Trail Map
Step2. Service Proxy, Discover, & Mesh
Step3. Service Mesh 솔루션
Step4. Service Mesh 구현화면 - Istio / linkerd
Step5. Multi-cluster (linkerd)
From Zero to Hero with Kafka Connect (Robin Moffat, Confluent) Kafka Summit L...confluent
Integrating Apache Kafka with other systems in a reliable and scalable way is often a key part of a streaming platform. Fortunately, Apache Kafka includes the Connect API that enables streaming integration both in and out of Kafka. Like any technology, understanding its architecture and deployment patterns is key to successful use, as is knowing where to go looking when things aren’t working. This talk will discuss the key design concepts within Kafka Connect and the pros and cons of standalone vs distributed deployment modes. We’ll do a live demo of building pipelines with Kafka Connect for streaming data in from databases, and out to targets including Elasticsearch. With some gremlins along the way, we’ll go hands-on in methodically diagnosing and resolving common issues encountered with Kafka Connect. The talk will finish off by discussing more advanced topics including Single Message Transforms, and deployment of Kafka Connect in containers.
History and Basics of containers, LXC, Docker and Kubernetes. This presentation is given to Engineering colleage students at VIT DevFest 2018. Beginner to Intermediate level.
Building zero data loss pipelines with apache kafkaAvinash Ramineni
Kafka is playing an increasingly important role in messaging and streaming systems and is becoming the defacto messaging platform in many enterprises. Managing and maintaining Kafka deployments and tuning the data pipelines for high-performance and scalability can become a challenging task.
In this session, we will discuss the lessons learned and the best practices for achieving zero data loss pipelines.
How netflix manages petabyte scale apache cassandra in the cloudVinay Kumar Chella
At Netflix, we manage petabytes of data in Apache Cassandra which must be reliably accessible to users in mere milliseconds. To achieve this, we have built sophisticated control planes that turn our persistence layer based on Apache Cassandra into a truly self-driving system. We will start with the user interface that Netflix developers use to interact with their Cassandra databases and dive deep into the automation that powers it all. From cluster creation, through scaling up, to cluster death, complex automation drives large fleets of virtual machines hosted on the AWS cloud. First, we will cover the basics of how Netflix deploys Apache Cassandra. In particular, this begins with how we mold Apache Cassandra to the Netflix philosophy of immutable infrastructure, including managing software and hardware upgrades in the face of ever-failing hardware. Then we will explore the concrete techniques needed for such a massive deployment, specifically pull-based control planes and auto-healing strategies. Next, we will cover how Netflix has automated complex but critical Apache Cassandra maintenance tasks such as continuous snapshot backups and always-on anti-entropy repair for keeping our datasets safe and consistent. Both of these systems have gone through multiple architectural evolutions, and we have learned many lessons along the way. Lastly, we will share some of the ways this has gone wrong, and what you can do to avoid them. We will cover a few case studies of major Cassandra outages at Netflix, their root cause, and what we learned from those incidents. At the end of this talk, we hope that participants leave with concrete understanding of the challenges in running massive scale Apache Cassandra as well as solid advice and techniques for building their own self-driving data persistence layer.
Building Cloud-Native App Series - Part 7 of 11
Microservices Architecture Series
Containers Docker Kind Kubernetes Istio
- Pods
- ReplicaSet
- Deployment (Canary, Blue-Green)
- Ingress
- Service
Building Cloud-Native App Series - Part 2 of 11
Microservices Architecture Series
Event Sourcing & CQRS,
Kafka, Rabbit MQ
Case Studies (E-Commerce App, Movie Streaming, Ticket Booking, Restaurant, Hospital Management)
to transfer data in network from one device to another with acceptable accuracy, so the system must guarantee the transmitted data should be identical to received data.
there should be no errors if any error occurs in how many ways it can be detected and corrected
Error Detection and correction concepts in Data communication and networksNt Arvind
single bit , burst error detection and correction in data communication networks , block coding ( hamming code , simple parity check code , Cyclic redundancy check-CRC , checksum , internet checksum etc
Concurrency in Distributed Systems : Leslie Lamport papersSubhajit Sahu
In computer science, concurrency is the ability of different parts or units of a program, algorithm, or problem to be executed out-of-order or in partial order, without affecting the final outcome. This allows for parallel execution of the concurrent units, which can significantly improve overall speed of the execution in multi-processor and multi-core systems. In more technical terms, concurrency refers to the decomposability property of a program, algorithm, or problem into order-independent or partially-ordered components or units.[1]
A number of mathematical models have been developed for general concurrent computation including Petri nets, process calculi, the parallel random-access machine model, the actor model and the Reo Coordination Language.
Neuro-symbolic is not enough, we need neuro-*semantic*Frank van Harmelen
Neuro-symbolic (NeSy) AI is on the rise. However, simply machine learning on just any symbolic structure is not sufficient to really harvest the gains of NeSy. These will only be gained when the symbolic structures have an actual semantics. I give an operational definition of semantics as “predictable inference”.
All of this illustrated with link prediction over knowledge graphs, but the argument is general.
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Albert Hoitingh
In this session I delve into the encryption technology used in Microsoft 365 and Microsoft Purview. Including the concepts of Customer Key and Double Key Encryption.
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Jeffrey Haguewood
Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows.
We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases.
This video focuses on the notifications, alerts, and approval requests using Slack for Bonterra Impact Management. The solutions covered in this webinar can also be deployed for Microsoft Teams.
Interested in deploying notification automations for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.
How world-class product teams are winning in the AI era by CEO and Founder, P...
Impossibility of Consensus with One Faulty Process - Papers We Love SF
1. Papers We Love
San Francisco Edition
July 24th, 2014
Henry Robinson
henry@cloudera.com / @henryr
2. • Software engineer at Cloudera since 2009
• My interests are in databases and distributed
systems
• I write about them - in particular, about papers in
those areas - at http://the-paper-trail.org
3. Papers We Love
San Francisco Edition
July 24th, 2014
Henry Robinson
henry@cloudera.com / @henryr
4. Papers We Love
San Francisco Edition
July 24th, 2014
Henry Robinson
henry@cloudera.com / @henryr
Papers of which we
are quite fond
5. • Impossibility of Distributed
Consensus with One Faulty
Process, by Fischer, Lynch
and Paterson (1985)
• Dijkstra award winner 2001
6. • Walk through the proof (leaving rigour for the paper
itself)
• Show how this gives rise to a framework for thinking
about distributed systems
8. • Consensus is the problem of having a set of
processes agree on a value proposed by one of
those processes
9. • Validity: the value agreed upon must have been
proposed by some process
• Termination: at least one non-faulty process
eventually decides
• Agreement: all deciding processes agree on the
same value
10. • Validity: the value agreed upon must have been
proposed by some process - safety
• Termination: at least one non-faulty process
eventually decides - liveness
• Agreement: all deciding processes agree on the
same value - safety
12. Replicated State Machines
Client
Node 1
Node 2
Node 3
N-2N-3
N =
S
N-1
N-2N-3
N =
S
N-1
N-2N-3
N =
S
N-1
1: Client proposes !
state N should !
be S
2: Magic consensus !
protocol
3: New state written to!
log
31. • The system model is the abstraction we layer over
messy computers and networks in order to actually
reason about them.
32. • Message deliveries are the only way that nodes
may communicate
• Messages are delivered in any order
• But are never lost (c.f. crash model vs. omission
model), and are always delivered exactly once
33. • Nodes do not have access to a shared clock.
• So cannot mutually estimate the passage of time
• Messages are the only way that nodes may co-
ordinate with each other
35. Some definitions
• Configuration: the state of every node in the system,
plus the set of undelivered (but sent) messages!
• Initial configuration: what each node in the system
would propose as the decisions at time 0
• Univalent: a state from which only one decision is
possible, no matter what messages are received (0-
valent and 1-valent can only decide 0 or 1 respectively)
• Bivalent: a state from which either decision value is still
possible.
42. 2-node system
C: 00!
V: 1
C: 01!
V: 0
C: 11!
V: 0
C: 10!
V: 1
(C:XY means process 0 has initial value X, process 1 has initial value
Y)
43. 2-node system
C: 00!
V: 1
C: 01!
V: 0
C: 11!
V: 0
C: 10!
V: 1
These two configurations
differ only at one node, but their
valencies are different
(C:XY means process 0 has initial value X, process 1 has initial value
Y)
44. 2-node system
C: 00!
V: 1
C: 01!
V: 0
C: 11!
V: 0
C: 10!
V: 1
These two configurations
differ only at one node, but their
valencies are different
(C:XY means process 0 has initial value X, process 1 has initial value
Y)
45. 2-node system
C: 00!
V: 1
C: 01!
V: 0
C: 11!
V: 0
C: 10!
V: 1
(C:XY means process 0 has initial value X, process 1 has initial value
Y)
I decided 1!
All executions of the protocol -
i.e. set of messages delivered
46. 2-node system
C: 00!
V: 1
C: 01!
V: 0
C: 11!
V: 0
C: 10!
V: 1
(C:XY means process 0 has initial value X, process 1 has initial value
Y)
I decided 0!
All executions of the protocol -
i.e. set of messages delivered
47. 2-node system
C: 00!
V: 1
C: 01!
V: 0
C: 11!
V: 0
C: 10!
V: 1
(C:XY means process 0 has initial value X, process 1 has initial value
Y)
I decided 0!
What if process 1 fails? Are the
configurations any different?
I decided 1!
48. 2-node system
C: 00!
V: 1
C: 01!
V: 0
C: 11!
V: 0
C: 10!
V: 1
(C:XY means process 0 has initial value X, process 1 has initial value
Y)
I decided 0!
For the remaining processes:
no difference in initial state, but
different outcome ?!
I decided 1!
Same
execution
53. • Consider the possibilities:
• If one of those configurations in D is bivalent,
we’re done
• Otherwise show that lack of bivalent state leads
to contradiction
• Do this by first showing that there must be both
0-valent and 1-valent configurations in D
• and that this leads to a contradiction
54. D
Configuration
C (bivalent)
0-valent!
e not received
0-valent!
e received
Either the protocol goes through
D before it reaches the 0-valent
configuration…
2. e is received
1. C moves to
0-valent configuration
before receiving e
55. D
Configuration
C (bivalent)
0-valent!
e not received
0-valent!
e received
Or the protocol gets to the
0-valent configuration after
receiving e
in which case this state also
must be 0-valent and in D
1. e is received
2. 0-valent state is arrived at
57. • There must be two configurations C0 and C1 that
are separated by a single message m where
receiving e in Ci moves the configuration to Di
• We will write that as Ci + e = Di
• So C0 + m = C1
• and C0 + m + e = C1 + e = D1
• and C0 + e = D0
58. • Now consider the destinations of m and e. If they
go to different processes, their receipt is
commutative
• C0 + m + e = D1
• C0 + e + m = D0 + m = D1
• Contradiction: D0 is 0-valent!
59. • Instead, e and m might go to the same process p.
• Consider a deciding computation R from the
original bivalent state C, where p does nothing (i.e.
looks like it failed)
• Since to get to D0 and D1, only e and m have been
received, only p took any steps to get there.
• So R can apply to both D0 and D1.
60. • Since D0 and D1 are both univalent, so the
configurations D0 + R and D1 + R are both
univalent.
61. • Now remember:
• A = C + R
• D1 = C + m + e
• D0 = C + e
• But what about
• C + R + m + e = A + m + e = D1 + R => 1-valent
• C + R + e = A + e = D0 + R => 0-valent
62. • Let e be some event that might be sent in configuration C. Then let D be the set
of all configurations where e is received last and let C be the set of
configurations where e has not been received.
• D either contains a bivalent configuration, or both 0- and 1-valent
configurations. If it contains a bivalent configuration, we’re done. So assume it
does not.
• Now there must be some C0 and C1 in C where C0 + e is 0-valent, but C1 + e
is 1-valent, and C1 = C0 + e’
• Consider two possibilities for the destination of e’ and e. If they are not the
same, then we can say C0 + e + e’ == C0 + e’ + e = C1 + e = D1 -> 1-valent.
But C0 + e -> 0-valent.
• If they are the same, then let A be the configuration reached by a deciding run
from C0 when p does nothing (looks like it failed). We can also apply that run
from D0 and D1 to get to E0 and E1. But we can get from A to either E0 or E1
by applying e or e’ + e. This is a contradiction.
64. !
“These results do not show that such problems
cannot be “solved” in practice; rather, they
point up the need for more refined models of
distributed computing that better reflect realistic
assumptions about processor and
communication timings, and for less stringent
requirements on the solution to such problems.
(For example, termination might be required
only with probability 1.) “
65. Paxos
• Paxos cleverly defers to its leader election scheme
• If leader election is perfect, so is Paxos!
• But perfect leader election is solvable iff consensus
is.
• Impossibilities all the way down…
66. Randomized Consensus
• Nice way to circumvent technical impossibilities:
make their probability vanishingly small
• Ben-Or gave an algorithm that terminates with
probability 1
• (But the rate of convergence might be high)
67. Failure Detectors
• Deep connection between the ability to tell if a
machine has failed, and consensus.
• Lots of research into ‘weak’ failure detectors, and
how weak they can be and still solve consensus
69. • FLP and CAP are not the same thing (see http://the-
paper-trail.org/blog/flp-and-cap-arent-the-same-
thing/)
• FLP is a stronger result, because the system model
has fewer restrictions (crash stop vs omission)
