In this talk, we explore how to construct resilient distributed systems on top of unreliable components. Starting, almost two decades ago, with Leslie Lamport’s work on organising parliament for a Greek island. We will take a journey to today’s datacenters and the systems powering companies like Google, Amazon and Microsoft. Along the way, we will face interesting impossibility results, machines acting maliciously and the complexity of today’s networks. Ultimately, we will discover how to reach agreement between many parties and from this, how to construct new fault-tolerance systems that we can depend upon everyday.
Reaching reliable agreement in an unreliable worldHeidi Howard
In this lecture, we explore how to construct resilient distributed systems on top of unreliable components. Starting, almost two decades ago, with Leslie Lamport’s work on organising parliament for a Greek island. We will take a journey to today’s datacenter and the systems powering companies like Google, Amazon and Microsoft. Along the way, we will face interesting impossibility results, machines acting maliciously and the complexity to today’s networks. We will discover how to reach agreement between many parties and from this, how we construct the fault-tolerance systems that we depend upon everyday.
This lecture was given on October 13th 2015 at the University of Cambridge, as part of the Research Students Lecture Series.
Flexible Paxos: Reaching agreement without majorities Heidi Howard
The Paxos algorithm is a widely adopted approach to achieving distributed consensus. Over decades it has been extensively researched, optimized and deployed in popular systems such as Raft, Zookeeper and Chubby. At its foundation, Paxos uses two phases, each requiring agreement from a majority of participants (known as quorums) to reliably reach consensus. In this talk, I will share the simple yet powerful result that each of the phases of Paxos may use non-intersecting quorums. This means that majorities are no longer necessary and that Paxos is a single point on a broad spectrum of possibilities for safely reaching consensus. This result therefore opens the door for a new breed of scalable and resilient consensus algorithms for performant production system.
Distributed Consensus: Making Impossible PossibleHeidi Howard
In this talk, we explore how to construct resilient distributed systems on top of unreliable components. Starting, almost two decades ago, with Leslie Lamport’s work on organising parliament for a Greek island. We will take a journey to today’s datacenters and the systems powering companies like Google, Amazon and Microsoft. Along the way, we will face interesting impossibility results, machines acting maliciously and the complexity of today’s networks. Ultimately, we will discover how to reach agreement between many parties and from this, how to construct new fault-tolerance systems that we can depend upon everyday.
This slide deck tries to evolve the challenges a developer can go through while designing a distributed system. It also explains a consensus algorithm called RAFT. Few slides also describe about consistent distributed key value store concepts and project like etcd using all these concepts.
Replication and Synchronization Algorithms for Distributed Databases - Lena W...distributed matters
This talk will provide in-depth background on strategies for replication and synchronization as implemented in modern distributed databases. The following topics will be covered: master-slave vs multi-master replication; epidemic protocols; two-phase commit vs Paxos; multiversion concurrency control; read and write quorums. A concise overview of implementations in current NoSQL databases will be presented.
Distributed Consensus A.K.A. "What do we eat for lunch?"Konrad Malawski
Distributed Consensus is everywhere! Even if not obvious at first, most apps nowadays are distributed systems, and these sometimes have to "agree on a value", this is where consensus algorithms come in. In this session we'll look at the general problem and solve a few example cases using the RAFT algorithm implemented using Akka's Actor and Cluster modules.
Infrastructure microservices such as Service Discovery and Routing need mechanisms to manage distributed state. The most common answer for this is the use of a central, consistent key value store such as Consul, Etcd or Zookeeper. These systems use consensus-based algorithms, such as Raft or Paxos, to provide consistency and failure tolerance.
I believe this is a dangerous direction for our industry, and instead we should be focusing on ease of use and reliabiligy. As such, Weave wants its infrastructure microservices to be decentralized, yet easy to install and run; our approach is inspired by the Internet, which is distributed and operates with no consensus. In this talk, I'll explain how we designed our Service Discovery and Address Management using Convergent Replicated Data Types (CRDTs) and Gossip, review the pros and cons of this concept, and how it compares to alternatives.
Reaching reliable agreement in an unreliable worldHeidi Howard
In this lecture, we explore how to construct resilient distributed systems on top of unreliable components. Starting, almost two decades ago, with Leslie Lamport’s work on organising parliament for a Greek island. We will take a journey to today’s datacenter and the systems powering companies like Google, Amazon and Microsoft. Along the way, we will face interesting impossibility results, machines acting maliciously and the complexity to today’s networks. We will discover how to reach agreement between many parties and from this, how we construct the fault-tolerance systems that we depend upon everyday.
This lecture was given on October 13th 2015 at the University of Cambridge, as part of the Research Students Lecture Series.
Flexible Paxos: Reaching agreement without majorities Heidi Howard
The Paxos algorithm is a widely adopted approach to achieving distributed consensus. Over decades it has been extensively researched, optimized and deployed in popular systems such as Raft, Zookeeper and Chubby. At its foundation, Paxos uses two phases, each requiring agreement from a majority of participants (known as quorums) to reliably reach consensus. In this talk, I will share the simple yet powerful result that each of the phases of Paxos may use non-intersecting quorums. This means that majorities are no longer necessary and that Paxos is a single point on a broad spectrum of possibilities for safely reaching consensus. This result therefore opens the door for a new breed of scalable and resilient consensus algorithms for performant production system.
Distributed Consensus: Making Impossible PossibleHeidi Howard
In this talk, we explore how to construct resilient distributed systems on top of unreliable components. Starting, almost two decades ago, with Leslie Lamport’s work on organising parliament for a Greek island. We will take a journey to today’s datacenters and the systems powering companies like Google, Amazon and Microsoft. Along the way, we will face interesting impossibility results, machines acting maliciously and the complexity of today’s networks. Ultimately, we will discover how to reach agreement between many parties and from this, how to construct new fault-tolerance systems that we can depend upon everyday.
This slide deck tries to evolve the challenges a developer can go through while designing a distributed system. It also explains a consensus algorithm called RAFT. Few slides also describe about consistent distributed key value store concepts and project like etcd using all these concepts.
Replication and Synchronization Algorithms for Distributed Databases - Lena W...distributed matters
This talk will provide in-depth background on strategies for replication and synchronization as implemented in modern distributed databases. The following topics will be covered: master-slave vs multi-master replication; epidemic protocols; two-phase commit vs Paxos; multiversion concurrency control; read and write quorums. A concise overview of implementations in current NoSQL databases will be presented.
Distributed Consensus A.K.A. "What do we eat for lunch?"Konrad Malawski
Distributed Consensus is everywhere! Even if not obvious at first, most apps nowadays are distributed systems, and these sometimes have to "agree on a value", this is where consensus algorithms come in. In this session we'll look at the general problem and solve a few example cases using the RAFT algorithm implemented using Akka's Actor and Cluster modules.
Infrastructure microservices such as Service Discovery and Routing need mechanisms to manage distributed state. The most common answer for this is the use of a central, consistent key value store such as Consul, Etcd or Zookeeper. These systems use consensus-based algorithms, such as Raft or Paxos, to provide consistency and failure tolerance.
I believe this is a dangerous direction for our industry, and instead we should be focusing on ease of use and reliabiligy. As such, Weave wants its infrastructure microservices to be decentralized, yet easy to install and run; our approach is inspired by the Internet, which is distributed and operates with no consensus. In this talk, I'll explain how we designed our Service Discovery and Address Management using Convergent Replicated Data Types (CRDTs) and Gossip, review the pros and cons of this concept, and how it compares to alternatives.
Distributed Consensus: Making Impossible Possible by Heidi howardJ On The Beach
In this talk, we explore how to construct resilient distributed systems on top of unreliable components. Starting, almost two decades ago, with Leslie Lamport’s work on organising parliament for a Greek island. We will take a journey to today’s datacenters and the systems powering companies like Google, Amazon and Microsoft. Along the way, we will face interesting impossibility results, machines acting maliciously and the complexity of today’s networks. Ultimately, we will discover how to reach agreement between many parties and from this, how to construct new fault-tolerance systems that we can depend upon everyday.
Distributed Consensus: Making the Impossible PossibleC4Media
Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/29faOI5.
Heidi Howard explores how to construct resilient distributed systems on top of unreliable components. Howard discusses today’s datacenters and the systems powering companies like Google, Amazon and Microsoft, as well as which algorithms are best suited to different situations. Filmed at qconlondon.com.
Heidi Howard is currently studying towards PhD at the Cambridge University, Computer Lab. Her research interest is fault-tolerance, consistency and consensus in modern distributed systems. Heidi has also previously worked as research assistant and undergraduate researcher on topics such as middlebox traversal, DNS, privacy preserving systems and wireless community networks.
Distributed Consensus: Making Impossible Possible by Heidi howardJ On The Beach
In this talk, we explore how to construct resilient distributed systems on top of unreliable components. Starting, almost two decades ago, with Leslie Lamport’s work on organising parliament for a Greek island. We will take a journey to today’s datacenters and the systems powering companies like Google, Amazon and Microsoft. Along the way, we will face interesting impossibility results, machines acting maliciously and the complexity of today’s networks. Ultimately, we will discover how to reach agreement between many parties and from this, how to construct new fault-tolerance systems that we can depend upon everyday.
Distributed Consensus: Making the Impossible PossibleC4Media
Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/29faOI5.
Heidi Howard explores how to construct resilient distributed systems on top of unreliable components. Howard discusses today’s datacenters and the systems powering companies like Google, Amazon and Microsoft, as well as which algorithms are best suited to different situations. Filmed at qconlondon.com.
Heidi Howard is currently studying towards PhD at the Cambridge University, Computer Lab. Her research interest is fault-tolerance, consistency and consensus in modern distributed systems. Heidi has also previously worked as research assistant and undergraduate researcher on topics such as middlebox traversal, DNS, privacy preserving systems and wireless community networks.
There have never been more commercial tools available for building distributed data apps — from cloud hosting services, to cloud-native databases, to cloud-based analytics platforms. So why is it still so hard to make a successful app with a global user base?
One of the toughest challenges cloud offerings take on is the problem of consensus, abstracting away most of the complexity. That's no small feat, given that this is a hard enough problem that people spend years getting a PhD just to understand it! Unfortunately, while buying off-the-shelf cloud services can accelerate the path to an MVP, it also makes optimization tough. How will we scale during a period of rapid user growth? How do we do I18n and l10n or guarantee a good UX for users on the other side of the world? How do we prevent replication that might get us into legal trouble?
In this talk, we'll consider several case studies of global apps (both successful and otherwise!), talk about the limitations of off-the-shelf consensus, and consider a future where everyday developers can use open source tools to build distributed data apps that are easier to reason about, maintain, and tune.
Shared on 5th Dec at SGInnovate with Swirlds Mance Harmon, Jordan Fried and Edgar Seah.
Hashgraph consensus, demo apps in Swirlds Java SDK, babble (unofficial golang implementation of Hashgraph) and their implications for distributed ledger technology.
[Paperreading] Paxos made easy (by sen han)PingCAP
This is a sub-project of open-source project TuringCell. Turing Cell Model is a computing model running on top of distributed consistency algorithms (such as Paxos/Raft). TuringCell is an open-source implementation of Turing Cell Model. This means that you can add features as high-availability, fault-tolerance and strong-consistency to existing software very easily. At the same time, TuringCell is an industry-friendly project, at its core is the force from an open, tolerant community. Wherever you are from, whichever language you speak, you are all welcome to equally join in, discuss and build TuringCell !
Paper: https://github.com/turingcell/paxos-made-easy/blob/feature/translation/README_en.md
An overview of Haystack's security features for low power IoT networks. Unlike most IoT stacks, when Haystack invented DASH7, security was an a priori principle and led to the most secure networking stack available in the low power, wide area networking (LPWAN) space today.
This was a talk, largely on Kamaelia & its original context given at a Free Streaming Workshop in Florence, Italy in Summer 2004. Many of the core
concepts still hold valid in Kamaelia today
In this slide deck, we go exploring the database landscape today and the common lego blocks that are used to build these different falvours of databses. We will dive through internals of a database, explore some choices and towards the end also explore some real world database architectures in view of the concepts (legos) we explored earlier.
Beyond the RTOS: A Better Way to Design Real-Time Embedded SoftwareQuantum Leaps, LLC
Embedded software developers from different industries are independently re-discovering patterns for building concurrent software that is safer, more responsive and easier to understand than naked threads of a Real-Time Operating System (RTOS). These best practices universally favor event-driven, asynchronous, non-blocking, encapsulated state machines instead of naked, blocking RTOS threads. This presentation explains the concepts related to this increasingly popular "reactive approach", and specifically how they apply to real-time embedded systems.
This is the keynote talk i delivered at GeekCamp.SG 2014
The main purpose of the talk is to create an awareness, if not existent, in the community when it comes to choosing and wanting to building a distributed system.
This presentation is not meant to be a survey of distributed computing through the ages but hopefully it serves as a good starting point in which the journeyman can start from.
I want to thank Jonas, CTO of Typesafe, as his work in Akka strongly influenced my own and i hope it would help you in the way his work helped me.
For further details contact:
N.RAJASEKARAN B.E M.S 9841091117,9840103301.
IMPULSE TECHNOLOGIES,
Old No 251, New No 304,
2nd Floor,
Arcot road ,
Vadapalani ,
Chennai-26.
This presentation overviews basic principles of high availability architectures and presents how to deploy in high availability FIWARE data management services.
Similar to Distributed Consensus: Making Impossible Possible [Revised] (20)
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...Ramesh Iyer
In today's fast-changing business world, Companies that adapt and embrace new ideas often need help to keep up with the competition. However, fostering a culture of innovation takes much work. It takes vision, leadership and willingness to take risks in the right proportion. Sachin Dev Duggal, co-founder of Builder.ai, has perfected the art of this balance, creating a company culture where creativity and growth are nurtured at each stage.
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
Key Trends Shaping the Future of Infrastructure.pdfCheryl Hung
Keynote at DIGIT West Expo, Glasgow on 29 May 2024.
Cheryl Hung, ochery.com
Sr Director, Infrastructure Ecosystem, Arm.
The key trends across hardware, cloud and open-source; exploring how these areas are likely to mature and develop over the short and long-term, and then considering how organisations can position themselves to adapt and thrive.
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
UiPath Test Automation using UiPath Test Suite series, part 3DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 3. In this session, we will cover desktop automation along with UI automation.
Topics covered:
UI automation Introduction,
UI automation Sample
Desktop automation flow
Pradeep Chinnala, Senior Consultant Automation Developer @WonderBotz and UiPath MVP
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
UiPath Test Automation using UiPath Test Suite series, part 4DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap.
The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies.
Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques
What will you get from this session?
1. Insights into SAP testing best practices
2. Heatmap utilization for testing
3. Optimization of testing processes
4. Demo
Topics covered:
Execution from the test manager
Orchestrator execution result
Defect reporting
SAP heatmap example with demo
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
2. Sometimes inconsistency is
not an option
• Distributed locking
• Financial services/ blockchain
• Safety critical systems
• Distributed scheduling and coordination
• Strongly consistent databases
Anything which requires guaranteed agreement
3. What is Consensus?
“The process by which we reach agreement over
system state between unreliable machines connected
by asynchronous networks”
4.
5. A walk through history
We are going to take a journey
through the developments in
distributed consensus, spanning
3 decades.
Bob
6. FLP Result
off to a slippery start
Impossibility of distributed
consensus with one faulty process
Michael Fischer, Nancy Lynch
and Michael Paterson
ACM SIGACT-SIGMOD
Symposium on Principles of
Database Systems
1983
7. FLP
We cannot guarantee agreement in an asynchronous
system where even one host might fail.
Why?
We cannot reliably detect failures. We cannot know
for sure the difference between a slow host/network
and a failed host
Note: We can still guarantee safety, the issue limited
to guaranteeing liveness.
8. Solution to FLP
In practice:
We accept that sometimes the system will not be
available. We mitigate this using timers and backoffs.
In theory:
We make weaker assumptions about the synchrony
of the system e.g. messages arrive within a year.
10. Viewstamped Replication
(Revisited)
In my view, the pioneer on the field of consensus.
Let one node be the ‘master’, rotating when failures
occur. Replicate requests for a state machine.
Now considered a variant of SMR + Multi-Paxos.
12. Paxos
The textbook consensus algorithm for reaching
agreement on a single value.
• two phase process: promise and commit
• each requiring majority agreement (aka quorums)
• 2 RRTs to agreement a single value
38. Paxos
Clients must wait two round trips (2 RTT) to the
majority of nodes. Sometimes longer.
The system will continue as long as a majority of
nodes are up
39. Multi-Paxos
Lamport’s leader-driven consensus algorithm
Paxos Made Moderately Complex
Robbert van Renesse and Deniz
Altinbuken
ACM Computing Surveys
April 2015
Not the original, but highly recommended
40. Multi-Paxos
Lamport’s insight:
Phase 1 is not specific to the request so can be done
before the request arrives and can be reused.
Implication:
Bob now only has to wait one RTT
48. Paxos Made Live
How google uses Paxos
Paxos Made Live - An Engineering
Perspective
Tushar Chandra, Robert Griesemer
and Joshua Redstone
ACM Symposium on Principles of
Distributed Computing
2007
49. Isn’t this a solved problem?
“There are significant gaps between the description
of the Paxos algorithm and the needs of a real-world
system.
In order to build a real-world system, an expert needs
to use numerous ideas scattered in the literature and
make several relatively small protocol extensions.
The cumulative effort will be substantial and the final
system will be based on an unproven protocol.”
50. Paxos Made Live
Paxos made live documents the challenges in
constructing Chubby, a distributed coordination
service, built using Multi-Paxos and SMR.
51. Challenges
• Handling disk failure and corruption
• Dealing with limited storage capacity
• Effectively handling read-only requests
• Dynamic membership & reconfiguration
• Supporting transactions
• Verifying safety of the implementation
52. Fast Paxos
Like Multi-Paxos, but faster
Fast Paxos
Leslie Lamport
Microsoft Research Tech Report
MSR-TR-2005-112
53. Fast Paxos
Paxos: Any node can commit a value in 2 RTTs
Multi-Paxos: The leader node can commit a value in
1 RTT
But, what about any node committing a value in 1
RTT?
54. Fast Paxos
We can bypass the leader node for many operations,
so any node can commit a value in 1 RTT.
However, we must increase the size of the quorum.
55. Zookeeper
The open source solution
Zookeeper: wait-free coordination
for internet-scale systems
Hunt et al
USENIX ATC 2010
Code: zookeeper.apache.org
56. Zookeeper
Consensus for the masses.
It utilizes and extends Multi-Paxos for
strong consistency.
Unlike “Paxos made live”, this is
clearly discussed and openly
available.
57. Egalitarian Paxos
Don’t restrict yourself unnecessarily
There Is More Consensus in
Egalitarian Parliaments
Iulian Moraru, David G. Andersen,
Michael Kaminsky
SOSP 2013
also see Generalized Consensus and Paxos
58. Egalitarian Paxos
The basis of SMR is that every replica of an
application receives the same commands in the
same order.
However, sometimes the ordering can be relaxed…
61. Egalitarian Paxos
Allow requests to be out-of-order if they are
commutative.
Conflict becomes much less common.
Works well in combination with Fast Paxos.
62. Raft Consensus
Paxos made understandable
In Search of an Understandable
Consensus Algorithm
Diego Ongaro and John
Ousterhout
USENIX Annual Technical
Conference
2014
63. Raft
Raft has taken the wider community by storm. Largely,
due to its understandable description.
It’s another variant of SMR with Multi-Paxos.
Key features:
• Really strong leadership - all other nodes are passive
• Various optimizations - e.g. dynamic membership
and log compaction
66. Ios
The issue with leader-driven algorithms like
Viewstamp Replication, Multi-Paxos, Zookeeper and
Raft is that throughput is limited to one node.
Ios allows a leader to safely and dynamically
delegate their responsibilities to other nodes in the
system.
67. Flexible Paxos
Paxos made scalable
Flexible Paxos: Quorum
intersection revisited
Heidi Howard, Dahlia Malkhi,
Alexander Spiegelman
ArXiv:1608.06696
68. Majorities are not needed
Usually, we use require majorities to agree so we can
guarantee that all quorums (groups) intersect.
This work shows that not all quorums need to
intersect. Only the ones used for replication and
leader election.
This applies to all algorithms in this class: Paxos,
Viewstamped Replication, Zookeeper, Raft etc..
69. The road we travelled
• 2 theoretical results: FLP & Flexible Paxos
• 2 popular ideas: CAP & Paxos made live
• 1 replication method: State machine Replication
• 8 consensus algorithms: Viewstamped
Replication, Paxos, Multi-Paxos, Fast Paxos,
Zookeeper, Egalitarian Paxos, Raft & Ios
70. How strong is the
leadership?
Strong
Leadership Leaderless
Paxos
Egalitarian
Paxos
Raft Viewstamped
Replication
Ios
Multi-Paxos
Fast Paxos
Leader with
Delegation
Leader only
when needed
Leader driven
Zookeeper
71. Who is the winner?
Depends on the award:
• Best for minimum latency: Viewstamped
Replication
• Most widely used open source project: Zookeeper
• Easiest to understand: Raft
• Best for WANs: Egalitarian Paxos
72. Future
1. More scalable and performant consensus
algorithms utilizing Flexible Paxos.
2. A clearer understanding of consensus and better
explained consensus algorithms.
3. Achieving consensus in challenge settings such
as geo-replicated systems.
73. Stops we drove passed
We have seen one path through history, but many
more exist.
• Alternative replication techniques e.g. chain
replication and primary backup replication
• Alternative failure models e.g. nodes acting
maliciously
• Alternative domains e.g. sensor networks, mobile
networks, between cores
74. Summary
Do not be discouraged by impossibility
results and dense abstract academic papers.
Don’t give up on consistency. Consensus is
achievable, even performant and scalable (if
done correctly)
Find the right algorithm for your specific
domain.
heidi.howard@cl.cam.ac.uk
@heidiann360