Foundations of cluster construction, more particularly the memberhsip service abstraction for building reliable distributed applications. This talk introduces failure detectors, dissemination strategies and consensus protocols
Modern applications are faced with the expectation of being always up. As a result, an increasing amount of businesses are transitioning to modern architectures such as the one of reactive microservice-based systems. As compelling as this sounds, this also means to embrace distribution so as to ensure fault-tolerance and scalability - which also means embracing the uncertainty and nondeterminism that comes with building networked applications: changes in link quality, network partitions and outages of individual nodes are scenarios that need to be addressed first-hand when designing such systems.
In this talk you will learn about the fundamental mechanisms that clustered systems need in order to operate in this uncertain world, such as failure detection, gossip-based cluster state propagation, split brain resolution and convergent replicated data types (CRDTs). We will look at these mechanisms in the context of Akka Cluster.
By the end of this talk you should have a solid understanding of the trade-offs that need to be made when building clustered applications. You will also get the sense that a networked application is not just one application, but the cooperation between individual processes that need to probe for each other’s availability and decide what to do when one or more of them are maybe unavailable all while not being able to talk with one another.
Is there anybody out there? Reactive Systems HamburgManuel Bernhardt
Foundations of cluster construction, more particularly the memberhsip service abstraction for building reliable distributed applications. This talk introduces failure detectors, dissemination strategies and consensus protocols
This is the keynote talk i delivered at GeekCamp.SG 2014
The main purpose of the talk is to create an awareness, if not existent, in the community when it comes to choosing and wanting to building a distributed system.
This presentation is not meant to be a survey of distributed computing through the ages but hopefully it serves as a good starting point in which the journeyman can start from.
I want to thank Jonas, CTO of Typesafe, as his work in Akka strongly influenced my own and i hope it would help you in the way his work helped me.
The 10 minutes presentation I gave at my PhD defence on 21.9.2015 in Amsterdam. Prof. Frank van Harmelen was my promoter. Prof. Ian Horrocks, prof. Manfred Hauswirth, prof. Geert-Jan Houben, Peter Boncz and prof. Guus Schreiber were my opponents.
Modern applications are faced with the expectation of being always up. As a result, an increasing amount of businesses are transitioning to modern architectures such as the one of reactive microservice-based systems. As compelling as this sounds, this also means to embrace distribution so as to ensure fault-tolerance and scalability - which also means embracing the uncertainty and nondeterminism that comes with building networked applications: changes in link quality, network partitions and outages of individual nodes are scenarios that need to be addressed first-hand when designing such systems.
In this talk you will learn about the fundamental mechanisms that clustered systems need in order to operate in this uncertain world, such as failure detection, gossip-based cluster state propagation, split brain resolution and convergent replicated data types (CRDTs). We will look at these mechanisms in the context of Akka Cluster.
By the end of this talk you should have a solid understanding of the trade-offs that need to be made when building clustered applications. You will also get the sense that a networked application is not just one application, but the cooperation between individual processes that need to probe for each other’s availability and decide what to do when one or more of them are maybe unavailable all while not being able to talk with one another.
Is there anybody out there? Reactive Systems HamburgManuel Bernhardt
Foundations of cluster construction, more particularly the memberhsip service abstraction for building reliable distributed applications. This talk introduces failure detectors, dissemination strategies and consensus protocols
This is the keynote talk i delivered at GeekCamp.SG 2014
The main purpose of the talk is to create an awareness, if not existent, in the community when it comes to choosing and wanting to building a distributed system.
This presentation is not meant to be a survey of distributed computing through the ages but hopefully it serves as a good starting point in which the journeyman can start from.
I want to thank Jonas, CTO of Typesafe, as his work in Akka strongly influenced my own and i hope it would help you in the way his work helped me.
The 10 minutes presentation I gave at my PhD defence on 21.9.2015 in Amsterdam. Prof. Frank van Harmelen was my promoter. Prof. Ian Horrocks, prof. Manfred Hauswirth, prof. Geert-Jan Houben, Peter Boncz and prof. Guus Schreiber were my opponents.
The Computer Science Behind a modern Distributed DatabaseArangoDB Database
What we see in the modern data store world is a race between different approaches to achieve a distributed and resilient storage of data. Every application needs a stateful layer which holds the data. There are several different necessary components which are anything but trivial to combine, and, of course, even more challenging when attempting to optimize for performance. Over the past years there has been significant progress in both the science and practical implementations of such data stores. In this talk Dan Larkin-York will introduce the audience to some of the challenges, address the difficulties of their interplay, and cover key approaches taken by some of the industry’s leaders (ArangoDB, Cassandra, CockroachDB, MarkLogic, and more).
OSDC 2018 | The Computer science behind a modern distributed data store by Ma...NETWAYS
What we see in the modern data store world is a race between different approaches to achieve a distributed and resilient storage of data. Most applications need a stateful layer which holds the data. There are at least three necessary ingredients which are everything else than trivial to combine and of course even more challenging when heading for an acceptable performance. Over the past years there has been significant progress in respect in both the science and practical implementations of such data stores. In his talk Max Neunhoeffer will introduce the audience to some of the needed ingredients, address the difficulties of their interplay and show four modern approaches of distributed open-source data stores.
Topics are:
– Challenges in developing a distributed, resilient data store
– Consensus, distributed transactions, distributed query optimization and execution
– The inner workings of ArangoDB, Cassandra, Cockroach and RethinkDB
The talk will touch complex and difficult computer science, but will at the same time be accessible to and enjoyable by a wide range of developers.
The computer science behind a modern disributed data storeJ On The Beach
What we see in the modern data store world is a race between different approaches to achieve a distributed and resilient storage of data. Every application needs a stateful layer which holds the data. There are at least three necessary components which are everything else than trivial to combine, and, of course, even more challenging when heading for an acceptable performance.
Over the past years there has been significant progress in both the science and practical implementations of such data stores. In his talk Max Neunhoeffer will introduce the audience to some of the needed ingredients, address the difficulties of their interplay and show four modern approaches of distributed open-source data stores (ArangoDB, Cassandra, Cockroach and RethinkDB).
On the topology of package dependency networks: A comparison of programming l...Tom Mens
This presentation is joint work by Alexandre Decan, Tom Mens and Maelick Claes (Software Engineering Lab, COMPLEXYS research institute, University of Mons). It was presented by Maelick during the International Workshop on Software Ecosystem Architectures (WEA 2016) in Copenhagen, on 29 November 2016.
Abstract of the accompanying paper (DOI 10.1145/1235):
Package-based software ecosystems are composed of thousands of interdependent software packages. Many empirical studies have focused on software packages belonging to a single software ecosystem, and suggest to generalise the results to more ecosystems. We claim that such a generalisation is not always possible, because the technical structure of software ecosystems can be very different, even if these ecosystems belong to the same domain. We confirm this claim through a study of three big and popular package-based programming language ecosystems: R’s CRAN archive network, Python’s PyPI distribution, and JavaScript’s NPM package manager. We study and compare the structure of their package dependency graphs and reveal some important differences that may make it difficult to generalise the findings of one ecosystem to another one.
A follow-up on this work can be found in the SANER 2017 paper by the same authors, entitled "An Empirical Comparison of Dependency Issues in OSS Packaging Ecosystems”
Distributed systems involve complex interactions among many components. This increases the possibilities of failures that could turn a whole system down. Software architects, designers, and developers need to architect, design, and program functional requirements thinking about possibility of failures and the need for a system to keep running despite failures. This presentation tackles but part of the problem, focusing on redundancy, different types of groups, replication, and eventual consistency, finishing with the presentation of CAP theorem.
Presentation delivered at IV Cloud Computing and Big Data Ent at Universdad Nacional de La Plata http://www.jcc.info.unlp.edu.ar/jcc2016/wordpress/index.php/cronograma/
Scala Days Copenhagen - 8 Akka anti-patterns you'd better be aware ofManuel Bernhardt
CAUTION: If you are responsible for an Akka system deployed to production, attending this talk may cause intense moments of self-doubt, stress and possibly panic.
Akka is a toolkit for building highly concurrent and distributed applications on the JVM using the actor model. Given the prevalence of frameworks over toolkits and models in the industry, it is easy to forget that the former will not prevent you from using them in any way you please – including ways that are possibly suboptimal or perhaps even harmful.
In this talk you'll learn about a few of the most common anti-patterns related to Akka usage. You'll also get to know about alternative and more appropriate solutions to use for each one of those anti-patterns. It should be noted that these suboptimal uses of Akka are not merely theoretical ponderings but real and recurring observations that the speaker made during a range of consulting projects.
More Related Content
Similar to Is there anybody out there? Scala Days Berlin 2018
The Computer Science Behind a modern Distributed DatabaseArangoDB Database
What we see in the modern data store world is a race between different approaches to achieve a distributed and resilient storage of data. Every application needs a stateful layer which holds the data. There are several different necessary components which are anything but trivial to combine, and, of course, even more challenging when attempting to optimize for performance. Over the past years there has been significant progress in both the science and practical implementations of such data stores. In this talk Dan Larkin-York will introduce the audience to some of the challenges, address the difficulties of their interplay, and cover key approaches taken by some of the industry’s leaders (ArangoDB, Cassandra, CockroachDB, MarkLogic, and more).
OSDC 2018 | The Computer science behind a modern distributed data store by Ma...NETWAYS
What we see in the modern data store world is a race between different approaches to achieve a distributed and resilient storage of data. Most applications need a stateful layer which holds the data. There are at least three necessary ingredients which are everything else than trivial to combine and of course even more challenging when heading for an acceptable performance. Over the past years there has been significant progress in respect in both the science and practical implementations of such data stores. In his talk Max Neunhoeffer will introduce the audience to some of the needed ingredients, address the difficulties of their interplay and show four modern approaches of distributed open-source data stores.
Topics are:
– Challenges in developing a distributed, resilient data store
– Consensus, distributed transactions, distributed query optimization and execution
– The inner workings of ArangoDB, Cassandra, Cockroach and RethinkDB
The talk will touch complex and difficult computer science, but will at the same time be accessible to and enjoyable by a wide range of developers.
The computer science behind a modern disributed data storeJ On The Beach
What we see in the modern data store world is a race between different approaches to achieve a distributed and resilient storage of data. Every application needs a stateful layer which holds the data. There are at least three necessary components which are everything else than trivial to combine, and, of course, even more challenging when heading for an acceptable performance.
Over the past years there has been significant progress in both the science and practical implementations of such data stores. In his talk Max Neunhoeffer will introduce the audience to some of the needed ingredients, address the difficulties of their interplay and show four modern approaches of distributed open-source data stores (ArangoDB, Cassandra, Cockroach and RethinkDB).
On the topology of package dependency networks: A comparison of programming l...Tom Mens
This presentation is joint work by Alexandre Decan, Tom Mens and Maelick Claes (Software Engineering Lab, COMPLEXYS research institute, University of Mons). It was presented by Maelick during the International Workshop on Software Ecosystem Architectures (WEA 2016) in Copenhagen, on 29 November 2016.
Abstract of the accompanying paper (DOI 10.1145/1235):
Package-based software ecosystems are composed of thousands of interdependent software packages. Many empirical studies have focused on software packages belonging to a single software ecosystem, and suggest to generalise the results to more ecosystems. We claim that such a generalisation is not always possible, because the technical structure of software ecosystems can be very different, even if these ecosystems belong to the same domain. We confirm this claim through a study of three big and popular package-based programming language ecosystems: R’s CRAN archive network, Python’s PyPI distribution, and JavaScript’s NPM package manager. We study and compare the structure of their package dependency graphs and reveal some important differences that may make it difficult to generalise the findings of one ecosystem to another one.
A follow-up on this work can be found in the SANER 2017 paper by the same authors, entitled "An Empirical Comparison of Dependency Issues in OSS Packaging Ecosystems”
Distributed systems involve complex interactions among many components. This increases the possibilities of failures that could turn a whole system down. Software architects, designers, and developers need to architect, design, and program functional requirements thinking about possibility of failures and the need for a system to keep running despite failures. This presentation tackles but part of the problem, focusing on redundancy, different types of groups, replication, and eventual consistency, finishing with the presentation of CAP theorem.
Presentation delivered at IV Cloud Computing and Big Data Ent at Universdad Nacional de La Plata http://www.jcc.info.unlp.edu.ar/jcc2016/wordpress/index.php/cronograma/
Scala Days Copenhagen - 8 Akka anti-patterns you'd better be aware ofManuel Bernhardt
CAUTION: If you are responsible for an Akka system deployed to production, attending this talk may cause intense moments of self-doubt, stress and possibly panic.
Akka is a toolkit for building highly concurrent and distributed applications on the JVM using the actor model. Given the prevalence of frameworks over toolkits and models in the industry, it is easy to forget that the former will not prevent you from using them in any way you please – including ways that are possibly suboptimal or perhaps even harmful.
In this talk you'll learn about a few of the most common anti-patterns related to Akka usage. You'll also get to know about alternative and more appropriate solutions to use for each one of those anti-patterns. It should be noted that these suboptimal uses of Akka are not merely theoretical ponderings but real and recurring observations that the speaker made during a range of consulting projects.
In this talk you'll learn about some of the many ways in which you can shoot yourself in the foot using the Akka toolkit. The anti-patterns are based on observations made in real-life consulting projects.
Beyond the buzzword: a reactive web-appliction in practiceManuel Bernhardt
This talk & live-coding session gives an insight into the why, what and how of reactive applications, and web-applications in particular. The first part is theoretical, whilst the second part is a live-coding session introducing the Play Framework, Akka and the Scala programming language all whilst demonstrating reactive patterns such as Futures, Actors, Pipes and Circuit Breakers.
Beyond the Buzzword - a reactive application in practiceManuel Bernhardt
Beyond the Buzzword presentation about building reactive web applications with Play Framework, Akka and Scala at the Slovak Scala User's Group, 26th of May 2016
3 things you must know to think reactive - Geecon Kraków 2015Manuel Bernhardt
Over the past few years, web-applications have started to play an increasingly important role in our lives. We expect them to be always available and the data to be always fresh. This shift into the realm of real-time data processing is now transitioning to physical devices, and Gartner predicts that the Internet of Things will grow to an installed base of 26 billion units by 2020.
As reactive architectures gain in popularity, more and more developers find themselves faced with the challenge of "thinking reactive". To leave behind the well-known concepts of mutable, object-oriented, imperative and synchronous programming in favour of immutable, functional, declarative and asynchronous programming requires quite a mind shift and it isn't obvious to take the plunge.
In this talk we will explore three concepts from the world of functional programming that are at the core of building reactive applications: immutability, higher-order functions and manipulating immutable collections. We will first see how the "traditional" mutable, object-oriented approach of doing things can be problematic when it comes to multi-core programming, and then how to apply them to asynchronous systems.
Over the past few years, web-applications have started to play an increasingly important role in our lives. We expect them to be always available and the data to be always fresh. This shift into the realm of real-time data processing is now transitioning to physical devices, and Gartner predicts that the Internet of Things will grow to an installed base of 26 billion units by 2020.
Reactive web-applications are an answer to the new requirements of high-availability and resource efficiency brought by this rapid evolution. On the JVM, a set of new languages and tools has emerged that enable the development of entirely asynchronous request and data handling pipelines. At the same time, container-less application frameworks are gaining increasing popularity over traditional deployment mechanisms.
This talk is going to give you an introduction into one of the most trending reactive web-application stack on the JVM, involving the Scala programming language, the concurrency toolkit Akka and the web-application framework Play. It will show you how functional programming techniques enable asynchronous programming, and how those technologies help to build robust and resilient web-applications.
My experience writing "Reactive Web-Applications with Play". This is my subjective view on writing a book with Manning. I'm not finished yet so there will likely be a second version of this presentation when the book is out.
Note if you weren't at the talk: the "know nothing" part refers to discovering the depth of a topic while writing, which I find to be one of the nicest things about writing (because I really like the subject matter)
Project Phoenix - From PHP to the Play Framework in 3 monthsManuel Bernhardt
This is an experience report about Project Phoenix, aiming at porting a platform to the Play Framework with Scala in the short time period of 3 months. The presentation was given at Devoxx UK 2014
This is a short introduction to the Scala programming language and its ecosystem, giving some background about how the language came to be and a quick taste of what it looks like. It also shortly introduces a few tools and technologies related to it
Introduction to the Scala programming language @ the Java Klassentreffen 2013 in Vienna and Linz organized by http://www.javaklassentreffen.at
Introducing the background, design goals, and some examples (from a Java perspective)
http://java.at/web/guest/jkt13
Harnessing WebAssembly for Real-time Stateless Streaming PipelinesChristina Lin
Traditionally, dealing with real-time data pipelines has involved significant overhead, even for straightforward tasks like data transformation or masking. However, in this talk, we’ll venture into the dynamic realm of WebAssembly (WASM) and discover how it can revolutionize the creation of stateless streaming pipelines within a Kafka (Redpanda) broker. These pipelines are adept at managing low-latency, high-data-volume scenarios.
Literature Review Basics and Understanding Reference Management.pptxDr Ramhari Poudyal
Three-day training on academic research focuses on analytical tools at United Technical College, supported by the University Grant Commission, Nepal. 24-26 May 2024
HEAP SORT ILLUSTRATED WITH HEAPIFY, BUILD HEAP FOR DYNAMIC ARRAYS.
Heap sort is a comparison-based sorting technique based on Binary Heap data structure. It is similar to the selection sort where we first find the minimum element and place the minimum element at the beginning. Repeat the same process for the remaining elements.
Student information management system project report ii.pdfKamal Acharya
Our project explains about the student management. This project mainly explains the various actions related to student details. This project shows some ease in adding, editing and deleting the student details. It also provides a less time consuming process for viewing, adding, editing and deleting the marks of the students.
Sachpazis:Terzaghi Bearing Capacity Estimation in simple terms with Calculati...Dr.Costas Sachpazis
Terzaghi's soil bearing capacity theory, developed by Karl Terzaghi, is a fundamental principle in geotechnical engineering used to determine the bearing capacity of shallow foundations. This theory provides a method to calculate the ultimate bearing capacity of soil, which is the maximum load per unit area that the soil can support without undergoing shear failure. The Calculation HTML Code included.
An Approach to Detecting Writing Styles Based on Clustering Techniquesambekarshweta25
An Approach to Detecting Writing Styles Based on Clustering Techniques
Authors:
-Devkinandan Jagtap
-Shweta Ambekar
-Harshit Singh
-Nakul Sharma (Assistant Professor)
Institution:
VIIT Pune, India
Abstract:
This paper proposes a system to differentiate between human-generated and AI-generated texts using stylometric analysis. The system analyzes text files and classifies writing styles by employing various clustering algorithms, such as k-means, k-means++, hierarchical, and DBSCAN. The effectiveness of these algorithms is measured using silhouette scores. The system successfully identifies distinct writing styles within documents, demonstrating its potential for plagiarism detection.
Introduction:
Stylometry, the study of linguistic and structural features in texts, is used for tasks like plagiarism detection, genre separation, and author verification. This paper leverages stylometric analysis to identify different writing styles and improve plagiarism detection methods.
Methodology:
The system includes data collection, preprocessing, feature extraction, dimensional reduction, machine learning models for clustering, and performance comparison using silhouette scores. Feature extraction focuses on lexical features, vocabulary richness, and readability scores. The study uses a small dataset of texts from various authors and employs algorithms like k-means, k-means++, hierarchical clustering, and DBSCAN for clustering.
Results:
Experiments show that the system effectively identifies writing styles, with silhouette scores indicating reasonable to strong clustering when k=2. As the number of clusters increases, the silhouette scores decrease, indicating a drop in accuracy. K-means and k-means++ perform similarly, while hierarchical clustering is less optimized.
Conclusion and Future Work:
The system works well for distinguishing writing styles with two clusters but becomes less accurate as the number of clusters increases. Future research could focus on adding more parameters and optimizing the methodology to improve accuracy with higher cluster values. This system can enhance existing plagiarism detection tools, especially in academic settings.
NO1 Uk best vashikaran specialist in delhi vashikaran baba near me online vas...Amil Baba Dawood bangali
Contact with Dawood Bhai Just call on +92322-6382012 and we'll help you. We'll solve all your problems within 12 to 24 hours and with 101% guarantee and with astrology systematic. If you want to take any personal or professional advice then also you can call us on +92322-6382012 , ONLINE LOVE PROBLEM & Other all types of Daily Life Problem's.Then CALL or WHATSAPP us on +92322-6382012 and Get all these problems solutions here by Amil Baba DAWOOD BANGALI
#vashikaranspecialist #astrologer #palmistry #amliyaat #taweez #manpasandshadi #horoscope #spiritual #lovelife #lovespell #marriagespell#aamilbabainpakistan #amilbabainkarachi #powerfullblackmagicspell #kalajadumantarspecialist #realamilbaba #AmilbabainPakistan #astrologerincanada #astrologerindubai #lovespellsmaster #kalajaduspecialist #lovespellsthatwork #aamilbabainlahore#blackmagicformarriage #aamilbaba #kalajadu #kalailam #taweez #wazifaexpert #jadumantar #vashikaranspecialist #astrologer #palmistry #amliyaat #taweez #manpasandshadi #horoscope #spiritual #lovelife #lovespell #marriagespell#aamilbabainpakistan #amilbabainkarachi #powerfullblackmagicspell #kalajadumantarspecialist #realamilbaba #AmilbabainPakistan #astrologerincanada #astrologerindubai #lovespellsmaster #kalajaduspecialist #lovespellsthatwork #aamilbabainlahore #blackmagicforlove #blackmagicformarriage #aamilbaba #kalajadu #kalailam #taweez #wazifaexpert #jadumantar #vashikaranspecialist #astrologer #palmistry #amliyaat #taweez #manpasandshadi #horoscope #spiritual #lovelife #lovespell #marriagespell#aamilbabainpakistan #amilbabainkarachi #powerfullblackmagicspell #kalajadumantarspecialist #realamilbaba #AmilbabainPakistan #astrologerincanada #astrologerindubai #lovespellsmaster #kalajaduspecialist #lovespellsthatwork #aamilbabainlahore #Amilbabainuk #amilbabainspain #amilbabaindubai #Amilbabainnorway #amilbabainkrachi #amilbabainlahore #amilbabaingujranwalan #amilbabainislamabad
Saudi Arabia stands as a titan in the global energy landscape, renowned for its abundant oil and gas resources. It's the largest exporter of petroleum and holds some of the world's most significant reserves. Let's delve into the top 10 oil and gas projects shaping Saudi Arabia's energy future in 2024.
Forklift Classes Overview by Intella PartsIntella Parts
Discover the different forklift classes and their specific applications. Learn how to choose the right forklift for your needs to ensure safety, efficiency, and compliance in your operations.
For more technical information, visit our website https://intellaparts.com
Is there anybody out there? Scala Days Berlin 2018
1. Is there anybody out there?
Scala Days Berlin 2018
Manuel Bernhardt
2. Disclaimer
[A career in distributed systems] is both exhilara6ng and
frustra6ng. When things work, it's like a symphony. When they
don't, it's like an eleventh-birthday party where half of the kids are
on speed.
— Jeff Darcy, HekaFS (formerly CloudFS) project lead
@elmanu - h+ps://manuel.bernhardt.io
3.
4. manuel.bernhardt.io
• Guiding companies to get started with
reac4ve systems and to keep them
running
• Lightbend consul4ng and training
partner, focus on Akka (Cluster,
Streams)
• Scuba-diver, currently stranded in
Austria
5. Mo#va#onal quote
Life is a single player game. You’re born alone. You’re going to die
alone. All of your interpreta8ons are alone. All your memories are
alone. You’re gone in three genera8ons and no one cares. Before
you showed up nobody cared. It’s all single player.
— Naval Ravikant
@elmanu - h+ps://manuel.bernhardt.io
6.
7.
8.
9. Key issues for building clusters
• discovery: who's there?
• fault detec0on: who's in trouble?
• load balancing: who can take up work?
@elmanu - h+ps://manuel.bernhardt.io
10. Key issues for building clusters
• discovery: who's there?
• fault detec0on: who's in trouble?
• load balancing: who can take up work?
Group membership
@elmanu - h+ps://manuel.bernhardt.io
14. Failure detector key
proper0es
• Completeness: crash-failure of any
group member is detected by all non-
faulty members
@elmanu - h+ps://manuel.bernhardt.io
15. Failure detector key
proper0es
• Completeness: crash-failure of any
group member is detected by all non-
faulty members
• Accuracy: no non-faulty group member
is declared as failed by any other non-
faulty group member (no false posi9ves)
@elmanu - h+ps://manuel.bernhardt.io
16. Failure detector key
proper0es
• Completeness: crash-failure of any group
member is detected by all non-faulty
members
• Accuracy: no non-faulty group member is
declared as failed by any other non-faulty
group member (no false posi9ves)
Also relevant in prac/ce:
• speed
• network message load
@elmanu - h+ps://manuel.bernhardt.io
17. Impossibility result
It is impossible for a failure detector algorithm to determinis3cally
achieve both completeness and accuracy over an asynchronous
unreliable network1
1
Chandra, Toueg: Unreliable failure detectors for reliable distributed systems (1996)
@elmanu - h+ps://manuel.bernhardt.io
18. Trade-offs
• Strong - Weak completeness: all / some non-faulty members
detect a crash
• Strong - Weak accuracy: there are no / some false-posi7ves
@elmanu - h+ps://manuel.bernhardt.io
19. Trade-offs
• Strong - Weak completeness: all / some non-faulty members
detect a crash
• Strong - Weak accuracy: there are no / some false-posi7ves
In prac(ce most applica(ons prefer strong completeness with a
weaker form of accuracy
@elmanu - h+ps://manuel.bernhardt.io
21. Phi Adap)ve Accrual
Failure Detector 3
• has a cool name
• adap.ve: adjusts to network condi.ons
• introduces the no.on of accrual failure
detec.on: suspicion value φ rather than
boolean (trusted or suspected)
• made popular by Cassandra
3
N. Hayashibara et al: The ϕ Accrual Failure Detector (2004)
@elmanu - h+ps://manuel.bernhardt.io
22. Phi Adap)ve Accrual
Failure Detector 3
Example: master and worker processes
• φ(w) > 8 stop sending new work to
the node
• φ(w) > 10 start to rebalance current
tasks of the worker to other nodes
• φ(w) > 12 remove the worker from
the list of nodes
3
N. Hayashibara et al: The ϕ Accrual Failure Detector (2004)
@elmanu - h+ps://manuel.bernhardt.io
23. New Adap)ve accrual
Failure Detector 4
• much simpler to calculate suspicion
level than Phi
• performs slightly be7er and more
adap)ve 5
5
h$ps://manuel.bernhardt.io/2017/07/26/a-new-adap=ve-accrual-
failure-detector-for-akka/
4
B. Satzger et al: A new adap3ve accrual failure detector for dependable
distributed systems (2007)
@elmanu - h+ps://manuel.bernhardt.io
24. SWIM Failure Detector
As you swim lazily through the milieu,
The secrets of the world will infect you 6
• has both a dissemina.on and a failure detec.on component
• scalable membership protocol
• members are first suspected and not immediately flagged as
failed
6
A. Das et al: SWIM: scalable weakly-consistent infec:on-style process group membership protocol (2002)
@elmanu - h+ps://manuel.bernhardt.io
25.
26.
27.
28.
29.
30.
31. Lifeguard Failure Detector
• based on SWIM, developed by
Hashicorp 7
• memberlist implementa;on 8
• extensions to the SWIM protocol
• dras;cally reduces the amount of false-
posi;ves
8
h$ps://github.com/hashicorp/memberlist
7
A. Dadgar et al: Lifeguard: SWIM-ing with Situa:onal Awareness (2017)
@elmanu - h+ps://manuel.bernhardt.io
37. Dissemina(on strategies:
mul(cast
• hardware / IP / UDP mul1cast: not
readily (or willingly) enabled in data
centres
• even if we had mul1cast support we'd
s1ll have quite a bit of work to do 23
23
X. Défago, A. Schiper, P. Urbán: Total Order Broadcast and MulDcast
Algorithms: Taxonomy and Survey
@elmanu - h+ps://manuel.bernhardt.io
38. Dissemina(on strategies:
gossip protocols
• based on the research done in the P2P
days 10 11 24 25
25
P. Rama, A. D. George, M. Radlinski, R. SubramaniyanL GEMS: Gosssip-
Enabled Monitoring Service for Heterogeneous Distributed Systems
24
I. Stoica, R. Morris, D. Karger, M. F. Kaashoek, H. Balakrishnan: Chord: A
Scalable Peer-to-peer Lookup Service for Internet ApplicaHons
11
S. Ranganathan et al: Gossip-Style Failure Detec:on and Distributed
Consensus for Scalable Heterogeneous Clusters (2000)
10
van Renesse et al: A gossip-style failure detec9on service (1998)
39. Gossip styles
• gossip to one node at random 10
10
van Renesse et al: A gossip-style failure detec9on service (1998)
@elmanu - h+ps://manuel.bernhardt.io
40. Gossip styles
• gossip to one node at random 10
• round-robin, binary round-robin, round-
robin with sequence check 11
11
S. Ranganathan et al: Gossip-Style Failure Detec:on and Distributed
Consensus for Scalable Heterogeneous Clusters (2000)
10
van Renesse et al: A gossip-style failure detec9on service (1998)
@elmanu - h+ps://manuel.bernhardt.io
41. Gossip styles
• gossip to one node at random 10
• round-robin, binary round-robin,
round-robin with sequence check 11
11
S. Ranganathan et al: Gossip-Style Failure Detec:on and Distributed
Consensus for Scalable Heterogeneous Clusters (2000)
10
van Renesse et al: A gossip-style failure detec9on service (1998)
@elmanu - h+ps://manuel.bernhardt.io
42. Gossip styles
• gossip to one node at random 10
• round-robin, binary round-robin, round-
robin with sequence check 11
• piggy-back on another protocol (e.g. on
a failure detector 6
): also called
infec&on-style gossip
6
A. Das et al: SWIM: scalable weakly-consistent infec:on-style process
group membership protocol (2002)
11
S. Ranganathan et al: Gossip-Style Failure Detec:on and Distributed
Consensus for Scalable Heterogeneous Clusters (2000)
10
van Renesse et al: A gossip-style failure detec9on service (1998)
@elmanu - h+ps://manuel.bernhardt.io
43. What do you even gossip about
@elmanu - h+ps://manuel.bernhardt.io
44. What do you even gossip about
@elmanu - h+ps://manuel.bernhardt.io
45. What do you even gossip about
@elmanu - h+ps://manuel.bernhardt.io
46. Example of gossip op.miza.ons
• Akka Cluster: gossip with a higher probability to nodes that have
not already seen a gossip
• Akka Cluster: speeds up gossip (3x) when less than half of the
members have seen the latest gossip
• Lifeguard: an<-entropy mechanism based on nodes doing a full
sync with a node at random (helps to speed up convergence a?er
a network par<<on)
@elmanu - h+ps://manuel.bernhardt.io
49. Impossibility result - group
membership
Group membership with a single group is impossible when there
are nodes that are suspected of having failed 9
9
Chandra et al: On the Impossibility of Group Membership (1996)
@elmanu - h+ps://manuel.bernhardt.io
50. It would be unwise to make
membership-related decisions while
there are processes suspected of
having crashed
@elmanu - h+ps://manuel.bernhardt.io
51. Reaching consensus: .me
• Lamport Clocks 12
: how do you order events in a distributed
system
12
L. Lamport: Time, Clocks, and the Ordering of Events in a Distributed System (1978)
@elmanu - h+ps://manuel.bernhardt.io
52. Reaching consensus: .me
• Lamport Clocks: how do you order events in a distributed
system
• Vector Clocks 13
: how do you order events in a distributed
system and flag concurrent event
13
F. Ma(ern: Virtual Time and Global States of Distributed Systems (1989)
@elmanu - h+ps://manuel.bernhardt.io
53. Reaching consensus: .me
• Lamport Clocks: how do you order events in a distributed system
• Vector Clocks: how do you order events in a distributed system and flag
concurrent events
• Version Vectors 14 15
and Do3ed Version Vectors 16
: very similar but
seman;cs primarily concerned with versioning and conflict detec;on in
replicas
16
N. Preguiça: Do1ed Version Vectors: Efficient Causality Tracking for Distributed Key-Value Stores (2012)
15
h%ps://haslab.wordpress.com/2011/07/08/version-vectors-are-not-vector-clocks/
14
D.S. Parker: Detec/on of mutual inconsistency in distributed systems (1983)
@elmanu - h+ps://manuel.bernhardt.io
54. Reaching consensus: .me
private[cluster] final case class Gossip(
members: immutable.SortedSet[Member],
overview: GossipOverview = GossipOverview(),
version: VectorClock = VectorClock(),
tombstones: Map[UniqueAddress, Gossip.Timestamp] = Map.empty)
@elmanu - h+ps://manuel.bernhardt.io
55. Reaching consensus:
replicated state machines
Any suffisciently complicated model class
contains an ad-hoc, informally-specified,
bug-ridden, slow implementa;on of half a
state machine
— Pete Forde
@elmanu - h+ps://manuel.bernhardt.io
56. Reaching consensus:
replicated state machines
This method allows one to implement any
desired form of mul4process
synchroniza4on in a distributed system
— Leslie Lamport 12
12
L. Lamport: Time, Clocks, and the Ordering of Events in a Distributed
System (1978)
@elmanu - h+ps://manuel.bernhardt.io
57. Reaching consensus:
protocols
• fault-tolerant distributed systems
• how do mul1ple servers agree on a
value?
• Paxos 19 20
, Ra@ 21
, CASPaxos 22
22
D. Rystsov: CASPaxos: Replicated State Machines without logs (2018)
21
D. Ongaro and J. Ousterhout: In Search of an Understandable
Consensus Algorithm (Extended Version) (2014)
20
L. Lamport: Paxos made simple (2001)
19
L. Lamport: The Part-Time Parliament (1998)
@elmanu - h+ps://manuel.bernhardt.io
58. Reaching consensus:
CRDTs
• Conflict-Free Replicated Data types 17
• strong eventual consistency
• two families:
• CmRDTs (commuta+vity of
opera+ons)
• CvRDTs (convergence of state - merge
funcBon)
17
F.B. Schneider: Implemen4ng Fault Tolerant Services Using the State
Machine Approach: A Tutorial (1990)
@elmanu - h+ps://manuel.bernhardt.io
59. Reaching consensus: conven/ons
• reaching a decision by not transmi2ng any informa4on
• example: Akka Cluster leader designa4on
/**
* INTERNAL API
* Orders the members by their address except that members with status
* Joining, Exiting and Down are ordered last (in that order).
*/
private[cluster] val leaderStatusOrdering: Ordering[Member] = ...
@elmanu - h+ps://manuel.bernhardt.io
68. Failure detectors
• Chandra, Toueg: Unreliable failure detectors for reliable distributed systems (1996)
• N. Hayashibara et al: The ϕ Accrual Failure Detector (2004)
• B. Satzger et al: A new adapNve accrual failure detector for dependable distributed systems
(2007)
• hQps://manuel.bernhardt.io/2017/07/26/a-new-adapNve-accrual-failure-detector-for-akka/
• A. Das et al: SWIM: scalable weakly-consistent infecNon-style process group membership
protocol (2002)
• A. Dadgar et al: Lifeguard: SWIM-ing with SituaNonal Awareness (2017)
• hQps://github.com/hashicorp/memberlist
@elmanu - h+ps://manuel.bernhardt.io
69. Impossibility results
• M.J. Fischer, N.A. Lynch, and M.S. Paterson: Impossibility of
distributed consensus with one faulty process (1985)
• Chandra, Toueg: Unreliable failure detectors for reliable
distributed systems (1996)
• Chandra et al: On the Impossibility of Group Membership (1996)
@elmanu - h+ps://manuel.bernhardt.io
70. Dissemina(on
• van Renesse et al: A gossip-style failure detec8on service (1998)
• S. Ranganathan et al: Gossip-Style Failure Detec8on and Distributed Consensus
for Scalable Heterogeneous Clusters (2000)
• P. Rama, A. D. George, M. Radlinski, R. SubramaniyanL GEMS: Gosssip-Enabled
Monitoring Service for Heterogeneous Distributed Systems (2002)
• X. Défago, A. Schiper, P. Urbán: Total Order Broadcast and Mul8cast Algorithms:
Taxonomy and Survey
• I. Stoica, R. Morris, D. Karger, M. F. Kaashoek, H. Balakrishnan: Chord: A Scalable
Peer-to-peer Lookup Service for Internet Applica8ons
@elmanu - h+ps://manuel.bernhardt.io
71. Consensus - )me
• L. Lamport: Time, Clocks, and the Ordering of Events in a Distributed
System (1978)
• F. MaJern: Virtual Time and Global States of Distributed Systems (1989)
• D.S. Parker: DetecNon of mutual inconsistency in distributed systems (1983)
• hJps://haslab.wordpress.com/2011/07/08/version-vectors-are-not-vector-
clocks/
• N. Preguiça: DoJed Version Vectors: Efficient Causality Tracking for
Distributed Key-Value Stores (2012)
@elmanu - h+ps://manuel.bernhardt.io
72. Consensus - protocols
• F.B. Schneider: Implemen3ng Fault Tolerant Services Using the State
Machine Approach: A Tutorial (1990)
• L. Lamport: The Part-Time Parliament (1998)
• L. Lamport: Paxos made simple (2001)
• D. Ongaro and J. Ousterhout: In Search of an Understandable
Consensus Algorithm (Extended Version) (2014)
• D. Rystsov: CASPaxos: Replicated State Machines without logs (2018)
@elmanu - h+ps://manuel.bernhardt.io
73. Consensus - misc
• M. Shapiro: Conflict-free Replicated Data Types (2011)
@elmanu - h+ps://manuel.bernhardt.io