A tutorial-like technical presentation that covers fundamental approaches for replication along with their advantages, disadvantages, comparisons with each other etc.
Client-Centric Consistency
Provide guarantees about ordering of operations only for a single client, i.e.
Effects of an operations depend on the client performing it
Effects also depend on the history of client’s operations
Applied only when requested by the client
No guarantees concerning concurrent accesses by different clients
Assumption:
Clients can access different replicas, e.g. mobile users
Replication in computing involves sharing information so as to ensure consistency between redundant resources, such as software or hardware components, to improve reliability, fault-tolerance, or accessibility.
Client-Centric Consistency
Provide guarantees about ordering of operations only for a single client, i.e.
Effects of an operations depend on the client performing it
Effects also depend on the history of client’s operations
Applied only when requested by the client
No guarantees concerning concurrent accesses by different clients
Assumption:
Clients can access different replicas, e.g. mobile users
Replication in computing involves sharing information so as to ensure consistency between redundant resources, such as software or hardware components, to improve reliability, fault-tolerance, or accessibility.
Conventional architectures coarsely comprise of a processor, memory system, and the datapath.
Each of these components present significant performance bottlenecks.
Parallelism addresses each of these components in significant ways.
Different applications utilize different aspects of parallelism - e.g., data itensive applications utilize high aggregate throughput, server applications utilize high aggregate network bandwidth, and scientific applications typically utilize high processing and memory system performance.
It is important to understand each of these performance bottlenecks.
An explicitly parallel program must specify concurrency and interaction between concurrent subtasks.
The former is sometimes also referred to as the control structure and the latter as the communication model.
This presentation is about Distributed Applications, points:
- Distribution
- Consistency
- Replication
- Distributed KV in Erlang
- Scaling
- Partitioning
- Observability
To Get any Project for CSE, IT ECE, EEE Contact Me @ 09849539085, 09966235788 or mail us - ieeefinalsemprojects@gmail.com-Visit Our Website: www.finalyearprojects.org
Error tolerant resource allocation and payment minimization for cloud systemIEEEFINALYEARPROJECTS
To Get any Project for CSE, IT ECE, EEE Contact Me @ 09849539085, 09966235788 or mail us - ieeefinalsemprojects@gmail.co¬m-Visit Our Website: www.finalyearprojects.org
IT IS ABOUT MULTIPROCESSING,COMMUNICATION BETWEEN THE PROCESS THROUGH MESSAGE PASSING AND SHARED MEMORY.SYNCHRONIZATION MECHANISM AND SYNCHRONIZATION USING SEMAPHORE
Yura Nikonovich is the first to set up a tradition with his research “CAP theorem and distributed systems”.
In the first part of the report, find his insights into what distributed systems are and what challenges he faced in the course of his work with them.
In the second part of the report Yura tells us about CAP theorem, it's varieties and the future of distributed databases.
The buffer manager is the software layer that is responsible for bringing pages from physical disk to main memory as needed. The buffer manages the available main memory by dividing the main memory into a collection of pages, which we called as buffer pool. The main memory pages in the buffer pool are called frames.
Replication in the wild ankara cloud meetup - feb 2017Onur Dayıbaşı
Replication, büyük verilere performanslı bir şekilde erişmek ve hata durumlarında veri kayıplarını önlemek için kullanılan bir tekniktir. Bu sunumda, özellikle NoSQL veritabanlarında sıkça kullanılan replication metodlarına göz atacağız. Replication metodlarını temel niteliklerine göre sınıflandırıp birbirlerine karşı avantaj / dezavantajlarını, hangi ihtiyaçlara uygun olduklarını, hangi problemleri çözüp hangi problemleri ortaya çıkardıklarını inceleyeceğiz. Sunum, tutorial havasında adım adım ilerleyen, takip etmesi kolay bir içeriğe sahiptir
Replication in the wild ankara cloud meetup - feb 2017AnkaraCloud
Replication, büyük verilere performanslı bir şekilde erişmek ve hata durumlarında veri kayıplarını önlemek için kullanılan bir tekniktir. Bu sunumda, özellikle NoSQL veritabanlarında sıkça kullanılan replication metodlarına göz atacağız. Replication metodlarını temel niteliklerine göre sınıflandırıp birbirlerine karşı avantaj / dezavantajlarını, hangi ihtiyaçlara uygun olduklarını, hangi problemleri çözüp hangi problemleri ortaya çıkardıklarını inceleyeceğiz. Sunum, tutorial havasında adım adım ilerleyen, takip etmesi kolay bir içeriğe sahiptir
Conventional architectures coarsely comprise of a processor, memory system, and the datapath.
Each of these components present significant performance bottlenecks.
Parallelism addresses each of these components in significant ways.
Different applications utilize different aspects of parallelism - e.g., data itensive applications utilize high aggregate throughput, server applications utilize high aggregate network bandwidth, and scientific applications typically utilize high processing and memory system performance.
It is important to understand each of these performance bottlenecks.
An explicitly parallel program must specify concurrency and interaction between concurrent subtasks.
The former is sometimes also referred to as the control structure and the latter as the communication model.
This presentation is about Distributed Applications, points:
- Distribution
- Consistency
- Replication
- Distributed KV in Erlang
- Scaling
- Partitioning
- Observability
To Get any Project for CSE, IT ECE, EEE Contact Me @ 09849539085, 09966235788 or mail us - ieeefinalsemprojects@gmail.com-Visit Our Website: www.finalyearprojects.org
Error tolerant resource allocation and payment minimization for cloud systemIEEEFINALYEARPROJECTS
To Get any Project for CSE, IT ECE, EEE Contact Me @ 09849539085, 09966235788 or mail us - ieeefinalsemprojects@gmail.co¬m-Visit Our Website: www.finalyearprojects.org
IT IS ABOUT MULTIPROCESSING,COMMUNICATION BETWEEN THE PROCESS THROUGH MESSAGE PASSING AND SHARED MEMORY.SYNCHRONIZATION MECHANISM AND SYNCHRONIZATION USING SEMAPHORE
Yura Nikonovich is the first to set up a tradition with his research “CAP theorem and distributed systems”.
In the first part of the report, find his insights into what distributed systems are and what challenges he faced in the course of his work with them.
In the second part of the report Yura tells us about CAP theorem, it's varieties and the future of distributed databases.
The buffer manager is the software layer that is responsible for bringing pages from physical disk to main memory as needed. The buffer manages the available main memory by dividing the main memory into a collection of pages, which we called as buffer pool. The main memory pages in the buffer pool are called frames.
Replication in the wild ankara cloud meetup - feb 2017Onur Dayıbaşı
Replication, büyük verilere performanslı bir şekilde erişmek ve hata durumlarında veri kayıplarını önlemek için kullanılan bir tekniktir. Bu sunumda, özellikle NoSQL veritabanlarında sıkça kullanılan replication metodlarına göz atacağız. Replication metodlarını temel niteliklerine göre sınıflandırıp birbirlerine karşı avantaj / dezavantajlarını, hangi ihtiyaçlara uygun olduklarını, hangi problemleri çözüp hangi problemleri ortaya çıkardıklarını inceleyeceğiz. Sunum, tutorial havasında adım adım ilerleyen, takip etmesi kolay bir içeriğe sahiptir
Replication in the wild ankara cloud meetup - feb 2017AnkaraCloud
Replication, büyük verilere performanslı bir şekilde erişmek ve hata durumlarında veri kayıplarını önlemek için kullanılan bir tekniktir. Bu sunumda, özellikle NoSQL veritabanlarında sıkça kullanılan replication metodlarına göz atacağız. Replication metodlarını temel niteliklerine göre sınıflandırıp birbirlerine karşı avantaj / dezavantajlarını, hangi ihtiyaçlara uygun olduklarını, hangi problemleri çözüp hangi problemleri ortaya çıkardıklarını inceleyeceğiz. Sunum, tutorial havasında adım adım ilerleyen, takip etmesi kolay bir içeriğe sahiptir
MySQL 5.7 clustering: The developer perspectiveUlf Wendel
(Compiled from revised slides of previous presentations - skip if you know the old presentations)
A summary on clustering MySQL 5.7 with focus on the PHP clients view and the PHP driver. Which kinds on MySQL clusters are there, what are their goal, how does wich one scale, what extra work does which clustering technique put at the client and finally, how the PHP driver (PECL/mysqlnd_ms) helps you.
MySQL 5.7 Fabric: Introduction to High Availability and Sharding Ulf Wendel
MySQL 5.7 has sharding built-in to MySQL. The free and open source MySQL Fabric utility simplifies the management of MySQL clusters of any kind. This includes MySQL Replication setup, monitoring, automatic failover, switchover and so fort for High Availability. Additionally, it offers measures to shard a MySQL database over many an arbitrary number of servers. Intelligent load balancer (updated drivers) take care of routing queries to the appropriate shards.
Talon systems - Distributed multi master replication strategySaptarshi Chatterjee
Data Replication is the process of storing data in more than one site or node.It is useful in improving the availability of data. The result is a distributed database in which users can access data relevant to their tasks without interfering with the work of others.Data Replication is generally performed to provide a consistent copy of data across all the database nodes.
Traditionally it’s done by copying data from one database server to another, so that all the servers can have the same data. Our implementation, proposes a completely different approach. Instead of copying data from one node to another, in our design , master replicas do not directly communicate between each other and work virtually independently for write queries. For read queries, an independent process consults all the replicas to constitute a quorum. and returns the result if a majority of the machines in the system response with the same result
Zookeeper vs Raft: Stateful distributed coordination with HA and Fault ToleranceAlluxio, Inc.
Big Data Bellevue & Cloudy With a Chance of Data Meetup
October 20, 2022
For more Alluxio events: https://alluxio.io/events/
Speakers:
David Zhu (Tech Lead Manager & PMC, Alluxio)
Jasmine Wang (Head of Community & DevRel, Alluxio)
Distributed systems are made up of many components such as authentication, a persistence layer, stateless services, load balancers, and stateful coordination services. These coordination services are central to the operation of the system, performing tasks such as maintaining system configuration state, ensuring service availability, name resolution, and storing other system metadata. Given their central role in the system it is essential that these systems remain available, fault tolerant and consistent. By providing a highly available file system-like abstraction as well as powerful recipes such as leader election, Apache Zookeeper is often used to implement these services. This talk will go over a generic example of stateful coordination service moving from Zookeeper to Raft.
MySQL Group Replication is a new 'synchronous', multi-master, auto-everything replication plugin for MySQL introduced with MySQL 5.7. It is the perfect tool for small 3-20 machine MySQL clusters to gain high availability and high performance. It stands for high availability because the fault of replica don't stop the cluster. Failed nodes can rejoin the cluster and new nodes can be added in a fully automatic way - no DBA intervention required. Its high performance because multiple masters process writes, not just one like with MySQL Replication. Running applications on it is simple: no read-write splitting, no fiddling with eventual consistency and stale data. The cluster offers strong consistency (generalized snapshot isolation).
It is based on Group Communication principles, hence the name.
DIY: A distributed database cluster, or: MySQL ClusterUlf Wendel
Live from the International PHP Conference 2013: MySQL Cluster is a distributed, auto-sharding database offering 99,999% high availability. It runs on Rasperry PI as well as on a cluster of multi-core machines. A 30 node cluster was able to deliver 4.3 billion (not million) read transactions per second in 2012. Take a deeper look into the theory behind all the MySQL replication/clustering solutions (including 3rd party) and learn how they differ.
Distributed systems involve complex interactions among many components. This increases the possibilities of failures that could turn a whole system down. Software architects, designers, and developers need to architect, design, and program functional requirements thinking about possibility of failures and the need for a system to keep running despite failures. This presentation tackles but part of the problem, focusing on redundancy, different types of groups, replication, and eventual consistency, finishing with the presentation of CAP theorem.
Presentation delivered at IV Cloud Computing and Big Data Ent at Universdad Nacional de La Plata http://www.jcc.info.unlp.edu.ar/jcc2016/wordpress/index.php/cronograma/
java.util.concurrent for Distributed Coordination - Berlin Expert Days 2019Ensar Basri Kahveci
The evolution of distributed coordination tools shows that high-level APIs ease implementation of coordination tasks, such as leader election, locking, synchronized actions. For instance, the Chubby paper highlights familiarity of lock-based interfaces. Similarly, Apache Curator hides complexity of ZooKeeper recipes behind Java APIs, while etcd and Consul implement concurrency primitives on their own.
A different path in this journey would be extending the long-lasting java.util.concurrent APIs, such as Lock, Semaphore, etc. Simplicity of these APIs makes them very useful in distributed coordination use cases.
Join this talk to explore Hazelcast’s brand new implementation of java.util.concurrent APIs on top of the Raft consensus algorithm. I will walk through code samples to demonstrate how Java locks, semaphores, etc. can be used in distributed environments that involve partial failures. I will also share our experience of how we coped with several challenges we faced while developing and testing our Raft implementation.
java.util.concurrent for Distributed Coordination, Riga DevDays 2019Ensar Basri Kahveci
The evolution of distributed coordination tools shows that high-level APIs ease implementation of coordination tasks, such as leader election, locking, synchronized actions. For instance, the Chubby paper highlights familiarity of lock-based interfaces. Similarly, Apache Curator hides complexity of ZooKeeper recipes behind Java APIs, while etcd and Consul implement concurrency primitives on their own.
A different path in this journey would be extending the long-lasting java.util.concurrent APIs, such as Lock, Semaphore, etc. Simplicity of these APIs makes them very useful in distributed coordination use cases.
Join this talk to explore Hazelcast’s brand new implementation of java.util.concurrent APIs on top of the Raft consensus algorithm. I will walk through code samples to demonstrate how Java locks, semaphores, etc. can be used in distributed environments that involve partial failures. I will also share our experience of how we coped with several challenges we faced while developing and testing our Raft implementation.
java.util.concurrent for Distributed Coordination, GeeCON Krakow 2019Ensar Basri Kahveci
The evolution of distributed coordination tools shows that high-level APIs ease implementation of coordination tasks, such as leader election, locking, synchronized actions. For instance, the Chubby paper highlights familiarity of lock-based interfaces. Similarly, Apache Curator hides complexity of ZooKeeper recipes behind Java APIs, while etcd and Consul implement concurrency primitives on their own.
A different path in this journey would be extending the long-lasting java.util.concurrent APIs, such as Lock, Semaphore, etc. Simplicity of these APIs makes them very useful in distributed coordination use cases.
Join this talk to explore Hazelcast’s brand new implementation of java.util.concurrent APIs on top of the Raft consensus algorithm. I will walk through code samples to demonstrate how Java locks, semaphores, etc. can be used in distributed environments that involve partial failures. I will also share our experience of how we coped with several challenges we faced while developing and testing our Raft implementation.
java.util.concurrent for Distributed Coordination, JEEConf 2019Ensar Basri Kahveci
The evolution of distributed coordination tools shows that high-level APIs ease implementation of coordination tasks, such as leader election, locking, synchronized actions. For instance, the Chubby paper highlights familiarity of lock-based interfaces. Similarly, Apache Curator hides complexity of ZooKeeper recipes behind Java APIs, while etcd and Consul implement concurrency primitives on their own.
A different path in this journey would be extending the long-lasting java.util.concurrent APIs, such as Lock, Semaphore, etc. Simplicity of these APIs makes them very useful in distributed coordination use cases.
Join this talk to explore Hazelcast’s brand new implementation of java.util.concurrent APIs on top of the Raft consensus algorithm. I will walk through code samples to demonstrate how Java locks, semaphores, etc. can be used in distributed environments that involve partial failures. I will also share our experience of how we coped with several challenges we faced while developing and testing our Raft implementation.
From AP to CP and Back: The Curious Case of Hazelcast (jdk.io 2018)Ensar Basri Kahveci
Distributed data stores cope with the challenges of managing replicated data. Building a distributed data store is hard, and clarifying its high level properties is even harder. Hazelcast makes use of several replication techniques to offer choices for the trade-offs between consistency, availability, and latency properties. In this talk, we take a closer look at these techniques from the perspective of the CAP and PACELC principles. We first break apart the main replication technique and challenge the rationale behind it. Then, we discuss what has gone wrong with it, and how we gradually extended and replaced it with other techniques to fix the issues.
Distributed Systems Theory for Mere Mortals - Topconf Dusseldorf October 2017Ensar Basri Kahveci
A talk about some of the core theoretical topics of distributed systems. It discusses system models, failure modes, time, the consensus problem, consistency models and high-level design principles: such as CAP and PACELC.
Democratizing Fuzzing at Scale by Abhishek Aryaabh.arya
Presented at NUS: Fuzzing and Software Security Summer School 2024
This keynote talks about the democratization of fuzzing at scale, highlighting the collaboration between open source communities, academia, and industry to advance the field of fuzzing. It delves into the history of fuzzing, the development of scalable fuzzing platforms, and the empowerment of community-driven research. The talk will further discuss recent advancements leveraging AI/ML and offer insights into the future evolution of the fuzzing landscape.
TECHNICAL TRAINING MANUAL GENERAL FAMILIARIZATION COURSEDuvanRamosGarzon1
AIRCRAFT GENERAL
The Single Aisle is the most advanced family aircraft in service today, with fly-by-wire flight controls.
The A318, A319, A320 and A321 are twin-engine subsonic medium range aircraft.
The family offers a choice of engines
Immunizing Image Classifiers Against Localized Adversary Attacksgerogepatton
This paper addresses the vulnerability of deep learning models, particularly convolutional neural networks
(CNN)s, to adversarial attacks and presents a proactive training technique designed to counter them. We
introduce a novel volumization algorithm, which transforms 2D images into 3D volumetric representations.
When combined with 3D convolution and deep curriculum learning optimization (CLO), itsignificantly improves
the immunity of models against localized universal attacks by up to 40%. We evaluate our proposed approach
using contemporary CNN architectures and the modified Canadian Institute for Advanced Research (CIFAR-10
and CIFAR-100) and ImageNet Large Scale Visual Recognition Challenge (ILSVRC12) datasets, showcasing
accuracy improvements over previous techniques. The results indicate that the combination of the volumetric
input and curriculum learning holds significant promise for mitigating adversarial attacks without necessitating
adversary training.
Industrial Training at Shahjalal Fertilizer Company Limited (SFCL)MdTanvirMahtab2
This presentation is about the working procedure of Shahjalal Fertilizer Company Limited (SFCL). A Govt. owned Company of Bangladesh Chemical Industries Corporation under Ministry of Industries.
Explore the innovative world of trenchless pipe repair with our comprehensive guide, "The Benefits and Techniques of Trenchless Pipe Repair." This document delves into the modern methods of repairing underground pipes without the need for extensive excavation, highlighting the numerous advantages and the latest techniques used in the industry.
Learn about the cost savings, reduced environmental impact, and minimal disruption associated with trenchless technology. Discover detailed explanations of popular techniques such as pipe bursting, cured-in-place pipe (CIPP) lining, and directional drilling. Understand how these methods can be applied to various types of infrastructure, from residential plumbing to large-scale municipal systems.
Ideal for homeowners, contractors, engineers, and anyone interested in modern plumbing solutions, this guide provides valuable insights into why trenchless pipe repair is becoming the preferred choice for pipe rehabilitation. Stay informed about the latest advancements and best practices in the field.
Sachpazis:Terzaghi Bearing Capacity Estimation in simple terms with Calculati...Dr.Costas Sachpazis
Terzaghi's soil bearing capacity theory, developed by Karl Terzaghi, is a fundamental principle in geotechnical engineering used to determine the bearing capacity of shallow foundations. This theory provides a method to calculate the ultimate bearing capacity of soil, which is the maximum load per unit area that the soil can support without undergoing shear failure. The Calculation HTML Code included.
Student information management system project report ii.pdfKamal Acharya
Our project explains about the student management. This project mainly explains the various actions related to student details. This project shows some ease in adding, editing and deleting the student details. It also provides a less time consuming process for viewing, adding, editing and deleting the marks of the students.
Hybrid optimization of pumped hydro system and solar- Engr. Abdul-Azeez.pdffxintegritypublishin
Advancements in technology unveil a myriad of electrical and electronic breakthroughs geared towards efficiently harnessing limited resources to meet human energy demands. The optimization of hybrid solar PV panels and pumped hydro energy supply systems plays a pivotal role in utilizing natural resources effectively. This initiative not only benefits humanity but also fosters environmental sustainability. The study investigated the design optimization of these hybrid systems, focusing on understanding solar radiation patterns, identifying geographical influences on solar radiation, formulating a mathematical model for system optimization, and determining the optimal configuration of PV panels and pumped hydro storage. Through a comparative analysis approach and eight weeks of data collection, the study addressed key research questions related to solar radiation patterns and optimal system design. The findings highlighted regions with heightened solar radiation levels, showcasing substantial potential for power generation and emphasizing the system's efficiency. Optimizing system design significantly boosted power generation, promoted renewable energy utilization, and enhanced energy storage capacity. The study underscored the benefits of optimizing hybrid solar PV panels and pumped hydro energy supply systems for sustainable energy usage. Optimizing the design of solar PV panels and pumped hydro energy supply systems as examined across diverse climatic conditions in a developing country, not only enhances power generation but also improves the integration of renewable energy sources and boosts energy storage capacities, particularly beneficial for less economically prosperous regions. Additionally, the study provides valuable insights for advancing energy research in economically viable areas. Recommendations included conducting site-specific assessments, utilizing advanced modeling tools, implementing regular maintenance protocols, and enhancing communication among system components.
Water scarcity is the lack of fresh water resources to meet the standard water demand. There are two type of water scarcity. One is physical. The other is economic water scarcity.
Cosmetic shop management system project report.pdfKamal Acharya
Buying new cosmetic products is difficult. It can even be scary for those who have sensitive skin and are prone to skin trouble. The information needed to alleviate this problem is on the back of each product, but it's thought to interpret those ingredient lists unless you have a background in chemistry.
Instead of buying and hoping for the best, we can use data science to help us predict which products may be good fits for us. It includes various function programs to do the above mentioned tasks.
Data file handling has been effectively used in the program.
The automated cosmetic shop management system should deal with the automation of general workflow and administration process of the shop. The main processes of the system focus on customer's request where the system is able to search the most appropriate products and deliver it to the customers. It should help the employees to quickly identify the list of cosmetic product that have reached the minimum quantity and also keep a track of expired date for each cosmetic product. It should help the employees to find the rack number in which the product is placed.It is also Faster and more efficient way.
Overview of the fundamental roles in Hydropower generation and the components involved in wider Electrical Engineering.
This paper presents the design and construction of hydroelectric dams from the hydrologist’s survey of the valley before construction, all aspects and involved disciplines, fluid dynamics, structural engineering, generation and mains frequency regulation to the very transmission of power through the network in the United Kingdom.
Author: Robbie Edward Sayers
Collaborators and co editors: Charlie Sims and Connor Healey.
(C) 2024 Robbie E. Sayers
2. REPLICATION
- Putting a data set into multiple nodes.
- Each replica has a full copy.
- A few reasons for replication:
- Performance
- Availability and fault tolerance
- Mostly used with partitioning.
3. NOTHING FOR FREE!
- Very easy to do when the data is immutable.
- Problems start when we have multiple copies
of the data and we want to update them.
- Two main difficulties
- Handling updates
- Handling failures
4. The dangers of replication and a solution
- Gray et al. [1] classify replication models by 2
parameters:
- Where to make updates: primary copy or update
anywhere
- When to make updates: eagerly or lazily
5. WHERE: PRIMARY COPY
- There is a single replica managing the updates.
- Concurrency control is easy.
- No conflicts and no conflict-handling logic.
- Updates are executed on the primary and
secondaries with the same order.
- When primary fails, a new primary is elected.
- Ensuring a single and good primary is hard.
6. WHERE: UPDATE ANYWHERE
- Each replica can initiate a transaction to make
an update.
- Complex concurrency control.
- Deadlocks or conflicts are possible.
- In practice, there is also multi-leader.
7. WHEN: EAGER REPLICATION
- Synchronously updates all replicas as part of
one atomic transaction.
- Provides strong consistency.
- Not very flexible. Degree of availability can
degrade on node failures.
- Consensus algorithms.
8. WHEN: LAZY REPLICATION
- Updates each replica with a separate
transaction.
- Updates can execute quite fast.
- Degree of availability is high.
- Eventual consistency.
- Data copies can diverge.
- Data loss or conflicts can occur.
10. WHERE?
WHEN?
PRIMARY COPY UPDATE ANYWHERE
EAGER
Multi Paxos [5]
etcd and Consul (RAFT) [6]
Zookeeper (Zab) [7]
Kafka
Paxos [5]
Hazelcast Cluster State Change [12]
LAZY
Hazelcast
MongoDB
ElasticSearch
Redis
Dynamo [4]
Cassandra
Riak
11. PRIMARY COPY + EAGER REPLICATION
- When the primary fails, secondaries are
guaranteed to be up to date.
- Raft, Kafka etc.
- Majority approach can be used.
- In Kafka, in-sync-replica set is maintained. [11]
- Secondaries can be used for reads.
12. UPDATE ANYWHERE + EAGER REPLICATION
- Each replica can initiate a new transaction.
- Concurrent transactions can compete with
each other.
- Possibility of deadlocks.
- In the basic Paxos algorithm, there is no
designated leader.
13. PRIMARy COPY + LAZY REPLICATION
- The primary copy can execute updates fast.
- Secondaries can fall behind the primary. It is
called replication lag.
- It can lead to data loss during leader failover, but
still no conflicts.
- Secondaries can be used for reads.
15. QUORUMS
- W + R > N
- W = 3, R = 1, N = 3
- W = 1, R = 3, N = 3
- W = 2, R = 2, N = 3
- If W or R is not met, consistency may be broken.
- Sloppy quorums and hinted handoff.
- Even if W and R are met, it can be still broken.
16. Conflict-free replicated data types (CRDTS)
- Special data types that achieve strong
eventual consistency and monotonicity [2]
- No conflicts
- Merge function has 3 properties:
- Commutative: A + B = B + A
- Associative: A + (B + C) = (A + B) + C
- Idempotent: f(f(x)) = f(x)
- Riak Data Types [3]
17. DISCARDING CONFLICTS: LAST WRITE WINS
- When 2 updates are concurrent, define an
arbitrary order among them.
- i.e., pretend that one of them is more recent.
- Attach a timestamp to each write.
- Cassandra uses physical timestamps [8], [9]
18. DETECTING CONFLICTS: VECTOR CLOCKS
- In Dynamo paper [4], each update is done
against a particular version of a data entry.
- Multiple versions of a data entry can exist together.
- Vector clocks [10] are used to track causality.
- The system can determine the authoritative version:
syntactic reconciliation
- The system cannot reconcile multiple versions:
semantic reconciliation
19. Resolving conflicts and EVENTUAL CONVERGENCE
- Write repair
- Read repair
- Anti-entropy
- Merkle trees
20. Recap
- We apply replication to make our systems
performant and fault tolerant.
- Replication suffers from core problems of
distributed systems.
- We can build many replication protocols that
vary on the 2 dimensions we discussed.
- No silver bullet. It is a world of trade-offs.
21. REFerences
[1] Gray, Jim, et al. "The dangers of replication and a solution." ACM SIGMOD Record 25.2 (1996): 173-182.
[2] Shapiro, Marc, et al. "Conflict-free replicated data types." Symposium on Self-Stabilizing Systems. Springer, Berlin, Heidelberg, 2011.
[3] http://docs.basho.com/riak/kv/2.2.0/learn/concepts/crdts/
[4] DeCandia, Giuseppe, et al. "Dynamo: amazon's highly available key-value store." ACM SIGOPS operating systems review 41.6 (2007): 205-220.
[5] Lamport, Leslie. "Paxos made simple." ACM Sigact News 32.4 (2001): 18-25.
[6] Ongaro, Diego, and John K. Ousterhout. "In Search of an Understandable Consensus Algorithm." USENIX Annual Technical Conference. 2014.
[7] Hunt, Patrick, et al. "ZooKeeper: Wait-free Coordination for Internet-scale Systems." USENIX annual technical conference. Vol. 8. 2010.
[8] http://www.datastax.com/dev/blog/why-cassandra-doesnt-need-vector-clocks
[9] https://aphyr.com/posts/299-the-trouble-with-timestamps
[10] Raynal, Michel, and Mukesh Singhal. "Logical time: Capturing causality in distributed systems." Computer 29.2 (1996): 49-56.
[11] http://kafka.apache.org/documentation.html#replication
[12] http://docs.hazelcast.org/docs/latest/manual/html-single/index.html#managing-cluster-and-member-states