What we see in the modern data store world is a race between different approaches to achieve a distributed and resilient storage of data. Every application needs a stateful layer which holds the data. There are at least three necessary components which are everything else than trivial to combine, and, of course, even more challenging when heading for an acceptable performance.
Over the past years there has been significant progress in both the science and practical implementations of such data stores. In his talk Max Neunhoeffer will introduce the audience to some of the needed ingredients, address the difficulties of their interplay and show four modern approaches of distributed open-source data stores (ArangoDB, Cassandra, Cockroach and RethinkDB).
OSDC 2018 | The Computer science behind a modern distributed data store by Ma...NETWAYS
What we see in the modern data store world is a race between different approaches to achieve a distributed and resilient storage of data. Most applications need a stateful layer which holds the data. There are at least three necessary ingredients which are everything else than trivial to combine and of course even more challenging when heading for an acceptable performance. Over the past years there has been significant progress in respect in both the science and practical implementations of such data stores. In his talk Max Neunhoeffer will introduce the audience to some of the needed ingredients, address the difficulties of their interplay and show four modern approaches of distributed open-source data stores.
Topics are:
– Challenges in developing a distributed, resilient data store
– Consensus, distributed transactions, distributed query optimization and execution
– The inner workings of ArangoDB, Cassandra, Cockroach and RethinkDB
The talk will touch complex and difficult computer science, but will at the same time be accessible to and enjoyable by a wide range of developers.
The Computer Science Behind a modern Distributed DatabaseArangoDB Database
What we see in the modern data store world is a race between different approaches to achieve a distributed and resilient storage of data. Every application needs a stateful layer which holds the data. There are several different necessary components which are anything but trivial to combine, and, of course, even more challenging when attempting to optimize for performance. Over the past years there has been significant progress in both the science and practical implementations of such data stores. In this talk Dan Larkin-York will introduce the audience to some of the challenges, address the difficulties of their interplay, and cover key approaches taken by some of the industry’s leaders (ArangoDB, Cassandra, CockroachDB, MarkLogic, and more).
Data FAIRport Skunkworks: Common Repository Access Via Meta-Metadata Descript...datascienceiqss
It would be useful to be able to discover what kinds of data are contained in the myriad general-purpose public data repositories. It would be even better if it were possible to query that data and/or have that data conform to a particular context-dependent data format. This was the ambition of the Data FAIRport project. I will be demonstrating the "strawman" demonstration of a fully-functional Data FAIRport, where the meta/data in a public repository can be "projected" into one of a number of different context-dependent formats, such that it can be cross-queried in combination with the (potentially "projected") data from other repositories.
The computational requirements of next generation sequencing is placing a huge demand on IT organisations .
Building compute clusters is now a well understood and relatively straightforward problem. However, NGS sequencing applications require large amounts of storage, and high IO rates.
This talk details our approach for providing storage for next-gen sequencing applications.
Talk given at BIO-IT World, Europe, 2009.
Data FAIRport Prototype & Demo - Presentation to Elsevier, Jul 10, 2015Mark Wilkinson
A discussion and demonstration of a functional Data FAIRport, using W3C's Linked Data Platform, Ruben Verborgh's Linked Data Fragments, and Hydra's hypermedia controlled vocabularies. This is the output of the "Skunkworks" working group of the larger Data FAIRport project (http://datafairport.org).
OSDC 2018 | The Computer science behind a modern distributed data store by Ma...NETWAYS
What we see in the modern data store world is a race between different approaches to achieve a distributed and resilient storage of data. Most applications need a stateful layer which holds the data. There are at least three necessary ingredients which are everything else than trivial to combine and of course even more challenging when heading for an acceptable performance. Over the past years there has been significant progress in respect in both the science and practical implementations of such data stores. In his talk Max Neunhoeffer will introduce the audience to some of the needed ingredients, address the difficulties of their interplay and show four modern approaches of distributed open-source data stores.
Topics are:
– Challenges in developing a distributed, resilient data store
– Consensus, distributed transactions, distributed query optimization and execution
– The inner workings of ArangoDB, Cassandra, Cockroach and RethinkDB
The talk will touch complex and difficult computer science, but will at the same time be accessible to and enjoyable by a wide range of developers.
The Computer Science Behind a modern Distributed DatabaseArangoDB Database
What we see in the modern data store world is a race between different approaches to achieve a distributed and resilient storage of data. Every application needs a stateful layer which holds the data. There are several different necessary components which are anything but trivial to combine, and, of course, even more challenging when attempting to optimize for performance. Over the past years there has been significant progress in both the science and practical implementations of such data stores. In this talk Dan Larkin-York will introduce the audience to some of the challenges, address the difficulties of their interplay, and cover key approaches taken by some of the industry’s leaders (ArangoDB, Cassandra, CockroachDB, MarkLogic, and more).
Data FAIRport Skunkworks: Common Repository Access Via Meta-Metadata Descript...datascienceiqss
It would be useful to be able to discover what kinds of data are contained in the myriad general-purpose public data repositories. It would be even better if it were possible to query that data and/or have that data conform to a particular context-dependent data format. This was the ambition of the Data FAIRport project. I will be demonstrating the "strawman" demonstration of a fully-functional Data FAIRport, where the meta/data in a public repository can be "projected" into one of a number of different context-dependent formats, such that it can be cross-queried in combination with the (potentially "projected") data from other repositories.
The computational requirements of next generation sequencing is placing a huge demand on IT organisations .
Building compute clusters is now a well understood and relatively straightforward problem. However, NGS sequencing applications require large amounts of storage, and high IO rates.
This talk details our approach for providing storage for next-gen sequencing applications.
Talk given at BIO-IT World, Europe, 2009.
Data FAIRport Prototype & Demo - Presentation to Elsevier, Jul 10, 2015Mark Wilkinson
A discussion and demonstration of a functional Data FAIRport, using W3C's Linked Data Platform, Ruben Verborgh's Linked Data Fragments, and Hydra's hypermedia controlled vocabularies. This is the output of the "Skunkworks" working group of the larger Data FAIRport project (http://datafairport.org).
Containers and Developer Defined Data Centers - Evan Powell - Keynote in Bang...CodeOps Technologies LLP
DevOps and Containers go hand in hand. DevOps industry is expected to benefit significantly benefit from the container eco-system and technology. This keynote talks about the challenges and opportunities around deploying containers into production use cases.
Basho and Riak at GOTO Stockholm: "Don't Use My Database."Basho Technologies
What are common use cases for NoSQL? When should I avoid NoSQL? When is RDBMS just fine?
This presentation, delivered at the GOTO NoSQL Roadshow events in London and Stockholm in November of 2011 by Basho co-founder and COO, Antony Falco, take a no-BS look at the tradeoffs one must make to gain the advantages offered by distributed databases like Riak.
UnConference for Georgia Southern Computer Science March 31, 2015Christopher Curtin
I presented to the Georgia Southern Computer Science ACM group. Rather than one topic for 90 minutes, I decided to do an UnConference. I presented them a list of 8-9 topics, let them vote on what to talk about, then repeated.
Each presentation was ~8 minutes, (Except Career) and was by no means an attempt to explain the full concept or technology. Only to wake up their interest.
Software and the Concurrency Revolution : NotesSubhajit Sahu
Highlighted notes of article while studying Concurrent Data Structures, CSE:
Software and the Concurrency Revolution
Herb Sutter
Software Architect, Microsoft
Software Development Consultant, www.gotw.ca/training
Herb Sutter is a prominent C++ expert. He is also a book author and was a columnist for Dr. Dobb's Journal. He joined Microsoft in 2002 as a platform evangelist for Visual C++ .NET, rising to lead software architect for C++/CLI.
In 1971, David Parnas wrote the great paper, "On the criteria to be used decomposing the system into parts," and yet the problem of breaking down big projects into small parts that work well together remains a struggle in the industry. The ability to decompose a problem space and in turn, compose a solution is essential to our work.
Things have gotten worse since 1971. With microservices, big data, and streaming systems, we're all going to be distributed systems engineers sooner or later. In distributed systems, effective decomposition has an even greater impact on the reliability, performance, and availability of our systems as it determines the frequency and weight of communication in the system.
This talk speaks to the essential considerations for defining and evaluating boundaries and behaviors in large-scale distributed systems. It will touch on topics such as bulkhead design and architectural evolution.
Cache Consistency – Requirements and its packet processing Performance implic...Michelle Holley
The audio starts at 9:00 due to a glitch while recording.
The topic can be as well stated as “All Processor Accesses are not created equally” – H/W and S/W Synergy – Methods and Mechanism:
- The developers in the audience, throughout the class, will wear different hats – first as a hard design engineer and look at platform design choices.
- Then they will wear the hat of driver developer and see the requirements and assumptions of driver on the hardware behavior.
- Then they will wear the hat of system architect and venture out to find out choices as how s/w and h/w can be in synergy and communicate – thereby potential optimum implementation can be made.
About the presenter: M Jay (Muthurajan Jayakumar) has worked with the DPDK team since 2009. He joined Intel in 1991 and has been in various roles and divisions: 64-bit CPU front side bus architect, 64 bit HAL developer, among others, before he joined the DPDK team. M Jay holds 21 US patents, both individually and jointly, all issued while working at Intel. M Jay was awarded the Intel Achievement Award in 2016, Intel's highest honor based on innovation and results.
Massively scalable ETL in real world applications: the hard wayJ On The Beach
Big Data examples always give the correct answers. However, in the real world, Big Data might be corrupt, contradictory or consist of so many small files it becomes extremely hard to keep track - let alone scale. A solid architecture will help to overcome many of the difficulties.
Floris will talk about a real-world implementation of a massively scalable ETL architecture. Two years ago, at the time of the implementation, Airflow just became part of Apache and still left many features to be desired for. However, requirements from the start were thousands of ETL tasks per day on average, but on occasion, this could become hundreds of thousands. The script-based method that was in place was already not capable to meet the requirements on a day to day basis and needed to be replaced as soon as possible. So this custom framework was rolled out in just 8 weeks of development time.
Traditional Big Data is done on Data you have. You load the data into a repository and perform map reduce or other style calculations on the data. However, certain industries need to perform complex operations on data you might not have. Data you can acquire, Data that can be shared with you, and Data that you can model are all types of data you may not have but may need to integrate instantly into a complex data analysis. Problem is: you may not even know you need this data until deep into the execution stack at runtime. This talk discusses a new functional language paradigm for dealing naturally with data you don’t have and about how to make all data first-class citizens, regardless of whether you have it or you don’t, and we will give a demo of a project written in Scala to deal exactly with this issue.
More Related Content
Similar to The computer science behind a modern disributed data store
Containers and Developer Defined Data Centers - Evan Powell - Keynote in Bang...CodeOps Technologies LLP
DevOps and Containers go hand in hand. DevOps industry is expected to benefit significantly benefit from the container eco-system and technology. This keynote talks about the challenges and opportunities around deploying containers into production use cases.
Basho and Riak at GOTO Stockholm: "Don't Use My Database."Basho Technologies
What are common use cases for NoSQL? When should I avoid NoSQL? When is RDBMS just fine?
This presentation, delivered at the GOTO NoSQL Roadshow events in London and Stockholm in November of 2011 by Basho co-founder and COO, Antony Falco, take a no-BS look at the tradeoffs one must make to gain the advantages offered by distributed databases like Riak.
UnConference for Georgia Southern Computer Science March 31, 2015Christopher Curtin
I presented to the Georgia Southern Computer Science ACM group. Rather than one topic for 90 minutes, I decided to do an UnConference. I presented them a list of 8-9 topics, let them vote on what to talk about, then repeated.
Each presentation was ~8 minutes, (Except Career) and was by no means an attempt to explain the full concept or technology. Only to wake up their interest.
Software and the Concurrency Revolution : NotesSubhajit Sahu
Highlighted notes of article while studying Concurrent Data Structures, CSE:
Software and the Concurrency Revolution
Herb Sutter
Software Architect, Microsoft
Software Development Consultant, www.gotw.ca/training
Herb Sutter is a prominent C++ expert. He is also a book author and was a columnist for Dr. Dobb's Journal. He joined Microsoft in 2002 as a platform evangelist for Visual C++ .NET, rising to lead software architect for C++/CLI.
In 1971, David Parnas wrote the great paper, "On the criteria to be used decomposing the system into parts," and yet the problem of breaking down big projects into small parts that work well together remains a struggle in the industry. The ability to decompose a problem space and in turn, compose a solution is essential to our work.
Things have gotten worse since 1971. With microservices, big data, and streaming systems, we're all going to be distributed systems engineers sooner or later. In distributed systems, effective decomposition has an even greater impact on the reliability, performance, and availability of our systems as it determines the frequency and weight of communication in the system.
This talk speaks to the essential considerations for defining and evaluating boundaries and behaviors in large-scale distributed systems. It will touch on topics such as bulkhead design and architectural evolution.
Cache Consistency – Requirements and its packet processing Performance implic...Michelle Holley
The audio starts at 9:00 due to a glitch while recording.
The topic can be as well stated as “All Processor Accesses are not created equally” – H/W and S/W Synergy – Methods and Mechanism:
- The developers in the audience, throughout the class, will wear different hats – first as a hard design engineer and look at platform design choices.
- Then they will wear the hat of driver developer and see the requirements and assumptions of driver on the hardware behavior.
- Then they will wear the hat of system architect and venture out to find out choices as how s/w and h/w can be in synergy and communicate – thereby potential optimum implementation can be made.
About the presenter: M Jay (Muthurajan Jayakumar) has worked with the DPDK team since 2009. He joined Intel in 1991 and has been in various roles and divisions: 64-bit CPU front side bus architect, 64 bit HAL developer, among others, before he joined the DPDK team. M Jay holds 21 US patents, both individually and jointly, all issued while working at Intel. M Jay was awarded the Intel Achievement Award in 2016, Intel's highest honor based on innovation and results.
Similar to The computer science behind a modern disributed data store (20)
Massively scalable ETL in real world applications: the hard wayJ On The Beach
Big Data examples always give the correct answers. However, in the real world, Big Data might be corrupt, contradictory or consist of so many small files it becomes extremely hard to keep track - let alone scale. A solid architecture will help to overcome many of the difficulties.
Floris will talk about a real-world implementation of a massively scalable ETL architecture. Two years ago, at the time of the implementation, Airflow just became part of Apache and still left many features to be desired for. However, requirements from the start were thousands of ETL tasks per day on average, but on occasion, this could become hundreds of thousands. The script-based method that was in place was already not capable to meet the requirements on a day to day basis and needed to be replaced as soon as possible. So this custom framework was rolled out in just 8 weeks of development time.
Traditional Big Data is done on Data you have. You load the data into a repository and perform map reduce or other style calculations on the data. However, certain industries need to perform complex operations on data you might not have. Data you can acquire, Data that can be shared with you, and Data that you can model are all types of data you may not have but may need to integrate instantly into a complex data analysis. Problem is: you may not even know you need this data until deep into the execution stack at runtime. This talk discusses a new functional language paradigm for dealing naturally with data you don’t have and about how to make all data first-class citizens, regardless of whether you have it or you don’t, and we will give a demo of a project written in Scala to deal exactly with this issue.
Acoustic Time Series in Industry 4.0: Improved Reliability and Cyber-Security...J On The Beach
Industry 4.0, aka the "Fourth Industrial Revolution," refers to the computerization of manufacturing. One important aspect of Industry 4.0 is the ability to monitor the health and reliability of a physical manufacturing plant using low-cost IoT sensors. For example, machine learning models can be trained to predict the physical degradation of a manufacturing system as a function of acoustic measurements obtained from strategically placed microphones; however, the same acoustic measurements can be used to reverse engineer proprietary information about the manufacturing process and/or precisely what is being manufactured at the time of recording. Thus, improved reliability and fault tolerance is achieved at the cost of what appears to be an unprecedented new class of security vulnerabilities related to the acoustic side channel.
As a case study, we report a novel acoustic side channel attack against a commercial DNA synthesizer, a commonly used instrument in fields such as synthetic biology. Using a smart phone-quality microphone placed on or in the near vicinity of a DNA synthesizer, we were able to determine with 88.07% accuracy the sequence of DNA being produced; using a database of biologically relevant known-sequences, we increased the accuracy of our model to 100%. An academic or industrial research project may use the synthetic DNA to engineer an organism with desired traits or functions; however, while the organism is still under development, prior to publication, patent, and/or copyright, the research remains vulnerable to academic intellectual property theft and/or industrial espionage. On the other hand, this attack could also be used for benevolent purposes, for example, to determine whether a suspected criminal or terrorist is engineering a harmful pathogen. Thus, it is essential to recognize both the benefits and risks inherent to the cyber-physical systems that will inevitably control Industry 4.0 manufacturing processes and to take steps to mitigate them whenever possible.
Where is the edge in IoT and how much can you do there? Data collection? Analytics? I’ll show you how to build and deploy an embedded IoT edge platform that can do data collection, analytics, dashboarding and much more. All using Open Source.
As IoT deployments move forward, the need to collect, analyze, and respond to data further out on the edge becomes a critical factor in the success – or failure – of any IoT project. Network bandwidth costs may be dropping, and storage is cheaper than ever, but at IoT scale, these costs can still quickly overrun a project’s budget and ultimately doom it to failure.
The more you centralize your data collection and storage, the higher these costs become. Edge data collection and analysis can dramatically lower these costs, plus decrease the time to react to critical sensor data. With most data platforms, it simply isn’t practical, or even possible, to push collection AND analytics to the edge. In this talk I’ll show how I’ve done exactly this with a combination of open source hardware – Pine64 – and open source software – InfluxDB – to build a practical, efficient and scalable data collection and analysis gateway device for IoT deployments. The edge is where the data is, so the edge is where the data collection and analytics needs to be.
Drinking from the firehose, with virtual streams and virtual actorsJ On The Beach
Event Stream Processing is a popular paradigm for building robust and performant systems in many different domains, from IoT to fraud detection to high-frequency trading. Because of the wide range of scenarios and requirements, it is difficult to conceptualize a unified programming model that would be equally applicable to all of them. Another tough challenge is how to build streaming systems with cardinalities of topics ranging from hundreds to billions while delivering good performance and scalability.
In this session, Sergey Bykov will talk about the journey of building Orleans Streams that originated in gaming and monitoring scenarios, and quickly expanded beyond them. He will cover the programming model of virtual streams that emerged as a natural extension of the virtual actor model of Orleans, the architecture of the underlying runtime system, the compromises and hard choices made in the process. Sergey will share the lessons learned from the experience of running the system in production, and future ideas and opportunities that remain to be explored.
Over the last twenty years, there has been a paradigm shift in software development: from meticulously planned release cycles to an experimental way of working in which lead times are becoming shorter and shorter.
How can Java ever keep up with this trend when we have Docker containers that are several hundred megabytes in size, with warm-up times of ten minutes or longer? In this talk, I'll demonstrate how we can use Quarkus so that we can create super small, super fast Java containers! This will give us better possibilities for scaling up and down - which can be a game-changer, especially in a serverless environment. It will also provide the shortest possible lead times, as well as a much better use of cloud performance with the added bonus of lower costs.
When Cloud Native meets the Financial SectorJ On The Beach
We live in our own bubble of microservices and endlessly horizontal scaling infrastructure, but there is still critical infrastructure that runs the world of financial systems depending on Windows boxes, FTP servers, and single-threaded protocols. This talk is about how to glue these two worlds together, what works for us and what doesn't.
The advancement of technology in the last decade or so has allowed astronomy to see exponential growth in data volumes. ESA's space telescope Euclid will gather high-resolution images of a third of the sky, ~850GB of data downloaded daily for 6 years, by 2032 ground-based telescope LSST will have generated 500PB of data and the radio telescope SKA will be producing more data per second than the entire internet worldwide. This talk will address the questions of what current techniques exist to address big data volumes, how the astronomical community will prepare for this big data wave, and what other challenges lie ahead?
The world is moving from a model where data sits at rest, waiting for people to make requests of it, to where data is constantly moving and streams of data flow to and from devices with or without human interaction. Decisions need to be made based on these streams of data in real-time, models need to be updated, and intelligence needs to be gathered. In this context, our old-fashioned approach of CRUD REST APIs serving CRUD database calls just doesn't cut it. It's time we moved to a stream-centric view of the world.
The TIPPSS Imperative for IoT - Ensuring Trust, Identity, Privacy, Protection...J On The Beach
Our increasingly connected world leveraging the Internet of Things (IoT) creates great value, in connected healthcare, smart cities, and more. The increasing use of IoT also creates great risk. We will discuss the challenges and risks we need to address as developers in TIPPSS - Trust, Identity, Privacy, Protection, Safety, and Security - for devices, systems and solutions we deliver and use. Florence leads IEEE workstreams on clinical IoT and data interoperability with blockchain addressing TIPPSS issues. She is an author of IEEE articles on "Enabling Trust and Security - TIPPSS for IoT" and "Wearables and Medical Interoperability - the Evolving Frontier", "TIPPSS for Smart Cities" in the 2017 book "Creating, Analysing and Sustaining Smarter Cities: A Systems Perspective" , and Editor in Chief for an upcoming book on "Women Securing the Future with TIPPSS for IoT."
Pushing AI to the Client with WebAssembly and BlazorJ On The Beach
Want to run your AI algorithms directly in the browser on the client-side? Now you can with WebAssembly and Blazor. Join us as we write code directly in WebAssembly. Then, we’ll look at Blazor and how you can use it, along with WebAssembly to run your tooling client side in the browser.
Want to run your AI algorithms directly in the browser on the client-side without the need for transpilers or browser plug-ins? Well, now you can with WebAssembly and Blazor. WebAssembly (WASM) is the W3C specification that will be used to provide the next generation of development tools for the web and beyond. Blazor is Microsoft’s experiment that allows ASP.Net developers to create web pages that do much of the scripting work in C# using WASM. Come join us as we learn to write code directly in WebAssembly’s human-readable format. Then, we’ll look at the current state of Blazor and how you can use it, along with WebAssembly to run your tooling client side in the browser.
RAFT protocol is a well-known protocol for consensus in Distributed Systems. Want to learn how consensus is achieved in a system with a large amount of data such as Axon Server’s Event Store? Join this talk to hear about all specifics regarding data replication in highly available Event Store!
Axon is a free and open source Java framework for writing Java applications following DDD, event sourcing, and CQRS principles. While especially useful in a microservices context, Axon provides great value in building structured monoliths that can be broken down into microservices when needed.
Axon Server is a messaging platform specifically built to support distributed Axon applications. One of its key benefits is storing events published by Axon applications. In not so rare cases, the number of these events is over millions, even billions. Availability of Axon Server plays a significant role in the product portfolio. To keep event replication reliable we chose RAFT protocol for consensus implementation of our clustering features.
In short, consensus involves multiple servers agreeing on values. Once they reach a decision on a value, that decision is final. Typical consensus algorithms make progress when any majority of their servers is available; for example, a cluster of 5 servers can continue to operate even if 2 servers fail. If more servers fail, they stop making progress (but will never return an incorrect result).
Join this talk to learn why we chose RAFT; what were our findings during the design, the implementation, and testing phase; and what does it mean to replicate an event store holding billions of events!
The Six Pitfalls of building a Microservices Architecture (and how to avoid t...J On The Beach
Thinking of moving to Microservices? Watch out! That quest is full of traps, social traps. If you are not able to handle it, you may be blocked by meetings, frustration, endless challenges that will make you miss the monolith. In this talk, I share my experience and mistakes, so you can avoid them.
Creating or migrating to a Microservices architecture might easily become a big mess, not only due to technical challenges but mostly because of human factors: it’s a major change in the software culture of a company. In this talk, I’ll share my past experience as the technical lead of an ambitious Microservices-based product, I’ll go through the parts we struggled with, and give you some advice on how to deal with what I call the Six Pitfalls:
The Common Patterns Phobia
The Book Club Cult
The Never-Decoupled Story
The Buzz Words Syndrome
The Agile Trap
The Conway’s Law Hackers
Instead of randomly injecting faults ( i.e. Chaos Monkey), what if we could order our experiments to perform min number of experiments for maximum yield? We present a solution(& results) to the problem of experiment selection using Lineage Driven Fault Injection to reduce the search space of faults.
Lineage Driven Fault Injection (LDFI) is a state of the art technique in chaos engineering experiment selection. LDFI since its inception has used an SAT solver under the hood which presents solutions to the decision problem (which faults to inject) in no particular order. As SRE’s we would like to perform experiments that reveal the bugs that the customers are most likely to hit first. In this talk, we present new improvements to LDFI that orders the experiment suggestions.
In the first the half of the talk we will show LDFI is a technique that can be widely used within an enterprise. We present the motivation for ordering the chaos experiments along with some prioritization we utilized while conducting the experiments. We also highlight how ordering is a general purpose technique that we can use to encode the peculiarities of a heterogeneous microservices architecture. LDFI can work in an enterprise by harnessing the observability infrastructure to model the redundancy of the system.
Next, we present experiments conducted within our organization using ordered LDFI and some preliminary results. We show examples of services where we discovered bugs, and how carefully controlling the order of experiments allowed LDFI to avoid running unnecessary experiments. We also present an example of an application where we declared the service shippable under crash stop model. We also present a comparison with Chaos Monkey and show how LDFI found the known bugs in a given application using orders of magnitude fewer experiments than a random fault injection tool like Chaos Monkey.
Finally, we discuss how we plan to take LDFI forward. We discuss open problems and possible solutions for scalarizing probabilities of failure, latency injection, integration with service mesh technologies like envoy for fine-grained fault injection, fault injection for stateful systems.
Key takeaways: 1) Understand how LDFI can be integrated in the enterprise by harnessing the observability infrastructure. 2) Limitations of LDFI w.r.t unordered solutions and why ordering matters for chaos engineering experiments. 3) Preliminary results of prioritized LDFI and a future direction for the community.
Complexity in systems should be defeated if it is possible to do. But the default nature of our computer systems are complex and servers are doomed to fail. In this talk, we will go through new approaches in modern architectures to design and evaluate new computer systems.
Interaction Protocols: It's all about good mannersJ On The Beach
Distributed systems collaborate to achieve collective goals via a system of rules. Rules that affords good hygiene, fault tolerance, effective communication and trusted feedback. These rules form protocols which enable the system to achieve its goals.
Distributed and concurrent systems can be considered a social group that collaborates to achieve collective goals. In order to collaborate a system of rules must be applied, that affords good hygiene, fault tolerance, and effective communication to coordinate, share knowledge, and provide feedback in a polite trusted manner. These rules form a number of protocols which enable the group to act as a system which is greater than the sum of the individual components.
In this talk, we will explore the history of protocols and their application when building distributed systems.
A race of two compilers: GraalVM JIT versus HotSpot JIT C2. Which one offers ...J On The Beach
Do you want to check the efficiency of the new, state of the art, GraalVM JIT Compiler in comparison to the old but mostly used JIT C2? Let’s have a side by side comparison from a performance standpoint on the same source code.
The talk reveals how traditional Just In Time Compiler (e.g. JIT C2) from HotSpot/OpenJDK internally manages runtime optimizations for hot methods in comparison to the new, state of the art, GraalVM JIT Compiler on the same source code, emphasizing all of the internals and strategies used by each Compiler to achieve better performance in most common situations (or code patterns). For each optimization, there is Java source code and corresponding generated assembly code in order to prove what really happens under the hood.
Each test is covered by a dedicated benchmark (JMH), timings and conclusions. Main topics of the agenda: - Scalar replacement - Null Checks - Virtual calls - Lock coarsening - Lock elision - Virtual calls - Scalar replacement - Lambdas - Vectorization (few cases)
The tools used during my research study are JITWatch, Java Measurement Harness, and perf. All test scenarios will be launched against the latest official Java release (e.g. version 11).
Leadership is easy when you're a manager, or an expert in a field, or a conference speaker! In a Kanban organisation, though, we "encourage acts of leadership at every level". In this talk, we look at what it means to be a leader in the uncertain, changing and high-learning environment of software development. We learn about the importance of safety in encouraging others to lead and follow, and how to get that safety using both technical and human practices; the necessity of a clear, compelling vision and provision of information on how we're achieving it; and the need to be able to ask awkward and difficult questions... especially the ones without easy answers.
Machine Learning: The Bare Math Behind LibrariesJ On The Beach
During this presentation, we will answer how much you’ll need to invest in a superhero costume to be as popular as Superman. We will generate a unique logo which will stand against the ever popular Batman and create new superhero teams. We shall achieve it using linear regression and neural networks.
Machine learning is one of the hottest buzzwords in technology today as well as one of the most innovative fields in computer science – yet people use libraries as black boxes without basic knowledge of the field. In this session, we will strip them to bare math, so next time you use a machine learning library, you’ll have a deeper understanding of what lies underneath.
During this session, we will first provide a short history of machine learning and an overview of two basic teaching techniques: supervised and unsupervised learning.
We will start by defining what machine learning is and equip you with an intuition of how it works. We will then explain the gradient descent algorithm with the use of simple linear regression to give you an even deeper understanding of this learning method. Then we will project it to supervised neural networks training.
Within unsupervised learning, you will become familiar with Hebb’s learning and learning with concurrency (winner takes all and winner takes most algorithms). We will use Octave for examples in this session; however, you can use your favourite technology to implement presented ideas.
Our aim is to show the mathematical basics of neural networks for those who want to start using machine learning in their day-to-day work or use it already but find it difficult to understand the underlying processes. After viewing our presentation, you should find it easier to select parameters for your networks and feel more confident in your selection of network type, as well as be encouraged to dive into more complex and powerful deep learning methods.
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdfPaige Cruz
Monitoring and observability aren’t traditionally found in software curriculums and many of us cobble this knowledge together from whatever vendor or ecosystem we were first introduced to and whatever is a part of your current company’s observability stack.
While the dev and ops silo continues to crumble….many organizations still relegate monitoring & observability as the purview of ops, infra and SRE teams. This is a mistake - achieving a highly observable system requires collaboration up and down the stack.
I, a former op, would like to extend an invitation to all application developers to join the observability party will share these foundational concepts to build on:
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...Ramesh Iyer
In today's fast-changing business world, Companies that adapt and embrace new ideas often need help to keep up with the competition. However, fostering a culture of innovation takes much work. It takes vision, leadership and willingness to take risks in the right proportion. Sachin Dev Duggal, co-founder of Builder.ai, has perfected the art of this balance, creating a company culture where creativity and growth are nurtured at each stage.
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
Generative AI Deep Dive: Advancing from Proof of Concept to ProductionAggregage
Join Maher Hanafi, VP of Engineering at Betterworks, in this new session where he'll share a practical framework to transform Gen AI prototypes into impactful products! He'll delve into the complexities of data collection and management, model selection and optimization, and ensuring security, scalability, and responsible use.
Essentials of Automations: Optimizing FME Workflows with ParametersSafe Software
Are you looking to streamline your workflows and boost your projects’ efficiency? Do you find yourself searching for ways to add flexibility and control over your FME workflows? If so, you’re in the right place.
Join us for an insightful dive into the world of FME parameters, a critical element in optimizing workflow efficiency. This webinar marks the beginning of our three-part “Essentials of Automation” series. This first webinar is designed to equip you with the knowledge and skills to utilize parameters effectively: enhancing the flexibility, maintainability, and user control of your FME projects.
Here’s what you’ll gain:
- Essentials of FME Parameters: Understand the pivotal role of parameters, including Reader/Writer, Transformer, User, and FME Flow categories. Discover how they are the key to unlocking automation and optimization within your workflows.
- Practical Applications in FME Form: Delve into key user parameter types including choice, connections, and file URLs. Allow users to control how a workflow runs, making your workflows more reusable. Learn to import values and deliver the best user experience for your workflows while enhancing accuracy.
- Optimization Strategies in FME Flow: Explore the creation and strategic deployment of parameters in FME Flow, including the use of deployment and geometry parameters, to maximize workflow efficiency.
- Pro Tips for Success: Gain insights on parameterizing connections and leveraging new features like Conditional Visibility for clarity and simplicity.
We’ll wrap up with a glimpse into future webinars, followed by a Q&A session to address your specific questions surrounding this topic.
Don’t miss this opportunity to elevate your FME expertise and drive your projects to new heights of efficiency.
UiPath Test Automation using UiPath Test Suite series, part 4DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap.
The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies.
Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques
What will you get from this session?
1. Insights into SAP testing best practices
2. Heatmap utilization for testing
3. Optimization of testing processes
4. Demo
Topics covered:
Execution from the test manager
Orchestrator execution result
Defect reporting
SAP heatmap example with demo
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Welocme to ViralQR, your best QR code generator.ViralQR
Welcome to ViralQR, your best QR code generator available on the market!
At ViralQR, we design static and dynamic QR codes. Our mission is to make business operations easier and customer engagement more powerful through the use of QR technology. Be it a small-scale business or a huge enterprise, our easy-to-use platform provides multiple choices that can be tailored according to your company's branding and marketing strategies.
Our Vision
We are here to make the process of creating QR codes easy and smooth, thus enhancing customer interaction and making business more fluid. We very strongly believe in the ability of QR codes to change the world for businesses in their interaction with customers and are set on making that technology accessible and usable far and wide.
Our Achievements
Ever since its inception, we have successfully served many clients by offering QR codes in their marketing, service delivery, and collection of feedback across various industries. Our platform has been recognized for its ease of use and amazing features, which helped a business to make QR codes.
Our Services
At ViralQR, here is a comprehensive suite of services that caters to your very needs:
Static QR Codes: Create free static QR codes. These QR codes are able to store significant information such as URLs, vCards, plain text, emails and SMS, Wi-Fi credentials, and Bitcoin addresses.
Dynamic QR codes: These also have all the advanced features but are subscription-based. They can directly link to PDF files, images, micro-landing pages, social accounts, review forms, business pages, and applications. In addition, they can be branded with CTAs, frames, patterns, colors, and logos to enhance your branding.
Pricing and Packages
Additionally, there is a 14-day free offer to ViralQR, which is an exceptional opportunity for new users to take a feel of this platform. One can easily subscribe from there and experience the full dynamic of using QR codes. The subscription plans are not only meant for business; they are priced very flexibly so that literally every business could afford to benefit from our service.
Why choose us?
ViralQR will provide services for marketing, advertising, catering, retail, and the like. The QR codes can be posted on fliers, packaging, merchandise, and banners, as well as to substitute for cash and cards in a restaurant or coffee shop. With QR codes integrated into your business, improve customer engagement and streamline operations.
Comprehensive Analytics
Subscribers of ViralQR receive detailed analytics and tracking tools in light of having a view of the core values of QR code performance. Our analytics dashboard shows aggregate views and unique views, as well as detailed information about each impression, including time, device, browser, and estimated location by city and country.
So, thank you for choosing ViralQR; we have an offer of nothing but the best in terms of QR code services to meet business diversity!
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
Key Trends Shaping the Future of Infrastructure.pdfCheryl Hung
Keynote at DIGIT West Expo, Glasgow on 29 May 2024.
Cheryl Hung, ochery.com
Sr Director, Infrastructure Ecosystem, Arm.
The key trends across hardware, cloud and open-source; exploring how these areas are likely to mature and develop over the short and long-term, and then considering how organisations can position themselves to adapt and thrive.
A tale of scale & speed: How the US Navy is enabling software delivery from l...sonjaschweigert1
Rapid and secure feature delivery is a goal across every application team and every branch of the DoD. The Navy’s DevSecOps platform, Party Barge, has achieved:
- Reduction in onboarding time from 5 weeks to 1 day
- Improved developer experience and productivity through actionable findings and reduction of false positives
- Maintenance of superior security standards and inherent policy enforcement with Authorization to Operate (ATO)
Development teams can ship efficiently and ensure applications are cyber ready for Navy Authorizing Officials (AOs). In this webinar, Sigma Defense and Anchore will give attendees a look behind the scenes and demo secure pipeline automation and security artifacts that speed up application ATO and time to production.
We will cover:
- How to remove silos in DevSecOps
- How to build efficient development pipeline roles and component templates
- How to deliver security artifacts that matter for ATO’s (SBOMs, vulnerability reports, and policy evidence)
- How to streamline operations with automated policy checks on container images
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
5. Resilience and Consensus
The Problem
A modern data store is distributed, because it needs to scale out and/or
be resilient.
6. Resilience and Consensus
The Problem
A modern data store is distributed, because it needs to scale out and/or
be resilient.
Different parts of the system need to agree on things.
7. Resilience and Consensus
The Problem
A modern data store is distributed, because it needs to scale out and/or
be resilient.
Different parts of the system need to agree on things.
Consensus is the art to achieve this as well as possible in software.
This is relatively easy, if things are good, but very hard, if:
8. Resilience and Consensus
The Problem
A modern data store is distributed, because it needs to scale out and/or
be resilient.
Different parts of the system need to agree on things.
Consensus is the art to achieve this as well as possible in software.
This is relatively easy, if things are good, but very hard, if:
the network has outages,
the network has dropped or delayed or duplicated packets,
disks fail (and come back with corrupt data),
machines fail (and come back with old data),
racks fail (and come back with or without data).
9. Resilience and Consensus
The Problem
A modern data store is distributed, because it needs to scale out and/or
be resilient.
Different parts of the system need to agree on things.
Consensus is the art to achieve this as well as possible in software.
This is relatively easy, if things are good, but very hard, if:
the network has outages,
the network has dropped or delayed or duplicated packets,
disks fail (and come back with corrupt data),
machines fail (and come back with old data),
racks fail (and come back with or without data).
(And we have not even talked about malicious attacks and enemy action.)
10. Paxos and Raft
Traditionally, one uses the Paxos Consensus Protocol (1998).
More recently, Raft (2013) has been proposed.
Paxos is a challenge to understand and to implement efficiently.
Various variants exist.
Raft is designed to be understandable.
11. Paxos and Raft
Traditionally, one uses the Paxos Consensus Protocol (1998).
More recently, Raft (2013) has been proposed.
Paxos is a challenge to understand and to implement efficiently.
Various variants exist.
Raft is designed to be understandable.
My advice:
First try to understand Paxos for some time (do not implement it!), then
enjoy the beauty of Raft,
12. Paxos and Raft
Traditionally, one uses the Paxos Consensus Protocol (1998).
More recently, Raft (2013) has been proposed.
Paxos is a challenge to understand and to implement efficiently.
Various variants exist.
Raft is designed to be understandable.
My advice:
First try to understand Paxos for some time (do not implement it!), then
enjoy the beauty of Raft, but do not implement it either!
13. Paxos and Raft
Traditionally, one uses the Paxos Consensus Protocol (1998).
More recently, Raft (2013) has been proposed.
Paxos is a challenge to understand and to implement efficiently.
Various variants exist.
Raft is designed to be understandable.
My advice:
First try to understand Paxos for some time (do not implement it!), then
enjoy the beauty of Raft, but do not implement it either!
Use some battle-tested implementation you trust!
14. Paxos and Raft
Traditionally, one uses the Paxos Consensus Protocol (1998).
More recently, Raft (2013) has been proposed.
Paxos is a challenge to understand and to implement efficiently.
Various variants exist.
Raft is designed to be understandable.
My advice:
First try to understand Paxos for some time (do not implement it!), then
enjoy the beauty of Raft, but do not implement it either!
Use some battle-tested implementation you trust!
But most importantly: DO NOT TRY TO INVENT YOUR OWN!
15. Raft in a slide
An odd number of servers each keep a persisted log of events.
16. Raft in a slide
An odd number of servers each keep a persisted log of events.
Everything is replicated to everybody.
17. Raft in a slide
An odd number of servers each keep a persisted log of events.
Everything is replicated to everybody.
They democratically elect a leader with absolute majority.
18. Raft in a slide
An odd number of servers each keep a persisted log of events.
Everything is replicated to everybody.
They democratically elect a leader with absolute majority.
Only the leader may append to the replicated log.
19. Raft in a slide
An odd number of servers each keep a persisted log of events.
Everything is replicated to everybody.
They democratically elect a leader with absolute majority.
Only the leader may append to the replicated log.
An append only counts when a majority has persisted and confirmed it.
20. Raft in a slide
An odd number of servers each keep a persisted log of events.
Everything is replicated to everybody.
They democratically elect a leader with absolute majority.
Only the leader may append to the replicated log.
An append only counts when a majority has persisted and confirmed it.
Very smart logic to ensure a unique leader and automatic recovery from failure.
21. Raft in a slide
An odd number of servers each keep a persisted log of events.
Everything is replicated to everybody.
They democratically elect a leader with absolute majority.
Only the leader may append to the replicated log.
An append only counts when a majority has persisted and confirmed it.
Very smart logic to ensure a unique leader and automatic recovery from failure.
It is all a lot of fun to get right, but it is proven to work.
22. Raft in a slide
An odd number of servers each keep a persisted log of events.
Everything is replicated to everybody.
They democratically elect a leader with absolute majority.
Only the leader may append to the replicated log.
An append only counts when a majority has persisted and confirmed it.
Very smart logic to ensure a unique leader and automatic recovery from failure.
It is all a lot of fun to get right, but it is proven to work.
One puts a key/value store on top, the log contains the changes.
25. Sorting
The Problem
Data stores need indexes. In practice, we need to sort things.
Most published algorithms are rubbish on modern hardware.
26. Sorting
The Problem
Data stores need indexes. In practice, we need to sort things.
Most published algorithms are rubbish on modern hardware.
The problem is no longer the
comparison computations but the data movement
.
27. Sorting
The Problem
Data stores need indexes. In practice, we need to sort things.
Most published algorithms are rubbish on modern hardware.
The problem is no longer the
comparison computations but the data movement
.
Since the time when I was a kid and have played with an Apple IIe,
compute power in one core has increased by ×20000
a single memory access by ×40
and now we have 32 cores in a CPU
this means computation has outpaced memory access by ×1280!
28. Idea for a parallel sorting algorithm: Merge Sort
Min−Heap:
sorted
merged
29. Idea for a parallel sorting algorithm: Merge Sort
Min−Heap:
sorted
merged
Nearly all comparisons hit the L2 cache!
30. Log structured merge trees (LSM-trees)
The Problem
People rightfully expect from a data store, that it
can hold more data than the available RAM,
works well with SSDs and spinning rust,
allows fast bulk inserts into large data sets, and
provides fast reads in a hot set that fits into RAM.
31. Log structured merge trees (LSM-trees)
The Problem
People rightfully expect from a data store, that it
can hold more data than the available RAM,
works well with SSDs and spinning rust,
allows fast bulk inserts into large data sets, and
provides fast reads in a hot set that fits into RAM.
Traditional B-tree based structures often fail to deliver with the last 2.
32. Log structured merge trees (LSM-trees)
(Source: http://www.benstopford.com/2015/02/14/log-structured-merge-trees/, Author: Ben Stopford, License: Creative Commons)
33. Log structured merge trees (LSM-trees)
LSM-trees — summary
writes first go into memtables,
all files are sorted and immutable,
compaction happens in the background,
merge sort can be used,
all writes use sequential I/O,
Bloom filters or Cuckoo filters for fast reads,
=⇒ good write throughput and reasonable read performance,
used in ArangoDB, BigTable, Cassandra, HBase, InfluxDB, LevelDB,
MongoDB, RocksDB, SQLite4 and WiredTiger, etc.
34. Hybrid Logical Clocks (HLC)
The Problem
Clocks in different nodes of distributed systems are not in sync.
35. Hybrid Logical Clocks (HLC)
The Problem
Clocks in different nodes of distributed systems are not in sync.
general relativity poses fundamental obstructions to synchronizity,
in practice, clock skew happens,
Google can use atomic clocks,
even with NTP (network time protocol) we have to live with ≈ 20ms.
36. Hybrid Logical Clocks (HLC)
The Problem
Clocks in different nodes of distributed systems are not in sync.
general relativity poses fundamental obstructions to synchronizity,
in practice, clock skew happens,
Google can use atomic clocks,
even with NTP (network time protocol) we have to live with ≈ 20ms.
Therefore, we cannot compare time stamps from different nodes!
37. Hybrid Logical Clocks (HLC)
The Problem
Clocks in different nodes of distributed systems are not in sync.
general relativity poses fundamental obstructions to synchronizity,
in practice, clock skew happens,
Google can use atomic clocks,
even with NTP (network time protocol) we have to live with ≈ 20ms.
Therefore, we cannot compare time stamps from different nodes!
Why would this help?
establish “happened after” relationship between events,
e.g. for conflict resolution, log sorting, detecting network delays,
time to live could be implemented easily.
38. Hybrid Logical Clocks (HLC)
The Idea
Every computer has a local clock, and we use NTP to synchronize.
39. Hybrid Logical Clocks (HLC)
The Idea
Every computer has a local clock, and we use NTP to synchronize.
If two events on different machines are linked by causality, the cause
should have a smaller time stamp than the effect.
40. Hybrid Logical Clocks (HLC)
The Idea
Every computer has a local clock, and we use NTP to synchronize.
If two events on different machines are linked by causality, the cause
should have a smaller time stamp than the effect.
causality ⇐⇒ a message is sent
Send a time stamp with every message. The HLC always returns a value
> max(local clock, largest time stamp ever seen).
41. Hybrid Logical Clocks (HLC)
The Idea
Every computer has a local clock, and we use NTP to synchronize.
If two events on different machines are linked by causality, the cause
should have a smaller time stamp than the effect.
causality ⇐⇒ a message is sent
Send a time stamp with every message. The HLC always returns a value
> max(local clock, largest time stamp ever seen).
Causality is preserved, time can “catch up” with logical time eventually.
http://muratbuffalo.blogspot.com.es/2014/07/hybrid-logical-clocks.html
42. Distributed ACID Transactions
Atomic either happens in its entirety or not at all
Consistent reading sees a consistent state, writing preser-
vers consistency
Isolated concurrent transactions do not see each other
Durable committed writes are preserved after shut-
down and crashes
43. Distributed ACID Transactions
Atomic either happens in its entirety or not at all
Consistent reading sees a consistent state, writing preser-
vers consistency
Isolated concurrent transactions do not see each other
Durable committed writes are preserved after shut-
down and crashes
(All relatively doable when transactions happen one after another!)
44. Distributed ACID Transactions
The Problem
In a distributed system:
How to make sure, that all nodes agree on whether
the transaction has happened? (Atomicity)
45. Distributed ACID Transactions
The Problem
In a distributed system:
How to make sure, that all nodes agree on whether
the transaction has happened? (Atomicity)
How to create a consistent snapshot across nodes? (Consistency)
46. Distributed ACID Transactions
The Problem
In a distributed system:
How to make sure, that all nodes agree on whether
the transaction has happened? (Atomicity)
How to create a consistent snapshot across nodes? (Consistency)
How to hide ongoing activities until commit? (Isolation)
47. Distributed ACID Transactions
The Problem
In a distributed system:
How to make sure, that all nodes agree on whether
the transaction has happened? (Atomicity)
How to create a consistent snapshot across nodes? (Consistency)
How to hide ongoing activities until commit? (Isolation)
How to handle lost nodes? (Durability)
48. Distributed ACID Transactions
The Problem
In a distributed system:
How to make sure, that all nodes agree on whether
the transaction has happened? (Atomicity)
How to create a consistent snapshot across nodes? (Consistency)
How to hide ongoing activities until commit? (Isolation)
How to handle lost nodes? (Durability)
We have to take replication, resilience and failover into account.
49. Distributed ACID Transactions
WITHOUT
Distributed databases without ACID transactions:
ArangoDB, BigTable, Couchbase, Datastax, Dynamo, Elastic, HBase,
MongoDB, RethinkDB, Riak, and lots more . . .
WITH
Distributed databases with ACID transactions:
(ArangoDB,) CockroachDB, FoundationDB, Spanner
50. Distributed ACID Transactions
WITHOUT
Distributed databases without ACID transactions:
ArangoDB, BigTable, Couchbase, Datastax, Dynamo, Elastic, HBase,
MongoDB, RethinkDB, Riak, and lots more . . .
WITH
Distributed databases with ACID transactions:
(ArangoDB,) CockroachDB, FoundationDB, Spanner
=⇒ Very few distributed engines promise ACID, because this is hard!
52. Distributed ACID Transactions
Basic Idea
Use Multi Version Concurrency Control (MVCC), i.e. multiple revisions of
a data item are kept.
Do writes and replication decentrally and distributed, without them
becoming visible from other transactions.
53. Distributed ACID Transactions
Basic Idea
Use Multi Version Concurrency Control (MVCC), i.e. multiple revisions of
a data item are kept.
Do writes and replication decentrally and distributed, without them
becoming visible from other transactions.
Then have some place, where there is a switch, which decides when the
transaction becomes visible.
54. Distributed ACID Transactions
Basic Idea
Use Multi Version Concurrency Control (MVCC), i.e. multiple revisions of
a data item are kept.
Do writes and replication decentrally and distributed, without them
becoming visible from other transactions.
Then have some place, where there is a switch, which decides when the
transaction becomes visible. These “switches” need to
be persisted somewhere (durability),
scale out (no bottleneck for commit/abort),
be replicated (no single point of failure),
be resilient in case of fail-over (fault-tolerance).
55. Distributed ACID Transactions
Basic Idea
Use Multi Version Concurrency Control (MVCC), i.e. multiple revisions of
a data item are kept.
Do writes and replication decentrally and distributed, without them
becoming visible from other transactions.
Then have some place, where there is a switch, which decides when the
transaction becomes visible. These “switches” need to
be persisted somewhere (durability),
scale out (no bottleneck for commit/abort),
be replicated (no single point of failure),
be resilient in case of fail-over (fault-tolerance).
Transaction visibility needs to be implemented (MVCC), time stamps play
a crucial role.