There are many reasons to use an eventually-consistent database — like Riak, Voldemort, or Cassandra — including increased availability, lower latency, and fault-tolerance. However, doing so requires a mental shift in how to structure client applications, and certain types of traditional data-structures, like sets, registers, and counters can’t be resolved simply in the face of race-conditions. It is difficult to achieve “logical monotonicity” except for the most trivial data-types.
That is, until the advent of Convergent Replicated Data Types (CRDTs). CRDTs are data-structures that tolerate eventual consistency. They replace traditional data-structure implementations and all have the property that, given any number of conflicting versions of the same datum, there is a single state on which they converge (monotonicity). This talk will discuss some of the most useful CRDTs and how to apply them to solve real-world data problems.
Redmineに全文検索エンジンGroongaサポートを追加する「Full Text Search」プラグインを紹介します。このプラグインを使えば数十万チケットがあるRedmineでも高速に目的の情報を見つけられます。高速になるという現状の話だけでなく、Redmine内のデータを有効活用するという今後の話もします。
OSMC 2022 | VictoriaMetrics: scaling to 100 million metrics per second by Ali...NETWAYS
The growth of observability trends and Kubernetes adoption generates more demanding requirements for monitoring systems. Volumes of time series data increase exponentially, and old solutions just can’t keep up with the pace. The talk will cover how and why we created a new open source time series database from scratch. Which architectural decisions, which trade-offs we had to take in order to match the new expectations and handle 100 million metrics per second with VictoriaMetrics. The talk will be interesting for software engineers and DevOps familiar with observability and modern monitoring systems, or for those who’re interested in building scalable high performant databases for time series.
Prometheus has become the defacto monitoring system for cloud native applications, with systems like Kubernetes and Etcd natively exposing Prometheus metrics. In this talk Tom will explore all the moving part for a working Prometheus-on-Kubernetes monitoring system, including kube-state-metrics, node-exporter, cAdvisor and Grafana. You will learn about the various methods for getting to a working setup: the manual approach, using CoreOSs Prometheus Operator, or using Prometheus Ksonnet Mixin. Tom will also share some little tips and tricks for getting the most out of your Prometheus monitoring, including the common pitfalls and what you should be alerting on.
Monitoring What Matters: The Prometheus Approach to Whitebox Monitoring (Berl...Brian Brazil
Often what you monitor and get alerted on is defined by your tools, rather than what makes the most sense to you and your organisation. Alerts on metrics such as CPU usage which are noisy and rarely spot real problems, while outages go undetected. Monitoring systems can also be challenging to maintain, and overall provide a poor return on investment.
In the past few years several new monitoring systems have appeared with more powerful semantics and which are easier to run, which offer a way to vastly improve how your organisation operates Prometheus is one such system. This talk will look at the monitoring ideal and how whitebox monitoring with a time series database, multi-dimensional labels and a powerful querying/alerting language can free you from midnight pages.
Redmineに全文検索エンジンGroongaサポートを追加する「Full Text Search」プラグインを紹介します。このプラグインを使えば数十万チケットがあるRedmineでも高速に目的の情報を見つけられます。高速になるという現状の話だけでなく、Redmine内のデータを有効活用するという今後の話もします。
OSMC 2022 | VictoriaMetrics: scaling to 100 million metrics per second by Ali...NETWAYS
The growth of observability trends and Kubernetes adoption generates more demanding requirements for monitoring systems. Volumes of time series data increase exponentially, and old solutions just can’t keep up with the pace. The talk will cover how and why we created a new open source time series database from scratch. Which architectural decisions, which trade-offs we had to take in order to match the new expectations and handle 100 million metrics per second with VictoriaMetrics. The talk will be interesting for software engineers and DevOps familiar with observability and modern monitoring systems, or for those who’re interested in building scalable high performant databases for time series.
Prometheus has become the defacto monitoring system for cloud native applications, with systems like Kubernetes and Etcd natively exposing Prometheus metrics. In this talk Tom will explore all the moving part for a working Prometheus-on-Kubernetes monitoring system, including kube-state-metrics, node-exporter, cAdvisor and Grafana. You will learn about the various methods for getting to a working setup: the manual approach, using CoreOSs Prometheus Operator, or using Prometheus Ksonnet Mixin. Tom will also share some little tips and tricks for getting the most out of your Prometheus monitoring, including the common pitfalls and what you should be alerting on.
Monitoring What Matters: The Prometheus Approach to Whitebox Monitoring (Berl...Brian Brazil
Often what you monitor and get alerted on is defined by your tools, rather than what makes the most sense to you and your organisation. Alerts on metrics such as CPU usage which are noisy and rarely spot real problems, while outages go undetected. Monitoring systems can also be challenging to maintain, and overall provide a poor return on investment.
In the past few years several new monitoring systems have appeared with more powerful semantics and which are easier to run, which offer a way to vastly improve how your organisation operates Prometheus is one such system. This talk will look at the monitoring ideal and how whitebox monitoring with a time series database, multi-dimensional labels and a powerful querying/alerting language can free you from midnight pages.
Getting Ready to Use Redis with Apache Spark with Dvir VolkSpark Summit
Getting Ready to use Redis with Apache Spark is a technical tutorial designed to address integrating Redis with an Apache Spark deployment to increase the performance of serving complex decision models. To set the context for the session, we start with a quick introduction to Redis and the capabilities Redis provides. We cover the basic data types provided by Redis and cover the module system. Using an ad serving use-case, we look at how Redis can improve the performance and reduce the cost of using complex ML-models in production. Attendees will be guided through the key steps of setting up and integrating Redis with Spark, including how to train a model using Spark then load and serve it using Redis, as well as how to work with the Spark Redis module. The capabilities of the Redis Machine Learning Module (redis-ml) will be discussed focusing primarily on decision trees and regression (linear and logistic) with code examples to demonstrate how to use these feature. At the end of the session, developers should feel confident building a prototype/proof-of-concept application using Redis and Spark. Attendees will understand how Redis complements Spark and how to use Redis to serve complex, ML-models with high performance.
2018 Jul 25th LINE Developer Meetup #41 in Fukuoka
Session Slide in English / セッションスライドです。
Graal in GraalVM - A New JIT Compiler
オラクル社からGraalVMというものが発表され、話題を呼んでいます。GraalVMはHotSpot VM上に新しいJITコンパイラGraalと言語実装用フレームワーク/ASTインタプリタであるTruffle、さらにネイティブイメージ作成機能とその実行に使われるSubstrateVMを併せ持ったものです。すでにTruffleを使用したJavaScriptやRuby、R、Pythonの実装も提供されており、これらの言語とJavaはコードから相互に呼び出しができます。このセッションではGraalVMを概観したあと、JITコンパイラGraalにとくに注力して解説します。GraalとTruffleはOracle Labsとヨハネス・ケプラー大学で共同研究されており、多くの論文が発表されています。HotSpotのJITコンパイラとパフォーマンスや構造などを比較しつつ、GraalのJITコンパイルのテクニックについてもいくつか触れます。とにかく、私がGraalをとても好きなのです。デモも実施しつつ、Graalのすごさを伝えられればと考えています。
There are many reasons to use an eventually-consistent database -- like Riak, Voldemort, or Cassandra -- including increased availability, lower latency, and fault-tolerance. However, doing so requires a mental shift in how to structure client applications, and certain types of traditional data-structures, like sets, registers, and counters can't be resolved simply in the face of race-conditions. It is difficult to achieve "logical monotonicity" except for the most trivial data-types.
That is, until the advent of Conflict-Free Replicated Data Types (CRDTs). CRDTs are data-structures that tolerate eventual consistency. They replace traditional data-structure implementations and all have the property that, given any number of conflicting versions of the same datum, there is a single state on which they converge (monotonicity). This talk will discuss some of the most useful CRDTs and how to apply them to solve real-world data problems.
Getting Ready to Use Redis with Apache Spark with Dvir VolkSpark Summit
Getting Ready to use Redis with Apache Spark is a technical tutorial designed to address integrating Redis with an Apache Spark deployment to increase the performance of serving complex decision models. To set the context for the session, we start with a quick introduction to Redis and the capabilities Redis provides. We cover the basic data types provided by Redis and cover the module system. Using an ad serving use-case, we look at how Redis can improve the performance and reduce the cost of using complex ML-models in production. Attendees will be guided through the key steps of setting up and integrating Redis with Spark, including how to train a model using Spark then load and serve it using Redis, as well as how to work with the Spark Redis module. The capabilities of the Redis Machine Learning Module (redis-ml) will be discussed focusing primarily on decision trees and regression (linear and logistic) with code examples to demonstrate how to use these feature. At the end of the session, developers should feel confident building a prototype/proof-of-concept application using Redis and Spark. Attendees will understand how Redis complements Spark and how to use Redis to serve complex, ML-models with high performance.
2018 Jul 25th LINE Developer Meetup #41 in Fukuoka
Session Slide in English / セッションスライドです。
Graal in GraalVM - A New JIT Compiler
オラクル社からGraalVMというものが発表され、話題を呼んでいます。GraalVMはHotSpot VM上に新しいJITコンパイラGraalと言語実装用フレームワーク/ASTインタプリタであるTruffle、さらにネイティブイメージ作成機能とその実行に使われるSubstrateVMを併せ持ったものです。すでにTruffleを使用したJavaScriptやRuby、R、Pythonの実装も提供されており、これらの言語とJavaはコードから相互に呼び出しができます。このセッションではGraalVMを概観したあと、JITコンパイラGraalにとくに注力して解説します。GraalとTruffleはOracle Labsとヨハネス・ケプラー大学で共同研究されており、多くの論文が発表されています。HotSpotのJITコンパイラとパフォーマンスや構造などを比較しつつ、GraalのJITコンパイルのテクニックについてもいくつか触れます。とにかく、私がGraalをとても好きなのです。デモも実施しつつ、Graalのすごさを伝えられればと考えています。
There are many reasons to use an eventually-consistent database -- like Riak, Voldemort, or Cassandra -- including increased availability, lower latency, and fault-tolerance. However, doing so requires a mental shift in how to structure client applications, and certain types of traditional data-structures, like sets, registers, and counters can't be resolved simply in the face of race-conditions. It is difficult to achieve "logical monotonicity" except for the most trivial data-types.
That is, until the advent of Conflict-Free Replicated Data Types (CRDTs). CRDTs are data-structures that tolerate eventual consistency. They replace traditional data-structure implementations and all have the property that, given any number of conflicting versions of the same datum, there is a single state on which they converge (monotonicity). This talk will discuss some of the most useful CRDTs and how to apply them to solve real-world data problems.
HBaseCon 2012 | Storing and Manipulating Graphs in HBaseCloudera, Inc.
Google’s original use case for BigTable was the storage and processing of web graph information, represented as sparse matrices. However, many organizations tend to treat HBase as merely a “web scale” RDBMS. This session will cover several use cases for storing graph data in HBase, including social networks and web link graphs, MapReduce processes like cached traversal, PageRank, and clustering and lastly will look at some lower-level modeling details like row key and column qualifier design, using FullContact’s graph processing systems as a real-world use case.
Consistency without Consensus: CRDTs in Production at SoundCloudC4Media
Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/1DKnwXr.
Peter Bourgon provides a practical introduction to Conflict-free Replicated Data Types (CRDTs) and describes a production CRDT system built at SoundCloud to serve several product features. Filmed at qconsf.com.
Peter Bourgon is a distributed systems engineer who has seen things. He works at SoundCloud, building and improving the infrastructure that powers the world's largest audio platform.
Guaranteeing Consensus in Distriubuted Systems with CRDTsSun-Li Beatteay
Consensus in distributed systems has been a debated topic every since programmers discovered they could run the same program on multiple machines. Researchers have been studying consensus for decades, resulting in numerous algorithms and white papers. Unfortunately, many of these algorithms are flawed and unreliable.
However, in 2011, a team of researchers published a paper on a novel approach to distributed consensus using Conflict-free Replicated Data Types (https://hal.inria.fr/inria-00609399v1...). This paper created quite a buzz as it showed that CRDTs were mathematically proven to guarantee consensus through "Strong Eventual Consistency." They also claimed to have solved the CAP conundrum.
This presentation dives into this seminal paper in order to answer the hard questions. What are CRDTs? How do they work? And most importantly, does it actually solve CAP? By the end of this talk, everyone in the audience will have a foundational understanding of CRDTs and how they can be applied to their own work.
Best of all, I will be explaining all of this is as simple language as possible. No advanced math degree required! Sound too good to be true? You'll just have to come see for yourself!
Embrace NoSQL and Eventual Consistency with RippleSean Cribbs
So, there's this "NoSQL" thing you may have heard of, and this related thing called "eventual consistency". Supposedly, they help you scale, but no one has ever explained why! Well, wonder no more! This talk will demystify NoSQL, eventual consistency, how they might help you scale, and -- most importantly -- why you should care.
We'll look closely at how Riak, a linearly-scalable, distributed and fault-tolerant NoSQL datastore, implements eventual consistency, and how you can harness it from Ruby via the slick Ripple client/ORM. When the talk is finished, you'll have the tools both to understand eventual consistency and to handle it like a pro inside your next Ruby application.
Here we describe how to "think" mapreduce not just "code" mapreduce. We solve some interesting problems using mapreduce (e.g. how to compute similarity between all pair of documents on the web, how to do k-means clustering using map-reduce, and how to find cliques in a graph using map-reduce). These solutions are simple, elegant, and open up new ways for people to actually use mapreduce more than just simple number crunching.
Incremental View Maintenance for openCypher QueriesGábor Szárnyas
Presented at the Fourth openCypher Implementers Meeting
Numerous graph use cases require continuous evaluation of queries over a constantly changing data set, e.g. fraud detection in financial systems, recommendations, and checking integrity constraints. For relational systems, incremental view maintenance has been researched for three decades, resulting in a wide body of literature. The property graph data model and the openCypher language, however, are recent developments, and therefore lack established techniques to perform efficient view maintenance. In this talk, we give an overview of the view maintenance problem for property graphs, discuss why it is particularly difficult and present an approach that tackles a meaningful subset of the language.
This case study gives an inside look at optimization of the MongoDB Perl driver, including custom benchmarking tools, step-by-step changes and results that will surprise and amaze. If you ever needed to optimize some Perl and wondered how people go about it, this talk is for you.
Flowr offers a scatter-gather approach, and works by submitting a web of jobs to the computing cluster using dependencies. It requires two simple data.frames as inputs and supports platforms like LSF, SGE, Torque and SLURM.
Concurrency in Ruby is all the rage these days, and people can't seem to agree
whether Threads, Fibers, event loops, or actors are the best solution. But did you ever consider that your *sequential* Ruby program might be concurrent, with nary a Thread, Fiber, or callback in sight? Well, it happened to me.
This is the story of how accidental concurrency (also known as re-entrancy) broke my brain multiple times over the course of two years, spawned flamewars on Twitter, long blog posts, and the various solutions I took to solve the problem. Along the way we'll illuminate some subtleties of concurrent programming in Ruby, differences between several Ruby implementations, and how we can all write code that is friendlier when accidental concurrency strikes.
A discussion of strategies for designing application schemas that use the Riak distributed key-value store.
Video available here: http://vimeo.com/17604126
Ruby on Rails is a powerful web framework that focuses on developer productivity. Riak is a friendly key-value store that is simple, flexible and scalable. Put them together and you have lots of exciting possibilities!
Moving to Riak involves a number of changes from the status quo of RDBMS systems, one of which is taking greater control over your schema design.
* How do you structure data when you don't have tables and foreign keys?
* When should you denormalize, add links, or create map-reduce queries?
* Where will Riak be a natural fit and where will it be challenging?
An introduction to Basho's Riak distributed data store, and the Ripple client in Ruby. Code samples from the demos are here: http://gist.github.com/365791
Many developers will be familiar with lex, flex, yacc, bison, ANTLR, and other related tools to generate parsers for use inside their own code. For recognizing computer-friendly languages, however, context-free grammars and their parser-generators leave a few things to be desired. This is about how the seemingly simple prospect of parsing some text turned into a new parser toolkit for Erlang, and why functional programming makes parsing fun and awesome
Software projects are rarely on-spec, on-time and on-budget, and the primary cause is miscommunication. As Martin Fowler says, there is a "yawning crevasse of doom" between stakeholders and developers, full of misunderstanding. How do you make sure that you're building something that adds value? How do you know you're building the thing that was asked for? How does your bottom line affect user experience?
Into the fray leaps Cucumber, a business-readable DSL combined with an awesome Ruby library that lets domain experts express business requirements as executable user stories. We'll cover outside-in, story-driven development with Cucumber, how to write effective stories, and how to make Cucumber work for your project.
(as given to CharlotteRuby on Jan 6, 2010)
Most developers will be familiar with lex, flex, yacc, bison, ANTLR, and other tools to generate parsers for use inside their own code. Erlang, the concurrent functional programming language, has its own pair, leex and yecc, for accomplishing most complicated text-processing tasks. This talk is about how the seemingly simple prospect of parsing text turned into a new parser toolkit for Erlang, and why functional programming makes parsing fun and awesome.
Reviews grammar and parsers and discusses my personal path toward writing my own packrat parser-generator for Erlang called neotoma.
Given to "Evil Robot Conference" on September 12, 2009.
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
UiPath Test Automation using UiPath Test Suite series, part 4DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap.
The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies.
Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques
What will you get from this session?
1. Insights into SAP testing best practices
2. Heatmap utilization for testing
3. Optimization of testing processes
4. Demo
Topics covered:
Execution from the test manager
Orchestrator execution result
Defect reporting
SAP heatmap example with demo
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Sudheer Mechineni, Head of Application Frameworks, Standard Chartered Bank
Discover how Standard Chartered Bank harnessed the power of Neo4j to transform complex data access challenges into a dynamic, scalable graph database solution. This keynote will cover their journey from initial adoption to deploying a fully automated, enterprise-grade causal cluster, highlighting key strategies for modelling organisational changes and ensuring robust disaster recovery. Learn how these innovations have not only enhanced Standard Chartered Bank’s data infrastructure but also positioned them as pioneers in the banking sector’s adoption of graph technology.
SAP Sapphire 2024 - ASUG301 building better apps with SAP Fiori.pdfPeter Spielvogel
Building better applications for business users with SAP Fiori.
• What is SAP Fiori and why it matters to you
• How a better user experience drives measurable business benefits
• How to get started with SAP Fiori today
• How SAP Fiori elements accelerates application development
• How SAP Build Code includes SAP Fiori tools and other generative artificial intelligence capabilities
• How SAP Fiori paves the way for using AI in SAP apps
Unlocking Productivity: Leveraging the Potential of Copilot in Microsoft 365, a presentation by Christoforos Vlachos, Senior Solutions Manager – Modern Workplace, Uni Systems
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Albert Hoitingh
In this session I delve into the encryption technology used in Microsoft 365 and Microsoft Purview. Including the concepts of Customer Key and Double Key Encryption.
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
Generative AI Deep Dive: Advancing from Proof of Concept to ProductionAggregage
Join Maher Hanafi, VP of Engineering at Betterworks, in this new session where he'll share a practical framework to transform Gen AI prototypes into impactful products! He'll delve into the complexities of data collection and management, model selection and optimization, and ensuring security, scalability, and responsible use.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024Neo4j
Neha Bajwa, Vice President of Product Marketing, Neo4j
Join us as we explore breakthrough innovations enabled by interconnected data and AI. Discover firsthand how organizations use relationships in data to uncover contextual insights and solve our most pressing challenges – from optimizing supply chains, detecting fraud, and improving customer experiences to accelerating drug discoveries.
In his public lecture, Christian Timmerer provides insights into the fascinating history of video streaming, starting from its humble beginnings before YouTube to the groundbreaking technologies that now dominate platforms like Netflix and ORF ON. Timmerer also presents provocative contributions of his own that have significantly influenced the industry. He concludes by looking at future challenges and invites the audience to join in a discussion.
Climate Impact of Software Testing at Nordic Testing DaysKari Kakkonen
My slides at Nordic Testing Days 6.6.2024
Climate impact / sustainability of software testing discussed on the talk. ICT and testing must carry their part of global responsibility to help with the climat warming. We can minimize the carbon footprint but we can also have a carbon handprint, a positive impact on the climate. Quality characteristics can be added with sustainability, and then measured continuously. Test environments can be used less, and in smaller scale and on demand. Test techniques can be used in optimizing or minimizing number of tests. Test automation can be used to speed up testing.
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
13. Safety / Liveness
Proving the Correctness of Multiprocess Programs - Leslie
Lamport (March 1977)
•Safety: “nothing bad happens”
(partial correctness)
14. Safety / Liveness
Proving the Correctness of Multiprocess Programs - Leslie
Lamport (March 1977)
•Safety: “nothing bad happens”
(partial correctness)
•Liveness: “something good
eventually happens” (termination)
15. Safety / Liveness
Proving the Correctness of Multiprocess Programs - Leslie
Lamport (March 1977)
•Safety: “nothing bad happens”
(partial correctness)
•Liveness: “something good
eventually happens” (termination)
“Safety and liveness: Eventual consistency is not safe” - Peter
Bailis
http://www.bailis.org/blog/safety-and-liveness-eventual-consistency-is-
not-safe/
22. Semantic
Resolution
• Your app knows the domain - use
business rules to resolve
• Amazon Dynamo’s shopping cart
23. Semantic
Resolution
• Your app knows the domain - use
business rules to resolve
• Amazon Dynamo’s shopping cart
“Ad hoc approaches have proven brittle and
error-prone”
There’s no ACID! But don’t worry, there’s no need to be upset, despite what you may have heard.\n
I think the fear people have about giving up ACID is really just a tendency to see things in black and white, because subtlety is much harder to understand and accept. Everyday in the wider technical community and the Internet we are presented with binary choices which are often not really in conflict, but either orthogonal albeit related concepts, or simply different ends of a spectrum. We too often perceive a Hegelian dialectic when one doesn’t exist (without the synthesis part!). \n\nAn important pair we need to understand, but is not frequently discussed outside of academia is safety and liveness.\n
I think the fear people have about giving up ACID is really just a tendency to see things in black and white, because subtlety is much harder to understand and accept. Everyday in the wider technical community and the Internet we are presented with binary choices which are often not really in conflict, but either orthogonal albeit related concepts, or simply different ends of a spectrum. We too often perceive a Hegelian dialectic when one doesn’t exist (without the synthesis part!). \n\nAn important pair we need to understand, but is not frequently discussed outside of academia is safety and liveness.\n
I think the fear people have about giving up ACID is really just a tendency to see things in black and white, because subtlety is much harder to understand and accept. Everyday in the wider technical community and the Internet we are presented with binary choices which are often not really in conflict, but either orthogonal albeit related concepts, or simply different ends of a spectrum. We too often perceive a Hegelian dialectic when one doesn’t exist (without the synthesis part!). \n\nAn important pair we need to understand, but is not frequently discussed outside of academia is safety and liveness.\n
I think the fear people have about giving up ACID is really just a tendency to see things in black and white, because subtlety is much harder to understand and accept. Everyday in the wider technical community and the Internet we are presented with binary choices which are often not really in conflict, but either orthogonal albeit related concepts, or simply different ends of a spectrum. We too often perceive a Hegelian dialectic when one doesn’t exist (without the synthesis part!). \n\nAn important pair we need to understand, but is not frequently discussed outside of academia is safety and liveness.\n
I think the fear people have about giving up ACID is really just a tendency to see things in black and white, because subtlety is much harder to understand and accept. Everyday in the wider technical community and the Internet we are presented with binary choices which are often not really in conflict, but either orthogonal albeit related concepts, or simply different ends of a spectrum. We too often perceive a Hegelian dialectic when one doesn’t exist (without the synthesis part!). \n\nAn important pair we need to understand, but is not frequently discussed outside of academia is safety and liveness.\n
I think the fear people have about giving up ACID is really just a tendency to see things in black and white, because subtlety is much harder to understand and accept. Everyday in the wider technical community and the Internet we are presented with binary choices which are often not really in conflict, but either orthogonal albeit related concepts, or simply different ends of a spectrum. We too often perceive a Hegelian dialectic when one doesn’t exist (without the synthesis part!). \n\nAn important pair we need to understand, but is not frequently discussed outside of academia is safety and liveness.\n
Safety and Liveness were terms which were defined for concurrent programs in this 1977 paper by Leslie Lamport. Colloquially, safety means that in the course of running your program, “nothing bad will happen” and liveness means that “something good will eventually happen”. Both are desirable properties, but sometimes enforcing one property may cause you to give up the other. I thought Peter Bailis stated this eloquently in his recent blog post that Eventual Consistency is not safe by itself - but a trivially satisfiable liveness property. That is, it helps keep your system available, but doesn’t make any guarantees about whether correct answers will be given at all. His larger point was that practical systems like Riak/Voldemort/Cassandra do make safety guarantees but tend not to state them. It’s not all “garbage”.\n
Safety and Liveness were terms which were defined for concurrent programs in this 1977 paper by Leslie Lamport. Colloquially, safety means that in the course of running your program, “nothing bad will happen” and liveness means that “something good will eventually happen”. Both are desirable properties, but sometimes enforcing one property may cause you to give up the other. I thought Peter Bailis stated this eloquently in his recent blog post that Eventual Consistency is not safe by itself - but a trivially satisfiable liveness property. That is, it helps keep your system available, but doesn’t make any guarantees about whether correct answers will be given at all. His larger point was that practical systems like Riak/Voldemort/Cassandra do make safety guarantees but tend not to state them. It’s not all “garbage”.\n
Safety and Liveness were terms which were defined for concurrent programs in this 1977 paper by Leslie Lamport. Colloquially, safety means that in the course of running your program, “nothing bad will happen” and liveness means that “something good will eventually happen”. Both are desirable properties, but sometimes enforcing one property may cause you to give up the other. I thought Peter Bailis stated this eloquently in his recent blog post that Eventual Consistency is not safe by itself - but a trivially satisfiable liveness property. That is, it helps keep your system available, but doesn’t make any guarantees about whether correct answers will be given at all. His larger point was that practical systems like Riak/Voldemort/Cassandra do make safety guarantees but tend not to state them. It’s not all “garbage”.\n
In an eventually consistent system, you tend to have multiple copies of the same datum, which means that it’s replicated. They also tend to allow loose coordination and things like sloppy quorums, since you don’t require expensive multi-phase commit protocols. This also makes them resilient to network partitions, which DO EXIST. Eventually consistent systems must also include means for state to move forward when staleness is detected. In Dynamo-like systems, this is usually done with read-repair, that is, writing the newer value to stale replicas when reading.\n
While not as simple to understand as an ACID system, eventual consistency has many practical benefits. When encountering failures, especially network-related ones, the system can more often remain available to reads and writes despite the failures. In the same vein, relying on dynamic participation in operations lends itself to systems with low, consistent latency because only promptly-responding replicas need to be considered.\n
Of course the tradeoff of those benefits, thanks to the CAP theorem, is that you sacrifice strict consistency. There is no total ordering of events in the system, you have no transactions, you have weak guarantees of delivery at best. This means it’s incredibly difficult to decide who wins when there are concurrent writes in the system. The solutions to the problem are both non-ideal, but they are generally: first, to throw one version out by applying an arbitrary ordering, usually a timestamp of sorts; second, to keep both values around and let the user decide. These are the approaches of Cassandra, and Riak/Voldemort respectively.\n
Of course the tradeoff of those benefits, thanks to the CAP theorem, is that you sacrifice strict consistency. There is no total ordering of events in the system, you have no transactions, you have weak guarantees of delivery at best. This means it’s incredibly difficult to decide who wins when there are concurrent writes in the system. The solutions to the problem are both non-ideal, but they are generally: first, to throw one version out by applying an arbitrary ordering, usually a timestamp of sorts; second, to keep both values around and let the user decide. These are the approaches of Cassandra, and Riak/Voldemort respectively.\n
So maybe you chose Riak or Voldemort, you get write conflicts (Riak calls them siblings). Now that you’ve got both values, how do you decide what the real state should be?\n
One strategy, which I call “semantic resolution”, is to say that your application encodes the domain of the problem and so it can use business rules to resolve the conflict. This is the strategy implemented by the “shopping cart” described in the Amazon Dynamo paper. It merges toward the maximum quantity of each item in the cart; however, it exhibits some problems -- namely that sometimes items that were removed from the cart can reappear! From Amazon’s point of view this is okay because it might encourage the customer to buy more, but it is a bewildering user-experience!\n\nFortunately, there is some interesting recent research about a more rigorous approach to eventual consistency.\n\n\n
...and that is Conflict-Free Replicated Data Types. This basically means that instead of strictly opaque values, the datastore provides useful abstract data structures. Since we’re in an eventually consistent system, the data structure is replicated to multiple locations, all of which act independently. But by far the most compelling part is that these data structures have the ability to resolve automatically toward a single value, given any number of conflicting values at individual replicas. CRDTs provide a strong safety property for eventually consistent systems that doesn’t sacrifice liveness in the process.\n
...and that is Conflict-Free Replicated Data Types. This basically means that instead of strictly opaque values, the datastore provides useful abstract data structures. Since we’re in an eventually consistent system, the data structure is replicated to multiple locations, all of which act independently. But by far the most compelling part is that these data structures have the ability to resolve automatically toward a single value, given any number of conflicting values at individual replicas. CRDTs provide a strong safety property for eventually consistent systems that doesn’t sacrifice liveness in the process.\n
...and that is Conflict-Free Replicated Data Types. This basically means that instead of strictly opaque values, the datastore provides useful abstract data structures. Since we’re in an eventually consistent system, the data structure is replicated to multiple locations, all of which act independently. But by far the most compelling part is that these data structures have the ability to resolve automatically toward a single value, given any number of conflicting values at individual replicas. CRDTs provide a strong safety property for eventually consistent systems that doesn’t sacrifice liveness in the process.\n
The theory behind what I’m going to talk about is the idea of bounded join semi-lattices, or “lattices” for short, and is rooted in the theory of monotonic logic. The definition I’m giving here comes from a recent paper by Neil Conway and others at UC-Berkeley.\n
A lattice is a triple of a set, a function, and a value. S is a set (possibly infinite) representing the possible values of the lattice. The upside-down T is the “least element” of the set. The “square U” is a binary operator over S that produces a least-upper bound of its operands that is also a member of S, also called the “join” or “merge” operator. The merge operator is commutative, associative, and idempotent. Finally, a lattice has the property such that for any two members of the set S, the merge operator creates a partial ordering over the set. This also means that merging any element with the least element is an identity operation.\n
A lattice is a triple of a set, a function, and a value. S is a set (possibly infinite) representing the possible values of the lattice. The upside-down T is the “least element” of the set. The “square U” is a binary operator over S that produces a least-upper bound of its operands that is also a member of S, also called the “join” or “merge” operator. The merge operator is commutative, associative, and idempotent. Finally, a lattice has the property such that for any two members of the set S, the merge operator creates a partial ordering over the set. This also means that merging any element with the least element is an identity operation.\n
A lattice is a triple of a set, a function, and a value. S is a set (possibly infinite) representing the possible values of the lattice. The upside-down T is the “least element” of the set. The “square U” is a binary operator over S that produces a least-upper bound of its operands that is also a member of S, also called the “join” or “merge” operator. The merge operator is commutative, associative, and idempotent. Finally, a lattice has the property such that for any two members of the set S, the merge operator creates a partial ordering over the set. This also means that merging any element with the least element is an identity operation.\n
A lattice is a triple of a set, a function, and a value. S is a set (possibly infinite) representing the possible values of the lattice. The upside-down T is the “least element” of the set. The “square U” is a binary operator over S that produces a least-upper bound of its operands that is also a member of S, also called the “join” or “merge” operator. The merge operator is commutative, associative, and idempotent. Finally, a lattice has the property such that for any two members of the set S, the merge operator creates a partial ordering over the set. This also means that merging any element with the least element is an identity operation.\n
A lattice is a triple of a set, a function, and a value. S is a set (possibly infinite) representing the possible values of the lattice. The upside-down T is the “least element” of the set. The “square U” is a binary operator over S that produces a least-upper bound of its operands that is also a member of S, also called the “join” or “merge” operator. The merge operator is commutative, associative, and idempotent. Finally, a lattice has the property such that for any two members of the set S, the merge operator creates a partial ordering over the set. This also means that merging any element with the least element is an identity operation.\n
A lattice is a triple of a set, a function, and a value. S is a set (possibly infinite) representing the possible values of the lattice. The upside-down T is the “least element” of the set. The “square U” is a binary operator over S that produces a least-upper bound of its operands that is also a member of S, also called the “join” or “merge” operator. The merge operator is commutative, associative, and idempotent. Finally, a lattice has the property such that for any two members of the set S, the merge operator creates a partial ordering over the set. This also means that merging any element with the least element is an identity operation.\n
Just for the sake of illustration, let’s look at one of the simpler lattices defined in Conway’s paper, the “lmax” lattice. The set of values in the lattice are the Real numbers. The merge function is defined as taking the maximum of the two values. The minimum value is negative infinity. I hope you can see that this definition is a lattice: nothing is less than negative infinity, and the merging of any two values trends toward positive infinity, without exceeding the seen values.\n
Let’s take another example for those who might be visual learners, the lset lattice. The set of values for the lattice are all simple sets, with the empty set being the minimum value. The merge function is set-union, which you should be able to see in this diagram, allow any ordering of operation delivery to eventually converge on the same value. This diagram doesn’t even show all of the possible orderings, in fact.\n\nNow why is this stuff important? Remember how we had conflicts and we needed a sane way to resolve those conflicts? Lattices are a generic type that give us determinism in how we merge our conflicts. In the case of the “lmax” lattice, if one value has 10 and another has 15, you pick 15 because it’s the larger one. This foundation gives us what we need to understand a larger study of the topic of conflict-resolution in eventual consistency.\n
The primary work on this research has been done by two researchers at INRIA and their colleagues in Portugal. Marc Shapiro also gave a great talk on the subject at Microsoft Research called &#x201C;Strong Eventual Consistency&#x201D; which you can easily find online.\n\nThe paper above is where I&#x2019;ve gotten most of the content and diagrams, but I&#x2019;ve tried to simplify the content so that we can get through it in the scope of this talk. If you want the real thing, search for <title>, it&#x2019;s free to download.\n
There are two flavors of CRDTs as you might have noticed. They both provide the same conflict-free property, but differ in their implementation strategy.\n\nConvergent types are based on a local modification of state, followed by forwarding the resulting state downstream, where a merge operation is performed at other replicas. The state itself encodes all information needed to converge. They are great for systems with weak message delivery guarantees - for example, a Dynamo-style system. Convergent types can also be resolved in clients, which is helpful for systems that do not provide rich datatypes.\n\nCommutative types, on the other hand, replicate commutative operations rather than state, and tend to rely on systems with reliable broadcast (that assures operations reach all replicas). Operations are generally not required to have a total ordering -- a local causal ordering is sufficient.\n
This diagram from the paper shows the basic format of a convergent, state based CRDT. Note how the mutation is applied locally, then forwarded downstream as a merge operation. As long as all replicas eventually receive states that include all mutations, they will converge on the same value. (The merge function is basically the merge function in a lattice.)\n
Again, in Commutative types forward operations to other replicas, not the state. Obviously, if an operation is not delivered, or applied out-of-order locally, the states don&#x2019;t converge. However, again, unlike the convergent type, a reliable broadcast channel is required. As long as functions f() and g() commute, state will converge.\n
A register is the simplest type of data structure - a memory cell storing an opaque value. It only supports two operations - &#x201C;assign&#x201D; and &#x201C;value&#x201D; (get and set). Concurrent updates will not commute (who should win?). We&#x2019;ve seen this problem before.\n
The two approaches to concurrent resolution are the same ones taken by Cassandra and Riak, respectively. That is, Last-Write-Wins (called an LWW-Register) and Multi-Valued (called MV-Register)-- keeping all divergent values. For resolution, LWW tend to use timestamps with a reasonable guarantee of ordering (which is difficult in practice, but in some systems sufficient). MV on the other hand, requires the more expensive version vector to resolve conflicts and produces the union of all divergent values (but it doesn&#x2019;t behave like a set!)\n
Counters are simply integers that are replicated and support the increment and decrement operations. Counters are useful for things like tracking the number of logged-in users, or click-throughs on an advertisement.\n\nThe simplest type of counter is a Commutative or operation-based type, since add and subtract are commutative, any delivery order is sufficient (ignoring over-/under-flow). The state-based counters are more interesting so we&#x2019;ll look at those.\n
A G-Counter only counts up and is basically a version vector (vector clock). Each replica increments its own pair only, the value is computed by summing the count of all replicas. Convergence is achieved by taking the maximum count for each replica. This is basically the Cassandra counters implementation.\n
A G-Counter only counts up and is basically a version vector (vector clock). Each replica increments its own pair only, the value is computed by summing the count of all replicas. Convergence is achieved by taking the maximum count for each replica. This is basically the Cassandra counters implementation.\n
A G-Counter only counts up and is basically a version vector (vector clock). Each replica increments its own pair only, the value is computed by summing the count of all replicas. Convergence is achieved by taking the maximum count for each replica. This is basically the Cassandra counters implementation.\n
A G-Counter only counts up and is basically a version vector (vector clock). Each replica increments its own pair only, the value is computed by summing the count of all replicas. Convergence is achieved by taking the maximum count for each replica. This is basically the Cassandra counters implementation.\n
A G-Counter only counts up and is basically a version vector (vector clock). Each replica increments its own pair only, the value is computed by summing the count of all replicas. Convergence is achieved by taking the maximum count for each replica. This is basically the Cassandra counters implementation.\n
A G-Counter only counts up and is basically a version vector (vector clock). Each replica increments its own pair only, the value is computed by summing the count of all replicas. Convergence is achieved by taking the maximum count for each replica. This is basically the Cassandra counters implementation.\n
A G-Counter only counts up and is basically a version vector (vector clock). Each replica increments its own pair only, the value is computed by summing the count of all replicas. Convergence is achieved by taking the maximum count for each replica. This is basically the Cassandra counters implementation.\n
A G-Counter only counts up and is basically a version vector (vector clock). Each replica increments its own pair only, the value is computed by summing the count of all replicas. Convergence is achieved by taking the maximum count for each replica. This is basically the Cassandra counters implementation.\n
A G-Counter only counts up and is basically a version vector (vector clock). Each replica increments its own pair only, the value is computed by summing the count of all replicas. Convergence is achieved by taking the maximum count for each replica. This is basically the Cassandra counters implementation.\n
A G-Counter only counts up and is basically a version vector (vector clock). Each replica increments its own pair only, the value is computed by summing the count of all replicas. Convergence is achieved by taking the maximum count for each replica. This is basically the Cassandra counters implementation.\n
A G-Counter only counts up and is basically a version vector (vector clock). Each replica increments its own pair only, the value is computed by summing the count of all replicas. Convergence is achieved by taking the maximum count for each replica. This is basically the Cassandra counters implementation.\n
A G-Counter only counts up and is basically a version vector (vector clock). Each replica increments its own pair only, the value is computed by summing the count of all replicas. Convergence is achieved by taking the maximum count for each replica. This is basically the Cassandra counters implementation.\n
A G-Counter only counts up and is basically a version vector (vector clock). Each replica increments its own pair only, the value is computed by summing the count of all replicas. Convergence is achieved by taking the maximum count for each replica. This is basically the Cassandra counters implementation.\n
A G-Counter only counts up and is basically a version vector (vector clock). Each replica increments its own pair only, the value is computed by summing the count of all replicas. Convergence is achieved by taking the maximum count for each replica. This is basically the Cassandra counters implementation.\n
PN-Counter - composed of two G-Counters - P for increments and N for decrements. The value is the difference between the values of the two G-Counters. The resolution is the pairwise resolution of the P and N counters.\n
Sets constitute one of the most basic data structures. Containers, Maps, and Graphs are all based on Sets. There are two operations, add and remove.\n
Like a G-Counter, a G-Set only grows in size. That is, it doesn&#x2019;t allow removal - its merge operation is a simple set-union, returning the maximal grouping without duplicates. Since add commutes with union, a G-Set can also be implemented as a commutative type. However, it&#x2019;s not an incredibly useful data-type on its own, but it can be part of another data structure.\n
Like a G-Counter, a G-Set only grows in size. That is, it doesn&#x2019;t allow removal - its merge operation is a simple set-union, returning the maximal grouping without duplicates. Since add commutes with union, a G-Set can also be implemented as a commutative type. However, it&#x2019;s not an incredibly useful data-type on its own, but it can be part of another data structure.\n
Like a G-Counter, a G-Set only grows in size. That is, it doesn&#x2019;t allow removal - its merge operation is a simple set-union, returning the maximal grouping without duplicates. Since add commutes with union, a G-Set can also be implemented as a commutative type. However, it&#x2019;s not an incredibly useful data-type on its own, but it can be part of another data structure.\n
Like a G-Counter, a G-Set only grows in size. That is, it doesn&#x2019;t allow removal - its merge operation is a simple set-union, returning the maximal grouping without duplicates. Since add commutes with union, a G-Set can also be implemented as a commutative type. However, it&#x2019;s not an incredibly useful data-type on its own, but it can be part of another data structure.\n
Like a G-Counter, a G-Set only grows in size. That is, it doesn&#x2019;t allow removal - its merge operation is a simple set-union, returning the maximal grouping without duplicates. Since add commutes with union, a G-Set can also be implemented as a commutative type. However, it&#x2019;s not an incredibly useful data-type on its own, but it can be part of another data structure.\n
Like a G-Counter, a G-Set only grows in size. That is, it doesn&#x2019;t allow removal - its merge operation is a simple set-union, returning the maximal grouping without duplicates. Since add commutes with union, a G-Set can also be implemented as a commutative type. However, it&#x2019;s not an incredibly useful data-type on its own, but it can be part of another data structure.\n
Like a G-Counter, a G-Set only grows in size. That is, it doesn&#x2019;t allow removal - its merge operation is a simple set-union, returning the maximal grouping without duplicates. Since add commutes with union, a G-Set can also be implemented as a commutative type. However, it&#x2019;s not an incredibly useful data-type on its own, but it can be part of another data structure.\n
Like a G-Counter, a G-Set only grows in size. That is, it doesn&#x2019;t allow removal - its merge operation is a simple set-union, returning the maximal grouping without duplicates. Since add commutes with union, a G-Set can also be implemented as a commutative type. However, it&#x2019;s not an incredibly useful data-type on its own, but it can be part of another data structure.\n
Like a G-Counter, a G-Set only grows in size. That is, it doesn&#x2019;t allow removal - its merge operation is a simple set-union, returning the maximal grouping without duplicates. Since add commutes with union, a G-Set can also be implemented as a commutative type. However, it&#x2019;s not an incredibly useful data-type on its own, but it can be part of another data structure.\n
Like a G-Counter, a G-Set only grows in size. That is, it doesn&#x2019;t allow removal - its merge operation is a simple set-union, returning the maximal grouping without duplicates. Since add commutes with union, a G-Set can also be implemented as a commutative type. However, it&#x2019;s not an incredibly useful data-type on its own, but it can be part of another data structure.\n
Like a G-Counter, a G-Set only grows in size. That is, it doesn&#x2019;t allow removal - its merge operation is a simple set-union, returning the maximal grouping without duplicates. Since add commutes with union, a G-Set can also be implemented as a commutative type. However, it&#x2019;s not an incredibly useful data-type on its own, but it can be part of another data structure.\n
Like a G-Counter, a G-Set only grows in size. That is, it doesn&#x2019;t allow removal - its merge operation is a simple set-union, returning the maximal grouping without duplicates. Since add commutes with union, a G-Set can also be implemented as a commutative type. However, it&#x2019;s not an incredibly useful data-type on its own, but it can be part of another data structure.\n
Like a G-Counter, a G-Set only grows in size. That is, it doesn&#x2019;t allow removal - its merge operation is a simple set-union, returning the maximal grouping without duplicates. Since add commutes with union, a G-Set can also be implemented as a commutative type. However, it&#x2019;s not an incredibly useful data-type on its own, but it can be part of another data structure.\n
Like a G-Counter, a G-Set only grows in size. That is, it doesn&#x2019;t allow removal - its merge operation is a simple set-union, returning the maximal grouping without duplicates. Since add commutes with union, a G-Set can also be implemented as a commutative type. However, it&#x2019;s not an incredibly useful data-type on its own, but it can be part of another data structure.\n
The second type of Set is a two-phase set, where a removed set member cannot be re-added. It is basically two G-Sets, one for add and one for remove. The removal set is sometimes called a tombstone set. To prevent spurious states (e.g. remove-before-add, making add have no effect), it has a precondition for remove that the local state must already contain the member.\n\nA special case of the 2P-Set is the U-Set. If the system can reasonably guarantee uniqueness, that is, the element will never be added again after removal, then the tombstone set is unnecessary. Uniqueness could be satisfied with a Lamport clock or suitably large RNG space.\n
The second type of Set is a two-phase set, where a removed set member cannot be re-added. It is basically two G-Sets, one for add and one for remove. The removal set is sometimes called a tombstone set. To prevent spurious states (e.g. remove-before-add, making add have no effect), it has a precondition for remove that the local state must already contain the member.\n\nA special case of the 2P-Set is the U-Set. If the system can reasonably guarantee uniqueness, that is, the element will never be added again after removal, then the tombstone set is unnecessary. Uniqueness could be satisfied with a Lamport clock or suitably large RNG space.\n
The second type of Set is a two-phase set, where a removed set member cannot be re-added. It is basically two G-Sets, one for add and one for remove. The removal set is sometimes called a tombstone set. To prevent spurious states (e.g. remove-before-add, making add have no effect), it has a precondition for remove that the local state must already contain the member.\n\nA special case of the 2P-Set is the U-Set. If the system can reasonably guarantee uniqueness, that is, the element will never be added again after removal, then the tombstone set is unnecessary. Uniqueness could be satisfied with a Lamport clock or suitably large RNG space.\n
The second type of Set is a two-phase set, where a removed set member cannot be re-added. It is basically two G-Sets, one for add and one for remove. The removal set is sometimes called a tombstone set. To prevent spurious states (e.g. remove-before-add, making add have no effect), it has a precondition for remove that the local state must already contain the member.\n\nA special case of the 2P-Set is the U-Set. If the system can reasonably guarantee uniqueness, that is, the element will never be added again after removal, then the tombstone set is unnecessary. Uniqueness could be satisfied with a Lamport clock or suitably large RNG space.\n
The second type of Set is a two-phase set, where a removed set member cannot be re-added. It is basically two G-Sets, one for add and one for remove. The removal set is sometimes called a tombstone set. To prevent spurious states (e.g. remove-before-add, making add have no effect), it has a precondition for remove that the local state must already contain the member.\n\nA special case of the 2P-Set is the U-Set. If the system can reasonably guarantee uniqueness, that is, the element will never be added again after removal, then the tombstone set is unnecessary. Uniqueness could be satisfied with a Lamport clock or suitably large RNG space.\n
The second type of Set is a two-phase set, where a removed set member cannot be re-added. It is basically two G-Sets, one for add and one for remove. The removal set is sometimes called a tombstone set. To prevent spurious states (e.g. remove-before-add, making add have no effect), it has a precondition for remove that the local state must already contain the member.\n\nA special case of the 2P-Set is the U-Set. If the system can reasonably guarantee uniqueness, that is, the element will never be added again after removal, then the tombstone set is unnecessary. Uniqueness could be satisfied with a Lamport clock or suitably large RNG space.\n
The second type of Set is a two-phase set, where a removed set member cannot be re-added. It is basically two G-Sets, one for add and one for remove. The removal set is sometimes called a tombstone set. To prevent spurious states (e.g. remove-before-add, making add have no effect), it has a precondition for remove that the local state must already contain the member.\n\nA special case of the 2P-Set is the U-Set. If the system can reasonably guarantee uniqueness, that is, the element will never be added again after removal, then the tombstone set is unnecessary. Uniqueness could be satisfied with a Lamport clock or suitably large RNG space.\n
The second type of Set is a two-phase set, where a removed set member cannot be re-added. It is basically two G-Sets, one for add and one for remove. The removal set is sometimes called a tombstone set. To prevent spurious states (e.g. remove-before-add, making add have no effect), it has a precondition for remove that the local state must already contain the member.\n\nA special case of the 2P-Set is the U-Set. If the system can reasonably guarantee uniqueness, that is, the element will never be added again after removal, then the tombstone set is unnecessary. Uniqueness could be satisfied with a Lamport clock or suitably large RNG space.\n
The second type of Set is a two-phase set, where a removed set member cannot be re-added. It is basically two G-Sets, one for add and one for remove. The removal set is sometimes called a tombstone set. To prevent spurious states (e.g. remove-before-add, making add have no effect), it has a precondition for remove that the local state must already contain the member.\n\nA special case of the 2P-Set is the U-Set. If the system can reasonably guarantee uniqueness, that is, the element will never be added again after removal, then the tombstone set is unnecessary. Uniqueness could be satisfied with a Lamport clock or suitably large RNG space.\n
The second type of Set is a two-phase set, where a removed set member cannot be re-added. It is basically two G-Sets, one for add and one for remove. The removal set is sometimes called a tombstone set. To prevent spurious states (e.g. remove-before-add, making add have no effect), it has a precondition for remove that the local state must already contain the member.\n\nA special case of the 2P-Set is the U-Set. If the system can reasonably guarantee uniqueness, that is, the element will never be added again after removal, then the tombstone set is unnecessary. Uniqueness could be satisfied with a Lamport clock or suitably large RNG space.\n
The second type of Set is a two-phase set, where a removed set member cannot be re-added. It is basically two G-Sets, one for add and one for remove. The removal set is sometimes called a tombstone set. To prevent spurious states (e.g. remove-before-add, making add have no effect), it has a precondition for remove that the local state must already contain the member.\n\nA special case of the 2P-Set is the U-Set. If the system can reasonably guarantee uniqueness, that is, the element will never be added again after removal, then the tombstone set is unnecessary. Uniqueness could be satisfied with a Lamport clock or suitably large RNG space.\n
The second type of Set is a two-phase set, where a removed set member cannot be re-added. It is basically two G-Sets, one for add and one for remove. The removal set is sometimes called a tombstone set. To prevent spurious states (e.g. remove-before-add, making add have no effect), it has a precondition for remove that the local state must already contain the member.\n\nA special case of the 2P-Set is the U-Set. If the system can reasonably guarantee uniqueness, that is, the element will never be added again after removal, then the tombstone set is unnecessary. Uniqueness could be satisfied with a Lamport clock or suitably large RNG space.\n
The second type of Set is a two-phase set, where a removed set member cannot be re-added. It is basically two G-Sets, one for add and one for remove. The removal set is sometimes called a tombstone set. To prevent spurious states (e.g. remove-before-add, making add have no effect), it has a precondition for remove that the local state must already contain the member.\n\nA special case of the 2P-Set is the U-Set. If the system can reasonably guarantee uniqueness, that is, the element will never be added again after removal, then the tombstone set is unnecessary. Uniqueness could be satisfied with a Lamport clock or suitably large RNG space.\n
The second type of Set is a two-phase set, where a removed set member cannot be re-added. It is basically two G-Sets, one for add and one for remove. The removal set is sometimes called a tombstone set. To prevent spurious states (e.g. remove-before-add, making add have no effect), it has a precondition for remove that the local state must already contain the member.\n\nA special case of the 2P-Set is the U-Set. If the system can reasonably guarantee uniqueness, that is, the element will never be added again after removal, then the tombstone set is unnecessary. Uniqueness could be satisfied with a Lamport clock or suitably large RNG space.\n
The second type of Set is a two-phase set, where a removed set member cannot be re-added. It is basically two G-Sets, one for add and one for remove. The removal set is sometimes called a tombstone set. To prevent spurious states (e.g. remove-before-add, making add have no effect), it has a precondition for remove that the local state must already contain the member.\n\nA special case of the 2P-Set is the U-Set. If the system can reasonably guarantee uniqueness, that is, the element will never be added again after removal, then the tombstone set is unnecessary. Uniqueness could be satisfied with a Lamport clock or suitably large RNG space.\n
The second type of Set is a two-phase set, where a removed set member cannot be re-added. It is basically two G-Sets, one for add and one for remove. The removal set is sometimes called a tombstone set. To prevent spurious states (e.g. remove-before-add, making add have no effect), it has a precondition for remove that the local state must already contain the member.\n\nA special case of the 2P-Set is the U-Set. If the system can reasonably guarantee uniqueness, that is, the element will never be added again after removal, then the tombstone set is unnecessary. Uniqueness could be satisfied with a Lamport clock or suitably large RNG space.\n
Tag each element in A and R with timestamp. Greatest timestamp wins out for each individual element. Could be implemented with Cassandra super-columns.\n\nFigure 12: LWW-element-Set; elements masked by one with a higher timestamp are elided (state-based)\n\n
Tag each added element uniquely (without exposing them). When removing, remove all seen and forward operation downstream with tags. State-based version would be based on U-Set.\n\n
You might notice we&#x2019;re going up in complexity here in terms of the types of data-structures. Graphs are incredibly useful for many problems, but also have a bunch of potential anomalies within them - concurrent add/removes of vertices and edges may not converge - that is, global invariants can&#x2019;t be guaranteed. For example, in the case of a DAG or linked-list where elements can be removed or added concurrently. Some anomalies may be removed via restricting the semantics, for example, making a graph add-only. I&#x2019;m not going to go into detail about how Graphs are implemented, but a simple one is the 2P2P graph, based on a pair of 2P-sets, one for vertices and one for edges. In the case where a vertex is removed, the most reliable (and intuitive) solution is to remove all attached edges, thus a 2P-Set paradigm works well for the components of a generic graph.\n\n\n
You might notice we&#x2019;re going up in complexity here in terms of the types of data-structures. Graphs are incredibly useful for many problems, but also have a bunch of potential anomalies within them - concurrent add/removes of vertices and edges may not converge - that is, global invariants can&#x2019;t be guaranteed. For example, in the case of a DAG or linked-list where elements can be removed or added concurrently. Some anomalies may be removed via restricting the semantics, for example, making a graph add-only. I&#x2019;m not going to go into detail about how Graphs are implemented, but a simple one is the 2P2P graph, based on a pair of 2P-sets, one for vertices and one for edges. In the case where a vertex is removed, the most reliable (and intuitive) solution is to remove all attached edges, thus a 2P-Set paradigm works well for the components of a generic graph.\n\n\n
You might notice we&#x2019;re going up in complexity here in terms of the types of data-structures. Graphs are incredibly useful for many problems, but also have a bunch of potential anomalies within them - concurrent add/removes of vertices and edges may not converge - that is, global invariants can&#x2019;t be guaranteed. For example, in the case of a DAG or linked-list where elements can be removed or added concurrently. Some anomalies may be removed via restricting the semantics, for example, making a graph add-only. I&#x2019;m not going to go into detail about how Graphs are implemented, but a simple one is the 2P2P graph, based on a pair of 2P-sets, one for vertices and one for edges. In the case where a vertex is removed, the most reliable (and intuitive) solution is to remove all attached edges, thus a 2P-Set paradigm works well for the components of a generic graph.\n\n\n
You might notice we&#x2019;re going up in complexity here in terms of the types of data-structures. Graphs are incredibly useful for many problems, but also have a bunch of potential anomalies within them - concurrent add/removes of vertices and edges may not converge - that is, global invariants can&#x2019;t be guaranteed. For example, in the case of a DAG or linked-list where elements can be removed or added concurrently. Some anomalies may be removed via restricting the semantics, for example, making a graph add-only. I&#x2019;m not going to go into detail about how Graphs are implemented, but a simple one is the 2P2P graph, based on a pair of 2P-sets, one for vertices and one for edges. In the case where a vertex is removed, the most reliable (and intuitive) solution is to remove all attached edges, thus a 2P-Set paradigm works well for the components of a generic graph.\n\n\n
\n
CRDTs tend to create a lot of garbage: tombstones grow and internal structures become unbalanced. In general, garbage collection is extremely difficult to do without synchronization. Luckily, this doesn&#x2019;t impact correctness, only efficiency and performance.\n
Client - have to come up with a common representation across languages, allocation of actor IDs is problematic, can only use state-based CRDTs.\nServer - no one implements them yet, really (Cassandra&#x2019;s counter has some anomalies), but we&#x2019;re working hard to bring them to Riak.\n