download for better quality - Learn about the sequence and traverse functions
through the work of Runar Bjarnason and Paul Chiusano, authors of Functional Programming in Scala https://www.manning.com/books/functional-programming-in-scala, and others (Martin Odersky, Derek Wyatt, Adelbert Chang)
Folding Unfolded - Polyglot FP for Fun and Profit - Haskell and ScalaPhilip Schwarz
(download for perfect quality) - See how recursive functions and structural induction relate to recursive datatypes.
Follow along as the fold abstraction is introduced and explained.
Watch as folding is used to simplify the definition of recursive functions over recursive datatypes
Part 1 - through the work of Richard Bird and Graham Hutton.
Errata:
slide 7, 11 fib(0) is 0,rather than 1
slide 23: was supposed to be followed by 2-3 slides recapitulating definitions of factorial and fibonacci with and without foldr, plus translation to scala
slide 36: concat not invoked in concat example
slides 48 and 49: unwanted 'm' in definition of sum
throughout: a couple of typographical errors
throughout: several aesthetic imperfections (wrong font, wrong font colour)
download for better quality - Learn about the sequence and traverse functions
through the work of Runar Bjarnason and Paul Chiusano, authors of Functional Programming in Scala https://www.manning.com/books/functional-programming-in-scala, and others (Martin Odersky, Derek Wyatt, Adelbert Chang)
Folding Unfolded - Polyglot FP for Fun and Profit - Haskell and ScalaPhilip Schwarz
(download for perfect quality) - See how recursive functions and structural induction relate to recursive datatypes.
Follow along as the fold abstraction is introduced and explained.
Watch as folding is used to simplify the definition of recursive functions over recursive datatypes
Part 1 - through the work of Richard Bird and Graham Hutton.
Errata:
slide 7, 11 fib(0) is 0,rather than 1
slide 23: was supposed to be followed by 2-3 slides recapitulating definitions of factorial and fibonacci with and without foldr, plus translation to scala
slide 36: concat not invoked in concat example
slides 48 and 49: unwanted 'm' in definition of sum
throughout: a couple of typographical errors
throughout: several aesthetic imperfections (wrong font, wrong font colour)
Folding Unfolded - Polyglot FP for Fun and Profit - Haskell and Scala - Part 5Philip Schwarz
(download for best quality slides) Gain a deeper understanding of why right folds over very large and infinite lists are sometimes possible in Haskell.
See how lazy evaluation and function strictness affect left and right folds in Haskell.
Learn when an ordinary left fold results in a space leak and how to avoid it using a strict left fold.
Errata:
slide 15: "as sharing is required" should be "as sharing is not required"
slide 43: 𝑠𝑓𝑜𝑙𝑑𝑙 (⊕) 𝑎 should be 𝑠𝑓𝑜𝑙𝑑𝑙 (⊕) 𝑒
This hands-on R course will guide users through a variety of programming functions in the open-source statistical software program, R. Topics covered include indexing, loops, conditional branching, S3 classes, and debugging. Full workshop materials available from http://projects.iq.harvard.edu/rtc/r-prog
Big picture of category theory in scala with deep dive into contravariant and...Piotr Paradziński
A big picture of category theory in Scala - starting from regular functors with additional structure (Apply, Applicative, Monad) to Comonads. Usually, we think about structures like Monoids in a monoidal category with particular tensor. In here I analyze just signatures of different abstractions.
Exploration of Contravariant functors as a way to model computation "backward" or abstract over input with the ability to prepend operation. Examples for predicates, sorting, show and function input (or any other function parameter except the last one).
Profunctors as abstraction unifying Functors and Contravariant functors to model both input and output. Example for Profunctor - function with one argument.
Relation to Bifunctors, Kan extensions, Adjunctions, and Free constructions.
Folding Unfolded - Polyglot FP for Fun and Profit - Haskell and Scala - Part ...Philip Schwarz
(download for best quality slides) Gain a deeper understanding of why right folds over very large and infinite lists are sometimes possible in Haskell.
See how lazy evaluation and function strictness affect left and right folds in Haskell.
Learn when an ordinary left fold results in a space leak and how to avoid it using a strict left fold.
This version eliminates some minor imperfections and corrects the following two errors:
slide 15: "as sharing is required" should be "as sharing is not required"
slide 43: 푠푓표푙푑푙 (⊕) 푎 should be 푠푓표푙푑푙 (⊕) 푒
Folding Unfolded - Polyglot FP for Fun and Profit - Haskell and Scala - Part ...Philip Schwarz
(download for flawless image quality) Can a left fold ever work over an infinite list? What about a right fold? Find out.
Learn about the other two functions used by functional programmers to implement mathematical induction: iterating and scanning.
Learn about the limitations of the accumulator technique and about tupling, a technique that is the dual of the accumulator trick.
N-Queens Combinatorial Problem - Polyglot FP for Fun and Profit – Haskell and...Philip Schwarz
See how the guard function has migrated from MonadPlus to Alternative and learn something about the latter.
Learn how to write a Scala program that draws an N-Queens solution board using the Doodle graphics library.
See how to write the equivalent Haskell program using the Gloss graphics library.
Learn how to use Monoid and Foldable to compose images both in Haskell and in Scala.
Link to part 1: https://www.slideshare.net/pjschwarz/nqueens-combinatorial-problem-polyglot-fp-for-fun-and-profit-haskell-and-scala-part-1
Errata:
On slide 22, the last line of the showQueens function should of course be show(solution).draw(frame) rather than show(solution).draw
On slide 43, it would be better if the definitions of the beside, above and on Monoids were also shown.
Functional Core and Imperative Shell - Game of Life Example - Haskell and ScalaPhilip Schwarz
See a program structure flowchart used to highlight how an FP program breaks down into a functional core and imperative shell
View a program structure flowchart for the Game of Life
See the code for Game of Life’s functional core and imperative shell, both in Haskell and in Scala.
Code:
https://github.com/philipschwarz/functional-core-imperative-shell-scala
https://github.com/philipschwarz/functional-core-imperative-shell-haskell
Sierpinski Triangle - Polyglot FP for Fun and Profit - Haskell and ScalaPhilip Schwarz
Take the very first baby steps on the path to doing graphics in Haskell and Scala.
Learn about a simple yet educational recursive algorithm producing images that are pleasing to the eye.
Learn how functional programs deal with the side effects required to draw images.
See how libraries like Gloss and Doodle make drawing Sierpinski’s triangle a doddle.
Code for this slide deck:
https://github.com/philipschwarz/sierpinski-triangle-haskell-gloss
https://github.com/philipschwarz/sierpinski-triangle-scala-cats-io
https://github.com/philipschwarz/sierpinski-triangle-scala-awt-and-doodle
Errata:
1. the title 'Sierpinski Triangle' on the front slide could be improved by replacing it with 'Sierpinski's Triangle'.
2. a couple of typos on two slides
3. the triangles drawn using Doodle are not equilateral, as intended but isosceles.
(UPDATE 2021-06-15 I opened PR https://github.com/creativescala/doodle/pull/99 and as a result, an equilateral triangle has now been added to Doodle: https://github.com/creativescala/doodle/commit/30d20efebcc2016942e9cdbae85fefca5b95fa3c).
Here is a corrected version of the deck: https://www.slideshare.net/pjschwarz/sierpinski-triangle-polyglot-fp-for-fun-and-profit-haskell-and-scala-with-minor-corrections
This ppt includes -
History
Introduction
R Basics
Download and Install R
GUI /IDE
Datatypes and Operators
Conditional Statements and Loops
Functions
Plotting
Features
Comparison with other programming languages
Merits / Advantages
Demerits / Disadvantages
Conclusion
Folding Unfolded - Polyglot FP for Fun and Profit - Haskell and Scala - Part 4Philip Schwarz
(download for flawless image quality) Can a left fold ever work over an infinite list? What about a right fold? Find out.
Learn about the other two functions used by functional programmers to implement mathematical induction: iterating and scanning.
Learn about the limitations of the accumulator technique and about tupling, a technique that is the dual of the accumulator trick.
Folding Unfolded - Polyglot FP for Fun and Profit - Haskell and Scala - Part 5Philip Schwarz
(download for best quality slides) Gain a deeper understanding of why right folds over very large and infinite lists are sometimes possible in Haskell.
See how lazy evaluation and function strictness affect left and right folds in Haskell.
Learn when an ordinary left fold results in a space leak and how to avoid it using a strict left fold.
Errata:
slide 15: "as sharing is required" should be "as sharing is not required"
slide 43: 𝑠𝑓𝑜𝑙𝑑𝑙 (⊕) 𝑎 should be 𝑠𝑓𝑜𝑙𝑑𝑙 (⊕) 𝑒
This hands-on R course will guide users through a variety of programming functions in the open-source statistical software program, R. Topics covered include indexing, loops, conditional branching, S3 classes, and debugging. Full workshop materials available from http://projects.iq.harvard.edu/rtc/r-prog
Big picture of category theory in scala with deep dive into contravariant and...Piotr Paradziński
A big picture of category theory in Scala - starting from regular functors with additional structure (Apply, Applicative, Monad) to Comonads. Usually, we think about structures like Monoids in a monoidal category with particular tensor. In here I analyze just signatures of different abstractions.
Exploration of Contravariant functors as a way to model computation "backward" or abstract over input with the ability to prepend operation. Examples for predicates, sorting, show and function input (or any other function parameter except the last one).
Profunctors as abstraction unifying Functors and Contravariant functors to model both input and output. Example for Profunctor - function with one argument.
Relation to Bifunctors, Kan extensions, Adjunctions, and Free constructions.
Folding Unfolded - Polyglot FP for Fun and Profit - Haskell and Scala - Part ...Philip Schwarz
(download for best quality slides) Gain a deeper understanding of why right folds over very large and infinite lists are sometimes possible in Haskell.
See how lazy evaluation and function strictness affect left and right folds in Haskell.
Learn when an ordinary left fold results in a space leak and how to avoid it using a strict left fold.
This version eliminates some minor imperfections and corrects the following two errors:
slide 15: "as sharing is required" should be "as sharing is not required"
slide 43: 푠푓표푙푑푙 (⊕) 푎 should be 푠푓표푙푑푙 (⊕) 푒
Folding Unfolded - Polyglot FP for Fun and Profit - Haskell and Scala - Part ...Philip Schwarz
(download for flawless image quality) Can a left fold ever work over an infinite list? What about a right fold? Find out.
Learn about the other two functions used by functional programmers to implement mathematical induction: iterating and scanning.
Learn about the limitations of the accumulator technique and about tupling, a technique that is the dual of the accumulator trick.
N-Queens Combinatorial Problem - Polyglot FP for Fun and Profit – Haskell and...Philip Schwarz
See how the guard function has migrated from MonadPlus to Alternative and learn something about the latter.
Learn how to write a Scala program that draws an N-Queens solution board using the Doodle graphics library.
See how to write the equivalent Haskell program using the Gloss graphics library.
Learn how to use Monoid and Foldable to compose images both in Haskell and in Scala.
Link to part 1: https://www.slideshare.net/pjschwarz/nqueens-combinatorial-problem-polyglot-fp-for-fun-and-profit-haskell-and-scala-part-1
Errata:
On slide 22, the last line of the showQueens function should of course be show(solution).draw(frame) rather than show(solution).draw
On slide 43, it would be better if the definitions of the beside, above and on Monoids were also shown.
Functional Core and Imperative Shell - Game of Life Example - Haskell and ScalaPhilip Schwarz
See a program structure flowchart used to highlight how an FP program breaks down into a functional core and imperative shell
View a program structure flowchart for the Game of Life
See the code for Game of Life’s functional core and imperative shell, both in Haskell and in Scala.
Code:
https://github.com/philipschwarz/functional-core-imperative-shell-scala
https://github.com/philipschwarz/functional-core-imperative-shell-haskell
Sierpinski Triangle - Polyglot FP for Fun and Profit - Haskell and ScalaPhilip Schwarz
Take the very first baby steps on the path to doing graphics in Haskell and Scala.
Learn about a simple yet educational recursive algorithm producing images that are pleasing to the eye.
Learn how functional programs deal with the side effects required to draw images.
See how libraries like Gloss and Doodle make drawing Sierpinski’s triangle a doddle.
Code for this slide deck:
https://github.com/philipschwarz/sierpinski-triangle-haskell-gloss
https://github.com/philipschwarz/sierpinski-triangle-scala-cats-io
https://github.com/philipschwarz/sierpinski-triangle-scala-awt-and-doodle
Errata:
1. the title 'Sierpinski Triangle' on the front slide could be improved by replacing it with 'Sierpinski's Triangle'.
2. a couple of typos on two slides
3. the triangles drawn using Doodle are not equilateral, as intended but isosceles.
(UPDATE 2021-06-15 I opened PR https://github.com/creativescala/doodle/pull/99 and as a result, an equilateral triangle has now been added to Doodle: https://github.com/creativescala/doodle/commit/30d20efebcc2016942e9cdbae85fefca5b95fa3c).
Here is a corrected version of the deck: https://www.slideshare.net/pjschwarz/sierpinski-triangle-polyglot-fp-for-fun-and-profit-haskell-and-scala-with-minor-corrections
This ppt includes -
History
Introduction
R Basics
Download and Install R
GUI /IDE
Datatypes and Operators
Conditional Statements and Loops
Functions
Plotting
Features
Comparison with other programming languages
Merits / Advantages
Demerits / Disadvantages
Conclusion
Folding Unfolded - Polyglot FP for Fun and Profit - Haskell and Scala - Part 4Philip Schwarz
(download for flawless image quality) Can a left fold ever work over an infinite list? What about a right fold? Find out.
Learn about the other two functions used by functional programmers to implement mathematical induction: iterating and scanning.
Learn about the limitations of the accumulator technique and about tupling, a technique that is the dual of the accumulator trick.
Accumulo Summit 2015: Rya: Optimizations to Support Real Time Graph Queries o...Accumulo Summit
Talk Abstract
The Resource Description Framework (RDF) is a standard model for expressing graph data for the World Wide Web. Developed by the W3C, RDF and related technologies such as OWL and SKOS provide a rich vocabulary for exchanging graph data in a machine understandable manner. As the size of available data continues to grow, there has been an increased desire for methods of storing very large RDF graphs within big data architectures. Rya is a government open source scalable RDF triple store built on top of Apache Accumulo. Originally developed by the Laboratory for Telecommunication Sciences and US Naval Academy, Rya is currently being used by a number of government agencies for storing, inferencing, and querying large amounts of RDF data.
As Rya’s user base has grown, there has been a stronger requirement for near real time query responsiveness over massive RDF graphs. In this talk, we detail several query optimization strategies the Rya team has pursued to better satisfy this requirement. We describe recent work allowing for the use of additional indices to eliminate large common joins within complex SPARQL queries. Additionally, we explain a number of statistics based optimizations to improve query planning. Specifically, we detail extensions to existing methods of estimating the selectivity of individual statement patterns (cardinality) and the selectivity of joining two statement patterns (join selectivity) to better fit a “big data” paradigm and utilize Accumulo. Finally, we share preliminary performance evaluation results for the optimizations that have been pursued.
Speaker
Caleb Meier
Engineer/Algorithm Developer, Parsons Corporation
Dr. Caleb Meier received a PhD from the University of California San Diego (UCSD) in Mathematics in 2012. For the past two years, he was a postdoctoral fellow at UCSD's Math department specializing in non-linear elliptic systems of partial differential equations. He received his undergraduate degree in Mathematics from Yale University in 2006. Dr. Meier is currently working as an engineer at Parsons Corporation, specializing in query optimization algorithms for large scale RDF graphs. He is an expert in semantic technologies, Accumulo, the Hadoop Ecosystem, and is actually more fun to be around than his bio suggests.
Community Finding with Applications on Phylogenetic Networks [Presentation]Luís Rita
[Master Thesis Presentation]
With the advent of high-throughput sequencing methods, new ways of visualizing and analyzing increasingly amounts of data are needed. Although some software already exist, they do not scale well or require advanced skills to be useful in phylogenetics.
The aim of this thesis was to implement three community finding algorithms – Louvain, Infomap and Layered Label Propagation (LLP); to benchmark them using two synthetic networks – Girvan-Newman (GN) and Lancichinetti-Fortunato-Radicchi (LFR); to test them in real networks, particularly, in one derived from a Staphylococcus aureus MLST dataset; to compare visualization frameworks – Cytoscape.js and D3.js, and, finally, to make it all available online (mscthesis.herokuapp.com).
Louvain, Infomap and LLP were implemented in JavaScript. Unless otherwise stated, next conclusions are valid for GN and LFR. In terms of speed, Louvain outperformed all others. Considering accuracy, in networks with well-defined communities, Louvain was the most accurate. For higher mixing, LLP was the best. Contrarily to weakly mixed, it is advantageous to increase the resolution parameter in highly mixed GN. In LFR, higher resolution decreases the accuracy of detection, independently of the mixing parameter. The increase of the average node degree enhanced partitioning accuracy and suggested detection by chance was minimized. It is computationally more intensive to generate GN with higher mixing or average degree, using the algorithm developed in the thesis or the LFR implementation. In S. aureus network, Louvain was the fastest and the most accurate in detecting the clusters of seven groups of strains directly evolved from the common ancestor.
Interactive Concept Mapping in ActiveMath (iCMap)Martin Homik
Describes a tool for conept mapping in mathematics that offers feedback, evaluation, and suggestions. It is part of the ActiveMath learning environment.
5th in the AskTOM Office Hours series on graph database technologies. https://devgym.oracle.com/pls/apex/dg/office_hours/3084
PGQL: A Query Language for Graphs
Learn how to query graphs using PGQL, an expressive and intuitive graph query language that's a lot like SQL. With PGQL, it's easy to get going writing graph analysis queries to the database in a very short time. Albert and Oskar show what you can do with PGQL, and how to write and execute PGQL code.
Functors, Comonads, and Digital Image ProcessingJustin Le
Exploring concepts in category theory and abstract algebra -- in particular, functors and comonads -- to solve problems in analyzing, modeling, and implementing digital image filters, with possible extensions to signal processing and numerical algorithms in general.
Future features for openCypher: Schema, Constraints, Subqueries, Configurable...openCypher
Presented at the First openCypher Implementers Meeting in Walldorf, Germany, February 2017 @ http://www.opencypher.org/blog/2017/03/31/first-ocim-blog/
Detecting paraphrases using recursive autoencodersFeynman Liang
Presentation on deep learning applied to natural language processing, presented at University of Cambridge Machine Learning Group's Research and Communication Club 2-11-2015 meeting.
UiPath Test Automation using UiPath Test Suite series, part 4DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap.
The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies.
Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques
What will you get from this session?
1. Insights into SAP testing best practices
2. Heatmap utilization for testing
3. Optimization of testing processes
4. Demo
Topics covered:
Execution from the test manager
Orchestrator execution result
Defect reporting
SAP heatmap example with demo
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
"Impact of front-end architecture on development cost", Viktor TurskyiFwdays
I have heard many times that architecture is not important for the front-end. Also, many times I have seen how developers implement features on the front-end just following the standard rules for a framework and think that this is enough to successfully launch the project, and then the project fails. How to prevent this and what approach to choose? I have launched dozens of complex projects and during the talk we will analyze which approaches have worked for me and which have not.
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualityInflectra
In this insightful webinar, Inflectra explores how artificial intelligence (AI) is transforming software development and testing. Discover how AI-powered tools are revolutionizing every stage of the software development lifecycle (SDLC), from design and prototyping to testing, deployment, and monitoring.
Learn about:
• The Future of Testing: How AI is shifting testing towards verification, analysis, and higher-level skills, while reducing repetitive tasks.
• Test Automation: How AI-powered test case generation, optimization, and self-healing tests are making testing more efficient and effective.
• Visual Testing: Explore the emerging capabilities of AI in visual testing and how it's set to revolutionize UI verification.
• Inflectra's AI Solutions: See demonstrations of Inflectra's cutting-edge AI tools like the ChatGPT plugin and Azure Open AI platform, designed to streamline your testing process.
Whether you're a developer, tester, or QA professional, this webinar will give you valuable insights into how AI is shaping the future of software delivery.
JMeter webinar - integration with InfluxDB and GrafanaRTTS
Watch this recorded webinar about real-time monitoring of application performance. See how to integrate Apache JMeter, the open-source leader in performance testing, with InfluxDB, the open-source time-series database, and Grafana, the open-source analytics and visualization application.
In this webinar, we will review the benefits of leveraging InfluxDB and Grafana when executing load tests and demonstrate how these tools are used to visualize performance metrics.
Length: 30 minutes
Session Overview
-------------------------------------------
During this webinar, we will cover the following topics while demonstrating the integrations of JMeter, InfluxDB and Grafana:
- What out-of-the-box solutions are available for real-time monitoring JMeter tests?
- What are the benefits of integrating InfluxDB and Grafana into the load testing stack?
- Which features are provided by Grafana?
- Demonstration of InfluxDB and Grafana using a practice web application
To view the webinar recording, go to:
https://www.rttsweb.com/jmeter-integration-webinar
Search and Society: Reimagining Information Access for Radical FuturesBhaskar Mitra
The field of Information retrieval (IR) is currently undergoing a transformative shift, at least partly due to the emerging applications of generative AI to information access. In this talk, we will deliberate on the sociotechnical implications of generative AI for information access. We will argue that there is both a critical necessity and an exciting opportunity for the IR community to re-center our research agendas on societal needs while dismantling the artificial separation between the work on fairness, accountability, transparency, and ethics in IR and the rest of IR research. Instead of adopting a reactionary strategy of trying to mitigate potential social harms from emerging technologies, the community should aim to proactively set the research agenda for the kinds of systems we should build inspired by diverse explicitly stated sociotechnical imaginaries. The sociotechnical imaginaries that underpin the design and development of information access technologies needs to be explicitly articulated, and we need to develop theories of change in context of these diverse perspectives. Our guiding future imaginaries must be informed by other academic fields, such as democratic theory and critical theory, and should be co-developed with social science scholars, legal scholars, civil rights and social justice activists, and artists, among others.
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf91mobiles
91mobiles recently conducted a Smart TV Buyer Insights Survey in which we asked over 3,000 respondents about the TV they own, aspects they look at on a new TV, and their TV buying preferences.
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
Let's dive deeper into the world of ODC! Ricardo Alves (OutSystems) will join us to tell all about the new Data Fabric. After that, Sezen de Bruijn (OutSystems) will get into the details on how to best design a sturdy architecture within ODC.
2. MODEL-DRIVEN ENGINEERING
Primarily for designing critical systems
Models are first class citizens during development
o SysML / requirements, statecharts, etc.
o Validation and code generation techniques for correctness
Technology:
Eclipse Modeling Framework (EMF)
Originally started at IBM as an implementation of the Object
Management Group’s (OMG) Meta Object Facility (MOF).
i.e. an object-oriented model
i.e. a property graph-like structure with a metamodel
3. MODEL VALIDATION
Implemented with model queries
Models are typed, attributed graphs
Typical queries
o Get two components connected by a particular edge
MATCH (r:R)…(s:S) WHERE NOT (r)-[:E]->(s)
o Check if two objects are reachable
MATCH (r:R)…(s:S) WHERE NOT (r)-[:E1|E2*]->(s)
o Property checks
MATCH (r:R)-->(s:S) WHERE r.a = 'x' OR (s:Y)
Complex graph queries
16. INCREMENTAL VIEW MAINTENANCE (IVM)
In many use cases…
queries are static
data changes slowly
-> views can be maintained incrementally
Graph applications
model validation
simulation
recommendation systems
fraud detection
17. INGRAPH: IVM ON PROPERTY GRAPHS
Idea: map to relational algebra and use standard IVM techniques
Challenging aspects
o Property graph data model
o Cypher language
Formalise the language in relational algebra
Use nested relational algebra -> closed on operations
Prototype tool: ingraph (OCIM1, OCIM2, GraphConnect talks)
Gábor Szárnyas, József Marton, Dániel Varró:
Formalising openCypher Graph Queries in Relational Algebra.
ADBIS 2017
20. INGRAPH
ingraph uses a procedural IVM approach: the Rete algorithm.
o Build caches for each operator
o Maintain caches upon changes
o Supports 15+ out of 25 LDBC BI queries
o Details to be published in a conference paper
o Extensible, but very heavy on memory
The rest of the talk focuses on the algebraic approach.
Gábor Szárnyas, József Marton et al.:
Incremental View Maintenance on Property Graphs.
arXiv preprint will be available on the 1st week of June
22. DELTA QUERIES AT A GLANCE
𝐺
evaluate query 𝑄
for each Δ𝐺
evaluate Δ𝑄
changes
Δ𝐺1, Δ𝐺2, …
𝑄(𝐺)
Δ𝑄(Δ𝐺1)
Δ𝑄(Δ𝐺2)
⇒ 𝑄(𝐺 + Δ𝐺1 + Δ𝐺2 + ⋯ )
𝑄 and Δ𝑄 are calculated by the same engine.
23. IMPLEMENTATION: TRIGGERS IN NEO4J
Event-driven programming in databases
Neo4j: TransactionEventHandler interface
o afterCommit(TransactionData data, T state)
o beforeCommit(TransactionData data)
o TransactionData contains Δ𝐺: createdNodes, deletedNodes, …
Only the updated state of the graph is accessible.
GraphAware framework: ImprovedTransactionData API
o Get properties and labels/types of deleted elements
Max de Marzi:
Triggers in Neo4j.
2015
Michal Bachman:
Neo4j Improved Transaction Event API.
2014
24. DERIVING DELTA QUERIES
a b
1 2
3 4
5 6
7 8
a b
1 2
5 6
7 8
𝑅 𝑚
Idea: given query 𝑄, derive delta queries Δ𝑄 and 𝛻𝑄, which
define positive and negative changes, respectively.
But: most IVM techniques are defined for relational algebra.
Notation:
𝑅 relation
Δ𝑅 positive changes
𝛻𝑅 negative changes
𝑅 𝑚
: maintained relation of 𝑅 ⇒ 𝑅 𝑚
= 𝑅 − 𝛻𝑅 + Δ𝑅
“−” denotes set minus (∖), “+” denotes set union (∪)
a b
1 2
3 4
5 6
𝑅 𝛻𝑅
Δ𝑅
27. T. Griffin, B. Kumar:
Algebraic Change Propagation for Semijoin and Outerjoin Queries.
SIGMOD Record 1998
DERIVING DELTA QUERIES
T. Griffin, L. Libkin:
Incremental Maintenance of Views with Duplicates.
SIGMOD 1995
T. Griffin, L. Libkin, H. Trickey: An Improved Algorithm for the
Incremental Recomputation of Active Relational Expressions.
TKDE 1997
X. Qian, G. Wiederhold:
Incremental Recomputation of Active Relational Expressions.
TKDE 1991
28. DELTA QUERIES
T. Griffin, L. Libkin:
Incremental Maintenance of Views with Duplicates.
SIGMOD 1995
The seminal paper
Δ/𝛻 delta queries
for joins, selections,
projections, etc.
Bag semantics
29. DELTA QUERIES
Semijoins, antijoins,
outer joins
Set semantics
Later publications,
e.g. Zhou-Larson’s
ICDE’07 paper
improved these
T. Griffin, B. Kumar:
Algebraic Change Propagation for Semijoin and Outerjoin Queries.
SIGMOD Record 1998
31. EXAMPLE QUERY #1
MATCH (v1)-[:a]->(v2)-[:b]->(v3)-[:c]->(v4)
RETURN v1, v2, v3, v4
Δ 𝑎 ⋈ 𝑏 ⋈ 𝑐
= 𝑎 𝑚
⋈ 𝑏 𝑚
⋈ Δ𝑐 + 𝑎 𝑚
⋈ Δ𝑏 ⋈ 𝑐 𝑚
+ Δ𝑎 ⋈ 𝑏 𝑚
⋈ 𝑐 𝑚
UNWIND $pcs AS pc
MATCH (v1)-[:a]->(v2)-[:b]->(v3)-[pc]->(v4)
RETURN v1, v2, v3, v4
$pcs -> pass lists of nodes/edges as parameters
// This only works in embedded mode, see neo4j/issues/10239
v1 v2 v3
:b:a
𝑎 𝑣1, 𝑣2
𝑏 𝑣2, 𝑣3
𝑐 𝑣3, 𝑣4
v4
:c
32. POSITIVE DELTA QUERY FOR 𝑎 ⋈ 𝑏 ⋈ 𝑐
// r1 = a⋈b⋈Δc
UNWIND $pcs AS pc
MATCH (v1)-[:a]->(v2)-[:b]->(v3)-[pc]->(v4)
RETURN v1, v2, v3, v4
UNION ALL
// r2 = a⋈Δb⋈c
UNWIND $pbs AS pb
MATCH (v1)-[:a]->(v2)-[pb]->(v3)-[:c]->(v4)
RETURN v1, v2, v3, v4
UNION ALL
// r3 = Δa⋈b⋈c
UNWIND $pas AS pa
MATCH (v1)-[pa]->(v2)-[:b]->(v3)-[:c]->(v4)
RETURN v1, v2, v3, v4
33. POSITIVE DELTA QUERY FOR 𝑎 ⋈ 𝑏 ⋈ 𝑐
Long WITH chains are cumbersome -> patterns+list comprehensions.
WITH [pc IN $pcs | // r1 = a⋈b⋈Δc
[(v1)-[:a]->(v2)-[:b]->(v3)-[pc]->(v4) |
[v1, v2, v3, v4]]]
[pb IN $pbs | // r2 = a⋈Δb⋈c
[(v1)-[:a]->(v2)-[pb]->(v3)-[:c]->(v4) |
[v1, v2, v3, v4]]] +
[pa IN $pas | // r3 = Δa⋈b⋈c
[(v1)-[pa]->(v2)-[:b]->(v3)-[:c]->(v4) |
[v1, v2, v3, v4]]] AS r
RETURN
r[0] AS v1, r[1] AS v2, r[2] AS v3, r[3] AS v4
34. EXAMPLE QUERY #2
MATCH (route:Route)
-[:FOLLOWS]->(swP:SwitchPosition)
-[:TARGET]->(sw:Switch)
-[:MONITORED_BY]->(sensor:Sensor)
WHERE NOT (route)-[:REQUIRES]->(sensor)
RETURN route, sensor, swP, sw
MATCH (v1)
-[:a]->(v2)
-[:b]->(v3)
-[:c]->(v4)
WHERE NOT (v1)-[:d]->(v4)
RETURN v1, v2, v3, v4
v1 v2
v3v4
:b
:c
:d
:a
:FOLLOWS
:MONITORED_BY
:TARGET:REQUIRES
sw: Switchsensor: Sensor
route: Route
swP:
SwitchPosition
42. WITH
[] AS pes, [] AS nes
WITH
[pe IN pes WHERE type(pe) = 'a'|pe] AS pas,
[pe IN pes WHERE type(pe) = 'b'|pe] AS pbs,
[pe IN pes WHERE type(pe) = 'c'|pe] AS pcs,
[pe IN pes WHERE type(pe) = 'd'|pe] AS pds,
[ne IN nes WHERE type(ne) = 'a'|ne] AS nas,
[ne IN nes WHERE type(ne) = 'b'|ne] AS nbs,
[ne IN nes WHERE type(ne) = 'c'|ne] AS ncs,
[ne IN nes WHERE type(ne) = 'd'|ne] AS nds
WITH
pas, nas, pbs, nbs, pcs, ncs, pds, nds,
[nd IN nds | startNode(nd)] AS nd_v1s,
[nd IN nds | endNode(nd)] AS nd_v4s
// calculating s1s...s4s
// s1s: (𝑎⋉∇𝑑)
UNWIND nd_v1s AS v1
MATCH (v1)-[:a]->(v2)
WITH
pas, nas, pbs, nbs, pcs, ncs, pds, nds,
nd_v1s, nd_v4s,
collect({v1: v1, v2: v2}) AS s1s
// s2s: (Δ𝑎⋉∇𝑑)
UNWIND pas AS pa
MATCH (v1)-[pa]->(v2)
WHERE v1 IN nd_v1s
WITH
pas, nas, pbs, nbs, pcs, ncs, pds, nds,
nd_v1s, nd_v4s,
s1s,
collect({v1: v1, v2: v2}) AS s2s
// s3s: (𝑐⋉∇𝑑)
UNWIND nd_v4s AS v4
MATCH (v3)-[:c]->(v4)
WITH
pas, nas, pbs, nbs, pcs, ncs, pds, nds,
nd_v1s, nd_v4s,
s1s, s2s,note
collect({v3: v3, v4: v4}) AS s3s
// s4s: (Δ𝑐⋉∇𝑑)
UNWIND pcs AS pc
MATCH (v3)-[pc]->(v4)
WHERE v4 IN nd_v1s
WITH
pas, nas, pbs, nbs, pcs, ncs, pds, nds,
nd_v1s, nd_v4s,
s1s, s2s, s3s,
collect({v3: v3, v4: v4}) AS s4s
// calculating r1...r7
WITH
pas, nas, pbs, nbs, pcs, ncs, pds, nds,
nd_v1s, nd_v4s, s1s, s2s, s3s, s4s
// r1: (𝑎⋉∇𝑑) ⋈ 𝑏 ⋈ (𝑐⋉∇𝑑)
UNWIND s1s AS s1
UNWIND s3s AS s3
WITH
pas, nas, pbs, nbs, pcs, ncs, pds, nds,
nd_v1s, nd_v4s, s1s, s2s, s3s, s4s,
s1.v1 AS v1, s1.v2 AS v2, s3.v3 AS v3, s3.v4
AS v4
WHERE (v2)-[:b]->(v3)
WITH
pas, nas, pbs, nbs, pcs, ncs, pds, nds,
nd_v1s, nd_v4s, s1s, s2s, s3s, s4s,
collect([v1, v2, v3, v4]) AS r1
// r2: -(𝑎⋉∇𝑑) ⋈ 𝑏 ⋈ (Δ𝑐⋉∇𝑑)
UNWIND s1s AS s1
UNWIND s4s AS s4
WITH
pas, nas, pbs, nbs, pcs, ncs, pds, nds,
nd_v1s, nd_v4s, s1s, s2s, s3s, s4s,
r1,
s1.v1 AS v1, s1.v2 AS v2, s4.v3 AS v3, s4.v4
AS v4
WHERE (v2)-[:b]->(v3)
WITH
pas, nas, pbs, nbs, pcs, ncs, pds, nds,
nd_v1s, nd_v4s, s1s, s2s, s3s, s4s,
r1,
collect([v1, v2, v3, v4]) AS r2
// r3: -(𝑎⋉∇𝑑) ⋈ Δ𝑏 ⋈ (𝑐⋉∇𝑑)
UNWIND s1s AS s1
UNWIND s3s AS s3
WITH
pas, nas, pbs, nbs, pcs, ncs, pds, nds,
nd_v1s, nd_v4s, s1s, s2s, s3s, s4s,
r1, r2,
s1.v1 AS v1, s1.v2 AS v2, s3.v3 AS v3, s3.v4
AS v4
MATCH (v2)-[b:b]->(v3)
WHERE b IN pbs
WITH
pas, nas, pbs, nbs, pcs, ncs, pds, nds,
nd_v1s, nd_v4s, s1s, s2s, s3s, s4s,
r1, r2,
collect([v1, v2, v3, v4]) AS r3
// r4: -(Δ𝑎⋉∇𝑑) ⋈ 𝑏 ⋈ (𝑐⋉∇𝑑)
UNWIND s2s AS s2
UNWIND s3s AS s3
WITH
pas, nas, pbs, nbs, pcs, ncs, pds, nds,
nd_v1s, nd_v4s, s1s, s2s, s3s, s4s,
r1, r2, r3,
s2.v1 AS v1, s2.v2 AS v2, s3.v3 AS v3, s3.v4
AS v4
MATCH (v2)-[b:b]->(v3)
WHERE b IN pbs
WITH
pas, nas, pbs, nbs, pcs, ncs, pds, nds,
nd_v1s, nd_v4s, s1s, s2s, s3s, s4s,
r1, r2, r3,
collect([v1, v2, v3, v4]) AS r4
// r5: 𝑎 ⋈ 𝑏 ⋈ Δ𝑐 ̅⋉ 𝑑
UNWIND pcs AS pc
MATCH (v1)-[:a]->(v2)-[:b]->(v3)-[:pc]->(v4)
WHERE NOT (v1)-[:d]->(v4)
WITH
pas, nas, pbs, nbs, pcs, ncs, pds, nds,
nd_v1s, nd_v4s, s1s, s2s, s3s, s4s,
r1, r2, r3, r4,
collect([v1, v2, v3, v4]) AS r5
// r6: 𝑎 ⋈ Δ𝑏 ⋈ 𝑐 ̅⋉ 𝑑
UNWIND pbs AS pb
MATCH (v1)-[:a]->(v2)-[:pb]->(v3)-[:c]->(v4)
WHERE NOT (v1)-[:d]->(v4)
WITH
pas, nas, pbs, nbs, pcs, ncs, pds, nds,
nd_v1s, nd_v4s, s1s, s2s, s3s, s4s,
r1, r2, r3, r4, r5,
collect([v1, v2, v3, v4]) AS r6
// r7: Δ𝑎 ⋈ 𝑏 ⋈ 𝑐 ̅⋉ 𝑑
UNWIND pas AS pa
MATCH (v1)-[:pa]->(v2)-[:b]->(v3)-[:c]->(v4)
WHERE NOT (v1)-[:d]->(v4)
WITH
pas, nas, pbs, nbs, pcs, ncs, pds, nds,
nd_v1s, nd_v4s, s1s, s2s, s3s, s4s,
r1, r2, r3, r4, r5, r6,
collect([v1, v2, v3, v4]) AS r7
WITH
r1 + r5 + r6 + r7 AS rp,
r2 + r3 + r4 AS rn
RETURN
[r IN rp WHERE NOT r IN rn] AS results
POSITIVE DELTA QUERY FOR 𝑎 ⋈ 𝑏 ⋈ 𝑐 ഥ⋉ 𝑑
43. POSITIVE DELTA QUERY FOR 𝑎 ⋈ 𝑏 ⋈ 𝑐 ഥ⋉ 𝑑
Workaround: knowing the change workload helps
o Only consider changes in Δ𝑑 and 𝛻𝑑
o Query is cleaner and much more efficient
UNWIND $nds AS nd
WITH
startNode(nd) AS v1,
endNode(nd) AS v4
MATCH (v1)-[:a]->(v2)-[:b]->(v3)-[:c]->(v4)
RETURN v1, v2, v3, v4
This can outperform recomputing the query from scratch.
44. LIST OPERATIONS IN CYPHER
Instead of subqueries, use chained queries and combine lists.
WITH [1, 1, 2, 2, 3] AS xs
UNWIND xs AS x
RETURN
collect(DISTINCT x) AS unique
WITH [1, 2, 3] AS xs, [2] AS ys
RETURN
xs + ys AS append,
[x IN xs WHERE NOT x IN ys] AS subtraction,
[x IN xs WHERE x IN ys] AS intersection
WITH [1, 1, 2, 2, 3] AS xs
RETURN
reduce(acc = [], x in xs |
acc + CASE x IN acc
WHEN false THEN [x]
ELSE []
END) AS unique
Get unique list in openCypher
45. DELTA QUERIES IN CYPHER
Delta queries are complex. Features that would be nice:
Subqueries //pattern comprehensions go some length
Named subqueries //help reusability
Subtracting lists //related: CIR-2017-180
Use collection elements or function results for matching
These are probably too much to ask.
-> recommended approach: compile directly to query plans.
MATCH (n)
WITH collect(n) AS ns
MATCH (ns[0])
RETURN *
MATCH (n)
WITH collect(n) AS ns
WITH ns[0] AS x
MATCH (x)
RETURN *
46. CHALLENGES FOR PROPERTY GRAPH QUERIES
Data model
NF2 (Non-First Normal Form): maps, lists
No schema (schema-optionality)
Graph structure
Queries
Nulls, antijoins and left outerjoins
Updates on property values
Aggregates on aggregates, non-distributive functions
Ordering + skip/limit
Reachability queries
47. CHALLENGES FOR PROPERTY GRAPH QUERIES
Data model
NF2
No schema
Graph
Queries
Nulls
Updates
Aggregates
Ordering
Reachability
A. Gupta, I. S. Mumick:
Materialized Views.
MIT Press, 1999
R. Chirkova, J. Yang:
Materialized Views.
Foundations and Trends
in Databases, 2012
Decades of research -> 2 long surveys
49. DBTOASTER
Shows the scale of the problem
Relational data model and SQL queries
R&D for ~5 years @ EPFL, Johns Hopkins, Cornell, etc.
Approach
o Queries over an algebraic ring
o Higher-order recursive IVM
Compiler in OCaml
Backend with code generation for C++, Scala/Spark
Christoph Koch et al.:
DBToaster: Higher-order Delta Processing for Dynamic, Frequently Fresh Views.
VLDB Journal 2014
50. FUTURE DIRECTIONS
Work out derivation rules for Expand/VarExpand, …
Automate delta query derivation
Integrate to Neo4j
Run performance experiments
o Train Benchmark (set semantics)
o LDBC Social Network Benchmark’s BI workload (bag semantics)
52. LDBC BENCHMARKS
Social Network Benchmark
o Business Intelligence workload published
o openCypher reference implementation
o Next goal: full conference paper
Graphalytics
o Competition is online at graphalytics.org
o Neo4j implementation (using the Graph Algorithms library) WIP
graphalytics-platforms-neo4j/pull/6
Gábor Szárnyas, Arnau Prat-Pérez, Alex Averbuch, József Marton et al.:
An early look at the LDBC Social Network Benchmark’s BI Workload.
GRADES-NDA at SIGMOD, 2018
ldbc/ldbc_snb_implementations
53. GRAPH ANALYTICS ON THE PANAMA PAPERS
Network science approach: multidimensional graph metrics
from social network analysis, biology, physics, etc.
Our work originally targeted software and system models.
Progress in 2018
o Q1: implemented adapters for Neo4j and CSV
o Q2 goal: analyse Panama papers and using metrics
Gábor Szárnyas, Zsolt Kővári, Ágnes Salánki, Dániel Varró:
Towards the Characterization of Realistic Models:
Evaluation of Multidisciplinary Graph Metrics,
MODELS 2016 ftsrg/model-analyzer
54. MAPPING CYPHER TO SQL
Evaluate graph queries in an RDB - similar to ORM
Approaches
o Cytosm: Cypher to SQL Mapper / gTop: graph topology
o GraphGen – extracting graphs from RDBs
o Ongoing work to map TCK to SQLite
B. A. Steer, A. Alnaimi, M. Lotz, F. Cuadrado, L. Vaquero, J. Varvenne:
Cytosm: Declarative Property Graph Queries Without Data Migration.
GRADES 2017
K. Xirogiannopoulos, V. Srinivas, A. Deshpande:
GraphGen: Adaptive Graph Processing using Relational Databases.
GRADES 2017
cytosm/cytosm
KonstantinosX/graphgen-project
55. NEO4J APOC LIBRARY
CSV loader that follows the schema of the neo4j-import tool
Goal
o Use headers to generate LOAD CSV commands.
o 1st pass: CALL apoc.import.csv.node(file, labels, …)
o 2nd pass: CALL apoc.import.csv.relationship(file, type, …)
Result
o Many corner cases -> ~700 LOC + tests
o Covers most use cases, but is very slow
o APOC PR pending
neo4j-apoc-procedures/pull/581 neo4j-documentation/pull/121