Presentation given at nosql east 2009 in Atlanta. Introduces the NOSQL space by offering a framework for categorization and discusses the benefits of graph databases. Oh, and also includes some tongue-in-cheek party poopers about sucky things in the NOSQL space.
This Presentation is about NoSQL which means Not Only SQL. This presentation covers the aspects of using NoSQL for Big Data and the differences from RDBMS.
Optimize the performance, cost, and value of databases.pptxIDERA Software
Today’s businesses run on data, making it essential for them to access data quickly and easily. This requirement means that databases must run efficiently at all times but keeping a database performing at its best remains a challenging task. Fortunately, database administrators (DBAs) can adopt many practices to achieve this goal, thus saving time and money.
This Presentation is about NoSQL which means Not Only SQL. This presentation covers the aspects of using NoSQL for Big Data and the differences from RDBMS.
Optimize the performance, cost, and value of databases.pptxIDERA Software
Today’s businesses run on data, making it essential for them to access data quickly and easily. This requirement means that databases must run efficiently at all times but keeping a database performing at its best remains a challenging task. Fortunately, database administrators (DBAs) can adopt many practices to achieve this goal, thus saving time and money.
Hadoop 3.0 has been years in the making, and now it's finally arriving. Andrew Wang and Daniel Templeton offer an overview of new features, including HDFS erasure coding, YARN Timeline Service v2, YARN federation, and much more, and discuss current release management status and community testing efforts dedicated to making Hadoop 3.0 the best Hadoop major release yet.
Apache Spark is a In Memory Data Processing Solution that can work with existing data source like HDFS and can make use of your existing computation infrastructure like YARN/Mesos etc. This talk will cover a basic introduction of Apache Spark with its various components like MLib, Shark, GrpahX and with few examples.
"The common use cases of Spark SQL include ad hoc analysis, logical warehouse, query federation, and ETL processing. Spark SQL also powers the other Spark libraries, including structured streaming for stream processing, MLlib for machine learning, and GraphFrame for graph-parallel computation. For boosting the speed of your Spark applications, you can perform the optimization efforts on the queries prior employing to the production systems. Spark query plans and Spark UIs provide you insight on the performance of your queries. This talk discloses how to read and tune the query plans for enhanced performance. It will also cover the major related features in the recent and upcoming releases of Apache Spark.
"
Spark SQL Tutorial | Spark Tutorial for Beginners | Apache Spark Training | E...Edureka!
This Edureka Spark SQL Tutorial will help you to understand how Apache Spark offers SQL power in real-time. This tutorial also demonstrates an use case on Stock Market Analysis using Spark SQL. Below are the topics covered in this tutorial:
1) Limitations of Apache Hive
2) Spark SQL Advantages Over Hive
3) Spark SQL Success Story
4) Spark SQL Features
5) Architecture of Spark SQL
6) Spark SQL Libraries
7) Querying Using Spark SQL
8) Demo: Stock Market Analysis With Spark SQL
This slide deck is used as an introduction to the internals of Apache Spark, as part of the Distributed Systems and Cloud Computing course I hold at Eurecom.
Course website:
http://michiard.github.io/DISC-CLOUD-COURSE/
Sources available here:
https://github.com/michiard/DISC-CLOUD-COURSE
Data Visualization Techniques in Power BIAngel Abundez
A progression from fundamental charts to more advanced ways to look at data. We end with Custom Visuals and R Visuals that extend this visualization platform.
Data Quality With or Without Apache Spark and Its EcosystemDatabricks
Few solutions exist in the open-source community either in the form of libraries or complete stand-alone platforms, which can be used to assure a certain data quality, especially when continuous imports happen. Organisations may consider picking up one of the available options – Apache Griffin, Deequ, DDQ and Great Expectations. In this presentation we’ll compare these different open-source products across different dimensions, like maturity, documentation, extensibility, features like data profiling and anomaly detection.
Data Lakehouse, Data Mesh, and Data Fabric (r1)James Serra
So many buzzwords of late: Data Lakehouse, Data Mesh, and Data Fabric. What do all these terms mean and how do they compare to a data warehouse? In this session I’ll cover all of them in detail and compare the pros and cons of each. I’ll include use cases so you can see what approach will work best for your big data needs.
Hadoop 3.0 has been years in the making, and now it's finally arriving. Andrew Wang and Daniel Templeton offer an overview of new features, including HDFS erasure coding, YARN Timeline Service v2, YARN federation, and much more, and discuss current release management status and community testing efforts dedicated to making Hadoop 3.0 the best Hadoop major release yet.
Apache Spark is a In Memory Data Processing Solution that can work with existing data source like HDFS and can make use of your existing computation infrastructure like YARN/Mesos etc. This talk will cover a basic introduction of Apache Spark with its various components like MLib, Shark, GrpahX and with few examples.
"The common use cases of Spark SQL include ad hoc analysis, logical warehouse, query federation, and ETL processing. Spark SQL also powers the other Spark libraries, including structured streaming for stream processing, MLlib for machine learning, and GraphFrame for graph-parallel computation. For boosting the speed of your Spark applications, you can perform the optimization efforts on the queries prior employing to the production systems. Spark query plans and Spark UIs provide you insight on the performance of your queries. This talk discloses how to read and tune the query plans for enhanced performance. It will also cover the major related features in the recent and upcoming releases of Apache Spark.
"
Spark SQL Tutorial | Spark Tutorial for Beginners | Apache Spark Training | E...Edureka!
This Edureka Spark SQL Tutorial will help you to understand how Apache Spark offers SQL power in real-time. This tutorial also demonstrates an use case on Stock Market Analysis using Spark SQL. Below are the topics covered in this tutorial:
1) Limitations of Apache Hive
2) Spark SQL Advantages Over Hive
3) Spark SQL Success Story
4) Spark SQL Features
5) Architecture of Spark SQL
6) Spark SQL Libraries
7) Querying Using Spark SQL
8) Demo: Stock Market Analysis With Spark SQL
This slide deck is used as an introduction to the internals of Apache Spark, as part of the Distributed Systems and Cloud Computing course I hold at Eurecom.
Course website:
http://michiard.github.io/DISC-CLOUD-COURSE/
Sources available here:
https://github.com/michiard/DISC-CLOUD-COURSE
Data Visualization Techniques in Power BIAngel Abundez
A progression from fundamental charts to more advanced ways to look at data. We end with Custom Visuals and R Visuals that extend this visualization platform.
Data Quality With or Without Apache Spark and Its EcosystemDatabricks
Few solutions exist in the open-source community either in the form of libraries or complete stand-alone platforms, which can be used to assure a certain data quality, especially when continuous imports happen. Organisations may consider picking up one of the available options – Apache Griffin, Deequ, DDQ and Great Expectations. In this presentation we’ll compare these different open-source products across different dimensions, like maturity, documentation, extensibility, features like data profiling and anomaly detection.
Data Lakehouse, Data Mesh, and Data Fabric (r1)James Serra
So many buzzwords of late: Data Lakehouse, Data Mesh, and Data Fabric. What do all these terms mean and how do they compare to a data warehouse? In this session I’ll cover all of them in detail and compare the pros and cons of each. I’ll include use cases so you can see what approach will work best for your big data needs.
Graph Search: The Power of Connected DataCodemotion
Today’s complex data is big, variably-structured and densely connected. In this session we’ll look at how size, structure and connectedness have converged to change the way we work with data. We’ll then go on to look at some of the new opportunities for creating end-user value that have emerged in a world of connected data, illustrated with graph search examples implemented using the Neo4j graph database.
OrientDB vs Neo4j - and an introduction to NoSQL databasesCurtis Mosters
NoSQL databases are a good alternative to common SQL technologies. Here you get an introduction and comparison of SQL vs NoSQL. Furthermore we have a look on Graph databases and especially OrientDB vs Neo4j.
Graph Databases - Where Do We Do the Modeling Part?DATAVERSITY
Graph processing and graph databases have been with us for a while. However, since their physical implementations are the same for every database in production (Node connected to node, or triplets), there's a perception that data modeling (and data modelers) have no role on projects where graph databases are used.
This month we'll talk about where graph databases are a best fit in a modern data architecture and where data models add value.
Graph databases are used to represent graph structures with nodes, edges and properties. Neo4j, an open-source graph database is reliable and fast for managing and querying highly connected data. Will explore how to install and configure, create nodes and relationships, query with the Cypher Query Language, importing data and using Neo4j in concert with SQL Server... Providing answers and insight with visual diagrams about connected data that you have in your SQL Server Databases!
This tutorial will provide you with a basic understanding of graph database technology and the ability to quickly begin development of a graph database application. You will have the capability to recognize graph-based problems and present the benefits of using graph technology for problem resolution.
The tutorial will give you an understanding of:
• Graph theory - origins and concepts
• Benefits of graph databases
• Different types of graph databases
• Typical graph database API
• Programming basics
• Use cases
Bring your laptops for a hands-on opportunity to practice some sample codes. A basic understanding of Java programming is a recommended prerequisite to understand this course. This session is led by the InfiniteGraph technical team and the demonstration code will be drawn from InfiniteGraph examples, however the broader educational presentation is product-neutral and not a commercial presentation of their products.
To participate in the hands-on portion of the graph tutorial users must have:
• Java programming experience
• Java Developer Kit (JDK)
• Current InfiniteGraph installed on laptop. (To download visit www.objectivity.com/infinitegraph)
• HelloGraph test – Upon installing IG, run HelloGraph to test the install. (HelloGraph can be found online at http://wiki.infinitegraph.com/2.1/w/index.php?title=Download_Sample_Code)
Leon Guzenda was one of the founding members of Objectivity in 1988 and one of the original architects of Objectivity/DB. He currently works with Objectivity's major customers to help them effectively develop and deploy complex applications and systems that use the industry's highest-performing, most reliable DBMS technology, Objectivity/DB. He also liaises with technology partners and industry groups to help ensure that Objectivity/DB remains at the forefront of database and distributed computing technology. Leon has more than 35 years experience in the software industry. At Automation Technology Products, he managed the development of the ODBMS for the Cimplex solid modeling and numerical control system. Before that, he was Principal Project Director for International Computers Ltd. in the United Kingdom, delivering major projects for NATO and leading multinationals. He was also design and development manager for ICL's 2900 IDMS product. He spent the first 7 years of his career working in defense and government systems. Leon has a B.S. degree in Electronic Engineering from the University of Wales.
Graph Database Management Systems provide an effective
and efficient solution to data storage in current scenarios
where data are more and more connected, graph models are
widely used, and systems need to scale to large data sets.
In this framework, the conversion of the persistent layer of
an application from a relational to a graph data store can
be convenient but it is usually an hard task for database
administrators. In this paper we propose a methodology
to convert a relational to a graph database by exploiting
the schema and the constraints of the source. The approach
supports the translation of conjunctive SQL queries over the
source into graph traversal operations over the target. We
provide experimental results that show the feasibility of our
solution and the efficiency of query answering over the target
database.
Intro to Graph Databases Using Tinkerpop, TitanDB, and GremlinCaleb Jones
A quick overview of the history, motivation, and uses of graph modeling and graph databases in various industries. Covers a brief introduction to graph databases with an emphasis on the Tinkerpop stack and Gremlin query language. These concepts are then solidified through a hands-on lab modeling a blog engine using Titan and Gremlin.
See more at http://allthingsgraphed.com.
PFE :: Application de gestion des dus d'enseignementNassim Bahri
Mon mémoire de PFE pour le projet Conception et développement d'une
application de gestion des dus d'enseignement pour l'Ecole Supérieure d'Economie Numérique Manouba. Le but de cette application est de centraliser les données de l'école d'une part (les parcours, les unités d'enseignement,...) et de faciliter l'affectation des charges horaire d'enseignement d'un autre part. Ce projet à été réalisé en adoptant Scrum comme étant une méthodologie de conception et de gestion de projet.
Relational databases were conceived to digitize paper forms and automate well-structured business processes, and still have their uses. But RDBMS cannot model or store data and its relationships without complexity, which means performance degrades with the increasing number and levels of data relationships and data size. Additionally, new types of data and data relationships require schema redesign that increases time to market.
A native graph database like Neo4j naturally stores, manages, analyzes, and uses data within the context of connections meaning Neo4j provides faster query performance and vastly improved flexibility in handling complex hierarchies than SQL.
This webinar explains why companies are shifting away from RDBMS towards graphs to unlock the business value in their data relationships.
Spring Data Graph is an integration library for the open source graph database Neo4j and has been around for over a year, evolving from its infancy as brainchild of Rod Johnson and Emil Eifrem. It supports transparent AspectJ based POJO to Graph Mapping, a Neo4jTemplate API and extensive support for Spring Data Repositories. It can work with an embedded graph database or with the standalone Neo4j Server.
The session starts with a short introduction to graph databases. Following that, the different approaches using Spring Data Graph are explored in the Cineasts.net web-app, a social movie database which is also the application of the tutorial in the Spring Data Graph Guidebook. The session will also cover creating a green-field project using the Spring Roo Addon for Spring Data Graph and deploying the App to CloudFoundry.
Django and Neo4j - Domain modeling that kicks assTobias Lindaaker
Presentation about using Neo4j from Django presented at OSCON 2010, Portland OR.
Sample code is available at: https://svn.neo4j.org/components/neo4j.py/trunk/src/examples/python/djangosites/blog/
A presentation of the Neo4j graph database given at QCon SF 2008. It describes why relational databases are increasingly unfit for many applications today and why graphs may be a good fit. It also covers the fundamentals of how to program with Neo4j.
using Spring and MongoDB on Cloud FoundryJoshua Long
This talk introduces how to build MongoDB applications with Spring Data MongoDB on Cloud Foundry. Spring Data provides rich support for easily building applications that work on multiple data stores.
While mathematicians have used graph theory since the 18th century to solve problems, the software patterns for graph data are new to most developers. To enable "mass adoption" of graph technology, we need to establish the right abstractions, access APIs, and data models.
RDF triples, while of paramount importance in establishing RDF graph semantics, are a low-level abstraction, much like using assembly language. For practical and productive “graph programming” we need something different.
Similarly, existing declarative graph query languages (such as SPARQL and Cypher) are not always the best way to access graph data, and sometimes you need a simpler interface (e.g., GraphQL), or even a different approach altogether (e.g., imperative traversals such as with Gremlin).
Ora Lassila is a Principal Graph Technologist in the Amazon Neptune graph database group. He has a long experience with graphs, graph databases, ontologies, and knowledge representation. He was a co-author of the original RDF specification as well as a co-author of the seminal article on the Semantic Web.
Startups in Sweden vs Startups in Silicon Valley, 2015 editionEmil Eifrem
Differences between running a startup in Sweden and a startup in Silicon Valley. Bonus: How Neo Technology (Neo4j) uses aspects of Scandinavian culture as a competitive advantage. Presented at the Nordic themed Monki Gräs in London, Jan of 2015.
Btw, we at Neo4j are hiring: http://neo4j.com/jobs/ :)
An Overview of the Emerging Graph Landscape (Oct 2013)Emil Eifrem
Recent years have seen an explosion of technologies for managing, processing and analyzing graphs, ranging from community projects like Apache Giraph, to vendor led products such as Neo4j and spin outs from established companies like Twitter’s FlockDB. The sheer number of technologies makes it difficult to keep track of these tools and what sets them apart, even for those of us who are active in the space!
But all graph technologies are not created equal. This session will provide a high level framework for making sense of the emerging graph landscape. It will describe the three dominant graph data models today, define top level categories like graph compute engines (Graphlab, Giraph, Pegasus, YarcData, etc) and graph databases (Neo4j, FlockDB, OrientDB, etc) and discuss common characteristics and important properties of each category.
Startups in Sweden vs Startups in Silicon ValleyEmil Eifrem
Differences between running a startup in Sweden and a startup in Silicon Valley. Presented at Stanford's "European Entrepreneurship & Innovation" (http://www.europeanentrepreneursatstanford.com) in Jan of 2012.
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
Generative AI Deep Dive: Advancing from Proof of Concept to ProductionAggregage
Join Maher Hanafi, VP of Engineering at Betterworks, in this new session where he'll share a practical framework to transform Gen AI prototypes into impactful products! He'll delve into the complexities of data collection and management, model selection and optimization, and ensuring security, scalability, and responsible use.
A tale of scale & speed: How the US Navy is enabling software delivery from l...sonjaschweigert1
Rapid and secure feature delivery is a goal across every application team and every branch of the DoD. The Navy’s DevSecOps platform, Party Barge, has achieved:
- Reduction in onboarding time from 5 weeks to 1 day
- Improved developer experience and productivity through actionable findings and reduction of false positives
- Maintenance of superior security standards and inherent policy enforcement with Authorization to Operate (ATO)
Development teams can ship efficiently and ensure applications are cyber ready for Navy Authorizing Officials (AOs). In this webinar, Sigma Defense and Anchore will give attendees a look behind the scenes and demo secure pipeline automation and security artifacts that speed up application ATO and time to production.
We will cover:
- How to remove silos in DevSecOps
- How to build efficient development pipeline roles and component templates
- How to deliver security artifacts that matter for ATO’s (SBOMs, vulnerability reports, and policy evidence)
- How to streamline operations with automated policy checks on container images
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Albert Hoitingh
In this session I delve into the encryption technology used in Microsoft 365 and Microsoft Purview. Including the concepts of Customer Key and Double Key Encryption.
Securing your Kubernetes cluster_ a step-by-step guide to success !KatiaHIMEUR1
Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster.
However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks.
In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...SOFTTECHHUB
The choice of an operating system plays a pivotal role in shaping our computing experience. For decades, Microsoft's Windows has dominated the market, offering a familiar and widely adopted platform for personal and professional use. However, as technological advancements continue to push the boundaries of innovation, alternative operating systems have emerged, challenging the status quo and offering users a fresh perspective on computing.
One such alternative that has garnered significant attention and acclaim is Nitrux Linux 3.5.0, a sleek, powerful, and user-friendly Linux distribution that promises to redefine the way we interact with our devices. With its focus on performance, security, and customization, Nitrux Linux presents a compelling case for those seeking to break free from the constraints of proprietary software and embrace the freedom and flexibility of open-source computing.
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
Removing Uninteresting Bytes in Software FuzzingAftab Hussain
Imagine a world where software fuzzing, the process of mutating bytes in test seeds to uncover hidden and erroneous program behaviors, becomes faster and more effective. A lot depends on the initial seeds, which can significantly dictate the trajectory of a fuzzing campaign, particularly in terms of how long it takes to uncover interesting behaviour in your code. We introduce DIAR, a technique designed to speedup fuzzing campaigns by pinpointing and eliminating those uninteresting bytes in the seeds. Picture this: instead of wasting valuable resources on meaningless mutations in large, bloated seeds, DIAR removes the unnecessary bytes, streamlining the entire process.
In this work, we equipped AFL, a popular fuzzer, with DIAR and examined two critical Linux libraries -- Libxml's xmllint, a tool for parsing xml documents, and Binutil's readelf, an essential debugging and security analysis command-line tool used to display detailed information about ELF (Executable and Linkable Format). Our preliminary results show that AFL+DIAR does not only discover new paths more quickly but also achieves higher coverage overall. This work thus showcases how starting with lean and optimized seeds can lead to faster, more comprehensive fuzzing campaigns -- and DIAR helps you find such seeds.
- These are slides of the talk given at IEEE International Conference on Software Testing Verification and Validation Workshop, ICSTW 2022.
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdfPaige Cruz
Monitoring and observability aren’t traditionally found in software curriculums and many of us cobble this knowledge together from whatever vendor or ecosystem we were first introduced to and whatever is a part of your current company’s observability stack.
While the dev and ops silo continues to crumble….many organizations still relegate monitoring & observability as the purview of ops, infra and SRE teams. This is a mistake - achieving a highly observable system requires collaboration up and down the stack.
I, a former op, would like to extend an invitation to all application developers to join the observability party will share these foundational concepts to build on:
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...UiPathCommunity
💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™:
See how to accelerate model training and optimize model performance with active learning
Learn about the latest enhancements to out-of-the-box document processing – with little to no training required
Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages
This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath.
Speakers:
👨🏫 Andras Palfi, Senior Product Manager, UiPath
👩🏫 Lenka Dulovicova, Product Program Manager, UiPath
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
5. Trend 2: connectedness
Giant
Global
Information connectivity
Graph
(GGG)
Ontologies
RDF
Folksonomies
Tagging
User-
Wikis
generated
content
Blogs
RSS
Hypertext
Text
documents web 1.0 web 2.0 “web 3.0”
1990 2000 2010 2020
6. Trend 3: semi-structure
Individualization of content!
In the salary lists of the 1970s, all elements had
exactly one job
In the salary lists of the 2000s, we need 5 job
columns! Or 8? Or 15?
Trend accelerated by the decentralization of
content generation that is the hallmark of the age
of participation (“web 2.0”)
7. Aside: RDBMS performance
Relational database
Salary List
Performance
Majority of
Webapps
Social network
Semantic
}
Trading
custom
Data complexity
14. Four (emerging) NOSQL categories
Key-value stores
Based on DHTs / Amazon's Dynamo paper
Data model: (global) collection of K-V pairs
Example: Dynomite, Voldemort, Tokyo
BigTable clones
Based on Google's BigTable paper
Data model: big table, column families
Example: Hbase, Hypertable
15. Four (emerging) NOSQL categories
Document databases
Inspired by Lotus Notes
Data model: collections of K-V collections
Example: CouchDB, MongoDB
Graph databases
Inspired by Euler & graph theory
Data model: nodes, rels, K-V on both
Example: AllegroGraph, VertexDB, Neo4j
17. NOSQL data models
Size
Key-value stores
Bigtable clones
Document
databases
Graph databases
(This is still billions of
90% nodes & relationships)
of
use
cases
Complexity
19. The Graph DB model: representation
Core abstractions: name = “Emil”
age = 29
Nodes sex = “yes”
Relationships between nodes
1 2
Properties on both
type = KNOWS
time = 4 years 3
type = car
vendor = “SAAB”
model = “95 Aero”
20. Example: The Matrix
name = “The Architect”
name = “Morpheus”
rank = “Captain”
name = “Thomas Anderson”
occupation = “Total badass” 42
age = 29
disclosure = public
KNOWS KNOWS CODED_BY
1 KN O
7 3 WS
13
S
KN name = “Cypher”
KNOW
OW last name = “Reagan”
S
name = “Agent Smith”
disclosure = secret version = 1.0b
age = 3 days age = 6 months language = C++
2
name = “Trinity”
21. Code (1): Building a node space
NeoService neo = ... // Get factory
// Create Thomas 'Neo' Anderson
Node mrAnderson = neo.createNode();
mrAnderson.setProperty( "name", "Thomas Anderson" );
mrAnderson.setProperty( "age", 29 );
// Create Morpheus
Node morpheus = neo.createNode();
morpheus.setProperty( "name", "Morpheus" );
morpheus.setProperty( "rank", "Captain" );
morpheus.setProperty( "occupation", "Total bad ass" );
// Create a relationship representing that they know each other
mrAnderson.createRelationshipTo( morpheus, RelTypes.KNOWS );
// ...create Trinity, Cypher, Agent Smith, Architect similarly
22. Code (1): Building a node space
NeoService neo = ... // Get factory
Transaction tx = neo.beginTx();
// Create Thomas 'Neo' Anderson
Node mrAnderson = neo.createNode();
mrAnderson.setProperty( "name", "Thomas Anderson" );
mrAnderson.setProperty( "age", 29 );
// Create Morpheus
Node morpheus = neo.createNode();
morpheus.setProperty( "name", "Morpheus" );
morpheus.setProperty( "rank", "Captain" );
morpheus.setProperty( "occupation", "Total bad ass" );
// Create a relationship representing that they know each other
mrAnderson.createRelationshipTo( morpheus, RelTypes.KNOWS );
// ...create Trinity, Cypher, Agent Smith, Architect similarly
tx.commit();
23. Code (1b): Defining RelationshipTypes
// In package org.neo4j.api.core
public interface RelationshipType
{
String name();
}
// In package org.yourdomain.yourapp
// Example on how to roll dynamic RelationshipTypes
class MyDynamicRelType implements RelationshipType
{
private final String name;
MyDynamicRelType( String name ){ this.name = name; }
public String name() { return this.name; }
}
// Example on how to kick it, static-RelationshipType-like
enum MyStaticRelTypes implements RelationshipType
{
KNOWS,
WORKS_FOR,
}
26. The Graph DB model: traversal
Traverser framework for name = “Emil”
high-performance traversing age = 29
sex = “yes”
across the node space
1 2
type = KNOWS
time = 4 years 3
type = car
vendor = “SAAB”
model = “95 Aero”
27. Example: Mr Anderson’s friends
name = “The Architect”
name = “Morpheus”
rank = “Captain”
name = “Thomas Anderson”
occupation = “Total badass” 42
age = 29
disclosure = public
KNOWS KNOWS CODED_BY
1 KN O
7 3 WS
13
S
KN name = “Cypher”
KNOW
OW last name = “Reagan”
S
name = “Agent Smith”
disclosure = secret version = 1.0b
age = 3 days age = 6 months language = C++
2
name = “Trinity”
28. Code (2): Traversing a node space
// Instantiate a traverser that returns Mr Anderson's friends
Traverser friendsTraverser = mrAnderson.traverse(
Traverser.Order.BREADTH_FIRST,
StopEvaluator.END_OF_GRAPH,
ReturnableEvaluator.ALL_BUT_START_NODE,
RelTypes.KNOWS,
Direction.OUTGOING );
// Traverse the node space and print out the result
System.out.println( "Mr Anderson's friends:" );
for ( Node friend : friendsTraverser )
{
System.out.printf( "At depth %d => %s%n",
friendsTraverser.currentPosition().getDepth(),
friend.getProperty( "name" ) );
}
29. name = “The Architect”
name = “Morpheus”
rank = “Captain”
name = “Thomas Anderson”
occupation = “Total badass” 42
age = 29
disclosure = public
KNOWS KNOWS CODED_BY
1 KN O
7 3 WS
13
S
KN name = “Cypher”
KNOW
OW last name = “Reagan”
S
name = “Agent Smith”
disclosure = secret version = 1.0b
age = 3 days age = 6 months language = C++
2
name = “Trinity”
$ bin/start-neo-example
Mr Anderson's friends:
At depth 1 => Morpheus
friendsTraverser = mrAnderson.traverse(
Traverser.Order.BREADTH_FIRST, At depth 1 => Trinity
StopEvaluator.END_OF_GRAPH, At depth 2 => Cypher
ReturnableEvaluator.ALL_BUT_START_NODE,
RelTypes.KNOWS,
At depth 3 => Agent Smith
Direction.OUTGOING ); $
30. Example: Friends in love?
name = “The Architect”
name = “Morpheus”
rank = “Captain”
name = “Thomas Anderson”
occupation = “Total badass” 42
age = 29
disclosure = public
KNOWS KNOWS CODED_BY
1 7 3 KN O
WS
13
S
KN
KNOW
name = “Cypher”
OW last name = “Reagan”
S
name = “Agent Smith”
LO disclosure = secret version = 1.0b
VE age = 6 months language = C++
S
2
name = “Trinity”
31. Code (3a): Custom traverser
// Create a traverser that returns all “friends in love”
Traverser loveTraverser = mrAnderson.traverse(
Traverser.Order.BREADTH_FIRST,
StopEvaluator.END_OF_GRAPH,
new ReturnableEvaluator()
{
public boolean isReturnableNode( TraversalPosition pos )
{
return pos.currentNode().hasRelationship(
RelTypes.LOVES, Direction.OUTGOING );
}
},
RelTypes.KNOWS,
Direction.OUTGOING );
32. Code (3a): Custom traverser
// Traverse the node space and print out the result
System.out.println( "Who’s a lover?" );
for ( Node person : loveTraverser )
{
System.out.printf( "At depth %d => %s%n",
loveTraverser.currentPosition().getDepth(),
person.getProperty( "name" ) );
}
33. name = “The Architect”
name = “Morpheus”
rank = “Captain”
name = “Thomas Anderson”
occupation = “Total badass” 42
age = 29
disclosure = public
KNOWS KNOWS KN O CODED_BY
1 7 3 WS
13
S
KN
KNOW
name = “Cypher”
OW last name = “Reagan”
S
name = “Agent Smith”
LO disclosure = secret version = 1.0b
VE age = 6 months language = C++
S 2
name = “Trinity”
$ bin/start-neo-example
new ReturnableEvaluator()
Who’s a lover?
{
public boolean isReturnableNode(
TraversalPosition pos)
At depth 1 => Trinity
{ $
return pos.currentNode().
hasRelationship( RelTypes.LOVES,
Direction.OUTGOING );
}
},
34. Bonus code: domain model
How do you implement your domain model?
Use the delegator pattern, i.e. every domain entity
wraps a Neo4j primitive:
// In package org.yourdomain.yourapp
class PersonImpl implements Person
{
private final Node underlyingNode;
PersonImpl( Node node ){ this.underlyingNode = node; }
public String getName()
{
return this.underlyingNode.getProperty( "name" );
}
public void setName( String name )
{
this.underlyingNode.setProperty( "name", name );
}
}
35. Domain layer frameworks
Qi4j (www.qi4j.org)
Framework for doing DDD in pure Java5
Defines Entities / Associations / Properties
Sound familiar? Nodes / Rel’s / Properties!
Neo4j is an “EntityStore” backend
NeoWeaver (http://components.neo4j.org/neo-weaver)
Weaves Neo4j-backed persistence into domain
objects in runtime (dynamic proxy / cglib based)
Veeeery alpha
36. Neo4j system characteristics
Disk-based
Native graph storage engine with custom binary
on-disk format
Transactional
JTA/JTS, XA, 2PC, Tx recovery, deadlock
detection, MVCC, etc
Scales up (what's the x and the y?)
Several billions of nodes/rels/props on single JVM
Robust
6+ years in 24/7 production
37. Social network pathExists()
12
~1k persons
3
7 1 Avg 50 friends per
person
pathExists(a, b) limit
36
41 77 depth 4
5
Two backends
Eliminate disk IO so
warm up caches
38. Social network pathExists()
2
Emil
1 5
7
Mike Kevin
3 John
Marcus
9 4
Bruce Leigh
# persons query time
Relational database 1 000 2 000 ms
Graph database (Neo4j) 1 000 2 ms
Graph database (Neo4j) 1 000 000 2 ms
39.
40.
41. Pros & Cons compared to RDBMS
+ No O/R impedance mismatch (whiteboard friendly)
+ Can easily evolve schemas
+ Can represent semi-structured info
+ Can represent graphs/networks (with performance)
- Lacks in tool and framework support
- Few other implementations => potential lock in
- No support for ad-hoc queries
+
42. More consequences
Ability to capture semi-structured information
=> allowing individualization of content
No predefined schema
=> easier to evolve model
=> can capture ad-hoc relationships
Can capture non-normative relations
=> easy to model specific links to specific sets
All state is kept in transactional memory
=> improves application concurrency
43. The Neo4j ecosystem
Neo4j is an embedded database
Tiny teeny lil jar file
Component ecosystem
index-util
neo-meta
neo-utils
pattern-match
sparql-engine
...
See http://components.neo4j.org
44. Language bindings
Neo4j.py – bindings for Jython and CPython
http://components.neo4j.org/neo4j.py
Neo4jrb – bindings for JRuby (incl RESTful API)
http://wiki.neo4j.org/content/Ruby
Clojure
http://wiki.neo4j.org/content/Clojure
Scala (incl RESTful API)
http://wiki.neo4j.org/content/Scala
… .NET? Erlang?
48. Scale out – replication
Rolling out Neo4j HA before end-of-year
Side note: ppl roll it today w/ REST frontends & onlinebackup
Master-slave replication, 1st configuration
MySQL style... ish
Except all instances can write, synchronously
between writing slave & master (strong consistency)
Updates are asynchronously propagated to the
other slaves (eventual consistency)
This can handle billions of entities...
… but not 100B
49. Scale out – partitioning
Sharding possible today
… but you have to do manual work
… just as with MySQL
Great option: shard on top of resilient, scalable
OSS app server , see: www.codecauldron.org
Transparent partitioning? Neo4j 2.0
100B? Easy to say. Sliiiiightly harder to do.
Fundamentals: BASE & eventual consistency
Generic clustering algorithm as base case, but
give lots of knobs for developers
50. How ego are you? (aka other impls?)
Franz’ AllegroGraph (http://agraph.franz.com)
Proprietary, Lisp, RDF-oriented but real graphdb
FreeBase graphd (http://bit.ly/13VITB)
In-house at Metaweb
Kloudshare (http://kloudshare.com)
Graph database in the cloud, still stealth mode
Google Pregel (http://bit.ly/dP9IP)
We are oh-so-secret
Some academic papers from ~10 years ago
G = {V, E} #FAIL
51. Conclusion
Graphs && Neo4j => teh awesome!
Available NOW under AGPLv3 / commercial license
AGPLv3: “if you’re open source, we’re open source”
If you have proprietary software? Must buy a
commercial license
But up to 1M primitives it’s free for all uses!
Download
http://neo4j.org
Feedback
http://lists.neo4j.org
53. Poop 1
Key-value stores?
=> the awesome
… if you have 1000s of BILLIONS records OR you
don't care about programmer productivity
What if you had no variables at all in your programs
except a single globally accessible hashtable?
Would your software be maintainable?
54. Poop 2
In a not-suck architecture...
… the only thing that makes sense is to have an
embedded database.
55. Poop 3
Exposing your data model on the wire is bad.
Period.
Adding a couple of buzzwords doesn't make it less
bad.
If it was bad with SQL-over-sockets (hint: it was)
then – surprise! – it's still bad even tho you use
Hype-compliant(tm) JSON-over-REST.
We don't want to couple everything to a specific
data model again!
56. Poop 4
In-memory database
What the hell?
That's an oxymoron!
Up next: ascii-only JPEG
Up next: loopback-only web server
If you're not durable, you're a cache!
If you happen to asynchronously spill over to disk,
you're a cache that asynchronously spills over to
disk.