Towards Temporal Graph Management and Analytics

•

1 like•77 views

LDBC council

Towards Temporal Graph Management and Analytics by Yinglong Xia (IBM) - 6th TUC Meeting (Barcelona)

IBM Research – Industries & Solutions – Business Solutions & Mathematical Sciences
Recent Updates on IBM System G
— GraphBIG and Temporal Data
Yinglong Xia
IBM T.J. Watson Research Center
Yorktown Heights, NY 10598

© 2014 International Business Machines Corporation
IBM Research – Industries & Solutions – Business Solutions & Mathematical Sciences
2
IBM T.J. Watson Research Center

© 2014 International Business Machines Corporation
IBM Research – Industries & Solutions – Business Solutions & Mathematical Sciences
3
Using LDBC-SNB for GraphBIG
• GraphBIG = Graph Benchmark Suite from IBM System G and GaTech HPArch
• A wide selection of workloads from both CPU and GPU
• Workload ranging from graph traversal to Gibbs Sampling on Bayesian Network
• Illustrating processor architecture impact using h/w performance counter
• Fix input data and implementation
• Show performance profiling at processor architecture level

© 2014 International Business Machines Corporation
IBM Research – Industries & Solutions – Business Solutions & Mathematical Sciences
4
Beyond the Benchmarking for Graph DBs
• Graph computing was barely considered in architecture design
• Increasing motivation due to popularity of graph analytics
• Impact of architecture requires fixed input data and analytic implementation

© 2014 International Business Machines Corporation
IBM Research – Industries & Solutions – Business Solutions & Mathematical Sciences
5

© 2014 International Business Machines Corporation
IBM Research – Industries & Solutions – Business Solutions & Mathematical Sciences
6
Demanding Graph
• Interactions of entities in many big data applications are naturally modeled by property graphs
• Evolution of graph structure and properties over time usually provides useful information, which needs
to be maintained for query or analytics
• Graph analytics market grows increasingly fast as well as the graph data size and complexity, but
near real time response is typically required
Xiaoyan Fu, Seok-Hee Hong, Nikola S. Nikolov, Xiaobin Shen, Ying Xin Wu and Kai Xu,
Visualization and Analysis of Email Networks, Proceedings of APVIS 2007, IEEE, pp.1-8, 2007.

© 2014 International Business Machines Corporation
IBM Research – Industries & Solutions – Business Solutions & Mathematical Sciences
7
Use Case: Forensic Analysis on Individual Status
• Recover the dynamics of individual status
• Evaluate status measures, anomalies, etc.
• Propagate known status measures
• Estimate labels for each person at each time stamp
• Aggregate the received measures
Chain Graph: A collection of graphs on  
contiguous time steps

© 2014 International Business Machines Corporation
IBM Research – Industries & Solutions – Business Solutions & Mathematical Sciences
8
Use Case: Bitemporal Data Exploration
• Support the valid dimension and the transaction dimension
• Audit trail of what you know and when did you know
• History of how history from a business perspective was stored in the
database
http://bitemporalmodeling.com/temporal-data-blog/

© 2014 International Business Machines Corporation
IBM Research – Industries & Solutions – Business Solutions & Mathematical Sciences
9
Graph Data Management
SparkseeNeo4j
Titan

© 2014 International Business Machines Corporation
IBM Research – Industries & Solutions – Business Solutions & Mathematical Sciences
10
Organization of Graph Store

© 2014 International Business Machines Corporation
IBM Research – Industries & Solutions – Business Solutions & Mathematical Sciences
11
Organize Temporal Graph Data Name Default Value
vertex_history Disabled
num_vertex_property_bundles 0
edge_history Disabled
num_edge_property_bundles 0
… …
Flag 
(uint8)
inEdge 
(uint64)
inEdge Count
(uint16)
outEdge 
(uint64)
outEdge Count 
(uint16)
Property 
(uint64)
Property Count 
(unit64)
History 
(unit64)
…
…
…
Vertex Record Table
inEdge List
Flag Property Property 
Count
History …
…
…
Prev Edge_list_buffer<EID,VID,LID>
…
…
Edge Record Table
Accessed Vertex  
Record by VID *  
izeof (VtxRec)
inEdgeCount * sizeof (<EID,VID,LID>) 
point to the buffer end
Accessed Edge  
Record by EID *  
izeof (EdgeRec)
Vertex Property Table
Prev property_buffer
…
…
PropertyCount 
point to the buffer end
Edge Property Table
Prev property_buffer
…
…
Name Default
Value
min_VID 0
max_VID
min_EID 0
Max_EID
… …
Local Configuration

© 2014 International Business Machines Corporation
IBM Research – Industries & Solutions – Business Solutions & Mathematical Sciences
12
Pointer Jumping in Temporal Graph Inference
• Converting a temporal graph into tridiagonal system
• Forward Gaussian elimination by propagation
• Backward substitution to produce solutions
• A Parallel Solution to Thomas Algorithm
• Apply pointer jumping to Thomas algorithm
• Logarithmic speedup
parallel solution to solve a tridiagonal linear system
• Converting a chain graph into tridiagonal system
• Forward Gaussian elimination by propagation
• Backward substitution to produce solutions  
• A Parallel Solution to Thomas Algorithm
• Apply pointer jumping to Thomas algorithm
• Logarithmic speedup
• Propagate belief among vertices within and cross
time stamps
Speedup wrt Gaussian  
Elimination: T3 / logT

© 2014 International Business Machines Corporation
IBM Research – Industries & Solutions – Business Solutions & Mathematical Sciences
13
Comments and Questions?

The Cost Approach case develops the example of a trended historical cost approach to valuation for a pool of patents. It is based on tangible experience incoming from TUDOR technology transfer activities. Although historical cost approaches to valuation are often presented as the simplest of all quantitative valuation methods, the reader will understand some drawbacks and key elements in order to ensure a valid valuation in such a framework. www.FITT-for-Innovation.eu

BIM For Free_28May2013_EdBartlett_BIM and lifecycle managementBIM Academy

20 billion triples in production

LDBC council

SPIMBENCH: A scalable, Schema-Aware Instance Matching Benchmark for the Seman...

LDBC council

Operational systems manage our finances, shopping, devices and much more. Adding real-time analytics to these systems enables them to instantly respond to changing conditions and provide immediate, targeted feedback. This use of analytics is called "operational intelligence," and the need for it is widespread. This talk will explain how in-memory computing techniques can be used to implement operational intelligence. It will show how an in-memory data grid integrated with a data-parallel compute engine can track events generated by a live system, analyze them in real time, and create alerts that help steer the system’s behavior. Code samples will demonstrate how an in-memory data grid employs object-oriented techniques to simplify the correlation and analysis of incoming events by maintaining an in-memory model of a live system. The talk also will examine simplifications offered by this approach over directly analyzing incoming event streams from a live system using complex event processing or Storm. Lastly, it will explain key requirements of the in-memory computing platform for operational intelligence, in particular real-time updating of individual objects and high availability using data replication, and contrast these requirements to the design goals for stream processing in Spark.

Data-Centric Approach for Project Delivery

AVEVA Group plc

Jacobs has used Endeavour (AVEVA NET) for more than 12 years for delivery of project data. The use has been primarily driven by customer or contract requirements for data handover, but over time both Jacobs’ project teams and customers have recognized the value of having trustworthy and complete data at the completion of a project, and is giving a focused effort to execute data-centric projects moving forward. To support this, Jacobs is implementing AVEVA Engineering to drive a data-centric collaboration between disciplines to enable greater work efficiencies. This game-changing approach using Endeavour and AVEVA Engineering will provide data alignment across the full project spectrum of EPC delivery. Presented by: Marc-Henri Cerar—Jacobs Discover how AVEVA can transform your business today www.aveva.com

Viewers also liked

Parallel and incremental materialisation of RDF/DATALOG in RDFOX

LDBC council

LDBC 6th TUC Meeting conclusions

LDBC council

MODAClouds Decision Support System for Cloud Service Selection

LDBC council

MarkLogic Overview and Use Cases

LDBC council

FOSDEM 2014: Social Network Benchmark (SNB) Graph Generator

LDBC council

E-Commerce and Graph-driven Applications: Experiences and Optimizations while...

LDBC council

Keynote IDEAS 2013 - Peter Boncz

LDBC council

LDBC Semantic Publishing Benchmark 2.0 evolution - Ontotext

LDBC council

LDBC 8th TUC Meeting: Introduction and status update

LDBC council

SADI: A design-pattern for “native” Linked-Data Semantic Web Services

LDBC council

Querying the Wikidata Knowledge Graph

LDBC council

Graph-TA 2013 - Josep Lluís Larriba Pey

LDBC council

Social Network Benchmark Interactive Workload

LDBC council

LDBC SNB Benchmark Auditing

LDBC council

LDBC SNB Benchmark Auditing

Ioan Toma

20 billion triples in production

Ioan Toma

MODAClouds Decision Support System for Cloud Service Selection

Ioan Toma

Social Network Benchmark Interactive Workload

Ioan Toma

SADI: A design-pattern for “native” Linked-Data Semantic Web Services

Ioan Toma

Lighthouse: Large-scale graph pattern matching on Giraph

LDBC council

Viewers also liked (20)

Parallel and incremental materialisation of RDF/DATALOG in RDFOX

LDBC 6th TUC Meeting conclusions

MODAClouds Decision Support System for Cloud Service Selection

MarkLogic Overview and Use Cases

FOSDEM 2014: Social Network Benchmark (SNB) Graph Generator

E-Commerce and Graph-driven Applications: Experiences and Optimizations while...

Keynote IDEAS 2013 - Peter Boncz

LDBC Semantic Publishing Benchmark 2.0 evolution - Ontotext

LDBC 8th TUC Meeting: Introduction and status update

SADI: A design-pattern for “native” Linked-Data Semantic Web Services

Querying the Wikidata Knowledge Graph

Graph-TA 2013 - Josep Lluís Larriba Pey

Social Network Benchmark Interactive Workload

LDBC SNB Benchmark Auditing

20 billion triples in production

MODAClouds Decision Support System for Cloud Service Selection

Social Network Benchmark Interactive Workload

SADI: A design-pattern for “native” Linked-Data Semantic Web Services

Lighthouse: Large-scale graph pattern matching on Giraph

Similar to Towards Temporal Graph Management and Analytics

IMCSummit 2015 - Day 1 Developer Track - Implementing Operational Intelligenc...

In-Memory Computing Summit

Data-Centric Approach for Project Delivery

AVEVA Group plc

New usage model for real-time analytics by Dr. WILLIAM L. BAIN at Big Data S...

Big Data Spain

HBaseCon 2015: Industrial Internet Case Study using HBase and TSDB

HBaseCon

This case study involves analysis of high-volume, continuous time-series aviation data from jet engines that consist of temperature, pressure, vibration and related parameters from the on-board sensors, joined with well-characterized slowly changing engine asset configuration data and other enterprise data for continuous engine diagnostics and analytics. This data is ingested via distributed fabric comprising transient containers, message queues and a columnar, compressed storage leveraging OpenTSDB over Apache HBase.

Don't Leave Your Traditional IBM Systems Out of Your IT Operations Efforts

Precisely

Many companies are modernizing their IT Operations capabilities by moving to a new platform. Some companies are motivated by a desire to become more proactive with their IT operations. Other companies find they have disparate processes and disconnected systems and need to move to a single platform for their IT operations. Whatever is motivating the modernization, it is critical that the new IT Operations process provides visibility into the entire infrastructure. Syncsort’s Ironstream solutions can provide a seamless integration for IBM Mainframe and IBM i IT Operations data to be included in these new IT Operations platforms. Watch this webcast to: Learn what IBM legacy system data needs to be integrated into the modern IT operations platforms like Splunk and ServiceNow Understand the pitfalls of common workarounds to this challenge Hear about actual real-world implementations and the benefits they are getting from Ironstream

Continuous Intelligence - Intersecting Event-Based Business Logic and ML

Paris Carbone

Modern data-driven business infrastructure is not as effective as it should be when it comes to critical decision making. Eng-to-End Data pipelines are composed out of fundamentally diverse pieces of tech, each focusing on a specific frontend (e.g., DataFrames, Tensors, Streams) and running in total isolation, thus, being highly unoptimised and complex to integrate with event-based business logic. Our research group has been looking into ways we can use advanced systems theory to compile, optimise and execute distributed functions in unison across the whole spectrum of data-driven programming, leading to a unified way to combine analytics and services all the way down to hardware execution and make continuous intelligence a reality. Key Takeaways Introducing the concept of Continuous Intelligence and why we are not there yet. Pinpointing weaknesses in the current way we structure data-driven pipelines today Explaining the potential of an Intermediate Representation (IR) and Shared Hardware Execution support to solve the problem. Presenting our vision on how this new tech can be used to radically change the way we declare and distill knowledge from data in a fast-changing world.

Lessons learned building a big data analytics engine, from proprietary to ope...

J On The Beach

Lessons learned building a big data analytics engine, from proprietary to open source by Álvaro Santamaria & Joel Brunger After spending four years building a proprietary all-in-one streaming analytics engine for financial services, it became clear that open-source was starting to pull ahead. Alvaro will talk about the challenges of creating an IT operations solution for financial services; what to build, what not to build, and how to use open source tools to get past the infrastructure and focus on the business problems that matter.

Making Hadoop Realtime by Dr. William Bain of Scaleout Software

Data Con LA

Hadoop has been widely embraced for its ability to economically store and analyze large data sets. Using parallel computing techniques like MapReduce, Hadoop can reduce long computation times to hours or minutes. This works well for mining large volumes of historical data stored on disk, but it is not suitable for gaining real-time insights from live operational data. Still, the idea of using Hadoop for real-time data analytics on live data is appealing because it leverages existing programming skills and infrastructure – and the parallel architecture of Hadoop itself. This presentation will describe how real-time analytics using Hadoop can be performed by combining an in-memory data grid (IMDG) with an integrated, stand-alone Hadoop MapReduce execution engine. This new technology delivers fast results for live data and also accelerates the analysis of large, static data sets.

Python business intelligence (PyData 2012 talk)

Stefan Urbanek

Analytics&IoT

Selvaraj Kesavan

Smart building mendix azure influx / smart City / IoT

Conclusion Connect enabling industry 4.0 with IoT

Scorecard Integration v1 MFGates, Map It Ralph

Brij Consulting, LLC

Servi sMART - Servi.ca - Smart Market Platform for Microservices

Stefan Ianta

StreamCentral for the IT Professional

Raheel Retiwalla

This presentation gives an overview of StreamCentral technology targeted for IT professionals. StreamCentral is software to model and build Big Data Solutions. StreamCentral consists of a Big Data Solutions Modeler that not only makes it easy to model traditional BI/DW and Big Data solutions but also auto deploys the model on the latest innovations in Big Data Management solutions (like HP Vertica and SQL Server Parallel Data Warehouse). StreamCentral Big Data Server executes the model definition in real-time. StreamCentral drastically reduces the time to market, risk and cost associated with building traditional BI/DW and Big Data solutions!

The Heatmap  - Why is Security Visualization so Hard?

Raffael Marty

The extent and impact of recent security breaches is showing that current approaches are just not working. But what can we do to protect our business? We have been advocating monitoring for a long time as a way to detect subtle, advanced attacks. However, products have failed to deliver on this promise. Current solutions don't scale in both data volume and analytical insights. In this presentation we will explore why it is so hard to come up with a security monitoring (or shall we call it security intelligence) approach that helps find sophisticated attackers in all the data collected. We are going to explore the question of how to visualize a billion events. We are going to look at a number of security visualization examples to illustrate the problem and some possible solutions. These examples will also help illustrate how data mining and user experience design help us get a handle of the security visualization challenges - enabling us to gain deep insight for a number of security use-cases.

Global C4IR-1 Masterclass Adryan - Zuehlke Engineering 2017

Justin Hayward

Fast Data at ING – the why, what and how of the streaming analytics platform ...

Bas Geerdink

Actionable Insights - Thompson

Prolifics

Neo4j: What's Under the Hood & How Knowing This Can Help You

Neo4j

Mindsphere: an open cloud-based IoT operating system for Industry

IIoTWorld

Similar to Towards Temporal Graph Management and Analytics (20)

IMCSummit 2015 - Day 1 Developer Track - Implementing Operational Intelligenc...

Data-Centric Approach for Project Delivery

New usage model for real-time analytics by Dr. WILLIAM L. BAIN at Big Data S...

HBaseCon 2015: Industrial Internet Case Study using HBase and TSDB

Don't Leave Your Traditional IBM Systems Out of Your IT Operations Efforts

Continuous Intelligence - Intersecting Event-Based Business Logic and ML

Lessons learned building a big data analytics engine, from proprietary to ope...

Making Hadoop Realtime by Dr. William Bain of Scaleout Software

Python business intelligence (PyData 2012 talk)

Analytics&IoT

Smart building mendix azure influx / smart City / IoT

Scorecard Integration v1 MFGates, Map It Ralph

Servi sMART - Servi.ca - Smart Market Platform for Microservices

StreamCentral for the IT Professional

The Heatmap  - Why is Security Visualization so Hard?

Global C4IR-1 Masterclass Adryan - Zuehlke Engineering 2017

Fast Data at ING – the why, what and how of the streaming analytics platform ...

Actionable Insights - Thompson

Neo4j: What's Under the Hood & How Knowing This Can Help You

Mindsphere: an open cloud-based IoT operating system for Industry

More from LDBC council

8th TUC Meeting - Juan Sequeda (Capsenta). Integrating Data using Graphs and ...

LDBC council

Juan Sequeda, Co-founder of Capsenta, gave an interesting talk on how can we integrate data using graphs and semantics (semantic data virtualization). As Mr. Sequeda said, the idea is to integrate data without needing to move it around. Juan started off his presentation talking about the huge gap that exists between the IT departments, guardians of the data and the business development departments, trying to extract insights about the data.

8th TUC Meeting - Zhe Wu (Oracle USA). Bridging RDF Graph and Property Graph...

LDBC council

8th TUC Meeting – Yinglong Xia (Huawei), Big Graph Analytics Engine

LDBC council

8th TUC Meeting – George Fletcher (TU Eindhoven), gMark: Schema-driven data a...

LDBC council

George Fletcher, Associate Professor at the Eindhoven University of Technology, presented gMark, an open-source framework for generating synthetic graph instances and workloads. The main focus of gMark has been to tailor different graph data management scenarios, often driven by query workloads. Such as multi-query optimization, workload-driven graph database physical design or mapping discovery and query rewriting data integration systems.

8th TUC Meeting – Marcus Paradies (SAP) Social Network Benchmark

LDBC council

Marcus Paradies, Software developer at SAP extended the talk Arnau Prats gave about the SNB, in this case about the Intelligence workload. In contrast with the 17+4 queries the Interactive workload has, the Business Intelligence (BI) workload consists on 24 queries that can be seen as OLAP-style against the OLTP-style of the Interactive one. The BI focuses on analytic queries and they touch the whole graph.

8th TUC Meeting - Sergey Edunov (Facebook). Generating realistic trillion-edg...

LDBC council

Weining Qian (ECNU). On Statistical Characteristics of Real-Life Knowledge Gr...

LDBC council

8th TUC Meeting - Peter Boncz (CWI). Query Language Task Force status

LDBC council

8th TUC Meeting | Lijun Chang (University of New South Wales). Efficient Subg...

LDBC council

8th TUC Meeting - Eugene I. Chong (Oracle USA). Balancing Act to improve RDF ...

LDBC council

8th TUC Meeting -

LDBC council

8th TUC Meeting - David Meibusch, Nathan Hawes (Oracle Labs Australia). Frapp...

LDBC council

8th TUC Meeting - Martin Zand University of Rochester Clinical and Translatio...

LDBC council

8th TUC Meeting - Tim Hegeman (TU Delft). Social Network Benchmark, Analytics...

LDBC council

More from LDBC council (14)