Apache Fluo is an open source implementation of Google's Percolator (which populates Google's search index) for Apache Accumulo. Fluo makes it possible to continuously join new data into large existing data sets without reprocessing all data. This is done with low latency. This talk will walk through the use case of building a derived graph from multiple graphs using Fluo. For example, multiple social network graphs (e.g. Twitter, Github, Google+, etc) could be combined into a single derived graph. Machine learning could link ids and inject these into Fluo. Fluo could continually join the linked ids with social network data to create virtual nodes, compute features, and export updates to a query system. A system like this would enable searches for virtual nodes with features that only exists in the combined view. — Speaker — Keith Turner Software Engineer, Peterson Technologies Keith Turner has been working with big data since 2004. Keith started working on Accumulo in 2008 and Fluo in 2013. Keith has an MS in Computer Science from Purdue and a BS in Computer Science from the University of Louisiana at Lafayette. — More Information — For more information see http://www.accumulosummit.com/

1. Continuously computing and
indexing a derived graph using
Apache Fluo
Keith Turner
Peterson Technologies

2. Percolator : Google’s Use Case
● Terabytes of new data coming in each day
● To build index: join terabytes of new data with petabytes of existing data.
● Joining new data with existing data via Map Reduce took multiple days.
● Using Percolator, index update time dropped from days to minutes.

3. Fluo Features
● Layer on top of Accumulo
● Snapshot Isolation : only see committed data
● Cross Row/Node Transactions
○ Read/write data from multiple nodes
○ Fail if two transactions modify same cell : collision
○ Correct in case of faults on multiple nodes
● Observers
○ User code, executes a transaction
○ Triggered by persistent notifications.
○ Observers can trigger other observers
○ Runs in parallel on many nodes

4. Fluo 101 - Architecture
Accumulo
HDFS
Zookeeper
YARN (Kubernetes and Mesos soon)
Client Cluster
Fluo Client
for App 1
Fluo Client
for App 1
Fluo Client
for App 2
Fluo Application 2Fluo Application 1
Fluo Worker
Observer1 Observer2
Fluo Oracle
Fluo Worker
ObserverA
Fluo Oracle
Fluo Worker
Observer1 Observer2
Table1 Table2

5. Derived graph overview

6. Graphs from multiple social networks
Twitter
T1
T2
T3
T4
Github
G1
G2
G3
G4
Facebook
F1
F2
F3 F5

7. Analytics determine aliases
Twitter
T1
T2
T3
T4
Github
G1
G2
G3
G4
Facebook
F1
F2
F3 F5
Aliases
A1
A2
A3
A4

8. Create derived graph
Twitter
T1
T2
T3
T4
Github
G1
G2
G3
G4
Facebook
F1
F2
F3 F5
Derived
A1
A2
A3
A4
F5

9. Add an edge
Twitter
T1
T2
T3
T4
Github
G1
G2
G3
G4
Facebook
F1
F2
F3
Derived
A1
A2
A3
A4
F5

10. Add edge in derived graph
Twitter
T1
T2
T3
T4
Github
G1
G2
G3
G4
Facebook
F1
F2
F3 F5
Derived
A1
A2
A3
A4
F5

11. Add attributes
Twitter
T1
T2
T3
T4
Github
G1
G2
G3
G4
Facebook
F1
F2
F3
Derived
A1
A2
A3
A4
F5
Location:
4 Privet Dr
Timezone:
GMT

12. Add attributes in derived graph
Twitter
T1
T2
T3
T4
Github
G1
G2
G3
G4
Facebook
F1
F2
F3 F5
Derived
A1
A2
A3
A4
F5
Location:
4 Privet Dr
Timezone:
GMT
Timezone:
GMT
Location:
4 Privet Dr

13. Putting it all together
Fluo Derived Graph
Application
Raw Graph Data
Changes
Alias analytics
Attribute analytics
Query System
Analytic System

14. Distribution of data on cluster
Server 1 Server 2 Server 3 Server 4 Server 5 Server 6
Input graph 1 (e.g. Twitter data)
Input graph 2 (e.g. Github data)
Derived graph
Input graph 3 (e.g. Facebook data)
Aliases
Attributes

15. Using Map Reduce to create derived graph
● Three to Four Joins/Map Reduce jobs
● Analysis/indexing of derived graph requires additional jobs
● When input data changes, must reprocess all data

16. Derived edges Map Reduce job #1
Input
Aliases
A1 F1, T1
A1 F1
A2 T2
A3 F3,T3
Edges
T1 T3
T3 T1
T1 T2
F1 F3
Output
Derived
Edges
Original
Edges
A1 T3 T1 T3
A3 T1 T3 T1
A1 T2 T1 T2
A1 F3 F1 F3

17. Derived edges Map Reduce job #2
Input
Aliases
A1 F1, T1
A1 F1
A2 T2
A3 F3,T3
Output
Derived
Edges
Original
Edges
A1 A3 T1 T3
A3 A1 T3 T1
A1 A2 T1 T2
A1 A3 F1 F3
Derived
Edges
Original
Edges
A1 T3 T1 T3
A3 T1 T3 T1
A1 T2 T1 T2
A1 F3 F1 F3

18. Unique edges Map Reduce job (optional)
Input
Output
Derived
Edges
Original
Edges
A1 A3 {T1->T3,F1->F3}
A1 A2 {T1->T2}
A3 A1 {T3->T1}
Derived
Edges
Original
Edges
A1 A3 T1 T3
A3 A1 T3 T1
A1 A2 T1 T2
A1 A3 F1 F3

19. Derived attributes Map Reduce job
Input
Aliases
A1 F1, T1
A1 F1
A2 T2
A3 F3,T3
Attributes
T1 {K1=V1}
F1 {K2=V2}
Output
Derived Attributes
A1 {T1.K1=V1,
F1.K2=V2}

20. Analysis/Indexing Map Reduce jobs ...
Input
Derived
Edges
Original
Edges
A1 A3 {T1->T3,F1->F3}
A1 A2 {T1->T2}
A3 A1 {T3->T1}
Derived Attributes
A1 {T1.K1=V1,
F1.K2=V2}
Output ????

21. Using Fluo to create derived graph
● Inputs
○ Raw edges
○ Raw node attributes
○ Aliases
● Supports adding and removing
○ Does not require reprocessing all data
● Outputs changes to derived graph

22. Adding a new edge

23. Fluo Data (stored in Accumulo table)
Twitter data
T1 alias A1
T3 alias A3
T5 alias A5
New Edge Transaction : T1->T5
Derived GraphGithub data
G1 alias A1
G2 alias A2
G7 alias A7

24. Fluo Data
Twitter data
T1 alias A1
T3 alias A3
T5 alias A5
New Edge Transaction : T1->T5
● Read Aliases
Derived GraphGithub data
G1 alias A1
G2 alias A2
G7 alias A7
Legend
Data WrittenData Read Notification

25. Fluo Data
Twitter data
T1 alias A1
T1 -> T5 A1:A5
T3 alias A3
T5 alias A5
T5 <- T1 A5:A1
New Edge Transaction : T1->T5
● Write Edges
Derived Graph
A1 -> A5 T1:T5 new
A5 <- A1 T5:T1 new
Github data
G1 alias A1
G2 alias A2
G7 alias A7
Legend
Data WrittenData Read Notification

26. Fluo Data
Twitter data
T1 alias A1
T1 -> T5 A1:A5
T3 alias A3
T5 alias A5
T5 <- T1 A5:A1
New Edge Transaction : T1->T5
● Notify nodes
Derived Graph
A1 -> A5 T1:T5 new
A5 <- A1 T5:T1 new
Github data
G1 alias A1
G2 alias A2
G7 alias A7
Legend
Data WrittenData Read Notification

27. Fluo Data
Twitter data
T1 alias A1
T1 -> T5 A1:A5
T3 alias A3
T5 alias A5
T5 <- T1 A5:A1
New Edge Transaction : T1->T5
● Commit
Derived Graph
A1 -> A5 T1:T5 new
A5 <- A1 T5:T1 new
Github data
G1 alias A1
G2 alias A2
G7 alias A7
Legend
Data WrittenData Read Notification

28. Processing Changes to a Derived Node

29. Fluo Data
Twitter data
T1 alias A1
T1 -> T5 A1:A5
T3 alias A3
T5 alias A5
T5 <- T1 A5:A1
Derived Node Transaction : A1
● Read changed edges
Derived Graph
A1 -> A5 T1:T5 new
A5 <- A1 T5:T1 new
Github data
G1 alias A1
G2 alias A2
G7 alias A7
Export Queue
Legend
Data WrittenData Read Notification

30. Fluo Data
Twitter data
T1 alias A1
T1 -> T5 A1:A5
T3 alias A3
T5 alias A5
T5 <- T1 A5:A1
Derived Node Transaction : A1
● Mark edge processed
● Queue for export
Derived Graph
A1 -> A5 T1:T5
A5 <- A1 T5:T1 new
Github data
G1 alias A1
G2 alias A2
G7 alias A7
Export Queue
+ A1->A5 Followers:0 Following:1
Legend
Data WrittenData Read Notification

31. Fluo Data
Twitter data
T1 alias A1
T1 -> T5 A1:A5
T3 alias A3
T5 alias A5
T5 <- T1 A5:A1
Derived Node Transaction : A1
● Commit
Derived Graph
A1 -> A5 T1:T5
A5 <- A1 T5:T1 new
Github data
G1 alias A1
G2 alias A2
G7 alias A7
Export Queue
+ A1->A5 Followers:0 Following:1
Legend
Data WrittenData Read Notification

32. Fluo Data
Twitter data
T1 alias A1
T1 -> T5 A1:A5
T3 alias A3
T5 alias A5
T5 <- T1 A5:A1
Derived Graph
A1 -> A5 T1:T5
A5 <- A1 T5:T1
Github data
G1 alias A1
G2 alias A2
G7 alias A7
Export Queue
+ A1->A5 Followers:0 Following:1
+ A5<-A1 Followers:1 Following:0

33. Processing an Alias Change

34. Fluo Data
Twitter data
T1 alias A1
T1 -> T5 A1:A5
T3 alias A3
T5 alias A7
T5 <- T1 A5:A1
Derived Graph
A1 -> A5 T1:T5
A1 -> A7 G1:G7
A5 <- A1 T5:T1
A7 <- A1 G7:G1
Github data
G1 alias A1
G1 -> G7 A1:A7
G2 alias A2
G7 alias A7
G7 <- G1 A7:A1
Alias Change Transaction : T5
Legend
Data WrittenData Read Notification

35. Fluo Data
Twitter data
T1 alias A1
T1 -> T5 A1:A5
T3 alias A3
T5 alias A7
T5 <- T1 A5:A1
Derived Graph
A1 -> A5 T1:T5
A1 -> A7 G1:G7
A5 <- A1 T5:T1
A7 <- A1 G7:G1
Github data
G1 alias A1
G1 -> G7 A1:A7
G2 alias A2
G7 alias A7
G7 <- G1 A7:A1
Alias Change Transaction : T5
● Read edges and alias
Legend
Data WrittenData Read Notification

36. Fluo Data
Twitter data
T1 alias A1
T1 -> T5 A1:A7
T3 alias A3
T5 alias A7
T5 <- T1 A7:A1
Derived Graph
A1 -> A5 T1:T5 deleted
A1 -> A7 G1:G7
A1 -> A7 T1:T5 new
A5 <- A1 T5:T1 deleted
A7 <- A1 G7:G1
A7 <- A1 T5:T1 new
Github data
G1 alias A1
G1 -> G7 A1:A7
G2 alias A2
G7 alias A7
G7 <- G1 A7:A1
Alias Change Transaction : T5
● Delete edges
● Insert edges
Legend
Data WrittenData Read Notification

37. Fluo Data
Twitter data
T1 alias A1
T1 -> T5 A1:A7
T3 alias A3
T5 alias A7
T5 <- T1 A7:A1
Derived Graph
A1 -> A5 T1:T5 deleted
A1 -> A7 G1:G7
A1 -> A7 T1:T5 new
A5 <- A1 T5:T1 deleted
A7 <- A1 G7:G1
A7 <- A1 T5:T1 new
Github data
G1 alias A1
G1 -> G7 A1:A7
G2 alias A2
G7 alias A7
G7 <- G1 A7:A1
Alias Change Transaction : T5
● Set notifications
Legend
Data WrittenData Read Notification

38. Fluo Data
Twitter data
T1 alias A1
T1 -> T5 A1:A7
T3 alias A3
T5 alias A7
T5 <- T1 A7:A1
Derived Graph
A1 -> A5 T1:T5 deleted
A1 -> A7 G1:G7
A1 -> A7 T1:T5 new
A5 <- A1 T5:T1 deleted
A7 <- A1 G7:G1
A7 <- A1 T5:T1 new
Github data
G1 alias A1
G1 -> G7 A1:A7
G2 alias A2
G7 alias A7
G7 <- G1 A7:A1
Alias Change Transaction : T5
● Commit
Legend
Data WrittenData Read Notification

39. Legend
Concurrent Aliases Change
Twitter data (time 1)
T1 alias A9
T1 -> T5 A1:A5
T3 alias A3
T5 alias A7
T5 <- T1 A5:A1
● Alias for T1 and T2 both change.
● Starts two transactions.
● Collision : one fails, one succeeds.
Twitter data (time2)
T1 alias A9
T1 -> T5 A1:A7
T1 -> T5 A9:A5
T3 alias A3
T5 alias A7
T5 <- T1 A5:A9
T5 <- T1 A7:A1
Twitter data (time 0)
T1 alias A1
T1 -> T5 A1:A5
T3 alias A3
T5 alias A5
T5 <- T1 A5:A1
Transaction 1
Changes
Transaction 2
Changes

40. Adding attribute not covered

41. Mixer prototype
● Supports add/remove of edges, aliases, and attributes.
● Exports changes to external query table.
○ Uses invert on export.
● Can lookup nodes in external query table
● Available soon on github
● Easy to run using MiniFluo and MiniAccumulo
○ Git clone
○ ./mixer.sh mini &> mini.log &
○ ./mixer.sh shell fluo.properties

43. Derived graph in Fluo
bob
tw:bob99
g+:bobE
gh:bob799
tw:alice95
loc=TX
g+:joe8
gh:jojo
gh:jeb
fb:joe9
gh:eAdam
gh:alice++
tz=CST
Bob in query table updated by Fluo
bob -> g+:joe8 followers=1,following=0,rawEdges=1
bob -> gh:jojo followers=1,following=1,rawEdges=1
bob -> tw:alice95 followers=1,following=0,loc=TX,rawEdges=1
Status up to here

44. Derived graph in Fluo
bob
tw:bob99
g+:bobE
gh:bob799
alice
tw:alice95
loc=TX
g+:joe8
gh:jojo
gh:jeb
fb:joe9
gh:eAdam
gh:alice++
tz=CST
Bob in query table updated by Fluo
bob -> alice followers=1,following=0,loc=TX,tz=CST,rawEdges=1
bob -> g+:joe8 followers=1,following=0,rawEdges=1
bob -> gh:jojo followers=1,following=1,rawEdges=1
bob -> tw:alice95 followers=1,following=0,loc=TX,rawEdges=1
Status up to here

45. Derived graph in Fluo
bob
tw:bob99
g+:bobE
gh:bob799
alice
tw:alice95
loc=TX
joe
g+:joe8
gh:jojo
gh:jeb
fb:joe9
gh:eAdam
gh:alice++
tz=CST
Bob in query table updated by Fluo
bob -> alice followers=1,following=0,loc=TX,tz=CST,rawEdges=1
bob -> joe followers=1,following=2,rawEdges=2
bob -> g+:joe8 followers=1,following=0,rawEdges=1
bob -> gh:jojo followers=1,following=1,rawEdges=1
bob -> tw:alice95 followers=1,following=0,loc=TX,rawEdges=1
Status up to here

46. Getting started with Fluo
● Fluo Tour
● Documentation on website
● Mailing list and IRC

47. https://fluo.apache.org