Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Tips for writing Apache Fluo
Applications
Keith Turner
Peterson Technologies
Percolator : Google’s Use Case
● Terabytes of new data coming in each day
● To build index: join new data with petabytes o...
Fluo Features
● Layer on top of Accumulo
● Snapshot Isolation : only see committed data.
● Cross Row Transactions
○ Read/w...
Fluo 101 - Architecture
Accumulo
HDFS
Zookeeper
YARN
Client Cluster
Fluo Client
for App 1
Fluo Client
for App 1
Fluo Clien...
Fluo Recipes Overview
● Separate project from Fluo
● Can have different release cadence and support older versions of Fluo...
Observers incrementally transform datasets
Input Dataset 1
Input Dataset 2
Derived Dataset
A
Derived Dataset
B
Observer Y
...
Uneven data -> low utilization
Tablet
Server 1
Tablet
Server 2
Tablet
Server 3
Tablet
Server 4
Tablet
Server 5
Tablet
Serv...
Balance is important
Evenly spread data -> maybe high utilization
Tablet
Server 1
Tablet
Server 2
Tablet
Server 3
Tablet
Server 4
Tablet
Server...
Many instances of observers running
Tablet
Server 1
Tablet
Server 2
Tablet
Server 3
Tablet
Server 4
Tablet
Server 5
Tablet...
Balancing details
● Fluo Recipes provides an API to specify table ranges to
balance
● Other recipes use this balancing API...
May still have hotspots
● Popular keys may cluster. Add short hash to prefix.
● <prefix>:<hash of key>:<key>
● p:a8j42:com...
Incrementing link counts
//When a document changes, determine changes in its links
foreach(url : newOutlinks)
C = tx.get(u...
Collisions : One TX fails
Observer 1
Processing
Doc2
URL 1
URL 2
URL 3
Transactions attempting to
update URL in link count...
Queue updates for url
//When a document changes, determine changes in its links
foreach(url : newOutlinks)
tx.set(url, ‘qu...
Use queues + multiple txs to avoid collisions
Observer 1
Processing
Doc2
URL 1
Update
Queue
URL 2
Update
Queue
URL 3
Updat...
Updates leave a trail
An Accumulo’s Tablet’s files
K1.1=45
K2.1=22
K3.1=99
.
.
A ton of
other key
values
Flush a file with some updates
An Accumulo’s Tablet’s files
K1.1=45
K2.1=22
K3.1=99
.
.
A ton of
other key
values
K1U.1=+1...
Flush a file with some deletes
An Accumulo’s Tablet’s files
K1.1=45
K2.1=22
K3.1=99
.
.
A ton of
other key
values
K1U.1=+1...
Compact smaller files into one
An Accumulo’s Tablet’s files
K1.1=45
K2.1=22
K3.1=99
.
.
A ton of
other key
values
K1.2=44
...
Compacting all files drops delete markers
An Accumulo’s Tablet’s files
K1.2=44
K2.2=20
K3.1=99
.
.
A ton of
other key
valu...
Compactions will clean it up
● Too expensive to compact all data
● Accumulation of delete markers will cause slow down
● A...
Fluo Table
Derived
dataset A
Updates
for
A
Derived
dataset B
Updates
For
B
Input
dataset 2
Input
dataset 1
Can compact
jus...
Evenly spread update data
Tablet
Server 1
Tablet
Server 2
Tablet
Server 3
Tablet
Server 4
Tablet
Server 5
Tablet
Server 6
...
Transient data
● Fluo Recipes provides transient data registry
○ Used by other recipes (export Q and CFM)
● Fluo Recipes h...
Collision Free Map Recipe
● Recipe that implements queued update pattern
● Like map reduce, but continuous
● Two transacti...
Exporting Data
● Export data from Fluo to external system
○ Useful for continuously updating an external user facing query...
Exporting Data from Transaction
oldV = tx1.get(K1,”old”)
newV = tx1.get(K1,”new”)
exportDiffsExternalSystem(K1,oldV,newV)
...
Export Queue Recipe
● Only export committed data.
● Transactions add key/values to export queue
○ Gives each export entry ...
Export Queue
Transaction tx1
oldV = tx1.get(K1,”old”)
newV = tx1.get(K1,”new”)
queueExport(K1,oldV,newV)
tx1.commit()
Key ...
Export Queue Observer
● Transactions add to export queue and
notify observer
● An observer will export data on queue
○ May...
Evenly spread export data
Tablet
Server 1
Tablet
Server 2
Tablet
Server 3
Tablet
Server 4
Tablet
Server 5
Tablet
Server 6
...
Invert on export
● Compute information to index in Fluo
● Invert information when exporting
● Assume want to build a syste...
Invert on export example
● URI1 incount changes from
90 to 115
● Queues URI1:90:115 for
export
● Makes 5 updates for one
e...
Webindex video
Same row different columns
Tablet A
Row 1
Row 2
Tablet B
Row 3
Row 4
TX1
Fluo Client L
TX2
Fluo Client N
TX3
Fluo Client P...
Row Locking
● Fluo uses Accumulo Conditional Mutations
● Tserver locks entire row to check conditions
● Concurrent transac...
How much work should an observer do?
● Reasons to do less work :
○ Has to fit into memory.
○ Should have a low probability...
Fluo Releases
● Fluo 1.0.0 Release vote passed on October 4th
● Fluo Recipes 1.0.0 will be released soon
● Did well on 3 d...
Fluo Tour
● Self guided set of exercises on website
● Considering stepping through it at hackathon tonight
Moved to Apache
https://fluo.apache.org
Upcoming SlideShare
Loading in …5
×

Accumulo Summit 2016: Tips for Writing Fluo Applications

375 views

Published on

Apache Fluo (incubating) is an open source implementation of Percolator (which populates Google's search index) for Apache Accumulo. Fluo makes it possible to update the results of a large-scale computation, index, or analytic as new data is discovered.

While working on developing Fluo, we made it a priority to attempt to develop applications for Fluo in tandem. Developing these applications resulted in many improvements to Fluo and we also learned a lot about writing Fluo applications. These lesson learned drove the implementation of the Fluo Recipes project. This talk will go over some of the lessons learned and Fluo Recipes. Hopefully this information will save time for anyone attempting to write a Fluo application.

Some of the areas that will be covered are:

• Organizing data for optimal performance
• Achieving high cluster utilization
• Exporting data from Fluo to query systems
• Avoiding performance degradation over time

— Speaker — 

Keith Turner
Software Engineer, Peterson Technologies

Keith Turner has been working with big data since 2004. Keith started working on Accumulo in 2008 and Fluo in 2013. Keith has an MS in Computer Science from Purdue and a BS in Computer Science from the University of Louisiana at Lafayette.

— More Information —

For more information see http://www.accumulosummit.com/

Published in: Data & Analytics
  • Be the first to comment

Accumulo Summit 2016: Tips for Writing Fluo Applications

  1. 1. Tips for writing Apache Fluo Applications Keith Turner Peterson Technologies
  2. 2. Percolator : Google’s Use Case ● Terabytes of new data coming in each day ● To build index: join new data with petabytes of existing data. ● Joining new data with existing data via Map Reduce took multiple days. ● Using Percolator, index update time dropped from days to minutes.
  3. 3. Fluo Features ● Layer on top of Accumulo ● Snapshot Isolation : only see committed data. ● Cross Row Transactions ○ Read/write data from multiple nodes ○ Fail if two transactions modify same cell : collision ○ Correct in case of faults on multiple nodes ● Observers ○ User code, executes a transaction ○ Triggered by persistent notifications. ○ Observers can trigger other observers ○ Runs in parallel on many nodes
  4. 4. Fluo 101 - Architecture Accumulo HDFS Zookeeper YARN Client Cluster Fluo Client for App 1 Fluo Client for App 1 Fluo Client for App 2 Fluo Application 2Fluo Application 1 Fluo Worker Observer1 Observer2 Fluo Oracle Fluo Worker ObserverA Fluo Oracle Fluo Worker Observer1 Observer2 Table1 Table2
  5. 5. Fluo Recipes Overview ● Separate project from Fluo ● Can have different release cadence and support older versions of Fluo ● Provides higher level abstractions that build on Fluo primitives ○ Collision Free Map : recipe for many-to-many updates ○ Export Queue : recipe for updating external systems in a fault tolerant manner ○ Recording TX : recipe for recording changes made by a TX ○ Each recipe is documented and tested ● Common code ○ Transient ranges ○ Table optimizations
  6. 6. Observers incrementally transform datasets Input Dataset 1 Input Dataset 2 Derived Dataset A Derived Dataset B Observer Y Observer X Fluo Client Fluo Client External System Observer Z
  7. 7. Uneven data -> low utilization Tablet Server 1 Tablet Server 2 Tablet Server 3 Tablet Server 4 Tablet Server 5 Tablet Server 6 Input dataset 1Input dataset 2 Derived dataset A Derived dataset B
  8. 8. Balance is important
  9. 9. Evenly spread data -> maybe high utilization Tablet Server 1 Tablet Server 2 Tablet Server 3 Tablet Server 4 Tablet Server 5 Tablet Server 6 Input dataset 1 Input dataset 2 Derived dataset A Derived dataset B
  10. 10. Many instances of observers running Tablet Server 1 Tablet Server 2 Tablet Server 3 Tablet Server 4 Tablet Server 5 Tablet Server 6 Input dataset 1 Input dataset 2 Derived dataset A Derived dataset B X X X X X X X X X X X X X X X X X Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y
  11. 11. Balancing details ● Fluo Recipes provides an API to specify table ranges to balance ● Other recipes use this balancing API ● Implementation uses Accumulo RegexGroupBalancer introduced in 1.7.0
  12. 12. May still have hotspots ● Popular keys may cluster. Add short hash to prefix. ● <prefix>:<hash of key>:<key> ● p:a8j42:com.example/popular-link ● Fluo Recipes can help hash and evenly spread data. ○ Generates tablet splits for hashed prefix ○ Generates balancing config for hashed prefix ○ Util code to add/remove hash+prefix
  13. 13. Incrementing link counts //When a document changes, determine changes in its links foreach(url : newOutlinks) C = tx.get(url, ‘inLinks’) tx.set(url, ‘inLinks’, C + 1) foreach(url : removedOutlinks) C = tx.get(url, ‘inLinks’) tx.set(url, ‘inLinks’, C - 1)
  14. 14. Collisions : One TX fails Observer 1 Processing Doc2 URL 1 URL 2 URL 3 Transactions attempting to update URL in link count. Do this by reading current count, adding or subtracting 1, writing back. Observer 1 Processing Doc1
  15. 15. Queue updates for url //When a document changes, determine changes in its links foreach(url : newOutlinks) tx.set(url, ‘queue:’+uuid, +1) foreach(url : removedOutlinks) tx.set(url, ‘queue:’+uuid, -1)
  16. 16. Use queues + multiple txs to avoid collisions Observer 1 Processing Doc2 URL 1 Update Queue URL 2 Update Queue URL 3 Update Queue Observer 1 Processing Doc1 Observer 2 Processing Queue Observer 2 Processing Queue Observer 2 Processing Queue URL 1 URL 2 URL 3
  17. 17. Updates leave a trail An Accumulo’s Tablet’s files K1.1=45 K2.1=22 K3.1=99 . . A ton of other key values
  18. 18. Flush a file with some updates An Accumulo’s Tablet’s files K1.1=45 K2.1=22 K3.1=99 . . A ton of other key values K1U.1=+1 K1U.2=-2 K2U.1=+5 K2U.2=-7
  19. 19. Flush a file with some deletes An Accumulo’s Tablet’s files K1.1=45 K2.1=22 K3.1=99 . . A ton of other key values K1U.1=+1 K1U.2=-2 K2U.1=+5 K2U.2=-7 K1.2=44 K2.2=20 K1U.1 K1U.2 K2U.1 K2U.2
  20. 20. Compact smaller files into one An Accumulo’s Tablet’s files K1.1=45 K2.1=22 K3.1=99 . . A ton of other key values K1.2=44 K2.2=20 K1U.1 K1U.2 K2U.1 K2U.2
  21. 21. Compacting all files drops delete markers An Accumulo’s Tablet’s files K1.2=44 K2.2=20 K3.1=99 . . A ton of other key values
  22. 22. Compactions will clean it up ● Too expensive to compact all data ● Accumulation of delete markers will cause slow down ● Accumulo supports compactions of ranges ● Organize data so that updates are in separate ranges. ● <update prefix>:<hash of key>:<key> ● <data prefix>:<hash of key>:<key> ● Use fluo scan raw command to see trail of data left by transactions
  23. 23. Fluo Table Derived dataset A Updates for A Derived dataset B Updates For B Input dataset 2 Input dataset 1 Can compact just these ranges.
  24. 24. Evenly spread update data Tablet Server 1 Tablet Server 2 Tablet Server 3 Tablet Server 4 Tablet Server 5 Tablet Server 6 Input dataset 1 Input dataset 2 Derived dataset A Updates Derived dataset A Derived dataset B Updates Derived dataset B
  25. 25. Transient data ● Fluo Recipes provides transient data registry ○ Used by other recipes (export Q and CFM) ● Fluo Recipes has a utility to compact transient ranges ○ Since data is evenly spread, compactions are also ○ Utility periodically compacts
  26. 26. Collision Free Map Recipe ● Recipe that implements queued update pattern ● Like map reduce, but continuous ● Two transactions ○ One queues updates to one or more values. ○ One processes queued updates. ● User provides two functions ○ Combiner ○ Update Observer
  27. 27. Exporting Data ● Export data from Fluo to external system ○ Useful for continuously updating an external user facing query system ● Assume external system needs to see every change
  28. 28. Exporting Data from Transaction oldV = tx1.get(K1,”old”) newV = tx1.get(K1,”new”) exportDiffsExternalSystem(K1,oldV,newV) //update old and new tx1.commit() Committ ed Key oldV newV True K1 9 13 False K1 13 17 True K1 13 21 True K1 21 37 Assume external system deletes old and inserts new. Oh no 17 is inserted and never deleted.
  29. 29. Export Queue Recipe ● Only export committed data. ● Transactions add key/values to export queue ○ Gives each export entry a sequence number. ○ Only makes it in queue if tx succeeds ● User provides an idempotent export function. ○ Passed list of (key,seq,value) ● Receiving system must handle out of order and redundant data. ○ Sequence can help ○ Use sequence number for timestamp when exporting to Accumulo ● Automatic balancing config, transient range config, and table splits.
  30. 30. Export Queue Transaction tx1 oldV = tx1.get(K1,”old”) newV = tx1.get(K1,”new”) queueExport(K1,oldV,newV) tx1.commit() Key Sequen ce oldV newV K1 6 9 13 K1 13 13 21 K1 45 21 37
  31. 31. Export Queue Observer ● Transactions add to export queue and notify observer ● An observer will export data on queue ○ May export same data multiple times Ke y Sequen ce old V new V K1 6 9 13 K1 17 13 21 K1 45 21 37 Example of data being exported twice ● TX1 : Export K1,6,9,13 ● TX1 : Export K1,17,13,21 ● TX1 : Fault ● TX2 : Export K1,6,9,13 ● TX2 : Export K1,17,13,21 ● TX2 : Export K1,45,21,37 ● TX2 : delete queued exports ● TX2 : commit
  32. 32. Evenly spread export data Tablet Server 1 Tablet Server 2 Tablet Server 3 Tablet Server 4 Tablet Server 5 Tablet Server 6 Input dataset 1 Input dataset 2 Derived dataset A Updates Derived dataset A Derived dataset B Updates Derived dataset B Export Queue
  33. 33. Invert on export ● Compute information to index in Fluo ● Invert information when exporting ● Assume want to build a system that can answer following. ○ For a domain, which page has the most inbound links? ○ For a page, how many inbound links does it have? ○ For all pages, which page has the most inbound links? ● Index number of inbound links three ways. ○ Do not need to build 3 indexes in fluo. ○ Compute # of inbound links in fluo and export. ○ On export, update 3 indexes. ● Fluo orchestrates reliably and incrementally updating external index.
  34. 34. Invert on export example ● URI1 incount changes from 90 to 115 ● Queues URI1:90:115 for export ● Makes 5 updates for one export queue entry Export actions Action Row Insert t:(99999-115) <uri> Insert d:<domain>:(99999-115 ) <uri> Insert p:<uri> incount 115 Delete t:(99999-90) <uri> Delete d:<domain>:(99999-90) <uri>
  35. 35. Webindex video
  36. 36. Same row different columns Tablet A Row 1 Row 2 Tablet B Row 3 Row 4 TX1 Fluo Client L TX2 Fluo Client N TX3 Fluo Client P TX4 Fluo Client R Col U Col V Col W Col X
  37. 37. Row Locking ● Fluo uses Accumulo Conditional Mutations ● Tserver locks entire row to check conditions ● Concurrent transactions against a single row can slow each other, even if they do not collide. Example Schemas Row Column Row Lock Contention <node1> edge:<node2> Likely <node1>/<node2> edge Unlikely
  38. 38. How much work should an observer do? ● Reasons to do less work : ○ Has to fit into memory. ○ Should have a low probability of collisions. ● Reasons to do more work : ○ Minimizing total # of seeks ○ Avoiding recomputation
  39. 39. Fluo Releases ● Fluo 1.0.0 Release vote passed on October 4th ● Fluo Recipes 1.0.0 will be released soon ● Did well on 3 day cluster test ● Will follow semver
  40. 40. Fluo Tour ● Self guided set of exercises on website ● Considering stepping through it at hackathon tonight
  41. 41. Moved to Apache https://fluo.apache.org

×