Active Data: A Programming Model to Manage Data Life Cycle Across Heterogeneous Systems and Infrastructures
Upcoming SlideShare
Loading in...5
×
 

Active Data: A Programming Model to Manage Data Life Cycle Across Heterogeneous Systems and Infrastructures

on

  • 1,408 views

 

Statistics

Views

Total Views
1,408
Views on SlideShare
1,408
Embed Views
0

Actions

Likes
1
Downloads
3
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Active Data: A Programming Model to Manage Data Life Cycle Across Heterogeneous Systems and Infrastructures Active Data: A Programming Model to Manage Data Life Cycle Across Heterogeneous Systems and Infrastructures Presentation Transcript

  • Active Data: A Programming Model to ManageData Life Cycle Across Heterogeneous Systemsand InfrastructuresA. Simonet, G. Fedak, M. RipeanuGilles.Fedak@inria.frINRIA, University of Lyon, FranceAvalon WG, Lyon, France, 13/05/2013G. Fedak (INRIA/LIP) Active Data Avalon 1 / 22
  • Outline of Topics1 IntroductionChallenges of Life-Cycle ManagementActive Data at a GlanceRelated Works2 The Model and ImplementationLife-Cycle ModelExecution Runtime3 ExperimentsPerformance EvaluationUse Cases4 ConclusionG. Fedak (INRIA/LIP) Active Data Avalon 2 / 22
  • Outline1 IntroductionChallenges of Life-Cycle ManagementActive Data at a GlanceRelated Works2 The Model and ImplementationLife-Cycle ModelExecution Runtime3 ExperimentsPerformance EvaluationUse Cases4 ConclusionG. Fedak (INRIA/LIP) Active Data Avalon 3 / 22
  • Introduction to Big Data• Huge and growing volume of information originating from multiplesources.• Impacts many scientific disciplines and industry branches• Large Scientific Instruments (LSST, LHC, OOOI), but not only (Sequencingmachines)• Internet and Social Network (Google, Facebook, Twitter, etc.)• Open Data (Open Library, Governemental, Genomics)→ impacts the whole process of scientific discovery (4thparadigm of science)G. Fedak (INRIA/LIP) Active Data Avalon 4 / 22
  • Big Data Challenges• Big Data creates several challenges :• how to scale the infrastructure : end-to-end performance improvement,inter-system optimization.• how to improve productivity of data-intensive scientist : workflow,programming language, quality of data provenance.• how to enable collaborative data science : incentive for data publication,data-sets sharing, collaborative workflow.• New models and software are needed to represent and manipulate largeand distributed scientific data-sets.G. Fedak (INRIA/LIP) Active Data Avalon 5 / 22
  • Focus on Data Life-CycleData Life Cycle: the course of operational stages through which data passfrom the time when they enter a system to the time when they leave it.We’re aiming at :• Simplify the programming of applications that implement data life cyclemanagement.• A model to capture the essential life cycle stages and properties:creation, deletion, faults, replication, error checking . . .• Allow to reason about data sets handled by heterogeneous software andinfrastructures.• Allows legacy systems to expose their intrinsic data life cycle.• Scalable performances and minimum performance overhead overexisting systems.G. Fedak (INRIA/LIP) Active Data Avalon 6 / 22
  • Active Data at a Glance• a life cycle model, inspired by Petri Net, which allows data managementsystems to expose data life cycle through a well-formalizedrepresentation• a programming model and a runtime environment, which allows toprogram applications by specifying for each step of the data life cycle, thecode that will be executed.CREATEDt1 WRITTENt2READt3LOOPt4t5DELETEDFigure: Representation of the“Write-Once, Read-Many” data lifecycle.G. Fedak (INRIA/LIP) Active Data Avalon 7 / 22
  • Active Data at a Glance• a life cycle model, inspired by Petri Net, which allows data managementsystems to expose data life cycle through a well-formalizedrepresentation• a programming model and a runtime environment, which allows toprogram applications by specifying for each step of the data life cycle, thecode that will be executed.CREATEDt1 WRITTENt2READt3LOOPt4t5DELETEDFigure: Representation of the“Write-Once, Read-Many” data lifecycle.representation of the “Write-Once, Read-Many” life cycle.A data item starts its life in the Created place, then iswritten once (Written place), may be read once or severaltimes (loop between Read and Loop places) and finallydeleted (Deleted place). In this example, the transitiont1 corresponds to the action of writing data, t2 and t3 aretriggered when reading data, t4 is a phony transition corre-sponding to a loop so that data can be read again, and t5corresponds to a data item deletion.We present a short code sample that illustrates how toautomatically perform curation when data get written in thesystem. Whenever transition t2 is triggered, the applicationcreates a md5 signature file.The programmer first writes the handler that contains thechecksum computation:TransitionHandler md5Handler = new ￿→ TransitionHandler ( ) {public void handler ( Lif eC ycle lc , S tring ￿→ transitionName , boolean i s L o c a l ) {MessageDigest md = ￿→ MessageDigest . g e t I n s t a n c e ( ”MD5”) ;Strin g path = getPath ( l c . getId ( ) ) ;InputStream input = new ￿→ FileInputStream ( path ) ;byte b u f f e r [ ] = new byte [ 2 0 4 8 ] ;int n = 0 ;while ( ( n = input . read ( b u f f e r ) ) > 0)md. update ( buffer , 0 , n) ;byte [ ] d i g e s t = md. d i g e s t ( ) ;BigInteger b i g I n t = new ￿→ BigInteger (1 , d i g e s t ) ;Strin g hash = b i g I n t . t o S t r i n g (16) ;while ( hash . length ( ) < 32 )hash = ”0 ” + hash ;OutputStream output = new ￿→ FileOutputStream ( path + ” . md5”) ;output . write ( hash . getBytes ( ) ) ;output . c l o s e ( ) ;}} ;Listing 1: md5 sum transition handlerG. Fedak (INRIA/LIP) Active Data Avalon 7 / 22
  • Related Works• Data-centric parallel programming languages (MapReduce, Dryad,Allpairs, Twister, PigLatin . . . )• Runtime execution environments for dynamic data : incrementalprocessing (Percolator), parallel stream processing (Nephele,MapReduce Online), workflow (Chimera)• Event based processing (Mace, libasync, Incontext)• Data Provenance addresses the issue of representation of data-setderivation (PASS, Open Provenance Model)• Data Management Software (BitDew, Chirp, MosaStore, Globus Online,DCache, iRods and many more)G. Fedak (INRIA/LIP) Active Data Avalon 8 / 22
  • Outline1 IntroductionChallenges of Life-Cycle ManagementActive Data at a GlanceRelated Works2 The Model and ImplementationLife-Cycle ModelExecution Runtime3 ExperimentsPerformance EvaluationUse Cases4 ConclusionG. Fedak (INRIA/LIP) Active Data Avalon 9 / 22
  • Life-Cycle Model: the Basics• A Petri Net is a 5-tuple PN = (P, T, F, W, M0) where:• P = {CREATED, DELETED, p1, p2, . . . , pm} is a finite set of places;• T = {t1, t2, . . . , tn} is a finite set of transitions;• Data items are represented by tokens •.• W : F → N+is a weight function which indicates how many tokens everytransition requires and how many token it produces.• Data Identification, we tag each token δ with a triplet (id, i, p) where id isa unique data identifier, identical for all the replica of a single data item, iis the replica number and p ∈ P is a place.• Data Creation a data item is created as a token in the CREATED placelabelled with the triplet (id, 1, CREATED).• Data ReplicationP1t1NEW INSTANCE2t222Figure: Data instance creation.G. Fedak (INRIA/LIP) Active Data Avalon 10 / 22
  • Life-Cycle Model: the Basics• A Petri Net is a 5-tuple PN = (P, T, F, W, M0) where:• P = {CREATED, DELETED, p1, p2, . . . , pm} is a finite set of places;• T = {t1, t2, . . . , tn} is a finite set of transitions;• Data items are represented by tokens •.• W : F → N+is a weight function which indicates how many tokens everytransition requires and how many token it produces.• Data Identification, we tag each token δ with a triplet (id, i, p) where id isa unique data identifier, identical for all the replica of a single data item, iis the replica number and p ∈ P is a place.• Data Creation a data item is created as a token in the CREATED placelabelled with the triplet (id, 1, CREATED).• Data ReplicationP1t1NEW INSTANCE2t222Figure: Data instance creation.G. Fedak (INRIA/LIP) Active Data Avalon 10 / 22
  • Composition of Life-Cycle Model• Life Cycle Composition To compose two independent life cycles A andB, we define the start places, SaA ⊆ PA, and stop places, SoB ⊆ PB.• when one token reaches SaA, it creates a new token in the place CREATEDof B. New tokens are labeled with the quadruplet (idA, idB, rB, qB).• When one token reaches SoB, it creates a new token in the place DELETEDof A.• Life Cycle termination We say that a data item terminates its life cycle,when all of its tokens reach the DELETED place.• Composed Life Cycles Termination We say that a data item terminatesa composed life cycle, represented by A and B, their start sets SaA andSaB, and stop sets SoA and SoB, when it terminates A and terminates B.G. Fedak (INRIA/LIP) Active Data Avalon 11 / 22
  • Composition of Life-Cycle Model• Life Cycle Composition To compose two independent life cycles A andB, we define the start places, SaA ⊆ PA, and stop places, SoB ⊆ PB.• when one token reaches SaA, it creates a new token in the place CREATEDof B. New tokens are labeled with the quadruplet (idA, idB, rB, qB).• When one token reaches SoB, it creates a new token in the place DELETEDof A.• Life Cycle termination We say that a data item terminates its life cycle,when all of its tokens reach the DELETED place.• Composed Life Cycles Termination We say that a data item terminatesa composed life cycle, represented by A and B, their start sets SaA andSaB, and stop sets SoA and SoB, when it terminates A and terminates B.G. Fedak (INRIA/LIP) Active Data Avalon 11 / 22
  • Execution RuntimeActive Data is composed of two parts:• the execution runtime• manages data life cycles, publishes transitions, triggers handler codeexecutions, and guarantees execution correctness• distributed system based on Publish/Subscribe• the programming interface (API)• allows data management systems to publish transitions• allows programmers to develop applications by registering their transitionhandlers• two kinds of transition subscription :• subscribe to a specific transition for any data items• subscribe to a specific data item and being notified for any life cycle transitions.G. Fedak (INRIA/LIP) Active Data Avalon 12 / 22
  • Integration with Data Management SystemsCreated t1 To Placet2Deletedt3Placedt4Loopt52t6t7Lostt8t9(a) Bitdew SchedulerCreatedt1Readyt2Startedt3Completedt4Deleted Invalidt5t6t7t8(b) Bitdew File TransferIN CREATEt1IN MOVED FROMt2t3IN MOVED TOt4IN CLOSE WRITEt5t6t7 t8t9 t10t11DELETEDt14t13CREATEDt12(c) inotifyGetPutCreatedt5t6t7t8Deleted(d) iRODSCreatedt1 t2Succeeded Failedt3 t4Deleted(e) Globus OnlineFigure 3: Data life cycle models for four data management system.structed from its documentation.Reading the source code of BitDew, we observe that dataitems are managed by instances of the Data class, and thisclass has the status variable which holds the data itemstate. Therefore, we simply deduce from the enumeration ofthe possible value of status the set of corresponding placesin the Petri Net (see Figure 2a and 2b). By further analyz-ing the source code, we construct the model and summarizehow high level DLM features are modelized using ActiveData model:Scheduling and replication Part of the complexity ofthe data life cycle in BitDew comes from the Data Schedulerthat schedules data on nodes. Whenever a data item isscheduled on a node, a new replica is created. We represent Figure 4: Average number of transitions per second handledby the Active Data Service• BitDew (INRIA), programmable environment fordata management.• inotify Linux kernel subsystem: notification systemfor file creation, modification, write, movement anddeletion.• iRODS (DICE, Univ. North Carolina), rule-orienteddata management system.• Globus Online (ANL) offers fast, simple andreliable service to transfer large volumes of data.• Data scheduling andreplication• Fault tolerance• Composition of File Transferand Data SchedulerG. Fedak (INRIA/LIP) Active Data Avalon 13 / 22
  • Outline1 IntroductionChallenges of Life-Cycle ManagementActive Data at a GlanceRelated Works2 The Model and ImplementationLife-Cycle ModelExecution Runtime3 ExperimentsPerformance EvaluationUse Cases4 ConclusionG. Fedak (INRIA/LIP) Active Data Avalon 14 / 22
  • Performance EvaluationData life cycle models for four data management system.we observe that dataData class, and thisholds the data itemm the enumeration ofcorresponding places). By further analyz-model and summarizedelized using Activeof the complexity ofm the Data Schedulernever a data item isreated. We representstate that creates ansses through it.tDew’s target archi-with frequent faults,tem is scheduled to asystem, it is markeded to an other node.ed, Lost, tosched-nd Data Schedulere Transfer Service areycles. A file transfera item, and a deletedonnect the two Petrip places as explainedansfer can be startedFigure 4: Average number of transitions per second handledby the Active Data ServiceSata II hard drive. Nodes are interconnected with GigabitEthernet and are running Linux 2.6.32.5.1 Performance EvaluationTo evaluate the performance in terms of throughput, la-tency and overhead, we conduct a set of benchmarks basedon the version 0.1.2 of the prototype1.Throughput is measured as the number of transitions thatActive Data is able to handle per second. In order to stressthe system we run one Active Data server and a varyingnumber of clients (one per cluster node), which publish 10,000transitions in a loop without a pause between the iterations.Figure: Average number of transitions per second handled by the Active Data ServiceLatencymed 90thcentile std devLocal 0.77 ms 0.81 ms 18.68 msEth. 1.25 ms 1.45 ms 12.97 msOverhead Eth.w/o AD with AD38.04 s 40.6 s (4.6%)Table: Latency in milliseconds for life cycle creation and transition publication andoverhead measured using BitDew file transfers (1K files) with and without Active Data.G. Fedak (INRIA/LIP) Active Data Avalon 15 / 22
  • Amazon S3 CacheScenario: Caching Amazon S3• evaluates the ability to rapidlyprototype a data managementapplication using Active Dataand BitDew.• Write-through cache policyexpressed using life-cycletransitions.• evaluated using a pseudomaster-worker application (10nodes, 200MB input)w cache w/o cache DifferenceIn 2350 Mb 2350 Mb 0 MbOut 0.15 Mb 1976.17 Mb 1976.02 Mb#Put 13 13 0#Get 0 20 20Dollars 0.3 0.53 0.23Integration with Data ManagemeCreated t1 To Placet2Deletedt3Placedt4Loopt52t6t7Lostt8t9(a) Bitdew SchedulerCreatedt1Readyt2Startedt3Completedt4Deleted Invalidt5t6t7t8(b) Bitdew File TransferIN CREATEt1IN MOVEDtIN CLOSE Wt5tt9t11DELETEt14CREATEDt12(c) inoFigure 3: Data life cycle models for four datastructed from its documentation.Reading the source code of BitDew, we observe that dataitems are managed by instances of the Data class, and thisclass has the status variable which holds the data itemstate. Therefore, we simply deduce from the enumeration ofthe possible value of status the set of corresponding placesin the Petri Net (see Figure 2a and 2b). By further analyz-ing the source code, we construct the model and summarizehow high level DLM features are modelized using ActiveData model:• BitDew (INRIA), programmable environment fordata management.• inotify Linux kernel subsystem: notification systemfor file creation, modification, write, movement anddeletion.• iRODS (DICE, Univ. North Carolina), rule-orienteddata management system.Figure: BitDew File Transfer• t1 If cache hit, the handlerserves the file from the cache;if cache miss, the handler getsthe file from Amazon S3• t4 the handler transfers thedata item to Amazon S3 +cache eviction policy.G. Fedak (INRIA/LIP) Active Data Avalon 16 / 22
  • Distributed Sensor Networks• evaluates the ability to developdistributed application basedon data life-cycle.• sensors: images acquisition,pre-processing, archiving tocentralized remote storage• use Active Data to implementdistributed data throttling123456789100 30 60 90 120 150 180 210TIF download Convertion to JPG JPG uploadIntegration with Data Management SystemsCreated t1 To Placet2Deletedt3Placedt4Loopt52t6t7Lostt8t9(a) Bitdew SchedulerCreatedt1Readyt2Startedt3Completedt4Deleted Invalidt5t6t7t8(b) Bitdew File TransferIN CREATEt1IN MOVED FROMt2t3IN MOVED TOt4IN CLOSE WRITEt5t6t7 t8t9 t10t11DELETEDt14t13CREATEDt12(c) inotifyGetPutCreatedt5t6t7t8Deleted(d) iRODSFigure 3: Data life cycle models for four data management system.structed from its documentation.Reading the source code of BitDew, we observe that dataitems are managed by instances of the Data class, and thisclass has the status variable which holds the data itemstate. Therefore, we simply deduce from the enumeration ofthe possible value of status the set of corresponding places• BitDew (INRIA), programmable environment fordata management.• inotify Linux kernel subsystem: notification systemfor file creation, modification, write, movement anddeletion.• Data schedulinreplication• Fault tolerance• t12: we check if the transition islocal or remote: if it is remotewe increment the local counterp.• t5: if the transition is local, weupload the file only if p < n.G. Fedak (INRIA/LIP) Active Data Avalon 17 / 22
  • Incremental MapReduceScenario: Incremental MapReduce• we investigate if an existingsystem can be optimized byleveraging on Active Data’sability to cope with dynamicdata.• Making MapReduceincremental : re-run map andreduce tasks only for the datainput chunks that havechanged• Evaluated using the wordcount application (10 mappers,5 reducers, 3.2 GB split in 200chunks)• Implemented usingBitDew-MapReduce• Active Data notifies that a filetransfer is modifying an inputchunk. The Mapper Transitionhandler flags the chunk asdirty.• the mapper executes again themap task on dirty chunk andsends the updatedintermediate results to thereducers.• Reducers proceed as usual tocompute again the final result.Fraction modified 20% 40% 60% 80%Update time 27% 49% 71% 94%G. Fedak (INRIA/LIP) Active Data Avalon 18 / 22
  • Data ProvenanceScenario: Data Provenance• take advantage Active Data’sunified view of data-sets overheterogeneous systems.• file transfers service (GlobusOnline) + metadata catalog(iRODS).• query iRods and obtain filetransfer information :endpoints, completion date,request time• To compose the two life cycles, the placeSUCCEEDED from Globus Online is a startplace which creates a token in the iRODSlife cycle model.n with Data Management SystemsPlaceacedLostt8t9dulerCreatedt1Readyt2Startedt3Completedt4Deleted Invalidt5t6t7t8(b) Bitdew File TransferIN CREATEt1IN MOVED FROMt2t3IN MOVED TOt4IN CLOSE WRITEt5t6t7 t8t9 t10t11DELETEDt14t13CREATEDt12(c) inotifyGetPutCreatedt5t6t7t8Deleted(d) iRODSCreatedt1 t2Succeeded Failedt3 t4Deleted(e) Globus OnlineFigure 3: Data life cycle models for four data management system.umentation.code of BitDew, we observe that datay instances of the Data class, and thiss variable which holds the data itemsimply deduce from the enumeration ofstatus the set of corresponding placesFigure 2a and 2b). By further analyz-A), programmable environment forment.ernel subsystem: notification systemn, modification, write, movement and• Data scheduling andreplication• Fault tolerance• t1 a handler to store the file in iRODS. Thetoken now contains Globus Online andiRods identifiers .• t5 the handler queries Globus Online to getfile transfer information and publish asiRods meta-data.G. Fedak (INRIA/LIP) Active Data Avalon 19 / 22
  • Outline1 IntroductionChallenges of Life-Cycle ManagementActive Data at a GlanceRelated Works2 The Model and ImplementationLife-Cycle ModelExecution Runtime3 ExperimentsPerformance EvaluationUse Cases4 ConclusionG. Fedak (INRIA/LIP) Active Data Avalon 20 / 22
  • ConclusionActive Data, a programming model for supporting data life cycle managementapplications• formal definition of the data life cycle allows unified view of data across heterogeneous systems andinfrastructures• runtime execution environment + API to publish transitions and execute transition handlers,• low overhead (< 4%) and high transition throughput (> 30K/sec)• ability to plug into legacy systems (data scheduler, file system, file transfer service, rule-basedmanagement system)• Evaluated with several complex use cases.Future Works:• Model: advanced representation of computations; collective operations on data sets.• Runtime: rollback mechanisms for fault-tolerant execution; distributed implementations of thepublish/subscribe substrate.• Application leveraging Active Data: a MapReduce runtime for dynamic data, incremental andasynchronous workflow (Swift/Mosastore)G. Fedak (INRIA/LIP) Active Data Avalon 21 / 22
  • Thank you !G. Fedak (INRIA/LIP) Active Data Avalon 22 / 22