Telegraph Cq English

1,171 views

Published on

Workshop on TelegraphCQ:
Concept of Data Stram Management System.
TelegraphCQ: the DSMS developped at Berkley, internal architecture.
Differences between tradition database.
Adaptive QUery Processing using the new concept of Eddies like a routing operator.
Troubles about join Streams (with no statistical data) and Relations; and the two solution: STAIR and SteMs.
STAIR: a join operator that allow internal state changing using primite function visible to eddies.
SteMs: helf-join operator that keep homogeneous tuples, internal state is decision-indipendent.
Eddies Routing Policy implemented with the (Waldspurger & Weihl [1994]) Lottery Scheduling.

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,171
On SlideShare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
0
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide
  • DBMS Open Source, PostgreSQL, da cui partire per implementare TelegraphCQ.Sviluppato alla Berkeley UniversityScritto in C/C++Licenza OpenBSDBasato sul codice di PostgreSQLVersione corrente: 2.1 su PostgreSQL 7.3.2Progetto chiuso nel 2006Importanti punti di interesse e caratteristicheSoftware http://telegraph.cs.berkeley.eduPapers http://db.cs.berkeley.edu/telegraphSpinoff commerciale Truviso
  • Versioni non bloccanti di operatori classici (sel, proj)Eddies: decidono routing tupla per tuplaFlux instradano le tuple fra le macchine di un cluster per supportare il parallelismo, il bilanciamento del carico e la tolleranza ai guasti
  • Le sorgenti devono identificarsi prima di inviare datiWrapper: funzioni user-definedCome devono essere processati i datiAll’interno del Wrapper Clearinghouse processPush sourcesIniziano una connessione con TelegraphCQPull sourcesIl wrapper inizia la connessionePull, per esempio si connette ad un server mail e controlla la posta ogni minutoDati da Wrapper differenti possono confluire nello stesso streamHeartbeat: tupla Punctuated senza dati, solo timestamp Quando arriva una tupla Punctuated non arriveranno dati antecedenti
  • I dati arrivati possono essere solo parti di tuple quindi è necessario bufferizzarle, ad ogni chiamata non è detto che avremo una tuplaWrapperState serve per fare comunicare funzioni utente con il WCH,Se arrivano meno campi saranno di default a NULL, e se ne arrivano troppi saranno troncati
  • Una sorta di repository temporaneo di datiOperatore di Half-Join che memorizza tuple omogeneeStatoindipendentedalle precedenti decisioni di routing (poiché non memorizza le tuple)Supporta le operazioniInserimento (build)Ricerca (probe)Cancellazione (eviction) utili per le windowSimili a Mjoin ma più adattiviSimile alla facile routing policy con le query con solo le selezioniSharing dello stato tra altre query continueNon memorizza i risultati intermediAumento del costo di computazione
  • The opportunity to impove:Optimizers pick a single plan for a queryHowever, different subsets of data may have very different statistical propertiesMay be more efficient to use different plans for different subsets of data
  • Telegraph Cq English

    1. 1. Berkley’sTelegraphCQ<br />Details about TelegraphCQ<br />Adaptivity: Eddies, SteMs and STAIR<br />Routing Policy: Lottery Scheduling<br />1<br />Friday, July 15, 2011<br />Alberto Minetti<br />Advanced Data Management @ DISI<br />Università degli Studi di Genova<br />
    2. 2. Data Stream Management System<br />Continuous, unbounded, rapid, time-varying streams of data elements<br />Occur in a variety of modern applications<br />Network monitoring and traffic engineering<br />Sensor networks, RFID tags<br />Telecom call records<br />Financial applications<br />Web logs and click-streams<br />Manufacturing processes<br />DSMS = Data Stream Management System<br />2<br />
    3. 3. The Begin:Telegraph<br />Several continuous queries <br />Several data streams<br />At the beginning in Java<br />Then C-based using PostgreSQL<br />No Distribuite Scheduling<br />Level of adaptivity doesn’t change against overloading<br />Ignore system resources<br />Data managemente fully in memory<br />3<br />
    4. 4. ReDesign:TelegraphCQ<br />Developed byBerkeley University<br />Written in C/C++<br />OpenBSD License<br />Based on PostgreSQL sources<br />Current Version: 0.6 (PostgreSQL 7.3.2)<br />Closed Project in 2006<br />Several points of interest and features<br />Software http://telegraph.cs.berkeley.edu<br />Papers http://db.cs.berkeley.edu/telegraph<br />Commercial spinoff Truviso<br />4<br />
    5. 5. TelegraphCQ Architecture<br />PostgreSQL backends<br />Many TelegraphCQ front-end<br />Only one TelegraphCQ back-end<br />Front-End<br />Fork for every client connection<br />Doesn’t hold streams<br />Parsing continuous query in the shared memory<br />Back-End has an Eddy<br />Joins query plans together<br />Can be shared among queries<br />Put results in the shared memory<br />5<br />
    6. 6. TelegraphCQ Architecture<br />Shared Memory<br />Query Plan Queue<br />TelegraphCQBack End<br />TelegraphCQBack End<br />TelegraphCQ Front End<br />Eddy Control Queue<br />Planner Parser Listener<br />Modules<br />Modules<br />Query Result Queues<br />Split<br />Mini-Executor<br />Proxy<br />CQEddy<br />CQEddy<br />}<br />Split<br />Split<br />Catalog<br />Scans<br />Scans<br />Shared Memory Buffer Pool<br />Wrappers<br />TelegraphCQ <br />Wrapper <br />ClearingHouse<br />Disk<br />6<br />Taken from Michael Franklin, UC Berkeley<br />
    7. 7. Tipologie di Moduli<br />Input and Caching (Relations and Streams)<br />Interface to external datasource<br />Wrapper for HTML, XML, FileSystem, proxy P2P<br />Remote Databases with caching support<br />Query Execution<br />Non-blocking of classical operators (sel, proj)<br />SteMs, STAIRs<br />Adaptive Routing<br />Reoptimize plan during execution<br />Eddies: choose routing tupla per tupla<br />Juggle: ordering on the fly (per value or timestamp),<br />Flux: routing among computer of a cluster<br />7<br />Fjords Framework<br />
    8. 8. Fjords<br />Framework inJava for Operators on Remote Data Streams<br />Interconnect modules<br />Support queues among modules<br />Non-blocking<br />Support for Relazions and Streams<br />8<br />
    9. 9. Stream in TelegraphCQ<br />Unarchived Stream<br />Never written on disk<br />In shared memory between executor and wrapper<br />Archived Stream<br />Append-only method to send tuples to system<br />No update, insert, delete; query aggregate with window<br />tcqtimeof type TIMESTAMP for window queries<br />With constraint TIMESTAMPCOLUMN<br />9<br />
    10. 10. DDL: Create Stream<br />10<br /> CREATE STREAM measurements (<br /> tcqtime TIMESTAMP TIMESTAMPCOLUMN, <br /> stationid INTEGER, <br /> speed REAL) <br /> TYPE ARCHIVED;<br /> CREATE STREAM tinydb (<br /> tcqtime TIMESTAMP TIMESTAMPCOLUMN,<br /> light REAL,<br /> temperature REAL)<br /> TYPE UNARCHIVED;<br />DROP STREAM measurements;<br />
    11. 11. Acquisizione di Dati<br />Sources must identify before sending datas<br />Wrapper: user-defined functions<br />How process datas<br />Inside Wrapper Clearinghouse process<br />Push sources<br />Begin a connection to TelegraphCQ<br />Pull sources<br />Thewrapper begin the connection<br />Different Wrapper Data can merge in the same stream<br />Heartbeat: Punctuated tuple without datas, only timestamp<br />When see a Punctuatedtuple no prior datas will come<br />11<br />
    12. 12. Wrappers nel WCH<br />Non-process-blocking over network socket (TCP/IP)<br />Wrapper funct called when there are datas on socket<br />Or when there are datas on buffer<br />If funct return a tuple for time (classic iterator)<br />Ritorns array ofPostgreSQL Datum<br />Init(WrapperState*) allocate resources and state<br />Next(WrapperState*) tuples are in the WrapperState<br />Done(WrapperState*) free resources and destroy state<br />All in the infrastructured memory of PostgreSQL<br />12<br />
    13. 13. DDL: Create Wrapper<br />13<br /> CREATE FUNCTION measurements_init(INTEGER)<br /> RETURNS BOOLEAN<br /> AS ‘libmeasurements.so,measurements_init’<br /> LANGUAGE ‘C’;<br /> CREATE WRAPPER mywrapper (<br />init=measurements_init,<br />next=measurements_next,<br />done=measurements_done);<br />
    14. 14. HtmlGet and WebQueryServer<br />HtmlGet allows the user to execute a series of Html GET or POST requests to arrive at a page, and then to extract data out of the page using a TElegraph Screen Scrapperdefinition file. Once extracted, this data can be output to a CSV file.<br />Welcome to the TESS homepage. TESS is the TElegraph Screen Scrapper. It is part of The Telegraph Project at UC Berkeley. TESS is a program that takes data from web forms (like search engines or database queries) and turns it into a representation that is usable by a database query processor.<br />http://telegraph.cs.berkeley.edu/tess/<br />14<br />
    15. 15. self-monitoring capability<br />Three special streams: info about system state<br />Support tointrospettive queries<br />Dynamic catalog<br />Queried as an any stream<br />tcq_queries(tcqtime, qrynum, qid, kind, qrystr)<br />tcq_operators(tcqtime, opnum, opid, numqrs, kind, qid, opstr, opkind, opdesc)<br />tcq_queues(tcqtime, opid, qkind, kind)<br />15<br />
    16. 16. Example<br />16<br />Welcome to psql 0.2, the PostgreSQL interactive terminal.<br /># CREATE SCHEMA traffic;<br /># CREATE STREAM traffic.measurements<br />(stationid INT,<br /> speed REAL,<br />tcqtime TIMESTAMP TIMESTAMPCOLUMN<br />) TYPE ARCHIVED;<br /># ALTER STREAM traffic.measurements ADD WRAPPER csvwrapper;<br />$ cat tcpdump.log | source.pl localhost 5533 csvwrapper,network.tcpdum<br />Default TelegraphCQ script written in Perl to simulate sources that send CSV datas<br />Default Port<br />
    17. 17. Load Shedding<br /> CREATE STREAM … TYPE UNARCHIVED ON OVERLOAD ????<br />BLOCK: stop it(default)<br />DROP: drop tuples<br />KEEP<br />COUNTS: keep the count of dropped tuples<br />REGHIST: build a fixed-grid histog. of shedded tuples<br />MYHIST: build a MHIST (multidimensional histog.)<br />WAVELET wavelet params<br />Build a wavelet histogram<br />SAMPLE: keep a Reservoir Sample<br />17<br />
    18. 18. LoadShedding:SummaryStream<br />For a stream named schema.stream<br />Automatically created two streams<br />schema.__stream_dropped<br />schema.__stream_kept<br />For WAVELET, MYHIST, REHIST, COUNTS<br />Schema contains:<br />Summary datas<br />Summary interval<br />For SAMPLE<br />Same schema with column __samplemult<br />Keep the number of effective tuples rappresented<br />18<br />
    19. 19. Quering TelegraphCQ:StreaQuel<br />19<br />Continuous Query for:<br />Standard Relations inherit from PostgreSQL<br />Data stream windowed (sliding, hopping, jumping)<br />RANGE BY specify the window dimension<br />SLIDE BY specify the update rate<br />START AT specify whenthe query will begin<br />optional<br />SELECT stream.color, COUNT(*)<br />FROM stream [RANGE BY ‘9’ SLIDE BY ‘1’]<br />GROUP BY stream.color<br />window<br />START OUPUT!<br />1<br />1<br />1<br />1<br />1<br />2<br />2<br />1<br />2<br />1<br />2<br />2<br />1<br />2<br />1<br />2<br />1<br />2<br />1<br />Adapted from Jarle Søberg<br />
    20. 20. Quering TelegraphCQ:StreaQuel (2)<br />wtime(*) returns the last timestamp in the window<br />Recoursive query using WITH [SQL 1999]<br />StreaQuel doesn’t allow subqueries<br />20<br />SELECT S.a, wtime(*)<br />FROM<br /> S [RANGE BY ’10 seconds’ SLIDE BY ’10 second’],<br /> R [RANGE BY ’10 seconds’ SLIDE BY ’10 second’]<br />WHERE S.a = R.a;<br />Data Stream<br />Window<br />…<br />10 seconds<br />10 seconds<br />
    21. 21. Net Monitor Windowed Query<br />21<br />SELECT (CASE when outgoing = true<br /> then src_ip else dst_ip end) as inside_ip ,<br /> (CASE when outgoing = true<br /> then dst_ip else src_ip end) as outside_ip,<br />sum(bytes_sent) + sum(bytes_recv) as bytes<br />FROM flow [RANGE BY $w SLIDE BY $w]<br />GROUP BY inside_ip, outside_ip<br />All active connections<br />SELECT src_ip, wtime(*),<br />COUNT(DISTINCT(dst_ip||dst_port)) AS fanout,<br />FROM flow [RANGE BY $w SLIDE BY $w]<br />WHERE outgoing = false<br />GROUP BY src_ip ORDER BY fanout DESC<br />LIMIT 100 PER WINDOW;<br />100 sources with the max number of connections<br />SELECT sum(bytes_sent), src_ip, wtime(*) AS now<br />FROM flow [RANGE BY $w SLIDE BY $w]<br />WHERE outgoing = false<br />GROUP BY src_ip ORDER BY sum(bytes_sent) DESC<br />LIMIT 100 PER WINDOW;<br />100 most significant sources of traffic<br />Taken from: İlkay Ozan Kay<br />
    22. 22. Evolutionary<br />Revolutionary<br />Adaptive Query Processing:Evolution<br />Static Late Inter Intra Per<br />Plans Binding Operator Operator Tuple<br />Traditional Dynamic QEP Query Scrambling Xjoin, DPHJ, Eddies<br />DBMS Parametric Mid-query Reopt. Convergent QP <br /> Competitive Progressive Opt.<br />Taken from: AmolDeshpande, Vijayshankar Raman<br />22<br />
    23. 23. Adaptive Query Processing:Current Background<br />Several plans<br />Parametric Queries<br />Continuous Queries<br />Focus on incremental output<br />Complex Queries (10-20 relazions in join)<br />Data Stream and asyncronous datas<br />Statistics not available<br />Interactive Queries and user preferences<br />Sharing of the state<br />Several operators<br />XML data and text<br />Wide area federations<br />23<br />
    24. 24. Adaptive Query Processing:System R<br />Repeat<br />Observate environment: daily/weekly (runstats)<br />Choose behaviour (optimizer)<br />If current plan is not the best plan (analyzer)<br />Actuate the new plan (executor)<br />Cost-based optimization<br />Runstats-optimize-execute -> high coarseness<br />Weekly Adaptivity!<br />Goal: adaptivity per tuple<br />Merge the 4 works<br />Measure<br />Actuate<br />Analyze<br />Plan<br />Taken from: Avnur, Hellerstein<br />24<br />
    25. 25. TelegraphCQ Executor:Eddy<br />Idea taken from fluid mechanic<br />continuously adaptive query processing mechanism<br />Eddy is a routing operator<br />Delineate which modules must visit before<br />After a tuple visit all modules, it can output<br />See tuples before and after each module (operator)<br />25<br />Measure<br />Eddy<br />Analyze<br />Actuate<br />Plan<br />Taken from: Amol Deshpande, Vijayshankar Raman<br />
    26. 26. Eddies:Correctness<br />Every tuple has two BitVector<br />Every position correspond to an operator<br />Ready: identify if tuple is ready for that operator<br />Eddy can delineate which tuples send to an operator<br />Done: identify if tuple was processed by that operator<br />Eddy avoids sending two times to same operator<br />When all Done bits are setted -> output<br />joined tuple have Ready and Donein bitwise-OR<br />Simple selections have Ready=complement(Done)<br />26<br />
    27. 27. Eddies:Simple Selection Routing<br />27<br />S<br />SELECT *FROM SWHERE S.a > 10 AND S.b < 15<br /> AND S.c = 15<br />σa<br />σb<br />S.b < 15 <br />S.a > 10<br />Eddy<br />σc<br />S.c = 15 <br />σaσbσc<br />σaσbσc<br />15 ; 0 ; 15<br />S1<br />0 0 0<br />1 1 1<br />1 0 0<br />01 1<br />1 1 0<br />0 0 1<br />1 1 1<br />0 0 0<br />a<br />15<br />b<br />0<br />Ready<br />Done<br />c<br />15<br />Adapted from Manuel Hertlein<br />
    28. 28. S >< T<br />Relation Binary Join:R >< S >< T<br />Join order<br />(R >< S) >< T<br />Alright with direct access to datas (index or seq.)<br />28<br />R >< S<br />T<br />R<br />S<br />Taken from Jarle Søberg<br />
    29. 29. Stream Binary JoinR >< S >< T<br />29<br />Join order<br />(R >< S) >< T<br />But if data are sent by sources...<br />S >< T<br />R >< S<br />Block or drop some tuples is inevitable!<br />Taken from Jarle Søberg<br />
    30. 30. On the fly optimization necessary<br />S >< T<br />30<br />Stream Binary JoinR >< S >< T<br />Often stream changes<br />Reoptimize require lot of time<br />Non dynamic enough!<br />R >< S<br />Taken from Jarle Søberg<br />
    31. 31. Stream Binary Join:Eddies<br />Using an Eddy<br />Initial behaviour of Telegraph<br />31<br />S >< T<br />eddy<br />R >< S<br />tuple-based adaptivity<br />Consider dynamic changes in the stram<br />Taken from Jarle Søberg<br />
    32. 32. Eddies:Sheduling Join problem<br />Sheduling on selectivity of joins doesn’t work<br />Example<br />32<br />|S E|<br />|EC|<br />>><br />E and Carrive early; Sis delayed<br />S<br />E<br />C<br />time<br />Taken from Amol Deshpande<br />
    33. 33. 33<br />|S E|<br />|E C|<br />SE<br />HashTable<br />E.Name<br />HashTable<br />S.Name<br />Eddy<br />S<br />E<br />Output<br />C<br />HashTable<br />C.Course<br />HashTable<br />E.Course<br />Eddy decides to route E to EC<br />EC<br />>><br />E and Carrive early; Sis delayed<br />S0<br />sent and received suggest<br />S Join E is better option<br />S<br />S<br />E<br />S0<br />S –S0<br />E<br />C<br />time<br />C<br />SE<br />S0E<br />(S –S0)E<br />Eddy learns the correct sizes<br />Too Late !!<br />Taken from Amol Deshpande<br />
    34. 34. 34<br />State got embedded as a<br />result of earlier routing <br />decisions<br />|S E|<br />|EC|<br />SE<br />HashTable<br />E.Name<br />HashTable<br />S.Name<br />EC<br />Eddy<br />S<br />E<br />Output<br />C<br />HashTable<br />C.Course<br />HashTable<br />E.Course<br />SE<br />EC<br />>><br />E and Carrive early; Sis delayed<br />S<br />E<br />C<br />C<br />SE<br />S<br />E<br />Execution Plan Used<br />Query is executed using the worse plan!<br />Too Late !!<br />Taken from Amol Deshpande<br />
    35. 35. STAIRStorage, Transformation and Access for Intermediate Results<br />35<br />S.Name STAIR<br />Build into<br />S.Name STAIR<br />HashTable<br />E.Name STAIR<br />HashTable<br />Eddy<br />S<br />Output<br />E<br />C<br />HashTable<br />HashTable<br />E.Course STAIR<br />C.Course STAIR<br />Probe into <br />E.Name STAIR<br />Show internal state of join to eddy<br />Provide primitive function to state management<br />Demotion<br />Promotion<br />Operation for<br />Insertion (build)<br />Lookup (probe)<br />s1<br />s1<br />s1<br />s1<br />Taken from Amol Deshpande<br />
    36. 36. STAIR: Demotion<br />36<br />e1<br />e1<br />e2c1<br />e2<br />s1e1<br />e2c1<br />e2<br />s1e1<br />S.Name STAIR<br />HashTable<br />E.Name STAIR<br />s1<br />Demoting di e2c1ae2:<br />Simple projection<br />HashTable<br />e1<br />e2c1<br />e2<br />Eddy<br />S<br />s1e1<br />E<br />Output<br />C<br />HashTable<br />e2<br />s1e1<br />e2c1<br />HashTable<br />c1<br />Can be tought of as undoing work<br />E.Course STAIR<br />C.Course STAIR<br />Adapted from Amol Deshpande<br />
    37. 37. STAIR: Promotion<br />37<br />Promotinge1 using EC<br />e1<br />e1<br />e1c1<br />e1<br />e1c1<br />S.Name STAIR<br />HashTable<br />E.Name STAIR<br />s1<br />Two arguments:<br /><ul><li>A tuple
    38. 38. A join to be used to promote this tuple</li></ul>HashTable<br />e1<br />e1c1<br />e2c1<br />Eddy<br />S<br />E<br />Output<br />C<br />HashTable<br />e2<br />s1e1<br />HashTable<br />c1<br />e1<br />Can be tought of as precomputation of work<br />E.Course STAIR<br />C.Course STAIR<br />Adapted from Amol Deshpande<br />
    39. 39. Demotion OR Promotion<br />38<br />Taken from Lifting the Burden of History from Adaptive Query Processing<br />Amol Deshpande and Joseph M. Hellerstein<br />
    40. 40. Demotion AND Promotion<br />Taken from Lifting the Burden of History from Adaptive Query Processing<br />Amol Deshpande and Joseph M. Hellerstein<br />39<br />
    41. 41. 40<br />S.Name STAIR<br />HashTable<br />|S E|<br />|EC|<br />S0<br />E<br />E<br />E<br />HashTable<br />E<br />E<br />E<br />C<br />C<br />C<br />S0E<br />Eddy decides to route E to EC<br />E.Course STAIR<br />>><br />E and Carrive early; Sis delayed<br />S0<br />E.Name STAIR<br />HashTable<br />S<br />E<br />E<br />Eddy<br />S<br />C<br />E<br />Output<br />C<br />time<br />E<br />HashTable<br />C<br />Eddy decides to migrateE<br />Eddy learns the correct selectivities<br />By promoting E using EC<br />C.Course STAIR<br />Adapted from Amol Deshpande<br />
    42. 42. 41<br />|S E|<br />|EC|<br />HashTable<br />E<br />C<br />S0E<br />E.Course STAIR<br />>><br />E and Carrive early; Sis delayed<br />S.Name STAIR<br />HashTable<br />S<br />S0<br />E.Name STAIR<br />HashTable<br />S<br />S –S0<br />S –S0<br />E<br />Eddy<br />S<br />C<br />(S –S0) EC<br />E<br />Output<br />C<br />time<br />E<br />HashTable<br />C<br />C.Course STAIR<br />Adapted from Amol Deshpande<br />
    43. 43. 42<br />EC<br />SE<br />C<br />S – S0<br />SE<br />EC<br />S0<br />E<br />E<br />C<br />HashTable<br />E<br />C<br />SE<br />E.Course STAIR<br />S.Name STAIR<br />HashTable<br />S<br />E.Name STAIR<br />HashTable<br />UNION<br />Eddy<br />S<br />E<br />Output<br />C<br />E<br />HashTable<br />C<br />Most of the data is<br />processed using the<br />correct plan<br />C.Course STAIR<br />Adapted from AmolDeshpande<br />
    44. 44. STAIR:Correctness<br />Theorem [3.1]: An eddy with STAIRs always produces the correct query result in spite of arbitrary applications of the promotion and demotion operations.<br />STAIRs will produce wvery resul tuple<br />There wull be no spurious duplicates<br />43<br />Taken from Lifting the Burden of History from Adaptive Query Processing<br />Amol Deshpande and Joseph M. Hellerstein<br />
    45. 45. State Module<br />A kind of temporary data repository<br />Half-Join operator that keep homogeneous tuple<br />State inside the operators is Decisions-Indipendent<br />Support the operations<br />Insertion (build)<br />Look-up (probe)<br />Deletion (eviction) [useful for windows]<br />Similar to a Mjoin but more adaptive<br />Sharing of the state among other continuous queries<br />But not storing intermediate results<br />Increase the computation cost significant<br />44<br />
    46. 46. Eddies Join with SteMs<br />45<br />T<br />R<br />S<br />eddy<br />R<br />T<br />S<br />More adaptivity<br />Eddy knows half-join<br />Different access method<br />Index access<br />Scan access<br />Simulate several join<br />On overloading?<br />Hash join (fast!!)<br />Index join (mem limit)<br />Join familty?<br />Hash join (equi-join)<br />B-tree join (<, <=, >)<br />Parametrica query can be tought as a join<br />Adapted from Jarle Søberg<br />
    47. 47. SteMs:Correctness <br />46<br />R<br />S<br />Correctness problem!<br />Possibile duplicates!!<br />Global unique sequence number<br />Only younger can probe<br />Taken from Jarle Søberg<br />
    48. 48. SteMs sliding Window<br />47<br />SELECT * <br />FROM Str [RANGE BY 5 Second<br /> SLIDE BY 2 Second],<br /> Relation,<br />WHERE Relation.a = Str.a<br />A|…….|18:49:36<br />R<br />B|…….|18:49:36<br />C|…….|18:49:37<br />A|…….|18:49:38<br />Keep the state for the sliding window (eviction)<br />At time 40, what will happen at time 42?<br />Instead rebuild all hash table it remove only old tuples and add new tuples<br />B|…….|18:49:39<br />C|…….|18:49:39<br />A|…….|18:49:40<br />B|…….|18:49:40<br />C|…….|18:49:40<br />Eviction!18:49:37<br />A|…….|18:49:41<br />B|…….|18:49:41<br />
    49. 49. Binary Join, STAIR, SteMComparison<br />48<br />select *<br />from customer c, orders o, lineitem l<br />where c.custkey = o.custkey and<br /> o.orderkey = l.orderkey and<br /> c.nationkey = 1 and<br /> c.acctbal > 9000 and<br /> l.shipdate > date ’1996-01-01’<br />Necessary Recomputation<br />NO adaptive<br />lineitemcoming with ascending shipdate<br />Initial routing (O >< L) >< C<br />Taken from Lifting the Burden of History from Adaptive Query Processing<br />Amol Deshpande and Joseph M. Hellerstein<br />
    50. 50. Eddies:Routing Policy <br />How to choose the best plan? Using routing<br />Every tuple is routed individually<br />Routing policy estabilish thesystem efficiency<br />Eddy has a tuple buffer with priorità<br />Initially they have low priority<br />Exiting form an operator they have higher priority<br />A tuple is sent to output as early as possible<br />Avoid system memory congestion<br />Allow low memory consumption<br />49<br />
    51. 51. Eddies’ Routing Policy:(old) Back-Pressure<br />Approach Naive:<br />Quick operator before<br />50<br />sel(s1) = sel(s2)<br />cost(s2) = 5<br />cost(s1) changes<br />cost(s1) = cost(s2)<br />sel(s2) = 50%<br />sel(s1) changes<br />Taken from: Avnur, Hellerstein<br />
    52. 52. Eddies’ Routing Policy:Lottery Scheduling<br />Waldspurger& Weihlin1994<br />Algorithm to sheduling shared resources<br />« rapidlyfocus availableresources»<br />Every operator begin with N tickets<br />Operator receive another ticket when take one tuple<br />Promote operators which waste tuples fast<br />Operator lose a ticket when returns one tuples<br />Promote operators with lowselettività<br />low: Operators that returns few tuples after processing many<br />When two operators compete for a tuple<br />The tuple is assigned to lottery winner operator<br />Never let op without tickets + randomexploration<br />51<br />
    53. 53. Eddies’ Routing Policy:Lottery Scheduling<br />Lottery Scheduling is better than Back-Pressure<br />52<br />cost(s1) = cost(s2)<br />sel(s2) = 50%<br />sel(s1) varia<br />Taken from: Avnur, Hellerstein<br />
    54. 54. Experiment<br />53<br />Stream: x with mean 40 and standard deviation 10<br />
    55. 55. 54<br />Experiment: Stream variation<br />Stream: x with mean 10 and standard deviation 0<br />
    56. 56. 55<br />Experiment: Stream variation (2)<br />Stream: x with mean 10 and standard deviation 0<br />
    57. 57. Other Works<br />Distribuited Eddies<br />Freddies: DHT-Based Adaptive Query Processing via Federated Eddies<br />Content-based Routing<br />Partial Results for Online Query Processing<br />Flux: An Adaptive Partitioning Operator for Continuous Query Systems<br />Java Support for Data-Intensive Systems: Experiences Building the Telegraph Dataflow System<br />Ripple Join for Online Aggregation<br />Highly Available, Fault-Tolerant, Parallel Dataflows<br />56<br />
    58. 58. Bibliography<br />TelegraphCQ: An Architectural Status Report<br />Continuous Dataflow Processing for an Uncertain World<br />Enabling Real-Time Querying of Live and Historical Stream Data<br />Declarative Network Monitoring with an Underprovisioned Query Processor<br />Lifting the Burden of History from Adaptive Query Processing [STAIRs]<br />Eddies: Continuously Adaptive Query Processing<br />Using State Modules for Adaptive Query Processing<br />E altri… http://telegraph.cs.berkeley.edu/papers.html<br />Telegraph team @ UC Berkley: Mike Franklin, Joe Hellerstein, Bryan Fulton, Sirish Chandrasekaran, Amol Deshpande, Ryan Huebsch, Edwin Mach, Garrett Jacobson,Sailesh Krishnamurthy, Boon Thau Loo, Nick Lanham, Sam Madden, Fred Reiss, Mehul Shah, Eric Shan, Kyle Stanek, Owen Cooper, David Culler, Lisa Hellerstein, Wei Hong, Scott Shenker, Torsten Suel, Ion Stoica, Doug Tygar, Hal Varian, Ron Avnur, David Yu Chen, Mohan Lakhamraju, Vijayshankar Raman<br />Lottery Scheduling: Flexible Proportional-Share Resource Management<br />Carl A. Waldspurger & William E. Weihl @ MIT <br />57<br />

    ×