BIRTE Panel at VLDB: Are we solving the core Problems in stream processing?

•

1 like•355 views

This presentation highlight recent developments in the Apache Flink community and recent related research publications. The presentation was held at the panel discussion of the BIRTE workshop at VLDB 2018 in Rio de Janeiro. More information can be found on http://db.cs.pitt.edu/birte2018/ Panel Title: Are we making any attempts towards solving the hardest problems in stream processing today? Panel Abstract: Most of today’s Internet applications are data-centric and generate vast amounts of data that needs to be processed and analyzed for detailed reporting, enhancing user experience and increasing monetization. Streaming data processing systems must be designed based on a varying set of requirements. The list of requirements can be categorized based on different properties of such systems: 1. Consistency: Does every record in the input (or equivalently an input event) need to be committed exactly-once or at-least-once or at-most-once to the output? Is the event committed atomically or eventually to all outputs? 2. Scale: How many events per second can the system process? Tens of events per second? Or Thousands? Millions? Billions or even more? Does the system auto-scale to a new workload? 3. Failure Resilience: What kind of failures is the system resilient to? Machine-level or partial datacenter-level or entire datacenter-level? Is it enough to ensure that the data processing system itself is failure-resistant? Does the output need to be stored in globally consistent way? Is the system resilient to a bug in input data, a bug in user’s business logic, etc? 4. Latency: How long does it take every event from the time it is generated to the time it is committed? Milliseconds or seconds or minutes or hours or days? Should we target SLOs for median latencies or 90th percentile or higher tail latencies? 5. Expressiveness: What kind of operations can the user express in the system? From simple stateless operations (e.g. filter) to complex joins or stateful operations (e.g. HAVING clause in SQL)? How flexible is the system to add more input sources and output sinks? 6. Cost: This includes not only hardware cost (CPU, RAM, Disk, network, etc) but also engineering design complexity, cost of production support to run as a service and providing SLOs for latency / completeness, etc. From a pure business perspective, all this cost needs to be justified by the value the end user gets. 7. Service: Does the system run as a service for the users? Multi-tenant? What kind of isolation (e.g. performance, security, etc) is provided amongst users? How is business logic isolated from infrastructure? How easy is it for users to modify business logic in a self-service way? Lots of systems provide a lambda architecture: Use stream processing for best-effort (approximate) analysis, and use batch processing (e.g. daily) for strong consistency, high reliability, etc. This represents an easy way out. But is it the right thing to do? [...]

Science

1 Jonas Traub, BIRTE @ VLDB, 20181 Jonas Traub, BIRTE @ VLDB, 2018
Are we solving the core problems
in stream processing?
Jonas Traub
Technische Universität Berlin / DFKI IAM
www.dima.tu-berlin.de | jonas.traub@tu-berlin.de
Panel Discussion with:
Manpreet Singh (Google)
Karthik Ramasamy (Stremlio)
C. Mohan (IBM)
Badrish Chandramouli (Microsoft)
Neng Lu (Twitter)
Alok Pareek (Striim)
Jonas Traub (TU-Berlin)

2 Jonas Traub, BIRTE @ VLDB, 20182 Jonas Traub, BIRTE @ VLDB, 2018
Are we solving the core problems
in stream processing?
Jonas Traub
Technische Universität Berlin / DFKI IAM
www.dima.tu-berlin.de | jonas.traub@tu-berlin.de

3 Jonas Traub, BIRTE @ VLDB, 20183 Jonas Traub, BIRTE @ VLDB, 2018
Are we solving the core problems
in stream processing?
Yes, we do!

4 Jonas Traub, BIRTE @ VLDB, 20184 Jonas Traub, BIRTE @ VLDB, 2018
Are we solving the core problems
in stream processing?
Yes, we do!
Apache Flink and
its success story
What are the core problems
and how are we solving them?
Examples

5 Jonas Traub, BIRTE @ VLDB, 2018
5
5 Jonas Traub, BIRTE @ VLDB, 2018
Apache Flink Timeline

6 Jonas Traub, BIRTE @ VLDB, 2018
6
6 Jonas Traub, BIRTE @ VLDB, 2018

7 Jonas Traub, BIRTE @ VLDB, 2018
Apache Flink - Stateful Computations over Data Streams
source: flink.apache.org
• Event-driven Applications
• Stream & Batch Analytics
• Data Pipelines & ETL
• Exactly-once state consistency
• Event-time processing
• Sophisticated late data handling
• Scale-out architecture
• Support for very large state
• Incremental checkpointing

8 Jonas Traub, BIRTE @ VLDB, 20188 Jonas Traub, BIRTE @ VLDB, 2018
Examples:
What are core problems and
how are we solving them?

9 Jonas Traub, BIRTE @ VLDB, 20189 Jonas Traub, BIRTE @ VLDB, 2018
Examples:
Expressiveness: Event-time processing and sophisticated late data handling
The Dataflow Model: A Practical Approach to Balancing Correctness, Latency, and Cost in Massive-
Scale, Unbounded, Out-of-Order Data Processing (Akidau et al.)

10 Jonas Traub, BIRTE @ VLDB, 201810 Jonas Traub, BIRTE @ VLDB, 2018
Examples:
Expressiveness: Event-time processing and sophisticated late data handling
Service: Common APIs and feature sets
The Dataflow Model: A Practical Approach to Balancing Correctness, Latency, and Cost in Massive-
Scale, Unbounded, Out-of-Order Data Processing (Akidau et al.)
Apache Beam: An advanced unified programming model
“Implement batch and streaming data processing jobs that run on any
execution engine. (beam.apache.org)”

11 Jonas Traub, BIRTE @ VLDB, 201811 Jonas Traub, BIRTE @ VLDB, 2018
Examples:
Consistency: Exactly-once state consistency
Expressiveness: Event-time processing and sophisticated late data handling
Service: Common APIs and feature sets
Lightweight asynchronous snapshots for distributed dataflows
P Carbone, G Fóra, S Ewen, S Haridi, K Tzoumas
State management in Apache Flink: consistent stateful distributed stream processing
P Carbone, S Ewen, G Fóra, S Haridi, S Richter, K Tzoumas
The Dataflow Model: A Practical Approach to Balancing Correctness, Latency, and Cost in Massive-
Scale, Unbounded, Out-of-Order Data Processing (Akidau et al.)
Apache Beam: An advanced unified programming model
“Implement batch and streaming data processing jobs that run on any
execution engine. (beam.apache.org)”

More from Jonas Traub

Computing aggregates over windows is at the core of virtually every stream processing job. Typical stream processing applications involve overlapping windows and, therefore, cause redundant computations. Several techniques prevent this redundancy by sharing partial aggregates among windows. However, these techniques do not support out-of-order processing and session windows. Out-of-order processing is a key requirement to deal with delayed tuples in case of source failures such as temporary sensor outages. Session windows are widely used to separate different periods of user activity from each other. Current versions of Apache Flink use Window Buckets to process stream aggregations with session windows and out-of-order tuples. This Approach does not share partial aggregates among overlapping windows. In our talk, we present Scotty, a high throughput operator for window discretization and aggregation in Apache Flink. Scotty splits streams into non-overlapping slices and computes partial aggregates per slice. These partial aggregates are shared among all overlapping windows including session windows. Scotty introduces the first slicing technique which (1) enables stream slicing for session windows in addition to tumbling and sliding windows and (2) processes out-of-order tuples efficiently. Scotty was first published at ICDE 2018 (http://www.user.tu-berlin.de/powibol/assets/publications/traub-scotty-icde-2018.pdf).

Flink Forward 2018: Efficient Window Aggregation with Stream Slicing

Jonas Traub

This poster was presented at ICDE 2018. Abstract: Computing aggregates over windows is at the core of virtually every stream processing job. Typical stream processing applications involve overlapping windows and, therefore, cause redundant computations. Several techniques prevent this redundancy by sharing partial aggregates among windows. However, these techniques do not support out-of-order processing and session windows. Out-of-order processing is a key requirement to deal with delayed tuples in case of source failures such as temporary sensor outages. Session windows are widely used to separate different periods of user activity from each other. In this paper, we present Scotty, a high throughput operator for window discretization and aggregation. Scotty splits streams into non-overlapping slices and computes partial aggregates per slice. These partial aggregates are shared among all concurrent queries with arbitrary combinations of tumbling, sliding, and session windows. Scotty introduces the first slicing technique which (1) enables stream slicing for session windows in addition to tumbling and sliding windows and (2) processes out-of-order tuples efficiently. Our technique is generally applicable to a broad group of dataflow systems which use a unified batch and stream processing model. Our experiments show that we achieve a throughput an order of magnitude higher than alternative state-of-the-art solutions.

Scotty: Efficient Window Aggregation for Out-of-Order Stream Processing

Jonas Traub

Paper: Scalable Detection of Concept Drifts on Data Streams with Parallel Adaptive Windowing Abstract: Machine learning techniques for data stream analysis suffer from concept drifts such as changed user preferences, varying weather conditions, or economic changes. These concept drifts cause wrong predictions and lead to incorrect business decisions. Concept drift detection methods such as adaptive windowing (Adwin) allow for adapting to concept drifts on the fly. In this paper, we examine Adwin in detail and point out its throughput bottlenecks. We then introduce several parallelization alternatives to address these bottlenecks. Our optimizations lead to a speedup of two orders of magnitude over the original Adwin implementation. Thus, we explore parallel adaptive windowing to provide scalable concept detection for high-velocity data streams with millions of tuples per second.

Scalable Detection of Concept Drifts on Data Streams with Parallel Adaptive W...

Jonas Traub

This poster was presented at the 21st International Conference on Extending Database Technology (EDBT), March 26-29, 2018. Paper: Efficient SIMD Vectorization for Hashing in OpenCL Abstract: Hashing is at the core of many efficient database operators such as hash-based joins and aggregations. Vectorization is a technique that uses Single Instruction Multiple Data (SIMD) instructions to process multiple data elements at once. Applying vectorization to hash tables results in promising speedups for build and probe operations. However, vectorization typically requires intrinsics – low-level APIs in which functions map to processorspecific SIMD instructions. Intrinsics are specific to a processor architecture and result in complex and difficult to maintain code. OpenCL is a parallel programming framework which provides a higher abstraction level than intrinsics and is portable to different processors. Thus, OpenCL avoids processor dependencies, which results in improved code maintainability. In this paper, we add efficient, vectorized hashing primitives to OpenCL. Our results show that OpenCL-based vectorization is competitive to intrinsics on CPUs but not on Xeon Phi coprocessors.

Efficient SIMD Vectorization for Hashing in OpenCL

Jonas Traub

About the Workshop: The Stream Reasoning Workshop took place from January 16th to 17th, 2018. Processing, querying and reasoning over streaming data is studied in different communities such as KR&R, Semantic Web, Databases, Stream Processing, Complex Event Processing, etc., where researchers have different perspectives and face different challenges. This workshop aims at advancing Stream Reasoning as research theme by bringing together these different views and goals. In addition to invited talks, the workshop will provide opportunities for all participants to engage in discussions on open problems and future directions. (http://www.ifi.uzh.ch/en/ddis/events/streamreasoning2018.html) About the Talk: Real-time sensor data enables diverse applications such as smart metering, traffic monitoring, and sport analysis. In the Internet of Things, billions of sensor nodes form a sensor cloud and offer data streams to analysis systems. However, it is impossible to transfer all available data with maximal frequencies to all applications. Therefore, we need to tailor data streams to the demand of applications. We contribute a technique that optimizes communication costs while maintaining the desired accuracy. Our technique schedules reads across huge amounts of sensors based on the data-demands of a huge amount of concurrent queries. We introduce user-defined sampling functions that define the data-demand of queries and facilitate various adaptive sampling techniques, which decrease the amount of transferred data. Moreover, we share sensor reads and data transfers among queries. Our experiments with real-world data show that our approach saves up to 87% in data transmissions.

UZH Stream Reasoning Workshop 2018: Optimized On-Demand Data Streaming from S...

Jonas Traub

JT@UCSB - On-Demand Data Streaming from Sensor Nodes and A quick overview of ...

Jonas Traub

We present I², an interactive development environment for real-time analysis pipelines, which is based on Apache Flink and Apache Zeppelin. The sheer amount of available streaming data frequently makes it impossible to visualize all data points at the same time. I² coordinates running Flink jobs and corresponding visualizations such that only the currently depicted data points are processed in Flink and transferred towards the front end. We show how Flink jobs can adapt to changed visualization properties at runtime to allow interactive data exploration on high bandwidth data streams. Moreover, we present a data reduction technique which minimizes data transfer while providing loss free time-series plots. We show I² in a live demonstration in which we replay recorded sensor data from a football match (ca. 12k event/s). I² was first presented at EDBT'17 where it was awarded as best demonstration. The demonstration is available as open source at github.com/TU-Berlin-DIMA/i2.

I²: Interactive Real-Time Visualization for Streaming Data with Apache Flink ...

Jonas Traub

I²: Interactive Real-Time Visualization for Streaming Data

Jonas Traub

LWA 2015: The Apache Flink Platform (Poster)

Jonas Traub

LWA 2015: The Apache Flink Platform for Parallel Batch and Stream Analysis

Jonas Traub

More from Jonas Traub (10)

Flink Forward 2018: Efficient Window Aggregation with Stream Slicing

Scotty: Efficient Window Aggregation for Out-of-Order Stream Processing

Scalable Detection of Concept Drifts on Data Streams with Parallel Adaptive W...

Efficient SIMD Vectorization for Hashing in OpenCL

UZH Stream Reasoning Workshop 2018: Optimized On-Demand Data Streaming from S...

JT@UCSB - On-Demand Data Streaming from Sensor Nodes and A quick overview of ...

I²: Interactive Real-Time Visualization for Streaming Data with Apache Flink ...

I²: Interactive Real-Time Visualization for Streaming Data

LWA 2015: The Apache Flink Platform (Poster)

LWA 2015: The Apache Flink Platform for Parallel Batch and Stream Analysis

Recently uploaded

Mammals have dominated Earth for approximately 55 Myr thanks to their adaptations and resilience to warming and cooling during the Cenozoic. All life will eventually perish in a runaway greenhouse once absorbed solar radiation exceeds the emission of thermal radiation in several billions of years. However, conditions rendering the Earth naturally inhospitable to mammals may develop sooner because of long-term processes linked to plate tectonics (short-term perturbations are not considered here). In ~250 Myr, all continents will converge to form Earth’s next supercontinent, Pangea Ultima. A natural consequence of the creation and decay of Pangea Ultima will be extremes in pCO2 due to changes in volcanic rifting and outgassing. Here we show that increased pCO2, solar energy (F⨀; approximately +2.5% W m−2 greater than today) and continentality (larger range in temperatures away from the ocean) lead to increasing warming hostile to mammalian life. We assess their impact on mammalian physiological limits (dry bulb, wet bulb and Humidex heat stress indicators) as well as a planetary habitability index. Given mammals’ continued survival, predicted background pCO2 levels of 410–816 ppm combined with increased F⨀ will probably lead to a climate tipping point and their mass extinction. The results also highlight how global landmass configuration, pCO2 and F⨀ play a critical role in planetary habitability.

Climate extremes likely to drive land mammal extinction during next supercont...

Sérgio Sacani

Microbial bio Synthesis of nanoparticles.pptx

Cherry

Alternative method of dissolution in-vitro in-vivo correlation and dissolutio...

Sahil Suleman

NUMERICAL Proof Of TIme Electron Theory.

syedmuneemqadri

Recent observations of galaxy clusters and groups with misalignments between their central AGN jets and X-ray cavities, or with multiple misaligned cavities, have raised concerns about the jet – bubble connection in cooling cores, and the processes responsible for jet realignment. To investigate the frequency and causes of such misalignments, we construct a sample of 16 cool core galaxy clusters and groups. Using VLBA radio data we measure the parsec-scale position angle of the jets, and compare it with the position angle of the X-ray cavities detected in Chandra data. Using the overall sample and selected subsets, we consistently find that there is a 30% – 38% chance to find a misalignment larger than ∆Ψ = 45◦ when observing a cluster/group with a detected jet and at least one cavity. We determine that projection may account for an apparently large ∆Ψ only in a fraction of objects (∼35%), and given that gas dynamical disturbances (as sloshing) are found in both aligned and misaligned systems, we exclude environmental perturbation as the main driver of cavity – jet misalignment. Moreover, we find that large misalignments (up to ∼ 90◦ ) are favored over smaller ones (45◦ ≤ ∆Ψ ≤ 70◦ ), and that the change in jet direction can occur on timescales between one and a few tens of Myr. We conclude that misalignments are more likely related to actual reorientation of the jet axis, and we discuss several engine-based mechanisms that may cause these dramatic changes.

Jet reorientation in central galaxies of clusters and groups: insights from V...

Sérgio Sacani

It should be no surprise that AI is treading a similar path to computing which began with single-purpose machines tasked for payroll calculations, banking transactions, or weapons targeting et al, but nothing more! It took decades for General Purpose Computing to emerge in the form of the now ubiquitous PC. Today, AI is still in a single-purpose/task-specific phase, and we have no general-purpose platforms, but their emergence is only a matter of time! Recent AI progress has seen a repeat of the media debate and alarmist warnings for our computing past, compounded by consequential advances in robotics. In turn, this has promoted numerous attempts to draw biological equivalences defining the time when machines will overtake humans. But without any workable definitions or framework that tend to little more than un/educated guesses. Recourse to IQ measures and the Touring test have proved to be irrelevant, and without a reference framework or formal characterisation, continued discussion and debate remain futile We therefore approach this AI problem from the bottom up by defining the simplest of machines and lifeforms to derive clues, pointers and basic boundary conditions . This sees a fundamental Entropic description emerge that is applicable to both machine and lifeforms. This presentation is suitable for professionals and the public alike, and is fully illustrated by high-quality graphics, animations and, movies. Inevitably, it contains some mathematics that non-practitioners will have to take on trust, but the focus is on defining the key characteristics, parameters, and important features of AI, our total dependence, and the future! Note: A 40 min session for a predominantly ley audience and not all the slides presented here were used on the day. Their inclusion here is in response to those audience members requesting more detail at the end of/during the event.

Quantifying Artificial Intelligence and What Comes Next!

University of Hertfordshire

INSIGHT Partner Profile: Tampere University

Steffi Friedrichs

PESTS OF GREEN MANURE CROPS AND FORAGE CROPS SESBANIA BORERS 1. Sesbania stem borer, Azygophleps scalaris, Cossidae, Lepidoptera Symptom of damage: Drying of sesbania plants. Nature of damage: Stoutly built larva bores into the stem and feeds, the faecal matter getting accumulated at the entrance hole. Egg: Laid in masses attached to the leaflets. Larva: Robust larva of 7 cm length, light brown with a reddish brown head. Pupa: Pupa has spines anteriorly with the help of which it moves up and down inside the tunnel in the stem. Adult: Sluggish yellowish brown moth. 2. Stem weevil, Alcidodes bubo, Curculionidae, Coleoptera Symptom of damage: Drying of the terminal branches or whole plant, lodging of the plants during monsoon seasons; affected stem with numerous bore holes and swelling symptoms. Presence of apodous grub inside the stem. Nature of damage: Grubs riddle into stem and cause thickenings on stem. The growth and vigour of the plants are impaired to a great extent. Egg: Laid on the stem. Grub : Apodous whitish grub. Adult: Reddish brown weevil with white transverse markings on the elytra. LEAF FEEDERS 3. Green semilooper, Pericyma glaucinans, Noctuidae, Lepidoptera Symptom of damage: Defoliation of plants preceding with leaflets damage. Nature of damage: Caterpillar feeds on the leaflets remaining along the ribs of leaves. Larva : Pale green, semilooper with broad yellow lateral stripes. Adult: Dark brown with black lines 4. Green looper, Semiothisa pervelgata, Semiothisa emersaria, Geometridae, Lepidoptera Symptom of damage: Defoliation of plants preceding with leaflets damage. Nature of damage: Larvae feed on the leaves. Egg: Oval greenish eggs laid singly on leaves. Larva : Pale green looper with white lines and when full grown becomes pink with horizontal bands of violet patches dorsally on the abdominal segments. Pupa: Pupates in the soil. Adult: White colour with three dark lines and spots on the wings.

Pests of Green Manures_Bionomics_IPM_Dr.UPR.pdf

PirithiRaju

GBSN - Microbiology (Unit 7) Microbiology in Everyday Life

Areesha Ahmad

Erythropoiesis- Dr.E. Muralinath-C Kalyan

muralinath2

Hemoglobin metabolism: C Kalyan & E. Muralinath

muralinath2

The highest priority recommendation of the Astro2020 Decadal Survey for space-based astronomy was the construction of an observatory capable of characterizing habitable worlds. In this paper series we explore the detectability of and interference from exomoons and exorings serendipitously observed with the proposed Habitable Worlds Observatory (HWO) as it seeks to characterize exoplanets, starting in this manuscript with Earth-Moon analog mutual events. Unlike transits, which only occur in systems viewed near edge-on, shadow (i.e., solar eclipse) and lunar eclipse mutual events occur in almost every star-planet-moon system. The cadence of these events can vary widely from ∼yearly to multiple events per day, as was the case in our younger Earth-Moon system. Leveraging previous space-based (EPOXI) lightcurves of a Moon transit and performance predictions from the LUVOIR-B concept, we derive the detectability of Moon analogs with HWO. We determine that Earth-Moon analogs are detectable with observation of ∼2-20 mutual events for systems within 10 pc, and larger moons should remain detectable out to 20 pc. We explore the extent to which exomoon mutual events can mimic planet features and weather. We find that HWO wavelength coverage in the near-IR, specifically in the 1.4 µm water band where large moons can outshine their host planet, will aid in differentiating exomoon signals from exoplanet variability. Finally, we predict that exomoons formed through collision processes akin to our Moon are more likely to be detected in younger systems, where shorter orbital periods and favorable geometry enhance the probability and frequency of mutual events.

Exomoons & Exorings with the Habitable Worlds Observatory I: On the Detection...

Sérgio Sacani

In-Pond Raceway Systems (IPRS) represent a cutting-edge method in pond aquaculture, effectively combining the advantages of confining fish in a restricted area within the pond with the efficiency of a flowing water system. This innovative approach facilitates water circulation within the pond, mimicking the natural flow found in riverine environments. By creating this dynamic water movement, IPRS significantly enhances the pond's productivity potential. The core principle of IPRS involves the installation of specific components that work in concert to circulate and refresh the water within the pond, all while preventing any discharge into the surrounding environment. This setup effectively utilizes a dividing partition, or baffle, to create a circular water flow pattern. As a result, the water undergoes continuous mixing and movement, optimizing conditions for fish growth and minimizing stagnation.

In-pond Race way systems for Aquaculture (IPRS).pptx

MAGOTI ERNEST

Mars is a particularly attractive candidate among known astronomical objects to potentially host life. Results from space exploration missions have provided insights into Martian geochemistry that indicate oxychlorine species, particularly perchlorate, are ubiquitous features of the Martian geochemical landscape. Perchlorate presents potential obstacles for known forms of life due to its toxicity. However, it can also provide potential benefits, such as producing brines by deliquescence, like those thought to exist on present-day Mars. Here we show perchlorate brines support folding and catalysis of functional RNAs, while inactivating representative protein enzymes. Additionally, we show perchlorate and other oxychlorine species enable ribozyme functions, including homeostasis-like regulatory behavior and ribozyme-catalyzed chlorination of organic molecules. We suggest nucleic acids are uniquely wellsuited to hypersaline Martian environments. Furthermore, Martian near- or subsurface oxychlorine brines, and brines found in potential lifeforms, could provide a unique niche for biomolecular evolution.

Emergent ribozyme behaviors in oxychlorine brines indicate a unique niche for...

Sérgio Sacani

MODERN PHYSICS_REPORTING_QUANTA_.....pdf

RevenJadePalma

Hadean zircons provide a potential record of Earth's earliest subduction 4.3 billion years ago. Itremains enigmatic how subduction could be initiated so soon after the presumably Moon‐forming giant impact(MGI). Earlier studies found an increase in Earth's core‐mantle boundary (CMB) temperature due to theaccumulation of the impactor's core, and our recent work shows Earth's lower mantle remains largely solid, withsome of the impactor's mantle potentially surviving as the large low‐shear velocity provinces (LLSVPs). Here,we show that a hot post‐impact CMB drives the initiation of strong mantle plumes that can induce subductioninitiation ∼200 Myr after the MGI. 2D and 3D thermomechanical computations show that a high CMBtemperature is the primary factor triggering early subduction, with enrichment of heat‐producing elements inLLSVPs as another potential factor. The models link the earliest subduction to the MGI with implications forunderstanding the diverse tectonic regimes of rocky planets.

A Giant Impact Origin for the First Subduction on Earth

Sérgio Sacani

Ostiguy & Panizza & Moffitt (eds.) - Populism in Global Perspective. A Perfor...

frank0071

SCHISTOSOMA HEAMATOBIUM life cycle .pdf

DebdattaGhosh6

The Scientific names of some important families of Industrial plants .pdf

Mohamed Said

Lec 1.b Totipotency and birth of tissue culture.ppt

asifali1111

Recently uploaded (20)

Climate extremes likely to drive land mammal extinction during next supercont...

Microbial bio Synthesis of nanoparticles.pptx

Alternative method of dissolution in-vitro in-vivo correlation and dissolutio...

NUMERICAL Proof Of TIme Electron Theory.

Jet reorientation in central galaxies of clusters and groups: insights from V...

Quantifying Artificial Intelligence and What Comes Next!

INSIGHT Partner Profile: Tampere University

Pests of Green Manures_Bionomics_IPM_Dr.UPR.pdf

GBSN - Microbiology (Unit 7) Microbiology in Everyday Life

Erythropoiesis- Dr.E. Muralinath-C Kalyan

Hemoglobin metabolism: C Kalyan & E. Muralinath

Exomoons & Exorings with the Habitable Worlds Observatory I: On the Detection...

In-pond Race way systems for Aquaculture (IPRS).pptx

Emergent ribozyme behaviors in oxychlorine brines indicate a unique niche for...

MODERN PHYSICS_REPORTING_QUANTA_.....pdf

A Giant Impact Origin for the First Subduction on Earth

Ostiguy & Panizza & Moffitt (eds.) - Populism in Global Perspective. A Perfor...

SCHISTOSOMA HEAMATOBIUM life cycle .pdf

The Scientific names of some important families of Industrial plants .pdf

Lec 1.b Totipotency and birth of tissue culture.ppt

BIRTE Panel at VLDB: Are we solving the core Problems in stream processing?

1. 1 Jonas Traub, BIRTE @ VLDB, 20181 Jonas Traub, BIRTE @ VLDB, 2018 Are we solving the core problems in stream processing? Jonas Traub Technische Universität Berlin / DFKI IAM www.dima.tu-berlin.de | jonas.traub@tu-berlin.de Panel Discussion with: Manpreet Singh (Google) Karthik Ramasamy (Stremlio) C. Mohan (IBM) Badrish Chandramouli (Microsoft) Neng Lu (Twitter) Alok Pareek (Striim) Jonas Traub (TU-Berlin)

2. 2 Jonas Traub, BIRTE @ VLDB, 20182 Jonas Traub, BIRTE @ VLDB, 2018 Are we solving the core problems in stream processing? Jonas Traub Technische Universität Berlin / DFKI IAM www.dima.tu-berlin.de | jonas.traub@tu-berlin.de

3. 3 Jonas Traub, BIRTE @ VLDB, 20183 Jonas Traub, BIRTE @ VLDB, 2018 Are we solving the core problems in stream processing? Yes, we do!

4. 4 Jonas Traub, BIRTE @ VLDB, 20184 Jonas Traub, BIRTE @ VLDB, 2018 Are we solving the core problems in stream processing? Yes, we do! Apache Flink and its success story What are the core problems and how are we solving them? Examples

5. 5 Jonas Traub, BIRTE @ VLDB, 2018 5 5 Jonas Traub, BIRTE @ VLDB, 2018 Apache Flink Timeline

6. 6 Jonas Traub, BIRTE @ VLDB, 2018 6 6 Jonas Traub, BIRTE @ VLDB, 2018

7. 7 Jonas Traub, BIRTE @ VLDB, 2018 Apache Flink - Stateful Computations over Data Streams source: flink.apache.org • Event-driven Applications • Stream & Batch Analytics • Data Pipelines & ETL • Exactly-once state consistency • Event-time processing • Sophisticated late data handling • Scale-out architecture • Support for very large state • Incremental checkpointing

8. 8 Jonas Traub, BIRTE @ VLDB, 20188 Jonas Traub, BIRTE @ VLDB, 2018 Examples: What are core problems and how are we solving them?

9. 9 Jonas Traub, BIRTE @ VLDB, 20189 Jonas Traub, BIRTE @ VLDB, 2018 Examples: Expressiveness: Event-time processing and sophisticated late data handling The Dataflow Model: A Practical Approach to Balancing Correctness, Latency, and Cost in Massive- Scale, Unbounded, Out-of-Order Data Processing (Akidau et al.)

10. 10 Jonas Traub, BIRTE @ VLDB, 201810 Jonas Traub, BIRTE @ VLDB, 2018 Examples: Expressiveness: Event-time processing and sophisticated late data handling Service: Common APIs and feature sets The Dataflow Model: A Practical Approach to Balancing Correctness, Latency, and Cost in Massive- Scale, Unbounded, Out-of-Order Data Processing (Akidau et al.) Apache Beam: An advanced unified programming model “Implement batch and streaming data processing jobs that run on any execution engine. (beam.apache.org)”

11. 11 Jonas Traub, BIRTE @ VLDB, 201811 Jonas Traub, BIRTE @ VLDB, 2018 Examples: Consistency: Exactly-once state consistency Expressiveness: Event-time processing and sophisticated late data handling Service: Common APIs and feature sets Lightweight asynchronous snapshots for distributed dataflows P Carbone, G Fóra, S Ewen, S Haridi, K Tzoumas State management in Apache Flink: consistent stateful distributed stream processing P Carbone, S Ewen, G Fóra, S Haridi, S Richter, K Tzoumas The Dataflow Model: A Practical Approach to Balancing Correctness, Latency, and Cost in Massive- Scale, Unbounded, Out-of-Order Data Processing (Akidau et al.) Apache Beam: An advanced unified programming model “Implement batch and streaming data processing jobs that run on any execution engine. (beam.apache.org)”

BIRTE Panel at VLDB: Are we solving the core Problems in stream processing?

Recommended

Recommended

More Related Content

More from Jonas Traub

More from Jonas Traub (10)

Recently uploaded

Recently uploaded (20)

BIRTE Panel at VLDB: Are we solving the core Problems in stream processing?