Storm over gearpump

•

0 likes•684 views

How to support binary Storm compatibility on Gearpump

Storm over Gearpump
Manu Zhang

About Me
● Software Engineer at Big Data Technology Group, Intel
● contributor to hadoop-nativetask
(https://github.com/apache/hadoop/tree/trunk/hadoop-mapreduce-project/hadoop
-mapreduce-client/hadoop-mapreduce-client-nativetask)
● written storm-benchmark (https://github.com/intel-hadoop/storm-benchmark)
● working on Gearpump (kafka connector, storm compatibility, state)
● maintain a list of awesome-streaming
(https://github.com/manuzhang/awesome-streaming)

Gearpump - Distributed Real-time Streaming Engine
● Akka / Actor Model
● Dynamic DAG
● Flow Control / Backpressure
● Low watermark
● At least once / Exactly-once
● High Availability
● Interactive Web UI

Gearpump Updates
● released 0.6.1 & 0.7.0
● new documentation site
● secure YARN and HBase
support
● Akka-stream compatibility
● Storm binary compatibility

Storm over Gearpump - Why
● Storm is widely used in the industry
● but has its limitations
● Gearpump is designed to overcome those limitations
● We want Storm users to benefit from Gearpump’s advanced features without any
cost

Storm over Gearpump - Features
● binary compatible (no recompilation is required) with Storm 0.9.x
● multi-lang
● DRPC
● KafkaSpout / KafkaBolt
● Trident (WIP)

Similarities of Gearpump and Storm
● per-message
● Topology / DAG
● similar user interface

Storm over Gearpump - Overview

Storm over Gearpump - DAG Translation

Storm over Gearpump - Task Execution

Storm over Gearpump - Flow Control
● Acker is removed
● Flow control with back pressure for both acked and unacked Storm topologies

Storm over Gearpump - At Least Once
● each message is tagged with
system time
● asynchronous non-blocking
ack through tracking global
minimum ack time
● support KafkaSpout for now

Performance
● SOL from storm-benchmark
● Storm 0.9.6
● 4-node 10GbE cluster
● 16 workers
● 48 Spouts and 48 Bolts

Future work
● submit Storm Job through Web UI
● Storm 0.10 support
● At Least Once support for more spouts
● Trident support

References
1. https://github.com/gearpump/gearpump
2. https://gearpump.io
3. https://storm.apache.org
4. How to use Akka to make a PERFECT Streaming system
5. https://www.typesafe.com/blog/gearpump-real-time-streaming-engine-using-akka
6. http://akka.io/docs/

More Related Content

What's hot

Building data pipelines is pretty hard! Building a multi-datacenter active-active real time data pipeline for multiple classes of data with different durability, latency and availability guarantees is much harder. Real time infrastructure powers critical pieces of Uber (think Surge) and in this talk we will discuss our architecture, technical challenges, learnings and how a blend of open source infrastructure (Apache Kafka and Samza) and in-house technologies have helped Uber scale.

Uber Real Time Data Analytics

Uber Real Time Data Analytics

Uber Real Time Data Analytics

ScyllaDB CEO Dor Laor lays out the ten million dollar engineering problem for distributed systems, and how only Scylla is architected to address the issue at the heart of Big Data ROI. He then introduces ScyllaDB's Glauber Costa and Packet's James Malachowski to reveal a new level of performance for a persistent NoSQL datastore. Dor concludes his talk with a bold proposition about how Scylla is uniquely positioned to help companies easily create and scale the software they need to achieve their vision.

Scylla Summit 2019 Keynote - Dor Laor - Beyond Cassandra

Scylla Summit 2019 Keynote - Dor Laor - Beyond Cassandra

Scylla Summit 2019 Keynote - Dor Laor - Beyond Cassandra

Advanced kapacitor

Advanced kapacitor

Advanced kapacitor

You should spend your time using the powerful Apache Flink ecosystem to get value from your data, not on your data processing infrastructure. Cloud environments can help you with this problem by providing managed services and infrastructure. Since Google Cloud Dataproc, Google's managed service to power the Apache big data ecosystem, runs Flink, you can easily combine the benefits of cloud with your Flink data pipelines. With new support for Flink and long-running streaming jobs, we will show you how you can set up a cluster and a streaming job in less than three minutes.

Flink Forward SF 2017: James Malone - Make The Cloud Work For You

Flink Forward SF 2017: James Malone - Make The Cloud Work For You

Flink Forward SF 2017: James Malone - Make The Cloud Work For You

PSUG #52 Dataflow and simplified reactive programming with Akka-streams

PSUG #52 Dataflow and simplified reactive programming with Akka-streams

PSUG #52 Dataflow and simplified reactive programming with Akka-streams

Stephane Manciot

Statsd introduction

Statsd introduction

Statsd introduction

Apache Beam lets you write data pipelines over unbounded, out-of-order, global-scale data that are portable across diverse backends including Apache Flink, Apache Apex, Apache Spark, and Google Cloud Dataflow. But not all use cases are pipelines of simple "map" and "combine" operations. Beam's new State API adds scalability and consistency to fine-grained stateful processing, all with Beam's usual portability. Examples of new use cases unlocked include: * Microservice-like streaming applications * Aggregations that aren't natural/efficient as an associative combiner * Fine control over retrieval and storage of intermediate values during aggregation * Output based on customized conditions, such as limiting to only "significant" changes in a learned model (resulting in potentially large cost savings in subsequent processing) This talk will introduce the new state and timer features in Beam and show how to use them to express common real-world use cases in a backend-agnostic manner.

Flink Forward SF 2017: Kenneth Knowles - Back to Sessions overview

Flink Forward SF 2017: Kenneth Knowles - Back to Sessions overview

Flink Forward SF 2017: Kenneth Knowles - Back to Sessions overview

DATA VISUALIZATION & ALERTING WITH GRAFANA Grafana is the leading graph and dashboard builder for visualizing time series, which is a great tool for visual monitoring of InfluxData. This session will provide an intro to Grafana and talk about adding data sources, creating dashboards and getting the most out of your data visualization. The talk will look into some new features Grafana has to offer, as well as explain why different graphs are important and specifically how you can use them to analyze data performance and troubleshoot operational issues.

Influx/Days 2017 San Francisco | Dan Cech

Influx/Days 2017 San Francisco | Dan Cech

Influx/Days 2017 San Francisco | Dan Cech

S3, Cassandra or Outer Space? Dumping Time Series Data using Spark - Demi Ben...

S3, Cassandra or Outer Space? Dumping Time Series Data using Spark - Demi Ben...

S3, Cassandra or Outer Space? Dumping Time Series Data using Spark - Demi Ben...

Codemotion Tel Aviv

Spacecrafts Made Simple: How Loft Orbital Delivers Unparalleled Speed-to-Space and Reliability Using Telegraf, InfluxDB Cloud and Google Cloud Today, access to space requires custom engineering, driving up costs, unpredictable schedule delays, and increased risk. Loft Orbital is changing that. Loft Orbital flies and operates customer payloads on their microsatellites as a service.. Companies turn to Loft Orbital when they want to focus on their end-use, with Loft Orbital operating their satellites using its mission agnostic, flexible operating system and interfacing technology. Loft Orbital's Payload Hub Technology provides clients with a modular payload adapter which can fly any payload on identical, commodity satellite buses it keeps in inventory while Cockpit, it's mission control system is used to operate all customer missions as a single constellation. By standardizing this technology, Loft Orbital has been able to deliver unparalleled speed-to-space without sacrificing reliability. Discover how Loft Orbital uses Telegraf, InfluxDB Cloud and Google Cloud to collect and store IoT sensor data from their equipment - including spacecrafts! In this webinar, Caleb MacLachlan will dive into: Loft Orbital's approach to QA-ing their code and enabling better performance monitoring Their methodology for monitoring their infrastructure, including servers and containers, and How a time series platform empowers long-term trend analysis

Spacecrafts Made Simple: How Loft Orbital Delivers Unparalleled Speed-to-Spac...

Spacecrafts Made Simple: How Loft Orbital Delivers Unparalleled Speed-to-Spac...

Spacecrafts Made Simple: How Loft Orbital Delivers Unparalleled Speed-to-Spac...

Building data product requires having Lambda Architecture to bridge the batch and streaming processing. AirStream is a framework built on top of Apache Spark to allow users to easily build data products at Airbnb. It proved Spark is impactful and useful in the production for mission-critical data products. On the streaming side, hear how AirStream integrates multiple ecosystems with Spark Streaming, such as HBase, Elasticsearch, MySQL, DynamoDB, Memcache and Redis. On the batch side, learn how to apply the same computation logic in Spark over large data sets from Hive and S3. The speakers will also go through a few production use cases, and share several best practices on how to manage Spark jobs in production.

Building Data Product Based on Apache Spark at Airbnb with Jingwei Lu and Liy...

Building Data Product Based on Apache Spark at Airbnb with Jingwei Lu and Liy...

Building Data Product Based on Apache Spark at Airbnb with Jingwei Lu and Liy...

Intro to Kapacitor for Alerting and Anomaly Detection

Intro to Kapacitor for Alerting and Anomaly Detection

Intro to Kapacitor for Alerting and Anomaly Detection

Key considerations in productionizing streaming applications

Key considerations in productionizing streaming applications

Key considerations in productionizing streaming applications

WRITING QUERIES (INFLUXQL AND TICK)

WRITING QUERIES (INFLUXQL AND TICK)

WRITING QUERIES (INFLUXQL AND TICK)

The mission of Uber is to make transportation as reliable as running water. The business is fundamentally driven by real-time data -- more than half of the employees in Uber, many of whom are non-technical, use SQL on a regular basis to analyze data and power their business decisions. We are building AthenaX, a stream processing platform built on top of Apache Flink to enable our users to write SQL to process real-time data efficiently and reliably at Uber's scale. Using Apache Calcite as query parser, AthenaX compiles the SQL down to Flink jobs. Leveraging Flink's unique streaming capabilities, AthenaX supports (1) consistent computations reliably thanks to at-least-once guarantees, (2) nontrivial analytics (e.g., windowing and joins) on multiple data sources, and (3) efficient and cost-effective executions in production through code generation and elastic scaling.

Flink Forward SF 2017: Bill Liu & Haohui Mai - AthenaX : Uber’s streaming pro...

Flink Forward SF 2017: Bill Liu & Haohui Mai - AthenaX : Uber’s streaming pro...

Flink Forward SF 2017: Bill Liu & Haohui Mai - AthenaX : Uber’s streaming pro...

OPTIMIZING THE TICK STACK

OPTIMIZING THE TICK STACK

OPTIMIZING THE TICK STACK

Optimizing InfluxDB Performance in the Real World by Dean Sheehan, Senior Dir...

Optimizing InfluxDB Performance in the Real World by Dean Sheehan, Senior Dir...

Optimizing InfluxDB Performance in the Real World by Dean Sheehan, Senior Dir...

The presentation gives a brief overview of the high-load service that stores users' actions. The given service is able to serve up to 240k writes per second in less than 2ms 95 percentile with just a few ScyllaDB nodes packed with HDDs. Hardware setup, cluster specification, live load numbers and latencies achieved are given. The problems we encountered with HDD setup are described along with the possible solutions to them.

High-Load Storage of Users’ Actions with ScyllaDB and HDDs

High-Load Storage of Users’ Actions with ScyllaDB and HDDs

High-Load Storage of Users’ Actions with ScyllaDB and HDDs

MySQL performance monitoring using Statsd and Graphite (PLUK2013)

MySQL performance monitoring using Statsd and Graphite (PLUK2013)

MySQL performance monitoring using Statsd and Graphite (PLUK2013)

spil-engineering

Talk Abstract As with all open-source databases, Accumulo developers often compete between building exciting new features and hacking on performance and stability. As the core features solidify and expand, we see many opportunities to improve performance. An effective methodology for performance improvement is scientific in nature, and follows a well-definite modeling and simulation approach, matching theory to experimentation in an iterative fashion. Ingest performance is one of the most differentiating characteristics of Accumulo. However, there is much room for improvement for typical ingest-heavy applications. Accumulo supports two mechanisms to bring data in: streaming ingest and bulk ingest. In bulk ingest, the goal is to maximize throughput without constraining latency. Bulk ingest involves creating a set of files that conform to Accumulo's internal RFile format and then registering those files with Accumulo. MapReduce provides a framework for generating, sorting, and storing key/value pairs, which form the primary elements of preparing RFiles for bulk ingest. MapReduce has been used many times over the years to break sorting records, such as Terasort. We can expect it is a reasonable choice for maximizing bulk ingest throughput. However, the theory often proves challenging to implement as there are many performance pitfalls along the way. In this talk, we dive deep into optimizing MapReduce for Accumulo bulk ingest. We share detailed theoretical and empirical performance models, we discuss techniques for profiling performance, and we suggest reusable techniques for squeezing the maximum performance out of enterprise-grade Accumulo bulk ingest. Speaker Chris McCubbin Director of Data Science, Sqrrl Chris is the Director of Data Science for Sqrrl. He has extensive experience with the Hadoop ecosystem and applying scientific computation algorithms to real-world datasets. Previously, Chris developed Big Data analysis tools for the Intelligence Community and applied artificial intelligence techniques to unmanned vehicle systems. He holds a MS in Computer Science and BS in Computer Science and Mathematics from the University of Maryland.

Accumulo Summit 2015: Performance Models for Apache Accumulo: The Heavy Tail ...

Accumulo Summit 2015: Performance Models for Apache Accumulo: The Heavy Tail ...

Accumulo Summit 2015: Performance Models for Apache Accumulo: The Heavy Tail ...

Accumulo Summit

What's hot (20)

Uber Real Time Data Analytics

Uber Real Time Data Analytics

Uber Real Time Data Analytics

Scylla Summit 2019 Keynote - Dor Laor - Beyond Cassandra

Scylla Summit 2019 Keynote - Dor Laor - Beyond Cassandra

Scylla Summit 2019 Keynote - Dor Laor - Beyond Cassandra

Advanced kapacitor

Advanced kapacitor

Advanced kapacitor

Flink Forward SF 2017: James Malone - Make The Cloud Work For You

Flink Forward SF 2017: James Malone - Make The Cloud Work For You

Flink Forward SF 2017: James Malone - Make The Cloud Work For You

PSUG #52 Dataflow and simplified reactive programming with Akka-streams

PSUG #52 Dataflow and simplified reactive programming with Akka-streams

PSUG #52 Dataflow and simplified reactive programming with Akka-streams

Statsd introduction

Statsd introduction

Statsd introduction

Flink Forward SF 2017: Kenneth Knowles - Back to Sessions overview

Flink Forward SF 2017: Kenneth Knowles - Back to Sessions overview

Flink Forward SF 2017: Kenneth Knowles - Back to Sessions overview

Influx/Days 2017 San Francisco | Dan Cech

Influx/Days 2017 San Francisco | Dan Cech

Influx/Days 2017 San Francisco | Dan Cech

S3, Cassandra or Outer Space? Dumping Time Series Data using Spark - Demi Ben...

S3, Cassandra or Outer Space? Dumping Time Series Data using Spark - Demi Ben...

S3, Cassandra or Outer Space? Dumping Time Series Data using Spark - Demi Ben...

Spacecrafts Made Simple: How Loft Orbital Delivers Unparalleled Speed-to-Spac...

Spacecrafts Made Simple: How Loft Orbital Delivers Unparalleled Speed-to-Spac...

Spacecrafts Made Simple: How Loft Orbital Delivers Unparalleled Speed-to-Spac...

Building Data Product Based on Apache Spark at Airbnb with Jingwei Lu and Liy...

Building Data Product Based on Apache Spark at Airbnb with Jingwei Lu and Liy...

Building Data Product Based on Apache Spark at Airbnb with Jingwei Lu and Liy...

Intro to Kapacitor for Alerting and Anomaly Detection

Intro to Kapacitor for Alerting and Anomaly Detection

Intro to Kapacitor for Alerting and Anomaly Detection

Key considerations in productionizing streaming applications

Key considerations in productionizing streaming applications

Key considerations in productionizing streaming applications

WRITING QUERIES (INFLUXQL AND TICK)

WRITING QUERIES (INFLUXQL AND TICK)

WRITING QUERIES (INFLUXQL AND TICK)

Flink Forward SF 2017: Bill Liu & Haohui Mai - AthenaX : Uber’s streaming pro...

Flink Forward SF 2017: Bill Liu & Haohui Mai - AthenaX : Uber’s streaming pro...

Flink Forward SF 2017: Bill Liu & Haohui Mai - AthenaX : Uber’s streaming pro...

OPTIMIZING THE TICK STACK

OPTIMIZING THE TICK STACK

OPTIMIZING THE TICK STACK

Optimizing InfluxDB Performance in the Real World by Dean Sheehan, Senior Dir...

Optimizing InfluxDB Performance in the Real World by Dean Sheehan, Senior Dir...

Optimizing InfluxDB Performance in the Real World by Dean Sheehan, Senior Dir...

High-Load Storage of Users’ Actions with ScyllaDB and HDDs

High-Load Storage of Users’ Actions with ScyllaDB and HDDs

High-Load Storage of Users’ Actions with ScyllaDB and HDDs

MySQL performance monitoring using Statsd and Graphite (PLUK2013)

MySQL performance monitoring using Statsd and Graphite (PLUK2013)

MySQL performance monitoring using Statsd and Graphite (PLUK2013)

Accumulo Summit 2015: Performance Models for Apache Accumulo: The Heavy Tail ...

Accumulo Summit 2015: Performance Models for Apache Accumulo: The Heavy Tail ...

Accumulo Summit 2015: Performance Models for Apache Accumulo: The Heavy Tail ...

Viewers also liked

Blr hadoop meetup

Blr hadoop meetup

Blr hadoop meetup

The total amount of data in the world almost doubles every 2 years. Storing data for offline processing is no longer a viable business model. In the past few years, new technologies for real-time data processing emerged. Microsoft Azure offers a comprehensive set of tools to ingest and process data in motion. In this presentation we will go over and learn how to collect data from devices, how to process data in real time using Azure Stream Analytic jobs, and how to produce and handle actionable insights.

Getting started with Azure Event Hubs and Stream Analytics services

Getting started with Azure Event Hubs and Stream Analytics services

Getting started with Azure Event Hubs and Stream Analytics services

Vladimir Bychkov

Landoop presenting how to simplify your ETL process using Kafka Connect for (E) and (L). Introducing KCQL - the Kafka Connect Query Language & how it can simplify fast-data (ingress & egress) pipelines. How KCQL can be used to set up Kafka Connectors for popular in-memory and analytical systems and live demos with HazelCast, Redis and InfluxDB. How to get started with a fast-data docker kafka development environment. Enhance your existing Cloudera (Hadoop) clusters with fast-data capabilities.

London Apache Kafka Meetup (Jan 2017)

London Apache Kafka Meetup (Jan 2017)

London Apache Kafka Meetup (Jan 2017)

Kafka connect

Andrew Stevenson

Not Only Streams for Akademia JLabs

Not Only Streams for Akademia JLabs

Not Only Streams for Akademia JLabs

Konrad Malawski

Processing IoT Data with Apache Kafka

Processing IoT Data with Apache Kafka

Processing IoT Data with Apache Kafka

Matthew Howlett

IoT Connected Brewery

IoT Connected Brewery

IoT Connected Brewery

Confluent kafka meetupseattle jan2017

Confluent kafka meetupseattle jan2017

Confluent kafka meetupseattle jan2017

Strata+Hadoop 2017 San Jose - The Rise of Real Time: Apache Kafka and the Str...

Strata+Hadoop 2017 San Jose - The Rise of Real Time: Apache Kafka and the Str...

Strata+Hadoop 2017 San Jose - The Rise of Real Time: Apache Kafka and the Str...

Extracting Insights from Data at Twitter

Extracting Insights from Data at Twitter

Extracting Insights from Data at Twitter

Apache kafka-a distributed streaming platform

Apache kafka-a distributed streaming platform

Apache kafka-a distributed streaming platform

In this session, we will discuss: * reactive architecture tenets * distributed “fast data” streams * application and analytics focused Data Lake Enterprise level concerns and the importance of holistic governance, operational management, and a Metadata Lake will be conceptually investigated. The next level of detail will be to explore what a prospective architecture looks like at scale with Terabytes of ingestion per day, how scale puts pressure on an architecture, and how to be successful without losing data in a mission critical system via resilient, self-healing, scalable technologies. DevOps and application architecture concerns will be first-class themes throughout. Reactive principles and technology will be the second act of this talk. Kafka. Akka. Spark. Various streaming technologies (Kafka Streams, Akka Streams, Spark Streaming) will be reviewed to identify what they are best suited for. The fast data pipeline discussion will center around Kafka, Akka, and Apache Flink (Lightbend Fast Data platform). We’ll also walk through an exciting addition to the Akka family, Alpakka, which is a Camel equivalent for Enterprise Integration Patterns. The final act will be to dive into the Data Lake, from both an analytics and application development perspective. Technologies used to explain concepts will include Amazon and Hadoop. A Data Lake may service multiple analytics consumers with various “views” (and access levels) of data. It may also be a participant of various applications, perhaps by acting as a centralized source for reference data or common middleware (in turn feeding the analytics aspect). The concept of the Metadata Lake to apply structure, meaning and purpose will be an over-arching success factor for a Data Lake. The difference between the Data Lake and Metadata Lake is conceptually similar to a Halocline… Various technologies (Iglu/Snowplow and more) will be discussed from a feature standpoint to flesh out the technology capabilities needed for Data Lake governance.

Building Reactive Fast Data & the Data Lake with Akka, Kafka, Spark

Building Reactive Fast Data & the Data Lake with Akka, Kafka, Spark

Building Reactive Fast Data & the Data Lake with Akka, Kafka, Spark

While the last few years have seen great advancements in computing paradigms for big data stores, there remains one critical bottleneck in this architecture - the ingestion process. Instead of immediate insights into the data, a poor ingestion process can cause headaches and problems to no end. On the other hand, a well-designed ingestion infrastructure should give you real-time visibility into how your systems are functioning at any given time. This can significantly increase the overall effectiveness of your ad-campaigns, fraud-detection systems, preventive-maintenance systems, or other critical applications underpinning your business. In this session we will explore various modes of ingest including pipelining, pub-sub, and micro-batching, and identify the use-cases where these can be applied. We will present this in the context of open source frameworks such as Apache Flume, Kafka, among others that can be used to build related solutions. We will also present when and how to use multiple modes and frameworks together to form hybrid solutions that can address non-trivial ingest requirements with little or no extra overhead. Through this discussion we will drill-down into details of configuration and sizing for these frameworks to ensure optimal operations and utilization for long-running deployments.

Big Data Day LA 2015 - Always-on Ingestion for Data at Scale by Arvind Prabha...

Big Data Day LA 2015 - Always-on Ingestion for Data at Scale by Arvind Prabha...

Big Data Day LA 2015 - Always-on Ingestion for Data at Scale by Arvind Prabha...

IoT Innovation Lab Berlin @relayr - Kay Lerch on Getting basics right for you...

IoT Innovation Lab Berlin @relayr - Kay Lerch on Getting basics right for you...

IoT Innovation Lab Berlin @relayr - Kay Lerch on Getting basics right for you...

Introduction to Structured Streaming

Introduction to Structured Streaming

Introduction to Structured Streaming

Dataflow with Apache NiFi - Apache NiFi Meetup - 2016 Hadoop Summit - San Jose

Dataflow with Apache NiFi - Apache NiFi Meetup - 2016 Hadoop Summit - San Jose

Dataflow with Apache NiFi - Apache NiFi Meetup - 2016 Hadoop Summit - San Jose

Comparison of various streaming technologies This meetup will take us through the various streaming technologies such as Storm, Flink, Infosphere Streams and Spark Streaming. Agenda • Characteristics of streaming technologies • Introduction to Apache Storm, Trident and Flink • Examples of Code and API • Deep-dive of Spark Streaming • Comparison of Spark Streaming with other streaming technologies • Benchmark of Spark Streaming (with code walkthrough) We will supplement theory concepts with sufficient examples

Comparison of various streaming technologies

Comparison of various streaming technologies

Comparison of various streaming technologies

Sachin Aggarwal

In this presentation, we are going to talk about the state of the art infrastructure we have established at Walmart Labs for the Search product using Spark Streaming and DataFrames. First, we have been able to successfully use multiple micro batch spark streaming pipelines to update and process information like product availability, pick up today etc. along with updating our product catalog information in our search index to up to 10,000 kafka events per sec in near real-time. Earlier, all the product catalog changes in the index had a 24 hour delay, using Spark Streaming we have made it possible to see these changes in near real-time. This addition has provided a great boost to the business by giving the end-costumers instant access to features likes availability of a product, store pick up, etc. Second, we have built a scalable anomaly detection framework purely using Spark Data Frames that is being used by our data pipelines to detect abnormality in search data. Anomaly detection is an important problem not only in the search domain but also many domains such as performance monitoring, fraud detection, etc. During this, we realized that not only are Spark DataFrames able to process information faster but also are more flexible to work with. One could write hive like queries, pig like code, UDFs, UDAFs, python like code etc. all at the same place very easily and can build DataFrame template which can be used and reused by multiple teams effectively. We believe that if implemented correctly Spark Data Frames can potentially replace hive/pig in big data space and have the potential of becoming unified data language. We conclude that Spark Streaming and Data Frames are the key to processing extremely large streams of data in real-time with ease of use.

Learnings Using Spark Streaming and DataFrames for Walmart Search: Spark Summ...

Learnings Using Spark Streaming and DataFrames for Walmart Search: Spark Summ...

Learnings Using Spark Streaming and DataFrames for Walmart Search: Spark Summ...

These slides discuss the main concepts of event & stream processing, as well as the related technologies on Microsoft Azure. We start by giving and overview of what Event & Stream Processing is. Then we describe the canonical architecture of a Stream Processing solution. We will delve into Message Queuing part of the solution. After that, we Introduce Apache Storm on HDInsight, as well as Azure Stream Analytics. We compare Apache Storm to Azure Stream Analytics, and finally conclude with useful resources

Real-Time Event & Stream Processing on MS Azure

Real-Time Event & Stream Processing on MS Azure

Real-Time Event & Stream Processing on MS Azure

Reactive integrations with Akka Streams

Reactive integrations with Akka Streams

Reactive integrations with Akka Streams

Konrad Malawski

Viewers also liked (20)

Blr hadoop meetup

Blr hadoop meetup

Blr hadoop meetup

Getting started with Azure Event Hubs and Stream Analytics services

Getting started with Azure Event Hubs and Stream Analytics services

Getting started with Azure Event Hubs and Stream Analytics services

London Apache Kafka Meetup (Jan 2017)

London Apache Kafka Meetup (Jan 2017)

London Apache Kafka Meetup (Jan 2017)

Kafka connect

Not Only Streams for Akademia JLabs

Not Only Streams for Akademia JLabs

Not Only Streams for Akademia JLabs

Processing IoT Data with Apache Kafka

Processing IoT Data with Apache Kafka

Processing IoT Data with Apache Kafka

IoT Connected Brewery

IoT Connected Brewery

IoT Connected Brewery

Confluent kafka meetupseattle jan2017

Confluent kafka meetupseattle jan2017

Confluent kafka meetupseattle jan2017

Strata+Hadoop 2017 San Jose - The Rise of Real Time: Apache Kafka and the Str...

Strata+Hadoop 2017 San Jose - The Rise of Real Time: Apache Kafka and the Str...

Strata+Hadoop 2017 San Jose - The Rise of Real Time: Apache Kafka and the Str...

Extracting Insights from Data at Twitter

Extracting Insights from Data at Twitter

Extracting Insights from Data at Twitter

Apache kafka-a distributed streaming platform

Apache kafka-a distributed streaming platform

Apache kafka-a distributed streaming platform

Building Reactive Fast Data & the Data Lake with Akka, Kafka, Spark

Building Reactive Fast Data & the Data Lake with Akka, Kafka, Spark

Building Reactive Fast Data & the Data Lake with Akka, Kafka, Spark

Big Data Day LA 2015 - Always-on Ingestion for Data at Scale by Arvind Prabha...

Big Data Day LA 2015 - Always-on Ingestion for Data at Scale by Arvind Prabha...

Big Data Day LA 2015 - Always-on Ingestion for Data at Scale by Arvind Prabha...

IoT Innovation Lab Berlin @relayr - Kay Lerch on Getting basics right for you...

IoT Innovation Lab Berlin @relayr - Kay Lerch on Getting basics right for you...

IoT Innovation Lab Berlin @relayr - Kay Lerch on Getting basics right for you...

Introduction to Structured Streaming

Introduction to Structured Streaming

Introduction to Structured Streaming

Dataflow with Apache NiFi - Apache NiFi Meetup - 2016 Hadoop Summit - San Jose

Dataflow with Apache NiFi - Apache NiFi Meetup - 2016 Hadoop Summit - San Jose

Dataflow with Apache NiFi - Apache NiFi Meetup - 2016 Hadoop Summit - San Jose

Comparison of various streaming technologies

Comparison of various streaming technologies

Comparison of various streaming technologies

Learnings Using Spark Streaming and DataFrames for Walmart Search: Spark Summ...

Learnings Using Spark Streaming and DataFrames for Walmart Search: Spark Summ...

Learnings Using Spark Streaming and DataFrames for Walmart Search: Spark Summ...

Real-Time Event & Stream Processing on MS Azure

Real-Time Event & Stream Processing on MS Azure

Real-Time Event & Stream Processing on MS Azure

Reactive integrations with Akka Streams

Reactive integrations with Akka Streams

Reactive integrations with Akka Streams

Similar to Storm over gearpump

HBase @ Twitter

HBase @ Twitter

HBase @ Twitter

Tez is the next generation Hadoop Query Processing framework written on top of YARN. Computation topologies in higher level languages like Pig/Hive can be naturally expressed in the new graph dataflow model exposed by Tez. Multi-stage queries can be expressed as a single Tez job resulting in lower latency for short queries and improved throughput for large scale queries. MapReduce has been the workhorse for Hadoop but its monolithic structure had made innovation slower. YARN separates resource management from application logic and thus enables the creation of Tez, a more flexible and generic new framework for data processing for the benefit of the entire Hadoop query ecosystem.

Apache Tez: Accelerating Hadoop Query Processing

Apache Tez: Accelerating Hadoop Query Processing

Apache Tez: Accelerating Hadoop Query Processing

DataWorks Summit

2013 Nov 20 Toronto Hadoop User Group (THUG) - Hadoop 2.2.0

2013 Nov 20 Toronto Hadoop User Group (THUG) - Hadoop 2.2.0

2013 Nov 20 Toronto Hadoop User Group (THUG) - Hadoop 2.2.0

Stream Processing Everywhere - What to use?

Stream Processing Everywhere - What to use?

Stream Processing Everywhere - What to use?

MapR Technologies

Hadoop 2 @ Twitter, Elephant Scale

Hadoop 2 @ Twitter, Elephant Scale

Hadoop 2 @ Twitter, Elephant Scale

DataWorks Summit

Hadoop 2 @Twitter, Elephant Scale. Presented at

Hadoop 2 @Twitter, Elephant Scale. Presented at

Hadoop 2 @Twitter, Elephant Scale. Presented at

lohitvijayarenu

Deep learning is useful for enterprises tasks in the field of speech recognition, image classification, AI chatbots and machine translation, just to name a few. In order to train deep learning/machine learning models, applications such as TensorFlow / MXNet / Caffe / XGBoost can be leveraged. And sometimes these applications will be used together to solve different problems. To make distributed deep learning/machine learning applications easily launched, managed, monitored. Hadoop community has introduced Submarine project along with other improvements such as first-class GPU support, container-DNS support, scheduling improvements, etc. These improvements make distributed deep learning/machine learning applications run on YARN as simple as running it locally, which can let machine-learning engineers focus on algorithms instead of worrying about underlying infrastructure. Also, YARN can better manage a shared cluster which runs deep learning/machine learning and other services/ETL jobs with these improvements. In this session, we will take a closer look at Submarine project as well as other improvements and show how to run these deep learning workloads on YARN with demos. Audiences can start trying running these workloads on YARN after this talk. Speakers: Sunil Govindan, Staff Engineer Hortonworks Zhankun Tank, Staff Engineer Hortonworks

Hadoop {Submarine} Project: Running Deep Learning Workloads on YARN

Hadoop {Submarine} Project: Running Deep Learning Workloads on YARN

Hadoop {Submarine} Project: Running Deep Learning Workloads on YARN

DataWorks Summit

Capturing live traffic in a typical micro-services setup is one of the industry standard strategies for various needs including testing, canary, and dark launches, but when it comes to databases, capturing and replaying live traffic is quite challenging but equally critical. Cassandra is one of the few databases which supports a live traffic capture and replay feature out of the box starting with 4.0. As part of this talk, Vinay Chella will dive deeper into the query logging framework that is introduced in Cassandra 4.0, and the features built on top of this framework, including full query logging, audit logging, and traffic replay. At end of this talk, you will learn how to use audit logging, live traffic capture and replay features in the upcoming release of Cassandra.

Live traffic capture and replay in cassandra 4.0

Live traffic capture and replay in cassandra 4.0

Live traffic capture and replay in cassandra 4.0

Vinay Kumar Chella

Each new version of the Oracle database includes improvements in the upgrade and patching utilities, forcing us to update our procedures to incorporate these changes. The Fleet Provisioning & Patching (FPP, formerly RHP) utility, together with the change in its licensing announced at OOW 2019 that makes it free in RAC, now makes it possible to centrally manage the software life cycle. This presentation shows examples of how to use FPP and different configuration options.

Automate Oracle database patches and upgrades using Fleet Provisioning and Pa...

Automate Oracle database patches and upgrades using Fleet Provisioning and Pa...

Automate Oracle database patches and upgrades using Fleet Provisioning and Pa...

02 ai inference acceleration with components all in open hardware: opencapi a...

02 ai inference acceleration with components all in open hardware: opencapi a...

02 ai inference acceleration with components all in open hardware: opencapi a...

Visual Mapping of Clickstream Data

Visual Mapping of Clickstream Data

Visual Mapping of Clickstream Data

DataWorks Summit

Apache Ambari BOF - OpenStack - Hadoop Summit 2013

Apache Ambari BOF - OpenStack - Hadoop Summit 2013

Apache Ambari BOF - OpenStack - Hadoop Summit 2013

Apache Tez - A unifying Framework for Hadoop Data Processing

Apache Tez - A unifying Framework for Hadoop Data Processing

Apache Tez - A unifying Framework for Hadoop Data Processing

DataWorks Summit

Apache Tez is a library to build data processing engines in Hadoop/YARN. It takes care of many common building blocks like scheduling, fault tolerance, speculation, security etc. so that the engine can focus on its core features. E.g. Apache Hive can focus on SQL optimization. There has been rapid adoption in projects like Hive, Pig, Flink, Cascading, Scalding and commercial products like Datameer and Syncsort. We will provide a brief overview of Tez and then look at new features for job monitoring in the Tez UI and performance debugging tools for Tez applications. Finally we will explore upcoming features like hybrid scheduling that open up new areas of performance and functionality.

Big Data Day LA 2015 - What's new and next in Apache Tez by Bikas Saha of Hor...

Big Data Day LA 2015 - What's new and next in Apache Tez by Bikas Saha of Hor...

Big Data Day LA 2015 - What's new and next in Apache Tez by Bikas Saha of Hor...

Ever wonder what Hadoop might look like in 12 months or 24 months or longer? Apache Hadoop MapReduce has undergone a complete re-haul to emerge as Apache Hadoop YARN, a generic compute fabric to support MapReduce and other application paradigms. As a result, Hadoop looks very different from itself 12 months ago. This talk will take you through some ideas for YARN itself and the many myriad ways it is really moving the needle for MapReduce, Pig, Hive, Cascading and other data-processing tools in the Hadoop ecosystem.

Hadoop past, present and future

Hadoop past, present and future

Hadoop past, present and future

2012 09-08-josug-jeff

2012 09-08-josug-jeff

2012 09-08-josug-jeff

Zheng (Jeff) Xu

Performance: Observe and Tune

Performance: Observe and Tune

Performance: Observe and Tune

Paul V. Novarese

HDFS presented by VIJAY

HDFS presented by VIJAY

HDFS presented by VIJAY

Introduction to Structured Streaming

Introduction to Structured Streaming

Introduction to Structured Streaming

AltaVault

Similar to Storm over gearpump (20)

HBase @ Twitter

HBase @ Twitter

HBase @ Twitter

Apache Tez: Accelerating Hadoop Query Processing

Apache Tez: Accelerating Hadoop Query Processing

Apache Tez: Accelerating Hadoop Query Processing

2013 Nov 20 Toronto Hadoop User Group (THUG) - Hadoop 2.2.0

2013 Nov 20 Toronto Hadoop User Group (THUG) - Hadoop 2.2.0

2013 Nov 20 Toronto Hadoop User Group (THUG) - Hadoop 2.2.0

Stream Processing Everywhere - What to use?

Stream Processing Everywhere - What to use?

Stream Processing Everywhere - What to use?

Hadoop 2 @ Twitter, Elephant Scale

Hadoop 2 @ Twitter, Elephant Scale

Hadoop 2 @ Twitter, Elephant Scale

Hadoop 2 @Twitter, Elephant Scale. Presented at

Hadoop 2 @Twitter, Elephant Scale. Presented at

Hadoop 2 @Twitter, Elephant Scale. Presented at

Hadoop {Submarine} Project: Running Deep Learning Workloads on YARN

Hadoop {Submarine} Project: Running Deep Learning Workloads on YARN

Hadoop {Submarine} Project: Running Deep Learning Workloads on YARN

Live traffic capture and replay in cassandra 4.0

Live traffic capture and replay in cassandra 4.0

Live traffic capture and replay in cassandra 4.0

Automate Oracle database patches and upgrades using Fleet Provisioning and Pa...

Automate Oracle database patches and upgrades using Fleet Provisioning and Pa...

Automate Oracle database patches and upgrades using Fleet Provisioning and Pa...

02 ai inference acceleration with components all in open hardware: opencapi a...

02 ai inference acceleration with components all in open hardware: opencapi a...

02 ai inference acceleration with components all in open hardware: opencapi a...

Visual Mapping of Clickstream Data

Visual Mapping of Clickstream Data

Visual Mapping of Clickstream Data

Apache Ambari BOF - OpenStack - Hadoop Summit 2013

Apache Ambari BOF - OpenStack - Hadoop Summit 2013

Apache Ambari BOF - OpenStack - Hadoop Summit 2013

Apache Tez - A unifying Framework for Hadoop Data Processing

Apache Tez - A unifying Framework for Hadoop Data Processing

Apache Tez - A unifying Framework for Hadoop Data Processing

Big Data Day LA 2015 - What's new and next in Apache Tez by Bikas Saha of Hor...

Big Data Day LA 2015 - What's new and next in Apache Tez by Bikas Saha of Hor...

Big Data Day LA 2015 - What's new and next in Apache Tez by Bikas Saha of Hor...

Hadoop past, present and future

Hadoop past, present and future

Hadoop past, present and future

2012 09-08-josug-jeff

2012 09-08-josug-jeff

2012 09-08-josug-jeff

Performance: Observe and Tune

Performance: Observe and Tune

Performance: Observe and Tune

HDFS presented by VIJAY

HDFS presented by VIJAY

HDFS presented by VIJAY

Introduction to Structured Streaming

Introduction to Structured Streaming

Introduction to Structured Streaming

AltaVault

Recently uploaded

FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...

FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...

FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...

Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf

Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf

Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf

As an SEO expert specializing in the IPTV and VPN niches with over five years of experience, I navigate the unique challenges of these industries adeptly. My strategic approach encompasses competitive analysis, targeted keyword research, content optimization, and high-quality backlink creation. My goal is to optimize my clients' online visibility, generating targeted organic traffic and maximizing their return on investment. With a results-driven approach and a passion for innovation, I'm poised to assist my clients in thriving in an ever-evolving digital landscape. <a href="https://iptvreel.com">

THE BEST IPTV in GERMANY for 2024: IPTVreel

THE BEST IPTV in GERMANY for 2024: IPTVreel

THE BEST IPTV in GERMANY for 2024: IPTVreel

Intro in Product Management - Коротко про професію продакт менеджера

Intro in Product Management - Коротко про професію продакт менеджера

Intro in Product Management - Коротко про професію продакт менеджера

IoT Analytics Company Presentation May 2024

IoT Analytics Company Presentation May 2024

IoT Analytics Company Presentation May 2024

Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)

Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)

Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)

Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...

Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...

Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...

Heather Hedden, Senior Consultant at Enterprise Knowledge, presented “Enterprise Knowledge Graphs: The Importance of Semantics” on May 9, 2024, at the annual Data Summit in Boston. In her presentation, Hedden describes the components of an enterprise knowledge graph and provides further insight into the semantic layer – or knowledge model – component, which includes an ontology and controlled vocabularies, such as taxonomies, for controlled metadata. While data experts tend to focus on the graph database components (RDF triple store or a label property graph), Hedden emphasizes they should not overlook the importance of the semantic layer.

Enterprise Knowledge Graphs - Data Summit 2024

Enterprise Knowledge Graphs - Data Summit 2024

Enterprise Knowledge Graphs - Data Summit 2024

Enterprise Knowledge

Extensible Python: Robustness through Addition - PyCon 2024

Extensible Python: Robustness through Addition - PyCon 2024

Extensible Python: Robustness through Addition - PyCon 2024

Patrick Viafore

The standard Salesforce Approval process can be limiting in many ways, especially in complex scenarios. What if there was a way to implement very flexible approvals where one can use Apex code to make data updates in unrelated records, dynamically generate next steps details, and compute assignees on the fly? And still use UI-based configurations to implement concrete approval processes. In this session, we will share ideas behind such a solution and show a few lines of code to get you started.

Custom Approval Process: A New Perspective, Pavel Hrbacek & Anindya Halder

Custom Approval Process: A New Perspective, Pavel Hrbacek & Anindya Halder

Custom Approval Process: A New Perspective, Pavel Hrbacek & Anindya Halder

Unlock the mysteries of successful Salesforce interviews in this insightful session hosted by Hugo Rosario (Salesforce Customer), a seasoned hiring manager that leads the Salesforce Department of multinational company with over 100 interviews under their belt. Step into the manager's chair and gain exclusive behind-the-scenes insights into what makes a Salesforce consultant stand out during the interview process. From deciphering the unspoken cues to mastering key strategies, we'll explore the intricacies of the interview process and provide practical tips for consultants looking to not only pass interviews but also thrive in their roles. Whether you're a seasoned professional or just starting your Salesforce journey, this session is your backstage pass to the secrets that hiring managers wish you knew.

Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...

Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...

Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...

How to differentiate Sales Cloud and CPQ on first glance might be tricky if you do not know where to look and what to look at. You will know :-) Managing the sales process within Salesforce is a common use case that can be managed with standart Sales Cloud. If you want to do entire quoting process you will find out Salesforce CPQ solution exists. What is then the difference if both can handle selling products? You will see comparison of 10 different features, which Sales Cloud and Salesforce CPQ handle differently. Simple question you will always remember if you should consider using Salesforce CPQ will be a cherry on top.

10 Differences between Sales Cloud and CPQ, Blanka Doktorová

10 Differences between Sales Cloud and CPQ, Blanka Doktorová

10 Differences between Sales Cloud and CPQ, Blanka Doktorová

Designing inclusive products is not only a social responsibility but also a business imperative. This talk delves into the journey of creating accessible hardware products that cater to diverse user needs. Key Topics Covered: 1. Introduction to Inclusive Design - Importance of accessibility in product design - Overview of Comcast's commitment to making products accessible to a wide audience 2. Case Study: Xfinity Large Button Voice Remote - Initial challenges and the evolution of the product - User research and feedback that shaped the design - Key features of the final product and their benefits 3. Designing for Diverse Needs - Understanding human-centered design and its historical context - The impact of designing for people with disabilities on overall product quality - Examples from other industries, such as architecture and industrial design 4. Integrating Accessibility from the Beginning - The cost and efficiency benefits of designing for accessibility from the start - The process of embedding accessibility as a core trait rather than an optional feature 5. Real-World Impact and Continuous Improvement - Insights from in-home studies with users having assistive needs - How continuous feedback and iterative design lead to better products - The role of inclusive research and development practices

Designing for Hardware Accessibility at Comcast

Designing for Hardware Accessibility at Comcast

Designing for Hardware Accessibility at Comcast

This presentation dives into the practical applications of machine learning within Google's operations, providing a comprehensive overview of how to leverage AI technologies to solve real-world business challenges. Key Points Covered: - Introduction to Machine Learning at Google: Discussion on the role of ML and its evolution in enhancing Google's operational efficiency. - Experience Sharing: Insights into the team's long-term engagement with machine learning projects and the impacts on Google’s operational strategies. - Practical Applications: Real-world examples of ML applications within Google’s daily operations, providing a blueprint to adapt similar strategies. - Challenges and Solutions: Discussion on the challenges faced during the implementation of ML projects and the strategic solutions employed to overcome them. - Future of ML at Google: Insights into future trends in machine learning at Google and how they plan to continue integrating AI into their ecosystem.

Strategic AI Integration in Engineering Teams

Strategic AI Integration in Engineering Teams

Strategic AI Integration in Engineering Teams

The Epson EcoTank L3210 is a high-performance and cost-efficient printer designed to meet the printing needs of both home users and small businesses. Equipped with Epson’s revolutionary EcoTank ink tank system, the Epson eliminates the need for traditional ink cartridges, thereby significantly reducing printing costs and plastic waste. With its PrecisionCore technology, this printer delivers sharp, vibrant prints for both documents and photos. Its user-friendly design ensures easy setup and operation, while its compact form factor saves valuable desk space. Whether it’s everyday printing jobs or creative projects, the Epson EcoTank L3210 provides a reliable and eco-friendly printing solution.

Buy Epson EcoTank L3210 Colour Printer Online.pptx

Buy Epson EcoTank L3210 Colour Printer Online.pptx

Buy Epson EcoTank L3210 Colour Printer Online.pptx

EasyPrinterHelp

The Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdf

The Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdf

The Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdf

In this session, we will showcase how to revolutionize automated testing for your software, automation, and QA teams with UiPath Test Suite. In part 1 of UiPath test automation using UiPath Test Suite – developer series, we will cover, Software testing overview What is software testing Why software testing is required Typical test types and levels Continuous testing and challenges Introduction to UiPath Test Suite UiPath Test Suite family of products Speaker: Atul Trikha, Chief Technologist & Solutions Architect, Peraton and UiPath MVP Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP

UiPath Test Automation using UiPath Test Suite series, part 1

UiPath Test Automation using UiPath Test Suite series, part 1

UiPath Test Automation using UiPath Test Suite series, part 1

Automation is transforming industries by streamlining operations and reducing costs. However, the success of automated systems hinges not only on their functionality but also on the user experience (UX). This talk, "The UX of Automated Processes," delves into the intricate relationship between automation and UX, providing a comprehensive guide to designing intuitive and effective automated systems. Key Topics Covered: 1. Introduction to Automated UX: * Understanding the importance of UX in automated systems * Real-world example: Robotic beer dispensers at a music festival 2. Step-by-Step Process for Enhancing Automated UX: * Step 0: Identifying main workflows and happy paths * Step 1: Conducting contextual inquiries to understand current user practices and edge cases * Step 2: Prioritizing issues and solutions through collaborative ideation sessions * Step 3: Prototyping and usability testing for both software and hardware 3. Prototyping and Testing: * Methods for simulating automated processes * Importance of user feedback in refining automated systems 4. Designing for Support Users: * Crafting the experience for support personnel who manage edge cases * Ensuring effective communication and issue prioritization for support users

The UX of Automation by AJ King, Senior UX Researcher, Ocado

The UX of Automation by AJ King, Senior UX Researcher, Ocado

The UX of Automation by AJ King, Senior UX Researcher, Ocado

New customer? New industry? New cloud? New team? A lot to handle! How to ensure the success of the project? Start it well! I've created the 3 areas of focus at the beginning of the project that helped me in multiple roles (BA, PO, and Consultant). Learn from real-world experiences and discover how these insights can empower you to deliver unparalleled value to your customers right from the project's start.

Powerful Start- the Key to Project Success, Barbara Laskowska

Powerful Start- the Key to Project Success, Barbara Laskowska

Powerful Start- the Key to Project Success, Barbara Laskowska

Intrigued by why some of the world's largest companies (Netflix, Google, Cisco, Twitter, Uber etc) are using gRPC? In this demo based talk we delve into the world of gRPC in .Net, what it does and why we should use it. We compare the interface with both Rest and graphQL. We will show you how to implement grpc server-side in .net and in the web. Finally, I will show you how the tooling helps you deliver powerful interfaces and interact with them quickly and simply.

Demystifying gRPC in .Net by John Staveley

Demystifying gRPC in .Net by John Staveley

Demystifying gRPC in .Net by John Staveley

Recently uploaded (20)

FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...

FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...

FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...

Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf

Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf

Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf

THE BEST IPTV in GERMANY for 2024: IPTVreel

THE BEST IPTV in GERMANY for 2024: IPTVreel

THE BEST IPTV in GERMANY for 2024: IPTVreel

Intro in Product Management - Коротко про професію продакт менеджера

Intro in Product Management - Коротко про професію продакт менеджера

Intro in Product Management - Коротко про професію продакт менеджера

IoT Analytics Company Presentation May 2024

IoT Analytics Company Presentation May 2024

IoT Analytics Company Presentation May 2024

Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)

Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)

Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)

Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...

Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...

Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...

Enterprise Knowledge Graphs - Data Summit 2024

Enterprise Knowledge Graphs - Data Summit 2024

Enterprise Knowledge Graphs - Data Summit 2024

Extensible Python: Robustness through Addition - PyCon 2024

Extensible Python: Robustness through Addition - PyCon 2024

Extensible Python: Robustness through Addition - PyCon 2024

Custom Approval Process: A New Perspective, Pavel Hrbacek & Anindya Halder

Custom Approval Process: A New Perspective, Pavel Hrbacek & Anindya Halder

Custom Approval Process: A New Perspective, Pavel Hrbacek & Anindya Halder

Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...

Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...

Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...

10 Differences between Sales Cloud and CPQ, Blanka Doktorová

10 Differences between Sales Cloud and CPQ, Blanka Doktorová

10 Differences between Sales Cloud and CPQ, Blanka Doktorová

Designing for Hardware Accessibility at Comcast

Designing for Hardware Accessibility at Comcast

Designing for Hardware Accessibility at Comcast

Strategic AI Integration in Engineering Teams

Strategic AI Integration in Engineering Teams

Strategic AI Integration in Engineering Teams

Buy Epson EcoTank L3210 Colour Printer Online.pptx

Buy Epson EcoTank L3210 Colour Printer Online.pptx

Buy Epson EcoTank L3210 Colour Printer Online.pptx

The Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdf

The Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdf

The Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdf

UiPath Test Automation using UiPath Test Suite series, part 1

UiPath Test Automation using UiPath Test Suite series, part 1

UiPath Test Automation using UiPath Test Suite series, part 1

The UX of Automation by AJ King, Senior UX Researcher, Ocado

The UX of Automation by AJ King, Senior UX Researcher, Ocado

The UX of Automation by AJ King, Senior UX Researcher, Ocado

Powerful Start- the Key to Project Success, Barbara Laskowska

Powerful Start- the Key to Project Success, Barbara Laskowska

Powerful Start- the Key to Project Success, Barbara Laskowska

Demystifying gRPC in .Net by John Staveley

Demystifying gRPC in .Net by John Staveley

Demystifying gRPC in .Net by John Staveley

Storm over gearpump

1. Storm over Gearpump Manu Zhang

2. About Me ● Software Engineer at Big Data Technology Group, Intel ● contributor to hadoop-nativetask (https://github.com/apache/hadoop/tree/trunk/hadoop-mapreduce-project/hadoop -mapreduce-client/hadoop-mapreduce-client-nativetask) ● written storm-benchmark (https://github.com/intel-hadoop/storm-benchmark) ● working on Gearpump (kafka connector, storm compatibility, state) ● maintain a list of awesome-streaming (https://github.com/manuzhang/awesome-streaming)

3. Gearpump - Distributed Real-time Streaming Engine ● Akka / Actor Model ● Dynamic DAG ● Flow Control / Backpressure ● Low watermark ● At least once / Exactly-once ● High Availability ● Interactive Web UI

4. Gearpump Updates ● released 0.6.1 & 0.7.0 ● new documentation site ● secure YARN and HBase support ● Akka-stream compatibility ● Storm binary compatibility

5. Storm over Gearpump - Why ● Storm is widely used in the industry ● but has its limitations ● Gearpump is designed to overcome those limitations ● We want Storm users to benefit from Gearpump’s advanced features without any cost

6. Storm over Gearpump - Features ● binary compatible (no recompilation is required) with Storm 0.9.x ● multi-lang ● DRPC ● KafkaSpout / KafkaBolt ● Trident (WIP)

7. Similarities of Gearpump and Storm ● per-message ● Topology / DAG ● similar user interface

8. Storm over Gearpump - Overview

9. Storm over Gearpump - DAG Translation

10. Storm over Gearpump - Task Execution

11. Storm over Gearpump - Flow Control ● Acker is removed ● Flow control with back pressure for both acked and unacked Storm topologies

12. Storm over Gearpump - At Least Once ● each message is tagged with system time ● asynchronous non-blocking ack through tracking global minimum ack time ● support KafkaSpout for now

13. Performance ● SOL from storm-benchmark ● Storm 0.9.6 ● 4-node 10GbE cluster ● 16 workers ● 48 Spouts and 48 Bolts

14. Future work ● submit Storm Job through Web UI ● Storm 0.10 support ● At Least Once support for more spouts ● Trident support

15. References 1. https://github.com/gearpump/gearpump 2. https://gearpump.io 3. https://storm.apache.org 4. How to use Akka to make a PERFECT Streaming system 5. https://www.typesafe.com/blog/gearpump-real-time-streaming-engine-using-akka 6. http://akka.io/docs/