Streaming, Fast and Slow_Caito Scherr

•

1 like•380 views

What if you were given 2 weeks to prepare for running your first marathon, and had to be able to keep up with the fastest runner? This is the story of a small team that jumped head first into building a top tier stream processing service on a tight timeline, and how they prepared for being able to keep up with the fastest components. This talk will be of interest to those who want to build larger, more complex stream processing applications (and quickly), but will also benefit anyone who is looking to adopt a new streaming technology (or use it in a way that is new to their company), at any scale. Other takeaways include pragmatic steps to leverage unique features of various stream processing frameworks to more effectively solve common, real world data streaming problems.

Data & Analytics

STREAMING, FAST AND SLOW
June 2020
Caito Scherr
Caito_200_OK

• Caito Scherr
• Portland, OR, USA
• Software Engineer
• Apache Flink
Who am I?
Caito_200_OK

Agenda
Background
Integration
Operations
Analytics
Caito_200_OK

Agenda
Caito_200_OK
Background
Integration
Operations
Analytics

Background
Caito_200_OK
Product A usage Product B usage Product C usage
Subscription Info Account Info
Data
Store,
UI
Data pipeline

… Like training for your ﬁrst
marathon
Caito_200_OK

Caito_200_OK
… Like training for a new
marathon

Caito_200_OK
… Like training for a new
marathon
Integration

Caito_200_OK
… Like training for a new
marathon
Integration Operations

Caito_200_OK
… Like training for a new
marathon
Integration Operations Analytics

Ecosystem
Some of the best features of stream processing for
your team can also be the biggest drawbacks for
teams that interact with you.
Caito_200_OK

Ecosystem
Start by assuming each team may be
treating their data differently from the next.
Caito_200_OK

Ecosystem
Optimize for: understanding your
application’s impact, and allowing that data to drive
the architecture and design process.
Caito_200_OK

Streaming Community
Caito_200_OK
Documentation that is interactive,
A safe space for others to experiment,
And, take a map!

Streaming Community
Caito_200_OK
Don’t underestimate the power of a good map

Resources
Can your resources handle this new challenge?
Caito_200_OK

• Pulsar Summit
• Jun, Rosalie & the
conference event
staff!
• Maureen Dugan
Thank you!
• Ruby Andrews
• Eric Shamow
• Adam Klein
• Matthew Scharr
• Max Aguirre
Caito_200_OK

❖ https://kafkaesque.io/performance-comparison-between-apache-
pulsar-and-kafka-latency/
❖ http://pulsar.apache.org/docs/en/concepts-messaging/
Credits
Caito_200_OK

"Four out of five CEOs do not trust the data upon which they base their decisions." For many years, insight-driven organizations have understood the importance of data in decision-making. They have also understood the importance of data governance by having invested heavily in DataOps technologies. However, the problem of trust persists. In this session, you will discover how to better govern the last mile on the data journey, de-risk Analytics, and thus ensure user trust. Watch the session here: https://youtu.be/FOd5nswyGSY

Fermentation equipment auction malaysia 30 june 2020

Justin Kadis

Sitecore Commerce Analytics

pratik satikunvar

Derek Pearcy - Reading Users' Minds For Fun And Profit

bolt peters

What users say will generally be different from what they do -- this is true, but what's a good strategy when you can't get to enough of your users? What if you could answer some really big questions by performing simple research on ALL of your users? This is the same style of approach taken by companies like Google and Zynga, to target user research efforts which have made them what they are today. Log analysis, done well, can seem like mind-reading. If you haven't done it before: there's nothing to fear.

Data-Driven Software Design

Patrick McKenzie

PowerBI for Finance Teams

Anthony Key

Building a Beer Recommender with Yhat (PAPIs.io - November 2014)

Austin Ogilvie

***** Python for Data Science Training: https://www.edureka.co/python ***** This Edureka video on "Python Certification" will give you a complete insight of Python Certification, how to crack it and hands-on knowledge on various real-time projects offered by Edureka Python Certification Training. This video will help you to learn following topics: 1. Why go for Python Certification? 2. Advantages of Python 3. Career Opportunities 4. Python Job Profiles 5. Edureka: Python Projects Check out our Python Training Playlist: https://goo.gl/Na1p9G

Charles Farina - Analytics Pros (All Things Data 2015)

Shuki Mann

Cross Device Measurement - All Things Data Conference

Charles Farina

Productionizing Data Science at Experience

Matt Mills

Discover SAP BusinessObjects BI 4.3

Wiiisdom

Charlotte's top analytics tips!

Kate Waite

MLops workshop AWS

Gili Nachum

Cpu and execution of instruction.baabtra.com - No. 1 supplier of quality freshers

[Oracle Innovation Summit Tokyo 2018] オラクルの考えるAI、そのアプローチ

オラクルエンジニア通信

http://www.oracle.co.jp/campaign/innovation/2018/ ----- 企業における人工知能（AI）の活用が進んでいます。このセッションでは、OracleのAIへの様々な取り組みを事例を交えながらご紹介させていただきます。また、株式会社GAUSS様にもご登壇いただき、OracleのAIソリューションを活用した、新たな取組と展開についてご紹介いただきます。

Practical AWS Fargate

Richard Harvey

Speed up your Machine Learning workflows with build-in algorithms

Julien SIMON

Speed up your Machine Learning workflows with built-in algorithms - Tel Aviv ...

Amazon Web Services

In machine learning, training large models on massive amount of data usually improved results. Our customers report however that training such models and deploying them is either operationally prohibitive or outright impossible for them. Amazon AI Algorithms is designed to solve this problem. It is a collection of distributed streaming ML algorithms that scale to any amount of data. They are fast and efficient because they distribute across CPU.GPU machines and share a collective distributed state via a highly-optimized parameter server. They scale to an infinite amount of data because they operate in the streaming model. This means they require only one pass over the data and never increase their resources consumption allowing training to be paused resumed and snapshotted and even for algorithms to consume kinesis streams directly providing an "always on" training mechanism. They are production ready. Trained models are automatically containerized and usable in production using Amazon SageMaker hosting. Finally, we provide a convenient SDK which allows scientists to create new algorithms which operate in this model and enjoy all the benefits above.

SAS A00-415 Exam | Study Tips & Tricks

PalakMazumdar1

Click Here---> https://bit.ly/3OVzwOC <---Get complete detail on A00-415 exam guide to crack SAS Viya Fundamentals of Programming. You can collect all information on A00-415 tutorial, practice test, books, study material, exam questions, and syllabus. Firm your knowledge on SAS Viya Fundamentals of Programming and get ready to crack A00-415 certification. Explore all information on A00-415 exam with number of questions, passing percentage and time duration to complete test.

Sap bpc cpm study materials

Sap Materials

Contact Us At : www.erpexams.com Email Us At : sapmaterials4u@gmail.com ERP Certification, ERP Education, ERP Training Materials, ERP Courses DECEMBER OFFER: Get 18% FLAT OFF on SPOT Purchase & Payments*** ***NOW Offer Left For Just First 50 Customers ONLY Have Questions ?? <<--CLICK HERE ::: Get ERP Certification Courses For Any Below ERP Module ::: NEW: BO 4.0, TZMDG1, EH&S, HANA 1.0, Banking 8.0 Latest : EHP5 / EHP4 - SD, MM, EWM, LE, Banking 7.0, SCM 7.0, HANA , ECC 6.0 IS-Media : IMD320,IMD700, CRM 7.0, BusinessObjects-BOU310, BOU320, BOW310,BOW320,BOE320,BOE310,BOE330,BOI300,BOX310,BOQ300,BOC310,BOC320,BOC330 - Latest 2012 v096 Courses, BPC 10.0 - BPC410, BPC420, BPC430,BPC440 - Latest 2009 Courses, BPC340, Real Estate, EWM, GTS, BPC-CPM (OutLooksoft), Citrix & VMware Study & Training Materials, GRC, SOA, GTS, Public Sector, Tutorials, BluePrints, v091, v092, v093, v096, v010, v097, Col96, Col95,Col93, Col92, Col91, Col84, Col81, Col74, Col72, Col71, Col62, Col10, Col97, HA100, HA200, HA300 Many More New Courses Available. C O N T A C T: sapmaterials4u@gmail.com (For Questions & Live Google Chat) : www.erpexams.com ( Click On Any Navigation Links For Module Course List Details & Price ) ( Oracle : R12 Courses, Hyperion Materials All Available, Ask Us For Details )

Sap bpc cpm study materials

ebay_SAP

Agile Lab_BigData_MeetupPaolo Platter

Massive Streaming Analytics with Spark Streaming

Paolo Platter

Rpa conference new delhi

gaurav

How to use the SAP Connector for Alteryx Connect

DVW Analytics

Google Analytics Powerups and Smartcuts

Charles Meaden

Is Using KoP (Kafka-on-Pulsar) a Good Idea? - Pulsar Summit SF 2022

StreamNative

So, you are a responsible software engineer building microservices for Apache Kafka, and life is good. Eventually, you hear the community talking about the outstanding experience they are having with Apache Pulsar features. They talk about infinite event stream retention, a rebalance-free architecture, native support for event processing, and multi-tenancy. Exciting, right? Most people would want to migrate their code to Pulsar. Especially when you know that Pulsar also supports Kafka clients natively via the protocol handler known as KoP — which enables the Kafka client APIs on Pulsar. But, as said before, you are responsible; and you don't believe in fairy tales, just like you don't believe that migrations like this happen effortlessly. This session will discuss the architecture behind protocol handlers, what it means having one enabled on Pulsar, and how the KoP works. It will detail the effort required to migrate a microservice written for Kafka to Pulsar, and whether the code need to change for this.

Building an Asynchronous Application Framework with Python and Pulsar - Pulsa...

StreamNative

This talk describes Klaviyo’s internal messaging system, an asynchronous application framework built around Pulsar that provides a set of high-quality tools for building business-critical asynchronous data flows in unreliable environments. This framework includes: a pulsar ORM and schema migrator for topic configuration; a retry/replay system; a versioned schema registry; a consumer framework oriented around preventing message loss and in hostile environments while maximizing observability; an experimental “online schema change” for topics; and more. Development of this system was informed by lessons learned during heavy use of datastores like RabbitMQ and Kafka, and frameworks like Celery, Spark, and Flink. In addition to the capabilities of this system, this talk will also cover (sometimes painful) lessons learned about the process of converting a heterogenous async-computing environment onto Pulsar and a unified model.

Similar to Streaming, Fast and Slow_Caito Scherr

Python Certification | Data Science with Python Certification | Python Online...

Edureka!

Charles Farina - Analytics Pros (All Things Data 2015)

Shuki Mann

Cross Device Measurement - All Things Data Conference

Charles Farina

Productionizing Data Science at Experience

Matt Mills

Discover SAP BusinessObjects BI 4.3

Wiiisdom

Charlotte's top analytics tips!

Kate Waite

MLops workshop AWS

Gili Nachum

Cpu and execution of instruction.baabtra.com - No. 1 supplier of quality freshers

[Oracle Innovation Summit Tokyo 2018] オラクルの考えるAI、そのアプローチ

オラクルエンジニア通信

Practical AWS Fargate

Richard Harvey

Speed up your Machine Learning workflows with build-in algorithms

Julien SIMON

Speed up your Machine Learning workflows with built-in algorithms - Tel Aviv ...

Amazon Web Services

SAS A00-415 Exam | Study Tips & Tricks

PalakMazumdar1

Sap bpc cpm study materials

Sap Materials

Sap bpc cpm study materials

ebay_SAP

Agile Lab_BigData_MeetupPaolo Platter

Massive Streaming Analytics with Spark Streaming

Paolo Platter

Rpa conference new delhi

gaurav

How to use the SAP Connector for Alteryx Connect

DVW Analytics

Google Analytics Powerups and Smartcuts

Charles Meaden

Similar to Streaming, Fast and Slow_Caito Scherr (20)

Python Certification | Data Science with Python Certification | Python Online...

Charles Farina - Analytics Pros (All Things Data 2015)

Cross Device Measurement - All Things Data Conference

Productionizing Data Science at Experience

Discover SAP BusinessObjects BI 4.3

Charlotte's top analytics tips!

MLops workshop AWS

Cpu and execution of instruction.

[Oracle Innovation Summit Tokyo 2018] オラクルの考えるAI、そのアプローチ

Practical AWS Fargate

Speed up your Machine Learning workflows with build-in algorithms

Speed up your Machine Learning workflows with built-in algorithms - Tel Aviv ...

SAS A00-415 Exam | Study Tips & Tricks

Sap bpc cpm study materials

Agile Lab_BigData_Meetup

Massive Streaming Analytics with Spark Streaming

Rpa conference new delhi

How to use the SAP Connector for Alteryx Connect

Google Analytics Powerups and Smartcuts

More from StreamNative

Is Using KoP (Kafka-on-Pulsar) a Good Idea? - Pulsar Summit SF 2022

StreamNative

Building an Asynchronous Application Framework with Python and Pulsar - Pulsa...

StreamNative

Blue-green deploys with Pulsar & Envoy in an event-driven microservice ecosys...

StreamNative

In this talk, learn how Toast leverages our Envoy control-plane to manage blue-green deploys of Pulsar consumers, and how this has helped drive adoption across the engineering organization. Dive into the history of Pulsar at Toast, starting from its introduction in 2019 to provide event-driven architecture across a rapidly scaling restaurant software platform. We will detail some of the hurdles that we encountered gaining buy-in across a diverse set of teams, and dive deep into how we enforce best practices and integrate with our service control plane.

Distributed Database Design Decisions to Support High Performance Event Strea...

StreamNative

Event streaming architectures launched a reexamination of applications and systems architectures across the board. We live in a world where answers are needed now in a constant real-time flow. Yet beyond the event streaming system itself, what are the corequisites to ensure our large scale distributed database systems can keep pace with this always-on, always-current real time flow of data? What are the requirements and expectations for this next tech cycle?

Simplify Pulsar Functions Development with SQL - Pulsar Summit SF 2022

StreamNative

Pulsar Functions is a succinct framework provided by Apache Pulsar to conduct real-time data processing. Its use cases include ETL pipeline, event-driven applications, and simple data analytics. While Pulsar Functions already provides an extremely simple programming interface, we want to further lower the barrier for users to access real-time data. Since SQL is one of the universal languages in the technology world and well accepted by the vast majority of data engineers, we decided to add a SQL expressing layer on top of Pulsar Functions runtime. In this talk, we will discuss the architecture and implementation of this new service. We will see how SQL syntax, Pulsar Functions, and Function Mesh can work together to deliver a unique user development experience for real-time data jobs in the cloud environment. We will also walk through use cases like filtering, routing, and projecting messages as well as integrating with the Pulsar IO Connectors framework.

Towards a ZooKeeper-less Pulsar, etcd, etcd, etcd. - Pulsar Summit SF 2022

StreamNative

Starting with version 2.10, the Apache ZooKeeper dependency has been eliminated and replaced with a pluggable framework that enables you to reduce the infrastructure footprint of Apache Pulsar by leveraging alternative metadata and coordination systems based on your deployment environment. In this talk, walk through the steps required to utilize the existing etcd service running inside Kubernetes to act as Pulsar's metadata store, thereby eliminating the need to run ZooKeeper entirely, leaving you with a Zookeeper-less Pulsar.

Validating Apache Pulsar’s Behavior under Failure Conditions - Pulsar Summit ...

StreamNative

Apache Pulsar is a highly available, distributed messaging system that provides guarantees of no message loss and strong message ordering with predictable read and write latency. In this talk, learn how this can be validated for Apache Pulsar Kubernetes deployments. Various failures are injected using Chaos Mesh to simulate network and other infrastructure failure conditions. There are many questions that are asked about failure scenarios, but it could be hard to find answers to these important questions. When a failure happens, how long does it take to recover? Does it cause unavailability? How does it impact throughput and latency? Are the guarantees of no message loss and strong message ordering kept, even when components fail? If a complete availability zone fails, is the system configured correctly to handle AZ failures? This talk will help you find answers to these questions and apply the tooling and practices to your own testing and validation.

Cross the Streams! Creating Streaming Data Pipelines with Apache Flink + Apac...

StreamNative

Despite what the Ghostbusters said, we’re going to go ahead and cross (or, join) the streams. This session covers getting started with streaming data pipelines, maximizing Pulsar’s messaging system alongside one of the most flexible streaming frameworks available, Apache Flink. Specifically, we’ll demonstrate the use of Flink SQL, which provides various abstractions and allows your pipeline to be language-agnostic. So, if you want to leverage the power of a high-speed, highly customizable stream processing engine without the usual overhead and learning curves of the technologies involved (and their interconnected relationships), then this talk is for you. Watch the step-by-step demo to build a unified batch and streaming pipeline from scratch with Pulsar, via the Flink SQL client. This means you don’t need to be familiar with Flink, (or even a specific programming language). The examples provided are built for highly complex systems, but the talk itself will be accessible to any experience level.

Message Redelivery: An Unexpected Journey - Pulsar Summit SF 2022

StreamNative

Apache Pulsar depends upon message acknowledgments to provide at-least-once or exactly-once processing guarantees. With these guarantees, any transmission between the broker and its producers and consumers requires an acknowledgment. But what happens if an acknowledgment is not received? Resending the message introduces the potential of duplicate processing and increases the likelihood of out or order processing. Therefore, it is critical to understand the Pulsar message redelivery semantics in order to prevent either of these conditions. In this talk, we will walk you through the redelivery semantics of Apache Pulsar, and highlight some of the control mechanisms available to application developers to control this behavior. Finally, we will present best practices for configuring message redelivery to suit various use cases.

Unlocking the Power of Lakehouse Architectures with Apache Pulsar and Apache ...

StreamNative

Lakehouses are quickly growing in popularity as a new approach to Data Platform Architecture bringing some of the long-established benefits from OLTP world to OLAP, including transactions, record-level updates/deletes, and changes streaming. In this talk, we will discuss Apache Hudi and how it unlocks possibilities of building your own fully open-source Lakehouse featuring a rich set of integrations with existing technologies, including Apache Pulsar. In this session, we will present: - What Lakehouses are, and why they are needed. - What Apache Hudi is and how it works. - Provide a use-case and demo that applies Apache Hudi’s DeltaStreamer tool to ingest data from Apache Pulsar.

Understanding Broker Load Balancing - Pulsar Summit SF 2022

StreamNative

Pulsar is a horizontally scalable messaging system, so the traffic in a logical cluster must be balanced across all the available Pulsar brokers as evenly as possible, in order to ensure full utilization of the broker layer. You can use multiple settings and tools to control the traffic distribution which requires a bit of context to understand how the traffic is managed in Pulsar. In this talk, we will walk you through the load balancing capabilities of Apache Pulsar, and highlight some of the control mechanisms available to control the distribution of load across the Pulsar brokers. Finally, we will discuss the various loading shedding strategies that are available. At the end of the talk, you will have a better understanding of how Pulsar's broker level auto-balancing works, and how to properly configure it to meet your workload demands.

Building an Asynchronous Application Framework with Python and Pulsar - Pulsa...

StreamNative

Pulsar's Journey in Yahoo!: On-prem, Cloud and Hybrid - Pulsar Summit SF 2022

StreamNative

In today’s world, we are seeing a big shift toward the Cloud. With this shift comes a big shift in the expectations we have for a messaging system, especially when the messaging system is presented as managed service in a large-scale, multi-tenant environment. For any large-scale enterprise, it’s very important to evaluate messaging system and be confident before expanding complex distributed data systems like Apache Pulsar from on-premise to elastically scalable, fully managed services on cloud services. We must consider aspects such as: migration from and integration with large-scale on-premise clusters, security, cost efficiency, and the cloud friendliness of the architecture, modeling cost and capacity, tenant isolation, deployment robustness, availability, monitoring, etc. Not every messaging system is built to be cloud-native and run as a managed service with cost efficiency. We have been running large-scale Apache Pulsar at Yahoo for the last 8 years on various platforms and hardware configurations while meeting application SLAs and serving more than 1M topics in a cluster. In this talk, we will talk about Pulsar’s journey in Yahoo! from an on-premise platform to a hybrid cloud and on-premise system. We will talk about Pulsar’s architecture and features that make Pulsar a good cloud-native messaging-system choice for any enterprise.

Event-Driven Applications Done Right - Pulsar Summit SF 2022

StreamNative

Pulsar Summit San Francisco is the event dedicated to Apache Pulsar. This one-day, action-packed event will include 5 keynotes, 12 breakout sessions, and 1 amazing happy hour. Speakers are from top companies, including Google, AWS, Databricks, Onehouse, StarTree, Intel, ScyllaDB, and more! It’s the perfect opportunity to network with Pulsar thought leaders in person. Join developers, architects, data engineers, DevOps professionals, and anyone who wants to learn about messaging and event streaming for this one-day, in-person event. Pulsar Summit San Francisco brings the Apache Pulsar Community together to share best practices and discuss the future of streaming technologies.

Pulsar @ Scale. 200M RPM and 1K instances - Pulsar Summit SF 2022

StreamNative

Our services team creates, builds, and maintains the as a service offering for base platform services within our organization. Several thousand applications use these custom services daily generating more than 700 million requests per minute. One of these services was our publish / subscriber offering, BQ with custom SDK and custom metrics based on Apache Pulsar. BQ is the core communication service within our organization, having more 200M RPM. All the core processes of the organization depend on this service for operation: the CDC of any of our RDBMS or NoSQL offering, all the eventing efforts of the organization, async communication between apps, notification systems, etc. The backend of the solution was Apache Pulsar running on EC2 on AWS and on top of that we built several components as wrappers of the actual backend, creating our own SDKs and abstractions and in many ways extending the features provided by Pulsar. We had a multi-cluster setup 100% on AWS, with custom Pulsar Docker images running on large ASG setups, along with our own wrapping and admin APIs and DBs. All of this in turn transformed the solution into a volatile solution.

Data Democracy: Journey to User-Facing Analytics - Pulsar Summit SF 2022

StreamNative

There is an increasing need to unleash analytical capabilities directly to the end-users to democratize decision-making. User-Facing Analytics is a new frontier that will shape the products of tomorrow and push the limits of existing technology. It demands a solution that will scale to millions of users to provide fast, real-time insights. In this session, Xiang will talk about his journey to build Apache Pinot to tackle the analytics problem space with the architectural changes and technology inventions made over the past decade. He will also talk about how other big data companies such as LinkedIn, Uber, and Stripe power their user-facing analytical applications.

Beam + Pulsar: Powerful Stream Processing at Scale - Pulsar Summit SF 2022

StreamNative

Welcome and Opening Remarks - Pulsar Summit SF 2022

StreamNative

Log System As Backbone – How We Built the World’s Most Advanced Vector Databa...

StreamNative

Milvus is an open-source vector database that leverages a novel data fabric to build and manage vector similarity search applications. As the world's most popular vector database, it has already been adopted in production by thousands of companies around the world, including Lucidworks, Shutterstock, and Cloudinary. With the launch of Milvus 2.0, the community aims to introduce a cloud-native, highly scalable and extendable vector similarity solution, and the key design concept is log as data. Milvus relies on Pulsar as the log pub/sub system. Pulsar helps Milvus to reduce system complexity by loosely decoupling each micro service, making the system stateless by disaggregating log storage and computation, which also makes the system further extendable. We will introduce the overview design, the implementation details of Milvus and its roadmap in this topic. Takeaways: 1) Get a general idea about what is a vector database and its real-world use cases. 2) Understand the major design principles of Milvus 2.0. 3) Learn how to build a complex system with the help of a modern log system like Pulsar.

MoP(MQTT on Pulsar) - a Powerful Tool for Apache Pulsar in IoT - Pulsar Summi...

StreamNative

MQTT (Message Queuing Telemetry Transport,) is a message protocol based on the pub/sub model with the advantages of compact message structure, low resource consumption, and high efficiency, which is suitable for IoT applications with low bandwidth and unstable network environments. This session will introduce MQTT on Pulsar, which allows developers users of MQTT transport protocol to use Apache Pulsar. I will share the architecture, principles and future planning of MoP, to help you understand Apache Pulsar's capabilities and practices in the IoT industry.

More from StreamNative (20)