Data Democracy: Journey to User-Facing Analytics - Pulsar Summit SF 2022

•

0 likes•22 views

There is an increasing need to unleash analytical capabilities directly to the end-users to democratize decision-making. User-Facing Analytics is a new frontier that will shape the products of tomorrow and push the limits of existing technology. It demands a solution that will scale to millions of users to provide fast, real-time insights. In this session, Xiang will talk about his journey to build Apache Pinot to tackle the analytics problem space with the architectural changes and technology inventions made over the past decade. He will also talk about how other big data companies such as LinkedIn, Uber, and Stripe power their user-facing analytical applications.

Technology

Pulsar Summit
San Francisco
Hotel Nikko
August 18 2022
Keynote
Data Democracy:
Journey to
User-Facing Analytics
Xiang Fu
Co-Founder • StarTree

About Me:
Co-Founder at StarTree, cloud-native platform to build the next generation of data
analytics applications for millions of users.
Founder and PMC of Apache Pinot, a realtime, distributed OLAP datastore
Previously, architect at Uber's data platform team solving streaming data serving,
processing, and analytics problems at a large scale.

Data Democracy - Are We There?
SQL Editors
Dashboard
Internal Facing Analytics
Operators
Analysts
Past — Present
Current technology has done a
great job delivering insights for
INTERNAL USERS
Analytical Data Apps
latency-sensitive
User Facing Analytics
Present — Future
Users Customers
To truly democratize data, we
need to deliver high quality
insights to EXTERNAL USERS
The Gap

Vanishing window of
opportunity for events
Time Value of Data
Value
Time

Event
Insight
Streaming Changed The Game…
Data warehouses & lakes
Hours to Days
Stream
Milliseconds to Seconds

And Started A Cycle
Streaming technologies like PULSAR increased speed and reduced costs
to store events, kicking of a cycle…
Collect more
events
Improved user
experience
Increase user
engagement

Streaming
Sources
Messaging
Pub-sub
Log
Aggregation
Streaming
Processing
Real Time
Analytics
Streaming Spawned New Use Cases

- Ingest data as soon as events happen
- Query that data as soon as it’s ingested
- Do above at scale.
In simple terms, we need to:
How Do We Do Real-time Analytics ?
Simple is HARD!

Ingestion Sources (Real-time, Batch, SQL)
Efficient compute and indexing powerhouse
Compressed and scalable storage (PB scale)
Advanced Query Support
Multi-Tenant and Distributed Architecture
Apache Pinot At A Glance

5000 queries/sec
~5ms average latency
<100ms 95th percentile
2016
After Pinot
5,000 Queries / sec
700M+ members
Before Pinot
1500 Queries / sec
200M+ members
2014
45X Improvement
in Efficiency
1000 Nodes
75 Nodes
Apache Pinot Impact

2013 2015 2019 2021
Started @ LinkedIn Apache Graduation
StarTree Founded
Open Source
Apache Pinot Timeline

40+
Companies
Slack Users
800
55k
Downloads
100+
Companies
Slack Users
2500+
1M+
Downloads
2020
2022
Apache Pinot Community Growth

INTERNAL FACING ANALYTICS
USER FACING ANALYTICS
Business Analysts
Platform Operators
Application Users
Business Partners
Food Delivery FinTech
Long Orders Insights
Nearby Orders in App,
Restaurants Manager Dashboard
Merchants Dashboard
Ledger Observability
Real Time Use Cases

events/sec
1M+
queries/sec
200K+
query
latency
Ms
data size
1PB+
rows
1T
query
latency
< 1s
data size
200TB+
queries/sec
30K+
query
latency
< 100ms
Conﬁdential - Do not duplicate or distribute without consent of StarTree Inc.
Apache Pinot At Scale

Democratizing data through
User-Facing Analytics

Contact Me:
Thank you!
Pulsar Summit
San Francisco
Hotel Nikko
August 18 2022
xiangfu@startree.ai
@xiangfu0

Enterprises are rapidly adopting stream computing backbones, in-memory data stores, change data capture, and other low-latency approaches for end-to-end applications. As businesses modernize their data architectures over the next several years, they will begin to evolve toward all-streaming architectures. In this webcast, Wikibon, Attunity, and MemSQL will discuss how enterprise data professionals should migrate their legacy architectures in this direction. They will provide guidance for migrating data lakes, data warehouses, data governance, and transactional databases to support all-streaming architectures for complex cloud and edge applications. They will discuss how this new architecture will drive enterprise strategies for operationalizing artificial intelligence, mobile computing, the Internet of Things, and cloud-native microservices. Link to the Wikibon report - wikibon.com/wikibons-2018-big-data-analytics-trends-forecast Link to Attunity Streaming CDC Book Download - http://www.bit.ly/cdcbook Link to MemSQL's Free Data Pipeline Book - http://go.memsql.com/oreilly-data-pipelines

Take Action: The New Reality of Data-Driven Business

Inside Analysis

The Briefing Room with Dr. Robin Bloor and WebAction Live Webcast on July 23, 2014 Watch the archive: https://bloorgroup.webex.com/bloorgroup/lsr.php?RCID=360d371d3a49ad256942f55350aa0a8b The waiting used to be the hardest part, but not anymore. Today’s cutting-edge enterprises can seize opportunities faster than ever, thanks to an array of technologies that enable real-time responsiveness across the spectrum of business processes. Early adopters are solving critical business challenges by enabling the rapid-fire design, development and production of very specific applications. Functionality can range from improved customer engagement to dynamic machine-to-machine interactions. Register for this episode of The Briefing Room to learn from veteran Analyst Dr. Robin Bloor, who will tout a new era in data-driven organizations, and why a data flow architecture will soon be critical for industry leaders. He’ll be briefed by Sami Akbay of WebAction, who will showcase his company’s real-time data management platform, which combines all the component parts needed to access, process and leverage data big and small. He’ll explain how this new approach can provide game-changing power to organizations of all types and sizes. Visit InsideAnlaysis.com for more information.

Modern Data Architectures for Business Outcomes

Amazon Web Services

The Cloud - What's different

Chen-Tien Tsai

Accelerate Self-Service Analytics with Data Virtualization and Visualization

Denodo

Watch full webinar here: https://bit.ly/39AhUB7 Enterprise organizations are shifting to self-service analytics as business users need real-time access to holistic and consistent views of data regardless of its location, source or type for arriving at critical decisions. Data Virtualization and Data Visualization work together through a universal semantic layer. Learn how they enable self-service data discovery and improve performance of your reports and dashboards. In this session, you will learn: - Challenges faced by business users - How data virtualization enables self-service analytics - Use case and lessons from customer success - Overview of the highlight features in Tableau

Financial Services Analytics on AWS

Amazon Web Services

Platforms, Cloud-Native Architectures, and APIs: Chicago Adapt or Die Keynote

Apigee | Google Cloud

The speed of innovation today creates tremendous opportunities for some, existential threats for others. Companies that win create their own success by leveraging modern data platforms. While architectures vary, the foundation is often in-memory, and the latency is real-time. Register for this Special Edition of The Briefing Room to hear veteran Analyst Dr. Robin Bloor explain how today's data platforms enable the modern enterprise in groundbreaking ways. He'll be briefed by Chris Hallenbeck of SAP who will demonstrate how forward-looking companies are leveraging real-time data platforms to achieve operational excellence, make decisions faster, and find new ways to innovate.

Tibco Augmented Intelligence - Analytics, IoT, Big Data, Streaming 20161025

Nicola Sandoli

Modern Data Architectures for Business Outcomes

Amazon Web Services

Real-time Visibility at Scale with Sumo Logic

Amazon Web Services

Legacy monitoring and troubleshooting tools can limit visibility and control over your infrastructure and applications. Organizations must find monitoring and troubleshooting tools that can scale with the volume, variety and velocity of data generated by today’s complex applications in order to keep pace with business demands. Our upcoming webinar will discuss how Sumo Logic helped Scripps Networks harness cloud-native machine data analytics to improve application quality and reliability on AWS. Sumo Logic allows IT operations teams to visualize and monitor workloads in real-time, identify issues and expedite root-cause analysis across the AWS environment. Join us to learn: • How to migrate from traditional on-premises data centers to AWS with confidence • How to improve the monitoring and troubleshooting of modern applications • How Scripps Networks, a leading content developer, used Sumo Logic to optimize their transition to AWS Who should attend: Developers, DevOps Director/Manager, IT Operations Director/Manager, Director of Cloud/Infrastructure, VP of Engineering

UiPath 23.4 Product Release Updates

DianaGray10

Join us for this Canadian Chapter Event where Dhruv Patel, Senior Sales Engineer at UiPath will walk you through the latest product release. Topics covered will include: What is in the new 23.4 Product Release What changes you should look out for Where you can find more information on the release How you can benefit from the updates We look forward to spending time with you and answering any questions you might have.

WSO2 ITALIA SMARTTALK #8 ASYNCAPI.pdf

Profesia Srl, Lynx Group

Re-Inventing Enterprise IT Around APIs & Apps

WSO2

Envisioning the Future Enterprise

WSO2

AWS Toronto Content Production symposium - Welcome

Amazon Web Services

Welcome to AWS for Content Production in the Cloud. This is an overview of AWS custom-tailored for Content Production firms, who want to bring their workflow to the AWS Cloud. In this presentation you will find a list of reference customers, the AWS Reference Workflow and a description of the topics which later detail how to implement rendering, video editing and content production using the AWS cloud.

Streaming Data and Stream Processing with Apache Kafka

confluent

Big Data Paris - A Modern Enterprise Architecture

MongoDB

Depuis les années 1980, le volume de données produit et le risque lié à ces données ont littéralement explosé. 90% des données existantes aujourd’hui ont été créé ces 2 dernières années, dont 80% sont non structurées. Avec plus d’utilisateurs et le besoin de disponibilité permanent, les risques sont beaucoup plus élevés. Quels sont les paramètres de bases de données qu’un décideur doit prendre en compte pour déployer ses applications innovantes?

Smart Enterprise Big Data Bus for the Modern Responsive Enterprise

DataWorks Summit

API and App Ecosystems - Build The Best: a deep dive

Cisco DevNet

Digital Transformation Mindset - More Than Just Technology

confluent

Many enterprises faced with silo’ed, batch-oriented, legacy systems struggle to compete in this new digital-first world. Adhering to the ‘If it’s not broken don’t fix it’ mentality leaves the door wide open for native digital challengers to grow and succeed. To stay competitive, your organization must respond in real time to every customer experience transaction, sale, and market movement. But how do you get there? First, you must change your mindset. As streaming platforms become central to data strategies, companies both small and large are re-thinking their enterprise architecture with real-time context at the forefront. Monoliths are evolving into microservices. Datacenters are moving to the cloud. What was once a ‘batch’ mindset is quickly being replaced with stream processing as the demands of the business impose real-time requirements on technology leaders. Join Argyle, in partnership with Confluent, in our 2018 CIO Virtual Event: The Digital Transformation Mindset – More Than Just Technology. During the webinar we’ll learn how leading companies across industries rely on a streaming platform to make event-driven architectures central to: • How data strategies and IT initiatives are improving the digital customer experiences • How executives are reducing risk with real time monitoring and anomaly detection • Increasing operational agility with microservices and IoT architectures within organizations

Real-Time Analytics with Confluent and MemSQL

SingleStore

#TDXRecap India tour

Shashank Srivatsavaya (ShashForce)

Confluent Partner Tech Talk with BearingPoint

confluent

Initiate Edinburgh 2019 - Big Data Meets AI

Amazon Web Services

Over 90% of today’s data has been generated in the last two years, and growth rates continue to climb. In this session, we’ll step through challenges and best practices with data capturing, how to derive meaningful insights to help predict the future, and common pitfalls in data analysis. Come discover how integrated solutions involving Amazon S3, AWS Glue, Amazon Redshift, Amazon Athena, Amazon EMR, Amazon Kinesis, and Amazon Machine Learning/Deep Learning result in effective data systems for data scientists and business users, alike.

Streaming Visualization

Guido Schmutz

Most data visualisation solutions today still work on data sources which are stored persistently in a data store, using the so called “data at rest” paradigms. More and more data sources today provide a constant stream of data, from IoT devices to Social Media streams. These data stream publish with high velocity and messages often have to be processed as quick as possible. For the processing and analytics on the data, so called stream processing solutions are available. But these only provide minimal or no visualisation capabilities. One was is to first persist the data into a data store and then use a traditional data visualisation solution to present the data. If latency is not an issue, such a solution might be good enough. An other question is which data store solution is necessary to keep up with the high load on write and read. If it is not an RDBMS but an NoSQL database, then not all traditional visualisation tools might already integrate with the specific data store. An other option is to use a Streaming Visualisation solution. They are specially built for streaming data and often do not support batch data. A much better solution would be to have one tool capable of handling both, batch and streaming data. This talk presents different architecture blueprints for integrating data visualisation into a fast data solution and highlights some of the products available to implement these blueprints.

Lunch and Learn and Sneakers

Bill Zajac

Is Using KoP (Kafka-on-Pulsar) a Good Idea? - Pulsar Summit SF 2022

StreamNative

So, you are a responsible software engineer building microservices for Apache Kafka, and life is good. Eventually, you hear the community talking about the outstanding experience they are having with Apache Pulsar features. They talk about infinite event stream retention, a rebalance-free architecture, native support for event processing, and multi-tenancy. Exciting, right? Most people would want to migrate their code to Pulsar. Especially when you know that Pulsar also supports Kafka clients natively via the protocol handler known as KoP — which enables the Kafka client APIs on Pulsar. But, as said before, you are responsible; and you don't believe in fairy tales, just like you don't believe that migrations like this happen effortlessly. This session will discuss the architecture behind protocol handlers, what it means having one enabled on Pulsar, and how the KoP works. It will detail the effort required to migrate a microservice written for Kafka to Pulsar, and whether the code need to change for this.

Building an Asynchronous Application Framework with Python and Pulsar - Pulsa...

StreamNative

This talk describes Klaviyo’s internal messaging system, an asynchronous application framework built around Pulsar that provides a set of high-quality tools for building business-critical asynchronous data flows in unreliable environments. This framework includes: a pulsar ORM and schema migrator for topic configuration; a retry/replay system; a versioned schema registry; a consumer framework oriented around preventing message loss and in hostile environments while maximizing observability; an experimental “online schema change” for topics; and more. Development of this system was informed by lessons learned during heavy use of datastores like RabbitMQ and Kafka, and frameworks like Celery, Spark, and Flink. In addition to the capabilities of this system, this talk will also cover (sometimes painful) lessons learned about the process of converting a heterogenous async-computing environment onto Pulsar and a unified model.

Similar to Data Democracy: Journey to User-Facing Analytics - Pulsar Summit SF 2022

A Winning Strategy for the Digital Economy

Eric Kavanagh

Tibco Augmented Intelligence - Analytics, IoT, Big Data, Streaming 20161025

Nicola Sandoli

Modern Data Architectures for Business Outcomes

Amazon Web Services

Real-time Visibility at Scale with Sumo Logic

Amazon Web Services

UiPath 23.4 Product Release Updates

DianaGray10

WSO2 ITALIA SMARTTALK #8 ASYNCAPI.pdf

Profesia Srl, Lynx Group

Re-Inventing Enterprise IT Around APIs & Apps

WSO2

Envisioning the Future Enterprise

WSO2

AWS Toronto Content Production symposium - Welcome

Amazon Web Services

Streaming Data and Stream Processing with Apache Kafka

confluent

Big Data Paris - A Modern Enterprise Architecture

MongoDB

Smart Enterprise Big Data Bus for the Modern Responsive Enterprise

DataWorks Summit

API and App Ecosystems - Build The Best: a deep dive

Cisco DevNet

Digital Transformation Mindset - More Than Just Technology

confluent

Real-Time Analytics with Confluent and MemSQL

SingleStore

#TDXRecap India tour

Shashank Srivatsavaya (ShashForce)

Confluent Partner Tech Talk with BearingPoint

confluent

Initiate Edinburgh 2019 - Big Data Meets AI

Amazon Web Services

Streaming Visualization

Guido Schmutz

Lunch and Learn and Sneakers

Bill Zajac

Similar to Data Democracy: Journey to User-Facing Analytics - Pulsar Summit SF 2022 (20)

A Winning Strategy for the Digital Economy

Tibco Augmented Intelligence - Analytics, IoT, Big Data, Streaming 20161025

Modern Data Architectures for Business Outcomes

Real-time Visibility at Scale with Sumo Logic

UiPath 23.4 Product Release Updates

WSO2 ITALIA SMARTTALK #8 ASYNCAPI.pdf

Re-Inventing Enterprise IT Around APIs & Apps

Envisioning the Future Enterprise

AWS Toronto Content Production symposium - Welcome

Streaming Data and Stream Processing with Apache Kafka

Big Data Paris - A Modern Enterprise Architecture

Smart Enterprise Big Data Bus for the Modern Responsive Enterprise

API and App Ecosystems - Build The Best: a deep dive

Digital Transformation Mindset - More Than Just Technology

Real-Time Analytics with Confluent and MemSQL

#TDXRecap India tour

Confluent Partner Tech Talk with BearingPoint

Initiate Edinburgh 2019 - Big Data Meets AI

Streaming Visualization

Lunch and Learn and Sneakers

More from StreamNative

Is Using KoP (Kafka-on-Pulsar) a Good Idea? - Pulsar Summit SF 2022

StreamNative

Building an Asynchronous Application Framework with Python and Pulsar - Pulsa...

StreamNative

Blue-green deploys with Pulsar & Envoy in an event-driven microservice ecosys...

StreamNative

In this talk, learn how Toast leverages our Envoy control-plane to manage blue-green deploys of Pulsar consumers, and how this has helped drive adoption across the engineering organization. Dive into the history of Pulsar at Toast, starting from its introduction in 2019 to provide event-driven architecture across a rapidly scaling restaurant software platform. We will detail some of the hurdles that we encountered gaining buy-in across a diverse set of teams, and dive deep into how we enforce best practices and integrate with our service control plane.

Distributed Database Design Decisions to Support High Performance Event Strea...

StreamNative

Event streaming architectures launched a reexamination of applications and systems architectures across the board. We live in a world where answers are needed now in a constant real-time flow. Yet beyond the event streaming system itself, what are the corequisites to ensure our large scale distributed database systems can keep pace with this always-on, always-current real time flow of data? What are the requirements and expectations for this next tech cycle?

Simplify Pulsar Functions Development with SQL - Pulsar Summit SF 2022

StreamNative

Pulsar Functions is a succinct framework provided by Apache Pulsar to conduct real-time data processing. Its use cases include ETL pipeline, event-driven applications, and simple data analytics. While Pulsar Functions already provides an extremely simple programming interface, we want to further lower the barrier for users to access real-time data. Since SQL is one of the universal languages in the technology world and well accepted by the vast majority of data engineers, we decided to add a SQL expressing layer on top of Pulsar Functions runtime. In this talk, we will discuss the architecture and implementation of this new service. We will see how SQL syntax, Pulsar Functions, and Function Mesh can work together to deliver a unique user development experience for real-time data jobs in the cloud environment. We will also walk through use cases like filtering, routing, and projecting messages as well as integrating with the Pulsar IO Connectors framework.

Towards a ZooKeeper-less Pulsar, etcd, etcd, etcd. - Pulsar Summit SF 2022

StreamNative

Starting with version 2.10, the Apache ZooKeeper dependency has been eliminated and replaced with a pluggable framework that enables you to reduce the infrastructure footprint of Apache Pulsar by leveraging alternative metadata and coordination systems based on your deployment environment. In this talk, walk through the steps required to utilize the existing etcd service running inside Kubernetes to act as Pulsar's metadata store, thereby eliminating the need to run ZooKeeper entirely, leaving you with a Zookeeper-less Pulsar.

Validating Apache Pulsar’s Behavior under Failure Conditions - Pulsar Summit ...

StreamNative

Apache Pulsar is a highly available, distributed messaging system that provides guarantees of no message loss and strong message ordering with predictable read and write latency. In this talk, learn how this can be validated for Apache Pulsar Kubernetes deployments. Various failures are injected using Chaos Mesh to simulate network and other infrastructure failure conditions. There are many questions that are asked about failure scenarios, but it could be hard to find answers to these important questions. When a failure happens, how long does it take to recover? Does it cause unavailability? How does it impact throughput and latency? Are the guarantees of no message loss and strong message ordering kept, even when components fail? If a complete availability zone fails, is the system configured correctly to handle AZ failures? This talk will help you find answers to these questions and apply the tooling and practices to your own testing and validation.

Cross the Streams! Creating Streaming Data Pipelines with Apache Flink + Apac...

StreamNative

Despite what the Ghostbusters said, we’re going to go ahead and cross (or, join) the streams. This session covers getting started with streaming data pipelines, maximizing Pulsar’s messaging system alongside one of the most flexible streaming frameworks available, Apache Flink. Specifically, we’ll demonstrate the use of Flink SQL, which provides various abstractions and allows your pipeline to be language-agnostic. So, if you want to leverage the power of a high-speed, highly customizable stream processing engine without the usual overhead and learning curves of the technologies involved (and their interconnected relationships), then this talk is for you. Watch the step-by-step demo to build a unified batch and streaming pipeline from scratch with Pulsar, via the Flink SQL client. This means you don’t need to be familiar with Flink, (or even a specific programming language). The examples provided are built for highly complex systems, but the talk itself will be accessible to any experience level.

Message Redelivery: An Unexpected Journey - Pulsar Summit SF 2022

StreamNative

Apache Pulsar depends upon message acknowledgments to provide at-least-once or exactly-once processing guarantees. With these guarantees, any transmission between the broker and its producers and consumers requires an acknowledgment. But what happens if an acknowledgment is not received? Resending the message introduces the potential of duplicate processing and increases the likelihood of out or order processing. Therefore, it is critical to understand the Pulsar message redelivery semantics in order to prevent either of these conditions. In this talk, we will walk you through the redelivery semantics of Apache Pulsar, and highlight some of the control mechanisms available to application developers to control this behavior. Finally, we will present best practices for configuring message redelivery to suit various use cases.

Unlocking the Power of Lakehouse Architectures with Apache Pulsar and Apache ...

StreamNative

Lakehouses are quickly growing in popularity as a new approach to Data Platform Architecture bringing some of the long-established benefits from OLTP world to OLAP, including transactions, record-level updates/deletes, and changes streaming. In this talk, we will discuss Apache Hudi and how it unlocks possibilities of building your own fully open-source Lakehouse featuring a rich set of integrations with existing technologies, including Apache Pulsar. In this session, we will present: - What Lakehouses are, and why they are needed. - What Apache Hudi is and how it works. - Provide a use-case and demo that applies Apache Hudi’s DeltaStreamer tool to ingest data from Apache Pulsar.

Understanding Broker Load Balancing - Pulsar Summit SF 2022

StreamNative

Pulsar is a horizontally scalable messaging system, so the traffic in a logical cluster must be balanced across all the available Pulsar brokers as evenly as possible, in order to ensure full utilization of the broker layer. You can use multiple settings and tools to control the traffic distribution which requires a bit of context to understand how the traffic is managed in Pulsar. In this talk, we will walk you through the load balancing capabilities of Apache Pulsar, and highlight some of the control mechanisms available to control the distribution of load across the Pulsar brokers. Finally, we will discuss the various loading shedding strategies that are available. At the end of the talk, you will have a better understanding of how Pulsar's broker level auto-balancing works, and how to properly configure it to meet your workload demands.

Building an Asynchronous Application Framework with Python and Pulsar - Pulsa...

StreamNative

Pulsar's Journey in Yahoo!: On-prem, Cloud and Hybrid - Pulsar Summit SF 2022

StreamNative

In today’s world, we are seeing a big shift toward the Cloud. With this shift comes a big shift in the expectations we have for a messaging system, especially when the messaging system is presented as managed service in a large-scale, multi-tenant environment. For any large-scale enterprise, it’s very important to evaluate messaging system and be confident before expanding complex distributed data systems like Apache Pulsar from on-premise to elastically scalable, fully managed services on cloud services. We must consider aspects such as: migration from and integration with large-scale on-premise clusters, security, cost efficiency, and the cloud friendliness of the architecture, modeling cost and capacity, tenant isolation, deployment robustness, availability, monitoring, etc. Not every messaging system is built to be cloud-native and run as a managed service with cost efficiency. We have been running large-scale Apache Pulsar at Yahoo for the last 8 years on various platforms and hardware configurations while meeting application SLAs and serving more than 1M topics in a cluster. In this talk, we will talk about Pulsar’s journey in Yahoo! from an on-premise platform to a hybrid cloud and on-premise system. We will talk about Pulsar’s architecture and features that make Pulsar a good cloud-native messaging-system choice for any enterprise.

Event-Driven Applications Done Right - Pulsar Summit SF 2022

StreamNative

Pulsar Summit San Francisco is the event dedicated to Apache Pulsar. This one-day, action-packed event will include 5 keynotes, 12 breakout sessions, and 1 amazing happy hour. Speakers are from top companies, including Google, AWS, Databricks, Onehouse, StarTree, Intel, ScyllaDB, and more! It’s the perfect opportunity to network with Pulsar thought leaders in person. Join developers, architects, data engineers, DevOps professionals, and anyone who wants to learn about messaging and event streaming for this one-day, in-person event. Pulsar Summit San Francisco brings the Apache Pulsar Community together to share best practices and discuss the future of streaming technologies.

Pulsar @ Scale. 200M RPM and 1K instances - Pulsar Summit SF 2022

StreamNative

Our services team creates, builds, and maintains the as a service offering for base platform services within our organization. Several thousand applications use these custom services daily generating more than 700 million requests per minute. One of these services was our publish / subscriber offering, BQ with custom SDK and custom metrics based on Apache Pulsar. BQ is the core communication service within our organization, having more 200M RPM. All the core processes of the organization depend on this service for operation: the CDC of any of our RDBMS or NoSQL offering, all the eventing efforts of the organization, async communication between apps, notification systems, etc. The backend of the solution was Apache Pulsar running on EC2 on AWS and on top of that we built several components as wrappers of the actual backend, creating our own SDKs and abstractions and in many ways extending the features provided by Pulsar. We had a multi-cluster setup 100% on AWS, with custom Pulsar Docker images running on large ASG setups, along with our own wrapping and admin APIs and DBs. All of this in turn transformed the solution into a volatile solution.

Beam + Pulsar: Powerful Stream Processing at Scale - Pulsar Summit SF 2022

StreamNative

Welcome and Opening Remarks - Pulsar Summit SF 2022

StreamNative

Log System As Backbone – How We Built the World’s Most Advanced Vector Databa...

StreamNative

Milvus is an open-source vector database that leverages a novel data fabric to build and manage vector similarity search applications. As the world's most popular vector database, it has already been adopted in production by thousands of companies around the world, including Lucidworks, Shutterstock, and Cloudinary. With the launch of Milvus 2.0, the community aims to introduce a cloud-native, highly scalable and extendable vector similarity solution, and the key design concept is log as data. Milvus relies on Pulsar as the log pub/sub system. Pulsar helps Milvus to reduce system complexity by loosely decoupling each micro service, making the system stateless by disaggregating log storage and computation, which also makes the system further extendable. We will introduce the overview design, the implementation details of Milvus and its roadmap in this topic. Takeaways: 1) Get a general idea about what is a vector database and its real-world use cases. 2) Understand the major design principles of Milvus 2.0. 3) Learn how to build a complex system with the help of a modern log system like Pulsar.

MoP(MQTT on Pulsar) - a Powerful Tool for Apache Pulsar in IoT - Pulsar Summi...

StreamNative

MQTT (Message Queuing Telemetry Transport,) is a message protocol based on the pub/sub model with the advantages of compact message structure, low resource consumption, and high efficiency, which is suitable for IoT applications with low bandwidth and unstable network environments. This session will introduce MQTT on Pulsar, which allows developers users of MQTT transport protocol to use Apache Pulsar. I will share the architecture, principles and future planning of MoP, to help you understand Apache Pulsar's capabilities and practices in the IoT industry.

Improvements Made in KoP 2.9.0 - Pulsar Summit Asia 2021

StreamNative

KoP(Kafka on Pulsar) is a protocol handler that provides a convenient method for Kafka clients to connect to Pulsar without any change. Since the first GA version (2.8.0) of KoP (Kafka on Pulsar) was released several months ago, there has been a lot of significant bug fixes and enhancements, such as the support for authorization, and lower version Kafka client. This talk will walk through the basics of KoP, then introduce the improvements in the 2.9.0 version.

More from StreamNative (20)