Log System As Backbone – How We Built the World’s Most Advanced Vector Database on Pulsar - Pulsar Summit Asia 2021

DevOps for Databricks

Applying DevOps to Databricks can be a daunting task. In this talk this will be broken down into bite size chunks. Common DevOps subject areas will be covered, including CI/CD (Continuous Integration/Continuous Deployment), IAC (Infrastructure as Code) and Build Agents. We will explore how to apply DevOps to Databricks (in Azure), primarily using Azure DevOps tooling. As a lot of Spark/Databricks users are Python users, will will focus on the Databricks Rest API (using Python) to perform our tasks.

Azure Synapse Analytics Overview (r2)

Azure Synapse Analytics is Azure SQL Data Warehouse evolved: a limitless analytics service, that brings together enterprise data warehousing and Big Data analytics into a single service. It gives you the freedom to query data on your terms, using either serverless on-demand or provisioned resources, at scale. Azure Synapse brings these two worlds together with a unified experience to ingest, prepare, manage, and serve data for immediate business intelligence and machine learning needs. This is a huge deck with lots of screenshots so you can see exactly how it works.

Keeping Up with the ELK Stack: Elasticsearch, Kibana, Beats, and Logstash

Version 7 of the Elastic Stack adds powerful new features to the popular open source platform for search, logging, and analytics. Come hear directly from Elastic engineers and architecture team members on powerful new additions like GIS functionality and frozen-tier search. Plus, hear about the full range of orchestration options for getting the most out of your deployments, however and wherever you choose to run them. This session is sponsored by Elastic.

Real-time Analytics with Apache Flink and Druid

Jan Graßegger

ML development brings many new complexities beyond the traditional software development lifecycle. Unlike in traditional software development, ML developers want to try multiple algorithms, tools, and parameters to get the best results, and they need to track this information to reproduce work. In addition, developers need to use many distinct systems to productionize models. To address these problems, many companies are building custom “ML platforms” that automate this lifecycle, but even these platforms are limited to a few supported algorithms and to each company’s internal infrastructure. In this session, we introduce MLflow, a new open source project from Databricks that aims to design an open ML platform where organizations can use any ML library and development tool of their choice to reliably build and share ML applications. MLflow introduces simple abstractions to package reproducible projects, track results, and encapsulate models that can be used with many existing tools, accelerating the ML lifecycle for organizations of any size. In this deep-dive session, through a complete ML model life-cycle example, you will walk away with: MLflow concepts and abstractions for models, experiments, and projects How to get started with MLFlow Understand aspects of MLflow APIs Using tracking APIs during model training Using MLflow UI to visually compare and contrast experimental runs with different tuning parameters and evaluate metrics Package, save, and deploy an MLflow model Serve it using MLflow REST API What’s next and how to contribute

Architect’s Open-Source Guide for a Data Mesh Architecture

Data Mesh is an innovative concept addressing many data challenges from an architectural, cultural, and organizational perspective. But is the world ready to implement Data Mesh? In this session, we will review the importance of core Data Mesh principles, what they can offer, and when it is a good idea to try a Data Mesh architecture. We will discuss common challenges with implementation of Data Mesh systems and focus on the role of open-source projects for it. Projects like Apache Spark can play a key part in standardized infrastructure platform implementation of Data Mesh. We will examine the landscape of useful data engineering open-source projects to utilize in several areas of a Data Mesh system in practice, along with an architectural example. We will touch on what work (culture, tools, mindset) needs to be done to ensure Data Mesh is more accessible for engineers in the industry. The audience will leave with a good understanding of the benefits of Data Mesh architecture, common challenges, and the role of Apache Spark and other open-source projects for its implementation in real systems. This session is targeted for architects, decision-makers, data-engineers, and system designers.

The Future of Data Science and Machine Learning at Scale: A Look at MLflow, D...

Many had dubbed 2020 as the decade of data. This is indeed an era of data zeitgeist. From code-centric software development 1.0, we are entering software development 2.0, a data-centric and data-driven approach, where data plays a central theme in our everyday lives. As the volume and variety of data garnered from myriad data sources continue to grow at an astronomical scale and as cloud computing offers cheap computing and data storage resources at scale, the data platforms have to match in their abilities to process, analyze, and visualize at scale and speed and with ease — this involves data paradigm shifts in processing and storing and in providing programming frameworks to developers to access and work with these data platforms. In this talk, we will survey some emerging technologies that address the challenges of data at scale, how these tools help data scientists and machine learning developers with their data tasks, why they scale, and how they facilitate the future data scientists to start quickly. In particular, we will examine in detail two open-source tools MLflow (for machine learning life cycle development) and Delta Lake (for reliable storage for structured and unstructured data). Other emerging tools such as Koalas help data scientists to do exploratory data analysis at scale in a language and framework they are familiar with as well as emerging data + AI trends in 2021. You will understand the challenges of machine learning model development at scale, why you need reliable and scalable storage, and what other open source tools are at your disposal to do data science and machine learning at scale.

Building Data Quality pipelines with Apache Spark and Delta Lake

Technical Leads and Databricks Champions Darren Fuller & Sandy May will give a fast paced view of how they have productionised Data Quality Pipelines across multiple enterprise customers. Their vision to empower business decisions on data remediation actions and self healing of Data Pipelines led them to build a library of Data Quality rule templates and accompanying reporting Data Model and PowerBI reports. With the drive for more and more intelligence driven from the Lake and less from the Warehouse, also known as the Lakehouse pattern, Data Quality at the Lake layer becomes pivotal. Tools like Delta Lake become building blocks for Data Quality with Schema protection and simple column checking, however, for larger customers they often do not go far enough. Notebooks will be shown in quick fire demos how Spark can be leverage at point of Staging or Curation to apply rules over data. Expect to see simple rules such as Net sales = Gross sales + Tax, or values existing with in a list. As well as complex rules such as validation of statistical distributions and complex pattern matching. Ending with a quick view into future work in the realm of Data Compliance for PII data with generations of rules using regex patterns and Machine Learning rules based on transfer learning.

Data & Analytics ReInvent Recap [AWS Basel Meetup - Jan 2023].pdf

Chris Bingham

After recapping the key data & analytics announcements from AWS re:Invent 2022, we look a little deeper at three key new services: • AWS DataZone • AWS Omics • AWS Clean Rooms And follow up with a demo of using AWS IoT ExpressLink hardware in conjunction with AWS IoT Core, Lambda, and Amplify to build a Gatsby web app that interacts with the AWS IoT ExpressLink demo badge via a device shadow.

Learn to Use Databricks for the Full ML Lifecycle

Machine learning development brings many new complexities beyond the traditional software development lifecycle. Unlike traditional software development, ML developers want to try multiple algorithms, tools and parameters to get the best results, and they need to track this information to reproduce work. In addition, developers need to use many distinct systems to productionize models. In this talk, learn how to operationalize ML across the full lifecycle with Databricks Machine Learning.

Introduction to AWS Glue

AWS Glue is a fully managed extract, transform, and load (ETL) service that makes it easy for customers to prepare and load their data for analytics. You can create and run an ETL job with a few clicks in the AWS Management Console. You simply point AWS Glue to your data stored on AWS, and AWS Glue discovers your data and stores the associated metadata (e.g. table definition and schema) in the AWS Glue Data Catalog. Once cataloged, your data is immediately searchable, queryable, and available for ETL. AWS Glue generates the code to execute your data transformations and data loading processes. Level: Intermediate Speakers: Ryan Malecky - Solutions Architect, EdTech, AWS Rajakumar Sampathkumar - Sr. Technical Account Manager, AWS

MLOps Using MLflow

MLflow is an MLOps tool that enables data scientist to quickly productionize their Machine Learning projects. To achieve this, MLFlow has four major components which are Tracking, Projects, Models, and Registry. MLflow lets you train, reuse, and deploy models with any library and package them into reproducible steps. MLflow is designed to work with any machine learning library and require minimal changes to integrate into an existing codebase. In this session, we will cover the common pain points of machine learning developers such as tracking experiments, reproducibility, deployment tool and model versioning. Ready to get your hands dirty by doing quick ML project using mlflow and release to production to understand the ML-Ops lifecycle.

How to Utilize MLflow and Kubernetes to Build an Enterprise ML Platform

In large enterprises, large solutions are sometimes required to tackle even the smallest tasks and ML is no different. At Comcast we are building a comprehensive, configuration based, continuously integrated and deployed platform for data pipeline transformations, model development and deployment. This is accomplished using a range of tools and frameworks such as Databricks, MLflow, Apache Spark and others. With a Databricks environment used by hundreds of researchers and petabytes of data, scale is critical to Comcast, so making it all work together in a frictionless experience is a high priority. The platform consists of a number of components: an abstraction for data pipelines and transformation to allow our data scientists the freedom to combine the most appropriate algorithms from different frameworks , experiment tracking, project and model packaging using MLflow and model serving via the Kubeflow environment on Kubernetes. The architecture, progress and current state of the platform will be discussed as well as the challenges we had to overcome to make this platform work at Comcast scale. As a machine learning practitioner, you will gain knowledge in: an example of data pipeline abstraction; ways to package and track your ML project and experiments at scale; and how Comcast uses Kubeflow on Kubernetes to bring everything together.

Choosing an HDFS data storage format- Avro vs. Parquet and more - StampedeCon...

StampedeCon

At the StampedeCon 2015 Big Data Conference: Picking your distribution and platform is just the first decision of many you need to make in order to create a successful data ecosystem. In addition to things like replication factor and node configuration, the choice of file format can have a profound impact on cluster performance. Each of the data formats have different strengths and weaknesses, depending on how you want to store and retrieve your data. For instance, we have observed performance differences on the order of 25x between Parquet and Plain Text files for certain workloads. However, it isn’t the case that one is always better than the others.

Azure Synapse Analytics Overview (r1)

ELK Stack

Eberhard Wolff

Databricks Platform.pptx

Alex Ivy

A Practical Enterprise Feature Store on Delta Lake

The feature store is a data architecture concept used to accelerate data science experimentation and harden production ML deployments. Nate Buesgens and Bryan Christian describe a practical approach to building a feature store on Delta Lake at a large financial organization. This implementation has reduced feature engineering “wrangling” time by 75% and has increased the rate of production model delivery by 15x. The approach described focuses on practicality. It is informed by innovative approaches such as Feast, but our primary goal is evolutionary extensions of existing patterns that can be applied to any Delta Lake architecture. Key Takeaways: – Understand the key use cases that motivate the feature store from both a data science and engineering perspective. – Consider edge cases where there may be opportunities for simplification such as “online” predictions. – Review a typical logical data model for a feature store and how that can be applied to your business domain. – Consider options for physical storage of the feature store in the Delta Lake. – Understand common access patterns including metadata-based feature discovery.

Data Lakehouse, Data Mesh, and Data Fabric (r2)

So many buzzwords of late: Data Lakehouse, Data Mesh, and Data Fabric. What do all these terms mean and how do they compare to a modern data warehouse? In this session I’ll cover all of them in detail and compare the pros and cons of each. They all may sound great in theory, but I'll dig into the concerns you need to be aware of before taking the plunge. I’ll also include use cases so you can see what approach will work best for your big data needs. And I'll discuss Microsoft version of the data mesh.

Introduction to MLflow

Vector Search for Data Scientists.pdf

ConnorShorten2

Analyzing Petabyte Scale Financial Data with Apache Pinot and Apache Kafka | ...

HostedbyConfluent

At Stripe, we operate a general ledger modeled as double-entry bookkeeping for all financial transactions. Warehousing such data is challenging due to its high volume and high cardinality of unique accounts. aFurthermore, it is financially critical to get up-to-date, accurate analytics over all records. Due to the changing nature of real time transactions, it is impossible to pre-compute the analytics as a fixed time series. We have overcome the challenge by creating a real time key-value store inside Pinot that can sustain half million QPS with all the financial transactions. We will talk about the details of our solution and the interesting technical challenges faced.

OpenStack Swift In the Enterprise

Hostway|HOSTING

Big Data Architecture Workshop - Vahid Amiri

datastack

What's hot

What’s New with Databricks Machine Learning

Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A...

Cathrine Wilhelmsen

MLflow: Infrastructure for a Complete Machine Learning Life Cycle with Mani ...

Architect’s Open-Source Guide for a Data Mesh Architecture

The Future of Data Science and Machine Learning at Scale: A Look at MLflow, D...

Building Data Quality pipelines with Apache Spark and Delta Lake

Data & Analytics ReInvent Recap [AWS Basel Meetup - Jan 2023].pdf

Chris Bingham

Learn to Use Databricks for the Full ML Lifecycle

Introduction to AWS Glue

MLOps Using MLflow

How to Utilize MLflow and Kubernetes to Build an Enterprise ML Platform

Choosing an HDFS data storage format- Avro vs. Parquet and more - StampedeCon...

StampedeCon

Azure Synapse Analytics Overview (r1)

ELK Stack

Eberhard Wolff

Databricks Platform.pptx

Alex Ivy

A Practical Enterprise Feature Store on Delta Lake

Data Lakehouse, Data Mesh, and Data Fabric (r2)

Introduction to MLflow

Vector Search for Data Scientists.pdf

ConnorShorten2

Analyzing Petabyte Scale Financial Data with Apache Pinot and Apache Kafka | ...

HostedbyConfluent

What's hot (20)

What’s New with Databricks Machine Learning

Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A...

MLflow: Infrastructure for a Complete Machine Learning Life Cycle with Mani ...

Architect’s Open-Source Guide for a Data Mesh Architecture

The Future of Data Science and Machine Learning at Scale: A Look at MLflow, D...

Building Data Quality pipelines with Apache Spark and Delta Lake

Data & Analytics ReInvent Recap [AWS Basel Meetup - Jan 2023].pdf

Learn to Use Databricks for the Full ML Lifecycle

Introduction to AWS Glue

MLOps Using MLflow

How to Utilize MLflow and Kubernetes to Build an Enterprise ML Platform

Choosing an HDFS data storage format- Avro vs. Parquet and more - StampedeCon...

Azure Synapse Analytics Overview (r1)

ELK Stack

Databricks Platform.pptx

A Practical Enterprise Feature Store on Delta Lake

Data Lakehouse, Data Mesh, and Data Fabric (r2)

Introduction to MLflow

Vector Search for Data Scientists.pdf

Analyzing Petabyte Scale Financial Data with Apache Pinot and Apache Kafka | ...

Similar to Log System As Backbone – How We Built the World’s Most Advanced Vector Database on Pulsar - Pulsar Summit Asia 2021

OpenStack Swift In the Enterprise

Hostway|HOSTING

Big Data Architecture Workshop - Vahid Amiri

datastack

Nuxeo Platform LTS 2015 Highlights

Nuxeo

Get a glimpse of the main features supported in Nuxeo Platform LTS 2015. With this LTS version of the Nuxeo Platform, we’re changing how we assign product version names and numbers. The name for each LTS version is now based on the release year. Nuxeo Platform LTS 2015 is the result of the four Fast Track releases throughout the past year. Highlights of Nuxeo Platform LTS 2015 include: - Nuxeo Live Connect: Native Integration with Google Drive & Dropbox - Content Analytics & Data Visualisation - Elasticsearch: API Passthrough, Hints for NXQL, Security - Massive Scalability with MongoDB Integration - New Document Viewer - Automation Scripting - Nuxeo Drive 2 - Automated Media Conversions

SpringPeople - Introduction to Cloud Computing

SpringPeople

Cloud computing is no longer a fad that is going around. It is for real and is perhaps the most talked about subject. Various players in the cloud eco-system have provided a definition that is closely aligned to their sweet spot –let it be infrastructure, platforms or applications. This presentation will provide an exposure of a variety of cloud computing techniques, architecture, technology options to the participants and in general will familiarize cloud fundamentals in a holistic manner spanning all dimensions such as cost, operations, technology etc

Nuxeo Fact Sheet

Nuxeo

Using Data Lakes

Database Technologies

Michel de Goede

(STG308) How EA, State Of Texas & H3 Biomedicine Protect Data

In this session, learn how enterprise customers use AWS storage services to address different storage requirements. Learn how Electronic Arts and H3 Biomedicine manage their data flow from on-premises systems to the cloud, giving them a centralized build system and storage flexibility by leveraging enterprise storage gateways. The State of Texas uses AWS and partner solutions to modernize and secure their office file services, and backup and recovery systems, achieving dramatic savings and productivity gains without compromising IT efficiency.

Semantic Technologies for Enterprise Cloud ManagementPeter Haase

Alluxio Data Orchestration Platform for the Cloud

Shubham Tagra

Using Data Lakes: Data Analytics Week SF

Data Analytics Week at the San Francisco Loft Using Data Lakes A data lake can be used as a source for both structured and unstructured data - but how? We'll look at using open standards including Spark and Presto with Amazon EMR, Amazon Redshift Spectrum and Amazon Athena to process and understand data. Speakers: John Mallory - Principal Business Development Manager Storage (Object), AWS Hemant Borole - Sr. Big Data Consultant, AWS

How companies-use-no sql-and-couchbase-10152013Dipti Borkar

Accelerating analytics in the cloud with the Starburst Presto + Alluxio stack

Alluxio, Inc.

Alluxio Tech Talk January 21, 2020 Speakers: Matt Fuller, Starburst Dipti Borkar, Alluxio With the advent of the public clouds and data increasingly siloed across many locations -- on premises and in the public cloud -- enterprises are looking for more flexibility and higher performance approaches to analyze their structured data. Join us for this tech talk where we’ll introduce the Starburst Presto, Alluxio, and cloud object store stack for building a highly-concurrent and low-latency analytics platform. This stack provides a strong solution to run fast SQL across multiple storage systems including HDFS, S3, and others in public cloud, hybrid cloud, and multi-cloud environments. You’ll learn more about: - The architecture of Presto, an open source distributed SQL engine - How the Presto + Alluxio stack queries data from cloud object storage like S3 for faster and more cost-effective analytics - Achieving data locality and cross-job caching with Alluxio regardless of where data is persisted

Modern data warehouse

Rakesh Jayaram

Estimating the Total Costs of Your Cloud Analytics Platform

DATAVERSITY

Organizations today need a broad set of enterprise data cloud services with key data functionality to modernize applications and utilize machine learning. They need a platform designed to address multi-faceted needs by offering multi-function Data Management and analytics to solve the enterprise’s most pressing data and analytic challenges in a streamlined fashion. They need a worry-free experience with the architecture and its components.

Data Lakehouse, Data Mesh, and Data Fabric (r1)

Introducing DocumentDB

Data Engineer, Patterns & Architecture The future: Deep-dive into Microservic...

Igor De Souza

With Industry 4.0, several technologies are used to have data analysis in real-time, maintaining, organizing, and building this on the other hand is a complex and complicated job. Over the past 30 years, we saw several ideas to centralize the database in a single place as the united and true source of data has been implemented in companies, such as Data wareHouse, NoSQL, Data Lake, Lambda & Kappa Architecture. On the other hand, Software Engineering has been applying ideas to separate applications to facilitate and improve application performance, such as microservices. The idea is to use the MicroService patterns on the date and divide the model into several smaller ones. And a good way to split it up is to use the model using the DDD principles. And that's how I try to explain and define DataMesh & Data Fabric.

Achieving Separation of Compute and Storage in a Cloud World

Alluxio, Inc.

Alluxio Tech Talk Feb 12, 2019 Speaker: Dipti Borkar, Alluxio The rise of compute intensive workloads and the adoption of the cloud has driven organizations to adopt a decoupled architecture for modern workloads – one in which compute scales independently from storage. While this enables scaling elasticity, it introduces new problems – how do you co-locate data with compute, how do you unify data across multiple remote clouds, how do you keep storage and I/O service costs down and many more. Enter Alluxio, a virtual unified file system, which sits between compute and storage that allows you to realize the benefits of a hybrid cloud architecture with the same performance and lower costs. In this webinar, we will discuss: - Why leading enterprises are adopting hybrid cloud architectures with compute and storage disaggregated - The new challenges that this new paradigm introduces - An introduction to Alluxio and the unified data solution it provides for hybrid environments

How companies use NoSQL and Couchbase - NoSQL Now 2013

Dipti Borkar

Similar to Log System As Backbone – How We Built the World’s Most Advanced Vector Database on Pulsar - Pulsar Summit Asia 2021 (20)

OpenStack Swift In the Enterprise

Big Data Architecture Workshop - Vahid Amiri

Nuxeo Platform LTS 2015 Highlights

SpringPeople - Introduction to Cloud Computing

Nuxeo Fact Sheet

Using Data Lakes

Database Technologies

(STG308) How EA, State Of Texas & H3 Biomedicine Protect Data

Semantic Technologies for Enterprise Cloud Management

Alluxio Data Orchestration Platform for the Cloud

Using Data Lakes: Data Analytics Week SF

How companies-use-no sql-and-couchbase-10152013

Accelerating analytics in the cloud with the Starburst Presto + Alluxio stack

Modern data warehouse

Estimating the Total Costs of Your Cloud Analytics Platform

Data Lakehouse, Data Mesh, and Data Fabric (r1)

Introducing DocumentDB

Data Engineer, Patterns & Architecture The future: Deep-dive into Microservic...

Achieving Separation of Compute and Storage in a Cloud World

How companies use NoSQL and Couchbase - NoSQL Now 2013

More from StreamNative

Is Using KoP (Kafka-on-Pulsar) a Good Idea? - Pulsar Summit SF 2022

So, you are a responsible software engineer building microservices for Apache Kafka, and life is good. Eventually, you hear the community talking about the outstanding experience they are having with Apache Pulsar features. They talk about infinite event stream retention, a rebalance-free architecture, native support for event processing, and multi-tenancy. Exciting, right? Most people would want to migrate their code to Pulsar. Especially when you know that Pulsar also supports Kafka clients natively via the protocol handler known as KoP — which enables the Kafka client APIs on Pulsar. But, as said before, you are responsible; and you don't believe in fairy tales, just like you don't believe that migrations like this happen effortlessly. This session will discuss the architecture behind protocol handlers, what it means having one enabled on Pulsar, and how the KoP works. It will detail the effort required to migrate a microservice written for Kafka to Pulsar, and whether the code need to change for this.

Building an Asynchronous Application Framework with Python and Pulsar - Pulsa...

This talk describes Klaviyo’s internal messaging system, an asynchronous application framework built around Pulsar that provides a set of high-quality tools for building business-critical asynchronous data flows in unreliable environments. This framework includes: a pulsar ORM and schema migrator for topic configuration; a retry/replay system; a versioned schema registry; a consumer framework oriented around preventing message loss and in hostile environments while maximizing observability; an experimental “online schema change” for topics; and more. Development of this system was informed by lessons learned during heavy use of datastores like RabbitMQ and Kafka, and frameworks like Celery, Spark, and Flink. In addition to the capabilities of this system, this talk will also cover (sometimes painful) lessons learned about the process of converting a heterogenous async-computing environment onto Pulsar and a unified model.

Blue-green deploys with Pulsar & Envoy in an event-driven microservice ecosys...

In this talk, learn how Toast leverages our Envoy control-plane to manage blue-green deploys of Pulsar consumers, and how this has helped drive adoption across the engineering organization. Dive into the history of Pulsar at Toast, starting from its introduction in 2019 to provide event-driven architecture across a rapidly scaling restaurant software platform. We will detail some of the hurdles that we encountered gaining buy-in across a diverse set of teams, and dive deep into how we enforce best practices and integrate with our service control plane.

Distributed Database Design Decisions to Support High Performance Event Strea...

Event streaming architectures launched a reexamination of applications and systems architectures across the board. We live in a world where answers are needed now in a constant real-time flow. Yet beyond the event streaming system itself, what are the corequisites to ensure our large scale distributed database systems can keep pace with this always-on, always-current real time flow of data? What are the requirements and expectations for this next tech cycle?

Simplify Pulsar Functions Development with SQL - Pulsar Summit SF 2022

Pulsar Functions is a succinct framework provided by Apache Pulsar to conduct real-time data processing. Its use cases include ETL pipeline, event-driven applications, and simple data analytics. While Pulsar Functions already provides an extremely simple programming interface, we want to further lower the barrier for users to access real-time data. Since SQL is one of the universal languages in the technology world and well accepted by the vast majority of data engineers, we decided to add a SQL expressing layer on top of Pulsar Functions runtime. In this talk, we will discuss the architecture and implementation of this new service. We will see how SQL syntax, Pulsar Functions, and Function Mesh can work together to deliver a unique user development experience for real-time data jobs in the cloud environment. We will also walk through use cases like filtering, routing, and projecting messages as well as integrating with the Pulsar IO Connectors framework.

Towards a ZooKeeper-less Pulsar, etcd, etcd, etcd. - Pulsar Summit SF 2022

Starting with version 2.10, the Apache ZooKeeper dependency has been eliminated and replaced with a pluggable framework that enables you to reduce the infrastructure footprint of Apache Pulsar by leveraging alternative metadata and coordination systems based on your deployment environment. In this talk, walk through the steps required to utilize the existing etcd service running inside Kubernetes to act as Pulsar's metadata store, thereby eliminating the need to run ZooKeeper entirely, leaving you with a Zookeeper-less Pulsar.

Validating Apache Pulsar’s Behavior under Failure Conditions - Pulsar Summit ...

Apache Pulsar is a highly available, distributed messaging system that provides guarantees of no message loss and strong message ordering with predictable read and write latency. In this talk, learn how this can be validated for Apache Pulsar Kubernetes deployments. Various failures are injected using Chaos Mesh to simulate network and other infrastructure failure conditions. There are many questions that are asked about failure scenarios, but it could be hard to find answers to these important questions. When a failure happens, how long does it take to recover? Does it cause unavailability? How does it impact throughput and latency? Are the guarantees of no message loss and strong message ordering kept, even when components fail? If a complete availability zone fails, is the system configured correctly to handle AZ failures? This talk will help you find answers to these questions and apply the tooling and practices to your own testing and validation.

Cross the Streams! Creating Streaming Data Pipelines with Apache Flink + Apac...

Despite what the Ghostbusters said, we’re going to go ahead and cross (or, join) the streams. This session covers getting started with streaming data pipelines, maximizing Pulsar’s messaging system alongside one of the most flexible streaming frameworks available, Apache Flink. Specifically, we’ll demonstrate the use of Flink SQL, which provides various abstractions and allows your pipeline to be language-agnostic. So, if you want to leverage the power of a high-speed, highly customizable stream processing engine without the usual overhead and learning curves of the technologies involved (and their interconnected relationships), then this talk is for you. Watch the step-by-step demo to build a unified batch and streaming pipeline from scratch with Pulsar, via the Flink SQL client. This means you don’t need to be familiar with Flink, (or even a specific programming language). The examples provided are built for highly complex systems, but the talk itself will be accessible to any experience level.

Message Redelivery: An Unexpected Journey - Pulsar Summit SF 2022

Apache Pulsar depends upon message acknowledgments to provide at-least-once or exactly-once processing guarantees. With these guarantees, any transmission between the broker and its producers and consumers requires an acknowledgment. But what happens if an acknowledgment is not received? Resending the message introduces the potential of duplicate processing and increases the likelihood of out or order processing. Therefore, it is critical to understand the Pulsar message redelivery semantics in order to prevent either of these conditions. In this talk, we will walk you through the redelivery semantics of Apache Pulsar, and highlight some of the control mechanisms available to application developers to control this behavior. Finally, we will present best practices for configuring message redelivery to suit various use cases.

Unlocking the Power of Lakehouse Architectures with Apache Pulsar and Apache ...

Lakehouses are quickly growing in popularity as a new approach to Data Platform Architecture bringing some of the long-established benefits from OLTP world to OLAP, including transactions, record-level updates/deletes, and changes streaming. In this talk, we will discuss Apache Hudi and how it unlocks possibilities of building your own fully open-source Lakehouse featuring a rich set of integrations with existing technologies, including Apache Pulsar. In this session, we will present: - What Lakehouses are, and why they are needed. - What Apache Hudi is and how it works. - Provide a use-case and demo that applies Apache Hudi’s DeltaStreamer tool to ingest data from Apache Pulsar.

Understanding Broker Load Balancing - Pulsar Summit SF 2022

Pulsar is a horizontally scalable messaging system, so the traffic in a logical cluster must be balanced across all the available Pulsar brokers as evenly as possible, in order to ensure full utilization of the broker layer. You can use multiple settings and tools to control the traffic distribution which requires a bit of context to understand how the traffic is managed in Pulsar. In this talk, we will walk you through the load balancing capabilities of Apache Pulsar, and highlight some of the control mechanisms available to control the distribution of load across the Pulsar brokers. Finally, we will discuss the various loading shedding strategies that are available. At the end of the talk, you will have a better understanding of how Pulsar's broker level auto-balancing works, and how to properly configure it to meet your workload demands.

Building an Asynchronous Application Framework with Python and Pulsar - Pulsa...

Pulsar's Journey in Yahoo!: On-prem, Cloud and Hybrid - Pulsar Summit SF 2022

In today’s world, we are seeing a big shift toward the Cloud. With this shift comes a big shift in the expectations we have for a messaging system, especially when the messaging system is presented as managed service in a large-scale, multi-tenant environment. For any large-scale enterprise, it’s very important to evaluate messaging system and be confident before expanding complex distributed data systems like Apache Pulsar from on-premise to elastically scalable, fully managed services on cloud services. We must consider aspects such as: migration from and integration with large-scale on-premise clusters, security, cost efficiency, and the cloud friendliness of the architecture, modeling cost and capacity, tenant isolation, deployment robustness, availability, monitoring, etc. Not every messaging system is built to be cloud-native and run as a managed service with cost efficiency. We have been running large-scale Apache Pulsar at Yahoo for the last 8 years on various platforms and hardware configurations while meeting application SLAs and serving more than 1M topics in a cluster. In this talk, we will talk about Pulsar’s journey in Yahoo! from an on-premise platform to a hybrid cloud and on-premise system. We will talk about Pulsar’s architecture and features that make Pulsar a good cloud-native messaging-system choice for any enterprise.

Event-Driven Applications Done Right - Pulsar Summit SF 2022

Pulsar Summit San Francisco is the event dedicated to Apache Pulsar. This one-day, action-packed event will include 5 keynotes, 12 breakout sessions, and 1 amazing happy hour. Speakers are from top companies, including Google, AWS, Databricks, Onehouse, StarTree, Intel, ScyllaDB, and more! It’s the perfect opportunity to network with Pulsar thought leaders in person. Join developers, architects, data engineers, DevOps professionals, and anyone who wants to learn about messaging and event streaming for this one-day, in-person event. Pulsar Summit San Francisco brings the Apache Pulsar Community together to share best practices and discuss the future of streaming technologies.

Pulsar @ Scale. 200M RPM and 1K instances - Pulsar Summit SF 2022

Our services team creates, builds, and maintains the as a service offering for base platform services within our organization. Several thousand applications use these custom services daily generating more than 700 million requests per minute. One of these services was our publish / subscriber offering, BQ with custom SDK and custom metrics based on Apache Pulsar. BQ is the core communication service within our organization, having more 200M RPM. All the core processes of the organization depend on this service for operation: the CDC of any of our RDBMS or NoSQL offering, all the eventing efforts of the organization, async communication between apps, notification systems, etc. The backend of the solution was Apache Pulsar running on EC2 on AWS and on top of that we built several components as wrappers of the actual backend, creating our own SDKs and abstractions and in many ways extending the features provided by Pulsar. We had a multi-cluster setup 100% on AWS, with custom Pulsar Docker images running on large ASG setups, along with our own wrapping and admin APIs and DBs. All of this in turn transformed the solution into a volatile solution.

Data Democracy: Journey to User-Facing Analytics - Pulsar Summit SF 2022

There is an increasing need to unleash analytical capabilities directly to the end-users to democratize decision-making. User-Facing Analytics is a new frontier that will shape the products of tomorrow and push the limits of existing technology. It demands a solution that will scale to millions of users to provide fast, real-time insights. In this session, Xiang will talk about his journey to build Apache Pinot to tackle the analytics problem space with the architectural changes and technology inventions made over the past decade. He will also talk about how other big data companies such as LinkedIn, Uber, and Stripe power their user-facing analytical applications.

Beam + Pulsar: Powerful Stream Processing at Scale - Pulsar Summit SF 2022

Welcome and Opening Remarks - Pulsar Summit SF 2022

MoP(MQTT on Pulsar) - a Powerful Tool for Apache Pulsar in IoT - Pulsar Summi...

MQTT (Message Queuing Telemetry Transport,) is a message protocol based on the pub/sub model with the advantages of compact message structure, low resource consumption, and high efficiency, which is suitable for IoT applications with low bandwidth and unstable network environments. This session will introduce MQTT on Pulsar, which allows developers users of MQTT transport protocol to use Apache Pulsar. I will share the architecture, principles and future planning of MoP, to help you understand Apache Pulsar's capabilities and practices in the IoT industry.

Improvements Made in KoP 2.9.0 - Pulsar Summit Asia 2021