Real-Time Analytics: Going Beyond Stream Processing With Apache Pinot

•

0 likes•42 views

Apache Pinot is a real-time analytics platform designed for user-facing applications. It provides excellent query performance on large, mutable datasets by using columnar storage and optimized indexing. Pinot handles real-time data ingestion from streaming and batch sources while supporting updates, deletes, and semi-structured data. It also offers a hybrid architecture that separates compute from storage for cost efficiency and scales from gigabytes to petabytes of data. Pinot aims to provide a full SQL interface and one platform for building various real-time analytics use cases with low latency and strong data consistency.

Real-time Analytics
Rong Rong
Software Engineer
- Going Beyond Stream Processing with Apache Pinot

Total users 700 Million
QPS 10000+
Latency SLA < 100 ms p99th
Freshness Seconds
Actionable Insights

Missed
orders
Inaccurate
orders
Downtime
Top selling items Menu item
Feedback
Actionable Insights
Restaurant Dashboard

● Real-Time analytics landscape is rapidly changing.
● OnLine Analytical Processing (OLAP) system are evolving with these trends.
Internal-Facing Analytics
Structured Data
Approximate Data / Query
Consistency
Slice-and-Dice Queries
User-Facing Analytics
Semi-Structured Data
Strong Data / Query
Consistency
Full-SQL semantics
GB-to-TB of Data TB-to-PB of Data
Evolution of Real-Time Analytics

Better Ingestion Better Queries Better Scalability
Requirements are changing!

Table
Pinot Data Segments
Col1
..
ca
jp
us
ca
…
ColN
..
…
…
…
…
…
. . . +
Indexes
Metadata
Columnar Store

browser
chrome
firefox
ie
firefox
ie
…
...
...
...
...
...
...
...
country browser ...
us chrome ...
ca firefox ...
jp ie ...
us firefox ...
ca ie ...
… … ...
Raw Data Column Based
country
us
ca
jp
us
ca
…
1
select count(*) from X
where country = us
Country
Browser
Star-Tree Index
US CA
IE C IE C
Optimized Index Algorithm

Filtering Optimizations
Inverted Index
Sorted Index
Range Index
JSON Index
Geo Index
Text Index
Aggregation Optimizations
Theta Sketches
StarTree Index
HyperLogLog
Optimized Index Algorithm

OLAP
Semi-Structured
Data
Pre-Process
Structured Data
OLAP
Semi-Structured
Data
Support Semi-Structured Data

OLAP
Real-Time
Insert
OLAP
Best Effort Queries
(Double Counting)
Accurate Queries
Real-Time
Insert + Updates +
Deletes
Support Strong Data Consistency

Support Strong Data Consistency
UPSERT data ingestions
Real-Time
Stream
K=1, V=5
K=1 ,V=1
S3 S4
S1 S2
Key Index Insert: (K=1, S=S1, Row=1)
Key Index Update: (K=1, S=S1, Row=1)
K=3, V=4
K=2, V=3
K=2, V=2 Key Index Insert: (K=2, S=S2, Row=2)
Key Index Update: (K=2, S=S3, Row=2)
Key Index Insert: (K=3, S=S4, Row=1)

OLAP
OLAP Queries
(Aggregations, Order By, Group
By, UDFs, …)
Full SQL Semantics !
(OLAP Queries + Joins + Nested
Queries + Window functions + …)
OLAP
Towards full-SQL semantics

Local Storage
Compute
OLAP Data Nodes
Cloud Storage
$$$$
$$
TBs - PBs
of Data
Local Storage
Compute
OLAP Data Nodes
$
GBs - TBs
of Data
Compute & Storage Decoupling

Pinot Servers
Pinot Brokers
Pinot Segments
Compute & Storage Decoupling

Pinot Servers
Cloud Storage
Pinot Brokers
Pinot Segments
Compute & Storage Decoupling

● Real-time analytics has evolved to power more user-facing,
actionable applications.
● Apache Pinot is our solution for real-time analytics.
Excellent Query Performance
Handles real-time data mutation
Ingest from Streaming, Batch, SQL sources
Semi-Structured Data support
Hybrid architecture for cost efﬁciency
One platform for building all kinds of use cases
Takeaway

My talk from GOTO Aarhus, 30th September 2014. Cogenta is a retail intelligence company which tracks ecommerce web sites around the world to provide competitive monitoring and analysis services to retailers. Using its proprietary crawler technology, Lucene and SQL Server, a stream of 20 million raw product data entries is captured and processed each day. This case study looks at how Cogenta uses Elasticsearch to break the shackles imposed by the RDBMS (and a limited budget) to make the data available in real time to its customers. Cogenta uses SQL as its canonical store & for complex reporting, and Elasticsearch for real-time processing & to drive its SaaS web applications. Elasticsearch is easy to use, delivers the powerful features of Lucene and enables the data & platform cost to scale linearly. But… synchronising your existing data in two places presents some interesting challenges such as aggregation and concurrency control. This talk will take a detailed look at how Cogenta how overcame those challenges, with a perpetually changing and asynchronously updated dataset. http://gotocon.com/aarhus-2014/presentation/Cogenta%20-%20Making%20Enterprise%20Data%20Available%20in%20Real%20Time%20with%20Elasticsearch

An Architect's guide to real time big data systems

Raja SP

Simplify and Scale Data Engineering Pipelines with Delta Lake

Databricks

We’re always told to ‘Go for the Gold!,’ but how do we get there? This talk will walk you through the process of moving your data to the finish fine to get that gold metal! A common data engineering pipeline architecture uses tables that correspond to different quality levels, progressively adding structure to the data: data ingestion (‘Bronze’ tables), transformation/feature engineering (‘Silver’ tables), and machine learning training or prediction (‘Gold’ tables). Combined, we refer to these tables as a ‘multi-hop’ architecture. It allows data engineers to build a pipeline that begins with raw data as a ‘single source of truth’ from which everything flows. In this session, we will show how to build a scalable data engineering data pipeline using Delta Lake, so you can be the champion in your organization.

Deploying your Data Warehouse on AWS

Amazon Web Services

February 2016 Webinar Series - Introduction to DynamoDB

Amazon Web Services

Amazon DynamoDB is a fast and flexible NoSQL database service for applications that need consistent, single-digit millisecond latency at any scale. It is a fully managed cloud database and supports both document and key-value store models. Its flexible data model and reliable performance make it a great fit for mobile, web, gaming, ad tech, IoT, and many other applications. Learning Objectives: Understand the differences between relational and non-relational databases Learn about common use cases for DynamoDB across gaming, ad tech, IoT, and more See how DynamoDB helps customers handle spikes in traffic and save development time for new feature launches Who Should Attend: Developers, IT Decision Makers, and Executives interested in learning more about Amazon Web Services’ serverless NoSQL service to scale mobile, web, IoT, ad tech, and gaming apps

Delta Lake is an open-source storage layer that brings reliability to data lakes. Delta Lake offers ACID transactions, scalable metadata handling, and unifies the streaming and batch data processing. It runs on top of your existing data lake and is fully compatible with Apache Spark APIs. In this talk, we will cover: * What data quality problems Delta helps address * How to convert your existing application to Delta Lake * How the Delta Lake transaction protocol works internally * The Delta Lake roadmap for the next few releases * How to get involved!

SnappyData, the Spark Database. A unified cluster for streaming, transactions...

SnappyData

Apache Spark 2.0 offers many enhancements that make continuous analytics quite simple. In this talk, we will discuss many other things that you can do with your Apache Spark cluster. We explain how a deep integration of Apache Spark 2.0 and in-memory databases can bring you the best of both worlds! In particular, we discuss how to manage mutable data in Apache Spark, run consistent transactions at the same speed as state-the-art in-memory grids, build and use indexes for point lookups, and run 100x more analytics queries at in-memory speeds. No need to bridge multiple products or manage, tune multiple clusters. We explain how one can take regulation Apache Spark SQL OLAP workloads and speed them up by up to 20x using optimizations in SnappyData. We then walk through several use-case examples, including IoT scenarios, where one has to ingest streams from many sources, cleanse it, manage the deluge by pre-aggregating and tracking metrics per minute, store all recent data in a in-memory store along with history in a data lake and permit interactive analytic queries at this constantly growing data. Rather than stitching together multiple clusters as proposed in Lambda, we walk through a design where everything is achieved in a single, horizontally scalable Apache Spark 2.0 cluster. A design that is simpler, a lot more efficient, and let’s you do everything from Machine Learning and Data Science to Transactions and Visual Analytics all in one single cluster.

SnappyData at Spark Summit 2017

Jags Ramnarayan

Apache Kylin @ Big Data Europe 2015

Seshu Adunuthula

Modern Data Architectures for Business Insights at Scale

Amazon Web Services

Learnings Using Spark Streaming and DataFrames for Walmart Search: Spark Summ...

Spark Summit

In this presentation, we are going to talk about the state of the art infrastructure we have established at Walmart Labs for the Search product using Spark Streaming and DataFrames. First, we have been able to successfully use multiple micro batch spark streaming pipelines to update and process information like product availability, pick up today etc. along with updating our product catalog information in our search index to up to 10,000 kafka events per sec in near real-time. Earlier, all the product catalog changes in the index had a 24 hour delay, using Spark Streaming we have made it possible to see these changes in near real-time. This addition has provided a great boost to the business by giving the end-costumers instant access to features likes availability of a product, store pick up, etc. Second, we have built a scalable anomaly detection framework purely using Spark Data Frames that is being used by our data pipelines to detect abnormality in search data. Anomaly detection is an important problem not only in the search domain but also many domains such as performance monitoring, fraud detection, etc. During this, we realized that not only are Spark DataFrames able to process information faster but also are more flexible to work with. One could write hive like queries, pig like code, UDFs, UDAFs, python like code etc. all at the same place very easily and can build DataFrame template which can be used and reused by multiple teams effectively. We believe that if implemented correctly Spark Data Frames can potentially replace hive/pig in big data space and have the potential of becoming unified data language. We conclude that Spark Streaming and Data Frames are the key to processing extremely large streams of data in real-time with ease of use.

Microsoft Power BI and Cortana Analytics user group meetings with Alteryx

Håkan Söderbom

MLOps with a Feature Store: Filling the Gap in ML Infrastructure

Data Science Milan

A Feature Store enables machine learning (ML) features to be registered, discovered, and used as part of ML pipelines, thus making it easier to transform and validate the training data that is fed into machine learning systems. Feature stores can also enable consistent engineering of features between training and inference, but to do so, they need a common data processing platform. The first Feature Stores, developed at hyperscale AI companies such as Uber, Airbnb, and Facebook, enabled feature engineering using domain specific languages, providing abstractions tailored to the companies’ feature engineering domains. However, a general purpose Feature Store needs a general purpose feature engineering, feature selection, and feature transformation platform. In this talk, we describe how we built a general purpose, open-source Feature Store for ML around dataframes and Apache Spark. We will demonstrate how data engineers can transform and engineers features from backend databases and data lakes, while data scientists can use PySpark to select and transform features into train/test data in a file format of choice (.tfrecords, .npy, .petastorm, etc) on a file system of choice (S3, HDFS). Finally, we will show how the Feature Store enables end-to-end ML pipelines to be factored into feature engineering and data science stages that each can run at different cadences. Bio: Fabio Buso is the head of engineering at Logical Clocks AB, where he leads the Feature Store development. Fabio holds a master's degree in cloud computing and services with a focus on data intensive applications, awarded by a joint program between KTH Stockholm and TU Berlin. Topics: feature store, MLOps.

Blueflood: Open Source Metrics Processing at CassandraEU 2013

gdusbabek

Getting Strated with Amazon Dynamo DB (Jim Scharf) - AWS DB Day

Amazon Web Services Korea

Open Source Reliability for Data Lake with Apache Spark by Michael Armbrust

Data Con LA

Abstract: Delta Lake is an open source storage layer that brings reliability to data lakes. Delta Lake offers ACID transactions, scalable metadata handling, and unifies streaming and batch data processing. It runs on top of your existing data lake and is fully compatible with Apache Spark APIs. In this talk, we will cover .All technical aspects of Delta Features .What’s coming .How to get started using it .How to contribute Bio: Michael Armbrust is committer and PMC member of Apache Spark and the original creator of Spark SQL. He currently leads the team at Databricks that designed and built Structured Streaming and Databricks Delta. He received his PhD from UC Berkeley in 2013, and was advised by Michael Franklin, David Patterson, and Armando Fox. His thesis focused on building systems that allow developers to rapidly build scalable interactive applications, and specifically defined the notion of scale independence. His interests broadly include distributed systems, large-scale structured storage and query optimization.

near real time search in e-commerce

Umesh Prasad

6° Sessione - Ambiti applicativi nella ricerca di tecnologie statistiche avan...

Jürgen Ambrosi

In questa sessione vedremo, con il solito approccio pratico di demo hands on, come utilizzare il linguaggio R per effettuare analisi a valore aggiunto, Toccheremo con mano le performance di parallelizzazione degli algoritmi, aspetto fondamentale per aiutare il ricercatore nel raggiungimento dei suoi obbiettivi. In questa sessione avremo la partecipazione di Lorenzo Casucci, Data Platform Solution Architect di Microsoft.

Modern Data Architectures for Real Time Analytics & Engagement

Amazon Web Services

How Totango uses Apache Spark

Oren Raboy

Unbreakable Sharepoint 2016 With SQL Server 2016 availability groups

Isabelle Van Campenhoudt

SQL Server is really the brain of SharePoint; in this session, Serge Luca (SharePoint MVP) and Isabelle Van Campenhoudt (SQL Server MVP) will give you an overview of what any SharePoint consultant and DBA need to know regarding business continuity in SharePoint 2013 & 2016. Of course SQL Server plays a major role in this story; the sessions will be animated with real & live demos. Topics covered: Concepts of business continuity SharePoint and Business continuity Patterns and anti-patterns SharePoint and SQL Server Always on Availability groups : what works, what doesn’t work (demos) Lessons learned from real projects SharePoint 2016 Min Role and Business continuity

O'Reilly Media Webcast: Building Real-Time Data Pipelines

SingleStore

As our customers tap into new sources of data or modify to existing data pipelines, we are often asked questions like: What technologies should we consider? Where can we reduce data latency? How can we simplify our data architecture? To eliminate the guesswork, we teamed up with Ben Lorica, Chief Data Scientist at O’Reilly Media to host a webcast centered around building real-time data pipelines.

Apache CarbonData+Spark to realize data convergence and Unified high performa...

Tech Triveni

Challenges in Data Analytics: Different application scenarios need different storage solutions: HBASE is ideal for point query scenarios but unsuitable for multi-dimensional queries. MPP is suitable for data warehouse scenarios but engine and data are coupled together which hampers scalability. OLAP stores used in BI applications perform best for Aggregate queries but full scan queries perform at a sub-optimal performance. Moreover, they are not suitable for real-time analysis. These distinct systems lead to low resource sharing and need different pipelines for data and application management.

Modern data architectures for real time analytics and engagement

Amazon Web Services

Metadata and Provenance for ML Pipelines with Hopsworks

Jim Dowling

AI/ML Infra Meetup | ML explainability in Michelangelo

Real-Time Analytics: Going Beyond Stream Processing With Apache Pinot

Recommended

Recommended

More Related Content

Similar to Real-Time Analytics: Going Beyond Stream Processing With Apache Pinot

Similar to Real-Time Analytics: Going Beyond Stream Processing With Apache Pinot (20)

More from Alluxio, Inc.

More from Alluxio, Inc. (20)

Recently uploaded

Recently uploaded (20)

Real-Time Analytics: Going Beyond Stream Processing With Apache Pinot