Queue Based Solr Indexing with Collection Management: Presented by Devansh Dhutia, Gannett Co.

•

3 likes•2,165 views

This document discusses Gannett's transition from synchronous to asynchronous indexing using queues. It summarizes Gannett's current indexing architecture, the problems with synchronous operations like outages during schema changes, and how moving to a queue-based system with RabbitMQ allows for faster schema changes, auto-scaling of workers, and more resilient indexing with minimal authoring impact. The queue-based approach includes real-time, batch, and prep queues to handle different types of document changes and updates in a decoupled, eventually consistent manner.

Technology

O C T O B E R 1 3 - 1 6 , 2 0 1 6 • A U S T I N , T X

Queue Based Indexing & Collection Management
Devansh Dhutia
Platform Architect

3
01
•  National & Local newspaper/media company
•  92+ Markets in 33 states

5
01
Agenda
•  Solr @ Gannett
•  Current State
•  Collection Management
•  Queuing Solution
•  Future Work
•  Questions

6
02
@
Site Search
CMS Search
Analytics
Personalization
40+ Applications
20M+
Integral pillar of Gannett’s Digital Platform
total
documents
800,000+ per
month
Growing rapidly
100,000+ requests
per minute
Highly Available
~100ms average
response time
Extremely Fast
8 nodes
256 gb memory
per availability zone

8
01
Current State
•  Synchronous Operations
•  Near Realtime
•  Time Consuming schema changes
•  Visible outage impact

9
01
Collection Management
•  Create Collection
•  Deploy Batch Indexer
•  Index new Collection
•  Update Alias to new Collection
•  Run catch up
•  Deploy Search/Index Apps

13
01
Outage Problems
•  Spinning Wheel
•  Duplicate content
•  Unable to ﬁnd new content
•  Frustrated editors
•  Ux & other presentation layers

14
01
Enter Queues
•  Asynchronous Write Operations
•  Near Realtime
•  Faster schema changes
•  Auto scale indexing workers
•  Low authoring outage impact
•  Eventually consistent

16
01
RabbitMQ
•  Clustered & Highly Available
•  FIFO
•  pub/sub model
•  Consistent Hash / Multiple Queues

18
01
Components
•  Realtime Queue
•  Batch Queue
•  Prep Queue
•  Deadletter Queue
•  Indexing Service
•  Prep mode
•  Batch Push Service

19
01
Future Work
•  Continuous Delivery of schema
•  Build payload in one zone only
•  Automated Deadletter handling
•  Earlier notiﬁcation of potential failure

20
01
Thank you
Interested in joining our team at Gannett?
http://www.gannett.com/careers

Treasure Data simplifies event analytics for the complex digital world. Our customers send us 1,000,000 events per second and issue 30,000+ Presto queries everyday to understand their customers better. One of the challenges is designing a cloud database with zero downtime to support a global customer base. We have achieved this goal by developing several open-source technologies; Fluentd and Embulk enable seamless log collection from stream/batch sources, and with MessagePack we can provide an extensible columnar store that accommodates future schema changes. Finally, Presto allows us to serve a wide variety of data processing our customers perform on our service. In this talk, I will present an overview of our system, and how our customers keep using Presto while collecting and extending their data set.

Data can be viewed as the exhaust of online activity. With the rise of cloud-based data platforms, barriers to data storage and transfer have crumbled. The demand for creative applications and learning from those datasets has accelerated. Rapid acceleration can quickly accrue disorder, and disorderly data design can turn the deepest data lake into an impenetrable swamp. In this talk, I will discuss the evolution of the data science workflow at Expedia with a special emphasis on Learning to Rank problems. From the heroic early days of ad-hoc Spark exploration to our first production sort model on the cloud, we will explore the process of industrializing the workflow. Layered over our story, I will share some best practices and suggestions on how to keep your data productive, or even pull your organization out of the data swamp.

Bullet: A Real Time Data Query Engine

DataWorks Summit

Bullet is an open sourced, lightweight, pluggable querying system for streaming data without a persistence layer implemented on top of Storm. It allows you to filter, project, and aggregate on data in transit. It includes a UI and WS. Instead of running queries on a finite set of data that arrived and was persisted or running a static query defined at the startup of the stream, our queries can be executed against an arbitrary set of data arriving after the query is submitted. In other words, it is a look-forward system. Bullet is a multi-tenant system that scales independently of the data consumed and the number of simultaneous queries. Bullet is pluggable into any streaming data source. It can be configured to read from systems such as Storm, Kafka, Spark, Flume, etc. Bullet leverages Sketches to perform its aggregate operations such as distinct, count distinct, sum, count, min, max, and average. An instance of Bullet is currently running at Yahoo against its user engagement data pipeline. We’ll highlight how it is powering internal use-cases such as web page and native app instrumentation validation. Finally, we’ll show a demo of Bullet and go over query performance numbers.

DataEngConf SF16 - High cardinality time series search

Hakka Labs

Meetup070416 Presentations

Ana Rebelo

Hoodie: How (And Why) We built an analytical datastore on Spark

Vinoth Chandar

Building a Vibrant Search Ecosystem @ Bloomberg: Presented by Steven Bower & ...

Lucidworks

Ubiquitous Solr - A Database's Not-So-Evil Twin: Presented by Ayon Sinha, Wal...

Lucidworks

Presto@Netflix Presto Meetup 03-19-15

Zhenxiao Luo

Leveraging the Power of Solr with Spark: Presented by Johannes Weigend, QAware

Lucidworks

The evolution of the big data platform @ Netflix (OSCON 2015)

Eva Tse

At Netflix, the big data platform is the foundation for analytics that drive all product decisions that directly impact our customer experience. As for scale, it is one of the top three largest services running at Netflix, in terms of compute power and data size. In this talk, we will take the audience through a journey to understand how we scale the platform to handle the increasing amount of data (over 500 billion events generated daily), the increasing demand of analytics (which translates to compute power), and the increasing number of users dependent on our platform to make business decisions.

Never Stop Exploring - Pushing the Limits of Solr: Presented by Anirudha Jadh...

Lucidworks

Евгений Бобров "Powered by OSS. Масштабируемая потоковая обработка и анализ б...

Fwdays

Технологии с открытым исходным кодом, такие как Microsoft Orleans и ElasticSearch, - ключевые элементы архитектуры YouScan. О том, как они помогают справляться с постоянно растущими объемами данных из социальных сетей, об эволюции архитектуры YouScan, я расскажу в данном докладе.

How Tencent Applies Apache Pulsar to Apache InLong - Pulsar Summit Asia 2021

StreamNative

Apache InLong is a one-stop data streaming platform, it chooses Apache Pulsar to cache data for forwarding sort. Apache Pulsar has great reliability and stability, which helps the InLong to be more confident for users. This session will share Tencent Big Data Team's journal of adopting Pulsar in their core data engine to process tens of billions of data integration. Besides, some problems they encountered during the process and the improvements on Pulsar they have made will also be shared as an example for future Pulsar users.

Streaming Aggregation in Solr - New Horizons for Search: Presented by Erick E...

Lucidworks

Виталий Бондаренко "Fast Data Platform for Real-Time Analytics. Architecture ...

Fwdays

We will start from understanding how Real-Time Analytics can be implemented on Enterprise Level Infrastructure and will go to details and discover how different cases of business intelligence be used in real-time on streaming data. We will cover different Stream Data Processing Architectures and discus their benefits and disadvantages. I'll show with live demos how to build Fast Data Platform in Azure Cloud using open source projects: Apache Kafka, Apache Cassandra, Mesos. Also I'll show examples and code from real projects.

Webinar 2017. Supercharge your analytics with ClickHouse. Alexander Zaitsev

Altinity Ltd

Kibana + timelion: time series with the elastic stack

Sylvain Wallez

Events Processing and Data Analysis with Lucidworks Fusion: Presented by Kira...

Lucidworks

Large Scale Log Analytics with Solr: Presented by Rafał Kuć & Radu Gheorghe, ...

Lucidworks

What's hot

Searching for Better Code: Presented by Grant Ingersoll, Lucidworks

Lucidworks

Presto meetup 2015-03-19 @Facebook

Treasure Data, Inc.

TweetMogaz - The Arabic Tweets Platform: Presented by Ahmed Adel, BADR

Lucidworks

Rental Cars and Industrialized Learning to Rank with Sean Downes

Databricks

Bullet: A Real Time Data Query Engine

DataWorks Summit

DataEngConf SF16 - High cardinality time series search

Hakka Labs

Meetup070416 Presentations

Ana Rebelo

Hoodie: How (And Why) We built an analytical datastore on Spark

Vinoth Chandar

Building a Vibrant Search Ecosystem @ Bloomberg: Presented by Steven Bower & ...

Lucidworks

Ubiquitous Solr - A Database's Not-So-Evil Twin: Presented by Ayon Sinha, Wal...

Lucidworks

Presto@Netflix Presto Meetup 03-19-15

Zhenxiao Luo

Leveraging the Power of Solr with Spark: Presented by Johannes Weigend, QAware

Lucidworks

The evolution of the big data platform @ Netflix (OSCON 2015)

Eva Tse

Never Stop Exploring - Pushing the Limits of Solr: Presented by Anirudha Jadh...

Lucidworks

Евгений Бобров "Powered by OSS. Масштабируемая потоковая обработка и анализ б...

Fwdays

How Tencent Applies Apache Pulsar to Apache InLong - Pulsar Summit Asia 2021

StreamNative

Streaming Aggregation in Solr - New Horizons for Search: Presented by Erick E...

Lucidworks

Виталий Бондаренко "Fast Data Platform for Real-Time Analytics. Architecture ...

Fwdays

Webinar 2017. Supercharge your analytics with ClickHouse. Alexander Zaitsev

Altinity Ltd

Kibana + timelion: time series with the elastic stack

Sylvain Wallez

What's hot (20)