Lucene at Yelp - By Sudarshan Gaikaiwari

•

1 like•464 views

Lucene is used at Yelp to power various search services including business search, phone search, and review search. The index is split across multiple machines ("federation") to address performance issues from disk seek latency and the large index size. Lucy is used for indexing and searching across the federated index. The federator combines results from shards. Techniques like successive geobounds relaxation and efficiently retrieving the top k hits based on binomial distribution are used to improve search performance.

Technology Entertainment & Humor

Bio

1. Over a decade of experience in information retrieval
2. Used IR techniques at Symantec's DLP group
3. Search Engineer at Yelp

Outline

1. Overview of search services at Yelp
2. Federation Motivation
3. Lucy Indexing
4. Lucy Searching
5. Efficiently Retrieving top k hits

Hard Disk Seek Latency
Disk seek 10,000,000 ns

Source Software Engineering Advice from
Building Large-Scale Distributed Systems
Jeffery Dean

RAM read latency

Main memory
reference
100 ns

Pinning Index in RAM

● vmtouch
● mlock
● http://hoytech.com/vmtouch/

Problem

Index is too large fit in memory on a single machine

Geographical Sharding drawbacks

1. Cumbersome manual process to determine shard boundary
2. No guarantee that a boundary can be found.

Federation

1. Split index across multiple machines
2. Shard on business id
3. TF-IDF scores from different machines should be
comparable

Mapping businesses to shards

1. Assigning businesses to shards

shard = shardlist[hash(business_id) % len(shardlist)]

Problems
1. Involves re-indexing all the businesses if we want to add a
new shard

Advantages

1. Flexibility (move vbuckets from one shard to another)
2. Split hot spot shards

Lucy Master Slave Architecture

Separate indexing (masters)
A master for each shard of a service

Searching (slaves)
A slave for every replica of a service

Federator: Combining results across
shards
1. Once we distribute an index across shards we need a
component which will search all these shards and combine
their results.
2. Written in Python (runs inside a python web process).
3. Uses Tornado IO loop to send requests to all shards.
4. The transfer protocol for the requests in JSON RPC

Executing queries

1. Gather the top results for a query
2. Collect attribute statitics for attributes like places, categories

Lucene

1. Efficiently executes queries over the index
2. Provides how relevant the business is to the words in the
query (word score)
3. Upgrading lucene to 2.9/3.1 is WIP

Efficiently Retrieving top k hits

1. When user moves through multiple pages the number of
hits to be returned increases

num hits = start + count

2. So if we need to retrieve 500 hits the naive way would be to
retrieve 500 hits from each shard and then sort them

Binomial Distribution
Probability (r of top k hits) are in a particular shard

Mean

Variance

Simulation

Formula Hits selected from each Results Missed (%)
shard
k = 100
p = 0.2

24 0.017

32 0.0001407

44 0.00000

Results

1. ~ 50% savings over 100 hits (44 hits requested from each
shard)
2. 77% savings over 1000 hits (228 hits requested from each
shard)

Future work

1. In memory index
2. Move towards real time search

Despite extensive research on cryptography, secure and efficient query processing over outsourced data remains an open challenge. We develop communication-efficient and information-theoretically secure algorithms for privacy-preserving aggregation queries using multi-party computation (MPC). Specifically, query processing techniques over secret-shared data outsourced by single or multiple database owners are developed. These algorithms allow a user to execute queries on the secret-shared database and also prevent the network and the (adversarial) clouds to learn the user’s queries, results, or the database. We further develop (non-mandatory) privacy-preserving result verification algorithms that detect malicious behaviors, and experimentally validate the efficiency of our approach over large datasets, the size of which prior approaches to secret-sharing or MPC systems have not scaled to.

AWS re:Invent 2016: Using MXNet for Recommendation Modeling at Scale (MAC306)

Amazon Web Services

For many companies, recommendation systems solve important machine learning problems. But as recommendation systems grow to millions of users and millions of items, they pose significant challenges when deployed at scale. The user-item matrix can have trillions of entries (or more), most of which are zero. To make common ML techniques practical, sparse data requires special techniques. Learn how to use MXNet to build neural network models for recommendation systems that can scale efficiently to large sparse datasets.

Deep Learning with TensorFlow: Understanding Tensors, Computations Graphs, Im...

Altoros

[DSC 2016] 系列活動：李宏毅 / 一天搞懂深度學習

台灣資料科學年會

深度學習 ( Deep Learning ) 是機器學習 ( Machine Learning ) 中近年來備受重視的一支，深度學習根源於類神經網路 ( Artificial Neural Network ) 模型，但今日深度學習的技術和它的前身已截然不同，目前最好的語音辨識和影像辨識系統都是以深度學習技術來完成，你可能在很多不同的場合聽過各種用深度學習做出的驚人應用 ( 例如：最近紅遍大街小巷的 AlphaGo )，聽完以後覺得心癢癢的，想要趕快使用這項強大的技術，卻不知要從何下手學習，那這門課就是你所需要的。這門課程將由台大電機系李宏毅教授利用短短的一天議程簡介深度學習。以下是課程大綱：什麼是深度學習深度學習的技術表面上看起來五花八門，但其實就是三個步驟：設定好類神經網路架構、訂出學習目標、開始學習，這堂課會簡介如何使用深度學習的工具 Keras，它可以幫助你在十分鐘內完成深度學習的程式。另外，有人說深度學習很厲害、有各種吹捧，也有人說深度學習只是個噱頭，到底深度學習和其他的機器學習方法有什麼不同呢？這堂課要剖析深度學習和其它機器學習方法相比潛在的優勢。深度學習的各種小技巧雖然現在深度學習的工具滿街都是，想要寫一個深度學習的程式只是舉手之勞，但要得到好的成果可不簡單，訓練過程中各種枝枝節節的小技巧才是成功的關鍵。本課程中將分享深度學習的實作技巧及實戰經驗。有記憶力的深度學習模型機器需要記憶力才能做更多事情，這段課程要講解遞迴式類神經網路 ( Recurrent Neural Network )，告訴大家深度學習模型如何可以有記憶力。深度學習應用與展望深度學習可以拿來做甚麼？怎麼用深度學習做語音辨識？怎麼用深度學習做問答系統？接下來深度學習的研究者們在意的是什麼樣的問題呢？本課程希望幫助大家不只能了解深度學習，也可以有效率地上手深度學習，用在手邊的問題上。無論是從未嘗試過深度學習的新手，還是已經有一點經驗想更深入學習，都可以在這門課中有所收穫。

Machine Learning, Deep Learning and Data Analysis Introduction

Te-Yen Liu

The document provides an introduction and overview of machine learning, deep learning, and data analysis. It discusses key concepts like supervised and unsupervised learning. It also summarizes the speaker's experience taking online courses and studying resources to learn machine learning techniques. Examples of commonly used machine learning algorithms and neural network architectures are briefly outlined.

Introduction To Tensorflow

Rayyan Khalid

PPT

butest

The document discusses using support vector machines (SVMs) for intrusion detection on virtual machines. It provides background on virtual machines, intrusion detection systems, and SVMs. It then describes using a two-class SVM approach on a synthetic dataset containing normal and abnormal workload data from virtual machines to detect intrusions. The SVM model performed well, accurately detecting intrusions in over 80% of cases. The document concludes that machine learning techniques like SVMs show promise for developing accurate intrusion detection systems for virtual machines.

Introduction to Neural Networks in Tensorflow

Nicholas McClure

Nick McClure gave an introduction to neural networks using Tensorflow. He explained the basic unit of neural networks as operational gates and how multiple gates can be combined. He discussed loss functions, learning rates, and activation functions. McClure also covered convolutional neural networks, recurrent neural networks, and applications such as image captioning and style transfer. He concluded by discussing resources for staying up to date with advances in machine learning.

This document summarizes a presentation about MontySolr, an extension that allows embedding CPython in Solr. It was created by Roman Chyla of CERN to connect Python and Java applications without compromises. MontySolr uses JCC to embed a Python interpreter in Java, allowing Python code to interface with Solr. This provides a robust, tested integration that works for any Python or C/C++ application and leverages the strengths of both Solr and Invenio.

Motion to allow discovery

J. Patrick Lucas

How to Gain Greater Business Intelligence from Lucene/Solr

lucenerevolution

Presented by Patrick Beaucamp | Bpm-Conseil - See conference video - http://www.lucidimagination.com/devzone/events/conferences/lucene-revolution-2012 Vanilla, an Open Source business intelligence application by bpm-conseil.com, offers unique features such as report indexing through an embedded Lucene integration. Using Vanilla and Lucene, developers can manage both report indexing and external document indexing, which ultimately saves end users time when they search for specific keywords such as "product code," or "customer code." Vanilla can build upon an existing Solr/Lucene installation that takes care of all the indexing processes while Vanilla takes care of the Reporting/Dashboard creation. During this presentation, attendees will learn how we moved from embed Lucene Api to a Solr/Lucene platform and all the technical and business benefits from this architecture in terms of clustering, caching and access mode.

NRI Residency Pari Chowk Gr. noida for confirmed allotment call 9540110008

Gaurav Arora

NR1 Residency is a luxury residential development located in a premium area of the city, near amenities. It offers affordable yet high-quality homes in a lush, peaceful environment. Residents can enjoy amenities such as a clubhouse, fitness center, pool, and landscaped gardens. The development aims to provide an indulgent living experience and invigorating community where residents can enjoy soulful relaxation.

The Seven Deadly Sins of Solr - By Jay Hill

lucenerevolution

Jay Hill from Lucid Imagination will be giving a presentation on common "sins" or anti-patterns that are seen in Lucene and Solr implementations. The document introduces Hill and Lucid Imagination, which provides commercial support for Lucene and Solr. It notes that there will be time for questions and discusses some of the sins that will be covered, including sloth, greed, pride, lust, envy, gluttony, and wrath.

KEYNOTE: Lucene / Solr road map

lucenerevolution

This document provides a roadmap and history of the Lucene and Solr projects: - It outlines key dates and developments from 1997 to the present for Lucene and from 2004 to the present for Solr, including major releases and new features. - Lucene 4.0 introduced flexible indexing, faster multithreaded indexing, and other performance improvements. Lucene 4.1-4.3 added additional features like compressed stored fields and term vectors. - The document discusses recent enhancements to Solr like document routing, seamless online shard splitting, and cloud enhancements; and previews future directions like greater scalability, more "NoSQL" capabilities, and improved analytics.

(CMP305) Deep Learning on AWS Made EasyCmp305

Amazon Web Services

This document provides an overview of deep learning and its applications. It discusses how deep learning can be used for image classification and how neural networks learn hierarchical representations from data. The document highlights some of the challenges of deep learning, such as the large amounts of data and computation required. It also covers how deep learning models can be deployed in production using services like Amazon Web Services to ensure low latency, high availability, and continuous learning.

Spark Based Distributed Deep Learning Framework For Big Data Applications

Humoyun Ahmedov

Deep Learning architectures, such as deep neural networks, are currently the hottest emerging areas of data science, especially in Big Data. Deep Learning could be effectively exploited to address some major issues of Big Data, such as fast information retrieval, data classification, semantic indexing and so on. In this work, we designed and implemented a framework to train deep neural networks using Spark, fast and general data flow engine for large scale data processing, which can utilize cluster computing to train large scale deep networks. Training Deep Learning models requires extensive data and computation. Our proposed framework can accelerate the training time by distributing the model replicas, via stochastic gradient descent, among cluster nodes for data resided on HDFS.

My Master's Thesis

Humoyun Ahmedov

The document proposes a distributed deep learning framework for big data applications built on Apache Spark. It discusses challenges in distributed computing and deep learning in big data. The proposed system addresses issues like concurrency, asynchrony, parallelism through a master-worker architecture with data and model parallelism. Experiments on sentiment analysis using word embeddings and deep networks on a 10-node Spark cluster show improved performance with increased nodes.

Distributed Deep Learning with Docker at Salesforce

Docker, Inc.

Jeff Hajewski, Salesforce - There is a wealth of information on building deep learning models with PyTorch or TensorFlow. Anyone interested in building a deep learning model is only a quick search away from a number of clear and well written tutorials that will take them from zero knowledge to having a working image classifier. But what happens when you need to deploy these models in a production setting? At Salesforce, we use TensorFlow models to help us provide customers with insights into their data, and we do this as close to real-time as possible. Designing these systems in a scalable manner requires overcoming a number of design challenges, but the core component is Docker. Docker enables us to design highly scalable systems by allowing us to focus on service interactions, rather than how our services will interact with the hardware. Docker is also at the core of our test infrastructure, allowing developers and data scientists to build and test the system in an end to end manner on their local machines. While some of this may sound complex, the core message is simplicity - Docker allows us to focus on the aspects of the system that matter, greatly simplifying our lives.

Introduction to Azure DocumentDB

Denny Lee

Denny Lee introduced Azure DocumentDB, a fully managed NoSQL database service. DocumentDB provides elastic scaling of throughput and storage, global distribution with low latency reads and writes, and supports querying JSON documents with SQL and JavaScript. Common scenarios that benefit from DocumentDB include storing product catalogs, user profiles, sensor telemetry, and social graphs due to its ability to handle hierarchical and de-normalized data at massive scale.

"Introducing Distributed Tracing in a Large Software System", Kostiantyn Sha...

Fwdays

Software systems are growing in size and complexity when the business is growing, and sometimes it is hard to figure out what is going on. Various teams make different changes for different business capabilities. Distributed Tracing is a useful way to look under the hood and see for yourself what operations are being performed, what services are used in a certain use case, and how performant are they. In this talk, I will present what Distributed Tracing is and how we introduced it into our software system with some tips and tricks on what you should focus on if you want to do the same.

The Key to Machine Learning is Prepping the Right Data with Jean Georges Perrin

Databricks

"Practical Machine Learning With Ruby" by Iqbal Farabi (ID Ruby Community)

Tech in Asia ID

Realtime search at Yammer

Boris Aleksandrovsky

This presentation is focused on the architecture, scalability concerns, performance bottlenecks, operational characteristics and lessons learned while designing and implementing Yammer distributed real-time search system. Yammer is an enterprise social network SaaS offering with over 100,000 networks (including 85% of the Fortune 100) and nearly 2 million users. The search system we developed scales well up to 1B messages and serves a foundation of knowledge base analysis services Yammer is developing.

Real Time Search at Yammer

Lucidworks (Archived)

Real-time Search at Yammer - By Aleksandrovsky Boris

lucenerevolution

See conference video - http://www.lucidimagination.com/devzone/events/conferences/revolution/2011 This talk will be focused on the architecture, scalability concerns, performance bottlenecks, operational characteristics and lessons learned while designing and implementing Yammer distributed real-time search system. Yammer is an enterprise social network SaaS offering with over 100,000 networks (including 85% of the Fortune 100) and nearly 2 million users. The search system we developed scales well up to 1B messages and serves a foundation of knowledge base analysis services Yammer is developing.

Puppetcamp Melbourne - puppetdb

m_richardson

Puppet Camp Melbourne 2014: Node Collaboration with PuppetDB

Puppet

The document discusses how PuppetDB can help with infrastructure provisioning and configuration challenges by providing a centralized database for Puppet data. PuppetDB stores facts, catalogs, and other data produced by Puppet runs. This data can be queried using the PuppetDB API or modules like PuppetDBquery to dynamically configure systems based on the states of other nodes and share information between nodes. Examples provided include using PuppetDB to populate files, load balance systems, and allow services to discover each other.

Conf orm - explain

Louise Grandjonc

- The document discusses performance problems that can arise from ORM use and how to identify them. It recommends examining database logs to see queries being executed. Different types of scans like sequential, index, and bitmap scans are explained. Techniques like select_related, prefetch_related, and using indexes are suggested to reduce queries. The EXPLAIN command is demonstrated to analyze query plans and identify optimizations.

Functional solid

Matt Stine

This document provides biographical information about the speaker Matt Stine and outlines the topics he will cover in his presentation. Stine will discuss how software designs tend to degrade over time due to various factors, and how following the SOLID principles of object-oriented design can help address this problem and make designs more functional in nature. He will also cover trends in how software systems evolve, different programming paradigms, and the ongoing quest for software design "best practices."

AI and Deep Learning

Subrat Panda, PhD

This talk was presented in Startup Master Class 2017 - http://aaiitkblr.org/smc/ 2017 @ Christ College Bangalore. Hosted by IIT Kanpur Alumni Association and co-presented by IIT KGP Alumni Association, IITACB, PanIIT, IIMA and IIMB alumni. My co-presenter was Biswa Gourav Singh. And contributor was Navin Manaswi. http://dataconomy.com/2017/04/history-neural-networks/ - timeline for neural networks

Viewers also liked

Cpython embedded in solr - By Roman Chyla

lucenerevolution

Motion to allow discovery

J. Patrick Lucas

How to Gain Greater Business Intelligence from Lucene/Solr

lucenerevolution

NRI Residency Pari Chowk Gr. noida for confirmed allotment call 9540110008

Gaurav Arora

The Seven Deadly Sins of Solr - By Jay Hill

lucenerevolution

KEYNOTE: Lucene / Solr road map

lucenerevolution

Viewers also liked (6)

Cpython embedded in solr - By Roman Chyla

Motion to allow discovery

How to Gain Greater Business Intelligence from Lucene/Solr

NRI Residency Pari Chowk Gr. noida for confirmed allotment call 9540110008

The Seven Deadly Sins of Solr - By Jay Hill

KEYNOTE: Lucene / Solr road map

Similar to Lucene at Yelp - By Sudarshan Gaikaiwari

(CMP305) Deep Learning on AWS Made EasyCmp305

Amazon Web Services

Spark Based Distributed Deep Learning Framework For Big Data Applications

Humoyun Ahmedov

My Master's Thesis

Humoyun Ahmedov

Distributed Deep Learning with Docker at Salesforce

Docker, Inc.

Introduction to Azure DocumentDB

Denny Lee

"Introducing Distributed Tracing in a Large Software System", Kostiantyn Sha...

Fwdays

The Key to Machine Learning is Prepping the Right Data with Jean Georges Perrin

Databricks

"Practical Machine Learning With Ruby" by Iqbal Farabi (ID Ruby Community)

Tech in Asia ID

Realtime search at Yammer

Boris Aleksandrovsky

Real Time Search at Yammer

Lucidworks (Archived)

Real-time Search at Yammer - By Aleksandrovsky Boris

lucenerevolution

Puppetcamp Melbourne - puppetdb

m_richardson

Puppet Camp Melbourne 2014: Node Collaboration with PuppetDB

Conf orm - explain

Functional solid

AI and Deep Learning

Beyond php - it's not (just) about the code

Wim Godden

SenseiDB

Volodymyr Zhabiuk

Sensei is a search engine and database built on top of Lucene. It provides full text search, relevance ranking, and faceting capabilities. It was originally developed at LinkedIn for their search and analytics needs and later open sourced. Sensei aims to provide fast updates, rich querying, and horizontal scalability improvements over traditional Lucene deployments. It uses a distributed architecture with indexing and querying separated.

Distributed deep learning_over_spark_20_nov_2014_ver_2.8

Vijay Srinivas Agneeswaran, Ph.D

Crm saturday madrid 2017 jordi montaña - test automation

Demian Raschkovan

The document discusses test automation for Dynamics 365 using FakeXrmEasy. It introduces Jordi Montaña as the presenter and provides an agenda covering why unit testing is important, why FakeXrmEasy is used, and how it works. FakeXrmEasy provides an in-memory database and query engine to mock Dynamics 365 and enable unit testing without complex mocking setup. It allows for a sustainable approach to facilitating change management. The presenter then demonstrates FakeXrmEasy through a code example.

Similar to Lucene at Yelp - By Sudarshan Gaikaiwari (20)

(CMP305) Deep Learning on AWS Made EasyCmp305

Spark Based Distributed Deep Learning Framework For Big Data Applications

My Master's Thesis

Distributed Deep Learning with Docker at Salesforce

Introduction to Azure DocumentDB

"Introducing Distributed Tracing in a Large Software System", Kostiantyn Sha...

The Key to Machine Learning is Prepping the Right Data with Jean Georges Perrin

"Practical Machine Learning With Ruby" by Iqbal Farabi (ID Ruby Community)

Realtime search at Yammer

Real Time Search at Yammer

Real-time Search at Yammer - By Aleksandrovsky Boris

Puppetcamp Melbourne - puppetdb

Puppet Camp Melbourne 2014: Node Collaboration with PuppetDB

Conf orm - explain

Functional solid

AI and Deep Learning

Beyond php - it's not (just) about the code

SenseiDB

Distributed deep learning_over_spark_20_nov_2014_ver_2.8

Crm saturday madrid 2017 jordi montaña - test automation

More from lucenerevolution

Text Classification Powered by Apache Mahout and Lucene

lucenerevolution

Presented by Isabel Drost-Fromm, Software Developer, Apache Software Foundation/Nokia Gate 5 GmbH at Lucene/Solr Revolution 2013 Dublin Text classification automates the task of filing documents into pre-defined categories based on a set of example documents. The first step in automating classification is to transform the documents to feature vectors. Though this step is highly domain specific Apache Mahout provides you with a lot of easy to use tooling to help you get started, most of which relies heavily on Apache Lucene for analysis, tokenisation and filtering. This session shows how to use facetting to quickly get an understanding of the fields in your document. It will walk you through the steps necessary to convert your text documents into feature vectors that Mahout classifiers can use including a few anecdotes on drafting domain specific features. Configure

State of the Art Logging. Kibana4Solr is Here!

lucenerevolution

Presented by Markus Klose, Search + Big Data Consultant SHI Elektronische Medien GmbH at Lucene/Solr Revolution 2013 Dublin Kibana4Solr is search-driven, scalable, browser based and extremely user friendly (also for non-technical users). Logs are everywhere. Any device, system or human can potentially produce a huge amount of information saved in logs. The amount of available logs and their semi-structured nature make a meaningful processing in real-time quite a difficult task. Thus, valuable business insights stored in logs might be not found. Kibana4Solr is a search-driven approach to handle that challenge. It offers user-friendly and browser-based dashboard which can be easily customized to particular needs. In the session the Kibana4Solr will be introduced. Some light will be shed on the architectural features of Kibana4Solr. Some ideas will be given in terms of possible business uses cases. And finally a live demo of Kibana4Solr will be shown. Configure

Search at Twitter

lucenerevolution

The document describes Twitter's search architecture. It discusses how Twitter uses modified versions of Lucene called Earlybird to build real-time and archive search indexes. The real-time indexes are partitioned and replicated across clusters. New tweets are continuously added and searchable with low latency. Archive indexes contain older tweets on HDFS and are optimized for throughput over low latency. The system uses an analyzer to preprocess tweets before indexing and a service called the Blender to merge search results.

Building Client-side Search Applications with Solr

lucenerevolution

Presented by Daniel Beach, Search Application Developer, OpenSource Connections Solr is a powerful search engine, but creating a custom user interface can be daunting. In this fast paced session I will present an overview of how to implement a client-side search application using Solr. Using open-source frameworks like SpyGlass (to be released in September) can be a powerful way to jumpstart your development by giving you out-of-the box results views with support for faceting, autocomplete, and detail views. During this talk I will also demonstrate how we have built and deployed lightweight applications that are able to be performant under large user loads, with minimal server resources.

Integrate Solr with real-time stream processing applications

lucenerevolution

The document discusses integrating Apache Storm with Apache Solr for real-time stream processing applications. It provides an example of building a Storm topology that listens to click events from a URL shortener, counts the frequency of pages in a time window, ranks the top sites, and persists the results to Solr for visualization. The key points covered are using Spring to simplify building Storm topologies, integrating with Solr for indexing and search, and unit testing streaming data providers.

Scaling Solr with SolrCloud

lucenerevolution

Configure your Solr cluster to handle hundreds of millions of documents without even noticing, handle queries in milliseconds, use Near Real Time indexing and searching with document versioning. Scale your cluster both horizontally and vertically by using shards and replicas. In this session you'll learn how to make your indexing process blazing fast and make your queries efficient even with large amounts of data in your collections. You'll also see how to optimize your queries to leverage caches as much as your deployment allows and how to observe your cluster with Solr administration panel, JMX, and third party tools. Finally, learn how to make changes to already deployed collections —split their shards and alter their schema by using Solr API.

Administering and Monitoring SolrCloud Clusters

lucenerevolution

Presented by Rafal Kuć, Consultant and Software engineer, , Sematext Group, Inc. Even though Solr can run without causing any troubles for long periods of time it is very important to monitor and understand what is happening in your cluster. In this session you will learn how to use various tools to monitor how Solr is behaving at a high level, but also on Lucene, JVM, and operating system level. You'll see how to react to what you see and how to make changes to configuration, index structure and shards layout using Solr API. We will also discuss different performance metrics to which you ought to pay extra attention. Finally, you'll learn what to do when things go awry - we will share a few examples of troubleshooting and then dissect what was wrong and what had to be done to make things work again.

Implementing a Custom Search Syntax using Solr, Lucene, and Parboiled

lucenerevolution

In a recent project with the United States Patent and Trademark Office, Opensource Connections was asked to prototype the next generation of patent search - using Solr and Lucene. An important aspect of this project was the implementation of BRS, a specialized search syntax used by patent examiners during the examination process. In this fast paced session we will relate our experiences and describe how we used a combination of Parboiled (a Parser Expression Grammar [PEG] parser), Lucene Queries and SpanQueries, and an extension of Solr's QParserPlugin to build BRS search functionality in Solr. First we will characterize the patent search problem and then define the BRS syntax itself. We will then introduce the Parboiled parser and discuss various considerations that one must make when designing a syntax parser. Following this we will describe the methodology used to implement the search functionality in Lucene/Solr. Finally, we will include an overview our syntactic and semantic testing strategies. The audience will leave this session with an understanding of how Solr, Lucene, and Parboiled may be used to implement their own custom search parser.

Using Solr to Search and Analyze Logs

lucenerevolution

Enhancing relevancy through personalization & semantic search

lucenerevolution

I. The document discusses how CareerBuilder uses Solr for search at scale, handling over 1 billion documents and 1 million searches per hour across 300 servers. II. It then covers traditional relevancy scoring in Solr, which is based on TF-IDF, as well as ways to boost documents, fields, and terms. III. Advanced relevancy techniques are described, including using custom functions to incorporate domain-specific knowledge into scoring, and context-aware weighting of relevancy parameters. Personalization and recommendation approaches are also summarized, including attribute-based and collaborative filtering methods.

Real-time Inverted Search in the Cloud Using Lucene and Storm

lucenerevolution

Building real-time notification systems is often limited to basic filtering and pattern matching against incoming records. Allowing users to query incoming documents using Solr's full range of capabilities is much more powerful. In our environment we needed a way to allow for tens of thousands of such query subscriptions, meaning we needed to find a way to distribute the query processing in the cloud. By creating in-memory Lucene indices from our Solr configuration, we were able to parallelize our queries across our cluster. To achieve this distribution, we wrapped the processing in a Storm topology to provide a flexible way to scale and manage our infrastructure. This presentation will describe our experiences creating this distributed, real-time inverted search notification framework.

Solr's Admin UI - Where does the data come from?

lucenerevolution

Like many Web-Applications in the past, the Solr Admin UI up until 4.0 was entirely server based. It used separate code on the server to generate their Dashboards, Overviews and Statistics. All that code had to be maintained and still ... you weren't really able to use that kind of data for the things you needed it for. It was wrapped into HTML, most of the time difficult to extract and changed the structure from time to time w/o announcement. After a short look back, we're going to look into the current state of the Solr Admin UI - a client-side application, running completely in your browser. We'll see how it works, where it gets its data from and how you can get the very same data and wire that into your own custom applications, dashboards and/oder monitoring systems.

Schemaless Solr and the Solr Schema REST API

lucenerevolution

Schemaless Solr allows documents to be indexed without pre-configuring fields in the schema. As documents are indexed, previously unknown fields are automatically added to the schema with inferred field types. This is implemented using Solr's managed schema, field value class guessing to infer types, and automatic schema field addition. The schema and newly added fields can be accessed via the Schema REST API, and the schema can be modified at runtime when configured as mutable. However, schemaless mode has limitations such as single field analyses and no way to change field types after initial inference.

High Performance JSON Search and Relational Faceted Browsing with Lucene

lucenerevolution

This document discusses high performance JSON search and relational faceted browsing using Lucene. It introduces SIREn, a Lucene plugin for indexing and searching JSON documents with a nested data model. SIREn uses tree labeling techniques to represent the JSON document structure and enable both full-text and structural queries. It also allows for relational faceted browsing across multiple record collections through pivot navigation and query rewriting. While BlockJoin supports some nested data in Lucene, SIREn has better scalability through its compression techniques and more flexibility through its schema-agnostic approach.

Text Classification with Lucene/Solr, Apache Hadoop and LibSVM

lucenerevolution

In this session we will show how to build a text classifier using the Apache Lucene/Solr with libSVM libraries. We classify our corpus of job offers into a number of predefined categories. Each indexed document (a job offer) then belongs to zero, one or more categories. Known machine learning techniques for text classification include naïve bayes model, logistic regression, neural network, support vector machine (SVM), etc. We use Lucene/Solr to construct the features vector. Then we use the libsvm library known as the reference implementation of the SVM model to classify the document. We construct as many one-vs-all svm classifiers as there are classes in our setting, then using the Hadoop MapReduce Framework we reconcile the result of our classifiers. The end result is a scalable multi-class classifier. Finally we outline how the classifier is used to enrich basic solr keyword search.

Faceted Search with Lucene

lucenerevolution

Faceted search is a powerful technique to let users easily navigate the search results. It can also be used to develop rich user interfaces, which give an analyst quick insights about the documents space. In this session I will introduce the Facets module, how to use it, under-the-hood details as well as optimizations and best practices. I will also describe advanced faceted search capabilities with Lucene Facets.

Recent Additions to Lucene Arsenal

lucenerevolution

Presented by Shai Erera, Researcher, IBM Lucene's arsenal has recently expanded to include two new modules: Index Sorting and Replication. Index sorting lets you keep an index consistently sorted based on some criteria (e.g. modification date). This allows for efficient search early-termination as well as achieve better index compression. Index replication lets you replicate a search index to achieve high-availability, fault tolerance as well as take hot index backups. In this talk we will introduce these modules, discuss implementation and design details as well as best practices.

Turning search upside down

lucenerevolution

As part of their work with large media monitoring companies, Flax has developed a technique for applying tens of thousands of stored Lucene queries to a document in under a second. We'll talk about how we built intelligent filters to reduce the number of actual queries applied and how we extended Lucene to extract the exact hit positions of matches, the challenges of implementation, and how it can be used, including applications that monitor hundreds of thousands of news stories every day.

Spellchecking in Trovit: Implementing a Contextual Multi-language Spellchecke...

lucenerevolution

Presented by Xavier Sanchez Loro, Ph.D, Trovit Search SL This session aims to explain the implementation and use case for spellchecking in Trovit search engine. Trovit is a classified ads search engine supporting several different sites, one for each on country and vertical. Our search engine supports multiple indexes in multiple languages, each with several millions of indexed ads. Those indexes are segmented in several different sites depending on the type of ads (homes, cars, rentals, products, jobs and deals). We have developed a multi-language spellchecking system using solr and lucene in order to help our users to better find the desired ads and avoid the dreaded 0 results as much as possible. As such our goal is not pure orthographic correction, but also suggestion of correct searches for a certain site.

Shrinking the haystack wes caldwell - final

lucenerevolution

The document discusses how Intelligent Software Solutions (ISS) uses Apache Solr and natural language processing (NLP) techniques to help their customers analyze large amounts of unstructured data. ISS develops innovative solutions for government customers dealing with thousands of data sources. Their approach involves acquiring content, indexing it in Solr for search and discovery, semantically enriching it using NLP techniques like named entity recognition and clustering, and presenting focused "data perspectives" for analysis. They leverage multiple NLP approaches like GATE/Gazetteers and OpenNLP/machine learning to complement each other's strengths in finding both known and unknown relevant information.

More from lucenerevolution (20)

Text Classification Powered by Apache Mahout and Lucene

State of the Art Logging. Kibana4Solr is Here!

Search at Twitter

Building Client-side Search Applications with Solr

Integrate Solr with real-time stream processing applications

Scaling Solr with SolrCloud

Administering and Monitoring SolrCloud Clusters

Implementing a Custom Search Syntax using Solr, Lucene, and Parboiled

Using Solr to Search and Analyze Logs

Enhancing relevancy through personalization & semantic search

Real-time Inverted Search in the Cloud Using Lucene and Storm

Solr's Admin UI - Where does the data come from?

Schemaless Solr and the Solr Schema REST API

High Performance JSON Search and Relational Faceted Browsing with Lucene

Text Classification with Lucene/Solr, Apache Hadoop and LibSVM

Faceted Search with Lucene

Recent Additions to Lucene Arsenal

Turning search upside down

Spellchecking in Trovit: Implementing a Contextual Multi-language Spellchecke...

Shrinking the haystack wes caldwell - final

Recently uploaded

Ocean lotus Threat actors project by John Sitima 2024 (1).pptx

SitimaJohn

Ocean Lotus cyber threat actors represent a sophisticated, persistent, and politically motivated group that poses a significant risk to organizations and individuals in the Southeast Asian region. Their continuous evolution and adaptability underscore the need for robust cybersecurity measures and international cooperation to identify and mitigate the threats posed by such advanced persistent threat groups.

CAKE: Sharing Slices of Confidential Data on Blockchain

Claudio Di Ciccio

Presented at the CAiSE 2024 Forum, Intelligent Information Systems, June 6th, Limassol, Cyprus. Synopsis: Cooperative information systems typically involve various entities in a collaborative process within a distributed environment. Blockchain technology offers a mechanism for automating such processes, even when only partial trust exists among participants. The data stored on the blockchain is replicated across all nodes in the network, ensuring accessibility to all participants. While this aspect facilitates traceability, integrity, and persistence, it poses challenges for adopting public blockchains in enterprise settings due to confidentiality issues. In this paper, we present a software tool named Control Access via Key Encryption (CAKE), designed to ensure data confidentiality in scenarios involving public blockchains. After outlining its core components and functionalities, we showcase the application of CAKE in the context of a real-world cyber-security project within the logistics domain. Paper: https://doi.org/10.1007/978-3-031-61000-4_16

UI5 Controls simplified - UI5con2024 presentation

Wouter Lemaire

Driving Business Innovation: Latest Generative AI Advancements & Success Story

Safe Software

Are you ready to revolutionize how you handle data? Join us for a webinar where we’ll bring you up to speed with the latest advancements in Generative AI technology and discover how leveraging FME with tools from giants like Google Gemini, Amazon, and Microsoft OpenAI can supercharge your workflow efficiency. During the hour, we’ll take you through: Guest Speaker Segment with Hannah Barrington: Dive into the world of dynamic real estate marketing with Hannah, the Marketing Manager at Workspace Group. Hear firsthand how their team generates engaging descriptions for thousands of office units by integrating diverse data sources—from PDF floorplans to web pages—using FME transformers, like OpenAIVisionConnector and AnthropicVisionConnector. This use case will show you how GenAI can streamline content creation for marketing across the board. Ollama Use Case: Learn how Scenario Specialist Dmitri Bagh has utilized Ollama within FME to input data, create custom models, and enhance security protocols. This segment will include demos to illustrate the full capabilities of FME in AI-driven processes. Custom AI Models: Discover how to leverage FME to build personalized AI models using your data. Whether it’s populating a model with local data for added security or integrating public AI tools, find out how FME facilitates a versatile and secure approach to AI. We’ll wrap up with a live Q&A session where you can engage with our experts on your specific use cases, and learn more about optimizing your data workflows with AI. This webinar is ideal for professionals seeking to harness the power of AI within their data management systems while ensuring high levels of customization and security. Whether you're a novice or an expert, gain actionable insights and strategies to elevate your data processes. Join us to see how FME and AI can revolutionize how you work with data!

みなさんこんにちはこれ何文字まで入るの？40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの？えこ...

名前です男

Monitoring and Managing Anomaly Detection on OpenShift.pdf

Tosin Akinosho

Monitoring and Managing Anomaly Detection on OpenShift Overview Dive into the world of anomaly detection on edge devices with our comprehensive hands-on tutorial. This SlideShare presentation will guide you through the entire process, from data collection and model training to edge deployment and real-time monitoring. Perfect for those looking to implement robust anomaly detection systems on resource-constrained IoT/edge devices. Key Topics Covered 1. Introduction to Anomaly Detection - Understand the fundamentals of anomaly detection and its importance in identifying unusual behavior or failures in systems. 2. Understanding Edge (IoT) - Learn about edge computing and IoT, and how they enable real-time data processing and decision-making at the source. 3. What is ArgoCD? - Discover ArgoCD, a declarative, GitOps continuous delivery tool for Kubernetes, and its role in deploying applications on edge devices. 4. Deployment Using ArgoCD for Edge Devices - Step-by-step guide on deploying anomaly detection models on edge devices using ArgoCD. 5. Introduction to Apache Kafka and S3 - Explore Apache Kafka for real-time data streaming and Amazon S3 for scalable storage solutions. 6. Viewing Kafka Messages in the Data Lake - Learn how to view and analyze Kafka messages stored in a data lake for better insights. 7. What is Prometheus? - Get to know Prometheus, an open-source monitoring and alerting toolkit, and its application in monitoring edge devices. 8. Monitoring Application Metrics with Prometheus - Detailed instructions on setting up Prometheus to monitor the performance and health of your anomaly detection system. 9. What is Camel K? - Introduction to Camel K, a lightweight integration framework built on Apache Camel, designed for Kubernetes. 10. Configuring Camel K Integrations for Data Pipelines - Learn how to configure Camel K for seamless data pipeline integrations in your anomaly detection workflow. 11. What is a Jupyter Notebook? - Overview of Jupyter Notebooks, an open-source web application for creating and sharing documents with live code, equations, visualizations, and narrative text. 12. Jupyter Notebooks with Code Examples - Hands-on examples and code snippets in Jupyter Notebooks to help you implement and test anomaly detection models.

Cosa hanno in comune un mattoncino Lego e la backdoor XZ?

Speck&Tech

ABSTRACT: A prima vista, un mattoncino Lego e la backdoor XZ potrebbero avere in comune il fatto di essere entrambi blocchi di costruzione, o dipendenze di progetti creativi e software. La realtà è che un mattoncino Lego e il caso della backdoor XZ hanno molto di più di tutto ciò in comune. Partecipate alla presentazione per immergervi in una storia di interoperabilità, standard e formati aperti, per poi discutere del ruolo importante che i contributori hanno in una comunità open source sostenibile. BIO: Sostenitrice del software libero e dei formati standard e aperti. È stata un membro attivo dei progetti Fedora e openSUSE e ha co-fondato l'Associazione LibreItalia dove è stata coinvolta in diversi eventi, migrazioni e formazione relativi a LibreOffice. In precedenza ha lavorato a migrazioni e corsi di formazione su LibreOffice per diverse amministrazioni pubbliche e privati. Da gennaio 2020 lavora in SUSE come Software Release Engineer per Uyuni e SUSE Manager e quando non segue la sua passione per i computer e per Geeko coltiva la sua curiosità per l'astronomia (da cui deriva il suo nickname deneb_alpha).

Presentation of the OECD Artificial Intelligence Review of Germany

innovationoecd

AI-Powered Food Delivery Transforming App Development in Saudi Arabia.pdf

Techgropse Pvt.Ltd.

In this blog post, we'll delve into the intersection of AI and app development in Saudi Arabia, focusing on the food delivery sector. We'll explore how AI is revolutionizing the way Saudi consumers order food, how restaurants manage their operations, and how delivery partners navigate the bustling streets of cities like Riyadh, Jeddah, and Dammam. Through real-world case studies, we'll showcase how leading Saudi food delivery apps are leveraging AI to redefine convenience, personalization, and efficiency.

Infrastructure Challenges in Scaling RAG with Custom AI models

Zilliz

Building Retrieval-Augmented Generation (RAG) systems with open-source and custom AI models is a complex task. This talk explores the challenges in productionizing RAG systems, including retrieval performance, response synthesis, and evaluation. We’ll discuss how to leverage open-source models like text embeddings, language models, and custom fine-tuned models to enhance RAG performance. Additionally, we’ll cover how BentoML can help orchestrate and scale these AI components efficiently, ensuring seamless deployment and management of RAG systems in the cloud.

How to Get CNIC Information System with Paksim Ga.pptx

danishmna97

20240609 QFM020 Irresponsible AI Reading List May 2024

Matthew Sinclair

Your One-Stop Shop for Python Success: Top 10 US Python Development Providers

akankshawande

Mariano G Tinti - Decoding SpaceX

Mariano Tinti

HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU

panagenda

Webinar Recording: https://www.panagenda.com/webinars/hcl-notes-und-domino-lizenzkostenreduzierung-in-der-welt-von-dlau/ DLAU und die Lizenzen nach dem CCB- und CCX-Modell sind für viele in der HCL-Community seit letztem Jahr ein heißes Thema. Als Notes- oder Domino-Kunde haben Sie vielleicht mit unerwartet hohen Benutzerzahlen und Lizenzgebühren zu kämpfen. Sie fragen sich vielleicht, wie diese neue Art der Lizenzierung funktioniert und welchen Nutzen sie Ihnen bringt. Vor allem wollen Sie sicherlich Ihr Budget einhalten und Kosten sparen, wo immer möglich. Das verstehen wir und wir möchten Ihnen dabei helfen! Wir erklären Ihnen, wie Sie häufige Konfigurationsprobleme lösen können, die dazu führen können, dass mehr Benutzer gezählt werden als nötig, und wie Sie überflüssige oder ungenutzte Konten identifizieren und entfernen können, um Geld zu sparen. Es gibt auch einige Ansätze, die zu unnötigen Ausgaben führen können, z. B. wenn ein Personendokument anstelle eines Mail-Ins für geteilte Mailboxen verwendet wird. Wir zeigen Ihnen solche Fälle und deren Lösungen. Und natürlich erklären wir Ihnen das neue Lizenzmodell. Nehmen Sie an diesem Webinar teil, bei dem HCL-Ambassador Marc Thomas und Gastredner Franz Walder Ihnen diese neue Welt näherbringen. Es vermittelt Ihnen die Tools und das Know-how, um den Überblick zu bewahren. Sie werden in der Lage sein, Ihre Kosten durch eine optimierte Domino-Konfiguration zu reduzieren und auch in Zukunft gering zu halten. Diese Themen werden behandelt - Reduzierung der Lizenzkosten durch Auffinden und Beheben von Fehlkonfigurationen und überflüssigen Konten - Wie funktionieren CCB- und CCX-Lizenzen wirklich? - Verstehen des DLAU-Tools und wie man es am besten nutzt - Tipps für häufige Problembereiche, wie z. B. Team-Postfächer, Funktions-/Testbenutzer usw. - Praxisbeispiele und Best Practices zum sofortigen Umsetzen

Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack

shyamraj55

“I’m still / I’m still / Chaining from the Block”

Claudio Di Ciccio

HCL Notes and Domino License Cost Reduction in the World of DLAU

panagenda

Webinar Recording: https://www.panagenda.com/webinars/hcl-notes-and-domino-license-cost-reduction-in-the-world-of-dlau/ The introduction of DLAU and the CCB & CCX licensing model caused quite a stir in the HCL community. As a Notes and Domino customer, you may have faced challenges with unexpected user counts and license costs. You probably have questions on how this new licensing approach works and how to benefit from it. Most importantly, you likely have budget constraints and want to save money where possible. Don’t worry, we can help with all of this! We’ll show you how to fix common misconfigurations that cause higher-than-expected user counts, and how to identify accounts which you can deactivate to save money. There are also frequent patterns that can cause unnecessary cost, like using a person document instead of a mail-in for shared mailboxes. We’ll provide examples and solutions for those as well. And naturally we’ll explain the new licensing model. Join HCL Ambassador Marc Thomas in this webinar with a special guest appearance from Franz Walder. It will give you the tools and know-how to stay on top of what is going on with Domino licensing. You will be able lower your cost through an optimized configuration and keep it low going forward. These topics will be covered - Reducing license cost by finding and fixing misconfigurations and superfluous accounts - How do CCB and CCX licenses really work? - Understanding the DLAU tool and how to best utilize it - Tips for common problem areas, like team mailboxes, functional/test users, etc - Practical examples and best practices to implement right away

AI 101: An Introduction to the Basics and Impact of Artificial Intelligence

IndexBug

Taking AI to the Next Level in Manufacturing.pdf

ssuserfac0301

Read Taking AI to the Next Level in Manufacturing to gain insights on AI adoption in the manufacturing industry, such as: 1. How quickly AI is being implemented in manufacturing. 2. Which barriers stand in the way of AI adoption. 3. How data quality and governance form the backbone of AI. 4. Organizational processes and structures that may inhibit effective AI adoption. 6. Ideas and approaches to help build your organization's AI strategy.

Recently uploaded (20)

Ocean lotus Threat actors project by John Sitima 2024 (1).pptx

CAKE: Sharing Slices of Confidential Data on Blockchain

UI5 Controls simplified - UI5con2024 presentation

Driving Business Innovation: Latest Generative AI Advancements & Success Story

Monitoring and Managing Anomaly Detection on OpenShift.pdf

Cosa hanno in comune un mattoncino Lego e la backdoor XZ?

Presentation of the OECD Artificial Intelligence Review of Germany

AI-Powered Food Delivery Transforming App Development in Saudi Arabia.pdf

Infrastructure Challenges in Scaling RAG with Custom AI models

How to Get CNIC Information System with Paksim Ga.pptx

20240609 QFM020 Irresponsible AI Reading List May 2024

Your One-Stop Shop for Python Success: Top 10 US Python Development Providers

Mariano G Tinti - Decoding SpaceX

HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU

Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack

“I’m still / I’m still / Chaining from the Block”

HCL Notes and Domino License Cost Reduction in the World of DLAU

AI 101: An Introduction to the Basics and Impact of Artificial Intelligence

Taking AI to the Next Level in Manufacturing.pdf

Lucene at Yelp - By Sudarshan Gaikaiwari

1. Lucene @ Yelp Sudarshan Gaikaiwari

2. Bio 1. Over a decade of experience in information retrieval 2. Used IR techniques at Symantec's DLP group 3. Search Engineer at Yelp

3. Outline 1. Overview of search services at Yelp 2. Federation Motivation 3. Lucy Indexing 4. Lucy Searching 5. Efficiently Retrieving top k hits

4. The services we provide

5. Lucy: business search

6. Lucy also powers phone search

7. Cathy: she 'talks' a lot

8. Listsearch: it searches lists....

9. Reviewsearch: it searches reviews....

10. DYM: did you really mean that?

11. Suggest: auto completion

12. Federation Motivation

13. Problem Search is too slow

14. Hard Disk Seek Latency Disk seek 10,000,000 ns Source Software Engineering Advice from Building Large-Scale Distributed Systems Jeffery Dean

15. RAM read latency Main memory reference 100 ns

16. Pinning Index in RAM ● vmtouch ● mlock ● http://hoytech.com/vmtouch/

17. Problem Index is too large fit in memory on a single machine

18. Geographical sharding

19. Geographical Sharding drawbacks 1. Cumbersome manual process to determine shard boundary 2. No guarantee that a boundary can be found.

20. Federation 1. Split index across multiple machines 2. Shard on business id 3. TF-IDF scores from different machines should be comparable

21. Mapping businesses to shards 1. Assigning businesses to shards shard = shardlist[hash(business_id) % len(shardlist)] Problems 1. Involves re-indexing all the businesses if we want to add a new shard

22. Virtual Nodes

23. Advantages 1. Flexibility (move vbuckets from one shard to another) 2. Split hot spot shards

24. Lucy Master Slave Architecture Separate indexing (masters) A master for each shard of a service Searching (slaves) A slave for every replica of a service

25. Lucy Indexing

26.

27.

28.

29.

30. Lucy Searching

31.

32. Federator: Combining results across shards 1. Once we distribute an index across shards we need a component which will search all these shards and combine their results. 2. Written in Python (runs inside a python web process). 3. Uses Tornado IO loop to send requests to all shards. 4. The transfer protocol for the requests in JSON RPC

33. Lucy Server

34.

35.

36. Tokens to Business Attributes

37. Executing queries 1. Gather the top results for a query 2. Collect attribute statitics for attributes like places, categories

38. Lucene 1. Efficiently executes queries over the index 2. Provides how relevant the business is to the words in the query (word score) 3. Upgrading lucene to 2.9/3.1 is WIP

39.

40. Successive geobounds relaxation

41. Successive geobounds relaxation

42. Federation

43. Efficiently Retrieving top k hits 1. When user moves through multiple pages the number of hits to be returned increases num hits = start + count 2. So if we need to retrieve 500 hits the naive way would be to retrieve 500 hits from each shard and then sort them

44. Distribution of hits in shards

45.

46. Probability a hit is in a shard

47. Binomial Distribution Probability (r of top k hits) are in a particular shard Mean Variance

48. Formula Std Deviation Formula

49. Simulation Formula Hits selected from each Results Missed (%) shard k = 100 p = 0.2 24 0.017 32 0.0001407 44 0.00000

50. Simulation Graph

51. Results 1. ~ 50% savings over 100 hits (44 hits requested from each shard) 2. 77% savings over 1000 hits (228 hits requested from each shard)

52. Future work 1. In memory index 2. Move towards real time search

53. Come Join Us!

54. Thank You smg@yelp.com

Lucene at Yelp - By Sudarshan Gaikaiwari

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (6)

Similar to Lucene at Yelp - By Sudarshan Gaikaiwari

Similar to Lucene at Yelp - By Sudarshan Gaikaiwari (20)

More from lucenerevolution

More from lucenerevolution (20)

Recently uploaded

Recently uploaded (20)

Lucene at Yelp - By Sudarshan Gaikaiwari