Some key goals in developing Lucense were to create a single search platform that could scale to handle large volumes of traffic and data while maintaining high performance, stability, and extensibility. Lucense was built from scratch based on learnings from other search engines like Sphinx and Solr. Developing Lucense provided several benefits including reducing hardware costs, improving search speeds and latencies, achieving 99.99% uptime, allowing quick functional enhancements through configuration changes, and exposing developers to advanced techniques that made them better developers. It also provided learnings around benchmarking, investigating

1. The Naukri Search Story
Aditya Varun Chadha
Search Team
Naukri.com, Info Edge

2. Genesis

3. Goals

Single Search Platform

Scale / Performance / Stability

Extensibility (quick turnaround)

4. Inspiration

Experience with other solutions (Sphinx/Solr)

Application / Product Requirements

Complex Filtering / Ranking / Sorting / Faceting…

Huge Volumes

High Traffic

5. Lucense

From Scratch: Disruptive, Daring
– all mistakes are our own

Applied Learnings from Working INSIDE other
engines

6. Impact

Reduced Hardware Costs to Half

Faster Search (lesser latencies)

Better Concurrency

7. Impact

99.99% uptime

Stability / Maintainability

8. Impact

Functional Enhancements = Configuration
Changes (quick dev turnaround)

Inversion of Control

State of the Art lib usage (Lucene 3.4)

latest features exposed

9. Impact

Better understanding of deep specifics in
information retrieval. Better arsenal to solve
search-like issues

Exposure to awe-inspiring design / coding
techniques

I am a better developer than I was before the
initiative

10. Learnings

Benchmark, Benchmark, Benchmark

TDD: T = {unit, perf, load}

Diamonds (and demons) are often found deep

11. Learnings

There is no magician inside your machine or
that external library

Read code as if you wrote it

Frameworks/Libs are pliable to extension / change

Know not only how/what to code, but also where to
code (Big O!)

12. Learnings

My team mates are geniuses

Don't think alone, involve the genius sitting next to
you

Articulating the problem in the right context is your
responsibility.

​
Fresh minds (to eat?) are why you work in a team

13. Learnings

Stay true to the Platform / Framework
constraint to ensure lasting extensibility

The Big O goes only so far. True love for the art
of computer programming goes beyond
asymptotic analysis

14. Glimpse of Specifics

Expressions

Arbitrary combination of primitive
variables/constants

Aggregations over data-sets, a-la MySQL group-by

Fast, light-weight, stateless (thread-safe)

Custom Aggregation Plugin framework

Featured Company / Featured Job / Add
Rotation etc.

Customizable Analysis of Text

Pluggable (Semantic) Augmentation/Replacement in
query/indexing

15. Specifics

Real Time Search

Auxillary Data Store (Actions / Links …)

True Distributed Search

Search Dashboard

Health

Trends

Deeper Infrastructural Support for Semantics

16. Co-Conspirors