Open Source Search Evolution

•

2 likes•1,179 views

Sematext Group, Inc.

From Gopher, WAIS, and Harvest to Lucene, Solr, SolrCloud, and Elasticsearch.

Technology

[Open Source]
Search Evolution
Otis Gospodnetić @otisg

Foci
1974 1995 now()
__________________________________________________________________________________________________________________
______
SEARCH

Then & Now
1990s 2014
WebGlimpse
Swish
Harvest
Ht://Dig
freeWAIS elasticsearch.

Still New?
elasticsearch.
…………………... 2000
…………………... 2004
…………………... 2010

Dominance
[Open Source]
Search Evolution

Big Cake
Big Data
Beyond Text
Memory Footprint
Distributed Model
Language Support
Indexing Speed, NRT
Relevance Algorithms

Language Support
Lucene 2004: ~ 20 languages
Lucene 2014: ~ 40 languages
most are stemmers

Relevance Models: VSM
TF IDF
For term i in document j
wi,j
= tfi,j
x log(N/dfi
)
tfi,j
= number of occurrences of i in j
dfi
= number of document containing i
N = total number of documents

Relevance Models: Pluggable
Lucene until 2011: 1 relevance model
Lucene 2014: 6 relevance models
got more?

Distributed Architecture
1 Master - N Slaves
good for scaling queries
not good for scaling data
Sharded index with replication
good for scaling queries
good for scaling data

Beyond Text
Geospatial Search
Classifier
Recommendation Engine
Key Value Store
NoSQL DB
Analytical DB

Recommender
Content Similarity
Collaborative Filtering

Key Value Store
id123 ⇒ manu:Apple desc:foo bar price:$111
id234 ⇒ manu:Sony desc:baz bam price:$222

NoSQL DB
Distributed
Replicated
Horizontally Scalable
Fast Retrieval
Searchable?

Gobble Gobble
If software is eating the world,
then [open source] search is gobbling it.
And has been for years.

Viewers also liked

Introduction to ElasticsearchSematext Group, Inc.

Side by Side with Elasticsearch and SolrSematext Group, Inc.

Side by Side with Elasticsearch & Solr, Part 2Sematext Group, Inc.

Running High Performance & Fault-tolerant Elasticsearch Clusters on DockerSematext Group, Inc.

Running High Performance and Fault Tolerant Elasticsearch Clusters on DockerSematext Group, Inc.

How to Run Solr on Docker and WhySematext Group, Inc.

Viewers also liked (6)

Introduction to Elasticsearch

Side by Side with Elasticsearch and Solr

Side by Side with Elasticsearch & Solr, Part 2

Running High Performance & Fault-tolerant Elasticsearch Clusters on Docker

Running High Performance and Fault Tolerant Elasticsearch Clusters on Docker

How to Run Solr on Docker and Why

Recently uploaded

How AI, OpenAI, and ChatGPT impact business and software.Curtis Poe

A Framework for Development in the AI AgeCprime

(How to Program) Paul Deitel, Harvey Deitel-Java How to Program, Early Object...AliaaTarek5

How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesThousandEyes

Assure Ecommerce and Retail Operations Uptime with ThousandEyesThousandEyes

TeamStation AI System Report LATAM IT Salaries 2024Lonnie McRorey

Time Series Foundation Models - current state and future directionsNathaniel Shimoni

Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24Mark Goldstein

TrustArc Webinar - How to Build Consumer Trust Through Data PrivacyTrustArc

A Journey Into the Emotions of Software DevelopersNicole Novielli

Merck Moving Beyond Passwords: FIDO Paris Seminar.pptxLoriGlavin3

Emixa Mendix Meetup 11 April 2024 about Mendix Native developmentPim van der Noll

Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024BookNet Canada

The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptxLoriGlavin3

Connecting the Dots for Information Discovery.pdfNeo4j

Scale your database traffic with Read & Write split using MySQL RouterMydbops

The Ultimate Guide to Choosing WordPress Pros and ConsPixlogix Infotech

From Family Reminiscence to Scholarly Archive .Alan Dix

[Webinar] SpiraTest - Setting New Standards in Quality AssuranceInflectra

Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptxLoriGlavin3

Recently uploaded (20)

How AI, OpenAI, and ChatGPT impact business and software.

A Framework for Development in the AI Age

(How to Program) Paul Deitel, Harvey Deitel-Java How to Program, Early Object...

How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes

Assure Ecommerce and Retail Operations Uptime with ThousandEyes

TeamStation AI System Report LATAM IT Salaries 2024

Time Series Foundation Models - current state and future directions

Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24

TrustArc Webinar - How to Build Consumer Trust Through Data Privacy

A Journey Into the Emotions of Software Developers

Merck Moving Beyond Passwords: FIDO Paris Seminar.pptx

Emixa Mendix Meetup 11 April 2024 about Mendix Native development

Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024

The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx

Connecting the Dots for Information Discovery.pdf

Scale your database traffic with Read & Write split using MySQL Router

The Ultimate Guide to Choosing WordPress Pros and Cons

From Family Reminiscence to Scholarly Archive .

[Webinar] SpiraTest - Setting New Standards in Quality Assurance

Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptx

Open Source Search Evolution

1. [Open Source] Search Evolution Otis Gospodnetić @otisg

2. Today

3. The Early Days

4. Even Earlier Days

5. Foci 1974 1995 now() __________________________________________________________________________________________________________________ ______ SEARCH

6. Otis Who? SEARCH

7. Then & Now 1990s 2014 WebGlimpse Swish Harvest Ht://Dig freeWAIS elasticsearch.

8. Still New? elasticsearch. …………………... 2000 …………………... 2004 …………………... 2010

9. Dominance [Open Source] Search Evolution

10. Big Cake Big Data Beyond Text Memory Footprint Distributed Model Language Support Indexing Speed, NRT Relevance Algorithms

11. Language Support: Stemming

12. Language Support: Lemmatization

13. Language Support: Morphology

14. Language Support Lucene 2004: ~ 20 languages Lucene 2014: ~ 40 languages most are stemmers

15. Relevance Models: VSM TF IDF For term i in document j wi,j = tfi,j x log(N/dfi ) tfi,j = number of occurrences of i in j dfi = number of document containing i N = total number of documents

16. Relevance Models: Pluggable Lucene until 2011: 1 relevance model Lucene 2014: 6 relevance models got more?

17. Distributed Architecture 1 Master - N Slaves good for scaling queries not good for scaling data Sharded index with replication good for scaling queries good for scaling data

18. Indexing Speed & NRT Search

19. Memory Footprint

20. Beyond Text Geospatial Search Classifier Recommendation Engine Key Value Store NoSQL DB Analytical DB

21. Geospatial Search

22. Classifier

23. Recommender Content Similarity Collaborative Filtering

24. Key Value Store id123 ⇒ manu:Apple desc:foo bar price:$111 id234 ⇒ manu:Sony desc:baz bam price:$222

25. NoSQL DB Distributed Replicated Horizontally Scalable Fast Retrieval Searchable?

26. Slicing & Dicing

27. Analytical Queries

28. Gobble Gobble If software is eating the world, then [open source] search is gobbling it. And has been for years.

29. FIN. Questions otis@sematext.com