The Rising Sun: Making the most of Solr power

•

1 like•1,212 views

The document appears to be slides from a presentation titled "THE RISING SUN: Making the most of Solr power" given at the code4lib '09 conference in Providence, RI. The presentation covers various topics related to optimizing performance and functionality in Solr such as performance measurement, memory usage, query parsing, data importing, request handlers, faceting, user interfaces, and the Solr community. It also mentions resources provided by Lucid Imagination such as blog posts, podcasts and articles on using Solr and Lucene.

Technology

http://ﬂickr.com/photos/thepma/640151954/
Wednesday, February 25, 2009 1

THE RISING SUN:
Making the most of Solr power

code4lib '09, Providence, RI
Erik Hatcher
erik.hatcher@lucidimagination.com

Wednesday, February 25, 2009 2

#9 Performance

Measure

Wednesday, February 25, 2009 3

#8 Memory

omitNorms
omitTf (1.4)
Wednesday, February 25, 2009 4

#7 Query Parsing

quot;It Dependsquot;
More Like This:
http://www.lucidimagination.com/blog/2009/02/22/
exploring-query-parsers/
Wednesday, February 25, 2009 5

#6 Data Import

DIH, Solr Cell,
CSV, LuSql, APIs,...
solr-ruby Mapper
Wednesday, February 25, 2009 6

Obligatory XML

Wednesday, February 25, 2009 7

#5 Request Handlers

Leverage Solr's
conﬁgurability
Wednesday, February 25, 2009 8

http://localhost:8983/solr/
timeline.xml

Wednesday, February 25, 2009 9

#4 Solr as IR toolkit

TermVectorRequestHandler
StatsComponent
Luke

Wednesday, February 25, 2009 10

#3 LocalSolr

geo searching

Wednesday, February 25, 2009 11

#2 Faceting

Solr 1.4 performance!
(*bows* to Yonik)
Multi-select
Wednesday, February 25, 2009 12

Multi-select faceting

http://www.lucidimagination.com/search/p:lucene,solr/s:email,wiki/l:user?q=marc
Wednesday, February 25, 2009 13

#1 User Interface

The Interface is
the App
Wednesday, February 25, 2009 14

Solritas

Velocity
templated
response
Wednesday, February 25, 2009 16

#0 Community

solr-user@lucene.apache.org

Wednesday, February 25, 2009 17

quot;LucidFindquot;

http://www.lucidimagination.com/search
ﬁnd@lucidimagination.com

Wednesday, February 25, 2009 18

Lucid Supports Solr
lucidimagination.com

Wednesday, February 25, 2009 19

Lucid Articles

Grant Ingersoll Erik Hatcher

Getting Started with Lucene Getting Started with Solr
(includes screencast), co-
authored with Jonathan
Debugging Relevance Issues in
Knudsen
Search

Sami Siren
Optimizing Findability in
Lucene and Solr
Content Extraction with Tika
Yonik Seeley
Mark Miller
Faceted Search with Solr
Scaling Lucene and Solr

Wednesday, February 25, 2009 20

Lucid Podcasts

Interviews with:
Doug Cutting (creator of Lucene)
Ryan McKinley (Solr committer)
Chris Hostetter (Solr committer)
More coming!

Wednesday, February 25, 2009 21

Lucid Blogging

Mark Miller: quot;Exploring Query Parsersquot;, quot;Highlighting
Highlighter Thoughtsquot;, quot;Investigating OOM and other
JVM issuesquot;, quot;Looking forward to new features in Solr
1.4quot;

Erik Hatcher: quot;acts_as_solr with rich document
indexingquot;

Grant Ingersoll: quot;Sorting, Faceting, and Schema Design
in Solrquot;

Wednesday, February 25, 2009 22

Pre-release ebook available now!

Wednesday, February 25, 2009 23

The Good Ol' Blues Brothers
Boys

Wednesday, February 25, 2009 24

Think *inside* the box. Inside the *search* box, that is. The "best"* search results incorporate many more factors than (just) textual matching and relevancy. Search experience owners manage query context rules, signals automatically feed back machine learned factors, users implicit and explicit behaviors filter and weight future interactions. Synergy emerges with several cooperating (just) searches. This talk will showcase and detail several (just) search examples including rules, typeahead/suggest, signals, and location awareness, bringing them all together into a cohesive search experience.

Lucene's Latest (for Libraries)

Erik Hatcher

Solr Indexing and Analysis Tricks

Erik Hatcher

This session will introduce and demonstrate several techniques for enhancing the search experience by augmenting documents during indexing. First we'll survey the analysis components available in Solr, and then we'll delve into using Solr's update processing pipeline to modify documents on the way in. The session will build on Erik's "Poor Man's Entity Extraction" blog at http://www.searchhub.org/2013/06/27/poor-mans-entity-extraction-with-solr/

Solr Powered Libraries

Erik Hatcher

Using Apache Lucene and Solr search technologies, information and knowledge have become vastly more searchable, findable, and accessible. Because scholars and researchers are some of the most demanding users of search systems, the problems encountered by the implementers are complex. For example, many of the applications built on these technologies also thrive on intentionally designed-in serendipitous discovery capabilities, bringing to light previously unknown, yet related and potentially interesting, content. Libraries and other public knowledge-sharing environments, such as Wikipedia, generally embrace "open source" and community improving contributions as core principles, making a lovely synergy with the power, features, and community-driven ecosystem provided by Lucene and Solr. This talk will introduce you to several Solr powered library-related systems, detail how they work, and leave you with lessons learned that can be applied to your applications.

Solr Query ParsingErik Hatcher

"Solr Update" at code4lib '13 - Chicago

Erik Hatcher

Solr is continually improving. Solr 4 was recently released, bringing dramatic changes in the underlying Lucene library and Solr-level features. It's tough for us all to keep up with the various versions and capabilities. This talk will blaze through the highlights of new features and improvements in Solr 4 (and up). Topics will include: SolrCloud, direct spell checking, surround query parser, and many other features. We will focus on the features library coders really need to know about.

Query Parsing - Tips and Tricks

Erik Hatcher

In this talk, Solr's built-in query parsers will be detailed included when and how to use them. Solr has nested query parsing capability, allowing for multiple query parsers to be used to generate a single query. The nested query parsing feature will be described and demonstrated. In many domains, e-commerce in particular, parsing queries often means interpreting which entities (e.g. products, categories, vehicles) the user likely means; this talk will conclude with techniques to achieve richer query interpretation.

Solr 4

Erik Hatcher

Solr 4.0 dramatically improves scalability, performance, and flexibility. An overhauled Lucene underneath sports near real-time (NRT) capabilities allowing indexed documents to be rapidly visible and searchable. Lucene’s improvements also include pluggable scoring, much faster fuzzy and wildcard querying, and vastly improved memory usage. These Lucene improvements automatically make Solr much better, and Solr magnifies these advances with “SolrCloud.” SolrCloud enables highly available and fault tolerant clusters for large scale distributed indexing and searching. There are many other changes that will be surveyed as well. This talk will cover these improvements in detail, comparing and contrasting to previous versions of Solr.

Solr Recipes

Erik Hatcher

Lucene for Solr Developers

Erik Hatcher

You’re Solr powered, and needing to customize its capabilities. Apache Solr is flexibly architected, with practically everything pluggable. Under the hood, Solr is driven by the well-known Apache Lucene. Lucene for Solr Developers will guide you through the various ways in which Solr can be extended, customized, and enhanced with a bit of Lucene API know-how. We’ll delve into improving analysis with custom character mapping, tokenizing, and token filtering extensions; show why and how to implement specialized query parsing, and how to add your own search and update request handling.

Introduction to Solr

Erik Hatcher

Apache Solr serves search requests at the enterprises and the largest companies around the world. Built on top of the top-notch Apache Lucene library, Solr makes indexing and searching integration into your applications straightforward. Solr provides faceted navigation, spell checking, highlighting, clustering, grouping, and other search features. Solr also scales query volume with replication and collection size with distributed capabilities. Solr can index rich documents such as PDF, Word, HTML, and other file types.

Solr FlairErik Hatcher

Introduction to Solr

Erik Hatcher

Lucene for Solr DevelopersErik Hatcher

Introduction to SolrErik Hatcher

Rapid Prototyping with SolrErik Hatcher

Lucene for Solr DevelopersErik Hatcher

What's New in Solr 3.x / 4.0Erik Hatcher

The Art of the Pitch: WordPress Relationships and Sales

Laura Byrne

Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes? All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.

Essentials of Automations: The Art of Triggers and Actions in FME

Safe Software

In this second installment of our Essentials of Automations webinar series, we’ll explore the landscape of triggers and actions, guiding you through the nuances of authoring and adapting workspaces for seamless automations. Gain an understanding of the full spectrum of triggers and actions available in FME, empowering you to enhance your workspaces for efficient automation. We’ll kick things off by showcasing the most commonly used event-based triggers, introducing you to various automation workflows like manual triggers, schedules, directory watchers, and more. Plus, see how these elements play out in real scenarios. Whether you’re tweaking your current setup or building from the ground up, this session will arm you with the tools and insights needed to transform your FME usage into a powerhouse of productivity. Join us to discover effective strategies that simplify complex processes, enhancing your productivity and transforming your data management practices with FME. Let’s turn complexity into clarity and make your workspaces work wonders!

Recently uploaded

The Art of the Pitch: WordPress Relationships and Sales

Laura Byrne

Essentials of Automations: The Art of Triggers and Actions in FME

Safe Software

FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf

FIDO Alliance

Elizabeth Buie - Older adults: Are we really designing for our future selves?

Nexer Digital

Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf

Paige Cruz

Monitoring and observability aren’t traditionally found in software curriculums and many of us cobble this knowledge together from whatever vendor or ecosystem we were first introduced to and whatever is a part of your current company’s observability stack. While the dev and ops silo continues to crumble….many organizations still relegate monitoring & observability as the purview of ops, infra and SRE teams. This is a mistake - achieving a highly observable system requires collaboration up and down the stack. I, a former op, would like to extend an invitation to all application developers to join the observability party will share these foundational concepts to build on:

GraphRAG is All You need? LLM & Knowledge Graph

Guy Korland

Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs. 1. Unifying Large Language Models and Knowledge Graphs: A Roadmap. https://arxiv.org/abs/2306.08302 2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs: https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/

Removing Uninteresting Bytes in Software Fuzzing

Aftab Hussain

Imagine a world where software fuzzing, the process of mutating bytes in test seeds to uncover hidden and erroneous program behaviors, becomes faster and more effective. A lot depends on the initial seeds, which can significantly dictate the trajectory of a fuzzing campaign, particularly in terms of how long it takes to uncover interesting behaviour in your code. We introduce DIAR, a technique designed to speedup fuzzing campaigns by pinpointing and eliminating those uninteresting bytes in the seeds. Picture this: instead of wasting valuable resources on meaningless mutations in large, bloated seeds, DIAR removes the unnecessary bytes, streamlining the entire process. In this work, we equipped AFL, a popular fuzzer, with DIAR and examined two critical Linux libraries -- Libxml's xmllint, a tool for parsing xml documents, and Binutil's readelf, an essential debugging and security analysis command-line tool used to display detailed information about ELF (Executable and Linkable Format). Our preliminary results show that AFL+DIAR does not only discover new paths more quickly but also achieves higher coverage overall. This work thus showcases how starting with lean and optimized seeds can lead to faster, more comprehensive fuzzing campaigns -- and DIAR helps you find such seeds. - These are slides of the talk given at IEEE International Conference on Software Testing Verification and Validation Workshop, ICSTW 2022.

GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...

Neo4j

Leonard Jayamohan, Partner & Generative AI Lead, Deloitte This keynote will reveal how Deloitte leverages Neo4j’s graph power for groundbreaking digital twin solutions, achieving a staggering 100x performance boost. Discover the essential role knowledge graphs play in successful generative AI implementations. Plus, get an exclusive look at an innovative Neo4j + Generative AI solution Deloitte is developing in-house.

Climate Impact of Software Testing at Nordic Testing Days

Kari Kakkonen

My slides at Nordic Testing Days 6.6.2024 Climate impact / sustainability of software testing discussed on the talk. ICT and testing must carry their part of global responsibility to help with the climat warming. We can minimize the carbon footprint but we can also have a carbon handprint, a positive impact on the climate. Quality characteristics can be added with sustainability, and then measured continuously. Test environments can be used less, and in smaller scale and on demand. Test techniques can be used in optimizing or minimizing number of tests. Test automation can be used to speed up testing.

GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...

Neo4j

Dr. Sean Tan, Head of Data Science, Changi Airport Group Discover how Changi Airport Group (CAG) leverages graph technologies and generative AI to revolutionize their search capabilities. This session delves into the unique search needs of CAG’s diverse passengers and customers, showcasing how graph data structures enhance the accuracy and relevance of AI-generated search results, mitigating the risk of “hallucinations” and improving the overall customer journey.

Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...

SOFTTECHHUB

The choice of an operating system plays a pivotal role in shaping our computing experience. For decades, Microsoft's Windows has dominated the market, offering a familiar and widely adopted platform for personal and professional use. However, as technological advancements continue to push the boundaries of innovation, alternative operating systems have emerged, challenging the status quo and offering users a fresh perspective on computing. One such alternative that has garnered significant attention and acclaim is Nitrux Linux 3.5.0, a sleek, powerful, and user-friendly Linux distribution that promises to redefine the way we interact with our devices. With its focus on performance, security, and customization, Nitrux Linux presents a compelling case for those seeking to break free from the constraints of proprietary software and embrace the freedom and flexibility of open-source computing.

Free Complete Python - A step towards Data Science

RinaMondal9

PCI PIN Basics Webinar from the Controlcase Team

ControlCase

20240605 QFM017 Machine Intelligence Reading List May 2024

Matthew Sinclair

UiPath Test Automation using UiPath Test Suite series, part 5

DianaGray10

Encryption in Microsoft 365 - ExpertsLive Netherlands 2024

Albert Hoitingh

20240607 QFM018 Elixir Reading List May 2024

Matthew Sinclair

Introduction to CHERI technology - Cybersecurity

mikeeftimakis1

GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024

Neo4j

GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024

Neo4j

Recently uploaded (20)

The Art of the Pitch: WordPress Relationships and Sales

Essentials of Automations: The Art of Triggers and Actions in FME

FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf

Elizabeth Buie - Older adults: Are we really designing for our future selves?

Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf

GraphRAG is All You need? LLM & Knowledge Graph

Removing Uninteresting Bytes in Software Fuzzing

GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...

Climate Impact of Software Testing at Nordic Testing Days

GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...

Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...

Free Complete Python - A step towards Data Science

PCI PIN Basics Webinar from the Controlcase Team

20240605 QFM017 Machine Intelligence Reading List May 2024

UiPath Test Automation using UiPath Test Suite series, part 5

Encryption in Microsoft 365 - ExpertsLive Netherlands 2024

20240607 QFM018 Elixir Reading List May 2024

Introduction to CHERI technology - Cybersecurity

GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024

GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024

The Rising Sun: Making the most of Solr power

1. http://ﬂickr.com/photos/thepma/640151954/ Wednesday, February 25, 2009 1

2. THE RISING SUN: Making the most of Solr power code4lib '09, Providence, RI Erik Hatcher erik.hatcher@lucidimagination.com Wednesday, February 25, 2009 2

3. #9 Performance Measure Wednesday, February 25, 2009 3

4. #8 Memory omitNorms omitTf (1.4) Wednesday, February 25, 2009 4

5. #7 Query Parsing quot;It Dependsquot; More Like This: http://www.lucidimagination.com/blog/2009/02/22/ exploring-query-parsers/ Wednesday, February 25, 2009 5

6. #6 Data Import DIH, Solr Cell, CSV, LuSql, APIs,... solr-ruby Mapper Wednesday, February 25, 2009 6

7. Obligatory XML Wednesday, February 25, 2009 7

8. #5 Request Handlers Leverage Solr's conﬁgurability Wednesday, February 25, 2009 8

9. http://localhost:8983/solr/ timeline.xml Wednesday, February 25, 2009 9

10. #4 Solr as IR toolkit TermVectorRequestHandler StatsComponent Luke Wednesday, February 25, 2009 10

11. #3 LocalSolr geo searching Wednesday, February 25, 2009 11

12. #2 Faceting Solr 1.4 performance! (*bows* to Yonik) Multi-select Wednesday, February 25, 2009 12

13. Multi-select faceting http://www.lucidimagination.com/search/p:lucene,solr/s:email,wiki/l:user?q=marc Wednesday, February 25, 2009 13

14. #1 User Interface The Interface is the App Wednesday, February 25, 2009 14

15. SolrJS Wednesday, February 25, 2009 15

16. Solritas Velocity templated response Wednesday, February 25, 2009 16

17. #0 Community solr-user@lucene.apache.org Wednesday, February 25, 2009 17

18. quot;LucidFindquot; http://www.lucidimagination.com/search ﬁnd@lucidimagination.com Wednesday, February 25, 2009 18

19. Lucid Supports Solr lucidimagination.com Wednesday, February 25, 2009 19

20. Lucid Articles Grant Ingersoll Erik Hatcher Getting Started with Lucene Getting Started with Solr (includes screencast), co- authored with Jonathan Debugging Relevance Issues in Knudsen Search Sami Siren Optimizing Findability in Lucene and Solr Content Extraction with Tika Yonik Seeley Mark Miller Faceted Search with Solr Scaling Lucene and Solr Wednesday, February 25, 2009 20

21. Lucid Podcasts Interviews with: Doug Cutting (creator of Lucene) Ryan McKinley (Solr committer) Chris Hostetter (Solr committer) More coming! Wednesday, February 25, 2009 21

22. Lucid Blogging Mark Miller: quot;Exploring Query Parsersquot;, quot;Highlighting Highlighter Thoughtsquot;, quot;Investigating OOM and other JVM issuesquot;, quot;Looking forward to new features in Solr 1.4quot; Erik Hatcher: quot;acts_as_solr with rich document indexingquot; Grant Ingersoll: quot;Sorting, Faceting, and Schema Design in Solrquot; Wednesday, February 25, 2009 22

23. Pre-release ebook available now! Wednesday, February 25, 2009 23

24. The Good Ol' Blues Brothers Boys Wednesday, February 25, 2009 24

The Rising Sun: Making the most of Solr power

Recommended

Recommended

More Related Content

More from Erik Hatcher

More from Erik Hatcher (20)

Recently uploaded

Recently uploaded (20)

The Rising Sun: Making the most of Solr power