Key topics when migrating from FAST to Solr, EuroCon 2010

Key topics when Migratng from FAST to Solr By Jan Høydahl cominvent as Apache Lucene EuroCon 05/21/10

Agenda  About Cominvent & Jan Høydahl  Quick overview of FAST ESP  The migraton step by step  Pain points  Q&A Apache Lucene EuroCon 05/21/10

Jan Høydahl: BIO ● Enterprise search consultant since 2000 ● Background in Telecom, Mobile services & sofware development ● Second FAST Global Services engineer ● Founder of Cominvent AS ● Lucid Imaginaton certfed instructor & partner ● FAST Certfed instructor Apache Lucene EuroCon 05/21/10 Logos represent projects I've been involved in, and ™ are © of respectve companies

Cominvent AS: Consultng  Vendor independent search consultng Apache Lucene EuroCon 05/21/10

Cominvent AS: Training  Certfed Solr Training Partner with Lucid Imaginaton  Certfed FAST ESP Training Partner Apache Lucene EuroCon 05/21/10 Photo: fuidpowerzone.com

Solr training Oslo June 1-3 Apache Lucene EuroCon 05/21/10

Assumptons  Decision to migrate to Solr is already done  This is not a "sales talk" for any partcular technology  Basic knowledge of Solr  None or limited knowledge of FAST ESP  Migraton to plain Solr or LucidWorks (LucidWorks Enterprise editon not considered) Apache Lucene EuroCon 05/21/10

Introducton to... ...for Solr people Apache Lucene EuroCon 05/21/10

Security Connectors Apache Lucene EuroCon 05/21/10

Apache Lucene EuroCon 05/21/10

FAST ESP architecture Apache Lucene EuroCon 05/21/10 Source: www.microsof.com

 Very strong & scalable document processing framework Format Language Linguistic Conversion Detection Normalization Entities Custom Taxonomy Sentiment Ontology Plug-in Search Alert PARIS (Reuters) - Venus Williams raced into the second round of the $11.25 million French Open Monday, brushing aside Bianka Lamade, 6-3, 6-3, in 65 minutes. Apache Lucene EuroCon 05/21/10

FAST Document Processors (DP)  DPs transform documents prior to indexing  This is diferent from Solr feld centric analysis  Examples of stages:  Encoding normalizaton, language identfcaton  Text extracton (HTML, PDF, MS Ofce, etc.)  Tokenizaton, lemmatzaton, entty extracton  DPs are chained in pipelines  ESP ships with lots useful DPs and pipelines  Writen in Python, very easy to script new ones Custom Taxonomy Sentiment Ontology Plug-in Apache Lucene EuroCon 05/21/10

Terminology Lucene/Solr FAST Replica Search row Shard Column Facet Navigator Spellcheck Did you mean Update processor Document processor Request Handler Query Transformer (QT) Response Writer Result Processor(RP)/TWM Apache Lucene EuroCon 05/21/10

Terminology Lucene/Solr FAST Schema Index profile Index segment Index partition Lucene IndexWriter/Rdr indexer/fsearch (RTS) ~Multi core ~Multi cluster (Documents receiving same Collection processing) Apache Lucene EuroCon 05/21/10

Important diferences Lucene/Solr FAST Most features query-time Most features index-time Field centric analysis Document centric analysis One language per field Multi lingual fields One Update handler per Format conversion in input type (XML, CSV) document pipeline Slim disk & memory Quite fat disk & memory footprint footprint One Java Web app 15-20 processes Apache Lucene EuroCon 05/21/10

Solr Architecture Thanks to Christan Moen/ATILIKA for graphics Apache Lucene EuroCon 05/21/10

The migraton... Apache Lucene EuroCon 05/21/10

Steps of the migraton  Review current features & architecture  Keep all features? Add new?  Install Solr and do a quick iteraton (1-2 days):  Draf schema.xml & solrconfg.xml  Dump & index some real data  Play around with queries – Solritas is nice here  Design spec covering all migraton areas:  Schema, Content, Feeding & Analysis  Frontends, Querying & API  Admin & Operatonal  Implement :) Apache Lucene EuroCon 05/21/10

Spreadsheet for planning the schema Apache Lucene EuroCon 05/21/10

Migratng index-profle -> Solr schema  ESP index profle -> Solr schema.xml  FAST felds example:  Solr equivalent:  Example: A feld with "tokenize=auto" in FAST → type="text"  Create new <feldType>'s as needed Apache Lucene EuroCon 05/21/10

Product facets & generic felds  With FAST you ofen use «generic1», «generic2» etc to model product facets which may vary between product groups. Front ends need logic to convert. Apache Lucene EuroCon 05/21/10

Product facets & generic felds  With Solr, using dynamic felds, each document can have as many facets you like.  Makes it easy to e.g. Introduce a new «color» facet for cars or a «MegaPixels» facet for digital cameras Apache Lucene EuroCon 05/21/10

Composite felds -> DisMax ReqHandler  FAST uses composite felds to search across multple felds, with weightng defned in Rank Profles  FAST's composite felds & rank profles can be modelled as Solr «DisMax» queries  Set suitable defaults in solrconfg.xml using named requesthandler instances.  In case of many felds & performance issues, use <copyField> to group similarly ranked felds!  Freshness boost, GEO boost etc handled through Functon Queries Apache Lucene EuroCon 05/21/10

Composite felds -> DisMax ReqHandler  Given a FAST composite feld / Rank Profle Apache Lucene EuroCon 05/21/10

Composite felds -> DisMax ReqHandler  This Solr query will do the same, confgureable per query:  qt=dismax  q=oslo  qf=ttle^5.0 teaser^1.5 body^0.1  bf=recip(rord(last_modifed),1,1000,1000) ... ... DisjunctonMaxQuery((teaser:foo^1.5 ||ttle:foo^5.0 ||body:foo^0.1)~0.01) DisjunctonMaxQuery((teaser:foo^1.5 ttle:foo^5.0 body:foo^0.1)~0.01) DisjunctonMaxQuery((teaser:bar^1.5 ||ttle:bar^5.0 ||body:bar^0.1)~0.01) DisjunctonMaxQuery((teaser:bar^1.5 ttle:bar^5.0 body:bar^0.1)~0.01) FunctonQuery(1000.0/(1.0*foat(top(rord(last_modifed))) FunctonQuery(1000.0/(1.0*foat(top(rord(last_modifed))) ... ... Apache Lucene EuroCon 05/21/10

Statc document boosts  FAST uses the «hwboost» feld to add a statc Quality boost to each document.  In Solr, you have more fexibility:  Add a boost to each document <doc boost="10.0">  Add a boost to each feld <feld name="ttle" boost="10.0">  Include any numeric document feld in a BoostFuncton bf=sum(sqrt(popularity)^100.0, statcboost^20.0) bf=sum(sqrt(popularity)^100.0, statcboost^20.0) Apache Lucene EuroCon 05/21/10

Navigator statstcs  FAST navigators provide statstcs metadata (min/max/avg/sum)  Soluton: Use the StatsComponent Apache Lucene EuroCon 05/21/10

Navigator auto-buckets  FAST numeric navigators give auto-bucketng based on  equal-frequency, equal-width, manual  Soluton:  Create a new feld which is pre-computed  Example: Document A has price=200.000, add pricerange="150.000 – 1.299.999"  Or use facet queries (expensive)  Or implement auto-bucketng and contribute the patch :-) Apache Lucene EuroCon 05/21/10

XRANK  FAST has a feature to boost documents satsfying an "XRANK" sub-query with a certain statc boost  In Solr, you can solve most XRANK use cases using FunctonQueries Apache Lucene EuroCon 05/21/10

Scope search  FAST ofers a feld type which holds arbitrary XML  Search in XPath-style: xml:companies:company:and(revenue:>1000, employees:>=100)  Have not found similar feld type in Lucene.  Anyone? Apache Lucene EuroCon 05/21/10

Migratng Connectors  FAST's connectors are many and mature  For simple use cases, consider Solr's DIH:  Supports DB, RSS, Web-services, Local flesystem  Additonally throgh Lucene Connectors Framework:  EMC Documentum, FileNet, JDBC, LiveLink, Patriarch (Memex), Meridio, SharePoint, RSS  New connectors should be writen for LCF -and be submited back to the community :) Apache Lucene EuroCon 05/21/10

Migratng Web Crawler  FAST's crawler is mature, performing & scalable  Solr has no built-in web crawler  Prepare for a lot of extra work migratng crawler  Alternatves:  The Apache Nutch crawler (steep learning curve)  Apache Droids  Heritx + Solr (example in Solr1.4 book)  OpenPipeline has a (very) simple crawler Apache Lucene EuroCon 05/21/10

Migratng document processing  Solr lacks a sophistcated processing pipeline.  Alternatves:  Solr's UpdateProcessorChain for simple pipelines:  Write a Solr UpdateProcessor (in Java, Jython etc, see SOLR-1725)  OpenPipeline for more advanced requirements:  Check out FindWise's talk  Integrated with Solr  LingPipe NamedEnttyExtractor plugin Apache Lucene EuroCon 05/21/10

Document processing examples  Binary documents with metadata  Actual customer request: Enrich library records with PDF content  Use Open Pipeline with Apache Tika processor  Implmenent Tika as an UpdateRequestProcessor (SOLR-1763)  Custom XML using FAST's XMLMapper  DIH's built-in XPath support  XSLT to Solr input XML  Write an new XMLMapper Update Request Handler? Apache Lucene EuroCon 05/21/10

Mult lingual  FAST is state of the art on linguistcs  FAST is language aware, e.g. the ttle feld is "analyzed" depending on detected language  Solr is not language aware  Each feld type has one and only one language  Most common soluton:  One feld type per language: text_no, text_en, text_de  Dynamic felds: <dynamicField name="*_en" type="text_en"..../>  Implement language awareness in applicaton layer (feeding + querying) Apache Lucene EuroCon 05/21/10

Mult lingual – advanced  FAST ships with Lemmatzaton for most languages  Solr ships with Stemming – has limitatons  Solutons for mult lingual needs:  Kstem is tghter. Free with  License 3rd party linguistcs  Example: BasisTech Rosete Linguistc Platorm Lemmatzaton, POS etc.. Apache Lucene EuroCon 05/21/10

Mult lingual – very advanced  FAST allows lemmatzaton by index expansion  This can be useful if your frontend does not know what languages are being queried, as all the word infectons are stored in the index.  There is no soluton for this in Solr today,  Workaround: DisMax query spanning all languages: q=eurocon&qf=text_en^2.0 text_no text_de text_it  Downside: This gets ugly and slow with increasing number of languages Apache Lucene EuroCon 05/21/10

Migratng Front ends / Query  Using a search middleware with Solr support? Lucky you!  If not, consider introducing one now:  Using FAST Java/.NET APIs?  Choose SolrJ or SolrNET/SolrSharp  Query language diferences. &fq= instead of flter()  Solr facets do not require session/state as FAST's Apache Lucene EuroCon 05/21/10

Result views  FAST uses "result-view" and "search profle" to specify what felds to return.  Migrate FAST's «views» into named RequestHandler confgs with all default presets  No need to defne felds to return up-front!, use f=a,b,c... Apache Lucene EuroCon 05/21/10

Operatons  Solr has no central admin-server (untl "SolrCloud")  For GUI installer, use  Multple cores – allows smooth schema upgrade etc.  No built-in query reportng, log analysis or monitoring. But have a look at: Apache Lucene EuroCon 05/21/10

Summary  Many migratons are (quite) straight-forward!  Warning fags  Mult-lingual and advanced linguistcs  Heavy use of Document Processing, including Entty Extracton  Scope search  Other enterprise complexites (security, connectors etc)  Follow a structured process  Quick prototyping  Design spec for each area  Don't forget to analyze logs and measure user satsfacton! Apache Lucene EuroCon 05/21/10

Thank You www.cominvent.com jh@cominvent.com www.twiter.com/cominvent linkedin.com/in/janhoy This presentaton licensed under CC-by-sa license Apache Lucene EuroCon 05/21/10 You must atribute Cominvent with name and link

http://www.cominvent.com	221
http://www.slideshare.net	31
http://www.lmodules.com	4
http://static.slidesharecdn.com	1
http://www.linkedin.com	1

Key topics when migrating from FAST to Solr, EuroCon 2010

by Cominvent AS on May 24, 2010

Accessibility

Categories

Upload Details

Usage Rights

5 Embeds 258

Statistics

Key topics when migrating from FAST to Solr, EuroCon 2010 Presentation Transcript