Your SlideShare is downloading. ×
Open source enterprise search and retrieval platform
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Open source enterprise search and retrieval platform

2,903

Published on

This technical talk describes the usage of Apache based open source software (Apache Lucene/SOLR, Apache Nutch, Apache Tika and Apache ServiceMix) within the implementation of an enterprise search and …

This technical talk describes the usage of Apache based open source software (Apache Lucene/SOLR, Apache Nutch, Apache Tika and Apache ServiceMix) within the implementation of an enterprise search and retrieval platform. The platform is the result of years of experience with enterprise search technologies combined with enterprise application integration and semantic (web) technologies, both within commercial and open source based environments. The talk will dive into the conceptual architecture of a typical search solutions based upon a real world use case, and will then present the accompanying framework that makes easy and swift implementations of enterprise search solutions possible, based upon this architecture. The architecture describes an innovative enterprise search solution, specifying all necessary components for collecting and indexing content (known in the architecture as the collection process, which consist of inbound, splitting, validating, filtering, enriching and indexing components) and publishing the content (known in the architecture as the publication process, which consists of inbound, validating, request enriching, searching, grouping, response enriching and presentation components). The framework can be seen as an orchestrator framework and contains all tools, components and default configurations and flow descriptions necessary to build enterprise solutions according to this architecture. The framework is entirely based upon open source technologies and are mainly Apache based

Published in: Technology
0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,903
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
128
Comments
0
Likes
4
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Datum 21 augustus 2010 Enterprise Search EAI Semantic Web Open Source Search & Retrieval Platform Marc Teutelink
  • 2. How Apache open source software is used during the implementation of an Enterprise Search and Retrieval Platform (Lucene/SOLR, Nutch, Tika, ServiceMix/Camel, Felix/Ace)
  • 3. Marc Teutelink marc.teutelink@luminis.eu @mteutelink •Software architect at Luminis •15+ years experience in software development; specialized in Enterprise Search, Enterprise Application Integration and Semantic Web technology •Currently writing “Enterprise Search in Action” for Manning (Mid-2011)
  • 4. Agenda •Enterprise Search • What is Enterprise Search: Functions and features • Challenges • Logical Architecture •Enterprise Search Solution • Technology Stack • Collection Process • Publication Process • Enricher framework • Deployment •Conclusion
  • 5. What is Enterprise Search? “Enterprise Search offers a solution for searching, finding and presenting enterprise related information in the larger sense of the word” Enterprise search is all about searching through documents from any type and format from any sources located anywhere with the upmost flexibility • Web search: limited to public documents on the web • Desktop search: limited to private documents on the local machine • Enterprise search: no limitations on document type and location
  • 6. Enterprise Search (features) •Information Sources and Types • Wide range of sources: local and remote filesystems, content repositories, e-mail, databases, internet, intranet and extranet • Type not limited: any type ranging from structured to unstructured data, text and binary formats and compound formats (zip) •Usage • Not limited to interactive use  automated business processes •Security • Integrations with enterprise security infrastructure •User Interaction and personalization • Identity enables more personalized search results
  • 7. Enterprise Search (features) •Extended metadata • More metadata  better and more precise search results • More control over schema (for example Dynamic Fields) •Ranking • More control over ranking: personalized ranking (group) •Data extraction and derivation • Extract data using various techniques: Xpath, Xquery • Derive data: using external knowledge models: RDBMS, RDF Store, Web Services • Conditional extraction & derivation •Managing and monitoring • On-the-fly management (JMX) • Real time monitoring
  • 8. Enterprise Search (features) •User Interfaces • Web search • All about selling advertisements to the mass • Generalistic & minimalistic screens; focus on adds • Enterprise search • All about finding: rich navigation; focus on quick find • Small targeted audience • Specialized and customized screens (use of ontologies, taxonomies and classifications) • Use of identity (results customized to user) and web 2.0 • Grouping • field collapsing, faceted search & clustering
  • 9. Enterprise Search (Challenges) •Performance and scalability •Rich functions and features •Managebility •Flexibility •Easy maintenance •Quick issue and problem solving •Reduce total cost of ownerschip
  • 10. Enterprise Search (Challenges) •Performance and scalability •Rich functions and features •Managebility •Flexibility •Easy maintenance •Quick issue and problem solving •Reduce total cost of ownerschip Commercial Search Engines?
  • 11. Enterprise Search (Challenges) •Performance and scalability •Rich functions and features •Managebility •Flexibility •Easy maintenance •Quick issue and problem solving •Reduce total cost of ownerschip Apache Based (Open Source) Search & Retrieval Platform
  • 12. Enterprise Search (Logical Architecture) Actors Search Engine Indexing Sources Content Inbound Pull (Crawling) Pull (Harvesting) Push (SOAP/ReST) Searching & Ordering Filtering Grouping Request Inbound HTTP/Get (URL) HTTP/Post (SOAP/ReST) API (Java,Perl,...) Request Validation Syntactic Semantic Request Enrichment Redirection (Suggestions) Enhancement (add/remove clauses) Response Enrichment Redirection (more like this) Enhancement (metadata, editorial) Response Outbound Stateless (XSLT, SolrJS) Statefull (Webapp Framework) Sorting Content Enrichment Extraction Enhancement Filtering Collection Process Publication Process Content Validation SemanticSyntactic
  • 13. Enterprise Search (Collection Process) Sources • Any document format • Any type • Structured and unstructured • Textual and binary • Compound • Residing Anywhere • Security Search Engine Indexing Sources Content Inbound Pull (Crawling) Pull (Harvesting) Push (SOAP/ReST) Content Enrichment Extraction Enhancement Filtering Collection Process Content Validation SemanticSyntactic
  • 14. Enterprise Search (Collection Process) Search Engine Indexing Sources Content Inbound Pull (Crawling) Pull (Harvesting) Push (SOAP/ReST) Content Enrichment Extraction Enhancement Filtering Collection Process Content Validation SemanticSyntactic Content Inbound • Pull (Crawling/Spidering) • Internet, intranet & extranet • Local and remote filesystems • Pull (Harvesting) • Databases • Content Repositories / Mgmt Systems • Webservices inbound • Push • Webservices (SOAP/REST) • Real time indexing
  • 15. Enterprise Search (Collection Process) Search Engine Indexing Sources Content Inbound Pull (Crawling) Pull (Harvesting) Push (SOAP/ReST) Content Enrichment Extraction Enhancement Filtering Collection Process Content Validation SemanticSyntactic Content Validation • Syntactic validation • Based on DTD / XML-Schema • Structure and limited content • Semantic validation • Based on algorithms: • Groovy, XPath, Regex, … • Think about exception handling • Placed anywhere in flow • During inbound: XML-Schema validation • After Enrichment: Validate derived metadata
  • 16. Enterprise Search (Collection Process) Search Engine Indexing Sources Content Inbound Pull (Crawling) Pull (Harvesting) Push (SOAP/ReST) Content Enrichment Extraction Enhancement Filtering Collection Process Content Validation SemanticSyntactic Content Enrichment • Extraction • Metadata • Content (free text of document) • Enhancing • Derive new and alter existing metadata • Filtering • Remove (parts of) metadata • Leverage external knowledge models • Conditional enrichment
  • 17. Enterprise Search (Collection Process) Search Engine Indexing Sources Content Inbound Pull (Crawling) Pull (Harvesting) Push (SOAP/ReST) Content Enrichment Extraction Enhancement Filtering Collection Process Content Validation SemanticSyntactic Indexing • Store in search engine(s) • Content based routing • Document boosting
  • 18. Enterprise Search (Publication Process) Request Inbound • HTTP/Get • URL based with parameters • Response in XML, JSON, … • HTTP/Post • XML (SOAP, REST) request • XML (SOAP, REST) response • API • Java, Perl, … • Wrappers on HTTP/Get Actors Search Engine Searching & Ordering Filtering Grouping Request Inbound HTTP/Get (URL) HTTP/Post (SOAP/ReST) API (Java,Perl,...) Request Validation Syntactic Semantic Request Enrichment Redirection (Suggestions) Enhancement (add/remove clauses) Response Enrichment Redirection (more like this) Enhancement (metadata, editorial) Response Outbound Stateless (XSLT, SolrJS) Statefull (Webapp Framework) Sorting Publication Process
  • 19. Enterprise Search (Publication Process) Actors Search Engine Searching & Ordering Filtering Grouping Request Inbound HTTP/Get (URL) HTTP/Post (SOAP/ReST) API (Java,Perl,...) Request Validation Syntactic Semantic Request Enrichment Redirection (Suggestions) Enhancement (add/remove clauses) Response Enrichment Redirection (more like this) Enhancement (metadata, editorial) Response Outbound Stateless (XSLT, SolrJS) Statefull (Webapp Framework) Sorting Publication Process Request Validation • Syntactic Validation • Correct Query syntax? • Semantic Validation • Correct Field Filters? • Based on algorithms: Groovy, Regex • Placed anywhere in flow • @inbound: XML-Schema validation • @enrichment: Validate derived request clauses
  • 20. Enterprise Search (Publication Process) Actors Search Engine Searching & Ordering Filtering Grouping Request Inbound HTTP/Get (URL) HTTP/Post (SOAP/ReST) API (Java,Perl,...) Request Validation Syntactic Semantic Request Enrichment Redirection (Suggestions) Enhancement (add/remove clauses) Response Enrichment Redirection (more like this) Enhancement (metadata, editorial) Response Outbound Stateless (XSLT, SolrJS) Statefull (Webapp Framework) Sorting Publication Process Request Enrichment • Redirection • Spelling suggestions • Metadata suggestions • Enhancing • Add/Remove clauses • Stemming, Synonyms, stop words
  • 21. Enterprise Search (Publication Process) Actors Search Engine Searching & Ordering Filtering Grouping Request Inbound HTTP/Get (URL) HTTP/Post (SOAP/ReST) API (Java,Perl,...) Request Validation Syntactic Semantic Request Enrichment Redirection (Suggestions) Enhancement (add/remove clauses) Response Enrichment Redirection (more like this) Enhancement (metadata, editorial) Response Outbound Stateless (XSLT, SolrJS) Statefull (Webapp Framework) Sorting Publication Process Searching & Ordering • Filtering • Field Search • Grouping • Add group information • Field collapsing, Faceted Search & Clustering • Sorting • Sort on Field • Ranking
  • 22. Enterprise Search (Publication Process) Actors Search Engine Searching & Ordering Filtering Grouping Request Inbound HTTP/Get (URL) HTTP/Post (SOAP/ReST) API (Java,Perl,...) Request Validation Syntactic Semantic Request Enrichment Redirection (Suggestions) Enhancement (add/remove clauses) Response Enrichment Redirection (more like this) Enhancement (metadata, editorial) Response Outbound Stateless (XSLT, SolrJS) Statefull (Webapp Framework) Sorting Publication Process Response Enrichment • Redirection • Suggestions • More like this • Enhancing • Add/Remove response fields • Schema information • Editorial information
  • 23. Enterprise Search (Publication Process) Actors Search Engine Searching & Ordering Filtering Grouping Request Inbound HTTP/Get (URL) HTTP/Post (SOAP/ReST) API (Java,Perl,...) Request Validation Syntactic Semantic Request Enrichment Redirection (Suggestions) Enhancement (add/remove clauses) Response Enrichment Redirection (more like this) Enhancement (metadata, editorial) Response Outbound Stateless (XSLT, SolrJS) Statefull (Webapp Framework) Sorting Publication Process Response outbound • Stateless • No security • XSLT, SolrJS • Statefull • Security • Web2.0 • Web Application Framework
  • 24. Technology Stack (Collection Process) •Use ESB for the flow: Apache ServiceMix with Camel • Leverage standard ESB components (Transformers, Validation, Splitter, Filter, Routers, Scripting) • Leverage standard ESB transports (WS, SMTP, JMS, JCR, JDBC, FILE) • Custom: Crawler Apache Nutch • Leverage only crawl framework • Extend NutchIndexWriter; asynchronously pushing crawled documents back into ESB flow (reply-to) •ESB Makes distributed flow possibleContent based routing •Hot deploy Easy maintenance •Reusing services across collection processes •Search Engine independent
  • 25. Collection Process Flow Content Indexer Content Inbound 2 1 Documents Message N D Document Messages D D Lucene/Solr INDEX HTTP Transport (Channel Adapter) Lucene/SOLR (SOLRJ) D SOLR Document Message Splitter Channel Content Validation Content Enrichment Enricher Content Filter Content Enricher Syntactic Validation (Channel Purger) Push Inbound (Message Endpoint) Semantic Validation (Channel Purger) Invalid Message Channel ! ? Invalid Message ChannelChannel Transformer (Message Translator)
  • 26. Technology Stack (Publication Process) •Use flow from Apache Lucene/Solr • Leverage standard Solr components (synonyms, stopwords, stemming, MLT, spelling, faceted search, …) • Custom components: using Solr’s extendability framework • Security: authority field in schema with Apache Shiro integration • Field filters (zipcode,…) •User interfaces • Stateless: SolrJs, XSLTResponseWriter & VelocityResponseWriter • Statefull: Apache Wicket with Spring
  • 27. Actors Search Engine Indexing Sources Content Inbound Pull (Crawling) Pull (Harvesting) Push (SOAP/ReST) Searching & Ordering Filtering Grouping Request Inbound HTTP/Get (URL) HTTP/Post (SOAP/ReST) API (Java,Perl,...) Request Validation Syntactic Semantic Request Enrichment Redirection (Suggestions) Enhancement (add/remove clauses) Response Enrichment Redirection (more like this) Enhancement (metadata, editorial) Response Outbound Stateless (XSLT, SolrJS) Statefull (Webapp Framework) Sorting Content Enrichment Extraction Enhancement Filtering Collection Process Publication Process Content Validation SemanticSyntactic Enterprise Search (Logical Architecture)
  • 28. Actors Search Engine Indexing Sources Content Inbound Pull (Crawling) Pull (Harvesting) Push (SOAP/ReST) Searching & Ordering Filtering Grouping Request Inbound HTTP/Get (URL) HTTP/Post (SOAP/ReST) API (Java,Perl,...) Request Validation Syntactic Semantic Request Enrichment Redirection (Suggestions) Enhancement (add/remove clauses) Response Enrichment Redirection (more like this) Enhancement (metadata, editorial) Response Outbound Stateless (XSLT, SolrJS) Statefull (Webapp Framework) Sorting Content Enrichment Extraction Enhancement Filtering Collection Process Publication Process Content Validation SemanticSyntactic Lucene/SOLR ServiceMix/Camel Nutch Apache WicketSolrJS/XSLT Enterprise Search (Logical Architecture)
  • 29. Actors Search Engine Indexing Sources Content Inbound Pull (Crawling) Pull (Harvesting) Push (SOAP/ReST) Searching & Ordering Filtering Grouping Request Inbound HTTP/Get (URL) HTTP/Post (SOAP/ReST) API (Java,Perl,...) Request Validation Syntactic Semantic Request Enrichment Redirection (Suggestions) Enhancement (add/remove clauses) Response Enrichment Redirection (more like this) Enhancement (metadata, editorial) Response Outbound Stateless (XSLT, SolrJS) Statefull (Webapp Framework) Sorting Content Enrichment Extraction Enhancement Filtering Collection Process Publication Process Content Validation SemanticSyntactic Enterprise Search (Logical Architecture) Luminis Enricher Framework
  • 30. Luminis Enricher Framework •Custom Enricher Framework • Existing ESB & SOLR enricher capabilities not sufficient. • Enriching = one or more actions (extraction, enhancing & filtering) performed on documents with fields • Same enricher to be used for: • Collection process: • Documents  enriching, filtering & splitting • Publication process: • Search requests’first-components’ searchcomponent • Search response’last-components’ searchcomponent
  • 31. Luminis Enricher Framework •Custom Enricher Framework • Existing ESB & SOLR enricher capabilities not sufficient. • Enriching = one or more actions (extraction, enhancing & filtering) performed on documents with fields • Same enricher to be used for: • Collection process: • Documents  enriching, filtering & splitting • Publication process: • Search requests’first-components’ searchcomponent • Search response’last-components’ searchcomponent Content Indexer Content Inbound 2 1 Documents Message N D Document Messages D D Lucene/Solr INDEX SOLR Indexer (Channel Adapter) Lucene/SOLR (SOLRJ) D SOLR Document Message Splitter Channel Content Validation Content Enrichment Enricher Content Filter Content Enricher Syntactic Validation (Channel Purger) Push Inbound (Message Endpoint) Semantic Validation (Channel Purger) Invalid Message Channel ! ? Invalid Message ChannelChannel
  • 32. Luminis Enricher Framework •Custom Enricher Framework • Existing ESB & SOLR enricher capabilities not sufficient. • Enriching = one or more actions (extraction, enhancing & filtering) performed on documents with fields • Same enricher to be used for: • Collection process: • Documents  enriching, filtering & splitting • Publication process: • Search requests’first-components’ searchcomponent • Search response’last-components’ searchcomponent Content Indexer Content Inbound 2 1 Documents Message N D Document Messages D D Lucene/Solr INDEX SOLR Indexer (Channel Adapter) Lucene/SOLR (SOLRJ) D SOLR Document Message Splitter Channel Content Validation Content Enrichment Enricher Content Filter Content Enricher Syntactic Validation (Channel Purger) Push Inbound (Message Endpoint) Semantic Validation (Channel Purger) Invalid Message Channel ! ? Invalid Message ChannelChannel <<SearchHandler>> RequestHandler "first-components" "components" "last-components" <<XML>> Response <<SearchComponent>> query <<SearchComponent>> facet <<SearchComponent>> mlt <<SearchComponent>> highlight <<SearchComponent>> stats <<SearchComponent>> debug <<SOLRQueryRequest>> Query <<XSLT>> XML2HTML <<QueryResponseWriter>> XSLTResponseWriter <<(X)HTML>> Resultaat
  • 33. Luminis Enricher Framework (architecture) •Pipe-and-filter architecture • Documents flow through series of actions • Output from one action is input to another action • Fields from input document can be used in action’s clauses: values in expressions filled by replacing velocity type patterns with field values •Conditional flows supported •Reuse of flows & Subflows supported
  • 34. Luminis Enricher Framework (architecture) •Pipe-and-filter architecture • Documents flow through series of actions • Output from one action is input to another action • Fields from input document can be used in action’s clauses: values in expressions filled by replacing velocity type patterns with field values •Conditional flows supported •Reuse of flows & Subflows supported Action (select C where ${B}) Action (remove A2) Document [[A1,A2],[B]] Document [[A1],[B]] Document [[A1],[B],[C1]] If [B=3] YES Action (select C where ${A}) Document [[A1],[B],[C2]] NO
  • 35. Luminis Enricher Framework (Configuration) •Enricher flow and expression configuration via XML based DSL • Conditional: if-then-else & switch-case-else (with regex support) • Actions: Add & remove fields and field values using expressions • Expression handlers currently supported: • Field • Function (execute methods via Java Reflection) • HttpClient (retrieve content by URL described by field values) • Xslt, Xpath, Xquery (external XML databases) • JDBC • SparQL (OpenRDF) • Apache Lucene/Solr • Apache Tika (Meta and Text extraction)
  • 36. Luminis Enricher Framework (Examples) <enricher name="Field" > <field name="a">AA1</field> <field name="b">BB1</field> <field name="b">BB2</field> <multivalue-field name="c">CC1</multivalue-field> <multivalue-field name="c">CC2</multivalue-field> <if test="field::c" pattern="CC2"> <then> <field name="e">EE1</field> </then> </if> <if test="field::a"> <then> <field name="f">FF1</field> </then> </if> <rename-field name="b">d</rename-field> <remove-field name="a"/> </enricher>
  • 37. Luminis Enricher Framework (Examples) <enricher name="Field" > <field name="a">AA1</field> <field name="b">BB1</field> <field name="b">BB2</field> <multivalue-field name="c">CC1</multivalue-field> <multivalue-field name="c">CC2</multivalue-field> <if test="field::c" pattern="CC2"> <then> <field name="e">EE1</field> </then> </if> <if test="field::a"> <then> <field name="f">FF1</field> </then> </if> <rename-field name="b">d</rename-field> <remove-field name="a"/> </enricher> <enricher name="XPath” xmlns:str="http://exslt.org/strings" xmlns:fn="http://www.w3.org/2005/xpath-functions" xmlns:html="http://www.w3.org/1999/xhtml"> field name="Description" expression-type="xpath"> //html:meta[@name='DC.description']/@content </field> <multivalue-field name="Type" expression-type="xpath"> //html:meta[@name='DC.type' and (@scheme='OVERHEIDbm.bekendmakingtypeGemeente' or @scheme='OVERHEIDbm.bekendmakingtypeProvincie' or @scheme='OVERHEIDbm.bekendmakingtypeWaterschap') ]/@content </multivalue-field> <field name="publisher" expression-type="xpath"> fn:string-join(('Blow, ', 'blow, ', 'thou ', 'winter ', 'wind!'), '') </field> <field name="publisher" expression-type="xpath"> fn:concat(//html:meta[@name='OVERHEID.organisationType']/@content, //html:meta[@name='DC.creator']/@content) </field> </enricher>
  • 38. Luminis Enricher Framework (Examples) <enricher name="Field" > <field name="a">AA1</field> <field name="b">BB1</field> <field name="b">BB2</field> <multivalue-field name="c">CC1</multivalue-field> <multivalue-field name="c">CC2</multivalue-field> <if test="field::c" pattern="CC2"> <then> <field name="e">EE1</field> </then> </if> <if test="field::a"> <then> <field name="f">FF1</field> </then> </if> <rename-field name="b">d</rename-field> <remove-field name="a"/> </enricher> <enricher name="XPath” xmlns:str="http://exslt.org/strings" xmlns:fn="http://www.w3.org/2005/xpath-functions" xmlns:html="http://www.w3.org/1999/xhtml"> field name="Description" expression-type="xpath"> //html:meta[@name='DC.description']/@content </field> <multivalue-field name="Type" expression-type="xpath"> //html:meta[@name='DC.type' and (@scheme='OVERHEIDbm.bekendmakingtypeGemeente' or @scheme='OVERHEIDbm.bekendmakingtypeProvincie' or @scheme='OVERHEIDbm.bekendmakingtypeWaterschap') ]/@content </multivalue-field> <field name="publisher" expression-type="xpath"> fn:string-join(('Blow, ', 'blow, ', 'thou ', 'winter ', 'wind!'), '') </field> <field name="publisher" expression-type="xpath"> fn:concat(//html:meta[@name='OVERHEID.organisationType']/@content, //html:meta[@name='DC.creator']/@content) </field> </enricher> <enricher name="SPARQL"> <field name="place">http://www.my.com/#channels</field> <field expression-type="sparql" repository="TESTRDF"> <![CDATA[ PREFIX skos: <http://www.w3.org/2004/02/skos/core#> SELECT ?definition WHERE { ?${place} skos:definition ?definition. } ]]> </field> </enricher>
  • 39. Luminis Enricher Framework (Examples) <enricher name="Field" > <field name="a">AA1</field> <field name="b">BB1</field> <field name="b">BB2</field> <multivalue-field name="c">CC1</multivalue-field> <multivalue-field name="c">CC2</multivalue-field> <if test="field::c" pattern="CC2"> <then> <field name="e">EE1</field> </then> </if> <if test="field::a"> <then> <field name="f">FF1</field> </then> </if> <rename-field name="b">d</rename-field> <remove-field name="a"/> </enricher> <enricher name="XPath” xmlns:str="http://exslt.org/strings" xmlns:fn="http://www.w3.org/2005/xpath-functions" xmlns:html="http://www.w3.org/1999/xhtml"> field name="Description" expression-type="xpath"> //html:meta[@name='DC.description']/@content </field> <multivalue-field name="Type" expression-type="xpath"> //html:meta[@name='DC.type' and (@scheme='OVERHEIDbm.bekendmakingtypeGemeente' or @scheme='OVERHEIDbm.bekendmakingtypeProvincie' or @scheme='OVERHEIDbm.bekendmakingtypeWaterschap') ]/@content </multivalue-field> <field name="publisher" expression-type="xpath"> fn:string-join(('Blow, ', 'blow, ', 'thou ', 'winter ', 'wind!'), '') </field> <field name="publisher" expression-type="xpath"> fn:concat(//html:meta[@name='OVERHEID.organisationType']/@content, //html:meta[@name='DC.creator']/@content) </field> </enricher> <enricher name="SPARQL"> <field name="place">http://www.my.com/#channels</field> <field expression-type="sparql" repository="TESTRDF"> <![CDATA[ PREFIX skos: <http://www.w3.org/2004/02/skos/core#> SELECT ?definition WHERE { ?${place} skos:definition ?definition. } ]]> </field> </enricher> <enricher name=”HttpAndTika"> <field name="content.url"><![CDATA[http://na.apachecon.com/c/acna2010/speakers/501]]></field> <field expression-type=”http" name="content.file">field:content.url</field> <field name="auteur" source="field::content.file">xpath://H1</field> <multivalue-field expression-type=”tika.meta” source="field::content.file”/> <field name=”content" expression-type=”tika.text” source="field::content.file”/> <switch test=”field::content.url <case pattern=".*.rijksweb.nl.*"><field name=”source">Rijksweb</field></case> <case pattern=".*.deventer.nl.*"><field name=”source">Gemeente Deventer</field></case> <case pattern="file:.*"><field name=”source">Locale Harde Schijf</field></case> <else><field name=”source">Overige</field></else> </switch> </enricher>
  • 40. Luminis Enricher Framework (Technology) •Enricher and expresion handlers are Java based OSGi services: • Hot pluggable and updatable • Flow and expression configuration changes no restart • Extendible: New expression handlers immediatly available in actions after installing OSGi bundle •Runs in Apache Felix • Collection Process: ServiceMix contains OSGi container • Publication Process: Custom OSGi loader for Lucene/Solr •Centralized & transactional provisioning (Apache Ace) ‑ Components & Configuration
  • 41. Deployment Architecture <<device>> Slave Publication Server (Slave2) <<Container>> Apache Tomcat Enricher (Luminis) Lucene/SOLR (Apache) Wicket (Apache) <<config>> SOLR::schema.xml <<config>> Luminis:Enricher.xml <<config>> SOLR::solrconfig.xml Felix OSGi (Apache) <<device>> Firewall <<device>> HTTP Load Balancer <<device>> Master Collection Server <<Container>> Apache Tomcat Enricher (Luminis) Nutch (Apache) ServiceMix (Apache) Tika (Apache) Lucene/SOLR (Apache) <<config>> SOLR::solrconfig.xml <<config>> Luminis:Enricher.xml <<config>> SOLR::schema.xml <<config>> servicenix::config.xml OpenRDF <<Data Container>> SQL <<Database>> Knowledge Models <<RDFTripleStore>> Knowledge Models <<HTTP>> <<HTTP>> <<HTTP>> <<JDBC>> <<HTTP>> Felix OSGi (Apache) <<HTTP>> <<HTTP/ReST>> <<HTTP/ReST>> <<device>> Deployment Server Ace (Apache) Felix OSGi (Apache) <<PROVISIONING>> <<JDBC>> <<device>> Slave Publication Server (Slave1) <<Container>> Apache Tomcat Enricher (Luminis) Lucene/SOLR (Apache) Wicket (Apache) <<config>> SOLR::schema.xml <<config>> Luminis:Enricher.xml <<config>> SOLR::solrconfig.xml Felix OSGi (Apache)
  • 42. Conclusions •Enterprise Search Solution is not Google search •Open Source paves the way; misses some ingredients • Useful ingredients: Lucene/Solr, Nutch, Tika, ServiceMix/Camel, Wicket, MySQL, OpenRDF, Felix/Ace • Missing ingredients: Enricher •Interesting developments: • Apache Chemistry (CMIS) • Apache Clerezza • Apache Nutch • Apache Connectors Framework (ManifoldCF)
  • 43. Questions & (answers?) Marc Teutelink marc.teutelink@luminis.eu @mteutelink MEAP December 2010 

×