(ATS4-PLAT05) Accelrys Catalog: A Search Index for AEP


Published on

Accelrys Catalog is a new technology for searching protocol databases and usage logs. It uses the Apache SOLR search engine to index protocol and usage information. As a pro client developer it lets you quickly find example protocols that use a particular component, as a web port user you can find popular protocols in a particular scientific domain, and as an administrator you can search for user protocols that verify compliance with IT policies or to prepare for server migration or upgrade. In this talk we will discuss how to configure and administer the search index, the algorithms employed in the search, and how the different types of end-users can get the most out of this technology.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

(ATS4-PLAT05) Accelrys Catalog: A Search Index for AEP

  1. 1. (ATS4-PLAT05) Accelrys Catalog: A Search Index for AEP Ton van Daelen Sr. Director, Platform Product Management ton.vandaelen@accelrys.com
  2. 2. The information on the roadmap and future software development efforts areintended to outline general product direction and should not be relied on in makinga purchasing decision.
  3. 3. Outline• Search use cases• Deployment architecture• Solr search index• Search syntax• Administration• Demo – Pro client UI – Web UI – Admin UI
  4. 4. Accelrys Catalog Vision Search from Pro Client Administer Examples that use the ‘Http Connector’ component Generate index PilotScript referencing ‘rsplit()’ Update frequency Pro Client – Pers Productivity Protocols using MAO data Admin Catalog Search Canned reports Generate index Security issues Update frequency Bad design Bad documentation Xml log Search from Web Port Recent protocols Popular protocols Protocols searching ‘Corporate Web User Database” Next steps: • Mail Users • Post report
  5. 5. The Size of the Challenge• 10-100 Pro client users• 50-1000 Web users• 1-10 servers• -> 5000+ protocols to be managed
  6. 6. Admin Use Cases …• Bad design practices. Find protocols that: – have shortcuts as copies – have saved checkpoints – store passwords – have components that are owner access only – don’t have top level parameters (Web Port) – have component with absolute file paths• Bad documentation practices. Find protocols that: – don’t have help text (or default help) – have components with missing captions
  7. 7. More Admin Use Cases• General queries. Find protocols: – with components that are deprecated (ad hoc / report) – not run in n days – not changed in n days – by client type (pro client, web port, web service, Notebook, Isentris, …) – with components with GUID x – with SQL components with specific DSN
  8. 8. Introduction to Text Searching• Unstructured or minimally- structured searches – Think “Google” – Keyword-based, non-relational; wide range of user input – Based on lookups using pre-built word (token) indexes
  9. 9. Introduction to Text Searching (cont’d)• Strategies to make searches more effective – Stop word removal: and, the, by, for, of, … – Stemming: startedstart, clusterscluster, etc. – Synonym aliasing: oncology=cancer, MB=megabyte, etc. (supported but only minimally implemented; extensible) – Language-specific document and query processing (support for Asian languages)
  10. 10. Apache Solr• Open source text search server• Part of Apache Software Foundation• Uses and extends Lucene Java search library• Hosted by a web application server• http://lucene.apache.org/solr/
  11. 11. Solr: Under the Hood…• Schema – XML specification of document fields and their types – Specifies how fields are tokenized and processed for indexing• Solr config file – XML specification of query and result set processing rules – E.g. field weights• Optional auxiliary files – Stop words, synonyms, protected words (unstemmed)• Host application container – For AEP this is Tomcat
  12. 12. Tokenization and Filtering• Tokenization options in Solr – Break on whitespace – Break on all non-letter characters – Break on case changes (for CamelCaseTokenization) – Break on character set changes (alphanum/ideographic/katakana)• Additional filters – Lowercase filter: converts all characters to lowercase – CJK bigram filter: outputs adjacent character pairs for Asian languages – Stem filter: applies stemming rules (many language-specific variants)• Field indexing and query processing use same tokenization – Better search results may be obtained by using slightly different analysis for indexing versus querying• See http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters
  13. 13. Customizing Solr
  14. 14. Mapping XMLDB to Solr Documents• XML Database = Component/Protocol Database• For each item in XMLDB, an indexing protocol – reads the item from the database – creates data record properties corresponding to Solr fields – joins in statistics from usage log – converts the data record to a JSON “document” – POSTs the document to Apache/Tomcat/Solr via HTTP• Weighting – Protocol name and description have higher weight – Proximity has higher weight
  15. 15. Some Catalog Fields (defined in schema) • name: protocol or component name • path: location in XMLDB • type: “component” or “protocol” • parameters: names of parameters • author: user who created protocol/component • modifieddate: data protocol/component last changed • runcount: number of times protocol has been run • lastrun: date protocol was last run • uses: list of components used by protocol • alltext: composite field for keyword search
  16. 16. Administration• Configure servers• Specify update interval• Manual rebuild
  17. 17. Configuring Accelrys Catalog• Configuration (admin portal) – AEP servers to index – Indexing schedule• Note – Indexer runs as scheduled service – Indexing takes ~1 to 3 minutes per 1000 XMLDB items – Two index copies; users can continue search while index is rebuilt – Tomcat and Solr automatically installed and launched with Apache
  18. 18. Limitations• Usage info can be incorrect because log file doesn’t store full protocol path (“Protocol 1” !)• No indexing at runtime – it can take a day before index is updated
  19. 19. Demo
  20. 20. Example Queries• MAO type:"Component“ – Any components referencing ‘MAO’• uses:"Xml Reader" -author:Accelrys – Components/protocols that have an xml reader and are not authored by Accelrys• lastrun:[*TO NOW-6MONTH] – Last run at least six months prior• runcount:0 – Never been run
  21. 21. Summary• Accelrys Catalog is powerful search technology built into AEP• Become a beta tester (beta-2)• Plan for 9.0 upgrade now• (ATS4-PLAT10) Planning your deployment for a 64 bit world