EMC Bay Area Momentum User Group 2011: Fast to xPlore, Andy Tran


Published on

EMC Bay Area Momentum User Group 2011: Fast to xPlore, Andy Tran

For more information visit:: https://community.emc.com/docs/DOC-10168

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Script:Why are we doing this?First, The new Documentum search functionality has been built from the ground up to improve search functionality and scalability over our prior Verity and FAST integrations. This search feature combines the high performance of EMC xDB with the innovative and trusted search capabilities of Apache Lucene -- providing comprehensive enterprise-ready search capabilities for content and case management solutions. It is significantly better than our currently integrated FAST Instream search.These enterprise ready capabilities include:Easy deployment and configurationScalable architecture and advanced data management capabilities that allow for easy and cost effective solutions for large deployments.Powerful administration and diagnostic capabilities available through xPlore Administrator UI.Support for VMware virtualization environmentsComprehensive options for High Availability and Disaster RecoveryIntegrated Documentum Security for improved query response timeDrill-down, native facet computation with security enforcementExtensive built-in metrics and analytics Support for NAS based disk technologies in addition to SAN/local
  • xPlore leverages best-of-breed technology for all of its components (xDB XML database, Lucene for base full text indexing, BASIS for linguistic analysis, and Stellent for content extraction). Lucene: Provides the index core for xPlore and is the most popular and advanced open source full text technology in Industry. It provides the basic inverted index technology for xPlore. xDB : This native XML database and the data foundation for xPlore. xDB is a full database providing storage services, concurrency control, and query optimization. Lucene has been integrated into xDB as a “multi-path” index and supports xDB’s transactional semantics. Unlike a relational database, an XML data easily handles flexible changing data models. Lucene backs this up with an ability to optimize queries over these ever changing models. Both of these combined provide a strong foundation to service Documentum’s extensible object models. In addition, the XML database provides a clean set of data services leveraged by xPlore. Most search engines rely on a smorgasbord of unconnected internal databases. This limits their ability to interconnect and relate data.Basis: Core linguistic analysis services for xPlore are provided by Basis technology. Basis is the leading vendor in this area supporting 80+ languages. It is used by many of the major search engine technologies. The breadth and depth of its offering is far above any open source alternative and so it significantly complements the services of xDB and Lucene.Stellent: A leading vendor in the content extraction space. Again, this technology is used by many leading search vendors.
  • Script: Customers using FAST Instream will be migrated to Documentum xPlore. To ease migration, Documentum xPlore is compatible with Content Server version: 6.5 SP2, 6.5 SP3 and 6.6 This allows a window where customers can continue to work with both FAST Instream and Documentum xPlore.For example, a customer on CS 6.5 SP2 may upgrade to Documentum xPlore and begin migration while still serving their clients with FAST InstreamCustomers who have built clients on 6.5 SP2 will be able to continue to use those clients with Documentum xPlore Evident from the roadmap, EMC has evolved to a more organic approach to enterprise search solutions - giving us the flexibility to implement critical large enterprise features For more information on xPlore and the migration process (including a migration checklist), please visit Powerlink.
  • If you use VQL, you will need to convert the corresponding VQL to Xquery. We will provide documentation and examples on how to do this.
  • Although xPlore does not typically require more hardware than FAST, there are a few situations in which you will require new hardware:You are planning to roll out xPlore in “Dual Mode”. In this case, your current FAST box will still be used for FAST until you complete your transition to xPlore. In this scenario, you will need hardware (possibly temporarily) until you complete your transition to xPlore.If your index and query needs have changed since you first rolled out FAST, your current hardware may be undersized. Often times, customers have seen their indices grow tremendously since they originally sized the hardware for FAST. If this is true, you need to ensure that your xPlore hardware is sized to handle your growing capacity.Virtual deployments – if you plan to leverage xPlore on Vmware, you need to ensure you have the appropriately sized hardware to support your xPlore deploymentYou do not currently have FAST deployed on Windows/Linux. Since xPlore only supports Windows/Linux, you need to secure appropriate hardware to support Windows or Linux.
  • Work Package 2 – Current Full-Text Environment AnalysisActivities:Understand the existing Documentum deploymentAnalyze the existing Documentum implementationGather existing requirements related to search & retrieval SLAGather any hardware requirementsAnalyze the content types and object counts which need to be indexedDetermine any application changes required for the existing Implementation to support xPloreGather any specific use cases Outputs:Requirements Document for Search functionalityConfiguration and Customization (if required) Analysis document.Work Package 3 – xPlore System Sizing and hardware recommendationsActivities:Determine deployment strategyDetermine the sizing and environment for the new xPlore systemDefine the Migration PathOutputs:Solution Design & Decisions documentXPlore Sizing DocumentActivities:Install any necessary plugins on the existing Documentum ImplementationInstall xPloreConfigure xPloreInstall and configure Index AgentStart IndexingTest SearchOutputs:xPlore deployment and re-index in Development EnvironmentInstallation Document
  • EMC Bay Area Momentum User Group 2011: Fast to xPlore, Andy Tran

    1. 1. FAST to xPlore Andy Tran Staff Systems Engineer Information Intelligence Group Aug 11, 2011 Bay Area Momentum User GroupInformation Intelligence Group
    2. 2. Agenda • FAST to xPlore? Why? • What is xPlore? • How do I migrate? • Information ResourcesInformation Intelligence Group
    3. 3. The FAST Instream wind-down • Documentum FAST integration has been in maintenance mode for some time • Microsoft ending FAST’s OEM offering (in general) – By end-of-year 2011 support will end for this version of Search within Documentum – -> FAST Instream 4.3.1 • Documentum 6.5 SP3, 6.6 will be the last releases that supports FAST – Documentum 6.7 Content Server compatible with 6.6 FAST but this configuration is highly discouraged • No new features or additional format support will be available for FAST integration  No extended support available for FASTInformation Intelligence Group EMC Partner and Internal Use Only
    4. 4. Documentum xPlore Enhances Content Server Powerful New Search Feature Improves Capability and Stability • Deployment ease • Scalable architecture • Powerful Administrator UI • Support for VMware • Comprehensive HA and DR • Integrated security • Native facet computation • Extensive metrics and analytics • Support for NAS and SANInformation Intelligence Group EMC Internal Use Only
    5. 5. Agenda • FAST to xPlore? Why? • What is xPlore? • How do I migrate? • Information ResourcesInformation Intelligence Group
    6. 6. Documentum xPlore Built from the ground up • Bring best of breed XML Database with powerful Apache Lucene Fulltext Engine • Provides structured and unstructured xPlore API search leveraging XML and XQuery Indexing Search standards Services Services Node/Data • Designed with Enterprise readiness, CPS Mgmt scalability and ingestion Admin Analytics Services • Advanced Data Management functionality necessary for large scale xDB API systems xDB query process & optimization • Industry leading linguistic technology xDB transaction, and comprehensive format filters index & page mgmt • Metrics and AnalyticsInformation Intelligence Group
    7. 7. xPlore Functional Overview • xPlore 1.0 supports Quality of Search: Language Support English, German, • xPlore leverages industry leading linguistic Spanish analysis technology that supports over 80+ languages – We will still be phasing in languages over releases • xPlore 1.1 adds as we are able to certify them French, Italian, Spanish, Japanese & • xPlore uses lemmatization to help improve retrieval Korean – Searches for “run” will find documents with “running” & “runs” – xPlore stores original form & lemma for each token to ensure at least exact match capability for all languages • Language identification done for metadata and content separately – i.e. meta data is Chinese, content is English – Meta data fields used for language id are configurable – “Default language” can be configured as fallback when language is not supportedInformation Intelligence Group
    8. 8. xPlore Functional Overview Document Format Support • xPlore supports over 150 unique file types – 800+ versions of formats (i.e. Word 2003, Word 2010) • Certain formats can be turned off for indexing if desired by modifying the dm_format object and setting can_index to ‘False’ – Meta data will still be indexedInformation Intelligence Group
    9. 9. Better Search through xPlore: Benefits for Existing Clients Existing Client Apps Benefit from xPlore Search Improvements No CHANGES REQUIRED Underprivileged user – xPlore leverages Native Security to greatly improve underprivileged user queries as only rows that meet security rules returned to Content Server – Queries that took minutes, now return in seconds Folder Descend – xPlore optimized for queries on deep folder structures – Comparison for search in structure with thousands of folders (time-to- display page of 200 results): xPlore = 0.97 secs FAST = 44.00 secsInformation Intelligence Group
    10. 10. xPlore Functional Overview xPlore Administrator • Web based Administrative UI for Documentum xPlore • Greatly simplifies day to day configuration and diagnostics of Documentum xPlore • Allows for administration of either single node or multinode deployments • Exposes power of metrics and auditing information via the reporting componentInformation Intelligence Group
    11. 11. xPlore Functional Overview xPlore Administrator: Reporting Mechanism Example Top N Slowest Queries • Leveraging metrics and audit information in xPlore, this report returns you the slowest queries for any time periodInformation Intelligence Group
    12. 12. Documentum xPlore Availability • xPlore in Controlled Release since July 2010 – Over 32 “Highly Qualified” Customers participating (highest ever for a CR Program) • Documentum xPlore 1.0 GA release Q4 (Oct) 2010 – Available independent of Documentum Stack Release – Support English, German & Chinese languages • Documentum xPlore 1.1 bundled with 6.7 Release (Q1 ‘2011) – Adds Support for French, Italian, Spanish, Japanese & Korean languages Part of Content Server: No additional license requiredInformation Intelligence Group EMC Partner and Internal Use Only
    13. 13. Agenda • FAST to xPlore? Why? • What is xPlore? • How do I migrate? • Information ResourcesInformation Intelligence Group
    14. 14. FAST to xPlore Transition Period Both FAST & Documentum xPlore Available FAST xPlore 6.5 SP1 6.5 SP2 6.5 SP3 6.6 6.7 • Transition Period – 2011 will be critical transition period as both FAST & xPlore will be supported on 6.5 SP2, 6.5 SP3 & 6.6 – Support of both engines allows customers to take advantage of Dual Mode Migrations • Platform Support – xPlore supports Windows/Linux only (Physical or Virtual) (32-bit/64-bit) – Content Server can be on all supported OS (Windows, Linux, Solaris, HP, AIX)Information Intelligence Group EMC Partner and Internal Use Only
    15. 15. Preparing to Migrate Migration Checklist Task Status Commit to Migrate ✔ Identify and Mitigate Risks Ensure Content Servers are 6.5 SP2 or later prior to Migration Define and Plan HA/DR Strategy for xPlore Determine Migration Strategy Secure Hardware & Staffing Resources Validate xPlore in Test Environment Put xPlore into ProductionInformation Intelligence Group EMC Internal Use Only
    16. 16. Mitigating Risks: xPlore Compatibility Lowers Migration Risks 6.5 SP2+ Content Server is “xPlore Enabled” – Simply plugin xPlore underneath 6.5 SP2+ allows Dual Mode indexing – Minor changes to Fulltext Query Plugin interface to identify underlying FT engine capabilities xPlore is compatible with existing clients – Existing Webtop, CenterStage, Desktop Client applications with work seamlessly with xPlore FAST – Only newer clients, however, will leverage new xPlore features (i.e Facets) xPlore Existing Query Language (FTDQL & SDC) work seamlessly with xPlore – Queries do not have to be changed to run against xPlore (with exception of VQL)Information Intelligence Group EMC Internal Use Only
    17. 17. Migration: High Availability & Disaster Recovery High Availability • FAST – only Active/Active HA with duplicate systems and data • xPlore has multiple options for High Availability – Active/Active dual system – Active/Passive with Clusters – N+1 Sparing Disaster Recovery • Hot (online) Backup • Domain, Node or Collection Level Backup/Restore • Integrated in xPlore Administrator but also available via APIs for integration into Enterprise Backup Systems (e.g. Networker)Information Intelligence Group
    18. 18. Migration Considerations Migration Methodology Two ways to migrate to Documentum xPlore: Straight Migration – Shutdown FAST – Install, configure and reindex with xPlore – Requires scheduled downtime to switch from FAST to xPlore Dual Mode Migration – Allows for FAST and xPlore to be available on the same repository for indexing & querying – Can slowly migrate users over from FAST to xPlore and eventually turn off FAST – Minimizes disruption to business and can be done without any downtime – Requires additional hardware (temporary)Information Intelligence Group
    19. 19. Determine Migration Strategy: Dual Mode Indexing FAST Index Agent Index Seamless Server – Existing 6.5 SP2+ repository with FAST untouched – Support both FAST and xPlore indices with the same repository Rolling Upgrade / Easy Content Transition Search User Server – Minimizes risk & disruption to business units Repository – Both FAST & xPlore Indices available for indexing & querying – Slowly migrate users from FAST to xPlore – Once all users migrated, turn FAST off Content Server Hardware Considerations Search User – Need to allocate additional hardware for xPlore until transition is complete – 2nd Content Server can run on same host as primary Index Agent xPloreInformation Intelligence Group EMC Internal Use Only
    20. 20. Migration: Securing Hardware and Staffing Resources • Hardware – Based on deployment strategy (physical vs virtual, HA, etc.) and use case requirements, size system for xPlore appropriately – Please refer to “Documentum Performance, Scalability, and Sizing - Part 2” session by Ed Bueché for more sizing details • Staffing – Make sure you budget for any staffing of migration effort for 2011Information Intelligence Group EMC Internal Use Only
    21. 21. Migration: Validate xPlore in Test Environment • Validate that issues seen with FAST have been resolved with xPlore – Pay close attention to areas you’ve historically had problems with Fulltext • Validate your sizing estimates in test • Test out your migration strategyInformation Intelligence Group EMC Internal Use Only
    22. 22. xPlore Readiness: Beta Program “I found it very easy and straight-forward. The Index Agent was infinitely easier to • Beta Program completed configure than in 5.3.” at end of April “Installers worked GREAT. Very – Extended to 6 months to simple! Very pleased with the install/configuration; well ensure exhaustive testing done!.” “I think the xPlore Admin UI is • Over 12 customers & the single best feature of xPlore, partners in beta program for me to trace a doc from docbase to xdb is great.” – Represent wide variety of use cases & deployment “Within the first few months of the scenarios program we knew xPlore was the way to go moving forward”Information Intelligence Group EMC Internal Use Only
    23. 23. Agenda • FAST to xPlore? Why? • What is xPlore? • How do I migrate? • Information ResourcesInformation Intelligence Group
    24. 24. Resources Posted to Powerlink Home > Products > Software D > Documentum Platform > Documentum xPlore/FAST Migration • Powerful New Documentum xPlore to Replace FAST Instream Search - Announcement • FAQ • xPlore Performance Benchmarks • White Paper: EMC Documentum xPlore Migration and Implementation—Best Practices Planning • Documentum Introduces Powerful New Search - Video • Migrating to New Search Documentum xPlore - VideoInformation Intelligence Group
    25. 25. EMC Community Website https://community.emc.com/docs/DOC-8945Information Intelligence Group
    26. 26. Beyond the Migration: Training • xPlore Administration (2 days) – Install, configure, and maintain a Documentum xPlore full-text Index Server and Index Agent – Migrate indexes from repositories presently indexed by FAST InStream • Available in Class room, Live-online, and Web-based • To register, go to http://mylearn.emc.comInformation Intelligence Group EMC Partner and Internal Use Only
    27. 27. We can accelerate migration and lower risks Short on experienced internal resources or bandwidth?  Application impact planning  Production sizing Complex FAST deployment?  Planning for high availability and disaster SI installed original FAST recovery environment?  Migration planning  Hardware and software setup FAST to xPlore Migration Service  Pre-production testing • Standard Approach  User acceptance testing  Production cutover • Culmination of our best practices • Part 1: Content migration & Initial user testing • Proven methods and insights • Part 2: User cutover and FAST cutoff – Beta program – Controlled release programInformation Intelligence Group EMC Partner and Internal Use Only
    28. 28. THANK YOU© Copyright 2011 EMC Corporation. All rights reserved. 28