Concept Searching ConceptClassifier For SharePoint
Webinar: The How To Guide For Taxonomies In Share Point
1. The How To Guide for Taxonomies in SharePoint 1
2. Project Performance CorporationJill HannemannJill Hanemann is an expert in information architecture, taxonomy design, and portal strategy for clients in the government, media, hospitality, and products industries. Jill is a frequent speaker on topics including taxonomy design, social computing, portal strategy, content management, and web usability. Don MillerDon Miller is a senior executive at ConceptSearching with over 20 years experience in knowledge management. He is a frequent speaker about Records Management and Information Architecture problems and solutions. Don has been a guest speaker at Taxonomy Boot Camp, Management Electronic Records and numerous SharePoint events about information organization and records management. Don works in the Life Science, Energy/Utility, Professional Services, Financial Services and Healthcare verticals but enjoys helping any company with an information or records management problem. Concept Searching • Don Miller • (408) 828-3400 • donm@conceptsearching.com 2
3. Agenda Quick Overview Reconfirm Demo Goals - We are only company with a native integration to 2010 for Term Store Management Show native integration into SharePoint 2010 Term Store for search Show native integration into SharePoint 2010 for Records Management and automatic content type updating Dynamic guided navigation within the search platform Show enterprise Taxonomy Management and auto-classification capabilities Building out new Taxonomies/Term Sets Term Store Management Enterprise Taxonomy Management Project Performance Corporation Concept Searching Summary Concept Searching • Don Miller • (408) 828-3400 • donm@conceptsearching.com
4. 4 Jill Hannemann – PPC Don Miller – Concept Searching February 23, 2011 How to automate Records Management and improve Search with guided navigation in SharePoint 2010 How To Guide for Taxonomies in SharePoint
5. Best Practices for Term Store Development and Managed Metadata Services to drive Business Value with SharePoint 2010 Jill Hannemann Project Performance Corporation 5 How To Guide for Taxonomies in SharePoint
6. 6 Donald T. Miller, VP of Business Development Concept Searching A demonstration of conceptClassifier and taxonomyManager for SharePoint 2010 for Term Store Management A practical demonstration of how to use conceptClassifier for SharePoint 2010 to automatically apply metadata for Records Management, Sensitive Information Removal and provide Guided Navigation to end users while leveraging the native capabilities of the Term Store How To Guide for Taxonomies in SharePoint
7. Agenda About Managed Metadata Service Designing Taxonomy and Metadata for Term Sets Using Keywords as Folksonomy Basic Governance Principles 7 How To Guide for Taxonomies in SharePoint
9. About Managed Metadata Service Managed Metadata Service Metadata taxonomies and terms can be shared across multiple SharePoint site collections Multiple manage metadata services can be created Enables search filtering Two types of terms: Managed terms – pre-defined by an enterprise administrator and may be hierarchical. Surfaced in the "managed metadata" column type Managed keywords – non-hierarchical words or phrases that have been added to SharePoint 2010 items by users (folksonomy) 9 How To Guide for Taxonomies in SharePoint 9
11. About Taxonomy Term Store Term sets – Behave as metadata facets Design a multi-taxonomy structure A facet/term set can have a hierarchical structure beneath it, following the behavior of a taxonomy Keyword term set – flat structure of enterprise keywords, can be leveraged as folksonomy. How to categorize content in a way that is intuitive for users? 11 How To Guide for Taxonomies in SharePoint
13. Faceted Browsing and Searching Faceted classification Assignment of multiple classifications to an object, enabling the classifications to be ordered in multiple ways, rather than in a single, pre-determined, taxonomic order. Faceted Browsing and Searching Uses faceted classification to allow users to explore and find information. Examples: Products by price ranges, brands, and vendors Corporate documents by department and location 13 How To Guide for Taxonomies in SharePoint
14. Faceted Browsing and Searching Find computers by category, weight, screen size, etc. 14 How To Guide for Taxonomies in SharePoint
15. Term Set Structure Group Term Set Parent Term 1 Child Term A Child Term B Parent Term 2 Business Unit Document Types Proposal Document Pink Team Red Team Resume Project Description Forms Presentations 15 How To Guide for Taxonomies in SharePoint
16. Editing Term Sets and Terms Group Term Set Parent Term Child Term 16 How To Guide for Taxonomies in SharePoint 16
18. Initial Planning: Discovering All Your Groups Determine Key Term Sets Define the Term Sets for document libraries Think about audience, business needs, content types Define the values/terms for each metadata field/term set 18 How To Guide for Taxonomies in SharePoint
20. Using Keywords as Folksonomy 20 How To Guide for Taxonomies in SharePoint 20
21. Keywords as Folksonomy “Folk + taxonomy” is a user-generated taxonomy. Collaborative Tagging Social Classification Social Indexing Social Tagging Folksonomy = = = = The practice of collaboratively creating and managing tags to categorize content. As opposed to traditional indexing, metadata is generated not only by experts but also by creators and consumers of the content. Usually, freely chosen keywords are used instead of a controlled vocabulary. Tag Clouds are used to visualize tags of a folksonomy. 21 How To Guide for Taxonomies in SharePoint
24. Taxonomy Governance Term set evolution Creating a new Group Term set creation/deletion Term creation/deletion/modification Governance committee representing Term Steering Committee Content Managers Content Contributors Ensure that the classification distribution and meaning of the Groups and Term Sets is not compromised Considering the volume of content that will be tagged using a Term Set schema, changes to terms should be formally presented as a business case and reviewed by the Steering Committee How To Guide for Taxonomies in SharePoint 24
25. Four Pillars of Governance In the Governance Plan, it is important to define: Establishing a vision Roles and responsibilities Policies and procedures Communications, education, and marketing How To Guide for Taxonomies in SharePoint 25
27. Policies and ProceduresProcedures All primary actions related to the Term Store and the application of terms to content in libraries/lists should be driven by policies and procedures Guidelines will empower Content Managers and Taxonomy Team members to manage the Term Store appropriately Intuitive procedures will ensure the system continues to be developed consistently Simple rules and workflows will reduce “bottlenecks” Determine: Who will have access to Term Store? Who can apply tags to content in libraries? Who can create keywords when tagging content? How To Guide for Taxonomies in SharePoint 27
28. Policies and ProceduresProcedures Basic workflow approval processes should be outlined for all aspects of the Term Store governance How To Guide for Taxonomies in SharePoint 28
29. Keys to Governance Loose vs. Tight How To Guide for Taxonomies in SharePoint 29
30. In Closing… 30 How To Guide for Taxonomies in SharePoint 30
37. Best Practices Define your Use Case Understand how and why you will be using taxonomy and metadata Start Small Select a business unit to begin classification within SharePoint Manage Scope Keep your Audience in Mind Recognize that users may think about and look for information in different ways Define Governance Roles, responsibilities, policies, and procedures Control Depth and Breadth A “flat” taxonomy ensures that users can find information quickly A focused taxonomy ensures that users can easily “digest” the scope of information Make a Long-Term Investment Taxonomy development is an iterative and on-going effort 32 How To Guide for Taxonomies in SharePoint 32
38. Enterprise Class Taxonomy Management and Auto-classification - Leveraging the Term Store for Organizational Metadata to Close Information and Records Management Capability Gaps in SharePoint
51. Enterprise Product Suite: conceptSearch, conceptTaxonomyManager, conceptClassifierConcept Searching • Don Miller • (408) 828-3400 • donm@conceptsearching.com
52.
53.
54. Search will return results based on the concept even if the exact terms are not contained in the document (i.e. ‘coronary artery surgery’, ‘heart surgery’)
55. Metadata can be used by any search engine index or any application/process that uses metadataTriple Baseball Three Heart Organ Center Bypass Highway Avoid Concept Searching • Don Miller • (408) 828-3400 • donm@conceptsearching.com
125. Can we all agree: METADATA IS A MUST HAVE…. Metadata is good and drives business value! How many terms per EMM Term Set/Store Library: X # of terms? 30,000 Terms per Term Set (1 Taxonomy) Tested to 1,000,000 Preferred Terms in an Enterprise Managed Metadata Service Concept Searching • Don Miller • (408) 828-3400 • donm@conceptsearching.com
126. Why do you need a term store management tool? Because Managed Metadata is building a database about your unstructured information and using the Term Store as the tool to organize it and find it with in SharePoint 2010 Legal Reporting Each value could be a metadata value that could be manually applied and used as a taxonomy facet. This is stored as a term with in a term set in 2010 HR Financial 39 How To Guide for Taxonomies in SharePoint
127. However, a manual metadata approach will fail 95%+ of the time Concept Searching • Don Miller • (408) 828-3400 • donm@conceptsearching.com
128.
129. Average return on investment minimum of 38% and runs as high as 600% (IDC)
153. Pulls hierarchy directly from Term Store, therefore updates are immediate and accurate for guided taxonomy navigation in FASTConcept Searching • Don Miller • (408) 828-3400 • donm@conceptsearching.com
154.
155. Enables import of FAST Entities into the conceptClassifier taxonomy manager to fine-tune them with metadata generated from your own content and nomenclature
156. Runs natively as a FAST Pipeline Stage eliminating integration and customization issues
161. Removes documents from search results that are confidential/sensitive through automatic Content Type updating and routing to secure server
162. Automatically tags content with both vocabulary and retention codes and respects SharePoint security that could prevent access to the document once it has been declared a record Concept Searching • Don Miller • (408) 828-3400 • donm@conceptsearching.com
164. Traditional manual approach is subjective, cumbersome and ineffective End user must select values from multiple term sets. Microsoft says to plan for 30,000 values in EMM aka Term Set Values
165. conceptClassifier for SharePoint is the only native Term Store Management tool for 2010 Build term sets/taxonomies here in SharePoint 2010 EMM. Plan for 30,000 values
166. Preferred terms and non preferred terms are managed in Taxonomy Manager TaxonomyManager is an intuitive and elegant to tool to manage how and when term sets are applied within SharePoint 2010 Deep capabilities to build out classification approaches including: standard term, phonetics, metadata, class ID, language, case sensitive, regular expression and boosting
167. An automated approach ensures accurate Records Management and improved Search Content is automatically organized against a normalized vocabulary and can be moved to additional record management libraries or can be managed in place based on metadata
168. conceptClassifier for FAST conceptClassifier for 2010 Product Suite provides intuitive guided navigation for FAST Multi value select with in a controlled vocabulary is the single fastest approach you can provide for end users to get access to the correct content
207. Enables import of FAST Entities into the conceptClassifier taxonomy manager to fine-tune them with metadata generated from your own content and nomenclature
208. Runs natively as a FAST Pipeline Stage eliminating integration and customization issues
213. Removes documents from search results that are confidential/sensitive through automatic Content Type updating and routing to secure server
214. Automatically tags content with both vocabulary and retention codes and respects SharePoint security that could prevent access to the document once it has been declared a record Concept Searching • Don Miller • (408) 828-3400 • donm@conceptsearching.com
232. Cons: high customization costs, increase in end-user labor costs, less end-user productivity, non-standardized application of metadata across enterprise
236. Acknowledge that it is a problem and do nothingwww.conceptsearching.com Concept Searching • Don Miller • (408) 828-3400 • donm@conceptsearching.com
237. Security Metadata, Auto-classification, Taxonomies Drive Business Value Semantic Metadata Tagging Increase Information Retrieval Precision for Search Records Retention Code Tagging Concept Classifier Automatic Content Type Updating Records Management Confidential Secure Data Appropriate Storage & Preservation Tagged for Search Collaboration Portal www.conceptsearching.com Concept Searching • Don Miller • (408) 828-3400 • donm@conceptsearching.com
241. Uses content types derived from metadata to drive individual and group access to data assets using inherent SharePoint Security;
242. Uses content types derived from metadata to drive migration of data assets to proper document libraries where RMS templates are automatically applied to restrict data asset usage.Concept Searching • Don Miller • (408) 828-3400 • donm@conceptsearching.com www.conceptsearching.com
Project Performance Corporation, part of the AEA group, is a management consulting firm integrating cutting edge IT expertise and management consulting for top government and Fortune 500 decision makers. Ms. Tabuchi is an expert in information architecture, taxonomy design, and portal strategy for clients in the government, media, hospitality, and products industries. She has significant experience in the design, maintenance, and content management of Microsoft Office SharePoint Server 2007 and Oracle Web Center portal and related systems. In addition, she possesses expertise in web usability and project strategy. She is a frequent speaker on topics including taxonomy design, social computing, portal strategy, content management, and web usability.
How does the taxonomy get created?Who is involved in it’s creation, maintenance over timeIf changes are necessary, how do we make those changes so the taxonomy evolves to ensure it continues to address user needsWhy is it important – amount of data, amount of users
Steering – ensure it meets needs of organizationTaxonomy Team – primary managers of taxonomy modificationContent Management Team – typically provide insight into what is missing, etc.Technical Team – make sure changes to the system are made properly
Now that we have the roles defined, what are the policies and procedures that ensure the maintenance and integrity of the taxonomy over time?
Traditional search assumes the end user knows what they are looking for, or must enter the ‘right’ combination of words to get the ‘right’ result.Knowledge workers need to identify content in the context of what they are seeking. The fundamental problem with search solutions is that they are based on an index of single words. Yet most queries are expressed in short patterns of words and not single words in isolation – which are highly ambiguous. In the example above, a search engine would identify all the documents that contained the words: triple, heart, bypass instead of documents that contained the concept of ‘triple heart bypass’. Since the concept has been identified, other documents that have related concepts will be identified even if they do not contain that exact phrase. The metadata generation issue is increasingly a growing concern in enterprises. Not only for search but also for records management, compliance, and enterprise content management. A comprehensive approach requires more than syntactic metadata and requiring end users to add rich metadata is haphazard and subjective at best. Since conceptClassifier for SharePoint is no longer restricted to keyword identification, compound term metadata can be automatically generated either when the content is created or ingested. The generation of metadata based on concepts extracts compound terms and keywords from a document or corpus of documents that are highly correlated to a particular concept. By identifying the most significant patterns in any text, these compound terms can then be used to generate non-subjective metadata based on an understanding of conceptual meaning. Compound term processing can address many challenges facing large enterprises and provide many benefits. Identification of concepts within a large corpus of information removes the ambiguity in search, eliminates inconsistent meta-tagging, and automatic classification and taxonomy management based on concept identification simplifies development and on-going maintenance.
It is important to note that metadata, auto-classification, and taxonomies are not applications – the business value of these tools are often integrated with other solutions – such as the offerings of the other participants in this panelLet’s look at where these tools can compliment other solutions and improve business processesCLICK: Migration:With the vast amounts of content - moving all content doesn’t make sense and using valuable resources to identify what should/should not be migrated isn’t a good use of time or moneyBefore the migration you can use these technologies to: Eliminate duplicate documentsIdentify documents that contain confidential or privacy dataIdentify and declare records Identify high value contentSavings: We had one client who needed to manually tag 45K marketing documents and estimated that it would take 6 months will 2 full-time people – with our tools it took 2 weeksCLICK Search:The age old problem is how to get end users to tag content – it’s estimated that less 50% of content is correctly indexed, meta tagged or efficiently searchable – it isn’t about what search engine you useStatistics still claim that end users spend 15% of their time duplicating information, 25% searching, and 40% can’t find what they need to do their jobsAutomatic generation of conceptual metadata removes the end user from the tagging process HUMANS WON’T TAG CONTENT THROUGH FORMS, PICKLISTS, DROP DOWNS BUT WE WILL ALWAYS FIND WAYS TO AVOID TAGGINGContent, once tagged can be provided to any search engine index to deliver more accurate search resultsUsing the taxonomy users can more efficiently find relevant information via the hierarchical structure Savings: 2.5 hours per day per userCLICK Records Management:The problem cited most frequently is inconsistent end user tagging in the declaration of recordsWith metadata generation and a taxonomy that mirrors the file plan – documents can be automatically declared records based on the concepts and descriptors within the documentBased on custom Content Types in SharePoint the document can be declared a record and routed to the RM repository Savings: $4 - $7.04 per document recordCLICK Data Privacy ProtectionTaxonomy(s) can be created to identify any organizationally defined confidential information When content is created or ingested the document can be identified as containing confidential information and using Content Type updating the document can be routed to a secure location and locked down using Windows Rights ManagementCost Avoidance: Average cost of a data exposure is $225K - $35 million
The Only Microsoft Solution that Runs Natively in ... FAST Search, SharePoint 2007, 2010, Windows Server R2 FCI, and Microsoft Office conceptClassifier provides the tools to rapidly build and easily manage unstructured content. Providing automatic conceptual metadata generation, automated classification and taxonomy management organizations can harness the power of content to not only improve findability within the FAST Search product suite, but drive additional business processes such as records management, compliance, and enforce governance. The Only FAST Search Solution that ... Automatically Generates Conceptual Metadata Utilizing our unique concept identification and extraction capabilities, conceptClassifier’s statistical engine can identify out-of-the box all the meaningful concepts resident within an organization’s own information repositories and automatically generate semantic metadata that is unique to organization and their nomenclature. The ability to automatically generate conceptual multi-word term metadata and placing those terms in the FAST Search index, the search can be performed with a higher degree of accuracy because the ambiguity inherent in single words is no longer a problem. Utilizing the Concept Searching technology framework, end users can now search on concepts, delivering a multi-dimensional view of relevant information and easily identify the relationships between content assets that otherwise may not have been found. The Only FAST Search Solution that ... Eliminates Manual Metadata Tagging The Only FAST Search Solution that... Delivers Innovative, Intuitive, & Rapidly Deployed Taxonomy Management Managed by Business Users
The Only Microsoft Solution that Runs Natively in ... FAST Search, SharePoint 2007, 2010, Windows Server R2 FCI, and Microsoft Office conceptClassifier provides the tools to rapidly build and easily manage unstructured content. Providing automatic conceptual metadata generation, automated classification and taxonomy management organizations can harness the power of content to not only improve findability within the FAST Search product suite, but drive additional business processes such as records management, compliance, and enforce governance. The Only FAST Search Solution that ... Automatically Generates Conceptual Metadata Utilizing our unique concept identification and extraction capabilities, conceptClassifier’s statistical engine can identify out-of-the box all the meaningful concepts resident within an organization’s own information repositories and automatically generate semantic metadata that is unique to organization and their nomenclature. The ability to automatically generate conceptual multi-word term metadata and placing those terms in the FAST Search index, the search can be performed with a higher degree of accuracy because the ambiguity inherent in single words is no longer a problem. Utilizing the Concept Searching technology framework, end users can now search on concepts, delivering a multi-dimensional view of relevant information and easily identify the relationships between content assets that otherwise may not have been found. The Only FAST Search Solution that ... Eliminates Manual Metadata Tagging The Only FAST Search Solution that... Delivers Innovative, Intuitive, & Rapidly Deployed Taxonomy Management Managed by Business Users
BY ADDRESSING THE TECHNOLOGY AND PROCESS INSTEAD OF THE HUMAN BEHAVIOR ORGANIZATIONS CAN IMPROVE SEARCH OUTCOMES, BRING ABOUT COMPLIANCE WITH INFORMATION AND RECORDS MANAGEMENT POLICIES AND DECREASE POTENTIAL DATA EXPOSURE EVENTS.IN THIS SCENARIO THE CLIENT IS USING SHAREPOINT (BUT IT CAN BE ANY REPOSITORY) THE END USER SIMPLY LOADS A DOCUMENT OR SET OF DOCUMENTS INTO SHAREPOINTCONCEPTCLASSIFIER AUTOMATICALLY APPLIES CONCEPTUAL METADATA FOUND WITHIN THE DOCUMENT SO IT CAN BE USED TO IMPROVE SEARCH, WHERE APPROPRIATE THE CORRECT CONTENT TYPE IS APPLIED TO ENABLE WINDOWS RIGHTS MANAGEMENT, KICK OFF WORKFLOWS, AND APPLY RECORDS RETENTION CODES FOR STORAGE AND PRESERVATION.FOR CONTENT RESIDING IN ARCHIVE AND BACKUP SYSTEMS CONCEPTCLASSIFIER APPLIES THE SAME CATEGORIES OF METADATA WITHOUT ANY END-USER INTERVENTION.
On this slide we content enters the MOSS environment either from multiple sources. Once in that environment and event handler triggers conceptClassifier for SharePoint to apply metadata to each data asset based upon the organizational metadata environment maintained in Taxonomy Manager. The metadata environment maintained in Taxonomy manager can include both organizational created metadata and third party metadata that has been aligned to organizational functions, data privacy and security guidelines, and records retention codes. Once metadata has been applied by conceptClassifier for SharePoint Custom Content Types that have been aligned to specific metadata tags are automatically applied. These two critical steps (automatic application of both metadata and Custom Content Types) provide the following value to organizations: Organizations can discern in real time “what is a document” and “what is a record” and can immediately take action relating to ensuring that declared records are stored in the right location and preserved for the correct period of time; Individual and group access permissions can be automatically applied to data assets/documents based on Custom Content Type; and, Data assets/documents can be automatically migrated to the appropriate document library for the automatic application of Windows Rights Management services to control data usage (i.e. can be viewed but not downloaded, e-mailed, printed, etc…).By automating the metadata and content type application processes organizations not only provide transparency and findability for their end-users, they can also ensure that appropriate document access permissions are applied across the enterprise while also controlling how documents are by end-users who have been granted access. In summary – Concept Searching’s enabling technology improves findability and reduces costs by enabling automated compliance with organizational e-Discovery, Records Management, and Information Management (data privacy and security) guidelines.
On this slide data assets and documents that have been automatically tagged with metadata and custom content types have been automatically migrated to document libraries based on their custom content type. Based on organizational data privacy and security guidelines document libraries containing sensitive information are now only accessible to certain individuals and groups and each library contains content where Windows Rights Management services have been applied to control how the documents in a particular library are used. When these documents are “checked-out” a key is issued that allows the end-user to access and use the data asset/document in a manner that has been pre-approved by the organization. This of course occurs after the Active Directory-Rights Management Services (AD-RMS) database and server already communicated with the MOSS farm and both publishing and user license credentials have been established and provisioned.For individuals who may attempt to access the SharePoint Content Database directly and by-pass AD-RMS protected content the linking of Custom Content Types to SharePoint Security Services prevents un-authorized users “back door” access to organizational content.