Downloaded 202 times
![[email_address]](https://image.slidesharecdn.com/apacheconus2008tikapaolomottadelli-1225998762707619-8/75/Content-analysis-for-ECM-with-Apache-Tika-2-2048.jpg)
![Q & A [email_address]](https://image.slidesharecdn.com/apacheconus2008tikapaolomottadelli-1225998762707619-8/75/Content-analysis-for-ECM-with-Apache-Tika-71-2048.jpg)
Uploaded byPaolo Mottadelli
Content analysis for ECM with Apache Tika
Tika is a toolkit for extracting metadata and text from various document formats. It allows developers to parse documents and extract metadata and text content in 3 main steps. Tika shields systems like Alfresco from needing to integrate many individual parsing components. Alfresco uses Tika to index content from various formats by passing file streams through Tika's parsers rather than using multiple custom transformers.
More Related Content
Content analysis for ECM with Apache Tika
- 1. Content analysis forECM with Apache Tika Paolo Mottadelli -
- 2.
- 3.
- 4.
- 5.
- 6.
- 7. A real worldchallenge ? ? ? Searching .docx .xlsx .pptx in Alfresco ECM
- 8.
- 9. What is Tika?Another Indian Lucene project? No.
- 10. What is Tika?It is a Toolkit
- 11.
- 12. A brief historyof Tika Sponsored by the Apache Lucene PMC
- 13. Tika organization Changingafter graduation
- 14. Getting Tika … and contributing
- 15.
- 16. The Parser interfacevoid parse(InputStream stream, ContentHandler handler, Metadata metadata) throws IOException, SAXException, TikaException;
- 17.
- 18.
- 19.
- 20. XHTML SAX events<html xmlns="http://www.w3.org/1999/xhtml"> <head> <title>...</title> </head> <body> ... </body> </html>
- 21. Why XHTML? Reflectthe structured text content of the document Not recreating the low level details For low level details use low level parser libs
- 22. ContentHandler (CH) andDecorators (CHD)
- 23.
- 24.
- 25. … moremetadata: HPSF
- 26.
- 27.
- 28. The AutoDetectParser Encapsulatesall Tika functionalities Can handle any type of document
- 29. Type Detection MimeTypetype = types.getMimeType(…);
- 30. tika-mimetypes.xml An example:Gzip <mime-type type="application/x-gzip"> <magic priority="40"> <match value="\037\213" type="string“ offset="0" /> </magic> <glob pattern="*.tgz" /> <glob pattern="*.gz" /> <glob pattern="*-gz" /> </mime-type>
- 31.
- 32. A really simpleexample InputStream input = MyTest.class.getResourceAsStream("testPPT.ppt"); Metadata metadata = new Metadata(); ContentHandler handler = new BodyContentHandler(); new OfficeParser ().parse(input, handler, metadata); String contentType = metadata.get(Metadata. CONTENT_TYPE) ; String title= metadata.get(Metadata. TITLE) ; String content = handler.toString() ;
- 33.
- 34.
- 35.
- 36.
- 37. ECM: what isit?
- 38. ECM: Manage IndexingCategorization * *
- 39. ECM: we loveSEARCHING!
- 40. ECM: we loveSEARCHING!
- 41. ECM: we loveSEARCHING!
- 42. Don’t do iton your own Tika shields ECM from using many single components
- 43.
- 44.
- 45.
- 46.
- 47. Alfresco Repository JSR-170Level2 Compatible
- 48. Repository Architecture ServicesComponents Storage Hibernate Content Lucene Content Index Database Search Node Node Content Query Index
- 49. Repository Architecture ServicesComponents Storage Hibernate Content Lucene Content Index Database Search Node Node Content Query Index
- 50.
- 51.
- 52.
- 53.
- 54.
- 55.
- 56. Step 2 for(ContentTransformer transformer : transformers) { long transformationTime = transformer.getTransformationTime(); if (bestTransformer == null || transformationTime < bestTime) { bestTransformer = transformer; bestTime = transformationTime; } } return bestTransformer; ContentTransformerRegistry Provides the most appropriate ContentTransformer
- 57. Step 2 (explained)Too many different ContentTransformer implementations
- 58. Step 3 Transformpublic void transformInternal(ContentReader reader, ContentWriter writer, TransformationOptions options) throws Exception { ... HSSFWorkbook workbook = new HSSFWorkbook(is); ... for (int i = 0; i < sheetCount; i++) { HSSFSheet sheet = workbook.getSheetAt(i); String sheetName = workbook.getSheetName(i); writeSheet(os, sheet, encoding); } ... } Example: PoiHssfContentTransformer
- 59. Step 3 (explained)Too many different ContentTransformer implementations ... again !?!
- 60. Step 4 Luceneindex creation ContentReader reader = contentService.getReader(nodeRef, propertyName); ContentTransformer transformer = contentService.getTransformer(reader.getMimetype(), MimetypeMap. MIMETYPE_TEXT_PLAIN); transformer.transform(reader, writer); reader = writer.getReader(); . . . . . . . . doc.add(new Field(attributeName, reader, Field.TermVector. NO));
- 61. Let’s do itusing Tika
- 62. Step 1 +Step 2 + Step 3 String name = “resource.doc” InputStream input = getResourceAsStream(name); Metadata metadata = new Metadata(); ContentHandler handler = new BodyContentHandler(); new AutoDetectParser().parse(input, handler, metadata); String title = metadata.get(Metadata. TITLE); String content = handler.toString();
- 63. Step 1 to4 (compressed) String name = “resource.doc” InputStream input = getResourceAsStream(name); Reader reader = new ParsingReader (input, name); . . . . . . doc.add(new Field(attributeName, reader , Field.TermVector. NO));
- 64.
- 65. Extension use caseAdding support for Microsoft Office Open XML Documents (Office 2007+)
- 66. Apache POI ApachePOI provides Text Extraction support for Office OpenXML formats and An advanced coverage of SpreadsheetML specification (WordprocessingML & PresentationML to come)
- 67. Apache POI ApachePOI status
- 68. Apache POI TextExtractorsPOIXMLDocument document; Package pkg = Package.open(stream); textExtractor = ExtractorFactory.createExtractor(pkg); if (textExtractor instanceof XSSFExcelExtractor) { setType(metadata, OOXML_EXCEL_MIMETYPE document = new XSSFWorkbook(pkg); } else if (textExtractor instanceof XWPFWordExtractor){…} else if (textExtractor instanceof XSLFPowerPointExtractor){…} setPOIXMLProperties(metadata, document);
- 69. Can we findit?
- 70.
- 71. Q & A[email_address]