Lightning talk :IBM Content Analytics with Enterprise Search - Wolfgang Jung

•

2 likes•1,979 views

See conference video - http://www.lucidimagination.com/devzone/events/conferences/ApacheLuceneEurocon2011 See and hear how IBM applies Lucence into their commercial software offerings. Hear about experience in development and advantages of this approach.

LIGHTNING TALKS
Powered by Lucene:
IBM Content Analytics with Enterprise Search

Wolfgang Jung

Barcelona, 19th October 2011 © 2011 IBM Corporation

IBM Content Analytics with Enterprise Search

Our agenda in the next 10 minutes
LIGHTNING TALKS
IBM is commited to Open Source
– Decade of contribution to the community.

Adoption of Apache Lucene to IBM Content Analytics
– The Why, What & examples.

Demonstration of IBM Content Analytics
– see the development results live.
Be enlightened !

2 © 2011 IBM Corporation

IBM Content Analytics with Enterprise Search

IBM is commited to Open Source

Decade of lineage and contributions to the open source community
– Apache Hadoop.
IBM‘s use of BigIndex for Search is mention in Chuck Lams‘s “Hadopp in Action”
– Apache Derby
– Apache Geronimo and Jetty
– Eclipse: Founded by IBM, PMC Board of Directors
– Apache UIMA: Unstructured Information Management Architecture.
Developed by IBM, Contributed to Apache
– Apache Jakarta: Lucene. PMC members
Significant contributions via IBM Lucene Extension Library (ILEL)
– Linux ... and more!

3 © 2011 IBM Corporation

IBM Content Analytics with Enterprise Search

Adoption of Apache Lucene
to IBM Content Analytics with Enterprise Search
The use of UIMA is existing since first release in 2005 of IBM OmniFind and later
IBM Content Analytics, continued into today‘s IBM Content Analytics with
Enterprise Search
http://www-01.ibm.com/software/data/content-management/analytics/uima.html

IBM‘s decision for the use of Lucene
–Index is a common technology and better to improve
–lower cost of maintenance
–advantage in incremental indexing
–extensibility

4 © 2011 IBM Corporation

IBM Content Analytics with Enterprise Search

Adoption of Apache Lucene
to IBM Content Analytics with Enterprise Search
IBM is a very active contributor. Look for PMC members:
–Michael McCandless; Shai Erera; Doron Cohen
http://lucene.apache.org/who.html

IBM extended Lucene based on our needs. Two examples already
contributed to community :
–Query Parser
–Facets

5 © 2011 IBM Corporation

IBM Content Analytics with Enterprise Search

Adoption of Apache Lucene
to IBM Content Analytics with Enterprise Search
On 13th December 2006, IBM and Yahoo! announced IBM OmniFind Yahoo! Edition, as
“no-cost, entry level enterprise search product developed to help eliminate financial and
technology barriers to intranet and Web search.”
http://www-03.ibm.com/press/us/en/pressrelease/20767.wss

This technology included Lucene as index technology and had full support by IBM
– 45,000+ downloads from the website http://omnifind.ibm.yahoo.net
– IBM support contracts for clients with “IBM Elite Support for OmniFind Yahoo Edition“
– Below 15 incidents regarding index technology

Technology is seen as success for IBM

6 © 2011 IBM Corporation

IBM Content Analytics with Enterprise Search

Content Analytics generates new insights and aggregates key
findings gathered from large data volumes in a visualized form

Extracted Concept
Claimant: Soft Tissue Injury
Automatic
Visualizing
Person Injury Body Part Location Results of concept evaluation
are displayed to the users
Noun Verb Noun Phrase Prep Phrase

Claus sprained his ankle on the step

Analysed documents
with identified concepts

Sources of Information
Internal (ECM, Files, DBMS, etc.)
and External (Social, News, etc.)

7 © 2011 IBM Corporation

IBM Content Analytics with Enterprise Search

Rapid Insights from Automotive Complaints

We will be using publically available data from the National Highway Traffic Safety Agency (NHTSA)
to demonstrate how IBM Content Analytics can be used to identify problems with automobiles.
NHTSA receives various reports about malfunctions, accidents, and other issues with automobiles
from dealerships, repair facilities, and from the general public. NHTSA publishes the data at
http://www.nhtsa.gov. For this demo we have created a collection from the NHTSA “complaints”
data spanning several years ending in early 2010. We will show how this and similar data can be
analyzed to arrive at rapid insights not possible by manually reading through the complaint records.

8 © 2011 IBM Corporation

IBM Content Analytics with Enterprise Search

See Content Analytics live!

9 © 2011 IBM Corporation

IBM Content Analytics with Enterprise Search

See Content Analytics live!

10 © 2011 IBM Corporation

IBM Content Analytics with Enterprise Search

Be enlightened !

11 © 2011 IBM Corporation

Продукт IBM Watson Analytics предоставляет мощные возможности анализа, доступные практически каждому. Автоматическая подготовка данных, прогнозная аналитика, отчетность, информационные панели, визуализация и функции совместной работы позволят вам полностью взять анализ данных под контроль. Далее вы сможете самостоятельно предпринять необходимые действия, чтобы решить проблему или воспользоваться возможностью, без помощи ИТ-специалистов и специалистов по обработке данных.

IBM Watson Analytics Presentation

Ian Balina

Predicting March Madness with IBM Watson Analytics

Ian Balina

Watson Analytics for HSE - Copy

Alexei Cherenkov

Ai, IBM Watson External

Jerry O'Brien

IBM Watson Overview

Penn State EdTech Network

Watson - Who What Why

None

Explore, analyze and interpret information for better business outcomes IBM Watson Explorer is a cognitive exploration solution that combines search and content analytics with unique cognitive computing capabilities to help users find and understand the information they need to work more efficiently and make better, more confident decisions. To learn more about IBM Watson Explorer visit ibm.biz/watsonexplorer.

Oltre l’intelligenza Artificiale: agire alla velocità del pensiero

Jürgen Ambrosi

In questo Webinar racconteremo come l’intelligenza cognitiva di IBM Watson si affianca alle Università e ai Centri di Ricerca per potenziare gli skill e le capacità di analisi e di comprensione dei dati e delle informazioni. Dare risposte concrete a problemi che incidono sulla nostra vita e il nostro lavoro, accedere ad un livello di conoscenza superiore grazie a nuove capacità cognitive anche questo è il nuovo modo di IBM per aiutare le Università e la Ricerca.

Watson AI platform for business - IBM Cloud

Sarmad Ibrahim

IBM Watson

Harshdeep Singh

Building Bots Using IBM Watson

Entrepreneur / Startup

Using Watson to build Cognitive IoT Apps on Bluemix

IBM

IBM Watson Explorer: Explore, analyze and interpret information for better bu...

Virginia Fernandez

AI future 2025 - IBM Watson Re

Sarmad Ibrahim

An AI Maturity Roadmap for Becoming a Data-Driven Organization

David Solomon

Modernizing the Analytics and Data Science Lifecycle for the Scalable Enterpr...

Data Con LA

Data Con LA 2020 Description It’s no secret that the roots of Data Science date back to the 1960’s and were first mainstreamed in the 1990’s with the emergence of Data Mining. This occurred when commercially affordable computers started offering the horsepower and storage necessary to perform advanced statistics to scale. However, the words “to scale” have evolved over time. The leap to “Big Data” is only one serial aspect of growth. Beyond the typical 1-off studies that catalyzed the field of Data Mining, Data Science now fulfills enterprise and multi-enterprise use cases spanning much broader and deeper data sets and integrations. For example, AI and Machine Learning frameworks can interoperate with a variety of other systems to drive alerting, feedback loops, predictive frameworks, prescriptive engines, continual learning, and more. The deployment of AI/ML processes themselves often involves integration with contemporary DevOps tools. Now segue to SEAL – the Scalable Enterprise Analytic Lifecycle. In this presentation, you’ll learn how to cover the major bases of a modern Data Science projects – and Citizen Data Science as well – from conception, learning, and evaluation through integration, implementation, monitoring, and continual improvement. And as the name implies, your deployments will be performant and scale as expected in today’s environments. Speaker Jeff Bertman, CTO, Dfuse Technologies

Big Data and Analytics: The IBM Perspective

The_IPA

Webinar - Comparative Analysis of Cloud based Machine Learning Platforms

BigDataCloud

Master the art of Data Science

InTTrust S.A.

Libera la potenza del Machine Learning

Jürgen Ambrosi

Un approccio completo di tipo cognitivo comprende tre componenti: un metodo, un ecosistema e una piattaforma. In questa sessione scopriremo come realizzare questo approccio grazie anche a Watson Data Platform, che aiuta i data scientist e gli esperti di business analytics a far “lavorare i dati” in un’ottica cognitive. In questo modo si può dare impulso alla crescita e al cambiamento aziendale. Ci concentreremo sulla possibilità di analizzare i dati provenienti dai Social Media per valutare la percezione dell’Amministrazione da parte di studenti, genitori, stampa, blogger… Al cuore della soluzione ci sono una serie di servizi disegnati per funzione aziendale (sviluppatori, data scientist, data engineers, comunicazione / marketing) e la capacità di imparare propria della tecnologia cognitiva, che completano l’architettura e aiutano a “comporre” nuove soluzioni di business.

InTTrust -IBM Artificial Intelligence Event

Michail Pagiatakis

Smw+ semantic enterprise wiki en_153

Semantic Enterprise Wiki SMWplus

"IBMs Open Source Strategy" by Adam Jollans @ eLiberatica 2009

eLiberatica

This is a presentation held at eLiberatica 2009. http://www.eliberatica.ro/2009/ One of the biggest events of its kind in Eastern Europe, eLiberatica brings community leaders from around the world to discuss about the hottest topics in FLOSS movement, demonstrating the advantages of adopting, using and developing Open Source and Free Software solutions. The eLiberatica organizational committee together with our speakers and guests, have graciously allowed media representatives and all attendees to photograph, videotape and otherwise record their sessions, on the condition that the photos, videos and recordings are licensed under the Creative Commons Share-Alike 3.0 License.

What's hot

Ml, AI and IBM Watson - 101 for Business

Jouko Poutanen

IBM Smarter AnalyticsAdrian Turcu

IBM Cognitive platform: IBM Watson

Daniela Zuppini

Watson and AnalyticsJorge W. Hago

Ibm big data-platform

IBM Sverige

What Watson Explorer is and How it works

Virginia Fernandez

Oltre l’intelligenza Artificiale: agire alla velocità del pensiero

Jürgen Ambrosi

Watson AI platform for business - IBM Cloud

Sarmad Ibrahim

IBM Watson

Harshdeep Singh

Building Bots Using IBM Watson

Entrepreneur / Startup

Using Watson to build Cognitive IoT Apps on Bluemix

IBM

IBM Watson Explorer: Explore, analyze and interpret information for better bu...

Virginia Fernandez

AI future 2025 - IBM Watson Re

Sarmad Ibrahim

An AI Maturity Roadmap for Becoming a Data-Driven Organization

David Solomon

Modernizing the Analytics and Data Science Lifecycle for the Scalable Enterpr...

Data Con LA

Big Data and Analytics: The IBM Perspective

The_IPA

Webinar - Comparative Analysis of Cloud based Machine Learning Platforms

BigDataCloud

Master the art of Data Science

InTTrust S.A.

Libera la potenza del Machine Learning

Jürgen Ambrosi

InTTrust -IBM Artificial Intelligence Event

Michail Pagiatakis

What's hot (20)

Ml, AI and IBM Watson - 101 for Business

IBM Smarter Analytics

IBM Cognitive platform: IBM Watson

Watson and Analytics

Ibm big data-platform

What Watson Explorer is and How it works

Oltre l’intelligenza Artificiale: agire alla velocità del pensiero

Watson AI platform for business - IBM Cloud

IBM Watson

Building Bots Using IBM Watson

Using Watson to build Cognitive IoT Apps on Bluemix

IBM Watson Explorer: Explore, analyze and interpret information for better bu...

AI future 2025 - IBM Watson Re

An AI Maturity Roadmap for Becoming a Data-Driven Organization

Modernizing the Analytics and Data Science Lifecycle for the Scalable Enterpr...

Big Data and Analytics: The IBM Perspective

Webinar - Comparative Analysis of Cloud based Machine Learning Platforms

Master the art of Data Science

Libera la potenza del Machine Learning

InTTrust -IBM Artificial Intelligence Event

Similar to Lightning talk :IBM Content Analytics with Enterprise Search - Wolfgang Jung

Smw+ semantic enterprise wiki en_153

Semantic Enterprise Wiki SMWplus

"IBMs Open Source Strategy" by Adam Jollans @ eLiberatica 2009

eLiberatica

Smw+tutorial berlin-fall-2011

Semantic Enterprise Wiki SMWplus

Flex 4.5 and mobile developmentMichael Chaize

Deploying Enterprise Search in PLM Context with Aras

Aras

Employ the Cloud for Efficient Content Analytics - 10 november 2011Samir Batla

Rosinski ibm ai overview with several examples of projects in the media and l...

FIAT/IFTA

Accelerating Apache MXNet Models on Apple Platforms Using Core ML - MCL311 - ...

Amazon Web Services

Running deep learning models on devices at the edge is one of the hottest trends in AI today. This workshop provides a tutorial on developing and training deep learning models with Apache MXNet and walks you through how to easily bring them into the Apple ecosystem of products. You will learn how to convert MXNet models easily and efficiently to formats that can be integrated into iOS/macOS applications. To participate in this workshop, attendees will require an Apple MacBook running the latest OS (10.13). An iPhone running iOS 11+ or higher to run Core ML and Apache MXNet is optional.

Breizh camp adobe flex et les mobiles

Michael Chaize

Splunk in 60 Minutes | Splunk Tutorial For Beginners | Splunk Training | Splu...

Edureka!

***** Splunk Training: https://www.edureka.co/splunk ***** This Edureka Live session on Splunk will help you understand the fundamentals for Splunk with a demo on Log Collection & Analysis. Below are the topics that will be discussed in this session: 1. Why Splunk? – Problems With Log Data 2. What Is Splunk? – Ultimate Soln. For Log Processing 3. How Does It Work? – Splunk Components 4. Hands-On:- Log Collection & Analysis

Open source, commercial or a co-existance strategy

IBM Rational software

Starting mobile development

Mihai Corlan

Mariana Alupului Inventionsmalupului

Adobe flash platform javaCh'ti JUG

Adobe flash platform java

Michael Chaize

Native extensions webinarimmanuelnoel

Jax2001 adobe keynote

Michael Chaize

The IBM Rational Insight Reporting Solution

Marc Nehme

Enterprise reporting with Rational and non-Rational products. This segment will discuss and demonstrate the benefits of using the Rational Insight solution for enterprise reporting from Rational and non-Rational data sources. One section will cover Insight’s flexibility and how it can leverage different data source integration methods such as REST and ODBC, to extract data not just from Rational products such as Requisite Pro and but also non-Rational and open source products such as Hudson, Bugzilla, Sonar, and Excel. Another section will cover the benefits of compiling all of this disparate data in an automated fashion from various data sources into one Insight dashboard view, which provides an organization with a higher level of visibility and detail into their software development projects and activities. A final section will cover best practices and overall benefits of implementing one unified reporting solution with Insight across an entire organization, showing the time, effort, and financial savings.

Convergence of mobility, analytics, social and cloud to drive innovation

Kerrie Holley

Inform: Targeting the Interest Graph

Vital.AI

Similar to Lightning talk :IBM Content Analytics with Enterprise Search - Wolfgang Jung (20)

Smw+ semantic enterprise wiki en_153

"IBMs Open Source Strategy" by Adam Jollans @ eLiberatica 2009

Smw+tutorial berlin-fall-2011

Flex 4.5 and mobile development

Deploying Enterprise Search in PLM Context with Aras

Employ the Cloud for Efficient Content Analytics - 10 november 2011

Rosinski ibm ai overview with several examples of projects in the media and l...

Accelerating Apache MXNet Models on Apple Platforms Using Core ML - MCL311 - ...

Breizh camp adobe flex et les mobiles

Splunk in 60 Minutes | Splunk Tutorial For Beginners | Splunk Training | Splu...

Open source, commercial or a co-existance strategy

Starting mobile development

Mariana Alupului Inventions

Adobe flash platform java

Native extensions webinar

Jax2001 adobe keynote

The IBM Rational Insight Reporting Solution

Convergence of mobility, analytics, social and cloud to drive innovation

Inform: Targeting the Interest Graph

More from lucenerevolution

Text Classification Powered by Apache Mahout and Lucene

lucenerevolution

Presented by Isabel Drost-Fromm, Software Developer, Apache Software Foundation/Nokia Gate 5 GmbH at Lucene/Solr Revolution 2013 Dublin Text classification automates the task of filing documents into pre-defined categories based on a set of example documents. The first step in automating classification is to transform the documents to feature vectors. Though this step is highly domain specific Apache Mahout provides you with a lot of easy to use tooling to help you get started, most of which relies heavily on Apache Lucene for analysis, tokenisation and filtering. This session shows how to use facetting to quickly get an understanding of the fields in your document. It will walk you through the steps necessary to convert your text documents into feature vectors that Mahout classifiers can use including a few anecdotes on drafting domain specific features. Configure

State of the Art Logging. Kibana4Solr is Here!

lucenerevolution

Presented by Markus Klose, Search + Big Data Consultant SHI Elektronische Medien GmbH at Lucene/Solr Revolution 2013 Dublin Kibana4Solr is search-driven, scalable, browser based and extremely user friendly (also for non-technical users). Logs are everywhere. Any device, system or human can potentially produce a huge amount of information saved in logs. The amount of available logs and their semi-structured nature make a meaningful processing in real-time quite a difficult task. Thus, valuable business insights stored in logs might be not found. Kibana4Solr is a search-driven approach to handle that challenge. It offers user-friendly and browser-based dashboard which can be easily customized to particular needs. In the session the Kibana4Solr will be introduced. Some light will be shed on the architectural features of Kibana4Solr. Some ideas will be given in terms of possible business uses cases. And finally a live demo of Kibana4Solr will be shown. Configure

Search at Twitterlucenerevolution

Building Client-side Search Applications with Solr

lucenerevolution

Presented by Daniel Beach, Search Application Developer, OpenSource Connections Solr is a powerful search engine, but creating a custom user interface can be daunting. In this fast paced session I will present an overview of how to implement a client-side search application using Solr. Using open-source frameworks like SpyGlass (to be released in September) can be a powerful way to jumpstart your development by giving you out-of-the box results views with support for faceting, autocomplete, and detail views. During this talk I will also demonstrate how we have built and deployed lightweight applications that are able to be performant under large user loads, with minimal server resources.

Integrate Solr with real-time stream processing applications

lucenerevolution

Presented by Timothy Potter, Founder, Text Centrix Storm is a real-time distributed computation system used to process massive streams of data. Many organizations are turning to technologies like Storm to complement batch-oriented big data technologies, such as Hadoop, to deliver time-sensitive analytics at scale. This talk introduces on an emerging architectural pattern of integrating Solr and Storm to process big data in real time. There are a number of natural integration points between Solr and Storm, such as populating a Solr index or supplying data to Storm using Solr’s real-time get support. In this session, Timothy will cover the basic concepts of Storm, such as spouts and bolts. He’ll then provide examples of how to integrate Solr into Storm to perform large-scale indexing in near real-time. In addition, we'll see how to embed Solr in a Storm bolt to match incoming tuples against pre-configured queries, commonly known as percolator. Attendees will come away from this presentation with a good introduction to stream processing technologies and several real-world use cases of how to integrate Solr with Storm.

Scaling Solr with SolrCloud

lucenerevolution

Configure your Solr cluster to handle hundreds of millions of documents without even noticing, handle queries in milliseconds, use Near Real Time indexing and searching with document versioning. Scale your cluster both horizontally and vertically by using shards and replicas. In this session you'll learn how to make your indexing process blazing fast and make your queries efficient even with large amounts of data in your collections. You'll also see how to optimize your queries to leverage caches as much as your deployment allows and how to observe your cluster with Solr administration panel, JMX, and third party tools. Finally, learn how to make changes to already deployed collections —split their shards and alter their schema by using Solr API.

Administering and Monitoring SolrCloud Clusters

lucenerevolution

Presented by Rafal Kuć, Consultant and Software engineer, , Sematext Group, Inc. Even though Solr can run without causing any troubles for long periods of time it is very important to monitor and understand what is happening in your cluster. In this session you will learn how to use various tools to monitor how Solr is behaving at a high level, but also on Lucene, JVM, and operating system level. You'll see how to react to what you see and how to make changes to configuration, index structure and shards layout using Solr API. We will also discuss different performance metrics to which you ought to pay extra attention. Finally, you'll learn what to do when things go awry - we will share a few examples of troubleshooting and then dissect what was wrong and what had to be done to make things work again.

Implementing a Custom Search Syntax using Solr, Lucene, and Parboiled

lucenerevolution

In a recent project with the United States Patent and Trademark Office, Opensource Connections was asked to prototype the next generation of patent search - using Solr and Lucene. An important aspect of this project was the implementation of BRS, a specialized search syntax used by patent examiners during the examination process. In this fast paced session we will relate our experiences and describe how we used a combination of Parboiled (a Parser Expression Grammar [PEG] parser), Lucene Queries and SpanQueries, and an extension of Solr's QParserPlugin to build BRS search functionality in Solr. First we will characterize the patent search problem and then define the BRS syntax itself. We will then introduce the Parboiled parser and discuss various considerations that one must make when designing a syntax parser. Following this we will describe the methodology used to implement the search functionality in Lucene/Solr. Finally, we will include an overview our syntactic and semantic testing strategies. The audience will leave this session with an understanding of how Solr, Lucene, and Parboiled may be used to implement their own custom search parser.

Using Solr to Search and Analyze Logs

lucenerevolution

Many of us tend to hate or simply ignore logs, and rightfully so: they’re typically hard to find, difficult to handle, and are cryptic to the human eye. But can we make logs more valuable and more usable if we index them in Solr, so we can search and run real-time statistics on them? Indeed we can, and in this session you’ll learn how to make that happen. In the first part of the session we’ll explain why centralized logging is important, what valuable information one can extract from logs, and we’ll introduce the leading tools from the logging ecosystems everyone should be aware of - from syslog and log4j to LogStash and Flume. In the second part we’ll teach you how to use these tools in tandem with Solr. We’ll show how to use Solr in a SolrCloud setup to index large volumes of logs continuously and efficiently. Then, we'll look at how to scale the Solr cluster as your data volume grows. Finally, we'll see how you can parse your unstructured logs and convert them to nicely structured Solr documents suitable for analytical queries.

Enhancing relevancy through personalization & semantic searchlucenerevolution

Real-time Inverted Search in the Cloud Using Lucene and Storm

lucenerevolution

Building real-time notification systems is often limited to basic filtering and pattern matching against incoming records. Allowing users to query incoming documents using Solr's full range of capabilities is much more powerful. In our environment we needed a way to allow for tens of thousands of such query subscriptions, meaning we needed to find a way to distribute the query processing in the cloud. By creating in-memory Lucene indices from our Solr configuration, we were able to parallelize our queries across our cluster. To achieve this distribution, we wrapped the processing in a Storm topology to provide a flexible way to scale and manage our infrastructure. This presentation will describe our experiences creating this distributed, real-time inverted search notification framework.

Solr's Admin UI - Where does the data come from?

lucenerevolution

Like many Web-Applications in the past, the Solr Admin UI up until 4.0 was entirely server based. It used separate code on the server to generate their Dashboards, Overviews and Statistics. All that code had to be maintained and still ... you weren't really able to use that kind of data for the things you needed it for. It was wrapped into HTML, most of the time difficult to extract and changed the structure from time to time w/o announcement. After a short look back, we're going to look into the current state of the Solr Admin UI - a client-side application, running completely in your browser. We'll see how it works, where it gets its data from and how you can get the very same data and wire that into your own custom applications, dashboards and/oder monitoring systems.

Schemaless Solr and the Solr Schema REST API

lucenerevolution

High Performance JSON Search and Relational Faceted Browsing with Lucene

lucenerevolution

Presented by Renaud Delbru, Co-Founder, SindiceTech In this presentation, we will discuss how Lucene and Solr can be used for very efficient search of tree-shaped schemaless document, e.g. JSON or XML, and can be then made to address both graph and relational data search. We will discuss the capabilities of SIREn, a Lucene/Solr plugin we have developed to deal with huge collections of tree-shaped schemaless documents, and how SIREn is built using Lucene extensibility capabilities (Analysis, Codec, Flexible Query Parser). We will compare it with Lucene's BlockJoin Query API in nested schemaless data intensive scenarios. We will then go through use cases that show how relational or graph data can be turned into JSON documents using Hadoop and Pig, and how this can be used in conjunction with SIREn to create relational faceting systems with unprecedented performance. Take-away lessons from this session will be awareness about using Lucene/Solr and Hadoop for relational and graph data search, as well as the awareness that it is now possible to have relational faceted browsers with sub-second response time on commodity hardware.

Text Classification with Lucene/Solr, Apache Hadoop and LibSVM

lucenerevolution

In this session we will show how to build a text classifier using the Apache Lucene/Solr with libSVM libraries. We classify our corpus of job offers into a number of predefined categories. Each indexed document (a job offer) then belongs to zero, one or more categories. Known machine learning techniques for text classification include naïve bayes model, logistic regression, neural network, support vector machine (SVM), etc. We use Lucene/Solr to construct the features vector. Then we use the libsvm library known as the reference implementation of the SVM model to classify the document. We construct as many one-vs-all svm classifiers as there are classes in our setting, then using the Hadoop MapReduce Framework we reconcile the result of our classifiers. The end result is a scalable multi-class classifier. Finally we outline how the classifier is used to enrich basic solr keyword search.

Faceted Search with Lucene

lucenerevolution

Faceted search is a powerful technique to let users easily navigate the search results. It can also be used to develop rich user interfaces, which give an analyst quick insights about the documents space. In this session I will introduce the Facets module, how to use it, under-the-hood details as well as optimizations and best practices. I will also describe advanced faceted search capabilities with Lucene Facets.

Recent Additions to Lucene Arsenal

lucenerevolution

Presented by Shai Erera, Researcher, IBM Lucene's arsenal has recently expanded to include two new modules: Index Sorting and Replication. Index sorting lets you keep an index consistently sorted based on some criteria (e.g. modification date). This allows for efficient search early-termination as well as achieve better index compression. Index replication lets you replicate a search index to achieve high-availability, fault tolerance as well as take hot index backups. In this talk we will introduce these modules, discuss implementation and design details as well as best practices.

Turning search upside down

lucenerevolution

As part of their work with large media monitoring companies, Flax has developed a technique for applying tens of thousands of stored Lucene queries to a document in under a second. We'll talk about how we built intelligent filters to reduce the number of actual queries applied and how we extended Lucene to extract the exact hit positions of matches, the challenges of implementation, and how it can be used, including applications that monitor hundreds of thousands of news stories every day.

Spellchecking in Trovit: Implementing a Contextual Multi-language Spellchecke...

lucenerevolution

Presented by Xavier Sanchez Loro, Ph.D, Trovit Search SL This session aims to explain the implementation and use case for spellchecking in Trovit search engine. Trovit is a classified ads search engine supporting several different sites, one for each on country and vertical. Our search engine supports multiple indexes in multiple languages, each with several millions of indexed ads. Those indexes are segmented in several different sites depending on the type of ads (homes, cars, rentals, products, jobs and deals). We have developed a multi-language spellchecking system using solr and lucene in order to help our users to better find the desired ads and avoid the dreaded 0 results as much as possible. As such our goal is not pure orthographic correction, but also suggestion of correct searches for a certain site.

Shrinking the haystack wes caldwell - finallucenerevolution

More from lucenerevolution (20)

Text Classification Powered by Apache Mahout and Lucene

State of the Art Logging. Kibana4Solr is Here!

Search at Twitter

Building Client-side Search Applications with Solr

Integrate Solr with real-time stream processing applications

Scaling Solr with SolrCloud

Administering and Monitoring SolrCloud Clusters

Implementing a Custom Search Syntax using Solr, Lucene, and Parboiled

Using Solr to Search and Analyze Logs

Enhancing relevancy through personalization & semantic search

Real-time Inverted Search in the Cloud Using Lucene and Storm

Solr's Admin UI - Where does the data come from?

Schemaless Solr and the Solr Schema REST API

High Performance JSON Search and Relational Faceted Browsing with Lucene

Text Classification with Lucene/Solr, Apache Hadoop and LibSVM

Faceted Search with Lucene

Recent Additions to Lucene Arsenal

Turning search upside down

Spellchecking in Trovit: Implementing a Contextual Multi-language Spellchecke...

Shrinking the haystack wes caldwell - final

Recently uploaded

Elevating Tactical DDD Patterns Through Object Calisthenics

Dorra BARTAGUIZ

After immersing yourself in the blue book and its red counterpart, attending DDD-focused conferences, and applying tactical patterns, you're left with a crucial question: How do I ensure my design is effective? Tactical patterns within Domain-Driven Design (DDD) serve as guiding principles for creating clear and manageable domain models. However, achieving success with these patterns requires additional guidance. Interestingly, we've observed that a set of constraints initially designed for training purposes remarkably aligns with effective pattern implementation, offering a more ‘mechanical’ approach. Let's explore together how Object Calisthenics can elevate the design of your tactical DDD patterns, offering concrete help for those venturing into DDD for the first time!

Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...

Thierry Lestable

Transcript: Selling digital books in 2024: Insights from industry leaders - T...

BookNet Canada

The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more. Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/ Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.

From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...

Product School

Designing Great Products: The Power of Design and Leadership by Chief Designe...

Product School

When stars align: studies in data quality, knowledge graphs, and machine lear...

Elena Simperl

FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf

FIDO Alliance

Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...

Jeffrey Haguewood

Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows. We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases. This video focuses on the notifications, alerts, and approval requests using Slack for Bonterra Impact Management. The solutions covered in this webinar can also be deployed for Microsoft Teams. Interested in deploying notification automations for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.

Key Trends Shaping the Future of Infrastructure.pdf

Cheryl Hung

Connector Corner: Automate dynamic content and events by pushing a button

DianaGray10

Here is something new! In our next Connector Corner webinar, we will demonstrate how you can use a single workflow to: Create a campaign using Mailchimp with merge tags/fields Send an interactive Slack channel message (using buttons) Have the message received by managers and peers along with a test email for review But there’s more: In a second workflow supporting the same use case, you’ll see: Your campaign sent to target colleagues for approval If the “Approve” button is clicked, a Jira/Zendesk ticket is created for the marketing design team But—if the “Reject” button is pushed, colleagues will be alerted via Slack message Join us to learn more about this new, human-in-the-loop capability, brought to you by Integration Service connectors. And... Speakers: Akshay Agnihotri, Product Manager Charlie Greenberg, Host

Monitoring Java Application Security with JDK Tools and JFR Events

Ana-Maria Mihalceanu

PCI PIN Basics Webinar from the Controlcase Team

ControlCase

LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...

DanBrown980551

Do you want to learn how to model and simulate an electrical network from scratch in under an hour? Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)! During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook. PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides: - A fully editable and extendable library for grid component modelling; - Visualization tools to display your network; - Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses; The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well. What you will learn during the webinar: - For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills; - For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.

Neuro-symbolic is not enough, we need neuro-*semantic*

Frank van Harmelen

Neuro-symbolic (NeSy) AI is on the rise. However, simply machine learning on just any symbolic structure is not sufficient to really harvest the gains of NeSy. These will only be gained when the symbolic structures have an actual semantics. I give an operational definition of semantics as “predictable inference”. All of this illustrated with link prediction over knowledge graphs, but the argument is general.

Assuring Contact Center Experiences for Your Customers With ThousandEyes

ThousandEyes

Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...

UiPathCommunity

💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™: See how to accelerate model training and optimize model performance with active learning Learn about the latest enhancements to out-of-the-box document processing – with little to no training required Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath. Speakers: 👨‍🏫 Andras Palfi, Senior Product Manager, UiPath 👩‍🏫 Lenka Dulovicova, Product Program Manager, UiPath

Essentials of Automations: Optimizing FME Workflows with Parameters

Safe Software

Are you looking to streamline your workflows and boost your projects’ efficiency? Do you find yourself searching for ways to add flexibility and control over your FME workflows? If so, you’re in the right place. Join us for an insightful dive into the world of FME parameters, a critical element in optimizing workflow efficiency. This webinar marks the beginning of our three-part “Essentials of Automation” series. This first webinar is designed to equip you with the knowledge and skills to utilize parameters effectively: enhancing the flexibility, maintainability, and user control of your FME projects. Here’s what you’ll gain: - Essentials of FME Parameters: Understand the pivotal role of parameters, including Reader/Writer, Transformer, User, and FME Flow categories. Discover how they are the key to unlocking automation and optimization within your workflows. - Practical Applications in FME Form: Delve into key user parameter types including choice, connections, and file URLs. Allow users to control how a workflow runs, making your workflows more reusable. Learn to import values and deliver the best user experience for your workflows while enhancing accuracy. - Optimization Strategies in FME Flow: Explore the creation and strategic deployment of parameters in FME Flow, including the use of deployment and geometry parameters, to maximize workflow efficiency. - Pro Tips for Success: Gain insights on parameterizing connections and leveraging new features like Conditional Visibility for clarity and simplicity. We’ll wrap up with a glimpse into future webinars, followed by a Q&A session to address your specific questions surrounding this topic. Don’t miss this opportunity to elevate your FME expertise and drive your projects to new heights of efficiency.

How world-class product teams are winning in the AI era by CEO and Founder, P...

Product School

DevOps and Testing slides at DASA Connect

Kari Kakkonen

State of ICS and IoT Cyber Threat Landscape Report 2024 preview

Prayukth K V

The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development. The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers: State of global ICS asset and network exposure Sectoral targets and attacks as well as the cost of ransom Global APT activity, AI usage, actor and tactic profiles, and implications Rise in volumes of AI-powered cyberattacks Major cyber events in 2024 Malware and malicious payload trends Cyberattack types and targets Vulnerability exploit attempts on CVEs Attacks on counties – USA Expansion of bot farms – how, where, and why In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East Why are attacks on smart factories rising? Cyber risk predictions Axis of attacks – Europe Systemic attacks in the Middle East Download the full report from here: https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/

Recently uploaded (20)

Elevating Tactical DDD Patterns Through Object Calisthenics

Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...

Transcript: Selling digital books in 2024: Insights from industry leaders - T...

From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...

Designing Great Products: The Power of Design and Leadership by Chief Designe...

When stars align: studies in data quality, knowledge graphs, and machine lear...

FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf

Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...

Key Trends Shaping the Future of Infrastructure.pdf

Connector Corner: Automate dynamic content and events by pushing a button

Monitoring Java Application Security with JDK Tools and JFR Events

PCI PIN Basics Webinar from the Controlcase Team

LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...

Neuro-symbolic is not enough, we need neuro-*semantic*

Assuring Contact Center Experiences for Your Customers With ThousandEyes

Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...

Essentials of Automations: Optimizing FME Workflows with Parameters

How world-class product teams are winning in the AI era by CEO and Founder, P...

DevOps and Testing slides at DASA Connect

State of ICS and IoT Cyber Threat Landscape Report 2024 preview

Lightning talk :IBM Content Analytics with Enterprise Search - Wolfgang Jung

2. IBM Content Analytics with Enterprise Search Our agenda in the next 10 minutes LIGHTNING TALKS IBM is commited to Open Source – Decade of contribution to the community. Adoption of Apache Lucene to IBM Content Analytics – The Why, What & examples. Demonstration of IBM Content Analytics – see the development results live. Be enlightened ! 2 © 2011 IBM Corporation

3. IBM Content Analytics with Enterprise Search IBM is commited to Open Source Decade of lineage and contributions to the open source community – Apache Hadoop. IBM‘s use of BigIndex for Search is mention in Chuck Lams‘s “Hadopp in Action” – Apache Derby – Apache Geronimo and Jetty – Eclipse: Founded by IBM, PMC Board of Directors – Apache UIMA: Unstructured Information Management Architecture. Developed by IBM, Contributed to Apache – Apache Jakarta: Lucene. PMC members Significant contributions via IBM Lucene Extension Library (ILEL) – Linux ... and more! 3 © 2011 IBM Corporation

4. IBM Content Analytics with Enterprise Search Adoption of Apache Lucene to IBM Content Analytics with Enterprise Search The use of UIMA is existing since first release in 2005 of IBM OmniFind and later IBM Content Analytics, continued into today‘s IBM Content Analytics with Enterprise Search http://www-01.ibm.com/software/data/content-management/analytics/uima.html IBM‘s decision for the use of Lucene –Index is a common technology and better to improve –lower cost of maintenance –advantage in incremental indexing –extensibility 4 © 2011 IBM Corporation

5. IBM Content Analytics with Enterprise Search Adoption of Apache Lucene to IBM Content Analytics with Enterprise Search IBM is a very active contributor. Look for PMC members: –Michael McCandless; Shai Erera; Doron Cohen http://lucene.apache.org/who.html IBM extended Lucene based on our needs. Two examples already contributed to community : –Query Parser –Facets 5 © 2011 IBM Corporation

6. IBM Content Analytics with Enterprise Search Adoption of Apache Lucene to IBM Content Analytics with Enterprise Search On 13th December 2006, IBM and Yahoo! announced IBM OmniFind Yahoo! Edition, as “no-cost, entry level enterprise search product developed to help eliminate financial and technology barriers to intranet and Web search.” http://www-03.ibm.com/press/us/en/pressrelease/20767.wss This technology included Lucene as index technology and had full support by IBM – 45,000+ downloads from the website http://omnifind.ibm.yahoo.net – IBM support contracts for clients with “IBM Elite Support for OmniFind Yahoo Edition“ – Below 15 incidents regarding index technology Technology is seen as success for IBM 6 © 2011 IBM Corporation

7. IBM Content Analytics with Enterprise Search Content Analytics generates new insights and aggregates key findings gathered from large data volumes in a visualized form Extracted Concept Claimant: Soft Tissue Injury Automatic Visualizing Person Injury Body Part Location Results of concept evaluation are displayed to the users Noun Verb Noun Phrase Prep Phrase Claus sprained his ankle on the step Analysed documents with identified concepts Sources of Information Internal (ECM, Files, DBMS, etc.) and External (Social, News, etc.) 7 © 2011 IBM Corporation

8. IBM Content Analytics with Enterprise Search Rapid Insights from Automotive Complaints We will be using publically available data from the National Highway Traffic Safety Agency (NHTSA) to demonstrate how IBM Content Analytics can be used to identify problems with automobiles. NHTSA receives various reports about malfunctions, accidents, and other issues with automobiles from dealerships, repair facilities, and from the general public. NHTSA publishes the data at http://www.nhtsa.gov. For this demo we have created a collection from the NHTSA “complaints” data spanning several years ending in early 2010. We will show how this and similar data can be analyzed to arrive at rapid insights not possible by manually reading through the complaint records. 8 © 2011 IBM Corporation

Lightning talk :IBM Content Analytics with Enterprise Search - Wolfgang Jung

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Lightning talk :IBM Content Analytics with Enterprise Search - Wolfgang Jung

Similar to Lightning talk :IBM Content Analytics with Enterprise Search - Wolfgang Jung (20)

More from lucenerevolution

More from lucenerevolution (20)

Recently uploaded

Recently uploaded (20)

Lightning talk :IBM Content Analytics with Enterprise Search - Wolfgang Jung