IR: Open source state

•

1 like•458 views

Introductory level presentation on Information Retrieval: Open source state. Helps the reader to comprehend what open source systems and tools are available for creating / managing own search engines. Provides a glimpse into research directions in IR, also solvable with open source solutions. These slides were presented in the University of Helsinki, as a guest lecture for the "Information Retrieval and Search Engines - Spring 2017" course.

Technology

IR: open source
state
Dmitry Kan, AlphaSense, Insider
Solutions
University of Helsinki, Information Retrieval and
Search Engines course, Feb 21, 2017

About me
● PhD in CS (Saint Petersburg State University), 2011
● Running a Search Engine team at AlphaSense since 2014
● Founded Insider Solutions in 2009: text analytics solutions +
consulting
● Co-committer on luke project: toolbox for Lucene index since 2013

What is AlphaSense
● Google for financial analysts
● Semantic research engine
● Edit, tag, annotate, share you data in a team
● Oracle, JP Morgan, Credit Suisse
● Engineering is 98% in Helsinki + 1% NYC + 1% India
● #1 fastest growing IT startup in Finland by Deloitte
(2015)
www.alpha-sense.com

● Founded 2009
● BigText Analytics APIs and on-premise solutions
○ Sentiment analysis: Russian, Chinese, English
○ Searchable trend extraction
● Consulting: startups and corporates
https://semanticanalyzer.info
Insider Solutions

Outline
● Search engine architecture
● Open source search ecosystem
● Research directions for applied IR

Search engine: building blocks
● Web crawler: Apache Nutch (based on Hadoop)
● Data ingestion pipeline: receiving, cleaning, data
extraction
● SolrCloud OR Elasticsearch (both based on Lucene)
● Shards: storing index on disk and / or memory

Search Engine Software Components
● Schema
● Query parser
● Scoring algorithm
● Snippet highlighter
● Index (on-disk or in-memory)

Content extraction
Apache Tika for parsing formats:
● Html, XML
● PDF
● Microsoft Office & iWorks document formats
● Audio, image, video
● Mail
● Source code

Inspecting Lucene index with Luke
Implemented by Andrzej Bialecki. Since 2013 → by Dmitry Kan (Finland) and
Tomoko Uchida (Japan)
● Perform index maintenance
● Prototype similarity functions
● Search for documents, reconstruct field values from the index
● Read index from HDFS (Hadoop’s distributed file system)
● Supports Apache Solr and Elasticsearch

Learning to rank: Solr
Contributed by Bloomberg
Machine learnt model for reranking documents based on user feedback
Trained on features: views, popularity, was hit in the title, length, can view on
mobile device?
LamdaMART, RankSVM

Word vectors with Lucene
Word2vec was released by Google to open source
Possible to train word2vec on Lucene index:
https://github.com/kojisekig/word2vec-lucene
● NO need to provide a text file besides Lucene index
● NO need to normalize text. Normalization already done in the index or
Analyzer does it for you when processing
● Use part of the index by specifying a filter query

References
1. Luke: https://github.com/DmitryKey/luke
2. My blog: http://dmitrykan.blogspot.fi/
3. Solr vs Elasticsearch (overview): https://sematext.com/blog/2015/01/30/solr-elasticsearch-comparison/
4. Solr vs Elasticsearch (in-depth): https://sematext.com/blog/2012/08/23/solr-vs-elasticsearch-part-1-overview/
5. Introduction to Apache Solr http://www.slideshare.net/ChristosManios/introduction-to-apache-solr-54076189
6. Word2vec-lucene: https://github.com/kojisekig/word2vec-lucene
7. Apache Tika: https://tika.apache.org/
8. Apache Solr: http://lucene.apache.org/solr/
9. Elasticsearch: https://github.com/elastic/elasticsearch
10. Learning to rank in Solr (video): https://www.youtube.com/watch?v=M7BKwJoh96s
11. Learning to rank in Solr (slides): https://lucidworks.com/2016/08/17/learning-to-rank-solr/
12. Word2vec: https://en.wikipedia.org/wiki/Word2vec#Analysis
13. Lucene scoring formula:
https://lucene.apache.org/core/4_0_0/core/org/apache/lucene/search/similarities/TFIDFSimilarity.html

Machine Learning in LINE Fukuoka DataLabs/ 機器學習在LINE福岡的各種應用 Wenchun Kao Summary： With increasing popularity of Machine Learning, it is easy to train Machine Learning models with external libraries but building systems to meet different needs is still a challenge. This talk will introduce how Machine Learning is applied to not only user-facing products but internal systems in LINE Fukuoka DataLabs. I will also share the life as a Machine Learning Engineer in LINE Fukuoka. 因為這幾年機器學習的熱潮，現在非常容易利用各式的函式庫訓練機器學習模型，但建立可應付不同需求的機器學習系統仍是一個困難的工程問題。此次主要分享在LINE福岡裡面機器學習如何被應用到各式的應用上。有使用者會接觸的產品，有些則是內部人員所用的系統。我也會分享作為一個機器學習工程師在LINE福岡的日常生活。

Introduction to Clova platform with Machine learning development in practice

LINE Corporation

Data Engineering in LINE Fukuoka DataLabs: A Newcomer’s View

LINE Corporation

Liangchi Hsieh Data Engineering in LINE Fukuoka DataLabs: A Newcomer’s View 談 LINE 福岡 DataLabs 的資料工程：一個新人的觀點 Summary: Data is very important property for modern company. Data engineering is the engineering practice to make data feasible for valuable use in business. As a Data Engineer newcomer joining LINE Fukuoka DataLabs team in less two months, I will talk about what data engineering looks like in LINE and what data engineer is supposed to do in LINE Fukuoka. 資料已是現代企業至關重要的資產之一。為了讓這些資料在企業運作中發揮價值，資料工程是不可或缺的工程實踐。作為剛剛加入 LINE 福岡 DataLabs 團隊不到兩個月的新進資料工程師，我將介紹我所知的 LINE 的資料工程概況，以及資料工程師在 LINE 福岡的工作內容。

VanyaSehgal_ResumeVANYA SEHGAL

WLIA - 2015 Fall Regional, Oshkosh WI

Wisconsin State Cartographer's Office

Talk given at the 2015 Fall Regional in Oshkosh WI. "An Approach to Address Parsing and Data Standardization" Abstract: Maintaining fully parsed address elements in your database can be one of the most beneficial steps toward achieving quality and consistency in addressing. Parsed address elements also serve a preparatory step in modeling an address toward NG9-1-1 supporting formats such as the FGDC address standard. In this talk, we’ll take a look at the approach we’ve used for parsing site addresses for the V1 Statewide Parcel Map, the role regular expressions played in this approach, and will unveil a suite of (free) ArcPy tools that can help you parse addresses, standardize field values, and achieve other tasks. Presenters: Codie See David Vogel

Divya Full Time ResumeMeenu Swaminathan

Incorporating Chinese Radicals Into Neural Machine Translation: Deeper Than C...

Lifeng (Aaron) Han

Resume ashay

ashay sharma

Entity Search is a search paradigm that aims to retrieve entities and all the information related to them. In the last few years the importance of this topic has become greater and greater due to the fact that 40% of the queries made by users mention specific entities nowdays. This talk wants to give a first overview of the state-of-the-art methods used for entities retrieval and then describe the new approach Anna has implemented and proposed in her master thesis. The novelty introduced with this work exploits two machine learning techniques: neural network and clustering.

DATA SCIENCE TRAINING IN CHENNAI

shivajirao12345

Chinese Character Decomposition for Neural MT with Multi-Word Expressions

Lifeng (Aaron) Han

MTEngine: Semantic-level Crowdsourced Machine Translation

Dmitry Kan

Видео к презентации: http://vk.com/mtengine В докладе представлен краудсорсинг проект, ориентированный на построение и улучшение системы машинного перевода. Отличительной чертой является применение компьютерной семантики русского языка. Также рассматривается статистический метод автоматической генерации переводных словарей.

SentiScan: система автоматической разметки тональности в social media

Dmitry Kan

Автоматический анализ тональности можно по праву считать подзадачей ИИ. В этом докладе мы рассмотрим проблематику создания системы SentiScan, коснёмся вопросов оценки качества, сопровождения, реальных кейсов и способов улучшения качества в полуавтоматическом режиме. Компания SemanticAnalyzer разработала API для распознавания объектной тональности в текстах на русском языке. Испробовать систему можно подключившись к API на сайте: https://www.mashape.com/dmitrykey/russiansentimentanalyzer

Social spam detection by SemanticAnalyzer GroupDmitry Kan

Lucene revolution eu 2013 dublin writeup

Dmitry Kan

Automatic Build Of Semantic Translational DictionaryDmitry Kan

Machine translation course program (in English)

Dmitry Kan

Introduction To Machine Translation 1Dmitry Kan

Linguistic component Sentiment Analyzer for the Russian language

Dmitry Kan

Semantic feature machine translation systemDmitry Kan

Solr onfitnesse learningfromberlinbuzzwordsDmitry Kan

Starget sentiment analyzer for English

Dmitry Kan

Linguistic component Lemmatizer for the Russian language

Dmitry Kan

Introduction To Machine TranslationDmitry Kan

NoSQL, Apache SOLR and Apache Hadoop

Dmitry Kan

NoSQL (Not Only SQL) is believed to be a superset of, or sometimes an intersecting set with, relational SQL databases. The concept itself is still shaping, but already now we can say for sure: NoSQL addresses the task of storing and retrieving the data of large volumes in the systems with high load. There is another very important angle in perceiving the concept: NoSQL systems can allow storing and efficient searching of the unstructured or semi-unstructured data, like completely raw or preprocessed documents. Using the example of one world-class document retrieval system Apache SOLR (performant HTTP wrapper around Apache Lucene) as a reference we will check upon its use cases, horizontal and vertical scalability, faceted search, distribution and load balancing, crawling, extendability, linguistic support, integration with relational databases and much more. Dmitry Kan will shortly touch upon *hot* topic of cloud computing using the famous project Apache Hadoop and will help the audience to see whether SOLR shines through the cloud.

Rule based approach to sentiment analysis at ROMIP 2011Dmitry Kan

Poster: Method for an automatic generation of a semantic-level contextual tra...Dmitry Kan

What's hot

Resume ashay

ashay sharma

Apply chinese radicals into neural machine translation: deeper than character...

Lifeng (Aaron) Han

ADAPT Centre and My NLP journey: MT, MTE, QE, MWE, NER, Treebanks, Parsing.

Lifeng (Aaron) Han

Entity Search on Virtual Documents Created with Graph Embeddings

Sease

DATA SCIENCE TRAINING IN CHENNAI

shivajirao12345

Chinese Character Decomposition for Neural MT with Multi-Word Expressions

Lifeng (Aaron) Han

What's hot (6)

Resume ashay

Apply chinese radicals into neural machine translation: deeper than character...

ADAPT Centre and My NLP journey: MT, MTE, QE, MWE, NER, Treebanks, Parsing.

Entity Search on Virtual Documents Created with Graph Embeddings

DATA SCIENCE TRAINING IN CHENNAI

Chinese Character Decomposition for Neural MT with Multi-Word Expressions

Viewers also liked

MTEngine: Semantic-level Crowdsourced Machine Translation

Dmitry Kan

SentiScan: система автоматической разметки тональности в social media

Dmitry Kan

Social spam detection by SemanticAnalyzer GroupDmitry Kan

Lucene revolution eu 2013 dublin writeup

Dmitry Kan

Automatic Build Of Semantic Translational DictionaryDmitry Kan

Machine translation course program (in English)

Dmitry Kan

Introduction To Machine Translation 1Dmitry Kan

Linguistic component Sentiment Analyzer for the Russian language

Dmitry Kan

Semantic feature machine translation systemDmitry Kan

Solr onfitnesse learningfromberlinbuzzwordsDmitry Kan

Starget sentiment analyzer for English

Dmitry Kan

Linguistic component Lemmatizer for the Russian language

Dmitry Kan

Introduction To Machine TranslationDmitry Kan

NoSQL, Apache SOLR and Apache Hadoop

Dmitry Kan

Rule based approach to sentiment analysis at ROMIP 2011Dmitry Kan

Poster: Method for an automatic generation of a semantic-level contextual tra...Dmitry Kan

Linguistic component Tokenizer for the Russian language

Dmitry Kan

Rule based approach to sentiment analysis at romip’11 slides

Dmitry Kan

Semantic Analysis: theory, applications and use cases

Dmitry Kan

Proformas informatica

yarlin04

Viewers also liked (20)

MTEngine: Semantic-level Crowdsourced Machine Translation

SentiScan: система автоматической разметки тональности в social media

Social spam detection by SemanticAnalyzer Group

Lucene revolution eu 2013 dublin writeup

Automatic Build Of Semantic Translational Dictionary

Machine translation course program (in English)

Introduction To Machine Translation 1

Linguistic component Sentiment Analyzer for the Russian language

Semantic feature machine translation system

Solr onfitnesse learningfromberlinbuzzwords

Starget sentiment analyzer for English

Linguistic component Lemmatizer for the Russian language

Introduction To Machine Translation

NoSQL, Apache SOLR and Apache Hadoop

Rule based approach to sentiment analysis at ROMIP 2011

Poster: Method for an automatic generation of a semantic-level contextual tra...

Linguistic component Tokenizer for the Russian language

Rule based approach to sentiment analysis at romip’11 slides

Semantic Analysis: theory, applications and use cases

Proformas informatica

Similar to IR: Open source state

SG_UserGroup_Oct20_2022_NLP_AzureLangStudio.pptx

PriyankaShah668821

Nirdesh_Developer_2.0_Years_6_months_ExpNirdesh Kulshreshtha

Gdsc IIIT Surat Orientation 2022.pdf

SparshJhariya2

From Academic Papers To Production : A Learning To Rank Story

Alessandro Benedetti

This talk is about the journey to bring Learning To Rank (LTR from now on) to the e-commerce domain in a real world scenario, including all the pitfalls and disillutions involved. LTR is a fantastic approach to solve complex ranking problems but industry domains are far from being the ideal world where those technologies were designed and experimented : open source software implementations are not working perfectly out of the box and require advanced tuning; industry training data is dirty, noisy and incomplete. This talk will guide you through the different phases and technologies involved in a LTR project with a pragmatic approach. Feature Engineering, Domain Modelling, Training Set Building, Model Training, Search Integration and Online Evaluation : each of them presents different challenges in the real world and must be carefully approached.

Sudipta_Mukherjee_Resume-Nov_2022.pdf

Sudipta Mukherjee

Resume (2)Romy Khetan

RohitSharmaResumeRohit Sharma

Sudipta mukherjee 2016_2017

Sudipta Mukherjee

Sudipta_Mukherjee_2016_2017Sudipta Mukherjee

Updated Ayushi Tongiya Resume(1)Ayushi Tongiya

ResumeAmanRajJuly2016Aman Raj

TejasveeBolisettyTejasvee Bolisetty

Collaborative Data Analysis with Taverna Workflows

Andrea Wiggins

Shrey_Kumar_Resume_01072016Shrey Kumar

Radhakrishnan MoniRadhakrishnan Moni

Discovering Emerging Tech through Graph Analysis - Henry Hwangbo @ GraphConne...

Neo4j

With the torrent of data available to us on the Internet, it's been increasingly difficult to separate the signal from the noise. We set out on a journey with a simple directive: Figure out a way to discover emerging technology trends. Through a series of experiments, trials, and pivots, we found our answer in the power of graph databases. We essentially built our "Emerging Tech Radar" on emerging technologies with graph databases being central to our discovery platform. Using a mix of NoSQL databases and open source libraries we built a scalable information digestion platform which touches upon multiple topics such as NLP, named entity extraction, data cleansing, cypher queries, multiple visualizations, and polymorphic persistence.

Resume sb

Surbhi Bhatnagar

Nirbhay SinghNirbhay Singh

Autopsy 3.0 - Open Source Digital Forensics Conference

Basis Technology

Autopsy 3.0 is a complete rewrite from Autopsy 2.0, and this talk will cover all of the things that are new about it. Multi-threaded ingest, triage, embedded databases, web artifact analysis, and indexed keyword search are just some of the new and exciting features. This talk is targeted towards both users and developers. Users will learn about the tool, and how they can use it. Developers will learn the basics of where they can incorporate their tools into the Autopsy workflow as modules. View more slides from the Open Source Digital Forensics Conference 2012 here: http://info.basistech.com/osdf-2012-slides

PriyankaDighe_Resume_newPriyanka Dighe

Similar to IR: Open source state (20)

SG_UserGroup_Oct20_2022_NLP_AzureLangStudio.pptx

Nirdesh_Developer_2.0_Years_6_months_Exp

Gdsc IIIT Surat Orientation 2022.pdf

From Academic Papers To Production : A Learning To Rank Story

Sudipta_Mukherjee_Resume-Nov_2022.pdf

Resume (2)

RohitSharmaResume

Sudipta mukherjee 2016_2017

Sudipta_Mukherjee_2016_2017

Updated Ayushi Tongiya Resume(1)

ResumeAmanRajJuly2016

TejasveeBolisetty

Collaborative Data Analysis with Taverna Workflows

Shrey_Kumar_Resume_01072016

Radhakrishnan Moni

Discovering Emerging Tech through Graph Analysis - Henry Hwangbo @ GraphConne...

Resume sb

Nirbhay Singh

Autopsy 3.0 - Open Source Digital Forensics Conference

PriyankaDighe_Resume_new

Recently uploaded

Neuro-symbolic is not enough, we need neuro-*semantic*

Frank van Harmelen

Neuro-symbolic (NeSy) AI is on the rise. However, simply machine learning on just any symbolic structure is not sufficient to really harvest the gains of NeSy. These will only be gained when the symbolic structures have an actual semantics. I give an operational definition of semantics as “predictable inference”. All of this illustrated with link prediction over knowledge graphs, but the argument is general.

Key Trends Shaping the Future of Infrastructure.pdf

Cheryl Hung

FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf

FIDO Alliance

UiPath Test Automation using UiPath Test Suite series, part 3

DianaGray10

Leading Change strategies and insights for effective change management pdf 1.pdf

OnBoard

GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...

James Anderson

Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management. The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM). Speakers: Bob Boule Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle. Gopinath Rebala Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.

When stars align: studies in data quality, knowledge graphs, and machine lear...

Elena Simperl

LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...

DanBrown980551

Do you want to learn how to model and simulate an electrical network from scratch in under an hour? Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)! During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook. PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides: - A fully editable and extendable library for grid component modelling; - Visualization tools to display your network; - Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses; The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well. What you will learn during the webinar: - For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills; - For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.

To Graph or Not to Graph Knowledge Graph Architectures and LLMs

Paul Groth

De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...

Product School

Monitoring Java Application Security with JDK Tools and JFR Events

Ana-Maria Mihalceanu

GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...

Sri Ambati

FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf

FIDO Alliance

Connector Corner: Automate dynamic content and events by pushing a button

DianaGray10

Here is something new! In our next Connector Corner webinar, we will demonstrate how you can use a single workflow to: Create a campaign using Mailchimp with merge tags/fields Send an interactive Slack channel message (using buttons) Have the message received by managers and peers along with a test email for review But there’s more: In a second workflow supporting the same use case, you’ll see: Your campaign sent to target colleagues for approval If the “Approve” button is clicked, a Jira/Zendesk ticket is created for the marketing design team But—if the “Reject” button is pushed, colleagues will be alerted via Slack message Join us to learn more about this new, human-in-the-loop capability, brought to you by Integration Service connectors. And... Speakers: Akshay Agnihotri, Product Manager Charlie Greenberg, Host

UiPath Test Automation using UiPath Test Suite series, part 4

DianaGray10

Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap. The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies. Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques What will you get from this session? 1. Insights into SAP testing best practices 2. Heatmap utilization for testing 3. Optimization of testing processes 4. Demo Topics covered: Execution from the test manager Orchestrator execution result Defect reporting SAP heatmap example with demo Speaker: Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP

From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...

Product School

Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024

Tobias Schneck

As AI technology is pushing into IT I was wondering myself, as an “infrastructure container kubernetes guy”, how get this fancy AI technology get managed from an infrastructure operational view? Is it possible to apply our lovely cloud native principals as well? What benefit’s both technologies could bring to each other? Let me take this questions and provide you a short journey through existing deployment models and use cases for AI software. On practical examples, we discuss what cloud/on-premise strategy we may need for applying it to our own infrastructure to get it to work from an enterprise perspective. I want to give an overview about infrastructure requirements and technologies, what could be beneficial or limiting your AI use cases in an enterprise environment. An interactive Demo will give you some insides, what approaches I got already working for real.

The Future of Platform Engineering

Jemma Hussein Allen

DevOps and Testing slides at DASA Connect

Kari Kakkonen

FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf

FIDO Alliance

Recently uploaded (20)