This document discusses text mining and fuzzy document classification using Elasticsearch. It describes building document class vectors for job roles like "Java programmer" and ".NET programmer" that include skills and technologies weighted by importance. The document demonstrates classifying an unknown document by tokenizing it, comparing the tokens to classification vectors, and calculating cosine similarity distances to classify the document.

1. Text mining
Fuzzy document classification
Using Elasticsearch
Lev Ozeryansky

2. Identity Card
• Merging of Ankor and We!
• Owned by Hilan (publicly traded in Tel Aviv Stock exchange)
• Fast growing IT integration company
• Over 2000 systems installed and maintained
• Over 1000 leading customers - Hi-tech, Industry, Academy, Banks,
Insurance,
• Strong technological team – over 45 engineers, professional services
and project managers
• Over 120 employees
• Four main divisions – Infrastructure, Big Data, Cloud, Cyber

3. Technology Edge

4. What is classification
• Document classification as document categorization.
• Using classification.
• Our classification data source.
• What we do with?
• Java programmer.
• .NET programmer.

5. Data source

6. The mathematics
• Let be class set
• Let be documents set
• Classification function

7. Classification method
• Cosine similarity
• Function

8. Build document class
vector
• Java programmer
• Java
• 5
• Hibernate
• .NET programmer
• C#
• 5
• Nhibernate

9. Let index classificators
• Add weight manually.
• For Java programmer:
• Java = 0.7
• 5 = 0.5
• Hibernate = 0.3
• For .NET programmer
• C# = 0.7
• 5 = 0.5
• Nhibernate = 0.3

10. DEMO

11. w-shingling
• In natural language processing a w-shingling is a set of
unique "shingles"—contiguous subsequences of tokens in
a document. (Wikipedia)
• Tokenization
• Elasticsearch analyze mechanism

12. DEMO

13. Classification process
• Tokens array.
• Classification query.
• Use terms query when terms array == tokens array
• Two vectors
• Vector of filtered tokens
• Classification vector

14. DEMO

15. Classification process
• SciPy to calculate distance.

16. Q&A