International Journal of Engineering and Science Invention (IJESI) is an international journal intended for professionals and researchers in all fields of computer science and electronics. IJESI publishes research articles and reviews within the whole field Engineering Science and Technology, new teaching methods, assessment, validation and the impact of new technologies and it will continue to provide information on the latest trends and developments in this ever-expanding subject. The publications of papers are selected through double peer reviewed to ensure originality, relevance, and readability. The articles published in our journal can be accessed online.
An Improved Similarity Matching based Clustering Framework for Short and Sent...IJECEIAES
Text clustering plays a key role in navigation and browsing process. For an efficient text clustering, the large amount of information is grouped into meaningful clusters. Multiple text clustering techniques do not address the issues such as, high time and space complexity, inability to understand the relational and contextual attributes of the word, less robustness, risks related to privacy exposure, etc. To address these issues, an efficient text based clustering framework is proposed. The Reuters dataset is chosen as the input dataset. Once the input dataset is preprocessed, the similarity between the words are computed using the cosine similarity. The similarities between the components are compared and the vector data is created. From the vector data the clustering particle is computed. To optimize the clustering results, mutation is applied to the vector data. The performance the proposed text based clustering framework is analyzed using the metrics such as Mean Square Error (MSE), Peak Signal Noise Ratio (PSNR) and Processing time. From the experimental results, it is found that, the proposed text based clustering framework produced optimal MSE, PSNR and processing time when compared to the existing Fuzzy C-Means (FCM) and Pairwise Random Swap (PRS) methods.
A Novel Multi- Viewpoint based Similarity Measure for Document ClusteringIJMER
International Journal of Modern Engineering Research (IJMER) is Peer reviewed, online Journal. It serves as an international archival forum of scholarly research related to engineering and science education.
International Journal of Modern Engineering Research (IJMER) covers all the fields of engineering and science: Electrical Engineering, Mechanical Engineering, Civil Engineering, Chemical Engineering, Computer Engineering, Agricultural Engineering, Aerospace Engineering, Thermodynamics, Structural Engineering, Control Engineering, Robotics, Mechatronics, Fluid Mechanics, Nanotechnology, Simulators, Web-based Learning, Remote Laboratories, Engineering Design Methods, Education Research, Students' Satisfaction and Motivation, Global Projects, and Assessment…. And many more.
Correlation Preserving Indexing Based Text ClusteringIOSR Journals
This document discusses a correlation preserving indexing (CPI) based text clustering method. CPI aims to find a low dimensional semantic subspace that maximizes correlation between similar documents while minimizing correlation between dissimilar documents. It is different from other methods like LSI and LPI that use Euclidean distance. The document outlines the CPI method and evaluates it on document clustering tasks, showing it doubles the accuracy of previous correlation-based methods. Hierarchical clustering algorithms are also discussed and compared to CPI in terms of evaluation metrics.
Privacy Preserving MFI Based Similarity Measure For Hierarchical Document Clu...IJORCS
The document proposes a privacy-preserving approach for hierarchical document clustering using maximal frequent item sets (MFI). First, MFI are identified from document collections using the Apriori algorithm to define clusters precisely. Then, the same MFI-based similarity measure is used to construct a hierarchy of clusters. This approach decreases dimensionality and avoids duplicate documents, thereby protecting individual copyrights. The methodology and algorithm are described in detail.
An Enhanced Suffix Tree Approach to Measure Semantic Similarity between Multi...iosrjce
1) The document discusses an approach to measure semantic similarity between multiple documents using an enhanced suffix tree. It involves preprocessing documents, constructing a suffix tree with documents' phrases as edges, calculating weights of shared nodes using TF-IDF, and applying cosine, dice, and hellinger similarity measures to determine pairwise document similarities.
2) The approach first preprocesses documents by removing stop words, special characters, and converting to lowercase. A suffix tree is constructed with documents' phrases as edges. Shared nodes in the tree represent common phrases between documents.
3) Node weights are calculated using TF-IDF, with higher weights given to rarer phrases. Several similarity measures (cosine, dice, hellinger) are then applied
This document summarizes a research paper that introduces a novel multi-viewpoint similarity measure for clustering text documents. The paper begins with background on commonly used similarity measures like Euclidean distance and cosine similarity. It then presents the novel multi-viewpoint measure, which considers multiple viewpoints (objects not assumed to be in the same cluster) rather than a single viewpoint. The paper proposes two new clustering criterion functions based on this measure and compares them to other algorithms on benchmark datasets. The goal is to develop a similarity measure and clustering methods that provide high-quality, consistent performance like k-means but can better handle sparse, high-dimensional text data.
Clustering the results of a search helps the user to overview the information returned. In this paper, we
look upon the clustering task as cataloguing the search results. By catalogue we mean a structured label
list that can help the user to realize the labels and search results. Labelling Cluster is crucial because
meaningless or confusing labels may mislead users to check wrong clusters for the query and lose extra
time. Additionally, labels should reflect the contents of documents within the cluster accurately. To be able
to label clusters effectively, a new cluster labelling method is introduced. More emphasis was given to
/produce comprehensible and accurate cluster labels in addition to the discovery of document clusters. We
also present a new metric that employs to assess the success of cluster labelling. We adopt a comparative
evaluation strategy to derive the relative performance of the proposed method with respect to the two
prominent search result clustering methods: Suffix Tree Clustering and Lingo.
we perform the experiments using the publicly available Datasets Ambient and ODP-239
O NTOLOGY B ASED D OCUMENT C LUSTERING U SING M AP R EDUCE ijdms
Nowadays, document clustering is considered as a da
ta intensive task due to the dramatic, fast increas
e in
the number of available documents. Nevertheless, th
e features that represent those documents are also
too
large. The most common method for representing docu
ments is the vector space model, which represents
document features as a bag of words and does not re
present semantic relations between words. In this
paper we introduce a distributed implementation for
the bisecting k-means using MapReduce programming
model. The aim behind our proposed implementation i
s to solve the problem of clustering intensive data
documents. In addition, we propose integrating the
WordNet ontology with bisecting k-means in order to
utilize the semantic relations between words to enh
ance document clustering results. Our presented
experimental results show that using lexical catego
ries for nouns only enhances internal evaluation
measures of document clustering; and decreases the
documents features from thousands to tens features.
Our experiments were conducted using Amazon ElasticMapReduce to deploy the Bisecting k-means
algorithm
An Improved Similarity Matching based Clustering Framework for Short and Sent...IJECEIAES
Text clustering plays a key role in navigation and browsing process. For an efficient text clustering, the large amount of information is grouped into meaningful clusters. Multiple text clustering techniques do not address the issues such as, high time and space complexity, inability to understand the relational and contextual attributes of the word, less robustness, risks related to privacy exposure, etc. To address these issues, an efficient text based clustering framework is proposed. The Reuters dataset is chosen as the input dataset. Once the input dataset is preprocessed, the similarity between the words are computed using the cosine similarity. The similarities between the components are compared and the vector data is created. From the vector data the clustering particle is computed. To optimize the clustering results, mutation is applied to the vector data. The performance the proposed text based clustering framework is analyzed using the metrics such as Mean Square Error (MSE), Peak Signal Noise Ratio (PSNR) and Processing time. From the experimental results, it is found that, the proposed text based clustering framework produced optimal MSE, PSNR and processing time when compared to the existing Fuzzy C-Means (FCM) and Pairwise Random Swap (PRS) methods.
A Novel Multi- Viewpoint based Similarity Measure for Document ClusteringIJMER
International Journal of Modern Engineering Research (IJMER) is Peer reviewed, online Journal. It serves as an international archival forum of scholarly research related to engineering and science education.
International Journal of Modern Engineering Research (IJMER) covers all the fields of engineering and science: Electrical Engineering, Mechanical Engineering, Civil Engineering, Chemical Engineering, Computer Engineering, Agricultural Engineering, Aerospace Engineering, Thermodynamics, Structural Engineering, Control Engineering, Robotics, Mechatronics, Fluid Mechanics, Nanotechnology, Simulators, Web-based Learning, Remote Laboratories, Engineering Design Methods, Education Research, Students' Satisfaction and Motivation, Global Projects, and Assessment…. And many more.
Correlation Preserving Indexing Based Text ClusteringIOSR Journals
This document discusses a correlation preserving indexing (CPI) based text clustering method. CPI aims to find a low dimensional semantic subspace that maximizes correlation between similar documents while minimizing correlation between dissimilar documents. It is different from other methods like LSI and LPI that use Euclidean distance. The document outlines the CPI method and evaluates it on document clustering tasks, showing it doubles the accuracy of previous correlation-based methods. Hierarchical clustering algorithms are also discussed and compared to CPI in terms of evaluation metrics.
Privacy Preserving MFI Based Similarity Measure For Hierarchical Document Clu...IJORCS
The document proposes a privacy-preserving approach for hierarchical document clustering using maximal frequent item sets (MFI). First, MFI are identified from document collections using the Apriori algorithm to define clusters precisely. Then, the same MFI-based similarity measure is used to construct a hierarchy of clusters. This approach decreases dimensionality and avoids duplicate documents, thereby protecting individual copyrights. The methodology and algorithm are described in detail.
An Enhanced Suffix Tree Approach to Measure Semantic Similarity between Multi...iosrjce
1) The document discusses an approach to measure semantic similarity between multiple documents using an enhanced suffix tree. It involves preprocessing documents, constructing a suffix tree with documents' phrases as edges, calculating weights of shared nodes using TF-IDF, and applying cosine, dice, and hellinger similarity measures to determine pairwise document similarities.
2) The approach first preprocesses documents by removing stop words, special characters, and converting to lowercase. A suffix tree is constructed with documents' phrases as edges. Shared nodes in the tree represent common phrases between documents.
3) Node weights are calculated using TF-IDF, with higher weights given to rarer phrases. Several similarity measures (cosine, dice, hellinger) are then applied
This document summarizes a research paper that introduces a novel multi-viewpoint similarity measure for clustering text documents. The paper begins with background on commonly used similarity measures like Euclidean distance and cosine similarity. It then presents the novel multi-viewpoint measure, which considers multiple viewpoints (objects not assumed to be in the same cluster) rather than a single viewpoint. The paper proposes two new clustering criterion functions based on this measure and compares them to other algorithms on benchmark datasets. The goal is to develop a similarity measure and clustering methods that provide high-quality, consistent performance like k-means but can better handle sparse, high-dimensional text data.
Clustering the results of a search helps the user to overview the information returned. In this paper, we
look upon the clustering task as cataloguing the search results. By catalogue we mean a structured label
list that can help the user to realize the labels and search results. Labelling Cluster is crucial because
meaningless or confusing labels may mislead users to check wrong clusters for the query and lose extra
time. Additionally, labels should reflect the contents of documents within the cluster accurately. To be able
to label clusters effectively, a new cluster labelling method is introduced. More emphasis was given to
/produce comprehensible and accurate cluster labels in addition to the discovery of document clusters. We
also present a new metric that employs to assess the success of cluster labelling. We adopt a comparative
evaluation strategy to derive the relative performance of the proposed method with respect to the two
prominent search result clustering methods: Suffix Tree Clustering and Lingo.
we perform the experiments using the publicly available Datasets Ambient and ODP-239
O NTOLOGY B ASED D OCUMENT C LUSTERING U SING M AP R EDUCE ijdms
Nowadays, document clustering is considered as a da
ta intensive task due to the dramatic, fast increas
e in
the number of available documents. Nevertheless, th
e features that represent those documents are also
too
large. The most common method for representing docu
ments is the vector space model, which represents
document features as a bag of words and does not re
present semantic relations between words. In this
paper we introduce a distributed implementation for
the bisecting k-means using MapReduce programming
model. The aim behind our proposed implementation i
s to solve the problem of clustering intensive data
documents. In addition, we propose integrating the
WordNet ontology with bisecting k-means in order to
utilize the semantic relations between words to enh
ance document clustering results. Our presented
experimental results show that using lexical catego
ries for nouns only enhances internal evaluation
measures of document clustering; and decreases the
documents features from thousands to tens features.
Our experiments were conducted using Amazon ElasticMapReduce to deploy the Bisecting k-means
algorithm
8 efficient multi-document summary generation using neural networkINFOGAIN PUBLICATION
This paper proposes a multi-document summarization system that uses bisect k-means clustering, an optimal merge function, and a neural network. The system first preprocesses input documents through stemming and removing stop words. It then applies bisect k-means clustering to group similar sentences. The clusters are merged using an optimal merge function to find important keywords. The NEWSUM algorithm is used to generate a primary summary for each keyword. A neural network trained on sentence classifications is then used to classify sentences in the primary summary as positive or negative. Only positively classified sentences are included in the final summary to improve accuracy. The system aims to generate a concise and accurate summary in a short period of time from multiple documents on a given topic.
SOURCE CODE RETRIEVAL USING SEQUENCE BASED SIMILARITYIJDKP
This document summarizes an approach to improve source code retrieval using structural information from source code. A lexical parser is developed to extract control statements and method identifiers from Java programs. A similarity measure is proposed that calculates the ratio of fully matching statements to partially matching statements in a sequence. Experiments show the retrieval model using this measure improves retrieval performance over other models by up to 90.9% relative to the number of retrieved methods.
USING ONTOLOGIES TO IMPROVE DOCUMENT CLASSIFICATION WITH TRANSDUCTIVE SUPPORT...IJDKP
Many applications of automatic document classification require learning accurately with little training
data. The semi-supervised classification technique uses labeled and unlabeled data for training. This
technique has shown to be effective in some cases; however, the use of unlabeled data is not always
beneficial.
On the other hand, the emergence of web technologies has originated the collaborative development of
ontologies. In this paper, we propose the use of ontologies in order to improve the accuracy and efficiency
of the semi-supervised document classification.
We used support vector machines, which is one of the most effective algorithms that have been studied for
text. Our algorithm enhances the performance of transductive support vector machines through the use of
ontologies. We report experimental results applying our algorithm to three different datasets. Our
experiments show an increment of accuracy of 4% on average and up to 20%, in comparison with the
traditional semi-supervised model.
Data mining , knowledge discovery is the process
of analyzing data from different perspectives and summarizing it
into useful information - information that can be used to increase
revenue, cuts costs, or both. Data mining software is one of a
number of analytical tools for analyzing data. It allows users to
analyze data from many different dimensions or angles, categorize
it, and summarize the relationships identified. Technically, data
mining is the process of finding correlations or patterns among
dozens of fields in large relational databases. The goal of
clustering is to determine the intrinsic grouping in a set of
unlabeled data. But how to decide what constitutes a good
clustering? It can be shown that there is no absolute “best”
criterion which would be independent of the final aim of the
clustering. Consequently, it is the user which must supply this
criterion, in such a way that the result of the clustering will suit
their needs.
For instance, we could be interested in finding
representatives for homogeneous groups (data reduction), in
finding “natural clusters” and describe their unknown properties
(“natural” data types), in finding useful and suitable groupings
(“useful” data classes) or in finding unusual data objects (outlier
detection).Of late, clustering techniques have been applied in the
areas which involve browsing the gathered data or in categorizing
the outcome provided by the search engines for the reply to the
query raised by the users. In this paper, we are providing a
comprehensive survey over the document clustering.
A Comparative Study of Centroid-Based and Naïve Bayes Classifiers for Documen...IJERA Editor
Assigning documents to related categories is critical task which is used for effective document retrieval. Automatic text classification is the process of assigning new text document to the predefined categories based on its content. In this paper, we implemented and performed comparison of Naïve Bayes and Centroid-based algorithms for effective document categorization of English language text. In Centroid Based algorithm, we used Arithmetical Average Centroid (AAC) and Cumuli Geometric Centroid (CGC) methods to calculate centroid of each class. Experiment is performed on R-52 dataset of Reuters-21578 corpus. Micro Average F1 measure is used to evaluate the performance of classifiers. Experimental results show that Micro Average F1 value for NB is greatest among all followed by Micro Average F1 value of CGC which is greater than Micro Average F1 of AAC. All these results are valuable for future research
With the rapid development in Geographic Information Systems (GISs) and their applications, more and
more geo-graphical databases have been developed by different vendors. However, data integration and
accessing is still a big problem for the development of GIS applications as no interoperability exists among
different spatial databases. In this paper we propose a unified approach for spatial data query. The paper
describes a framework for integrating information from repositories containing different vector data sets
formats and repositories containing raster datasets. The presented approach converts different vector data
formats into a single unified format (File Geo-Database “GDB”). In addition, we employ “metadata” to
support a wide range of users’ queries to retrieve relevant geographic information from heterogeneous and
distributed repositories. Such an employment enhances both query processing and performance.
International Journal of Engineering Research and Applications (IJERA) is an open access online peer reviewed international journal that publishes research and review articles in the fields of Computer Science, Neural Networks, Electrical Engineering, Software Engineering, Information Technology, Mechanical Engineering, Chemical Engineering, Plastic Engineering, Food Technology, Textile Engineering, Nano Technology & science, Power Electronics, Electronics & Communication Engineering, Computational mathematics, Image processing, Civil Engineering, Structural Engineering, Environmental Engineering, VLSI Testing & Low Power VLSI Design etc.
Hierarchal clustering and similarity measures along with multi representationeSAT Journals
Abstract All clustering methods have to assume some cluster relationship on the list of data objects that they really are applied on. Graph-Based Document Clustering works with frequent senses rather than frequent keywords used in traditional text mining techniques.Similarity between a pair of objects can be defined either explicitly or implicitly. With this paper, we analyzed existing multi-viewpoint based similarity measure and two related clustering methods. The main difference between a traditional dissimilarity/similarity measure and ours could be that the former uses merely a single viewpoint, which is the origin, even though the latter utilizes many viewpoints, which you ll find are objects assumed to not have the very same cluster using the two objects being measured. Using multiple viewpoints, more informative assessment of similarity could well be achieved. Theoretical analysis and empirical study are conducted to back up this claim. Two criterion functions for document clustering are proposed dependent on this wonderful measure. We compare them several well-known clustering algorithms which use other popular similarity measures on various document collections confirming the good sides of our proposal. Keywords –Multiview Cluster, Document id, ClusterDistance
This document summarizes several papers on document clustering techniques. It discusses hierarchical clustering and similarity measures, as well as multi-representation clustering. Several clustering algorithms are examined, including K-means clustering and graph-based clustering. The document also analyzes similarity measures like multi-viewpoint similarity and evaluates the performance of different clustering methods on document collections.
With ever increasing number of documents on web and other repositories, the task of organizing and
categorizing these documents to the diverse need of the user by manual means is a complicated job, hence
a machine learning technique named clustering is very useful. Text documents are clustered by pair wise
similarity of documents with similarity measures like Cosine, Jaccard or Pearson. Best clustering results
are seen when overlapping of terms in documents is less, that is, when clusters are distinguishable. Hence
for this problem, to find document similarity we apply link and neighbor introduced in ROCK. Link
specifies number of shared neighbors of a pair of documents. Significantly similar documents are called as
neighbors. This work applies links and neighbors to Bisecting K-means clustering in identifying seed
documents in the dataset, as a heuristic measure in choosing a cluster to be partitioned and as a means to
find the number of partitions possible in the dataset. Our experiments on real-time datasets showed a
significant improvement in terms of accuracy with minimum time.
This document discusses GCUBE indexing, which is a method for indexing and aggregating spatial/continuous values in a data warehouse. The key challenges addressed are defining and aggregating spatial/continuous values, and efficiently representing, indexing, updating and querying data that includes both categorical and continuous dimensions. The proposed GCUBE approach maps multi-dimensional data to a linear ordering using the Hilbert curve, and then constructs an index structure on the ordered data to enable efficient query processing. Empirical results show the GCUBE indexing offers significant performance advantages over alternative approaches.
1) The document discusses different clustering algorithms for text summarization including hierarchical clustering, query-based summarization, graph theoretic clustering, fuzzy c-means clustering, and DBSCAN clustering.
2) These algorithms are evaluated based on performance parameters like precision, recall, time complexity, space complexity, and summary quality.
3) The algorithm found to perform best based on these evaluations will be suggested as the better algorithm for query-dependent text document summarization.
International Journal of Engineering Research and Applications (IJERA) is a team of researchers not publication services or private publications running the journals for monetary benefits, we are association of scientists and academia who focus only on supporting authors who want to publish their work. The articles published in our journal can be accessed online, all the articles will be archived for real time access.
Our journal system primarily aims to bring out the research talent and the works done by sciaentists, academia, engineers, practitioners, scholars, post graduate students of engineering and science. This journal aims to cover the scientific research in a broader sense and not publishing a niche area of research facilitating researchers from various verticals to publish their papers. It is also aimed to provide a platform for the researchers to publish in a shorter of time, enabling them to continue further All articles published are freely available to scientific researchers in the Government agencies,educators and the general public. We are taking serious efforts to promote our journal across the globe in various ways, we are sure that our journal will act as a scientific platform for all researchers to publish their works online.
A rough set based hybrid method to text categorizationNinad Samel
This document summarizes a hybrid text categorization method that combines Latent Semantic Indexing (LSI) and Rough Sets theory to reduce the dimensionality of text data and generate classification rules. It introduces LSI to reduce the feature space of text documents represented as high-dimensional vectors. Then it applies Rough Sets theory to the reduced feature space to locate a minimal set of keywords that can distinguish document classes and generate multiple knowledge bases for classification instead of a single one. The method is tested on text categorization tasks and shown to improve accuracy over previous Rough Sets approaches.
This document describes a proposed concept-based mining model that aims to improve document clustering and information retrieval by extracting concepts and semantic relationships rather than just keywords. The model uses natural language processing techniques like part-of-speech tagging and parsing to extract concepts from text. It represents concepts and their relationships in a semantic network and clusters documents based on conceptual similarity rather than term frequency. The model is evaluated using singular value decomposition to increase the precision of key term and phrase extraction.
New proximity estimate for incremental update of non uniformly distributed cl...IJDKP
The conventional clustering algorithms mine static databases and generate a set of patterns in the form of
clusters. Many real life databases keep growing incrementally. For such dynamic databases, the patterns
extracted from the original database become obsolete. Thus the conventional clustering algorithms are not
suitable for incremental databases due to lack of capability to modify the clustering results in accordance
with recent updates. In this paper, the author proposes a new incremental clustering algorithm called
CFICA(Cluster Feature-Based Incremental Clustering Approach for numerical data) to handle numerical
data and suggests a new proximity metric called Inverse Proximity Estimate (IPE) which considers the
proximity of a data point to a cluster representative as well as its proximity to a farthest point in its vicinity.
CFICA makes use of the proposed proximity metric to determine the membership of a data point into a
cluster.
Complete agglomerative hierarchy document’s clustering based on fuzzy luhn’s ...IJECEIAES
Agglomerative hierarchical is a bottom up clustering method, where the distances between documents can be retrieved by extracting feature values using a topic-based latent dirichlet allocation method. To reduce the number of features, term selection can be done using Luhn’s Idea. Those methods can be used to build the better clusters for document. But, there is less research discusses it. Therefore, in this research, the term weighting calculation uses Luhn’s Idea to select the terms by defining upper and lower cut-off, and then extracts the feature of terms using gibbs sampling latent dirichlet allocation combined with term frequency and fuzzy Sugeno method. The feature values used to be the distance between documents, and clustered with single, complete and average link algorithm. The evaluations show the feature extraction with and without lower cut-off have less difference. But, the topic determination of each term based on term frequency and fuzzy Sugeno method is better than Tsukamoto method in finding more relevant documents. The used of lower cut-off and fuzzy Sugeno gibbs latent dirichlet allocation for complete agglomerative hierarchical clustering have consistent metric values. This clustering method suggested as a better method in clustering documents that is more relevant to its gold standard.
A CONCEPTUAL METADATA FRAMEWORK FOR SPATIAL DATA WAREHOUSEIJDKP
Metadata represents the information about data to be stored in Data Warehouses. It is a mandatory
element of Data Warehouse to build an efficient Data Warehouse. Metadata helps in data integration,
lineage, data quality and populating transformed data into data warehouse. Spatial data warehouses are
based on spatial data mostly collected from Geographical Information Systems (GIS) and the transactional
systems that are specific to an application or enterprise. Metadata design and deployment is the most
critical phase in building of data warehouse where it is mandatory to bring the spatial information and
data modeling together. In this paper, we present a holistic metadata framework that drives metadata
creation for spatial data warehouse. Theoretically, the proposed metadata framework improves the
efficiency of accessing of data in response to frequent queries on SDWs. In other words, the proposed
framework decreases the response time of the query and accurate information is fetched from Data
Warehouse including the spatial information
Text document clustering and similarity detection is the major part of document management, where every document should be identified by its key terms and domain knowledge. Based on the similarity, the documents are grouped into clusters. For document similarity calculation there are several approaches were proposed in the existing system. But the existing system is either term based or pattern based. And those systems suffered from several problems. To make a revolution in this challenging environment, the proposed system presents an innovative model for document similarity by applying back propagation time stamp algorithm. It discovers patterns in text documents as higher level features and creates a network for fast grouping. It also detects the most appropriate patterns based on its weight and BPTT performs the document similarity measures. Using this approach, the document can be categorized easily. In order to perform the above, a new approach is used. This helps to reduce the training process problems. The above framework is named as BPTT. The BPTT has implemented and evaluated using dot net platform with different set of datasets.
SEARCH OF INFORMATION BASED CONTENT IN SEMI-STRUCTURED DOCUMENTS USING INTERF...ijcsitcejournal
This paper proposes a semi-structured information retrieval model based on a new method for calculation
of similarity. We have developed CASISS (Calculation of Similarity of Semi-Structured documents)
method to quantify how two given texts are similar. This new method identifies elements of semi-structured
documents using elements descriptors. Each semi-structured document is pre-processed before the
extraction of a set of descriptors for each element, which characterize the contents of elements.It can be
used to increase the accuracy of the information retrieval process by taking into account not only the
presence of query terms in the given document but also the topology (position continuity) of these terms.
This document summarizes LinkedIn's approach to personalizing users' news feeds to improve relevance and address scalability challenges. It describes generating affinity scores for viewer-activity type, viewer-actor, and their combination. These scores are computed offline and stored to quickly retrieve for online scoring, avoiding computation overhead. Experiments show personalization improves click-through rates.
This document summarizes a research paper that proposes two approaches called LLORMA for constructing local low-rank matrix approximations. The approaches approximate an observed matrix M as a weighted sum of several low-rank matrices, where each low-rank matrix is accurate in a local region of M. This relaxes the assumption that M has global low-rank structure. The paper analyzes the accuracy of the local low-rank modeling approaches and shows they improve prediction accuracy over classical low-rank approximation methods on recommendation tasks.
8 efficient multi-document summary generation using neural networkINFOGAIN PUBLICATION
This paper proposes a multi-document summarization system that uses bisect k-means clustering, an optimal merge function, and a neural network. The system first preprocesses input documents through stemming and removing stop words. It then applies bisect k-means clustering to group similar sentences. The clusters are merged using an optimal merge function to find important keywords. The NEWSUM algorithm is used to generate a primary summary for each keyword. A neural network trained on sentence classifications is then used to classify sentences in the primary summary as positive or negative. Only positively classified sentences are included in the final summary to improve accuracy. The system aims to generate a concise and accurate summary in a short period of time from multiple documents on a given topic.
SOURCE CODE RETRIEVAL USING SEQUENCE BASED SIMILARITYIJDKP
This document summarizes an approach to improve source code retrieval using structural information from source code. A lexical parser is developed to extract control statements and method identifiers from Java programs. A similarity measure is proposed that calculates the ratio of fully matching statements to partially matching statements in a sequence. Experiments show the retrieval model using this measure improves retrieval performance over other models by up to 90.9% relative to the number of retrieved methods.
USING ONTOLOGIES TO IMPROVE DOCUMENT CLASSIFICATION WITH TRANSDUCTIVE SUPPORT...IJDKP
Many applications of automatic document classification require learning accurately with little training
data. The semi-supervised classification technique uses labeled and unlabeled data for training. This
technique has shown to be effective in some cases; however, the use of unlabeled data is not always
beneficial.
On the other hand, the emergence of web technologies has originated the collaborative development of
ontologies. In this paper, we propose the use of ontologies in order to improve the accuracy and efficiency
of the semi-supervised document classification.
We used support vector machines, which is one of the most effective algorithms that have been studied for
text. Our algorithm enhances the performance of transductive support vector machines through the use of
ontologies. We report experimental results applying our algorithm to three different datasets. Our
experiments show an increment of accuracy of 4% on average and up to 20%, in comparison with the
traditional semi-supervised model.
Data mining , knowledge discovery is the process
of analyzing data from different perspectives and summarizing it
into useful information - information that can be used to increase
revenue, cuts costs, or both. Data mining software is one of a
number of analytical tools for analyzing data. It allows users to
analyze data from many different dimensions or angles, categorize
it, and summarize the relationships identified. Technically, data
mining is the process of finding correlations or patterns among
dozens of fields in large relational databases. The goal of
clustering is to determine the intrinsic grouping in a set of
unlabeled data. But how to decide what constitutes a good
clustering? It can be shown that there is no absolute “best”
criterion which would be independent of the final aim of the
clustering. Consequently, it is the user which must supply this
criterion, in such a way that the result of the clustering will suit
their needs.
For instance, we could be interested in finding
representatives for homogeneous groups (data reduction), in
finding “natural clusters” and describe their unknown properties
(“natural” data types), in finding useful and suitable groupings
(“useful” data classes) or in finding unusual data objects (outlier
detection).Of late, clustering techniques have been applied in the
areas which involve browsing the gathered data or in categorizing
the outcome provided by the search engines for the reply to the
query raised by the users. In this paper, we are providing a
comprehensive survey over the document clustering.
A Comparative Study of Centroid-Based and Naïve Bayes Classifiers for Documen...IJERA Editor
Assigning documents to related categories is critical task which is used for effective document retrieval. Automatic text classification is the process of assigning new text document to the predefined categories based on its content. In this paper, we implemented and performed comparison of Naïve Bayes and Centroid-based algorithms for effective document categorization of English language text. In Centroid Based algorithm, we used Arithmetical Average Centroid (AAC) and Cumuli Geometric Centroid (CGC) methods to calculate centroid of each class. Experiment is performed on R-52 dataset of Reuters-21578 corpus. Micro Average F1 measure is used to evaluate the performance of classifiers. Experimental results show that Micro Average F1 value for NB is greatest among all followed by Micro Average F1 value of CGC which is greater than Micro Average F1 of AAC. All these results are valuable for future research
With the rapid development in Geographic Information Systems (GISs) and their applications, more and
more geo-graphical databases have been developed by different vendors. However, data integration and
accessing is still a big problem for the development of GIS applications as no interoperability exists among
different spatial databases. In this paper we propose a unified approach for spatial data query. The paper
describes a framework for integrating information from repositories containing different vector data sets
formats and repositories containing raster datasets. The presented approach converts different vector data
formats into a single unified format (File Geo-Database “GDB”). In addition, we employ “metadata” to
support a wide range of users’ queries to retrieve relevant geographic information from heterogeneous and
distributed repositories. Such an employment enhances both query processing and performance.
International Journal of Engineering Research and Applications (IJERA) is an open access online peer reviewed international journal that publishes research and review articles in the fields of Computer Science, Neural Networks, Electrical Engineering, Software Engineering, Information Technology, Mechanical Engineering, Chemical Engineering, Plastic Engineering, Food Technology, Textile Engineering, Nano Technology & science, Power Electronics, Electronics & Communication Engineering, Computational mathematics, Image processing, Civil Engineering, Structural Engineering, Environmental Engineering, VLSI Testing & Low Power VLSI Design etc.
Hierarchal clustering and similarity measures along with multi representationeSAT Journals
Abstract All clustering methods have to assume some cluster relationship on the list of data objects that they really are applied on. Graph-Based Document Clustering works with frequent senses rather than frequent keywords used in traditional text mining techniques.Similarity between a pair of objects can be defined either explicitly or implicitly. With this paper, we analyzed existing multi-viewpoint based similarity measure and two related clustering methods. The main difference between a traditional dissimilarity/similarity measure and ours could be that the former uses merely a single viewpoint, which is the origin, even though the latter utilizes many viewpoints, which you ll find are objects assumed to not have the very same cluster using the two objects being measured. Using multiple viewpoints, more informative assessment of similarity could well be achieved. Theoretical analysis and empirical study are conducted to back up this claim. Two criterion functions for document clustering are proposed dependent on this wonderful measure. We compare them several well-known clustering algorithms which use other popular similarity measures on various document collections confirming the good sides of our proposal. Keywords –Multiview Cluster, Document id, ClusterDistance
This document summarizes several papers on document clustering techniques. It discusses hierarchical clustering and similarity measures, as well as multi-representation clustering. Several clustering algorithms are examined, including K-means clustering and graph-based clustering. The document also analyzes similarity measures like multi-viewpoint similarity and evaluates the performance of different clustering methods on document collections.
With ever increasing number of documents on web and other repositories, the task of organizing and
categorizing these documents to the diverse need of the user by manual means is a complicated job, hence
a machine learning technique named clustering is very useful. Text documents are clustered by pair wise
similarity of documents with similarity measures like Cosine, Jaccard or Pearson. Best clustering results
are seen when overlapping of terms in documents is less, that is, when clusters are distinguishable. Hence
for this problem, to find document similarity we apply link and neighbor introduced in ROCK. Link
specifies number of shared neighbors of a pair of documents. Significantly similar documents are called as
neighbors. This work applies links and neighbors to Bisecting K-means clustering in identifying seed
documents in the dataset, as a heuristic measure in choosing a cluster to be partitioned and as a means to
find the number of partitions possible in the dataset. Our experiments on real-time datasets showed a
significant improvement in terms of accuracy with minimum time.
This document discusses GCUBE indexing, which is a method for indexing and aggregating spatial/continuous values in a data warehouse. The key challenges addressed are defining and aggregating spatial/continuous values, and efficiently representing, indexing, updating and querying data that includes both categorical and continuous dimensions. The proposed GCUBE approach maps multi-dimensional data to a linear ordering using the Hilbert curve, and then constructs an index structure on the ordered data to enable efficient query processing. Empirical results show the GCUBE indexing offers significant performance advantages over alternative approaches.
1) The document discusses different clustering algorithms for text summarization including hierarchical clustering, query-based summarization, graph theoretic clustering, fuzzy c-means clustering, and DBSCAN clustering.
2) These algorithms are evaluated based on performance parameters like precision, recall, time complexity, space complexity, and summary quality.
3) The algorithm found to perform best based on these evaluations will be suggested as the better algorithm for query-dependent text document summarization.
International Journal of Engineering Research and Applications (IJERA) is a team of researchers not publication services or private publications running the journals for monetary benefits, we are association of scientists and academia who focus only on supporting authors who want to publish their work. The articles published in our journal can be accessed online, all the articles will be archived for real time access.
Our journal system primarily aims to bring out the research talent and the works done by sciaentists, academia, engineers, practitioners, scholars, post graduate students of engineering and science. This journal aims to cover the scientific research in a broader sense and not publishing a niche area of research facilitating researchers from various verticals to publish their papers. It is also aimed to provide a platform for the researchers to publish in a shorter of time, enabling them to continue further All articles published are freely available to scientific researchers in the Government agencies,educators and the general public. We are taking serious efforts to promote our journal across the globe in various ways, we are sure that our journal will act as a scientific platform for all researchers to publish their works online.
A rough set based hybrid method to text categorizationNinad Samel
This document summarizes a hybrid text categorization method that combines Latent Semantic Indexing (LSI) and Rough Sets theory to reduce the dimensionality of text data and generate classification rules. It introduces LSI to reduce the feature space of text documents represented as high-dimensional vectors. Then it applies Rough Sets theory to the reduced feature space to locate a minimal set of keywords that can distinguish document classes and generate multiple knowledge bases for classification instead of a single one. The method is tested on text categorization tasks and shown to improve accuracy over previous Rough Sets approaches.
This document describes a proposed concept-based mining model that aims to improve document clustering and information retrieval by extracting concepts and semantic relationships rather than just keywords. The model uses natural language processing techniques like part-of-speech tagging and parsing to extract concepts from text. It represents concepts and their relationships in a semantic network and clusters documents based on conceptual similarity rather than term frequency. The model is evaluated using singular value decomposition to increase the precision of key term and phrase extraction.
New proximity estimate for incremental update of non uniformly distributed cl...IJDKP
The conventional clustering algorithms mine static databases and generate a set of patterns in the form of
clusters. Many real life databases keep growing incrementally. For such dynamic databases, the patterns
extracted from the original database become obsolete. Thus the conventional clustering algorithms are not
suitable for incremental databases due to lack of capability to modify the clustering results in accordance
with recent updates. In this paper, the author proposes a new incremental clustering algorithm called
CFICA(Cluster Feature-Based Incremental Clustering Approach for numerical data) to handle numerical
data and suggests a new proximity metric called Inverse Proximity Estimate (IPE) which considers the
proximity of a data point to a cluster representative as well as its proximity to a farthest point in its vicinity.
CFICA makes use of the proposed proximity metric to determine the membership of a data point into a
cluster.
Complete agglomerative hierarchy document’s clustering based on fuzzy luhn’s ...IJECEIAES
Agglomerative hierarchical is a bottom up clustering method, where the distances between documents can be retrieved by extracting feature values using a topic-based latent dirichlet allocation method. To reduce the number of features, term selection can be done using Luhn’s Idea. Those methods can be used to build the better clusters for document. But, there is less research discusses it. Therefore, in this research, the term weighting calculation uses Luhn’s Idea to select the terms by defining upper and lower cut-off, and then extracts the feature of terms using gibbs sampling latent dirichlet allocation combined with term frequency and fuzzy Sugeno method. The feature values used to be the distance between documents, and clustered with single, complete and average link algorithm. The evaluations show the feature extraction with and without lower cut-off have less difference. But, the topic determination of each term based on term frequency and fuzzy Sugeno method is better than Tsukamoto method in finding more relevant documents. The used of lower cut-off and fuzzy Sugeno gibbs latent dirichlet allocation for complete agglomerative hierarchical clustering have consistent metric values. This clustering method suggested as a better method in clustering documents that is more relevant to its gold standard.
A CONCEPTUAL METADATA FRAMEWORK FOR SPATIAL DATA WAREHOUSEIJDKP
Metadata represents the information about data to be stored in Data Warehouses. It is a mandatory
element of Data Warehouse to build an efficient Data Warehouse. Metadata helps in data integration,
lineage, data quality and populating transformed data into data warehouse. Spatial data warehouses are
based on spatial data mostly collected from Geographical Information Systems (GIS) and the transactional
systems that are specific to an application or enterprise. Metadata design and deployment is the most
critical phase in building of data warehouse where it is mandatory to bring the spatial information and
data modeling together. In this paper, we present a holistic metadata framework that drives metadata
creation for spatial data warehouse. Theoretically, the proposed metadata framework improves the
efficiency of accessing of data in response to frequent queries on SDWs. In other words, the proposed
framework decreases the response time of the query and accurate information is fetched from Data
Warehouse including the spatial information
Text document clustering and similarity detection is the major part of document management, where every document should be identified by its key terms and domain knowledge. Based on the similarity, the documents are grouped into clusters. For document similarity calculation there are several approaches were proposed in the existing system. But the existing system is either term based or pattern based. And those systems suffered from several problems. To make a revolution in this challenging environment, the proposed system presents an innovative model for document similarity by applying back propagation time stamp algorithm. It discovers patterns in text documents as higher level features and creates a network for fast grouping. It also detects the most appropriate patterns based on its weight and BPTT performs the document similarity measures. Using this approach, the document can be categorized easily. In order to perform the above, a new approach is used. This helps to reduce the training process problems. The above framework is named as BPTT. The BPTT has implemented and evaluated using dot net platform with different set of datasets.
SEARCH OF INFORMATION BASED CONTENT IN SEMI-STRUCTURED DOCUMENTS USING INTERF...ijcsitcejournal
This paper proposes a semi-structured information retrieval model based on a new method for calculation
of similarity. We have developed CASISS (Calculation of Similarity of Semi-Structured documents)
method to quantify how two given texts are similar. This new method identifies elements of semi-structured
documents using elements descriptors. Each semi-structured document is pre-processed before the
extraction of a set of descriptors for each element, which characterize the contents of elements.It can be
used to increase the accuracy of the information retrieval process by taking into account not only the
presence of query terms in the given document but also the topology (position continuity) of these terms.
This document summarizes LinkedIn's approach to personalizing users' news feeds to improve relevance and address scalability challenges. It describes generating affinity scores for viewer-activity type, viewer-actor, and their combination. These scores are computed offline and stored to quickly retrieve for online scoring, avoiding computation overhead. Experiments show personalization improves click-through rates.
This document summarizes a research paper that proposes two approaches called LLORMA for constructing local low-rank matrix approximations. The approaches approximate an observed matrix M as a weighted sum of several low-rank matrices, where each low-rank matrix is accurate in a local region of M. This relaxes the assumption that M has global low-rank structure. The paper analyzes the accuracy of the local low-rank modeling approaches and shows they improve prediction accuracy over classical low-rank approximation methods on recommendation tasks.
This document discusses dimensionality reduction techniques for machine learning. It introduces Fisher Linear Discriminant analysis, which seeks projection directions that maximize separation between classes while minimizing within-class variance. It describes using the means and scatter measures of each class to define a cost function that is maximized to find the optimal projection direction. Principal Component Analysis is also briefly mentioned as another technique for dimensionality reduction.
Distributed Time Travel for Feature Generation by DB Tsai and Prasanna Padman...Spark Summit
This document describes Netflix's use of distributed time travel for feature generation using data snapshots. Key points:
1. Netflix uses data snapshots of online services stored in S3 to generate features offline for model training and experimentation, allowing ideas to be tested on historical data quickly before deploying live tests.
2. A "DeLorean" system selects contexts, takes snapshots of data from services like viewing history and playlists, and provides batch APIs to access snapshot data for offline experiments.
3. Feature encoders generate features using the snapshot data without calling live systems, and features are stored in Parquet files in S3. Successful models are then deployed online.
4. This approach significantly reduces the time
Distributed Time Travel for Feature Generation at Netflixsfbiganalytics
Learning is an analytic process of exploring the past in order to predict the future. Hence, being able to travel back in time to create feature is critical for machine learning projects to be successful. At Netflix, we spend significant time and effort experimenting with new features and new ways of building models. This involves generating features for our members from different regions over multiple days. To enable this, we built a time machine using Apache Spark that computes features for any arbitrary time in the recent past. The first step of building this time machine is to snapshot the data from various micro services on a regular basis. We built a general purpose workflow orchestration and scheduling framework optimized for machine learning pipelines and used it to run the snapshot and model training workflows. Snapshot data is then consumed by feature encoders to compute various features for offline experimentation and model training. Crucially, the same feature encoders are used in both offline model building and online scoring for production or A/B tests. Building this time machine helped us try new ideas quickly without placing stress on production services and without having to wait for data accumulation of the newly-implemented features. Moreover, building it with Apache Spark empowered us to both scale up the data size by an order of magnitude and train and validate the models in less time. Finally, using Apache Zeppelin notebook, we are able to interactively prototype features and run experiments.
Speaker Bio: DB Tsai
DB Tsai is an Apache Spark committer and a Senior Research Engineer working on Personalized Recommendation Algorithms at Netflix. Prior to joining Netflix, DB was a Lead Machine Learning Engineer at Alpine Data Labs, where He implemented several algorithms including Linear Regression and Binary/Multinomial Logistic Regression with Elastici-Net (L1/L2) regularization using LBFGS/OWL-QN optimizers in Apache Spark. DB was a Ph.D. candidate in Applied Physics at Stanford University. He holds a Master’s degree in Electrical Engineering from Stanford University.
Introduction to Machine Learning on Apache Spark MLlib by Juliet Hougland, Se...Cloudera, Inc.
Spark MLlib is a library for performing machine learning and associated tasks on massive datasets. With MLlib, fitting a machine-learning model to a billion observations can take only a few lines of code, and leverage hundreds of machines. This talk will demonstrate how to use Spark MLlib to fit an ML model that can predict which customers of a telecommunications company are likely to stop using their service. It will cover the use of Spark's DataFrames API for fast data manipulation, as well as ML Pipelines for making the model development and refinement process easier.
Reactive Feature Generation with Spark and MLlib by Jeffrey Smith (1)Spark Summit
The document discusses reactive feature generation pipelines for machine learning. It describes extracting features from raw data, transforming those features, and composing the steps into reusable pipelines. The pipelines can be supervised by connecting to reactive database drivers, running on cluster managers, and validating generated features to ensure quality. The goal is to build responsive, resilient, and elastic machine learning systems using these reactive strategies.
Qcon SF 2013 - Machine Learning & Recommender Systems @ Netflix ScaleXavier Amatriain
The document summarizes Netflix's approach to machine learning and recommender systems. It discusses how Netflix uses algorithms like SVD and Restricted Boltzmann Machines on a massive scale to power highly personalized recommendations. Over 75% of what people watch on Netflix comes from recommendations. Netflix collects a huge amount of data from over 40 million subscribers and uses both offline, online, and nearline computation across cloud services to train models and power recommendations in real-time at scale. The key is combining more data, smarter models, accurate metrics, and optimized system architectures.
Textual Data Partitioning with Relationship and Discriminative AnalysisEditor IJMTER
Data partitioning methods are used to partition the data values with similarity. Similarity
measures are used to estimate transaction relationships. Hierarchical clustering model produces tree
structured results. Partitioned clustering produces results in grid format. Text documents are
unstructured data values with high dimensional attributes. Document clustering group ups unlabeled text
documents into meaningful clusters. Traditional clustering methods require cluster count (K) for the
document grouping process. Clustering accuracy degrades drastically with reference to the unsuitable
cluster count.
Textual data elements are divided into two types’ discriminative words and nondiscriminative
words. Only discriminative words are useful for grouping documents. The involvement of
nondiscriminative words confuses the clustering process and leads to poor clustering solution in return.
A variation inference algorithm is used to infer the document collection structure and partition of
document words at the same time. Dirichlet Process Mixture (DPM) model is used to partition
documents. DPM clustering model uses both the data likelihood and the clustering property of the
Dirichlet Process (DP). Dirichlet Process Mixture Model for Feature Partition (DPMFP) is used to
discover the latent cluster structure based on the DPM model. DPMFP clustering is performed without
requiring the number of clusters as input.
Document labels are used to estimate the discriminative word identification process. Concept
relationships are analyzed with Ontology support. Semantic weight model is used for the document
similarity analysis. The system improves the scalability with the support of labels and concept relations
for dimensionality reduction process.
Clustering Algorithm with a Novel Similarity MeasureIOSR Journals
This document proposes a new multi-viewpoint based similarity measure for clustering text documents that aims to overcome limitations of existing measures. Existing measures use a single viewpoint to measure similarity between documents, but the proposed measure uses multiple viewpoints to ensure clusters exhibit all relationships between documents. The empirical study found that using a multi-viewpoint similarity measure forms more meaningful clusters by capturing more informative relationships between documents.
This document discusses web document clustering using a hybrid approach in data mining. It begins with an abstract describing the huge amount of data on the internet and need to organize web documents into clusters. It then discusses requirements for document clustering like scalability, noise tolerance, and ability to present concise cluster summaries. Different existing document clustering approaches are described, including text-based and link-based approaches. The proposed approach uses a concept-based mining model along with hierarchical agglomerative clustering and link-based algorithms to cluster web documents based on both their content and hyperlinks. This hybrid approach aims to provide more relevant clustered documents to users than previous methods.
This document discusses hierarchical clustering and similarity measures for document clustering. It summarizes that hierarchical clustering creates a hierarchical decomposition of data objects through either agglomerative or divisive approaches. The success of clustering depends on the similarity measure used, with traditional measures using a single viewpoint, while multiviewpoint measures use different viewpoints to increase accuracy. The paper then focuses on applying a multiviewpoint similarity measure to hierarchical clustering of documents.
This document summarizes a research paper on applying a multiviewpoint-based similarity measure to hierarchical document clustering. It begins by introducing document clustering and hierarchical clustering. It then discusses traditional similarity measures used for clustering and introduces a new multiviewpoint-based similarity measure (MVS) that uses multiple reference points to more accurately assess similarity. The paper applies MVS to both hierarchical and k-means clustering algorithms and evaluates the accuracy, precision, and recall of the resulting clusters. It finds that hierarchical clustering with MVS achieves better performance than k-means clustering with MVS based on these evaluation metrics.
A Competent and Empirical Model of Distributed ClusteringIRJET Journal
This document discusses distributed document clustering. It begins with an introduction to how documents are stored and indexed in computers. It then discusses different clustering algorithms like hierarchical and k-means clustering that are used to group similar documents. The document proposes a new framework for efficiently clustering text documents stored across different distributed resources. It argues that traditional clustering algorithms cannot perfectly cluster text data in decentralized systems. The framework uses properties of traditional algorithms with the ability to cluster in distributed systems.
TEXT CLUSTERING USING INCREMENTAL FREQUENT PATTERN MINING APPROACHIJDKP
Text mining is an emerging research field evolving from information retrieval area. Clustering and
classification are the two approaches in data mining which may also be used to perform text classification
and text clustering. The former is supervised while the later is un-supervised. In this paper, our objective is
to perform text clustering by defining an improved distance metric to compute the similarity between two
text files. We use incremental frequent pattern mining to find frequent items and reduce dimensionality.
The improved distance metric may also be used to perform text classification. The distance metric is
validated for the worst, average and best case situations [15]. The results show the proposed distance
metric outperforms the existing measures.
Semantics-based clustering approach for similar research area detectionTELKOMNIKA JOURNAL
The manual process of searching out individuals in an already existing
research field is cumbersome and time-consuming. Prominent and rookie
researchers alike are predisposed to seek existing research publications in
a research field of interest before coming up with a thesis. From
extant literature, automated similar research area detection systems have
been developed to solve this problem. However, most of them use
keyword-matching techniques, which do not sufficiently capture the implicit
semantics of keywords thereby leaving out some research articles. In this
study, we propose the use of ontology-based pre-processing, Latent Semantic
Indexing and K-Means Clustering to develop a prototype similar research area
detection system, that can be used to determine similar research domain
publications. Our proposed system solves the challenge of high dimensionality
and data sparsity faced by the traditional document clustering technique. Our
system is evaluated with randomly selected publications from faculties
in Nigerian universities and results show that the integration of ontologies
in preprocessing provides more accurate clustering results.
FAST FUZZY FEATURE CLUSTERING FOR TEXT CLASSIFICATION cscpconf
Feature clustering is a powerful method to reduce the dimensionality of feature vectors for text
classification. In this paper, Fast Fuzzy Feature clustering for text classification is proposed. It
is based on the framework proposed by Jung-Yi Jiang, Ren-Jia Liou and Shie-Jue Lee in 2011.
The word in the feature vector of the document is grouped into the cluster in less iteration. The
numbers of iterations required to obtain cluster centers are reduced by transforming clusters
center dimension from n-dimension to 2-dimension. Principle Component Analysis with slit
change is used for dimension reduction. Experimental results show that, this method improve
the performance by significantly reducing the number of iterations required to obtain the cluster
center. The same is being verified with three benchmark datasets
The document summarizes research on multi-document summarization using EM clustering. It begins with an introduction to the topic and issues with existing techniques. It then proposes using Expectation-Maximization (EM) clustering to identify clusters, which improves over other methods by identifying latent semantic variables between sentences. The architecture involves preprocessing, EM clustering, mutual reinforcement ranking algorithms RARP and RDRP, summarization, and post-processing. Experimental results on DUC2007 data show EM clustering identifies more clusters and sentences than affinity propagation clustering. The technique aims to improve summarization accuracy by better capturing semantic relationships between sentences.
The World Wide Web holds a large size of different information. Sometimes while searching the World Wide Web, users always do not gain the type of information they expect. In the subject of information extraction, extracting semantic relationships between terms from documents become a challenge. This paper proposes a system helps in retrieving documents based on the query expansion and tackles the extracting of semantic relationships from biological documents. This system retrieved documents that are relevant to the input terms then it extracts the existence of a relationship. In this system, we use Boolean model and the pattern recognition which helps in determining the relevant documents and determining the place of the relationship in the biological document. The system constructs a term-relation table that accelerates the relation extracting part. The proposed method offers another usage of the system so the researchers can use it to figure out the relationship between two biological terms through the available information in the biological documents. Also for the retrieved documents, the system measures the percentage of the precision and recall.
The World Wide Web holds a large size of different information. Sometimes while searching the World Wide Web, users always do not gain the type of information they expect. In the subject of information extraction, extracting semantic relationships between terms from documents become a challenge. This paper proposes a system helps in retrieving documents based on the query expansion and tackles the extracting of semantic relationships from biological documents. This system retrieved documents that are relevant to the input terms then it extracts the existence of a relationship. In this system, we use Boolean model and the pattern recognition which helps in determining the relevant documents and determining the place of the relationship in the biological document. The system constructs a term-relation table that accelerates the relation extracting part. The proposed method offers another usage of the system so the researchers can use it to figure out the relationship between two biological terms through the available information in the biological documents. Also for the retrieved documents, the system measures the percentage of the precision and recall.
A Semantic Retrieval System for Extracting Relationships from Biological Corpusijcsit
The World Wide Web holds a large size of different information. Sometimes while searching the World Wide Web, users always do not gain the type of information they expect. In the subject of information extraction, extracting semantic relationships between terms from documents become a challenge. This
paper proposes a system helps in retrieving documents based on the query expansion and tackles the extracting of semantic relationships from biological documents. This system retrieved documents that are relevant to the input terms then it extracts the existence of a relationship. In this system, we use Boolean
model and the pattern recognition which helps in determining the relevant documents and determining the place of the relationship in the biological document. The system constructs a term-relation table that accelerates the relation extracting part. The proposed method offers another usage of the system so the
researchers can use it to figure out the relationship between two biological terms through the available information in the biological documents. Also for the retrieved documents, the system measures the percentage of the precision and recall.
IRJET- Cluster Analysis for Effective Information Retrieval through Cohesive ...IRJET Journal
This document discusses using document clustering to improve information retrieval systems. It proposes a framework with four steps: 1) the information retrieval system retrieves documents based on a user query, 2) a similarity measure is used to determine document similarity, 3) the documents are clustered based on similarity, and 4) the clusters are ranked based on relevance to the query. The goal of clustering is to group relevant documents together to help users more easily find needed information. Different clustering algorithms are reviewed, noting that hierarchical clustering and overlapping clusters may improve search results over other methods.
This document provides an overview and summary of Pankaj Jajoo's 2008 master's thesis on improving document clustering algorithms. The thesis explores two approaches: 1) preprocessing the graph representation of documents to remove noise before applying standard graph partitioning algorithms, and 2) clustering words first before clustering documents to reduce noise. Experimental results on three datasets show these approaches improve clustering quality over standard K-Means clustering. The thesis provides background on clustering, reviews existing document clustering methods, and describes the two new algorithms and evaluation of their performance.
Recent Trends in Incremental Clustering: A ReviewIOSRjournaljce
This document provides a review of recent trends in incremental clustering algorithms. It discusses clustering methods based on both similarity measures and those not based on similarity measures. Specific incremental clustering algorithms covered include single-pass clustering, k-nearest neighbors clustering, suffix tree clustering, incremental DBSCAN, and ICIB (incremental clustering based on information bottleneck theory). The document also reviews various techniques for clustering, including particle swarm optimization, ant colony optimization, and genetic algorithms. Applications of genetic algorithm based clustering are discussed.
The International Journal of Engineering and Science (IJES)theijes
The International Journal of Engineering & Science is aimed at providing a platform for researchers, engineers, scientists, or educators to publish their original research results, to exchange new ideas, to disseminate information in innovative designs, engineering experiences and technological skills. It is also the Journal's objective to promote engineering and technology education. All papers submitted to the Journal will be blind peer-reviewed. Only original articles will be published.
This document proposes using an enhanced suffix tree approach to measure semantic similarity between multiple documents. It involves preprocessing documents by removing stop words, special characters, and converting to lowercase. Phrases are extracted and used to construct a suffix tree, where internal nodes represent phrases shared across documents. Term frequency-inverse document frequency (tf-idf) is used to calculate weights for internal nodes. Cosine, Dice, and Hellinger similarity measures are then applied to calculate pairwise similarities between documents based on the weighted internal nodes. The approach aims to efficiently and accurately measure semantic similarity between documents.
Annotation Approach for Document with Recommendation ijmpict
An enormous number of organizations generate and share textual descriptions of their products, facilities, and activities. Such collections of textual data comprise a significant amount of controlled information, which residues buried in the unstructured text. Whereas information extraction systems simplify the extraction of structured associations, they are frequently expensive and incorrect, particularly when working on top of text that does not comprise any examples of the targeted structured data. Projected an alternative methodology that simplifies the structured metadata generation by recognizing documents that are possible to contain information of awareness and this data will be beneficial for querying the database. Moreover, we intend algorithms to extract attribute-value pairs, and similarly devise new mechanisms to map such pairs to manually created schemes. We apply clustering technique to the item content information to complement the user rating information, which improves the correctness of collaborative similarity, and solves the cold start problem.
Scaling Down Dimensions and Feature Extraction in Document Repository Classif...ijdmtaiir
-In this study a comprehensive evaluation of two
supervised feature selection methods for dimensionality
reduction is performed - Latent Semantic Indexing (LSI) and
Principal Component Analysis (PCA). This is gauged against
unsupervised techniques like fuzzy feature clustering using
hard fuzzy C-means (FCM) . The main objective of the study is
to estimate the relative efficiency of two supervised techniques
against unsupervised fuzzy techniques while reducing the
feature space. It is found that clustering using FCM leads to
better accuracy in classifying documents in the face of
evolutionary algorithms like LSI and PCA. Results show that
the clustering of features improves the accuracy of document
classification
Similar to International Journal of Engineering and Science Invention (IJESI) (20)
Conversational agents, or chatbots, are increasingly used to access all sorts of services using natural language. While open-domain chatbots - like ChatGPT - can converse on any topic, task-oriented chatbots - the focus of this paper - are designed for specific tasks, like booking a flight, obtaining customer support, or setting an appointment. Like any other software, task-oriented chatbots need to be properly tested, usually by defining and executing test scenarios (i.e., sequences of user-chatbot interactions). However, there is currently a lack of methods to quantify the completeness and strength of such test scenarios, which can lead to low-quality tests, and hence to buggy chatbots.
To fill this gap, we propose adapting mutation testing (MuT) for task-oriented chatbots. To this end, we introduce a set of mutation operators that emulate faults in chatbot designs, an architecture that enables MuT on chatbots built using heterogeneous technologies, and a practical realisation as an Eclipse plugin. Moreover, we evaluate the applicability, effectiveness and efficiency of our approach on open-source chatbots, with promising results.
Northern Engraving | Nameplate Manufacturing Process - 2024Northern Engraving
Manufacturing custom quality metal nameplates and badges involves several standard operations. Processes include sheet prep, lithography, screening, coating, punch press and inspection. All decoration is completed in the flat sheet with adhesive and tooling operations following. The possibilities for creating unique durable nameplates are endless. How will you create your brand identity? We can help!
AI in the Workplace Reskilling, Upskilling, and Future Work.pptxSunil Jagani
Discover how AI is transforming the workplace and learn strategies for reskilling and upskilling employees to stay ahead. This comprehensive guide covers the impact of AI on jobs, essential skills for the future, and successful case studies from industry leaders. Embrace AI-driven changes, foster continuous learning, and build a future-ready workforce.
Read More - https://bit.ly/3VKly70
Getting the Most Out of ScyllaDB Monitoring: ShareChat's TipsScyllaDB
ScyllaDB monitoring provides a lot of useful information. But sometimes it’s not easy to find the root of the problem if something is wrong or even estimate the remaining capacity by the load on the cluster. This talk shares our team's practical tips on: 1) How to find the root of the problem by metrics if ScyllaDB is slow 2) How to interpret the load and plan capacity for the future 3) Compaction strategies and how to choose the right one 4) Important metrics which aren’t available in the default monitoring setup.
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor IvaniukFwdays
At this talk we will discuss DDoS protection tools and best practices, discuss network architectures and what AWS has to offer. Also, we will look into one of the largest DDoS attacks on Ukrainian infrastructure that happened in February 2022. We'll see, what techniques helped to keep the web resources available for Ukrainians and how AWS improved DDoS protection for all customers based on Ukraine experience
AppSec PNW: Android and iOS Application Security with MobSFAjin Abraham
Mobile Security Framework - MobSF is a free and open source automated mobile application security testing environment designed to help security engineers, researchers, developers, and penetration testers to identify security vulnerabilities, malicious behaviours and privacy concerns in mobile applications using static and dynamic analysis. It supports all the popular mobile application binaries and source code formats built for Android and iOS devices. In addition to automated security assessment, it also offers an interactive testing environment to build and execute scenario based test/fuzz cases against the application.
This talk covers:
Using MobSF for static analysis of mobile applications.
Interactive dynamic security assessment of Android and iOS applications.
Solving Mobile app CTF challenges.
Reverse engineering and runtime analysis of Mobile malware.
How to shift left and integrate MobSF/mobsfscan SAST and DAST in your build pipeline.
"What does it really mean for your system to be available, or how to define w...Fwdays
We will talk about system monitoring from a few different angles. We will start by covering the basics, then discuss SLOs, how to define them, and why understanding the business well is crucial for success in this exercise.
What is an RPA CoE? Session 2 – CoE RolesDianaGray10
In this session, we will review the players involved in the CoE and how each role impacts opportunities.
Topics covered:
• What roles are essential?
• What place in the automation journey does each role play?
Speaker:
Chris Bolin, Senior Intelligent Automation Architect Anika Systems
Session 1 - Intro to Robotic Process Automation.pdfUiPathCommunity
👉 Check out our full 'Africa Series - Automation Student Developers (EN)' page to register for the full program:
https://bit.ly/Automation_Student_Kickstart
In this session, we shall introduce you to the world of automation, the UiPath Platform, and guide you on how to install and setup UiPath Studio on your Windows PC.
📕 Detailed agenda:
What is RPA? Benefits of RPA?
RPA Applications
The UiPath End-to-End Automation Platform
UiPath Studio CE Installation and Setup
💻 Extra training through UiPath Academy:
Introduction to Automation
UiPath Business Automation Platform
Explore automation development with UiPath Studio
👉 Register here for our upcoming Session 2 on June 20: Introduction to UiPath Studio Fundamentals: https://community.uipath.com/events/details/uipath-lagos-presents-session-2-introduction-to-uipath-studio-fundamentals/
Dandelion Hashtable: beyond billion requests per second on a commodity serverAntonios Katsarakis
This slide deck presents DLHT, a concurrent in-memory hashtable. Despite efforts to optimize hashtables, that go as far as sacrificing core functionality, state-of-the-art designs still incur multiple memory accesses per request and block request processing in three cases. First, most hashtables block while waiting for data to be retrieved from memory. Second, open-addressing designs, which represent the current state-of-the-art, either cannot free index slots on deletes or must block all requests to do so. Third, index resizes block every request until all objects are copied to the new index. Defying folklore wisdom, DLHT forgoes open-addressing and adopts a fully-featured and memory-aware closed-addressing design based on bounded cache-line-chaining. This design offers lock-free index operations and deletes that free slots instantly, (2) completes most requests with a single memory access, (3) utilizes software prefetching to hide memory latencies, and (4) employs a novel non-blocking and parallel resizing. In a commodity server and a memory-resident workload, DLHT surpasses 1.6B requests per second and provides 3.5x (12x) the throughput of the state-of-the-art closed-addressing (open-addressing) resizable hashtable on Gets (Deletes).
"$10 thousand per minute of downtime: architecture, queues, streaming and fin...Fwdays
Direct losses from downtime in 1 minute = $5-$10 thousand dollars. Reputation is priceless.
As part of the talk, we will consider the architectural strategies necessary for the development of highly loaded fintech solutions. We will focus on using queues and streaming to efficiently work and manage large amounts of data in real-time and to minimize latency.
We will focus special attention on the architectural patterns used in the design of the fintech system, microservices and event-driven architecture, which ensure scalability, fault tolerance, and consistency of the entire system.
Essentials of Automations: Exploring Attributes & Automation ParametersSafe Software
Building automations in FME Flow can save time, money, and help businesses scale by eliminating data silos and providing data to stakeholders in real-time. One essential component to orchestrating complex automations is the use of attributes & automation parameters (both formerly known as “keys”). In fact, it’s unlikely you’ll ever build an Automation without using these components, but what exactly are they?
Attributes & automation parameters enable the automation author to pass data values from one automation component to the next. During this webinar, our FME Flow Specialists will cover leveraging the three types of these output attributes & parameters in FME Flow: Event, Custom, and Automation. As a bonus, they’ll also be making use of the Split-Merge Block functionality.
You’ll leave this webinar with a better understanding of how to maximize the potential of automations by making use of attributes & automation parameters, with the ultimate goal of setting your enterprise integration workflows up on autopilot.
Lee Barnes - Path to Becoming an Effective Test Automation Engineer.pdfleebarnesutopia
So… you want to become a Test Automation Engineer (or hire and develop one)? While there’s quite a bit of information available about important technical and tool skills to master, there’s not enough discussion around the path to becoming an effective Test Automation Engineer that knows how to add VALUE. In my experience this had led to a proliferation of engineers who are proficient with tools and building frameworks but have skill and knowledge gaps, especially in software testing, that reduce the value they deliver with test automation.
In this talk, Lee will share his lessons learned from over 30 years of working with, and mentoring, hundreds of Test Automation Engineers. Whether you’re looking to get started in test automation or just want to improve your trade, this talk will give you a solid foundation and roadmap for ensuring your test automation efforts continuously add value. This talk is equally valuable for both aspiring Test Automation Engineers and those managing them! All attendees will take away a set of key foundational knowledge and a high-level learning path for leveling up test automation skills and ensuring they add value to their organizations.
Must Know Postgres Extension for DBA and Developer during MigrationMydbops
Mydbops Opensource Database Meetup 16
Topic: Must-Know PostgreSQL Extensions for Developers and DBAs During Migration
Speaker: Deepak Mahto, Founder of DataCloudGaze Consulting
Date & Time: 8th June | 10 AM - 1 PM IST
Venue: Bangalore International Centre, Bangalore
Abstract: Discover how PostgreSQL extensions can be your secret weapon! This talk explores how key extensions enhance database capabilities and streamline the migration process for users moving from other relational databases like Oracle.
Key Takeaways:
* Learn about crucial extensions like oracle_fdw, pgtt, and pg_audit that ease migration complexities.
* Gain valuable strategies for implementing these extensions in PostgreSQL to achieve license freedom.
* Discover how these key extensions can empower both developers and DBAs during the migration process.
* Don't miss this chance to gain practical knowledge from an industry expert and stay updated on the latest open-source database trends.
Mydbops Managed Services specializes in taking the pain out of database management while optimizing performance. Since 2015, we have been providing top-notch support and assistance for the top three open-source databases: MySQL, MongoDB, and PostgreSQL.
Our team offers a wide range of services, including assistance, support, consulting, 24/7 operations, and expertise in all relevant technologies. We help organizations improve their database's performance, scalability, efficiency, and availability.
Contact us: info@mydbops.com
Visit: https://www.mydbops.com/
Follow us on LinkedIn: https://in.linkedin.com/company/mydbops
For more details and updates, please follow up the below links.
Meetup Page : https://www.meetup.com/mydbops-databa...
Twitter: https://twitter.com/mydbopsofficial
Blogs: https://www.mydbops.com/blog/
Facebook(Meta): https://www.facebook.com/mydbops/
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...Jason Yip
The typical problem in product engineering is not bad strategy, so much as “no strategy”. This leads to confusion, lack of motivation, and incoherent action. The next time you look for a strategy and find an empty space, instead of waiting for it to be filled, I will show you how to fill it in yourself. If you’re wrong, it forces a correction. If you’re right, it helps create focus. I’ll share how I’ve approached this in the past, both what works and lessons for what didn’t work so well.
International Journal of Engineering and Science Invention (IJESI)
1. International Journal of Engineering Science Invention
ISSN (Online): 2319 – 6734, ISSN (Print): 2319 – 6726
www.ijesi.org Volume 3 Issue 1ǁ January 2014 ǁ PP.18-23
A Unified Model for Document Clustering Based on Dirichlet
Process Mixture Model
1
S.Princiya , 2, M.Prabakaran
Ratnavel subramaniam college of Engineering and technology
, Assistant Professor, Ratnavel subramaniam college of Engineering and technology,
ABSTRACT: Text document clustering provides an effective technique to organize a huge amount of retrieval
results by grouping documents in a small number of meaningful classes. In unsupervised clustering method the
unlabeled input data is used to estimate the parameter values. In a semi supervised document clustering both
labeled and unlabeled input data is used for document clustering. Traditional semi supervised clustering uses
only finite user supervision in the form of labeled instances and pairwise instance. A semi supervised clustering
with feature supervision is proposed in this paper. Feature supervision method is used for labeling of
documents. A variational inference algorithm uses the Dirichlet Process Mixture model for the document
collection as well as the supervision of the cluster at the same time. The proposed method finds the number of
clusters automatically which is an optimal solution for time consuming.
INDEX TERMS: variational inference algorithm, semi supervised clustering, feature supervision, Dirichlet
Polynomial Allocation (DPA) model, Dirichlet Process Mixture Model (DPM)
I. INTRODUCTION
Document Clustering Is The Process Of Collecting Similar Documents Into Meaningful Clusters.
Clustering Analysis Is Traditionally Considered As An Unsupervised Learning Method. Documents Within The
Same Clusters Are More Similar Than Those In Different Clusters. The Must-Link And Cannot-Link
Constraints Between Instances Are Considered Before Performing The Clustering. Different Users May Want
To Arrange The Document Collection In Their Own Point Of View Instead Of A Universal One. Consider A
Collection Of News Articles About Books. One User May Like To Arrange The Collection By Author While
Another User May Want To Organize It By Alphabet Order. This Is Not Possible In Unsupervised
Clustering.Another Issue In Document Clustering Is To Determine The Number Of Document Clusters K
Automatically. In Most Of The Previous Document Clustering Method, K Is Predefined Before The Clustering
Process. However Users Need To Search The Entire Document To Estimate The K. This Takes Much More
Time When The Size Of The Document Is Large. Furthermore Erroneous Evaluation Of K Might Easily Betray
The Clustering Process.Grouping Of Documents Into An Optimal Number Of Clusters Is Done By Dirichlet
Process Mixture Model (DPM). It Shows The Auspicious Results By Determining The Number Of Clusters
Automatically When The Number Of Clusters Is Unknown. When A New Data Point Enters, It Either Increases
From Previous Clusters Or Initiate A New Cluster. DPM Model Based Clustering Process Considers Both The
Data Possibility And The Clustering Property Of The DP Preceding That Data Points Are More Likely To Be
Related To Popular And Large Clusters. This Feature Makes The DPM Model Particularly Adapt For Document
Clustering. However, There Is No Work Inspecting The DPM Model For Document Clustering Due To The
High Dimensional Representation Of The Text Documents. Each Document Consists Of Huge Amounts Of
Words Containing Both Relevant Words And Irrelevant Words. Only Relevant Words Are Used For Clustering.
The Involvement Of Irrelevant Words Complicates The Clustering Process. Therefore During Document
Clustering, It Is Necessary To Separate The Irrelevant Words Specifically When The K Is Unknown.Variational
Inference Algorithm Is Used To Deduce DPM Parameters. When The Document Data Set Is Large, It Is Harder
To Apply The Gibbs Sampling Algorithm. It Also Needs Long Time To Converge. Variational Inference
Algorithm Is Used To Deduce The Document Collection Structure In A Quick Manner. In DPMFP Approach,
We Need To Deduce Both Document Collection Structure And Supervision Of Document Words At The Same
Time. Therefore A Traditional Variational Inference Algorithm Cannot Apply In DPM Model.To Overcome
The Problem Of Document Clustering A New Approach Called Dirichlet Process Mixture Model With Feature
Supervision (DPMFS) Is Designed In This Paper. To Simplify The Process Of Parameter Estimation, A
Dirichlet Polynomial Allocation With Feature Supervision Namely DPAFS, Used To Approximate The DPMFS
Model. The Supervision Algorithm Performs Two Types Of Supervision. The First Document Supervision
Algorithm Involves In Labeling The Documents. It Specifies The Document Constraints As Must-Link Or
Cannot-Link. Then The Feature Supervision Method Involves In Labeling The Features. Variational Inference
www.ijesi.org
18 | Page
2. A Unified Model For Document Clustering Based…
Algorithm Is Derived From The DPAFS Model. The Variational Inference Algorithm Uses The DPM Model
For The Clustering Process. The Documents Which Are Labeled Are Used For Clustering. It Finds The Number
Of Clusters And Forms The Cluster Automatically.The rest of the paper is organized as follows. Section II
presents a description about the previous research which is relevant to document clustering. Section III involves
the detailed description about the proposed method. Section IV presents the performance analysis. This paper
concludes in Section V.
II. RELATED WORK
This section analysis the previous work performed in document clustering algorithms. Document
clustering is used for retrieval of information from the text document and web documents. R. Huang, et al
presented a Dirichlet process for finding the number of clusters automatically. Documents were automatically
partitioned into discriminating words and nondiscriminative words. An Inference algorithm was explored to
deduce the document collection structure and seperation of document words at the same time [1]. Nott, et al
explored a modified method on the sequential updating and greedy search (SUGS) algorithm Wang and Dunson
for proper Dirichlet process mixture models. The SUGS algorithm was mutated within a variational Bayes
framework which was used to collect different approximations of the posterior distribution. It provided a
probability distribution to allocate data to a cluster [2]. Chen, et al proposed an effective Fuzzy Frequent
Itemset-based Document Clustering (F2IDC).
This approach combines the fuzzy association rule mining with the background knowledge embedded
in WordNet, which improve the quality of document clustering [3]. Gil-García and Pons-Porrata presented two
clustering algorithms particularly dynamic hierarchical compact and dynamic hierarchical star. The first
algorithm creates disjoint hierarchies of clusters, while the second algorithm obtains overlapped hierarchies.
This method offer a solution for hierarchical clustering in dynamic environment effectively and offer hierarchies
easier to browse than traditional algorithms [4]. Muhammad, et al compared two approaches to document
clustering based on suffix tree data model and quality of the results are analysed.
First model excerpt phrases from documents and uses a similarity measure based on common suffix
tree to cluster the documents. Second model extracts the similar word sequence from the document and uses the
common word meaning sequence to perform the compact representation. Finally document clustering method is
used to cluster the compact documents. To perform the actual clustering steps, agglomerative hierarchical
document clustering was used [5]. Jayabharathy, et al proposed a modified semantic-based model and topic
detection method. Bisecting K-means algorithm was used to excerpt the related terms as concepts for concept
based document cluster. Tester theory was used for determining the meaningful labels for the document
clusters [6] . Zhao, et al proposed a constrained DBSCAN algorithm, which includes instant-level constraints, to
obtained the better clustering performance. To access the user feedback, an effective learning approach is
applied by selecting descriptive document pairs [7]. Chen, et al formulated an efficient Fuzzy Frequent ItemsetBased Hierarchical Clustering (F2IHC) approach. To upgrade the clustering accuracy, fuzzy association rule
mining algorithm is used [8]. Mahdavi and Abolhassani presented a Harmony K-means algorithm (HKA) that
deals with the work related to document clustering based on Harmony Search (HS) optimization method. It also
compare the HKA with other meta-heurastic and model based document clustering approaches [9]. Taiping, et
al proposed a correlation preserving indexing (CPI), which is achieved in the correlation similarity measure
space. In this method, correlations between the documents in the local patches are maximized and the
correlations between the documents outside the patches are minimized together. The explored CPI method can
efficiently determine the intrinsic structures embedded in high-dimensional document space [10] . Forsati, et al
proposed a novel document clustering algorithms based on the Harmony Search (HS) optimization method. HS
based clustering algorithm obtain the nearby optimal clusters within a acceptable time. To achieve better
clustering harmony clustering is unified with the K-means algorithm in three ways[11]. Deng, et al presented a
new approach called Locally Concept Factorization (LCCF) which captures the local geometry of the document
submanifold. The documents associated with the same concept can be well clustered in this method [12]. Sun, et
al proposed a new method for document clustering by grouping consequential topics from the document set and
weighted acceptable features. It uses cluster correlation to remove the dissimilarity documents in clusters [13].
III. DIRICHLET POLYNOMIAL ALLOCATION WITH FEATURE SUPERVISION
(DPAFS)
A. Dirichlet Process Mixture Model
DPM model is a flexible mixture model which is used to count infinite mixture model. This paper
proposes the infinite mixture model by first characterize the simple finite mixture model.Each data point is
derived from one of K fixed unknown distributions in the fixed mixture model. Let us take be the parameter
www.ijesi.org
19 | Page
3. A Unified Model For Document Clustering Based…
from which the document
is generated. Since the number of K is always unknown, we assume that the data
point
follows a general mixture model in which
is generated from a distribution B. The comparison is as
follows.
(1)
(2)
where
is the number of data points and
is the distribution of
given .
The probability distribution described above is always unknown. The generative mixture model is reduced to
finite mixture model if
is a discrete distribution on a finite set of values. Dirichlet process mixture model
places a Dirichlet process prior on the unknown distribution in the nonparametric Bayesian analysis. is
considered as a mixture distribution with a random number of components.
The hierarchical Bayesian requirement of the DPM model is described below.
(3)
(4)
(5)
where
represents a DP with a base distribution
and a positive scaling parameter . Possibly,
the mean of the DP and is the inverse variance.
The pictorial representation of the DPM model is shown in Fig.1.
is
Fig.1. Hierarchical structure of DPM model
B. Dirichlet Polynomial Allocation Model
When the number of mixture components is taken as infinity, the DPM model can be derived as the
limit of a sequence of finite mixture models. Dirichlet Polynomial Allocation (DPA) model is one of the
eminent approximations of the DPM model. The common model for the DPA model is as follows:
where M is the number of mixture components. R is an M-dimensional vector indicating the mixing proportions
for components given a Dirichlet prior with symmetric parameter
.
is an integer illustrate the latent
component allocation of the data point
. For every component, the parameter
indicates the distribution of
data points from that component.
C. Algorithm Description
Algorithm: Semi-supervised Clustering with Feature
Supervision
Input: Set of data points X
www.ijesi.org
20 | Page
4. A Unified Model For Document Clustering Based…
Output: K clusters
Method:
Step 1: Perform dual supervision, i.e., document supervision and feature supervision
Step 2: Obtain the labeled feature set WL and the document seed set S or must-link set M and cannot-link set C
Step 3: if Combine Linked Documents
Step 4: Perform basic Clustering using the learned weights
Step 5: else
Step 6: Perform feature reweighting based on labeled feature set WL.
Step 7: Cluster the documents using semi-supervised clustering algorithm.
Step 8: end if
The procedure of semi supervised clustering with feature supervision is shown in algorithm 1.
D. Semi supervised clustering with Feature Supervision
Traditional semi-supervised method employs user supervision in the form of pairwise constraints.
Adding feature supervision to semi supervised clustering provides dual supervision for clustering. Both
supervision takes place together before the clustering algorithm begins. The variational inference algorithm is
derived from the DPAFS model to infer the document collection structure and the separation of document words
at the same time. To find the cluster structure variational inference algorithm is used. The probability
distributions are calculated by using the term frequencies. The conditional probability distribution for each
document is calculated by using the variational inference algorithm. After finding the approximation for all the
documents the document is decided in which cluster it belongs. In the document where it achieved highest
approximation value the document belongs to that cluster. So the user can give the input as the words to get the
relevant clusters. The architecture of the system is shown in the Fig.2. In data mining data preprocessing is the
first step for document clustering. Here the 20 newsgroups dataset considered for the clustering. For the labeling
of documents semi supervised method with feature supervision algorithm is used. It differentiates the document
constraints as must-link and cannot-link. Data preprocessing is used to remove unwanted data and noise in data
sets. Two steps mainly stemming and removal of stopword. The stemming process removes the suffixes
appended with each word. Stopword process removes the article and prepositions. To find the term frequencies
we should handle the words which are repeated. The clusters form based upon the frequencies of the terms. To
find the term frequency first of all the words in the documents is collected. The document can contain the
repeated words throughout the document. So to handle the repeated words we should find the distinct words in
the documents. The distinct words are the words which are not repeated comes out in the documents. So the
term frequency is constructed from these distinct words. Finally Variational inference algorithm is used for
cluster identification.
Fig.2. System Architecture
II. PERFORMANCE ANALYSIS
This section presents the performance evaluation of the proposed DPAFS method. The performance is
evaluated based on the following measures:
www.ijesi.org
21 | Page
5. A Unified Model For Document Clustering Based…
A. TIME CONSUMPTION
Fig.3. Illustrate the time taken for cluster formation in semi supervised method and unsupervised method.
USDM refers to an Unsupervised document method. SSDM refers to a Semi supervised document method.
From the figure it is observed that the forming of clusters in the unsupervised method takes more time when
compared to the semi supervised method.
Fig.3. Time taken for cluster formation
B. Number of clusters
In the proposed method the number of clusters should be calculated previously. If the number
of clusters is at a high level, then the document cluster provides efficient result. Fig.4. Illustrates the number of
cluster formation in unsupervised document clustering and semi supervised document clustering. From the
figure it is observed that the number of clusters is high in semi supervised document method.
Fig.4. Number of Clusters in USDM and SSDM
C. Must-Link Feature
Fig.5 illustrates the comparison of number of must-link features in semi supervised document method and
unsupervised document clustering.
www.ijesi.org
22 | Page
6. A Unified Model For Document Clustering Based…
Fig.5. Number of must-link words in USDM and SSDM
III. CONCLUSION
In this paper, an approach is proposed that handles both document cluster and feature supervision at the
same time. Semi supervised document clustering with feature supervision, which asks the user to label the
features by indicating whether they discriminate among clusters. A document cluster approach is investigated
supported the DPM model that teams documents into capricious range clusters. Our experiment shows that our
approach acquires high cluster accuracy and affordable supervision of document words. The comparison
between our approach and progressive approaches indicates that our approach is strong and effective for
document cluster. Our analysis of the experiment result also shows that the DPM model with automatic feature
supervision methodology may effectively discover word supervision and improve the document cluster quality.
REFERENCES
[1]
[2]
[3]
[4]
[5]
[6]
[7]
[8]
[9]
[10]
[11]
[12]
[13]
R. Huang, et al., "Dirichlet Process Mixture Model for Document Clustering with Feature Partition," 2012.
D. Nott, et al., "A sequential algorithm for fast fitting of Dirichlet process mixture models," arXiv preprint arXiv:1301.2897,
2013.
C.-L. Chen, et al., "An integration of fuzzy association rules and WordNet for document clustering," Knowledge and information
systems, vol. 28, pp. 687-708, 2011.
R. Gil-García and A. Pons-Porrata, "Dynamic hierarchical algorithms for document clustering," Pattern Recognition Letters, vol.
31, pp. 469-477, 2010.
R. Muhammad, et al., "A comparison of two suffix tree-based document clustering algorithms," in Information and Emerging
Technologies (ICIET), 2010 International Conference on, 2010, pp. 1-5.
J. Jayabharathy, et al., "Document clustering and topic discovery based on semantic similarity in scientific literature," in
Communication Software and Networks (ICCSN), 2011 IEEE 3rd International Conference on, 2011, pp. 425-429.
W. Zhao, et al., "Effective semi-supervised document clustering via active learning with instance-level constraints," Knowledge
and information systems, vol. 30, pp. 569-587, 2012.
C.-L. Chen, et al., "Mining fuzzy frequent itemsets for hierarchical document clustering," Information Processing &
Management, vol. 46, pp. 193-211, 2010.
M. Mahdavi and H. Abolhassani, "Harmony K-means algorithm for document clustering," Data Mining and Knowledge
Discovery, vol. 18, pp. 370-391, 2009/06/01 2009.
Z. Taiping, et al., "Document Clustering in Correlation Similarity Measure Space," Knowledge and Data Engineering, IEEE
Transactions on, vol. 24, pp. 1002-1013, 2012.
R. Forsati, et al., "Efficient stochastic algorithms for document clustering," Information Sciences, vol. 220, pp. 269-291, 2013.
C. Deng, et al., "Locally Consistent Concept Factorization for Document Clustering," Knowledge and Data Engineering, IEEE
Transactions on, vol. 23, pp. 902-913, 2011.
P. Sun, et al., "Document Clustering Method Using Weighted Semantic Features and Cluster Similarity," in Digital Game and
Intelligent Toy Enhanced Learning (DIGITEL), 2010 Third IEEE International Conference on, 2010, pp. 185-187.
www.ijesi.org
23 | Page