1. Weave-D
A cognitive approach towards data
accumulation and fusion
Thushan Ganegedara
Ruwan Gunarathne
Lasindu Vidana Pathiranage
Buddhima Wijeweera
2. Why Weave-D?
Growth of amount of information
Handle
data
Temporal
Multi-
modal
Prevent
catastrophic
interference
Incremental
learning
algorithms
Visualizing
information
Intuitive
Simple
Apply
previous
knowledge
to acquire
new
knowledge
Conceptualization
Generalizati
on of
acquired
knowledge
??
??
3. What is Weave-D?
Accumulate
data (i.e.
Images, Text)
Feature
Extraction
Incremental
learning
Link
generation
Query & Visualize UI
Accumulates temporal, multi-
modal or multi source data in an
organized manner
Extract information from data (ex.
Color, Edge, Shape information of
images)
Incrementally learn using IKASL
algorithm
Links represent relationships
between multi-modal data
4. Major Research Problems
• Integrating incremental learning algorithm to
the selected artificial perception model
• What are the potential performance
improvements for selected unsupervised
learning algorithm?
• What are the suitable feature extraction
techniques for images and text?
• How to visualize complex learning outcomes to
user?
5. Major Challenges
• General
▫ Limited resources and novelty of the algorithms
▫ Finding suitable datasets
• Image Feature Extraction
▫ Deciding the best colors space to represent images
▫ Researching shape descriptor implementations
• Text Feature Extraction Techniques
▫ Researching suitable text feature extraction
techniques
6. Major Challenges (cont.)
• Unsupervised Learning Algorithms
▫ Implementing IKASL
▫ Testing and verifying correctness of IKASL
• Researching information visualization tools to fit
our requirements
7. Project Scope
• The proposed system will be implemented for
handling only Image and Text inputs
• System will be designed to be used by data
analysts
• System will,
▫ Extract feature vectors of images and texts
▫ Acquire knowledge using data input to Weave-D
▫ Generate links between data
8. Project Scope
• Persistence technique (ex. SQL DB,XML, etc.)
will be used to store acquired knowledge and
generated links
• Provide an interface for users to query/visualize
information
9. Assumptions & Limitations
• Selected features(e.g. color, shape,…) provide a
good representation of the data (e.g. images,…)
• In artificial perception model, perception at a
certain layer, can be represented by one most
significant feature from the layer below
• Input data should be compatible with feature
extractors (i.e. Type, Format, …)
• Tools required (e.g. Feature extraction,
Information Visualization) can be utilized in the
project with no/slight modifications
10. Deliverables
• JAVA implementation of the proposed system
including several sub-components
• Documentation
Incremental
learning
Information
persistence
Information
linking
Information
visualization
• Research Proposal • Literature Review
• Project Scope Document • Architectural Document
• Project Report • User Manual
16. Artificial Perception Model [1]
• Inspired by human perceptive and cognitive
system
• Close resemblance to human brain
• Key features
▫ Supports multiple modalities
▫ Ability to generate high-level perceptions by
aggregating input stimuli belonging to multiple
modalities
▫ Conceptualization of information
[1] Bamunusinghe, Jeewanee, and Damminda Alahakoon. "Artificial Visual Percepts for Image Understanding." In
Proceedings of the International Conference on Intelligent Systems. 2010.
19. Self-Organizing Maps (SOM) [2]
• Visualization technique which reduces the
dimensions of data to help humans understand
high dimensional data.
• Self-Organizing Map (SOM) is a type of
unsupervised artificial neural network.
• Topology preserving map
[2] Kohonen, Teuvo. "The self-organizing map." Proceedings of the IEEE 78, no. 9 (1990):1464-1480
21. Growing Self-Organizing Maps
(GSOMs) [3]
• GSOM is an extension of Self-Organizing maps (SOM),
which is very popular in knowledge discovery applications.
• GSOM algorithm overcomes several limitations of SOM.
• The main advantage of GSOM over SOM is that, GSOM has
the ability to grow and modify the shape to represent the
data space better.
• Other similar work are,
▫ Growing Cell Structures (GCS’s)
▫ Neural Gas Algorithm (NGA)
▫ Incremental Grid Growing (IGG)
[3] Alahakoon, Damminda, Saman K. Halgamuge, and Bala Srinivasan. "Dynamic self-organizing maps with controlled
growth for knowledge discovery." Neural Networks, IEEE Transactions on 11, no. 3 (2000): 601-614
22. GSOM Algorithm
• GSOM is an unsupervised neural network, which
is initialized with four nodes and develops to
represent the input data space.
• There are three main phases which can be
distinguished in GSOM algorithm
▫ Initialization phase
▫ Growing phase
▫ Smoothing phase
23. Initialization Phase
• Starting four nodes will be initialized with
random values from the input vector space.
(0,1) (1,1)
(0,0) (1,0)
25. Smoothing phase
• Growing phase stops when new node growth
saturates
• Reduce learning rate and fix a small starting
neighborhood.
• Find winner and adapt the weights of winner
and neighbors in the same way as in growing
phase.
27. IKASL Algorithm [4]
• Most current Hebbian rule based algorithms do not
encompass incremental learning and life-long
learning
• Hebbian rule based unsupervised incremental
learning algorithm
• Is both stable and plastic
• Can be understood as an n-layer structure
• A single layer comprises 2 sub-layers
▫ Learning Layer
▫ Generalized Layer
[4] De Silva, Daswin, and Damminda Alahakoon. "Incremental knowledge acquisition and self learning from text." In
Neural Networks (IJCNN), The 2010 International Joint Conference on, pp. 1-8. IEEE, 2010.
31. Image Feature Extraction
• In project we do not directly interact with raw
images
• There are lots of redundant data in images
• The solution is feature extraction techniques
• This transformation process of input data to a set of
feature vectors is known as feature extraction
• The Moving Picture Expert Group (MPEG) was
established and it has developed several
implementations
• In MPEG-7: Multimedia content description
interface was created
32. MPEG-7 Descriptors [5-7]
• Descriptors: a core set of quantitative measures
of audio-visual features
• Some of MPEG-7 Descriptors are,
▫ Dominant Colour Descriptor
▫ Colour Layout Descriptor
▫ Edge Histogram Descriptors
[5] Ortiz, Edward, Cesar Pantoja, and María Trujillo. "An MPEG-7 Browser." InLatin American Conference on Networked
Electronic Media. 2009.
[6] Wu, Peng, Yong Man Ro, Chee Sun Won, and Yanglim Choi. "Texture descriptors in MPEG-7." In Computer Analysis of
Images and Patterns, pp. 21-28. Springer Berlin Heidelberg, 2001
[7] Chatzichristofis, Savvas A., Yiannis S. Boutalis, and Mathias Lux. "Img (rummager): An interactive content based image
retrieval system." In Similarity Search and Applications, 2009. SISAP'09. Second International Workshop on, pp. 151-153.
IEEE, 2009.
38. Shape Descriptors
[8] Bosch, Anna, Andrew Zisserman, and Xavier Munoz. "Representing shape with a spatial pyramid kernel." In
Proceedings of the 6th ACM international conference on Image and video retrieval, pp. 401-408. ACM, 2007.
• PHOG Descriptor [8]
▫ Outcomes
Local shape (Given by each divided region)
Spatial layout (Given by HOGs of regions of finer spatial grids)
39. Shape Descriptors
• GIST Descriptor [9]
▫ A holistic representation of an image
▫ Spatial Envelope
Described by boundary of surface of image and inner
textures
Properties
Naturalness, Openness, Roughness, Ruggedness,
Expansion
▫ Estimating spatial envelope properties
By calculating the energy spectrum of the image
(DFT)
[9] Oliva, Aude, and Antonio Torralba. "Modeling the shape of the scene: A holistic representation of the spatial envelope."
International journal of computer vision 42, no. 3 (2001): 145-175.
40. Text Feature Extraction
• Suitable text feature extraction techniques are
limited, why?
• Technique
• document is encoded as a histogram of words [10]
• select the set of keywords which are usually regarded
as an important keys, to create a feature vector [11]
• using WordNet lexical to create the feature vector [12]
• using uClassify web-service to create the feature
vector
[10] Kaski, Samuel, Timo Honkela, Krista Lagus, and Teuvo Kohonen. "WEBSOM–self-organizing maps of document
collections." Neurocomputing 21, no. 1 (1998): 101-117.
[11] Chumwatana, Todsanai, K. Wong, and Hong Xie. "A SOM-Based Document Clustering Using Frequent Max Substring
for Non-Segmented Texts." Journal of Intelligent Learning Systems & Applications 2 (2010): 117-125.
[12] Gharib, Tarek F., Mohammed M. Fouad, Abdulfattah Mashat, and Ibrahim Bidawi. "Self Organizing Map-based
Document Clustering Using WordNet Ontologies." International Journal of Computer Science 9 (2012).
42. uClassify output
docs sport games society Recreation Arts Science Business Computers Health Home
doc1 95.5 4.3 0.1 0 0 0 0 0 0 0
doc2 0 0 0 0 0 84 16 0 0 0
“Football refers to a number of sports that involve, to varying
degrees, kicking a ball with the foot to score a goal. The most popular
of these sports worldwide is association football, more commonly
known as just "football" or "soccer". Unqualified, the word football
applies to whichever form of football is the most popular in the
regional context in which the word appears, including association
football, as well as American football, Australian rules football,
Canadian football, Gaelic football, rugby league, rugby union and
other related games. These variations of football are known as
football codes.”
http://en.wikipedia.org/wiki/Football
44. Information Visualization
• The process of showing information in more intuitive
manner
• Today data analysts preferred to use computer generated
models
• Information Visualization can be represented by
following taxonomy
Visualizing
Tools
2D
2D
perspective
3D
2D perspective 3D
perspective
47. Existing Similar Systems
• Watson is an artificial intelligence computer
system capable of answering questions in
natural language.
IBM Watson [17]
[17] IBM Watson. n.d. http://www-03.ibm.com/innovation/us/watson/index.shtml (accessed April 28, 2013).
48. Significance of Watson
• The ability to discern double meanings of words,
puns, rhymes, and inferred hints.
• Extremely rapid responses
• The ability to process vast amounts of information to
make complex and subtle logical connections
Limitations of Watson
• Cannot process multi-modal data
• Cannot build a higher level perception of its data
• Watson does not learn incrementally
• Requires complex infrastructure
49. Contributions of Project Members
Major Task(s) Contributor
Implement SOM Ruwan
Research and Implement GSOM Thushan
Testing GSOM Lasindu
Research IKASL Thushan
Research Fuzzy integral Lasindu
Research Image Feature Extraction
Color Buddhima
Edge Ruwan
Shape Thushan
Research Text Feature Extraction Ruwan
Research WSD Lasindu
Research Information Visualization Buddhima
50. References
[13] Gephi, an open source graph visualization and manupulation software. n.d.
http://gephi.org/ (accessed April 28, 2013).
[14] BioLayout Express 3D. n.d. http://www.biolayout.org/ (accessed April 28, 2013).
[15] Ubigraph: Free dynamic graph visulization software. n.d.
http://ubietylab.net/ubigraph/ (accessed April 28, 2013).
[16] Secrier, Maria. Arena3D. n.d. http://arena3d.org/ (accessed April 28, 2013).
Based on artificial perception model proposed by Jeewani [1]Accumulate real-time, temporal, multi-modal or multi source data in an organized manner.Enable data analysts to elicit information about how patterns of data change over time, by means of exploratory data mining techniques.Allow users to query and visualize information in an intuitive and simple manner.
How to build relationships between multi-modal information?This is important because a perception at an instance is a combination of sensory info from several modalitiesHow to build relationships between multi-modal information?What is the best color space to represent images?IKASL perf. ImprovementsLearning rateDisparity measureFuzzy integral for aggregation
Image feature extractionFixing bugs of the found feature extraction librariesWordnetResult of wordnet lexical category based feature vectors was undesirableText feature extraction techniquesAchieving Word Sense Disambiguation GSOMMeasuring the cluster quality
IKASLVerifying results of IKASL algorithm
LimitationsThe size of the grid and the number of nodes need to be predetermined The analysts not being aware of the topology of the data spaceThe input vectors need to be compared with weights of each and every neuron. Since the map size is very large initially, it might cause a computational overhead.
Growing Cell Structures (GCS’s) The main advantage over Self Organizing Map, is the ability of the model to automatically find a suitable network structure and sizeStarts with a triangle of cells at random positions in Rnand grows over the non-zero probability space.The algorithm results in a network graph structure G=(V,E)Works well with relatively low-dimensional data, but the mapping cannot be guaranteed to be planar for high-dimensional data. This causes problems in visualizing high-dimensional dataNeural Gas Algorithm (NGA) An unsupervised self-generating neural network inspired by SOMThe network starts with a fixed number of units floating in the input vector space. When the inputs are presented to the network, units are adapted and connections are created between the winning units and the closest competitor.The dimensionality of the Neural Gas depends on the respective locality of the input data. Therefore, the network can develop different dimensionality for different sections, which can result in visualization difficulties.Incremental Grid Growing (IGG)IGG network starts with a small number of initial nodes and generates nodes from the boundary of the network using a growth heuristic Connections are added/removed when an internode weight difference drops/increases a threshold value.Adding nodes only at the boundary allows the IGG network to always maintain a two-dimensional structure, which results in easy visualization.
Square shape let the map to grow in any direction depending on the input values. It represents a good starting position to implement a 2-D lattice structure. Growth Threshold (GT) value will be calculated using the number of Dimensions in the input and Spread Factor (SF) to initiate node generation. A high GT value will result in less spread of map, on the other hand a low GT will produce a well-spread map. HErr Value is initialized to ‘0‘ and it will keep track of the highest accumulated quantization error value in the network.
Present InputFind winner based on Euclidian distancesWeight adaptation of winner and neighboring weights (Neighborhood radius and learning rate is exponentially reduced in the next iterations)Difference between Winner and input is accumulated as the Error value of the WinnerIf 𝐻𝐸𝑟𝑟≤𝐺𝑇 and ‘i’ is a boundary node =>grow, if non-boundary =>distribute weightsInitialize new node weight vectors and Learning RateRepeat steps until all inputs have been presentedand node growth is reduced to a minimumlevel.
Input data are repeatedly entered to the network until convergence is achieved. The smoothing phase is stopped when the error values of the nodes in the map become very small. During the Smoothing Phase, node growth is not possible.
SOM - The size of the grid and the number of nodes need to be predetermined In GSOM, having fewer nodes at the early stages and localized weight adaptation results in less processing time. GSOM does not require an ordering phase, which further reduces processing time. Number of nodes required to represent a set of data is less than that of a comparable SOM.
Stable and plasticSimultaneously Learn new things while not disrupting past learnings significantly. Activate similar areas for similar inputs, but have ability to deform
Why std. tools not suitable
Watson was specifically developed to answer questions on the quiz show ―Jeopardy! Jeopardy! is an American television quiz show which consists of numerous questions in every domain.In 2011, Watson competed on Jeopardy against two of the former Jeopardy Champions and received the first prize of $1 million.
IBM selected Jeopardy as the ultimate test of Watson’s capabilities because it relied on many human cognitive abilities traditionally seen beyond the capability of computers, such as;Within a human, these capabilities come from a lifetime of participation in human interaction and decision-making.For a computer, replicating these capabilities was an enormous challenge.Watson winning Jeopardy shows the power of next generation cognitive systems.Today Watson is used widely in many domains. Few example domains are healthcare, finance, etc.
