A Pen Based Intelligent System for Educating Arabic Handwriting Deep Learning

A Pen Based Intelligent System for Educating Arabic Handwriting
A Pen Based Intelligent System
for Educating Arabic Handwriting

Introduction
Previous Related Work
Background and Preliminaries
Arabic Handwritten Digits Recognition System
Arabic Handwritten Characters Recognition System
Intelligent Arab Teaching System
Conclusion and Future Work

Problem
Definition
Objectives
Hypothesis
Research
Questions

 Traditional Learning

 Complexity of Arabic characters.
 Expertise needs in traditional recognition systems.
 Traditional hand-crafted features.
 Arabic handwritten characters mistakes.

Description Template character Sample character
Missing stroke error
Extra stroke error
Broken stroke error

Description Template character Sample character
Concatenated stroke error
Stroke order error
Direction Error

 The main goal of this work is to build and develop an intelligent
tutor system for detecting Arabic preschool children handwriting
difficulty based on immediate feedback.
 The second goal of this work is to use deep learning architectures
to recognize Arabic handwritten characters and digits.

 Our hypothesis is that applying Convolutional neural networks and
stacked auto-encoder to classify Arabic handwritten characters and
digits.
 We expect that a simple Convolutional neural network and stacked
Autoencoder will success to obtain competitive results.
 We believe that implementing deep learning for this domain
will be moderately easy.

 Is training a deep learning architectures on handwritten characters
and digits better than computing numeric features from handwritten
characters and digits and training a simpler classifier ?
 Is deep learning feasible with the resources we have ? Is there any
advantage to use GPU acceleration ?
 Can we simplify the pipeline for Arabic handwritten characters and
digits recognition ?

 Can preprocessing and features extraction be replaced by more
layers on deep learning ?
 What are the best parameters for our deep learning architectures ?
 What are the advantages of using a automatic feedback to
determine handwriting stroke mistakes for Arab children ?
 Are deep learning architectures are a good option for future
research ?

Arabic Handwritten
Digits Recognition
Arabic Handwritten
Characters Recognition
Intelligent
Tutoring Systems

Authors Year Method Dataset Error Rate
Alwzwazy et al. 2016 CNN 46,612 4.3%
Kathirvalavakumar
and Palaniappan
2015 K-NN 6670 1.3%
Takruri et al. 2014 SVM 3510 12%
AlKhateeb et al. 2014 DBN 70,000 14.74%
Majdi Salameh 2014 Fuzzy 2000 5%
 CNN: Convolutional Neural Network
 SVM: Support Vector Machine
 DBN: Dynamic Bayesian Network

Pandi selvi and
Meyyappan
2013 NN Samples 4%
Mahmoud 2008 SVM 21120
0.15% and
2.16%
Melhaoui et al. 2011 CL 600 1%
 NN: Neural Network
 SVM: Support Vector Machine
 CL: Characteristics Loci

Hussien et al. 2015 HNN 8 22.75%
ElAdel et al. 2015 CNN 6000 6.08%
Elleuch et al. 2015 DBN 6600 2.10%
Shatnawi and
Abdallah
2015 K-NN 1824 26.6%
 CNN: Convolutional Neural Network
 HNN: Hopfild Neural Network
 DBN: Dynamic Bayesian Network

Kef et al. 2015
Fuzzy
NN
3840 6.20%
Alabodi and Li 2014 GF 3840 6.70%
Lawgali et al. 2014
DCT
NN
6033 9.27%
 NN: Neural Network
 DCT: Discrete Cosine Transform
 GF: geometrical features

Authors Year Language Method
Will Tang et al. 2014 Chinese special relation matrix
H. Bezine & A.
Alimi
2013 Arabic Attributed Relational Graph
Fork and Chan 2013 Latin Algorithms
Priyankara et al. 2013 Latin logical and spatial relationships
Hammadi et al. 2012 Arabic
Graph matching
A* algorithm
Chea et al. 2012 Latin Chain code and direction code

Authors Year Language Method
Neo et. al. 2012 Latin Chain code and direction code
Chen et al. 2007 Chinese
Feature extraction
spatial relationships
Hu et al. 2007 Chinese
graph matching technique
A* algorithm
Kai-Tai Tang et
al.
2006 Chinese
Hungarian method
Euclidean distance
Tang and Leung 2006 Chinese Feature extraction techniques

Pattern Recognition
Neural Network
Activation Functions
Deep Learning
Stacked Autoencoder
Convolutional Neural Network
Intelligent Tutoring Systems

Neural Network
Deep Learning
Stacked Autoencoder
Pattern Recognition

 Pattern recognition is a branch of machine learning
 Machine learning is divided into two main types:
 supervised learning
 unsupervised learning
 Supervised learning learn from labeled training data
 UnSupervised learning learn from unlabeled training data

 Artificial neural networks or simply neural networks are one of the
most important nonlinear recognition classifiers used today.
Axon
Terminal Branches
of Axon
Dendrites
S
x1
x2
w1
w2
wn
xn
x3 w3

 Common
nonlinear
activation
functions

Most deep networks use Rectified Linear Unit (ReLU)
 ReLU trains much faster
 RelU more expressive than logistic function
 ReLU prevents the gradient vanishing problem.

Recognition
Neural Network
Deep Learning
Stacked Autoencoder

 Deep learning (DL) is a hierarchical structure network which through
simulates the human brain’s structure to extract the internal and
external input data’s features

 The main component in Stacked Autoencoder is combined
Autoencoder. Autoencoder is a simple three-layer neural network
including an encoder and a decoder where output units are directly
connected back to input units.
 In Autoencoder: the number of input units equal the number of
output units.

 Hidden Layer Equation:
 Sigmoid Equation:
 Output Layer Equation:

 The first sparse auto-encoder produce the primary feature.

 The second sparse auto-encoder produce the secondary feature.

 Soft-max classifier:

 Proposed Stack Auto-Encoder architecture:

 Convolution Neural Networks (CNN) is supervised learning and a family
of multi-layer neural networks particularly designed for use on two
dimensional data, such as images and videos.
 A CNN consists of a number of layers:
 Convolutional layers.
 Pooling Layers.
 Fully-Connected Layers.

 Convolutional layer acts as a feature extractor that extracts features
of the inputs such as edges, corners , endpoints.

 Feature extraction layer
10-1
10-1
10-1
Convolve with Activation
Kernel

features
 Feature extraction layer

 The pooling layer reduces the resolution of the image that
reduce the precision of the translation (shift and distortion) effect.

ConvInput Pooling

 fully connected layer have full
connections to all activations in the
previous layer.
 Fully connect layer act as classifier.

 Intelligent tutoring systems (ITS) are computational agents whose
purpose is to facilitate learning, usually without the help of a
human teacher.
 ITS products can be organized on three categories:
 Read Systems
 Guided Systems
 Immediate Error Detection Systems

 Read systems are static not interactive systems; read-only because it
cannot provide the practice of writing.

 The Guided systems allow children to practice writing in a guided
method and on-line.

The immediate error detection Systems gives access to the
children to practice free writing mode, and provide
immediately a feedback to indicate if there are any errors
in the writing.

Arabic
Handwritten Digit
Dataset
Convolutional
Neural Network
based on LeNet-5
Stacked
Autoencoder
Convolutional
Neural Network
Optimized

 The MADBase is a modified version of the ADBase
benchmark that has the same format as MNIST benchmark.
 MADBase is composed of 70,000 digits written by 700 writers.
 The databases is partitioned into two sets:
 60,000 Training Data
 10,000 Testing Data

Training Data
Testing Data

 CCN based on LeNet-5 architecture was used with an 8 layers
including one input layer, one output layer, two Convolutional
layers and two sub-sampling, two fully connected layers as multi-
layer perceptron hidden layers for nonlinear classification.

 The experiments outcomes was performed by MATLAB 2016a
programming environment.
 Why MATLAB 2016b:
 Neural Network Toolbox (contain deep leaning algorithms)
 Statistics and Machine Learning Toolbox
 Image processing Toolbox

 Misclassification Rate

 Confusion Matrix:

 A Stacked Autoencoder (SAE) is a neural network consisting of
multiple layers of sparse Auto-encoders in which the outputs of each
layer is wired to the inputs of the successive layer.

 Size of input layer is 784 x 60,000
 Hidden layer for primary feature is 392
 Train Autoencoder with 60,000 training set
 Encode The training data with Autoencoder to produce the features
 Encoder outcome is 392 x 60,000 features
The First Autoencoder:

 Hidden layer for secondary feature is 196
 Train Autoencoder with 60,000 features set
 Encode The training features with Autoencoder to produce the features
 Encoder outcome is 196 x 60,000 features
The Second Autoencoder:

 Train a soft-max layer to classify the 196 x 60,000 feature vectors.
 The soft-max layer is trained to produce 10 output class.
The Soft-max Classifier:

 The proposed stacked Autoencoder produce ten output Arabic digits

 Confusion Matrix
 Feed Testing dataset to
proposed Stacked Autoencoder
 Misclassification error is 2.2%

 To produce better outcomes,
fine-tuning was used to update
all SAE parameters.
 Feed Training dataset to
 Feed Testing dataset to
 Misclassification error is 1.5%

Authors Database Images Error Rate
Takruri et al. Private 3510 12%
AlKhateeb et al. ADBase 70000 14.74%
Majdi Salameh Fonts 2000 5%
Melhaoui et al. Private 600 1%
Pandi selvi & Meyyappan Private Samples 4%
Mahmoud Private 21120 0.15%
Kathirvalavakumar & Palaniappan Private 6670 1.3%
Alwzwazy et al. Private 46,612 4.3%
Our Approach MADBase 70000 1.5%

 We built a new CNN architecture:
 INPUT → CONV → RELU → Max-pooling → CONV → RELU →
Max-pooling → FC → RELU → FC → Output

 The CNN architecture

 Error Rate= 0.8%

 The total of wrong
classification is 85 from 10k.

Authors Database Images Error Rate
Takruri et al. Private 3510 12%
AlKhateeb et al. ADBase 70000 14.74%
Majdi Salameh Fonts 2000 5%
Melhaoui et al. Private 600 1%
Pandi selvi & Meyyappan Private Samples 4%
Mahmoud Private 21120 0.15%
Kathirvalavakumar & Palaniappan Private 6670 1.3%
Alwzwazy et al. Private 46,612 4.3%
Our Approach MADBase 70000 0.85%

Arabic Handwritten
Characters DataSet
Stacked
Autoencoder
Convolutional
Neural Network

 We collect a dataset that composed of 16,800 characters written by 60
participants, the age range is between 19 to 40 years.
 The forms were scanned at the resolution of 300 dpi. Each block is
segmented automatically using Matlab 2016a to determining the
coordinates for each block.
 The database is partitioned into two sets: a training set (13,440
characters to 480 images per class) and a test set (3,360 characters to
120 images per class).

 Each participant wrote each
character (from ’alef’ to
’yeh’) ten times on two forms

 The different shapes of some
Arabic characters

 Train Autoencoder with 13,440 training
set
 Encode The training data with
Autoencoder to produce the features
 Encoder outcome is 512 x 13,440
features
The First Autoencoder

 Train Autoencoder with 13,440 features
 Encode The features data with
Autoencoder to produce the features
 Encoder outcome is 256 x 13,440
features
The Second Autoencoder

 Train a soft-max layer to classify the 256
x 13,440 feature vectors.
 The soft-max layer is trained to produce
28 output class.
The Soft-max classifier

 The proposed stacked Autoencoder
produce 28 output Arabic characters

 # of Misclassification
= 1208 from 3,360
Class 1 2 3 4 5 6 7
Arabic Character alef beh teh theh jeem hah khah
Correct Classification 111 86 60 64 67 66 65
Wrong Classification 9 34 60 56 53 54 55
Classification Accuracy 92.5% 71.7% 50% 53.3% 55.8% 55% 54.2%
Miss-Classification 7.5% 28.3% 50% 46.7% 44.2% 45% 45.8%
Class 8 9 10 11 12 13 14
Arabic Character dal thal reh zain seen sheen sad
Classification Accuracy 70.8% 62.5% 77.5% 64.2% 65.0% 63.3% 53.3%
Miss-Classification 29.2% 37.5% 22.5% 35.8% 35.0% 36.7% 46.7%
Class 15 16 17 18 19 20 21
Arabic Character dad tah zah ain ghain feh qaf
Class 22 23 24 25 26 27 28
Arabic Character kaf lam meem noon heh waw yeh

 Classification Matrix
Class 1 2 3 4 5 6 7
Arabic Character alef beh teh theh jeem hah khah
Classification Accuracy 100% 96.70% 91.70% 91.70% 95.80% 97.50% 93.30%
Class 8 9 10 11 12 13 14
Arabic Character dal thal reh zain seen sheen sad
Classification Accuracy 95.00% 91.70% 100% 87.50% 97.50% 95.80% 98.70%
Class 15 16 17 18 19 20 21
Arabic Character dad tah zah ain ghain feh qaf
Class 22 23 24 25 26 27 28
Arabic Character kaf lam meem noon heh waw yeh

 The total of wrong
classification is 173 from
3,360.

Authors Database Images Accuracy Rate
Hussien et al. Private 8 Letters 77.25%
ElAdel et al. IESK-arDB 6000 93.92%
Elleuch et al. HACDB 6600 97.9%
Shatnawi and Abdallah Private 1824 73.4%
Kef et al. IFN/ENIT 3840 93.8%
Alabodi and Li IFN/ENIT 3840 93.3%
Lawgali et al. IFN/ENIT 6033 90.73%
Our Approach Private 16800 94.85%

Interfaces
Architecture
Components
Database
Agents
Results

 The AKT system implementation follows the MVC design pattern.
 The Model–View–Controller (MVC) is a software architectural
pattern used to design a software system.
View
Controller
Model
Database

We Develop two interfaces:
 Children & Tutor Interface
 Learning Interface

Intelligent Arab Teaching is a multi-agent system based on
three components:
 Learning Agent
 Feedback Agent
 Evaluation Agent

Learning agent based on four components:
1) stroke number
2) stroke similarity
3) stroke order
4) stroke direction

Stroke Number Arabic characters
One ،‫ح،د،ر،س،ص،ط،ع،ل،م‬‫ﻫ‬‫،و‬
Two ‫أ،ب،ج،خ،ذ،ز،ض،ظ،غ،ف،ك،ن‬
Three ‫ت،ق،ي‬
Four ‫ث،ش‬

Basic stroke
Similar
characters
Basic stroke
Similar
characters
‫ٮ‬ ‫ب،ت،ث‬ ‫ط‬ ‫ط،ظ‬
‫ح‬ ‫ج،ح،خ‬ ‫ص‬ ‫ص،ض‬
‫د‬ ‫د،ذ‬ ‫ع‬ ‫ع،غ‬
‫ر‬ ‫ر،ز‬ ‫ٯ‬ ‫ف،ق‬
‫س‬ ‫س،ش‬ ‫ل‬ ‫ل،ك‬

Arabic Character Stroke #1
‫ح‬ ‫ح‬
‫د‬ ‫د‬
‫ر‬ ‫ر‬
‫س‬ ‫س‬
‫ص‬ ‫ص‬
‫ع‬ ‫ع‬
‫ل‬ ‫ل‬
‫م‬ ‫م‬
‫ﻫ‬ ‫ﻫ‬
‫و‬ ‫و‬ Stroke #2
‫أ‬ ‫ا‬ ‫ء‬
‫ب‬ ‫ب‬ .
‫ج‬ ‫ح‬ .
‫خ‬ ‫ح‬ .
‫ذ‬ ‫د‬ .
‫ز‬ ‫ر‬ .
‫ط‬ ‫ﺻ‬‫ـ‬ ‫ا‬
‫ض‬ ‫ص‬ .
‫غ‬ ‫ع‬ .
‫ف‬ ‫ٯ‬ .
‫ك‬ ‫لـ‬ ‫ء‬
‫ن‬ ‫ں‬ . Stroke #3
‫ت‬ ‫ٮ‬ . .
‫ظ‬ ‫ﺻ‬‫ـ‬ ‫ا‬ .
‫ق‬ ‫ٯ‬ . .
‫ي‬ ‫ى‬ . . Stroke #4
‫ث‬ ‫ٮ‬ . . .
‫ش‬ ‫س‬ . . .

Feedback agent based on:
1) Stroke direction detector
2) Stroke order detector

Intelligent Arab
Teaching system
was used chain
code to encode
a movement.
Code Angle (θ) Direction
C0 355°<θ<5° Right
C1 5°<θ<85° Up Right
C2 85° <θ< 95° Up
C3 95°<θ< 175° Up Left
C4 175° <θ< 185° Left
C5 185° <θ< 265° Down Left
C6 265° <θ< 275° Down
C7 275° <θ< 355° Down Right

 The freeman chain code algorithm is defined in the following three
steps:
Step 1: The absolute difference between y-axis
Step 2: The absolute difference between x-axis
Step 3: Calculate the angle

When the preschool children put his/her finger on touch
screen, the system detected the sequence of x−y point
coordinate.
When The children move his/her finger up, the system
store those sequence of points as stroke.

In this part, Intelligent Arab Teaching system indicates
the children level of understanding of learning
handwriting character concepts.
Intelligent Arab Teaching system use fuzzy logic to
evaluate Arabic children.

 The Fuzzy Rules
Example

 Fuzzy System & membership functions

 Arabic Character Difficulty Membership function

 Arabic Character Error Membership function

 Time Consumed Membership Function

 Arab Children Age Membership function

 Children Evaluation Membership Function

 Directional stroke error of
character Seen.

 Stroke position of Arabic
character Teh.

 Extra stroke error of
Arabic character Seen.

 The Egyptian preschool children and their tutors from Benha city
were asked to use the Intelligent Arab Teaching system as part of
their learning of write Arabic alphabet.
 200 questionnaires were collected from the educators & some
children in the experimental groups after the computerized training
program.
 The results specify that educators rated the Intelligent Arab Teaching
very highly on acceptability for both likeability and ease of use.

1) Improve children handwriting
‫تحسين‬‫اداء‬‫كتابة‬‫االطفال‬
(a) No (b) Neutral (c) Yes
(‫ج‬)‫نعم‬(‫ب‬)‫عادى‬(‫أ‬)‫ال‬
2) Easy to learn Application
‫سهولة‬‫تعلم‬‫التطبيق‬
3) Easy to use
‫سهولة‬‫االستخدام‬
4) Interactive with kids
‫متفاعل‬‫مع‬‫االطفال‬
# Yes Neutral No
1 85% 10% 5%
2 90% 5% 5%
3 90% 5% 5%
4 75% 10% 15%

In this presentation, we have demonstrated the
effectiveness of deep learning for Arabic handwritten
Arabic characters and digits recognition.
Compared to other machine learning architectures, SAE
and CNN have better performance in both images and
big data of images.

 In Arabic handwritten digits based on MADBase dataset:
 We achieved misclassification error rates 12% using CNN
based on LeNet-5.
 We achieved misclassification error rates 1.5% using SAE.
 We achieved misclassification error rates 0.85% using CNN.

In Arabic handwritten characters based on our dataset:
 We achieve 36% misclassification error rates using
SAE.
 We achieve 5.15% misclassification error rates using
CNN.

An intelligent tutoring system for handwriting Education
was developed, called Intelligent Arab Teaching system.
The main purpose of Intelligent Arab Teaching is to help
Arab preschool children to diagnose their handwriting
mistakes.

Work on Arabic handwritten word recognition using deep
learning techniques.
Improving the performance of handwritten Arabic
character recognition.
Improving our Intelligent Arab Teaching system to detect
all types of the Arabic handwriting learning mistakes.

facebook.com/mloey
mohamedloey@gmail.com
twitter.com/mloey
linkedin.com/in/mloey
mloey@fci.bu.edu.eg

A Pen Based Intelligent System for Educating Arabic Handwriting Deep Learning

More Related Content

What's hot

Viewers also liked

Similar to A Pen Based Intelligent System for Educating Arabic Handwriting Deep Learning

More from Mohamed Loey

Recently uploaded

A Pen Based Intelligent System for Educating Arabic Handwriting Deep Learning