Ying Xie & Pooja Chenna, Kennesaw State University, present at the 2015 HPCC Systems Engineering Summit Community Day. Deep learning has been emerged as a new breakthrough in the area of Machine Learning and revitalized the research in Artificial Intelligence (AI). Deep learning techniques have been widely used in image recognition and natural language processing. In this presentation, we will show our implementation of an important deep neural network architecture that is called Deep Belief Network in ECL. We will further illustrate how to apply our implementation to solve the problem of network intrusion detection based on massive data sets on HPCC Systems. Additionally, we will report our study on how to optimally configure a stacked auto encoder, another deep learning architecture, on HPCC Systems for the purpose of both supervised and unsupervised learning on different types of data sets.

1. Studies of HPCC Systems® from
Machine Learning Perspectives
Ying Xie, Pooja Chenna, Ken Hoganson
Department of Computer Science
Kennesaw State University

2. Outline
• Leverage and Enhance Deep Learning Capability of HPCC
Systems®
• Comparative Studies of HPCC Systems® and Hadoop Systems

3. Leverage and Enhance Deep Learning
Capability of HPCC Systems®
• Deep learning has been emerged as a recent breakthrough in
the area of Machine Learning and revitalized research
activities in Artificial Intelligence (AI).
• The development of deep learning techniques has been
primarily driven by big data analytics.
• Since HPCC Systems is a well established big data platform, it
is very important to leverage and enhance the deep learning
capability of the HPCC Systems platform.

4. Images are copied from neuralnetworksanddeeplearning.com
• From Neural Network to Deep Learning
A typical multilayer perceptron

5. A deep neural architecture

6. Stacked Autoencoder

7. Images are copied from ufldl.stanford.edu
• Stacked Autoencoder

8. This code was implemented by Maryam Mousaarab Najafabadi from F

9. Deep Belief Network

10. Restricted Boltzmann Machine
Images are copied from http://deeplearning4j.org/

11. Deep Belief Network

12. Visualize High Dimensional Data

13. • Iris Data

14. Use DBN to conduct dimension reduction

17. • Breast Cancer Wisconsin (Original) Data Set

18. Use DBN to conduct dimension reduction

21. • Glass Data

22. Use stacked auto-encoder to conduct dimension reduction

27. • Based on the visualization, we may even gain
some good ideas what classification algorithm
may work well

28. Classification Algorithm #Miss-classified Instances
Logistic 72
MLP 66
Simple Logistic 70
7NN 76
5NN 70
3NN 64

29. Mapping Data to Higher Dimensional Spaces

30. • Blood Transfusion Data

32. • Classification performance on higher dimensional space
Original Space (4 Dim)
(#Miss-classified Instances)
Higher Dim (10 Dim)
(#Miss-classified Instances)
Logistic 171 157
MLP 160 166
Mapped to higher dimensional space by Stacked Auto-Encoder
Mapped to higher dimensional space DBN

33. • Wine Data

34. • Classification performance on higher dimensional space
Original Space (13 Dim)
(#Miss-classified Instances)
Higher Dim (15 Dim)
(#Miss-classified Instances)
Logistic 10 6
MLP 4 4
Original Space (13 Dim)
(#Miss-classified Instances)
Higher Dim (15 Dim)
(#Miss-classified Instances)
Logistic 10 7
MLP 4 2
Mapped to higher dimensional space by Stacked Auto-Encoder

35. • For a given data set, we can explore which
combination of deep learning mapping
techniques, dimensional space, and
supervised learning model may yield best
classification result

36. • For instance – Breast Cancer Data
3 Dim. Space
(#Miss-classified
Instances)
6 Dim. Space
(#Miss-classified
Instances)
9 Dim. Space
(#Miss-classified
Instances)
12 Dim. Space
(#Miss-classified
Instances)
Logistic 22 21 24 25
MLP 21 22 30 21
3 Dim. Space
(#Miss-classified
Instances)
6 Dim. Space
(#Miss-classified
Instances)
9 Dim. Space
(#Miss-classified
Instances)
12 Dim. Space
(#Miss-classified
Instances)
Logistic 22 23 24 25
MLP 20 21 30 22
Mapped to different dimensional spaces by Stacked Auto-Encoder
Mapped to different dimensional spaces by DBN

37. • Our next step: try to implement a meta
supervised learning algorithm on HPCC
– This algorithm will automatically map the given data
to different dimensional spaces by using both stacked
auto-encoder and DBN
– Then classification models will be trained on all
dimensional spaces in a distributed manner
– Cross-validation will be used to select the best
performed model as the final output.

38. Implementation of Deep Belief Network on
HPCC

39. Our Implementations of Deep Learning
on HPCC
• Restricted Boltzmann Machine (RBM) with
Contrastive Divergence learning algorithm
• Deep Belief Network by stacking RBMs
• Supervised Deep Belief Network

40. Machine Learning Routines
• Utility Module
• Matrix Library
• Dense Matrix Library
• PBblas

41. Restricted Boltzmann Machine - RBM
v
h
v’
h’

42. w(t+1) = w(t) + α(vhT – v’h’T)

44. Stacking Boltzmann Machines -
Deep Belief Network

45. Final Output
Input Parameters
• Iris Data Sample

46. Supervised Boltzmann Machine –
Deep Belief Network
y – actual output
h – hidden samples
v – visible samples

47. Supervised Deep Belief Network

48. • Our ultimate goal is to implement a full-stack
of deep learning techniques on HPCC and
conduct a wide range of experiences to show
how powerful the deep learning engine on
HPCC will be.

49. Comparative Studies of HPCC and Hadoop

50. • HPCC and Hadoop clusters on CSCloud
– HPCC cluster :
• 5 thor nodes
• 5 roxie nodes
• 2 middle-ware nodes
• 1 landing zone node for uploading files
– Hadoop cluster
• 1 job-tracker / name-node
• 1 support system (Web UI, hadoop ecosystem, etc)
• 4 worker nodes: task-tracker / data-node

51. • Algorithms for comparison
– Text Processing Algorithms:
• Word Count
• Inverted Index
– Machine Learning Algorithms:
• Supervised Learning Algorithm - Random Forests
• Unsupervised Learning Algorithm - KMeans
– Graph Algorithm:
• Page Rank

52. Text Processing Algorithms
Data: Authorized version of Bible downloaded from http://av1611.com/
HPCC Implementation of Inverted Index:
http://www.dabhand.org/ECL/construct_a_simple_bible_search.htm (implemented by David
Bayliss, Chief Data Scientist and VP of LexisNexis Risk Solutions)
Hadoop Implementation of Inverted Index: Victor Guana and Joshua Davidson. On Comparing
Inverted Index Parallel Implementations Using Map/Reduce Technical Report. May 09, 2012.
Algorithm HPCC Hadoop
Word Count 1.003 seconds 23.466 seconds
Inverted Index 34.205 seconds 27.047 seconds

53. Machine Learning Algorithms
Data: KDD Network Intrusion Dataset
Reference: C. Blake and C. J. Merz. UCI Repository of machine learning databases. Irvine,
CA: University of California, Department of Information and Computer Science.
[http://www.ics.uci.edu/~mlearn/MLRepository.html]
Description:
– Total number of Instances : 4000000
– Used instances : 20394 randomly picked
– Number of Attributes : 42
Hadoop Libraries – Apache Mahout:
– Version : CDH-5.4.2-1 (Cloudera)
HPCC Machine Learning Library

54. Efficiency:
Algorithm HPCC Hadoop
Random Forests 1 minutes 50 seconds 18 seconds
KMeans 36.675 seconds 1 min 45 seconds

55. Graph Algorithm
Data Sets: Randomly generated graph with 25 nodes with maximum degree 5.
HPCC Implementation of Pageranking – Our team’s implementation
Hadoop Implementation Pageranking - http://blog.xebia.com/2011/09/wiki-
pagerank-with-hadoop/
Algorithm HPCC Hadoop
Page Rank 29.817 seconds 36 minutes

56. THANK YOU