This document describes WebCat, an app that automatically classifies websites into predefined categories using Apache Spark. It trains a Naive Bayes classifier on over 2,500 previously categorized websites and their crawled text features. New websites are categorized by extracting their text features and having the model make predictions. The high-level architecture includes link collection and crawling services that scale independently, with the classified data stored in a database and trained on Spark. Suggested improvements include allowing users to provide feedback to update the model, upload their own training data, define custom categories, and implement a public API.

1. Website classification
using Apache Spark
Amith Nambiar

2. Demo of the WebCat app

3. Business problem
Automatically classify new websites into one or more
predefined categories.

4. Why?
Web logs collected from data providers have new websites
popping up everyday. And these need to be categorized
before they are presented to customers in reports - daily.

5. Website classification using
Apache Spark's MLlib.

6. Training Data
Starting point was already categorised data in the
form:
URL, category_id
www.linux.com, 10 -> (Computers and Internet)
www.coles.com.au, 20 -> (Shopping and Classifieds)

7. Training Data
Developed a crawler to crawl each of the categorised websites
2,550 websites picked for initial training and test data.
URL, Category_Id -> URL, Category_Id, Features
www.coles.com.au, 10 ->
www.coles.com.au, 10, groceri deliv kitchen bench custom receiv deliveri first
spend onlin liquorland cole card cole insur apparel cole credit card locat hour
look hervey hervey today normal store hour monday friday 8am special store hour
saturday decemb sunday decemb store store search suburb postcod search
suburb postcod select locat suburb locat found pleas store store state recip
inspir recip tast cole partner tast weekli plan easier visit tast cook month cole
magazin everyday ingredi sensat meal famili friend latest cole cole handi video
recip creativ kitchen visit cole youtub rang rang product product bakeri dairi fresh
fruit cole mobil card heston liquor special diet gluten kosher foodtruck term condit
corpor respons corpor respons supplier commit work …
Crawled, Stemmed and
removed stop words from
the data for the website

8. Bayes's theorem

9. Website classification using
Naive Bayes
Naive Bayes Classifier are a family of simple probabilistic
classifiers based on applying Bayes' theorem with strong
(naive) independence assumptions between the features.

10. tf-idf for weighting
In information retrieval, tf–idf, short for term frequency–inverse document frequency,
is a numerical statistic that is intended to reflect how important a word is to a document
in a collection or corpus
https://en.wikipedia.org/wiki/Tf-idf

11. tf-idf
The tf-idf value increases proportionally to the number of times a word appears in the
document, but is offset by the frequency of the word in the corpus, which helps to adjust for
the fact that some words appear more frequently in general.

12. Training Data from Database/HDFS
TermDoc RDD’s
tf-idf’s
Array of LabeledPoint(classId, vector)
Calculate tf-idf’s on the features.
Create a LabelPoint for each of the
training data row
model = NaiveBayes.train(labelPoints)
Train the NaiveBayes Model
model.predict(feature_vector)
Predict class
New Data e.g “Automotive”
Each row of Training data (website) is
turned into this form:
(ClassId, Sparse Vector) in the form:
5.0, [100,(1,44,..),(0.3,0.12,…)]

13. API first for Data science
http://engineering.pivotal.io/post/api-first-for-data-science/

14. High Level Architecture of WebCat

15. High level architecture of WebCat
Webcat
App
Queues/Topics
Link Collector Service Link Crawler Service
Classification Service
Training Data
Database
Apache Spark
Categorize
www.coles.com.au
Category is
“Shopping and
Classifieds”
Category is
“Shopping and
Classifieds”
Scale the Crawler
service independent
of the rest of the
services

16. WebCat dashboard on PWS - Pivotal Web Services
Note that the crawler service is scaled up to 6 instances for better performance.

17. Ideas for improving WebCat?

18. User feedback loop to update the model on
incorrect predictions
Webcat
App
Queue with topics
Link Collector Service Link Crawler Service
Classification Service
Training Data
Database
Apache Spark
Categorise
www.bmw.com.
We think it is
“Electronics” - Did we
get it right?

19. Upload your own data - (website, category) pairs
Webcat
App
Queue with topics
Link Collector Service Link Crawler Service
Classification Service
Training Data
Database
Apache Spark
I know kogan.com.au
belongs to category
“Shopping and
Classifieds” - add it to
the training data please.
More data = Better predictions?

20. User defined categories
e.g realestate.com.au -> “Real Estate”
Webcat
App
Queue with topics
Link Collector Service Link Crawler Service
Classification Service
Training Data
Database
Apache Spark
Create New
Category
“Real Estate”

21. Provide a publicly available API for
categorised websites
Webcat
App
Queue with topics
Link Collector Service Link Crawler Service
Classification Service
Training Data
Database
Apache Spark
GET /websites/{id}/category
GET /websites/{id}/features
…

22. WebCat on Apache Madlib
http://madlib.incubator.apache.org/