Python 實作 Feature Selection

1. 30 分鐘學會
實作 Python Feature Selection
James CC Huang

2. Disclaimer
• 只有實作
• 沒有數學
• 沒有統計
Source: Internet

3. Warming Up
• 聽說這場分享不會有人問問題 (把講者釘在台上)
• 原 session 只講 40 分鐘，但是今天的分享給了 2 小時
• 考驗我的記憶力和理解力
• 講者講了一大堆名詞但沒有講實作 (不可能有時間講)
• 我用 Python 實作範例
• 希望大家如果跟我一樣，不搞理論也不搞數學統計，回家用剪貼的就可
以用 scikit-learn 做 feature selection

4. Reinventing the Wheel?
Source: P.60 http://www.slideshare.net/tw_dsconf/ss-62245351

5. 進行 Machine Learning 和 Deep Learning…
• 到底需不需要懂背後的數學、統計、理論…？
• 推廣及普及 Machine Learning / Deep Learning
• 工具的易用性及快速開發
• 正反方意見都有
• 正方例子：談到投入大演算 ”… 你會認為這需要繁重的數
學和嚴謹的理論工作，其實不然，反倒這所需要的是從
艱深的數學理論抽離，以便能看到學習現象的整體模
式。” (大演算 The Master Algorithm, P. 40)
• 反方例子：Deep Neural Networks - A Developmental
Perspective (slides, video)

6. 2014 – 2016
台灣資料科學”愛好者”年會
我的分享
一、連續 3 年吃便當的經驗
二、2016 聽完 Feature Engineering in Machine Learning 演講後夢到的東西

7. 三年的進化
• 參加的人愈來愈多
• [不負責任目測] 與會者平均年齡愈來愈大 XD
• 內容愈來愈多、場次愈來愈多
• 演講者身份的改變：教授和來自研究單位變多
• Deep Learning 這個詞出現頻率大幅增加
• $$ 愈來愈貴
• 朝向使用者付費
• 部分付費課程也會持續開課
• 便當沒有進化（都是同樣那幾家）

8. http://datasci.tw/agenda.php

9. http://datasci.tw/agenda.php

10. http://datasci.tw/agenda.php

11. http://datasci.tw/agenda.php

12. http://datasci.tw/agenda.php

13. http://datasci.tw/agenda.php

14. http://datasci.tw/agenda.php

15. Feature Engineering in Machine Learning
Session (Speaker: 李俊良)
Source: http://www.slideshare.net/tw_dsconf/feature-engineering-in-machine-learning

16. 用 Feature Engineering 可否判斷出寫作風
格？
• 羅琳化名寫小說 曝光後銷量飆升
http://www.bbc.com/zhongwen/trad/uk_study/2013/07/130714_ro
wling_novel
• “曾有書評評價新書《杜鵑鳥在呼喚》是部「才華橫溢的處女作」，還有
書評盛讚這名男性作者，能如此精湛地描述女性的服裝。”
• “… 出版( 3 個月)的這部小說，已經售出1500冊。但亞馬遜網站報道說，
周日正午12點後，該書的銷售量飆增，增速高達500000%。”
• 原投影片 P. 14 (Source:
http://www.slideshare.net/tw_dsconf/feature-engineering-in-
machine-learning)

17. Find Word / Doc Similarity with
Deep Learning
Using word2vec and Gensim (Python)

18. Goal (or Problem to Solve)
• Problem: Tech Support engineers (TS) want to “precisely” categorize
support cases. The task is being performed manually by TS engineers.
• Goal: Automatically categorize support case.
• What I have:
• 156 classified cases (with “so-called” correct issue categories)
• Support cases in database
• Challenges:
• Based on current data available, supervised classification algorisms can‘t be
applied.
• Clustering may not 100% achieve the goal.
• What about Deep Learning?

19. Gensim (word2vec implementation in Python)
from os import listdir
import gensim
LabeledSentence = gensim.models.doc2vec.LabeledSentence
docLabels = []
docLabels = [f for f in listdir(“../corpora/2016/”) if f.endswith(‘.txt’)]
data = []
for doc in docLabels:
data.append(open(“../corpora/2016/” + doc, ‘r’))
class LabeledLineSentence(object):
def __init__(self, doc_list, labels_list):
self.labels_list = labels_list
self.doc_list = doc_list
def __iter__(self):
for idx, doc in enumerate(self.doc_list):
yield LabeledSentence(words=doc.read().split(),
labels=[self.labels_list[idx]])

20. Gensim (Cont’d)
it = LabeledLineSentence(data, docLabels)
model = gensim.models.Doc2Vec(alpha=0.025,
min_alpha=0.025)
model.build_vocab(it)
for epoch in range(10):
model.train(it)
model.alpha -= 0.002
model.min_alpha = model.alpha
# find most similar support case
print model.most_similar(“00111105”)

21. 江湖傳言
• 用 Deep Learning 就不需要做 feature selection，因為 deep learning
會自動幫你決定
• From Wikipedia (https://en.wikipedia.org/wiki/Deep_learning):
• “One of the promises of deep learning is replacing handcrafted features with
efficient algorithms for unsupervised or semi-supervised feature learning and
hierarchical feature extraction.”
• 真 的 有 這 麼 神 奇 嗎 ？

22. Feature selection for Iris Dataset as Example
• Iris dataset attributes
1. sepal length in cm
2. sepal width in cm
3. petal length in cm
4. petal width in cm
5. class:
-- Iris Setosa
-- Iris Versicolour
-- Iris Virginica

23. Feature Selection - LASSO
>>> from sklearn.linear_model import Lasso
>>> from sklearn.datasets import load_iris
>>> from sklearn.feature_selection import SelectFromModel
>>> iris = load_iris()
>>> X, y = iris.data, iris.target
>>> print X.shape
(150, 4)
>>> clf = Lasso(alpha=0.01)
>>> sfm = SelectFromModel(clf, threshold=0.25)
>>> sfm.fit(X, y)
>>> n_features = sfm.transform(X).shape[1]
>>> print n_features
2
petal width & petal length

24. Feature Selection - LASSO (Cont’d)
>>> scaler = StandardScaler()
>>> X = scaler.fit_transform(X)
>>> names = iris["feature_names"]
>>> lasso = Lasso(alpha=0.01, positive=True)
>>> lasso.fit(X, y)
>>> print (sorted(zip(map(lambda x: round(x, 4),
lasso.coef_), names), reverse=True))
[(0.47199999999999998, 'petal width (cm)'),
(0.3105, 'petal length (cm)'), (0.0, 'sepal
width (cm)'), (0.0, 'sepal length (cm)')]

25. Feature Selection – Random Forest
>>> from sklearn.datasets import load_iris
>>> from sklearn.ensemble import RandomForestRegressor
>>> iris = load_iris()
>>> X, y = iris.data, iris.target
>>> print (X.shape)
(150, 4)
>>> names = iris["feature_names"]
>>> rf = RandomForestRegressor()
>>> rf.fit(X, y)
>>> print (sorted(zip(map(lambda x: round(x, 4),
rf.feature_importances_), names), reverse=True))
[(0.50729999999999997, 'petal width (cm)'), (0.47870000000000001,
'petal length (cm)'), (0.0091000000000000004, 'sepal width (cm)'),
(0.0048999999999999998, 'sepal length (cm)')]

26. Dimension Reduction - PCA
>>> from sklearn.datasets import load_iris
>>> from sklearn.decomposition import PCA as pca
>>> from sklearn.preprocessing import StandardScaler
>>> iris = load_iris()
>>> X, y = iris.data, iris.target
>>> X = StandardScaler().fit_transform(X)
>>> sklearn_pca = pca(n_components=2)
>>> sklearn_pca.fit_transform(X)
>>> print (sklearn_pca.components_)
[[ 0.52237162 -0.26335492 0.58125401 0.56561105]
[-0.37231836 -0.92555649 -0.02109478 -0.06541577]]

27. There are many others…
這次分享就是僅是把原講者所提到的方式實際做出來
簡單的我做完了, 難的就留給大家去發掘~

28. Reference

29. scikit-learn
• Feature selection
http://scikit-learn.org/stable/modules/feature_selection.html
• sklearn.linear_model.Lasso
http://scikit-
learn.org/stable/modules/generated/sklearn.linear_model.Lasso.html
• sklearn.decomposition.PCA http://scikit-
learn.org/stable/modules/generated/sklearn.decomposition.PCA.htm
l

30. Gensim
• https://radimrehurek.com/gensim/index.html

31. HoG (Histogram of Oriented Gradients)
• Python code example http://scikit-
image.org/docs/dev/auto_examples/plot_hog.html

32. An Introduction to Variable and Feature
Selection
• Author: Isabelle Guyon and Andre Elisseeff
• PDF download:
http://jmlr.csail.mit.edu/papers/volume3/guyon03a/guyon03a.pdf