This document describes a system for recognizing easily confused traditional Chinese medicine (TCM) herbs using convolutional neural networks (CNNs) on smartphones. The system was trained on a dataset of 2400 herb images from 24 categories collected using smartphone cameras. A hierarchical clustering CNN approach achieved higher accuracy than a standard CNN, with accuracy over 97% for most herb categories. Testing the system on different smartphone models found varying accuracy rates due to differences in camera quality. Data augmentation techniques like image rotation and brightness adjustment were also found to improve classification accuracy over using a single phone's dataset.

1. 基於CNN對易混淆中藥的手機辨識系統
Recognition of Easily-confused TCM Herbs
Using Convolutional Neural Network
On The Smartphone
Kun-chan Lan (藍崑展)
National Cheng Kung University
Joint work with Min-Chun Hu and
Juei-Chun Weng
1GTC Taiwan 2017

2. A little about me
• Background in sensor network (aka. IoT)
•2011: experienced TCM
•2013: started doing research on TCM
• smartphone APPs for TCM
• Tongue diagnosis (https://lens.csie.ncku.edu.tw/~john/)
• AR-based acupoint localization
(https://www.youtube.com/watch?time_continue=1&v=RyzKMuo3Gjo)
• TCM Herb recognition
•2015: studying TCM at China Medical University
(中國醫藥⼤學)
2
GTC Taiwan 2017

3. TCM 101
• Based on thousands of years of clinical experiences
• Data -> model (similar to DNN?)
• Treat by symptom 症(personalized treatment)
• Considering individual constitution and the interaction with the
environment
• Western Medicine : Treat by disease 病(same treatment for same disease)
• Four diagnoses (四診) : collect biometrics using sensors on the human body
• Inspection (望)
• Listen and smell (聞)
• Inquiry (問)
• Palpation (切)
3
GTC Taiwan 2017

4. Chinese Herbal Medicine
• Traditional Chinese medicine (TCM) originated in China and has
evolved over 5000 years. TCM is one of Complementary Medicines
(互補醫學) recognized by World Health Organization (WHO)
• Chinese Herbal Medicine (CHM) is one of the important therapies in
TCM (⼀針, ⼆灸, 三湯藥)
4
GTC Taiwan 2017

5. Easily-confused herbs
5
山藥 木薯
黃耆 紅耆
人參 西洋參
川木通 關木通
川母貝(松貝) 平母貝
黃芩 綠黃芩
GTC Taiwan 2017

6. 黃耆 vs. 紅耆
•Some TCM herbs have similar shape and
color but different utilities and cost.
6
GTC Taiwan 2017

7. Smartphone to the rescue?
Information
Illustrated handbooks Smartphones
7
GTC Taiwan 2017

8. Internet
A simple client-server framework
Pre-trained
Clustering
Model
Pre-trained
Classification
Model
CHM
Info.
Image
Preprocessing
Predict
Result
server
8
GTC Taiwan 2017

9. Prior work on TCM herb recognition
Tao et al. Liu et al. Sun et al. Ours
Category 18 8 95 24
Confused Herbs Pair 1 0 2 10
Method Hand-Crafted Method Hand-Crafted Method CNN Hierarchical Clustering
CNN
Implemented on
smartphone
No No No Yes
9
GTC Taiwan 2017

10. What we did ( a demo)
• 山藥 vs. 木
• 黃耆 vs. 紅耆
• GTC-demo_video.wmv
10
Test1 Test2 Test3 Test4 Test5 Avg.
Xiaomi 3.083 2.535 2.755 2.856 2.508 2.7474(s)
Asus 2.594 2.907 3.133 2.294 2.820 2.7496(s)
Smartphones
recognition time
GTC Taiwan 2017

11. Why Deep Learning?
• With traditional hand-crafted methods, It is not easy to find
representative features for easily-confused TCM herbs.
• Deep learning can automatically learn about the features.
Color?
Shape?
Texture?
11
GTC Taiwan 2017

12. CNN-CaffeNet
24
12
GTC Taiwan 2017

13. Dataset (中藥飲片)
• CHM dataset collected by iPhone6 camera.
• 2400 images of 24 CHMs
• 1440 images for training
• 960 images for testing
山藥(A1) 木薯(A2)
黃耆(B1) 紅耆(B2)
人參(C1) 西洋參(C2) 綠衣枳實(F1) 枳實(F2)
川木通(D1) 關木通(D2)
川母貝(松貝) (E1) 平母貝(E2)
川烏(G1) 草烏(G2)
黃芩(H1) 綠黃芩(H2)
半夏(I1) 水半夏(I2)
石蓮子(J1) 苦石蓮(J2)
川牛膝(K1) 味牛膝(K2)
北板藍根(L1) 南板藍根(L2)
13
GTC Taiwan 2017

14. Experimental Environment
• INTEL i7-4790 CPU & 16GB RAM
• NVIDIA GTX 1060
• Python
• Caffe
14
GTC Taiwan 2017

15. Result
Naïve CNN Method
15
Training Phase
Testing Phase
Input Image
Pre-trained
CNN Model
CNN
Model
Feature Extraction
Training Images
…
GTC Taiwan 2017

16. 16
80%
84%
88%
92%
96%
100%
A1 A2 B1 B2 C1 C2 D1 D2 E1 E2 F1 F2 G1 G2 H1 H2 I1 I2 J1 J2 K1 K2 L1 L2 Avg
CNN HCNN by AP algorithm (Average) HCNN by illustrated handbook
poor results for some herbs (green bars)
GTC Taiwan 2017

17. Hierarchical Clustering CNN
17
GTC Taiwan 2017

18. Result
Hierarchical Clustering CNN Method
18
Training Phase
Testing Phase
Input Image
Second-layer
CNN-based
Classification-1 ModelFirst-layer
CNN-based
Clustering
Model
First-layer
Pre-trained
Clustering
Model
Second-layer
Pre-trained
Classification Model
Training Images
…
…
CNN-based
Classification-n Model
Second-layer
If there are more
than one category
in the group
Data clustering
GTC Taiwan 2017

19. Clustering: Affinity Propagation
Training Images
…
Affinity Propagation
algorithm
Feature
Extraction
Each kind of herbs
randomly samples
images.
Each kind of herbs decides
an final exemplar.
If the exemplar of two kind
of herbs are the same, we
cluster two herbs into a
group.
1
2
3
19
. "Clustering by passing messages between data points".
Science. 315 (5814): GTC Taiwan 2017

20. current results
20GTC Taiwan 2017

21. Usefulness of CNN?
• Hand-Crafted Method
• SIFT
• HOG
• LBP
• SVM(classifier)
CNN method
• CaffeNet model
(5 conv layers)
• VGG16 model
(13 conv layers)
Test time(s)
1.93684
5.95769
Using five-fold cross validation to calculate accuracy
21
Method Accuracy
LBP+SVM 86.85%
HOG+SVM 75.31%
SIFT+SVM 70.83%
Method Accuracy
CNN[CaffeNet] 95.69%
CNN[VGG16] 95.63%
GTC Taiwan 2017

22. Effect of Fine-tune
• Fine-tune by pre-trained CaffeNet model (based on ImageNet
data)
22
0.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
70.0%
80.0%
90.0%
100.0%
0 1 2 4 6 8 10 20 30 40 50 60 70 80 90 100 110 120 130 140 150 160 170 180 190 200 210 220 230 240 250
Accuracy
Iterations
Fine-tune Re-train
GTC Taiwan 2017

23. CNN vs. HCNN
80%
84%
88%
92%
96%
100%
A1 A2 B1 B2 C1 C2 D1 D2 E1 E2 F1 F2 G1 G2 H1 H2 I1 I2 J1 J2 K1 K2 L1 L2 Avg
CNN HCNN by AP algorithm (Average) HCNN by illustrated handbook
Using five-fold cross validation to calculate accuracy
95.63%
97.54%
97.85%
88.80%
93.55%
94.5%
80%
84%
88%
92%
96%
100%
A1 A2 B1 B2 C1 C2 D1 D2 E1 E2 F1 F2 G1 G2 H1 H2 I1 I2 J1 J2 K1 K2 L1 L2 Avg
CNN HCNN by AP algorithm (Average) HCNN by illustrated handbook
1 2 3 4 5 6 7 8 9 10
97.65% 97.48% 97.85% 97.08% 97.85% 97.88% 97.48% 97.23% 97.65% 97.23% 23
GTC Taiwan 2017

24. Effect of Smartphones?
iPhone Xiaomi Samsung Asus
24
GTC Taiwan 2017

25. Effect of Smartphones
A1 A2 B1 B2 C1 C2 D1 D2 E1 E2 F1 F2 G1 G2 H1 H2 I1 I2 J1 J2 K1 K2 L1 L2 Avg.
iPhone 100.00% 95.00% 85.00% 80.00% 92.50% 77.50% 95.00% 97.50% 97.50% 97.50% 90.00% 95.00% 92.50% 97.50% 92.50% 85.00% 80.00% 92.50% 100.00% 100.00% 100.00% 92.50% 87.50% 100.00% 92.60%
Xiaomi 100.00% 75.00% 95.00% 70.00% 62.50% 60.00% 82.50% 100.00% 87.50% 62.50% 85.00% 77.50% 100.00% 70.00% 65.00% 87.50% 100.00% 60.00% 90.00% 95.00% 87.50% 92.50% 92.50% 90.00% 82.81%
Samsung 100.00% 95.00% 85.00% 72.50% 100.00% 60.00% 100.00% 100.00% 67.50% 97.50% 100.00% 100.00% 95.00% 97.50% 100.00% 47.50% 97.50% 80.00% 100.00% 100.00% 100.00% 100.00% 87.50% 80.00% 90.10%
Asus 100.00% 70.00% 90.00% 62.50% 70.00% 35.00% 87.50% 100.00% 100.00% 85.00% 97.50% 55.00% 100.00% 87.50% 50.00% 95.00% 82.50% 67.50% 77.50% 100.00% 90.00% 90.00% 87.50% 82.50% 81.77%
Avg 100.00% 83.75% 88.75% 71.25% 81.25% 58.13% 91.25% 99.38% 88.13% 85.63% 93.13% 81.88% 96.88% 88.13% 76.88% 78.75% 90.00% 75.00% 91.88% 98.75% 94.38% 93.75% 88.75% 88.13% 86.82%
25
iPhone iPhone
Xiaomi
Samsung
Asus
value value
The number of pixels
The number of pixels
GTC Taiwan 2017

26. Not enough data => data augmentation?
Zoom In
Zoom OutClockwise Rotation
Counter-clockwise Rotation Darken
Brighten
26
GTC Taiwan 2017

27. Data Augmentation
(1). iPhone camera (The original training data)
iPhone
1440 images
(2). iPhone camera + AUG*2(Rotation)
(3). iPhone camera + AUG*4(Rotation+Size)
(4). iPhone camera + AUG*6(Rotation+Size+Brightness)
iPhone
1440 images
iPhone
10080 images
iPhone
7200 images
iPhone
4320 images
(6). 4 smartphones camera + AUG*6
iPhone
1440 images
Xiaomi
1440 images
Samsung
1440 images
ASUS
1440 images
iPhone
10080 images
Xiaomi
10080 images
Samsung
10080 images
ASUS
10080 images
(5). 4 smartphones camera
iPhone
1440 images
Xiaomi
1440 images
Samsung
1440 images
ASUS
1440 images
Model trained by 6 different training data
27
GTC Taiwan 2017

28. data augmentation vs. adding more phone data
70.00%
75.00%
80.00%
85.00%
90.00%
95.00%
100.00%
iPhone Xiaomi Samsung Asus Average
1 2 3 4 5 6
Training Data
1. iPhone camera
2. iPhone camera + AUG*2(Rotation)
3. iPhone camera + AUG*4(Rotation + Size)
4. iPhone camera + AUG*6(Rotation + Size + Brightness)
5. 4 smartphones camera
6. 4 smartphones camera + AUG*6
iPhone Xiaomi Samsung Asus Average
1 92.60% 82.81% 90.10% 81.77% 86.82%
2 92.81% 84.48% 88.13% 84.27% 87.42%
3 94.27% 85.21% 88.96% 84.58% 88.26%
4* 94.48% 88.96% 91.02% 90.52% 91.24%
5* 94.06% 93.02% 95.31% 93.85% 94.06%
6 96.04% 95.83% 96.25% 94.90% 95.76%
28
GTC Taiwan 2017

29. Conclusions
• Automatic recognition of 24 easily-confused CHMs on the smartphone.
• Compared to traditional hand-crafted method, CNN works better!
• We propose a hierarchical CNN method which automatically clusters the
herbs using AP algorithm. This brings an accuracy improvement up to 5%
for some TCM herbs
• Differences between phones need to be considered when designing image
recognition Apps on the phone
29
GTC Taiwan 2017

30. Future work
• Short term
• Collect data for all 300+ TCM
herbs
• Try with more different phones
under different lighting conditions
• Long term
• A TCM robot assistant
30
GTC Taiwan 2017