Honeydew is an AI assistant that uses an SVM model with three feature extraction methods to predict meeting times from emails. An evaluation of 171 emails found the model predicted times correctly except for a few cases where there was insufficient training data. A separate test of 30 emails found Honeydew predicted times with 100% accuracy, outperforming other calendar assistants, demonstrating the effectiveness of its time prediction module. The main challenge is accurately extracting time phrases from emails.

1. Honeydew Time Prediction Evaluation
1. Introduction
Honeydew time prediction module is using SVM as the training algorithm, http://www.csie.ntu.edu.tw/~cjlin/libsvm/
Here are the trainer parameters.
C 1.0 cost, the parameter C of C-SVC, epsilon-SVR, and
nu-SVR
cache_size 40.0 cache memory size in MB (default 100)
coef0 0.0 coef0 in kernel function
degree 3 degree in kernel function
eps 0.0010 tolerance of termination criterion (default 0.001)
gamma 0.0 gamma in kernel function
kernel_type 0 type of kernel function, linear: u'*v
nr_weight 0 the parameter C of class i to weight*C, for C-SVC
nu 0.5 the parameter nu of nu-SVC, one-class SVM, and
nu-SVR (default 0.5)
p 0.1 the epsilon in loss function of epsilon-SVR
probability 0 whether to train a SVC or SVR model for probability
estimates, 0 or 1 (default 0)
shrinking 1 whether to use the shrinking heuristics, 0 or 1
(default 1)
svm_type 0 C-SVC

2. Honeydew Evaluation has two steps. One is Honeydew Time Module Accuracy. The other is overall honeydew performance among three systems.
In evaluation 1, we only consider the accuracy in Honeydew Time Module. We are using 171 emails for leave-one-out cross validation for three different feature
selection methods.
In evaluation 2, we use 30 new emails (not the 171 training examples for honeydew time prediction module) to do the performance comparison.

3. 2. Evaluation 1, Time Prediction Module in Honeydew
There are three methods for feature extraction in Honeydew Time Prediction Module.
1. Bag of Words (Token)
2. Token with position information
3. Token and Token with position information (1&2)
These are leave-one-out cross validation result of Honeydew Time Prediction
The total number of examples is 171. We got 13 errors in the original Only Token, 5 errors in Only TokenPos and Token&TokenPos
Incorrect example table
Only Hour Minute Slot Only TokenPos Hour Minute Slot Token & TokenPos Hour Minute Slot
Token (13 errors) (5 errors) (5 errors)
10am-11:30am Ten Thirty AM
8:30 - 11:00 Eleven Thirty AM
11:30am-12:30pm null Thirty AM
4-5 null Zero PM
9-12noon Twelve Zero AM
01:30 p.m. to null Thirty PM 01:30 p.m. to null Thirty PM 01:30 p.m. to 02:30 null Thirty PM
02:30 p.m. 02:30 p.m. p.m.
3:30 - 5 Three Zero PM
10:30-11:15 Ten Fifteen AM

4. 9-12 Twelve Zero AM
3:15 PM Three null PM 3:15 PM Three Zero PM 3:15 PM Three null PM
6 pm null Zero PM 6 pm null Zero PM 6 pm null Zero PM
5 pm null Zero PM
5pm null Zero PM
noon to 4pm null Zero PM noon to 4pm null Zero PM
Noon null Zero PM Noon null Zero PM
There are still five incorrect examples in method 2 and 3:
01:30 p.m. to 02:30 p.m. can’t decide hour
All of the Time Interval phrases are correct except the “01:30 p.m. to 02:30 p.m.” This is because 01 is different from 1, which needs more training examples
(appeared only once).
It shows that the position information is very useful for Time Interval Phrase.
3:15 PM, can’t decide minute
In this cross validation, we got two training examples that have “15”.
10:30-11:15
1:15 pm - 2:00 pm
But we still don’t know why 3:15PM can’t get correct minute because the other two got correct prediction.
6 pm, can’t decide hour
” 6 “appears only once in the 171 training data. So that is also because of lack of training data.

5. The following are all “noon” phrases in 171 training data. The three columns show all the errors that related two “noon” phrase in three different methods.
Only Token Only TokenPos Token & TokenPos
noon to 4pm noon to 4pm //can’t decide hour noon to 4pm // can’t decide hour
Noon Noon //can’t decide hour Noon // can’t decide hour
noon - 1 pm noon - 1 pm noon - 1 pm
noon - 1 p noon - 1 p noon - 1 p
9-12noon //Error because predict noon not 9 9-12noon 9-12noon
9:00 a.m. - noon 9:00 a.m. - noon 9:00 a.m. - noon
3. Evaluation 2, Honeydew Performance with VIO and other approaches
Comparison of Honeydew/Tea/Gmail “Calendar”
Result
Software Date Correct Time Correct Both
Honeydew 8(vio got 8) 13(vio got 13) 7
Tea 4 7 3
Gmail “Calendar” 5(allow 8) 4(allow 8) 3
Tea choose the earliest time

6. From the result, we can see Honeydew Time Prediction Module got 100% accuracy. Although the test data is not big but it still demonstrated the performance of
honeydew time prediction module. Besides that, VIO is the major reason that reduces honeydew’s accuracy. That means for email meeting information extraction,
the most difficult problem is how to extract time phrase. After we got time phrase, the prediction is not a hard problem.
The other two systems, TEA and Gmail “Add to Calendar” have worse performance than honeydew.