This document discusses recognizing happiness from mobile phone data. Researchers collected call logs, text messages, and Bluetooth data from 117 subjects over 10 months. They extracted features related to phone usage, diversity of contacts, activity levels, and regularity. The top features were personality traits, weather data, and Bluetooth proximity. A random forest classifier recognized daily happiness with 80% accuracy based on these features, approaching methods using multiple data modes. Future work should focus on multi-step model development.

1. Happiness Recognition
from Mobile Phone Data
Andrey Bogomolov1 Bruno Lepri2 Fabio Pianesi2
1University of Trento,
Via Sommarive, 5
I-38123 Povo - Trento, Italy
2Fondazione Bruno Kessler
Via Sommarive, 18
I-38050 Povo - Trento, Italy
EmoPAR group meeting 2013-JUN-19, Trento, Italy.
1 / 49

2. Outline
Introduction
Problem Statement
Source Data
Recognition Model
Basic Features
Final Feature Space
Results
Limitations
Summary
2 / 49

3. Happiness as an emotional state – why is it important?
Your ideas?. . .
3 / 49

4. General Problem Statement
Inputs
Pervasive technology data.
Multimodal data.
Outputs
Emotion recognition.
Mood recognition.
Personality recognition.
4 / 49

5. Our Problem Statement
Inputs
Smartphone call log.
Smartphone sms log.
Smartphone Bluetooth proximity hits.
Outputs
Daily happiness recognition.
3-classes: {not happy, neutral, happy}.
5 / 49

6. Data Collection
117 subjects
dates: 21 February, 2010 – 16 July, 2011
Source Space Dataset: Living Laboratory
phone calls 33497
sms 22587
Bluetooth hits 1460939
6 / 49

7. Happiness Data
Recorded Happiness Scores Density
0.0
0.2
0.4
0.6
0.8
0 2 4 6
Score
Density
7 / 49

8. Happiness Data
Descriptive Statistics
number of records 12991
mean 4.84
standard deviation 1.26
median 5.00
mean average deviation 1.48
min 1.00
max 7.00
range 6.00
skew -0.39
kurtosis -0.07
8 / 49

9. Happiness Data
Within-person and between-subject variance
0 1 2 3 4 5 6
0.00.20.40.60.81.01.21.4
density.default(x = r1[, 1])
Variance
Density
Within−Person Variance
Between−Person Variance
9 / 49

10. Feature Space
Basic Features
General Phone Usage
Diversity
Active Behaviors
Regularity
10 / 49

11. Feature Space
Basic Features: General Phone Usage
1. Total Number of Calls (Outgoing+Incoming)
2. Total Number of Incoming Calls
3. Total Number of Outgoing Calls
4. Total Number of Missed Calls
5. Number of SMS received
6. Number of SMS sent
11 / 49

12. Feature Space
Basic Features: Diversity
7. Number of Unique Contacts Called
8. Number of Unique Contacts who Called
9. Number of Unique Contacts Communicated with (Incoming+Outgoing)
10. Number of Unique Contacts Associated with Missed Calls
11. Entropy of Call Contacts
12. Call Contacts to Interactions Ratio
13. Number of Unique Contacts SMS received from
14. Number of Unique Contacts SMS sent to
15. Entropy of SMS Contacts
16. Sms Contacts to Interactions Ratio
12 / 49

13. Feature Space
Basic Features: Active Behaviors
17. Percent Call During the Night
18. Percent Call Initiated
19. Sms response rate
20. Sms response latency
21. Percent SMS Initiated
13 / 49

14. Feature Space
Basic Features: Regularity
22. Average Inter-event Time for Calls (time elapsed between two events)
23. Average Inter-event Time for SMS (time elapsed between two events)
24. Variance Inter-event Time for Calls (time elapsed between two events)
25. Variance Inter-event Time for SMS (time elapsed between two events)
14 / 49

15. Feature Space
Proximity Features
General Bluetooth Proximity
1. Number of Bluetooth IDs
2. Times most common Bluetooth ID is seen
3. Bluetooth IDs accounting for n% of IDs seen
4. Bluetooth IDs seen for more than k time slots
5. Time interval for which a Bluetooth ID is seen
6. Entropy of Bluetooth contacts
Diversity
7. Contacts to interactions ratio
Regularity
8. Average Bluetooth interactions inter-event time
(time elapsed between two events)
9. Variance of the Bluetooth interactions inter-event time
(time elapsed between two events)
15 / 49

16. Feature Space Innovation
“Background Noise” Features
a) peoples activity, as detected through their smartphones
b) the weather conditions {humidity, wind speed, pressure,
total precipitation and visibility}
c) personality traits {“Big Five"}
Functional Innovation
a) time domain – sliding window functions
b) Miller-Madow correction for entropy calculation
ˆHMM(θ) ≡ −
p
i=1
θML,i log θML,i +
ˆm − 1
2N
16 / 49

17. Top-30 Features
-1 0 1 MeanDecreaseAccuracy MeanDecreaseGini
meanTemperature 0.0099 0.0033 0.0094 0.0082 154.8379
humidity 0.0047 -0.0033 0.0115 0.0074 149.7668
pressure -0.0002 0.0008 0.0015 0.0011 149.0302
windSpeed 0.0028 0.0017 0.0051 0.0040 142.7727
visibility 0.0024 -0.0009 0.0051 0.0034 120.8683
neuroticism 0.0690 0.0288 0.0399 0.0419 90.5721
conscientiousness 0.0659 0.0480 0.0668 0.0627 90.2708
extraversion 0.0511 0.0357 0.0472 0.0454 76.9467
openness 0.0656 0.0340 0.0406 0.0429 73.9181
totalPrecipitation 0.0007 -0.0000 0.0012 0.0009 73.5273
agreeableness 0.0536 0.0282 0.0235 0.0289 70.7261
bluetoothQ95TimeForWhichIdSeen 0.0233 0.0120 0.0149 0.0155 22.4018
bluetoothQ90TimeForWhichIdSeen 0.0161 0.0082 0.0141 0.0131 19.8364
smsRepliedEventsLatencyMedian 0.0093 0.0085 0.0096 0.0093 18.7719
bluetoothIdsMoreThan04TimeSlotsSeen 0.0114 0.0066 0.0124 0.0110 15.6738
bluetoothMaxTimeForWhichIdSeen 0.0141 0.0072 0.0095 0.0097 15.2546
bluetoothTotalEntropyMillerMadow 0.0058 0.0017 0.0035 0.0035 13.9000
bluetoothTotalEntropyShannon 0.0065 0.0009 0.0060 0.0050 13.1826
callMeanInterEventTimePerDay -0.0003 -0.0000 0.0014 0.0009 13.1180
incomingAndOutgoingCallsPerDay -0.0000 -0.0002 0.0024 0.0015 12.3962
bluetoothQ50TimeForWhichIdSeen 0.0104 0.0104 0.0091 0.0095 12.2270
callStandardDeviationInterEventTimePerDay 0.0004 -0.0005 0.0007 0.0004 10.1723
bluetoothIdsMoreThan19TimeSlotsSeen 0.0087 0.0033 0.0089 0.0077 9.7388
incomingCallsPerDay 0.0003 -0.0005 0.0009 0.0005 9.6572
outgoingContactsToInteractionsRatioPerDay 0.0005 -0.0004 0.0013 0.0009 9.2016
callsInitiatedRatioPerDay -0.0001 -0.0001 0.0014 0.0008 9.0245
entropyMillerMadowCallsOutgoingWindow3Days 0.0001 -0.0008 0.0014 0.0008 8.7199
bluetoothIdsMoreThan09TimeSlotsSeen 0.0074 0.0046 0.0040 0.0046 8.6006
bluetoothQ75TimeForWhichIdSeen 0.0027 0.0018 0.0032 0.0028 8.4454
outgoingCallsPerDay -0.0001 -0.0001 0.0012 0.0007 8.3368
17 / 49

18. Top-20 Features Correlation
agreeableness
bluetoothIdsMoreThan04TimeSlotsSeen
bluetoothMaxTimeForWhichIdSeen
bluetoothQ90TimeForWhichIdSeen
bluetoothQ95TimeForWhichIdSeen
bluetoothTotalEntropyMillerMadow
bluetoothTotalEntropyShannon
callMeanInterEventTimePerDay
conscientiousness
extraversion
humidity
incomingAndOutgoingCallsPerDay
meanTemperature
neuroticism
openness
pressure
smsRepliedEventsLatencyMedian
totalPrecipitation
visibility
windSpeed
agreeablenessbluetoothIdsMoreThan04TimeSlotsSeenbluetoothMaxTimeForWhichIdSeenbluetoothQ90TimeForWhichIdSeenbluetoothQ95TimeForWhichIdSeenbluetoothTotalEntropyMillerMadowbluetoothTotalEntropyShannoncallMeanInterEventTimePerDayconscientiousnessextraversionhumidityincomingAndOutgoingCallsPerDaymeanTemperatureneuroticismopennesspressuresmsRepliedEventsLatencyMediantotalPrecipitationvisibilitywindSpeed
Var1
Var2
0.0
0.5
1.0
value
18 / 49

19. Results
Final Classifier Performance Metrics Comparison
Training set Test set
Accuracy 0.8081 0.8036
Kappa 0.5879 0.5743
AccuracyLower 0.8004 0.7878
AccuracyUpper 0.8156 0.8187
AccuracyNull 0.6415 0.6419
AccuracyPValue 2.139e-303 8.826e-73
McnemarPValue 5.647e-208 1.738e-57
19 / 49

20. Results
Final Classifier Confusion Matrix for Training Set
-1 0 1
-1 782 119 75
0 153 1170 145
1 600 903 6448
Final Classifier Confusion Matrix for Test Set
-1 0 1
-1 197 30 14
0 34 274 37
1 152 243 1616
20 / 49

21. Results
What We Learnt: Final Model ROC curve
Specificity
Sensitivity
0.00.20.40.60.81.0
1.0 0.8 0.6 0.4 0.2 0.0
AUC: 0.844
21 / 49

22. Limitations
Data
Data loss not registered as NA’s
Battery
Temporal resolution
Model
Requires personality data
Requires 1 week data collection period
Not tested on diverse cultural groups
22 / 49

23. Summary
Automatic recognition of people’s daily happiness from
mobile phone data is feasible.
Accuracy is approaching the results of multi-modal
obtrusive methods.
Future work should be focused on multi-step recognition
model development.
23 / 49

24. Thank you!
{andrey.bogomolov@unitn.it}
24 / 49

25. References I
Aharony, Nadav, Wei Pan, Cory Ip, Inas Khayal, and
Alex Pentland (2011). “Social fMRI: Investigating and shaping
social mechanisms in the real world”. In: Pervasive and Mobile
Computing 7.6, pp. 643–659.
Anguita, Davide, Alessandro Ghio, Luca Oneto, Xavier Parra,
and Jorge L. Reyes-Ortiz (2012). “Human Activity Recognition
on Smartphones Using a Multiclass Hardware-Friendly Support
Vector Machine”. In: IWAAL, pp. 216–223.
Bickmore, Timothy W., Rosalind, and W. Picard (2005).
“Establishing and Maintaining Long-Term Human-Computer
Relationships”. In: ACM Transactions on Computer Human
Interaction 12, pp. 293–327.
Bishop, C. M. (1995). “Training with Noise is Equivalent to
Tikhonov Regularization”. In: Neural Computation 7.1,
pp. 108–116.
25 / 49

26. References II
Both, Fiemke, Mark Hoogendoorn, Michel C. A. Klein, and
Jan Treur (2010). “Computational modeling and analysis of the
role of physical activity in mood regulation and depression”. In:
Proceedings of the 17th international conference on Neural
information processing: theory and algorithms - Volume Part I.
ICONIP’10. Sydney, Australia: Springer-Verlag, pp. 270–281.
Brave, Scott and Clifford Nass (2003). “The human-computer
interaction handbook”. In: ed. by Julie A. Jacko and
Andrew Sears. Hillsdale, NJ, USA: L. Erlbaum Associates Inc.
Chap. Emotion in human-computer interaction, pp. 81–96.
Breiman, Leo (Oct. 2001). “Random Forests”. In: Mach. Learn.
45.1, pp. 5–32.
Broek, Egon L. (Jan. 2013). “Ubiquitous emotion-aware
computing”. In: Personal Ubiquitous Comput. 17.1, pp. 53–67.
Buhmann, Martin D. and M. D. Buhmann (2003). Radial Basis
Functions. New York, NY, USA: Cambridge University Press.
26 / 49

27. References III
Busso, Carlos, Zhigang Deng, Serdar Yildirim, Murtaza Bulut,
Chul Min Lee, Abe Kazemzadeh, Sungbok Lee,
Ulrich Neumann, and Shrikanth Narayanan (2004). “Analysis of
emotion recognition using facial expressions, speech and
multimodal information”. In: in Sixth International Conference
on Multimodal Interfaces ICMI 2004. ACM Press, pp. 205–211.
Campbell, Angus, Philip E. Converse, and Willard L Rodgers
(1976). The quality of American life : perceptions, evaluations,
and satisfactions / Angus Campbell, Philip E. Converse, Willard
L. Rodgers. English. Russell Sage Foundation New York, xi,
583 p. :
Carlson, Charles R., Frank P. Gantz, and John C. Masters
(1983). “Adults’ emotional states and recognition of emotion in
young children”. English. In: Motivation and Emotion 7.1,
pp. 81–101.
27 / 49

28. References IV
Carver, Charles S. and Michael F. Scheier (Jan. 1990). “Origins
and Functions of Positive and Negative Affect: A
Control-Process View”. In: Psychological Review 97.1,
pp. 19–35.
Chittaranjan, Gokul, Jan Blom, and Daniel Gatica-Perez (Mar.
2013). “Mining large-scale smartphone data for personality
studies”. In: Personal Ubiquitous Comput. 17.3, pp. 433–450.
Cohen, Jacob (Apr. 1960). “A Coefficient of Agreement for
Nominal Scales”. In: Educational and Psychological
Measurement 20.1, pp. 37–46.
Cohn, Jeffrey F. (2007). “Foundations of human computing:
facial expression and emotion”. In: Proceedings of the ICMI
2006 and IJCAI 2007 international conference on Artifical
intelligence for human computing. ICMI’06/IJCAI’07. Banff,
Canada: Springer-Verlag, pp. 1–16.
28 / 49

29. References V
Costa, P T and R R McCrae (1992). Revised NEO personality
inventory (NEO PI-R) and neo five-factor inventory (NEO-FFI).
Vol. 0. October. Psychological Assessment Resources,
pp. 17–20.
— (1995). “Domains and facets: hierarchical personality
assessment using the revised NEO personality inventory.” In:
Journal of Personality Assessment 64.1, pp. 21–50.
Costa, Paul T. and Robert R. McCrae (1980). “Influence of
extraversion and neuroticism on subjective well-being: Happy
and unhappy people”. In: Journal of Personality and Social
Psychology 38 (4), pp. 668–678.
Cowie, R., E. Douglas-Cowie, K. Karpouzis, G. Caridakis,
M. Wallace, and S. Kollias (2008). “Recognition of Emotional
States in Natural Human-Computer Interaction”. English. In:
Multimodal User Interfaces. Ed. by Dimitrios Tzovaras. Signals
and Commmunication Technologies. Springer Berlin
Heidelberg, pp. 119–153.
29 / 49

30. References VI
Cowie, Roddy, Ellen Douglas-Cowie, Nicolas Tsapatsoulis,
George Votsis, Stefanos Kollias, Winfried Fellenz, and
John G Taylor (2001). “Emotion recognition in human-computer
interaction”. In: Signal Processing Magazine, IEEE 18.1,
pp. 32–80.
DeNeve, K.M. and H. Cooper (1998). “The happy personality: a
meta-analysis of 137 personality traits and subjective
well-being.” In: Psychol Bull 124.2, pp. 197–229.
Denissen, Jaap J. A., Ligaya Butalid, Lars Penke, and Marcel
A. G. Van Aken (2008). “The effects of weather on daily mood:
A multilevel approach.” In: Emotion Researcher 8.5,
pp. 662–667.
Diener, Ed (1994). “Assessing subjective well-being: Progress
and opportunities”. English. In: Social Indicators Research
31.2, pp. 103–157.
Diener, Ed and Martin E.P. Seligman (2002). “Very Happy
People”. In: Psychological Science 13.1, pp. 81–84.
30 / 49

31. References VII
Dolan, R. J. (Nov. 2002). “Emotion, cognition, and behavior.”
In: Science (New York, N.Y.) 298.5596.
Dong, Wen, Bruno Lepri, and Alex Pentland (2011). “Modeling
the co-evolution of behaviors and social relationships using
mobile phone data”. In: MUM, pp. 134–143.
Eagle, Nathan and Alex Pentland (2006). “Reality mining:
sensing complex social systems”. In: Personal and Ubiquitous
Computing 10.4, pp. 255–268.
Eagle, Nathan, Alex (Sandy) Pentland, and David Lazer (2009).
“Inferring friendship network structure by using mobile phone
data”. In: Proceedings of the National Academy of Sciences
106.36, pp. 15274–15278.
Feldman, Lisa A (1995). “Valence focus and arousal focus:
Individual differences in the structure of affective experience”.
In: Journal of personality and social psychology 69,
pp. 153–153.
31 / 49

32. References VIII
FUNF Open Sensing Framework. (Access date: 12-JUN-2013).
http://www.funf.org.
Garcia, Salvador and Francisco Herrera (Dec. 2008). “An
Extension on "Statistical Comparisons of Classifiers over
Multiple Data Sets" for all Pairwise Comparisons”. In: Journal
of Machine Learning Research 9, pp. 2677–2694.
Gray, J. A. (1991). “Neural systems, emotion, and personality.”
English. In: Neurobiology of learning, emotion, and affect /
editor, John Madden IV. Raven Press New York, pp. 273–306.
Greenberg, Ariel M., William G. Kennedy, and Nathan Bos,
eds. (2013). Social Computing, Behavioral-Cultural Modeling
and Prediction - 6th International Conference, SBP 2013,
Washington, DC, USA, April 2-5, 2013. Proceedings.
Vol. 7812. Lecture Notes in Computer Science. Springer.
Gunes, H. and M. Pantic (2010). “Automatic, Dimensional and
Continuous Emotion Recognition”. In: Int’l Journal of Synthetic
Emotion 1.1, pp. 68–99.
32 / 49

33. References IX
Gunes, H., B. Schuller, M. Pantic, and R. Cowie (2011).
“Emotion Representation, Analysis and Synthesis in
Continuous Space: A Survey”. In: Proceedings of IEEE
International Conference on Automatic Face and Gesture
Recognition (FG’11), EmoSPACE 2011 - 1st International
Workshop on Emotion Synthesis, rePresentation, and Analysis
in Continuous spacE. Santa Barbara, CA, USA, pp. 827–834.
Hand, DavidJ. and RobertJ. Till (2001). “A Simple
Generalisation of the Area Under the ROC Curve for Multiple
Class Classification Problems”. English. In: Machine Learning
45 (2), pp. 171–186.
Hernandez, Javier, Mohammed E. Hoque, and
Rosalind W. Picard (2012). “Mood Meter: large-scale and
long-term smile monitoring system”. In: ACM SIGGRAPH 2012
Emerging Technologies. SIGGRAPH ’12. Los Angeles,
California: ACM, 15:1–15:1.
33 / 49

34. References X
John, Oliver P. and Sanjay Srivastava (1999). “The Big Five trait
taxonomy: History, measurement, and theoretical
perspectives”. In: Handbook of Personality: Theory and
Research. Ed. by Lawrence A. Pervin and Oliver P. John.
Second. New York: Guilford Press, pp. 102–138.
Kapoor, Ashish, Winslow Burleson, and Rosalind W. Picard
(Aug. 2007). “Automatic prediction of frustration”. In: Int. J.
Hum.-Comput. Stud. 65.8, pp. 724–736.
Kapoor, Ashish and Rosalind W. Picard (2005). “Multimodal
affect recognition in learning environments”. In: Proceedings of
the 13th annual ACM international conference on Multimedia.
MULTIMEDIA ’05. Hilton, Singapore: ACM, pp. 677–682.
Keller, Matthew C., Barbara L. Fredrickson, Oscar Ybarra,
Stephane Cote, Kareem Johnson, Joe Mikels, Anne Conway,
and Tor Wager (Sept. 2005). “A Warm Heart and a Clear Head:
The Contingent Effects of Weather on Mood and Cognition”.
In: Psychological Science 16.9, pp. 724–731.
34 / 49

35. References XI
Krzanowski, Wojtek J. (2005). “Misclassification Rates”. In:
Encyclopedia of Statistics in Behavioral Science.
Lane, Nicholas, Mashfiqui Mohammod, Mu Lin, Xiaochao Yang,
Hong Lu, Shahid Ali, Afsaneh Doryab, Ethan Berke,
Tanzeem Choudhury, and Andrew Campbell (Apr. 2012).
“BeWell: A Smartphone Application to Monitor, Model and
Promote Wellbeing”. In: IEEE.
Lane, Nicholas D., Emiliano Miluzzo, Hong Lu, Daniel Peebles,
Tanzeem Choudhury, and Andrew T. Campbell (Sept. 2010). “A
survey of mobile phone sensing”. In: Comm. Mag. 48.9,
pp. 140–150.
Lee, Gary R. and Masako Ishii-Kuntz (Dec. 1987). “Social
Interaction, Loneliness, and Emotional Well-Being among the
Elderly”. In: Research on Aging 9.4, pp. 459–482.
35 / 49

36. References XII
Lester, Jonathan, Tanzeem Choudhury, and Gaetano Borriello
(2006). “A Practical Approach to Recognizing Physical
Activities”. In: Pervasive Computing. Ed. by KennethP. Fishkin,
Bernt Schiele, Paddy Nixon, and Aaron Quigley. Vol. 3968.
Lecture Notes in Computer Science. Springer Berlin
Heidelberg, pp. 1–16.
LEWIS, M.E. and J.M.E. HAVILAND-JONES (2004). Handbook
of Emotions. Guilford Publications, Incorporated.
Lin, Wei-Jiun and James J. Chen (Jan. 2013).
“Class-imbalanced classifiers for high-dimensional data”. In:
Briefings in Bioinformatics 14.1, pp. 13–26.
Lyubomirsky, Sonja, Laura King, and Ed Diener (Nov. 2005).
“The Benefits of Frequent Positive Affect: Does Happiness Lead
to Success?” In: Psychological Bulletin 131.6, pp. 803–855.
36 / 49

37. References XIII
Lyubomirsky, Sonja, Chris Tkach, and Robin M. Dimatteo
(2006). “WHAT ARE THE DIFFERENCES BETWEEN
HAPPINESS AND SELF-ESTEEM?” In: Social Indicators
Research 78, pp. 363–404.
Ma, Yuanchao, Bin Xu, Yin Bai, Guodong Sun, and Run Zhu
(2012). “Daily Mood Assessment Based on Mobile Phone
Sensing”. In: Wearable and Implantable Body Sensor Networks
(BSN), 2012 Ninth International Conference on, pp. 142–147.
Madan, Anmol, Manuel Cebrián, David Lazer, and
Alex Pentland (2010). “Social sensing for epidemiological
behavior change”. In: UbiComp, pp. 291–300.
McCrae, Robert R. and Paul T. Costa (1991). “Adding Liebe
und Arbeit: The Full Five-Factor Model and Well-Being”. In:
Personality and Social Psychology Bulletin 17.2, pp. 227–232.
Miller, G. (1955). “Note on the bias of information estimates”.
In: Information theory in psychology: Problems and methods
2.95, p. 100.
37 / 49

38. References XIV
Miller, Geoffrey (2012). “The Smartphone Psychology
Manifesto”. In: Perspectives on Psychological Science 7.3,
pp. 221–237.
Montjoye, Yves-Alexandre, Jordi Quoidbach, Florent Robic, and
Alex(Sandy) Pentland (2013a). “Predicting Personality Using
Novel Mobile Phone-Based Metrics”. In: Social Computing,
Behavioral-Cultural Modeling and Prediction. Ed. by
ArielM. Greenberg, WilliamG. Kennedy, and NathanD. Bos.
Vol. 7812. Lecture Notes in Computer Science. Springer Berlin
Heidelberg, pp. 48–55.
Montjoye, Yves-Alexandre de, Jordi Quoidbach, Florent Robic,
and Alex Pentland (2013b). “Predicting Personality Using Novel
Mobile Phone-Based Metrics”. In: SBP. Ed. by
Ariel M. Greenberg, William G. Kennedy, and Nathan Bos.
Vol. 7812. Lecture Notes in Computer Science. Springer,
pp. 48–55.
38 / 49

39. References XV
Moturu, Sai T., Inas Khayal, Nadav Aharony, Wei Pan, and
Alex Pentland (2011). “Using Social Sensing to Understand the
Links between Sleep, Mood, and Sociability”. In:
SocialCom/PASSAT, pp. 208–214.
Muaremi, Amir, Bert Arnrich, and Gerhard Tröster (2012). “A
Survey on Measuring Happiness with Smart Phones”. In:
Myers, David G. and Ed Diener (1995). “Who Is Happy?” In:
Psychological Science 6.1, pp. 10–19.
Myers, I.B. and P.B. Myers (1980). Gifts differing: understanding
personality type. Davies-Black Pub.
Nicolaou, M. A., H. Gunes, and M. Pantic (2011).
“Output-Associative RVM Regression for Dimensional and
Continuous Emotion Prediction”. In: Proceedings of IEEE
International Conference on Automatic Face and Gesture
Recognition (FG’11). Santa Barbara, CA, USA, pp. 16–23.
Nicu Sebe Erwin Bakker, Ira Cohen Theo Gevers and
Thomas Huang. Bimodal Emotion Recognition.
39 / 49

40. References XVI
Oikonomopoulos, A., I. Patras, M. Pantic, and N. Paragios
(2007). “Trajectory-based Representation of Human Actions”.
In: Lecture Notes in Artificial Intelligence, Special Volume on
Artificial Intelligence for Human Computing. Ed. by T. Huang,
A. Nijholt, A. Pentland, and M. Pantic. Vol. 4451, pp. 133–154.
Paninski, Liam (June 2003). “Estimation of entropy and mutual
information”. In: Neural Comput. 15.6, pp. 1191–1253.
Pantic, M. (2005). “Affective Computing”. In: in The
Encyclopedia of Multimedia Technology and Networking, M.
Pagani, Ed. Hershy, USA: Idea Group Reference. Vol. 1,
pp. 8–14.
— (2009a). “Affective Computing (revisited)”. In: in The
Encyclopedia of Multimedia Technology and Networking - 2nd
Edition, M. Pagani, Ed., Hershy, USA: Idea Group Reference.
Vol. 1, pp. 15–21.
40 / 49

41. References XVII
Pantic, M. (2009b). “Machine analysis of facial behaviour:
Naturalistic and dynamic behaviour”. In: Philosophical
Transactions of Royal Society B 364, pp. 3505–3513.
Pantic, M., A. Pentland, A. Nijholt, and T. Huang (2007).
“Human Computing and Machine Understanding of Human
Behavior: A Survey”. In: Lecture Notes in Artificial Intelligence,
Special Volume on Artificial Intelligence for Human Computing.
Ed. by T. Huang, A. Nijholt, A. Pentland, and M. Pantic.
Vol. 4451, pp. 47–71.
Pantic, M., R. Cowie, F. D’ericco, D. Heylen, M. Mehu,
C. Pelachaud, I. Poggi, M. Schroder, and A. Vinciarelli (2011).
“Social Signal Processing: The Research Agenda”. In: Visual
Analysis of Humans, pp. 511–538.
Pentland, Alex and Andrew Liu (Jan. 1999). “Modeling and
prediction of human behavior”. In: Neural Comput. 11.1,
pp. 229–242.
41 / 49

42. References XVIII
Picard, Rosalind W. (1997). Affective computing. Cambridge,
MA, USA: MIT Press.
Quercia, Daniele (2013). Donât Worry, Be Happy: The
Geography of Happiness on Facebook. http://profzero.
org/publications/websci13_dontworry.pdf.
Quercia, Daniele, Jonathan Ellis, Licia Capra, and
Jon Crowcroft (2012). “Tracking "gross community happiness"
from tweets”. In: CSCW, pp. 965–968.
Rabbi, Mashfiqui, Shahid Ali, Tanzeem Choudhury, and
Ethan Berke (2011). “Passive and In-Situ assessment of mental
and physical well-being using mobile sensors”. In: Proceedings
of the 13th international conference on Ubiquitous computing.
UbiComp ’11. Beijing, China: ACM, pp. 385–394.
42 / 49

43. References XIX
Rachuri, Kiran K., Mirco Musolesi, Cecilia Mascolo,
Peter J. Rentfrow, Chris Longworth, and Andrius Aucinas
(2010). “EmotionSense: a mobile phones based adaptive
platform for experimental social psychology research”. In:
UbiComp, pp. 281–290.
Robert LiKamWa Yunxin Liu, Nicholas D. Lane Lin Zhong
(2013). MoodScope: Building a Mood Sensor from Smartphone
Usage Patterns.
http://niclane.org/pubs/mobisys_moodscope.pdf.
Salah, Albert Ali and Bruno Lepri (2011). “Challenges of
Human Behavior Understanding for Inducing Behavioral
Change”. In: AmI Workshops, pp. 249–251.
43 / 49

44. References XX
Salah, AlbertAli, Javier Ruiz-del Solar, Ãetin Meriçli, and
Pierre-Yves Oudeyer (2012). “Human Behavior Understanding
for Robotics”. In: Human Behavior Understanding. Ed. by
AlbertAli Salah, Javier Ruiz-del Solar, Ãetin Meriçli, and
Pierre-Yves Oudeyer. Vol. 7559. Lecture Notes in Computer
Science. Springer Berlin Heidelberg, pp. 1–16.
Sanderson C.A., Cantor N. (1999). “A life task perspective on
personality coherence: Stability versus change in tasks, goals,
strategies, and outcomes.” In: D. Cervone & Y. Shoda (Eds.),
The Coherence of Personality: Social-cognitive Bases of
Consistency, Variability, and Organization. New York, NY, USA:
Guilford Press, pp. 372–392.
Schuller, B., R.J. Villar, G. Rigoll, and M. Lang (2005).
“Meta-Classifiers in Acoustic and Linguistic Feature
Fusion-Based Affect Recognition”. In: Acoustics, Speech, and
Signal Processing, 2005. Proceedings. (ICASSP ’05). IEEE
International Conference on. Vol. 1, pp. 325–328.
44 / 49

45. References XXI
Sebe, Nicu, Michael S. Lew, Ira Cohen, Ashutosh Garg, and
Thomas S. Huang (2002). “Emotion Recognition Using a
Cauchy Naive Bayes Classifier”. In: in Proc. ICPR, pp. 17–20.
Soleymani, M. and M. Pantic (2013). “Emotionally Aware TV”.
In: Proceedings of TVUX-2013: Workshop on Exploring and
Enhancing the User Experience for TV at ACM CHI 2013.
Paris, France.
Soleymani, M., M. Pantic, and T. Pun (2012). “Multimodal
Emotion Recognition in Response to Videos”. In: IEEE Trans.
on Affective Computing 3.2. in press, pp. 211–223.
Staiano, Jacopo, Bruno Lepri, Ramanathan Subramanian,
Nicu Sebe, and Fabio Pianesi (2011). “Automatic modeling of
personality states in small group interactions”. In: Proceedings
of the 19th ACM international conference on Multimedia. MM
’11. Scottsdale, Arizona, USA: ACM, pp. 989–992.
45 / 49

46. References XXII
Staiano, Jacopo, Bruno Lepri, Nadav Aharony, Fabio Pianesi,
Nicu Sebe, and Alex Pentland (2012). “Friends don’t lie:
inferring personality traits from social network structure”. In:
Proceedings of the 2012 ACM Conference on Ubiquitous
Computing. ACM, pp. 321–330.
Stehman, Stephen V. (1997). “Selecting and interpreting
measures of thematic classification accuracy”. In: Remote
Sensing of Environment 62.1, pp. 77–89.
Track your Happiness Web Application. (Access date:
12-JUN-2013). http://www.trackyourhappiness.org.
Tsutsui, Yoshiro (2012). “Weather and Individual Happiness”.
In: Weather, Climate, and Society 5.1, pp. 70–82.
Unser, Michael (2000). “Sampling–50 years after Shannon”. In:
Proceedings of the IEEE, pp. 569–587.
Vapnik, Vladimir N. (1995). The nature of statistical learning
theory. New York, NY, USA: Springer-Verlag New York, Inc.
46 / 49

47. References XXIII
Vitters, Joar (2001). “Personality traits and subjective
well-being: emotional stability, not extraversion, is probably the
important predictor”. In: Personality and Individual Differences
31.6, pp. 903 –914.
Wagner, Johannes, Florian Lingenfelser, Elisabeth Andre,
Jonghwa Kim, and Thurid Vogt (2011a). “Exploring Fusion
Methods for Multimodal Emotion Recognition with Missing
Data”. In: IEEE Transactions on Affective Computing 2.4,
pp. 206–218.
Wagner, Johannes, Elisabeth Andre, Florian Lingenfelser, and
Jonghwa Kim (Oct. 2011b). “Exploring Fusion Methods for
Multimodal Emotion Recognition with Missing Data”. In: IEEE
Trans. Affect. Comput. 2.4, pp. 206–218.
Wandishin, Matthew S. and Steven J. Mullen (2009). Multiclass
ROC Analysis. American Meteorological Society.
47 / 49

48. References XXIV
Wang, Yongjin and Ling Guan (2008). “Recognizing Human
Emotional State From Audiovisual Signals”. In: Multimedia,
IEEE Transactions on 10.4, pp. 659–668.
Zajonc, Robert B and Daniel N McIntosh (1992). “Emotions
research: Some promising questions and some questionable
promises”. In: Psychological Science, pp. 70–74.
Zeng, Z., M. Pantic, and T. Huang (2009). “Emotion Recognition
based on Multimodal Information”. In: Affective Information
Processing. Ed. by T. Tan and J. Tao. Springer, pp. 241–266.
Zeng, Z., M. Pantic, G. Roisman, and T. Huang (2009). “A
Survey of Affect Recognition Methods: Audio, Visual, and
Spontaneous Expressions”. In: IEEE Transactions on Pattern
Analysis and Machine Intelligence 31.1, pp. 39–58.
48 / 49

49. References XXV
Zeng, Zhihong, Jilin Tu, Ming Liu, Tong Zhang,
Nicholas Rizzolo, Zhenqiu Zhang, Thomas S. Huang,
Dan Roth, and Stephen Levinson (2004). “Bimodal HCI-related
affect recognition”. In: Proceedings of the 6th international
conference on Multimodal interfaces, pp. 137–143.
49 / 49