Daily Happiness Recognition from Mobile Phone Data


Published on

Published in: Technology, Business
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Daily Happiness Recognition from Mobile Phone Data

  1. 1. Happiness Recognitionfrom Mobile Phone DataAndrey Bogomolov1 Bruno Lepri2 Fabio Pianesi21University of Trento,Via Sommarive, 5I-38123 Povo - Trento, Italy2Fondazione Bruno KesslerVia Sommarive, 18I-38050 Povo - Trento, ItalyEmoPAR group meeting 2013-JUN-19, Trento, Italy.1 / 49
  2. 2. OutlineIntroductionProblem StatementSource DataRecognition ModelBasic FeaturesFinal Feature SpaceResultsLimitationsSummary2 / 49
  3. 3. Happiness as an emotional state – why is it important?Your ideas?. . .3 / 49
  4. 4. General Problem StatementInputsPervasive technology data.Multimodal data.OutputsEmotion recognition.Mood recognition.Personality recognition.4 / 49
  5. 5. Our Problem StatementInputsSmartphone call log.Smartphone sms log.Smartphone Bluetooth proximity hits.OutputsDaily happiness recognition.3-classes: {not happy, neutral, happy}.5 / 49
  6. 6. Data Collection117 subjectsdates: 21 February, 2010 – 16 July, 2011Source Space Dataset: Living Laboratoryphone calls 33497sms 22587Bluetooth hits 14609396 / 49
  7. 7. Happiness DataRecorded Happiness Scores Density0. 2 4 6ScoreDensity7 / 49
  8. 8. Happiness DataDescriptive Statisticsnumber of records 12991mean 4.84standard deviation 1.26median 5.00mean average deviation 1.48min 1.00max 7.00range 6.00skew -0.39kurtosis -0.078 / 49
  9. 9. Happiness DataWithin-person and between-subject variance0 1 2 3 4 5 = r1[, 1])VarianceDensityWithin−Person VarianceBetween−Person Variance9 / 49
  10. 10. Feature SpaceBasic FeaturesGeneral Phone UsageDiversityActive BehaviorsRegularity10 / 49
  11. 11. Feature SpaceBasic Features: General Phone Usage1. Total Number of Calls (Outgoing+Incoming)2. Total Number of Incoming Calls3. Total Number of Outgoing Calls4. Total Number of Missed Calls5. Number of SMS received6. Number of SMS sent11 / 49
  12. 12. Feature SpaceBasic Features: Diversity7. Number of Unique Contacts Called8. Number of Unique Contacts who Called9. Number of Unique Contacts Communicated with (Incoming+Outgoing)10. Number of Unique Contacts Associated with Missed Calls11. Entropy of Call Contacts12. Call Contacts to Interactions Ratio13. Number of Unique Contacts SMS received from14. Number of Unique Contacts SMS sent to15. Entropy of SMS Contacts16. Sms Contacts to Interactions Ratio12 / 49
  13. 13. Feature SpaceBasic Features: Active Behaviors17. Percent Call During the Night18. Percent Call Initiated19. Sms response rate20. Sms response latency21. Percent SMS Initiated13 / 49
  14. 14. Feature SpaceBasic Features: Regularity22. Average Inter-event Time for Calls (time elapsed between two events)23. Average Inter-event Time for SMS (time elapsed between two events)24. Variance Inter-event Time for Calls (time elapsed between two events)25. Variance Inter-event Time for SMS (time elapsed between two events)14 / 49
  15. 15. Feature SpaceProximity FeaturesGeneral Bluetooth Proximity1. Number of Bluetooth IDs2. Times most common Bluetooth ID is seen3. Bluetooth IDs accounting for n% of IDs seen4. Bluetooth IDs seen for more than k time slots5. Time interval for which a Bluetooth ID is seen6. Entropy of Bluetooth contactsDiversity7. Contacts to interactions ratioRegularity8. Average Bluetooth interactions inter-event time(time elapsed between two events)9. Variance of the Bluetooth interactions inter-event time(time elapsed between two events)15 / 49
  16. 16. Feature Space Innovation“Background Noise” Featuresa) peoples activity, as detected through their smartphonesb) the weather conditions {humidity, wind speed, pressure,total precipitation and visibility}c) personality traits {“Big Five"}Functional Innovationa) time domain – sliding window functionsb) Miller-Madow correction for entropy calculationˆHMM(θ) ≡ −pi=1θML,i log θML,i +ˆm − 12N16 / 49
  17. 17. Top-30 Features-1 0 1 MeanDecreaseAccuracy MeanDecreaseGinimeanTemperature 0.0099 0.0033 0.0094 0.0082 154.8379humidity 0.0047 -0.0033 0.0115 0.0074 149.7668pressure -0.0002 0.0008 0.0015 0.0011 149.0302windSpeed 0.0028 0.0017 0.0051 0.0040 142.7727visibility 0.0024 -0.0009 0.0051 0.0034 120.8683neuroticism 0.0690 0.0288 0.0399 0.0419 90.5721conscientiousness 0.0659 0.0480 0.0668 0.0627 90.2708extraversion 0.0511 0.0357 0.0472 0.0454 76.9467openness 0.0656 0.0340 0.0406 0.0429 73.9181totalPrecipitation 0.0007 -0.0000 0.0012 0.0009 73.5273agreeableness 0.0536 0.0282 0.0235 0.0289 70.7261bluetoothQ95TimeForWhichIdSeen 0.0233 0.0120 0.0149 0.0155 22.4018bluetoothQ90TimeForWhichIdSeen 0.0161 0.0082 0.0141 0.0131 19.8364smsRepliedEventsLatencyMedian 0.0093 0.0085 0.0096 0.0093 18.7719bluetoothIdsMoreThan04TimeSlotsSeen 0.0114 0.0066 0.0124 0.0110 15.6738bluetoothMaxTimeForWhichIdSeen 0.0141 0.0072 0.0095 0.0097 15.2546bluetoothTotalEntropyMillerMadow 0.0058 0.0017 0.0035 0.0035 13.9000bluetoothTotalEntropyShannon 0.0065 0.0009 0.0060 0.0050 13.1826callMeanInterEventTimePerDay -0.0003 -0.0000 0.0014 0.0009 13.1180incomingAndOutgoingCallsPerDay -0.0000 -0.0002 0.0024 0.0015 12.3962bluetoothQ50TimeForWhichIdSeen 0.0104 0.0104 0.0091 0.0095 12.2270callStandardDeviationInterEventTimePerDay 0.0004 -0.0005 0.0007 0.0004 10.1723bluetoothIdsMoreThan19TimeSlotsSeen 0.0087 0.0033 0.0089 0.0077 9.7388incomingCallsPerDay 0.0003 -0.0005 0.0009 0.0005 9.6572outgoingContactsToInteractionsRatioPerDay 0.0005 -0.0004 0.0013 0.0009 9.2016callsInitiatedRatioPerDay -0.0001 -0.0001 0.0014 0.0008 9.0245entropyMillerMadowCallsOutgoingWindow3Days 0.0001 -0.0008 0.0014 0.0008 8.7199bluetoothIdsMoreThan09TimeSlotsSeen 0.0074 0.0046 0.0040 0.0046 8.6006bluetoothQ75TimeForWhichIdSeen 0.0027 0.0018 0.0032 0.0028 8.4454outgoingCallsPerDay -0.0001 -0.0001 0.0012 0.0007 8.336817 / 49
  18. 18. Top-20 Features CorrelationagreeablenessbluetoothIdsMoreThan04TimeSlotsSeenbluetoothMaxTimeForWhichIdSeenbluetoothQ90TimeForWhichIdSeenbluetoothQ95TimeForWhichIdSeenbluetoothTotalEntropyMillerMadowbluetoothTotalEntropyShannoncallMeanInterEventTimePerDayconscientiousnessextraversionhumidityincomingAndOutgoingCallsPerDaymeanTemperatureneuroticismopennesspressuresmsRepliedEventsLatencyMediantotalPrecipitationvisibilitywindSpeedagreeablenessbluetoothIdsMoreThan04TimeSlotsSeenbluetoothMaxTimeForWhichIdSeenbluetoothQ90TimeForWhichIdSeenbluetoothQ95TimeForWhichIdSeenbluetoothTotalEntropyMillerMadowbluetoothTotalEntropyShannoncallMeanInterEventTimePerDayconscientiousnessextraversionhumidityincomingAndOutgoingCallsPerDaymeanTemperatureneuroticismopennesspressuresmsRepliedEventsLatencyMediantotalPrecipitationvisibilitywindSpeedVar1Var20.00.51.0value18 / 49
  19. 19. ResultsFinal Classifier Performance Metrics ComparisonTraining set Test setAccuracy 0.8081 0.8036Kappa 0.5879 0.5743AccuracyLower 0.8004 0.7878AccuracyUpper 0.8156 0.8187AccuracyNull 0.6415 0.6419AccuracyPValue 2.139e-303 8.826e-73McnemarPValue 5.647e-208 1.738e-5719 / 49
  20. 20. ResultsFinal Classifier Confusion Matrix for Training Set-1 0 1-1 782 119 750 153 1170 1451 600 903 6448Final Classifier Confusion Matrix for Test Set-1 0 1-1 197 30 140 34 274 371 152 243 161620 / 49
  21. 21. ResultsWhat We Learnt: Final Model ROC curveSpecificitySensitivity0. 0.8 0.6 0.4 0.2 0.0AUC: 0.84421 / 49
  22. 22. LimitationsDataData loss not registered as NA’sBatteryTemporal resolutionModelRequires personality dataRequires 1 week data collection periodNot tested on diverse cultural groups22 / 49
  23. 23. SummaryAutomatic recognition of people’s daily happiness frommobile phone data is feasible.Accuracy is approaching the results of multi-modalobtrusive methods.Future work should be focused on multi-step recognitionmodel development.23 / 49
  24. 24. Thank you!{andrey.bogomolov@unitn.it}24 / 49
  25. 25. References IAharony, Nadav, Wei Pan, Cory Ip, Inas Khayal, andAlex Pentland (2011). “Social fMRI: Investigating and shapingsocial mechanisms in the real world”. In: Pervasive and MobileComputing 7.6, pp. 643–659.Anguita, Davide, Alessandro Ghio, Luca Oneto, Xavier Parra,and Jorge L. Reyes-Ortiz (2012). “Human Activity Recognitionon Smartphones Using a Multiclass Hardware-Friendly SupportVector Machine”. In: IWAAL, pp. 216–223.Bickmore, Timothy W., Rosalind, and W. Picard (2005).“Establishing and Maintaining Long-Term Human-ComputerRelationships”. In: ACM Transactions on Computer HumanInteraction 12, pp. 293–327.Bishop, C. M. (1995). “Training with Noise is Equivalent toTikhonov Regularization”. In: Neural Computation 7.1,pp. 108–116.25 / 49
  26. 26. References IIBoth, Fiemke, Mark Hoogendoorn, Michel C. A. Klein, andJan Treur (2010). “Computational modeling and analysis of therole of physical activity in mood regulation and depression”. In:Proceedings of the 17th international conference on Neuralinformation processing: theory and algorithms - Volume Part I.ICONIP’10. Sydney, Australia: Springer-Verlag, pp. 270–281.Brave, Scott and Clifford Nass (2003). “The human-computerinteraction handbook”. In: ed. by Julie A. Jacko andAndrew Sears. Hillsdale, NJ, USA: L. Erlbaum Associates Inc.Chap. Emotion in human-computer interaction, pp. 81–96.Breiman, Leo (Oct. 2001). “Random Forests”. In: Mach. Learn.45.1, pp. 5–32.Broek, Egon L. (Jan. 2013). “Ubiquitous emotion-awarecomputing”. In: Personal Ubiquitous Comput. 17.1, pp. 53–67.Buhmann, Martin D. and M. D. Buhmann (2003). Radial BasisFunctions. New York, NY, USA: Cambridge University Press.26 / 49
  27. 27. References IIIBusso, Carlos, Zhigang Deng, Serdar Yildirim, Murtaza Bulut,Chul Min Lee, Abe Kazemzadeh, Sungbok Lee,Ulrich Neumann, and Shrikanth Narayanan (2004). “Analysis ofemotion recognition using facial expressions, speech andmultimodal information”. In: in Sixth International Conferenceon Multimodal Interfaces ICMI 2004. ACM Press, pp. 205–211.Campbell, Angus, Philip E. Converse, and Willard L Rodgers(1976). The quality of American life : perceptions, evaluations,and satisfactions / Angus Campbell, Philip E. Converse, WillardL. Rodgers. English. Russell Sage Foundation New York, xi,583 p. :Carlson, Charles R., Frank P. Gantz, and John C. Masters(1983). “Adults’ emotional states and recognition of emotion inyoung children”. English. In: Motivation and Emotion 7.1,pp. 81–101.27 / 49
  28. 28. References IVCarver, Charles S. and Michael F. Scheier (Jan. 1990). “Originsand Functions of Positive and Negative Affect: AControl-Process View”. In: Psychological Review 97.1,pp. 19–35.Chittaranjan, Gokul, Jan Blom, and Daniel Gatica-Perez (Mar.2013). “Mining large-scale smartphone data for personalitystudies”. In: Personal Ubiquitous Comput. 17.3, pp. 433–450.Cohen, Jacob (Apr. 1960). “A Coefficient of Agreement forNominal Scales”. In: Educational and PsychologicalMeasurement 20.1, pp. 37–46.Cohn, Jeffrey F. (2007). “Foundations of human computing:facial expression and emotion”. In: Proceedings of the ICMI2006 and IJCAI 2007 international conference on Artificalintelligence for human computing. ICMI’06/IJCAI’07. Banff,Canada: Springer-Verlag, pp. 1–16.28 / 49
  29. 29. References VCosta, P T and R R McCrae (1992). Revised NEO personalityinventory (NEO PI-R) and neo five-factor inventory (NEO-FFI).Vol. 0. October. Psychological Assessment Resources,pp. 17–20.— (1995). “Domains and facets: hierarchical personalityassessment using the revised NEO personality inventory.” In:Journal of Personality Assessment 64.1, pp. 21–50.Costa, Paul T. and Robert R. McCrae (1980). “Influence ofextraversion and neuroticism on subjective well-being: Happyand unhappy people”. In: Journal of Personality and SocialPsychology 38 (4), pp. 668–678.Cowie, R., E. Douglas-Cowie, K. Karpouzis, G. Caridakis,M. Wallace, and S. Kollias (2008). “Recognition of EmotionalStates in Natural Human-Computer Interaction”. English. In:Multimodal User Interfaces. Ed. by Dimitrios Tzovaras. Signalsand Commmunication Technologies. Springer BerlinHeidelberg, pp. 119–153.29 / 49
  30. 30. References VICowie, Roddy, Ellen Douglas-Cowie, Nicolas Tsapatsoulis,George Votsis, Stefanos Kollias, Winfried Fellenz, andJohn G Taylor (2001). “Emotion recognition in human-computerinteraction”. In: Signal Processing Magazine, IEEE 18.1,pp. 32–80.DeNeve, K.M. and H. Cooper (1998). “The happy personality: ameta-analysis of 137 personality traits and subjectivewell-being.” In: Psychol Bull 124.2, pp. 197–229.Denissen, Jaap J. A., Ligaya Butalid, Lars Penke, and MarcelA. G. Van Aken (2008). “The effects of weather on daily mood:A multilevel approach.” In: Emotion Researcher 8.5,pp. 662–667.Diener, Ed (1994). “Assessing subjective well-being: Progressand opportunities”. English. In: Social Indicators Research31.2, pp. 103–157.Diener, Ed and Martin E.P. Seligman (2002). “Very HappyPeople”. In: Psychological Science 13.1, pp. 81–84.30 / 49
  31. 31. References VIIDolan, R. J. (Nov. 2002). “Emotion, cognition, and behavior.”In: Science (New York, N.Y.) 298.5596.Dong, Wen, Bruno Lepri, and Alex Pentland (2011). “Modelingthe co-evolution of behaviors and social relationships usingmobile phone data”. In: MUM, pp. 134–143.Eagle, Nathan and Alex Pentland (2006). “Reality mining:sensing complex social systems”. In: Personal and UbiquitousComputing 10.4, pp. 255–268.Eagle, Nathan, Alex (Sandy) Pentland, and David Lazer (2009).“Inferring friendship network structure by using mobile phonedata”. In: Proceedings of the National Academy of Sciences106.36, pp. 15274–15278.Feldman, Lisa A (1995). “Valence focus and arousal focus:Individual differences in the structure of affective experience”.In: Journal of personality and social psychology 69,pp. 153–153.31 / 49
  32. 32. References VIIIFUNF Open Sensing Framework. (Access date: 12-JUN-2013).http://www.funf.org.Garcia, Salvador and Francisco Herrera (Dec. 2008). “AnExtension on "Statistical Comparisons of Classifiers overMultiple Data Sets" for all Pairwise Comparisons”. In: Journalof Machine Learning Research 9, pp. 2677–2694.Gray, J. A. (1991). “Neural systems, emotion, and personality.”English. In: Neurobiology of learning, emotion, and affect /editor, John Madden IV. Raven Press New York, pp. 273–306.Greenberg, Ariel M., William G. Kennedy, and Nathan Bos,eds. (2013). Social Computing, Behavioral-Cultural Modelingand Prediction - 6th International Conference, SBP 2013,Washington, DC, USA, April 2-5, 2013. Proceedings.Vol. 7812. Lecture Notes in Computer Science. Springer.Gunes, H. and M. Pantic (2010). “Automatic, Dimensional andContinuous Emotion Recognition”. In: Int’l Journal of SyntheticEmotion 1.1, pp. 68–99.32 / 49
  33. 33. References IXGunes, H., B. Schuller, M. Pantic, and R. Cowie (2011).“Emotion Representation, Analysis and Synthesis inContinuous Space: A Survey”. In: Proceedings of IEEEInternational Conference on Automatic Face and GestureRecognition (FG’11), EmoSPACE 2011 - 1st InternationalWorkshop on Emotion Synthesis, rePresentation, and Analysisin Continuous spacE. Santa Barbara, CA, USA, pp. 827–834.Hand, DavidJ. and RobertJ. Till (2001). “A SimpleGeneralisation of the Area Under the ROC Curve for MultipleClass Classification Problems”. English. In: Machine Learning45 (2), pp. 171–186.Hernandez, Javier, Mohammed E. Hoque, andRosalind W. Picard (2012). “Mood Meter: large-scale andlong-term smile monitoring system”. In: ACM SIGGRAPH 2012Emerging Technologies. SIGGRAPH ’12. Los Angeles,California: ACM, 15:1–15:1.33 / 49
  34. 34. References XJohn, Oliver P. and Sanjay Srivastava (1999). “The Big Five traittaxonomy: History, measurement, and theoreticalperspectives”. In: Handbook of Personality: Theory andResearch. Ed. by Lawrence A. Pervin and Oliver P. John.Second. New York: Guilford Press, pp. 102–138.Kapoor, Ashish, Winslow Burleson, and Rosalind W. Picard(Aug. 2007). “Automatic prediction of frustration”. In: Int. J.Hum.-Comput. Stud. 65.8, pp. 724–736.Kapoor, Ashish and Rosalind W. Picard (2005). “Multimodalaffect recognition in learning environments”. In: Proceedings ofthe 13th annual ACM international conference on Multimedia.MULTIMEDIA ’05. Hilton, Singapore: ACM, pp. 677–682.Keller, Matthew C., Barbara L. Fredrickson, Oscar Ybarra,Stephane Cote, Kareem Johnson, Joe Mikels, Anne Conway,and Tor Wager (Sept. 2005). “A Warm Heart and a Clear Head:The Contingent Effects of Weather on Mood and Cognition”.In: Psychological Science 16.9, pp. 724–731.34 / 49
  35. 35. References XIKrzanowski, Wojtek J. (2005). “Misclassification Rates”. In:Encyclopedia of Statistics in Behavioral Science.Lane, Nicholas, Mashfiqui Mohammod, Mu Lin, Xiaochao Yang,Hong Lu, Shahid Ali, Afsaneh Doryab, Ethan Berke,Tanzeem Choudhury, and Andrew Campbell (Apr. 2012).“BeWell: A Smartphone Application to Monitor, Model andPromote Wellbeing”. In: IEEE.Lane, Nicholas D., Emiliano Miluzzo, Hong Lu, Daniel Peebles,Tanzeem Choudhury, and Andrew T. Campbell (Sept. 2010). “Asurvey of mobile phone sensing”. In: Comm. Mag. 48.9,pp. 140–150.Lee, Gary R. and Masako Ishii-Kuntz (Dec. 1987). “SocialInteraction, Loneliness, and Emotional Well-Being among theElderly”. In: Research on Aging 9.4, pp. 459–482.35 / 49
  36. 36. References XIILester, Jonathan, Tanzeem Choudhury, and Gaetano Borriello(2006). “A Practical Approach to Recognizing PhysicalActivities”. In: Pervasive Computing. Ed. by KennethP. Fishkin,Bernt Schiele, Paddy Nixon, and Aaron Quigley. Vol. 3968.Lecture Notes in Computer Science. Springer BerlinHeidelberg, pp. 1–16.LEWIS, M.E. and J.M.E. HAVILAND-JONES (2004). Handbookof Emotions. Guilford Publications, Incorporated.Lin, Wei-Jiun and James J. Chen (Jan. 2013).“Class-imbalanced classifiers for high-dimensional data”. In:Briefings in Bioinformatics 14.1, pp. 13–26.Lyubomirsky, Sonja, Laura King, and Ed Diener (Nov. 2005).“The Benefits of Frequent Positive Affect: Does Happiness Leadto Success?” In: Psychological Bulletin 131.6, pp. 803–855.36 / 49
  37. 37. References XIIILyubomirsky, Sonja, Chris Tkach, and Robin M. Dimatteo(2006). “WHAT ARE THE DIFFERENCES BETWEENHAPPINESS AND SELF-ESTEEM?” In: Social IndicatorsResearch 78, pp. 363–404.Ma, Yuanchao, Bin Xu, Yin Bai, Guodong Sun, and Run Zhu(2012). “Daily Mood Assessment Based on Mobile PhoneSensing”. In: Wearable and Implantable Body Sensor Networks(BSN), 2012 Ninth International Conference on, pp. 142–147.Madan, Anmol, Manuel Cebrián, David Lazer, andAlex Pentland (2010). “Social sensing for epidemiologicalbehavior change”. In: UbiComp, pp. 291–300.McCrae, Robert R. and Paul T. Costa (1991). “Adding Liebeund Arbeit: The Full Five-Factor Model and Well-Being”. In:Personality and Social Psychology Bulletin 17.2, pp. 227–232.Miller, G. (1955). “Note on the bias of information estimates”.In: Information theory in psychology: Problems and methods2.95, p. 100.37 / 49
  38. 38. References XIVMiller, Geoffrey (2012). “The Smartphone PsychologyManifesto”. In: Perspectives on Psychological Science 7.3,pp. 221–237.Montjoye, Yves-Alexandre, Jordi Quoidbach, Florent Robic, andAlex(Sandy) Pentland (2013a). “Predicting Personality UsingNovel Mobile Phone-Based Metrics”. In: Social Computing,Behavioral-Cultural Modeling and Prediction. Ed. byArielM. Greenberg, WilliamG. Kennedy, and NathanD. Bos.Vol. 7812. Lecture Notes in Computer Science. Springer BerlinHeidelberg, pp. 48–55.Montjoye, Yves-Alexandre de, Jordi Quoidbach, Florent Robic,and Alex Pentland (2013b). “Predicting Personality Using NovelMobile Phone-Based Metrics”. In: SBP. Ed. byAriel M. Greenberg, William G. Kennedy, and Nathan Bos.Vol. 7812. Lecture Notes in Computer Science. Springer,pp. 48–55.38 / 49
  39. 39. References XVMoturu, Sai T., Inas Khayal, Nadav Aharony, Wei Pan, andAlex Pentland (2011). “Using Social Sensing to Understand theLinks between Sleep, Mood, and Sociability”. In:SocialCom/PASSAT, pp. 208–214.Muaremi, Amir, Bert Arnrich, and Gerhard Tröster (2012). “ASurvey on Measuring Happiness with Smart Phones”. In:Myers, David G. and Ed Diener (1995). “Who Is Happy?” In:Psychological Science 6.1, pp. 10–19.Myers, I.B. and P.B. Myers (1980). Gifts differing: understandingpersonality type. Davies-Black Pub.Nicolaou, M. A., H. Gunes, and M. Pantic (2011).“Output-Associative RVM Regression for Dimensional andContinuous Emotion Prediction”. In: Proceedings of IEEEInternational Conference on Automatic Face and GestureRecognition (FG’11). Santa Barbara, CA, USA, pp. 16–23.Nicu Sebe Erwin Bakker, Ira Cohen Theo Gevers andThomas Huang. Bimodal Emotion Recognition.39 / 49
  40. 40. References XVIOikonomopoulos, A., I. Patras, M. Pantic, and N. Paragios(2007). “Trajectory-based Representation of Human Actions”.In: Lecture Notes in Artificial Intelligence, Special Volume onArtificial Intelligence for Human Computing. Ed. by T. Huang,A. Nijholt, A. Pentland, and M. Pantic. Vol. 4451, pp. 133–154.Paninski, Liam (June 2003). “Estimation of entropy and mutualinformation”. In: Neural Comput. 15.6, pp. 1191–1253.Pantic, M. (2005). “Affective Computing”. In: in TheEncyclopedia of Multimedia Technology and Networking, M.Pagani, Ed. Hershy, USA: Idea Group Reference. Vol. 1,pp. 8–14.— (2009a). “Affective Computing (revisited)”. In: in TheEncyclopedia of Multimedia Technology and Networking - 2ndEdition, M. Pagani, Ed., Hershy, USA: Idea Group Reference.Vol. 1, pp. 15–21.40 / 49
  41. 41. References XVIIPantic, M. (2009b). “Machine analysis of facial behaviour:Naturalistic and dynamic behaviour”. In: PhilosophicalTransactions of Royal Society B 364, pp. 3505–3513.Pantic, M., A. Pentland, A. Nijholt, and T. Huang (2007).“Human Computing and Machine Understanding of HumanBehavior: A Survey”. In: Lecture Notes in Artificial Intelligence,Special Volume on Artificial Intelligence for Human Computing.Ed. by T. Huang, A. Nijholt, A. Pentland, and M. Pantic.Vol. 4451, pp. 47–71.Pantic, M., R. Cowie, F. D’ericco, D. Heylen, M. Mehu,C. Pelachaud, I. Poggi, M. Schroder, and A. Vinciarelli (2011).“Social Signal Processing: The Research Agenda”. In: VisualAnalysis of Humans, pp. 511–538.Pentland, Alex and Andrew Liu (Jan. 1999). “Modeling andprediction of human behavior”. In: Neural Comput. 11.1,pp. 229–242.41 / 49
  42. 42. References XVIIIPicard, Rosalind W. (1997). Affective computing. Cambridge,MA, USA: MIT Press.Quercia, Daniele (2013). Donât Worry, Be Happy: TheGeography of Happiness on Facebook. http://profzero.org/publications/websci13_dontworry.pdf.Quercia, Daniele, Jonathan Ellis, Licia Capra, andJon Crowcroft (2012). “Tracking "gross community happiness"from tweets”. In: CSCW, pp. 965–968.Rabbi, Mashfiqui, Shahid Ali, Tanzeem Choudhury, andEthan Berke (2011). “Passive and In-Situ assessment of mentaland physical well-being using mobile sensors”. In: Proceedingsof the 13th international conference on Ubiquitous computing.UbiComp ’11. Beijing, China: ACM, pp. 385–394.42 / 49
  43. 43. References XIXRachuri, Kiran K., Mirco Musolesi, Cecilia Mascolo,Peter J. Rentfrow, Chris Longworth, and Andrius Aucinas(2010). “EmotionSense: a mobile phones based adaptiveplatform for experimental social psychology research”. In:UbiComp, pp. 281–290.Robert LiKamWa Yunxin Liu, Nicholas D. Lane Lin Zhong(2013). MoodScope: Building a Mood Sensor from SmartphoneUsage Patterns.http://niclane.org/pubs/mobisys_moodscope.pdf.Salah, Albert Ali and Bruno Lepri (2011). “Challenges ofHuman Behavior Understanding for Inducing BehavioralChange”. In: AmI Workshops, pp. 249–251.43 / 49
  44. 44. References XXSalah, AlbertAli, Javier Ruiz-del Solar, Ãetin Meriçli, andPierre-Yves Oudeyer (2012). “Human Behavior Understandingfor Robotics”. In: Human Behavior Understanding. Ed. byAlbertAli Salah, Javier Ruiz-del Solar, Ãetin Meriçli, andPierre-Yves Oudeyer. Vol. 7559. Lecture Notes in ComputerScience. Springer Berlin Heidelberg, pp. 1–16.Sanderson C.A., Cantor N. (1999). “A life task perspective onpersonality coherence: Stability versus change in tasks, goals,strategies, and outcomes.” In: D. Cervone & Y. Shoda (Eds.),The Coherence of Personality: Social-cognitive Bases ofConsistency, Variability, and Organization. New York, NY, USA:Guilford Press, pp. 372–392.Schuller, B., R.J. Villar, G. Rigoll, and M. Lang (2005).“Meta-Classifiers in Acoustic and Linguistic FeatureFusion-Based Affect Recognition”. In: Acoustics, Speech, andSignal Processing, 2005. Proceedings. (ICASSP ’05). IEEEInternational Conference on. Vol. 1, pp. 325–328.44 / 49
  45. 45. References XXISebe, Nicu, Michael S. Lew, Ira Cohen, Ashutosh Garg, andThomas S. Huang (2002). “Emotion Recognition Using aCauchy Naive Bayes Classifier”. In: in Proc. ICPR, pp. 17–20.Soleymani, M. and M. Pantic (2013). “Emotionally Aware TV”.In: Proceedings of TVUX-2013: Workshop on Exploring andEnhancing the User Experience for TV at ACM CHI 2013.Paris, France.Soleymani, M., M. Pantic, and T. Pun (2012). “MultimodalEmotion Recognition in Response to Videos”. In: IEEE Trans.on Affective Computing 3.2. in press, pp. 211–223.Staiano, Jacopo, Bruno Lepri, Ramanathan Subramanian,Nicu Sebe, and Fabio Pianesi (2011). “Automatic modeling ofpersonality states in small group interactions”. In: Proceedingsof the 19th ACM international conference on Multimedia. MM’11. Scottsdale, Arizona, USA: ACM, pp. 989–992.45 / 49
  46. 46. References XXIIStaiano, Jacopo, Bruno Lepri, Nadav Aharony, Fabio Pianesi,Nicu Sebe, and Alex Pentland (2012). “Friends don’t lie:inferring personality traits from social network structure”. In:Proceedings of the 2012 ACM Conference on UbiquitousComputing. ACM, pp. 321–330.Stehman, Stephen V. (1997). “Selecting and interpretingmeasures of thematic classification accuracy”. In: RemoteSensing of Environment 62.1, pp. 77–89.Track your Happiness Web Application. (Access date:12-JUN-2013). http://www.trackyourhappiness.org.Tsutsui, Yoshiro (2012). “Weather and Individual Happiness”.In: Weather, Climate, and Society 5.1, pp. 70–82.Unser, Michael (2000). “Sampling–50 years after Shannon”. In:Proceedings of the IEEE, pp. 569–587.Vapnik, Vladimir N. (1995). The nature of statistical learningtheory. New York, NY, USA: Springer-Verlag New York, Inc.46 / 49
  47. 47. References XXIIIVitters, Joar (2001). “Personality traits and subjectivewell-being: emotional stability, not extraversion, is probably theimportant predictor”. In: Personality and Individual Differences31.6, pp. 903 –914.Wagner, Johannes, Florian Lingenfelser, Elisabeth Andre,Jonghwa Kim, and Thurid Vogt (2011a). “Exploring FusionMethods for Multimodal Emotion Recognition with MissingData”. In: IEEE Transactions on Affective Computing 2.4,pp. 206–218.Wagner, Johannes, Elisabeth Andre, Florian Lingenfelser, andJonghwa Kim (Oct. 2011b). “Exploring Fusion Methods forMultimodal Emotion Recognition with Missing Data”. In: IEEETrans. Affect. Comput. 2.4, pp. 206–218.Wandishin, Matthew S. and Steven J. Mullen (2009). MulticlassROC Analysis. American Meteorological Society.47 / 49
  48. 48. References XXIVWang, Yongjin and Ling Guan (2008). “Recognizing HumanEmotional State From Audiovisual Signals”. In: Multimedia,IEEE Transactions on 10.4, pp. 659–668.Zajonc, Robert B and Daniel N McIntosh (1992). “Emotionsresearch: Some promising questions and some questionablepromises”. In: Psychological Science, pp. 70–74.Zeng, Z., M. Pantic, and T. Huang (2009). “Emotion Recognitionbased on Multimodal Information”. In: Affective InformationProcessing. Ed. by T. Tan and J. Tao. Springer, pp. 241–266.Zeng, Z., M. Pantic, G. Roisman, and T. Huang (2009). “ASurvey of Affect Recognition Methods: Audio, Visual, andSpontaneous Expressions”. In: IEEE Transactions on PatternAnalysis and Machine Intelligence 31.1, pp. 39–58.48 / 49
  49. 49. References XXVZeng, Zhihong, Jilin Tu, Ming Liu, Tong Zhang,Nicholas Rizzolo, Zhenqiu Zhang, Thomas S. Huang,Dan Roth, and Stephen Levinson (2004). “Bimodal HCI-relatedaffect recognition”. In: Proceedings of the 6th internationalconference on Multimodal interfaces, pp. 137–143.49 / 49