Y.-C. Yu et al.: Magic Mirror Table for Social-Emotion Alleviation in the Smart Home 127expression recognition is an important part for automaticfeedback of information about the viewer’s internal emotionalstate, intention, or social communication.The facial recognition system is also available for smartphones. As reported by Terzis, et al. , their system canachieve a success rate of 89 % to recognize six basic emotions(happy, angry, sad, surprised, scared, and disgusted) plusneutral based on facial expressions.C. Alleviative EffectsBased on the valence-arousal (V-A) mood plane ,human affective response or state can be represented withvalence and arousal coordinates. Valence is typicallycharacterized as the feeling of a person from pleasant orpositive to unpleasant or negative. Arousal is the activation ofthe emotion.Music therapy is a clinical practice that lets the client hearand touch to sense music in order to achieve physical andpsychological treatment effects . As  indicates,“Familiar Songs Method” is effective enough to improve thearousal level of senile dementia. Another case of musictherapy treatment is applied to hypertensive Patients .Variability of heart rate can be used for evaluating theeffects of music therapy on anxiety reduction of patients .The study of Cheng et al.  confirms that music therapy onpatients with dental anxiety can obviously relieve patients’symptoms and dreadful emotion. They also point out thatdifferent types of music should be chosen for different groupsof people.III. SYSTEM DESIGN FOR MAGIC MIRROR TABLEThe proposed system includes several hardware andsoftware components. The following briefly describe thesecomponents with the applied techniques.A. Overview of the Proposed SystemFig. 1 is the prototype of the proposed minor system. Themirror itself is an LCD monitor mounted with a one-waymirror in front of the monitor. If the monitor turns off, theone-way mirror acts as a regular mirror. On the other hand, ifthe monitor turns on, the mirror is transparent for viewers tosee the screen of the monitor. The power of the monitor iscontrolled by the system based on the state of the operation.On top of the monitor is a camera for detecting the emotion ofthe viewer based on his/her facial expression. To guaranteesufficient lighting for the camera, the system also controls anLED light source. The system connects to speaker(s) by anUSB control box. The control box also connects to motionsensors and the LED light source. The system controller is acomputer mounted on the base of the mirror (and monitor).The speakers are typically placed on both sides of the mirrorbase. The proposed system operates in four different modes,as described below.(1) Mirror mode: When the system is in this mode, itsprimary function is to act as a mirror. In the meantime,however, the camera captures the user’s facial expression. Thesystem then analyzes the identity of the user and his/her facialexpression. Based on the analysis, the system may enter to thealleviation mode. Alternatively, the user may choose thereminder mode to view the planned calendar.(2) Alleviation mode: When the system is in this mode,suitable text messages are displayed on the LCD monitor forthe user to view. In addition, a voice message followed bybackground music is played through the speakers.(3) Reminder mode: When the system is in this mode, themonitor shows the text message with a voice message tosignal the user of the present mode. In this mode, the user canmanage his/her personal calendar to check for any futureevents. When done, the user can exit this mode so that systemmay enter the mirror mode again.(4) Silent mode: The system is in this mode if it is in theinitialization phase or the user is out of the operation range.When in this mode, the camera does not capture pictures andthe LCD monitor always turns off.B. Detection of the Presence of the ViewerIn a typical house, the space is divided by room walls.Therefore, it is easy to detect the distance between the viewerand the mirror. To simplify the design, instead of distancemeasuring sensor, we deploy the motion sensors on the entryFig. 1. The prototype of the proposed system.Fig. 2. The deployment of motion Sensors for detecting the presence ofthe viewer.
128 IEEE Transactions on Consumer Electronics, Vol. 58, No. 1, February 2012or main pathway of the room, as shown in Fig. 2. If the sensorclose to the room door detects a body motion, the systemleaves the silent mode and enters the mirror mode. Such adesign simplifies the problem of detecting the viewer andgives the system additional time for mode transition.C. Operation of the Proposed SystemThe basic operations of the system are outlined below: When a viewer walks into the room, the motion sensordetects the motion. Then, the system enters the mirrormode. The digital camera begins to capture the imagesequences. The system uses features extracted from theface of the viewer to find his/her identify. Based on theidentity information, the system fetches the emotionalprofile of the viewer for emotion classification in thenext step. If the identity is identified, the system checks eachframe of image sequences for facial emotion recognition.The detailed method is given in the next subsection. Ifthe emotion belongs to negative, the system calculatesthe temporal emotion energy. When the accumulatedenergy is greater than a threshold, the system switchesto the alleviation mode. The alleviating mode consists of several steps. First,emotionally supporting sentences are selected from thedatabase based on the personal profiles. For example, asentence such as “Dont worry, things will get better” maybe displayed on the mirror (monitor) and simultaneouslyread out from the speakers. Then, the system playsfavorite background music about sunshine. For such amechanism, the database for positive words (sentences) isbuilt in advance. Similarly, the database of backgroundmusic is also collected and selected by the individual userbefore the actual operation of the system. The combination of the voice and music usually beginswith fade-in background music. Then, the voice ofpositive words (sentences). Finally, the backgroundmusic only. In a typical case, the background musicoften lasts for several minutes. In addition to the alleviation mode, the system can beoperated as an event reminder. When in this mode, thesystem checks all planned schedule to determinewhether a reminder message should be provided or not.If so, the system displays a message and reads themessage with a voice. If necessary, the user may instructthe system to switch to the reminder mode. Then, themonitor can display todays To-do-list and eventcalendar. The operation of the reminder mode is similar to thatgiven in the home calendar service  proposed by theauthors. The snapshot of the schedule interface is shownin Fig. 3. To use the calendar, the user drags and drops atime slot on the screen to schedule the selected event. Tocheck the calendar, the user clicks a time slot to checkthe event.D. Social Emotion RecognitionIn the proposed system, a high resolution digital camera(Full HD 1080p video recording) is mounted on top of themirror to capture the sequence of face images. Following theapproached proposed by , we use the head-trackingalgorithm to find the bounding box of the face. The detailedcontour of the face is obtained inside the face box by the skin-color blob detector with convex region calculation. Thedetector detects and clusters the pixels of the face based on theYCbCr color space. As  proposed, we also use theelliptical model for skin-color modeling. The pixels arecalculated by a model function. For each pixel with Y, Cb, andCr components, we calculate its intermediate values ),( yxusing the following equation (Y component is not used):yxcCrcCbyxcossinsincos(1)where xc =109.38, yc =152.02, and =2.53. A pixel isconsidered as a skin pixel to be clustered together to form askin blob if it satisfies the following constraint:1)()(222becyaecx yx (2)where xec =1.60, yec =2.41, a =25.39, and b =14.03, Finally,skin blobs fuse together to form a convex face area.After the face area is determined, the facial features can bedetected based on the point-based face model and anatomyalgorithm . We extracted geometric features of the facesuch as eyes, mouth, nose, corners of the eyes, mouth shape,etc. These features are related to facial expression in atemporal consequence. In our prototype system, the front-view face model is composed of 12 features from a set of 19facial points, as shown in Fig. 4. Because our system is usedin real time, we use fewer facial points and features to reducethe complexity of the algorithm. The recognition rate can beimproved with more features and training data.To recognize the emotion of the viewer based on the facialexpression, we use a back-propagation neural network (BPNN) as the classifier to identify six basic emotions (happy,angry, sad, surprised, scared, and disgusted). In the homeenvironment, however, the service of magic mirror is only forhome members. Therefore, each home member has his/herdedicated neural network for performance consideration. Eachnetwork had 12 input nodes, with each corresponding to the12 input facial features. The output layer contains 7 nodes (sixFig. 3. The home calendar in the reminder mode.
Y.-C. Yu et al.: Magic Mirror Table for Social-Emotion Alleviation in the Smart Home 129emotions plus neutral) to represent different emotioncategories. There are one hidden layers and the number ofhidden nodes is 14, 28, or 35.Other factors affecting the recognition performance are learningrate , momentum value , and the parameter in the activationfunction. In the training process, different numbers of neurons (14,28, or 35) in the hidden layers are used. The values of , , and are also experimentally determined with repeated training. Theerror criterion is set at 110-9and the maximum number of epochsused for training varies from 160 to 1100.The facial features are normalized against the viewer’s distanceand angle to the mirror or angle before sent to the neural network.Hyperbolic tangent sigmoid (TanSig) and logarithmic sigmoid(LogSig) functions are used for the hidden-layer neurons andoutput neurons, respectively. At the output neuron, an output of 0.6or higher is set to 1, otherwise set to zero.For the training data, each home member is taken 30 pictures ofhis/her front view with different emotions. The size of the images is640 480 pixels with 24-bit color intensity encoding. For bettertraining performance, we manually remove unqualified images.E. Calculation of Temporal Emotion EnergyIn our system, we define sad, scared, and disgusted asnegative emotion. Since we have a sequence of images, wedefine the temporal emotion energy as the accumulation oftemporal change of emotion. When the accumulated negativeenergy is greater than a threshold, the system goes into thealleviation mode. The threshold is experimentally determinedand may be fine-tuned by the viewer’s historical data.The temporal emotion energy is calculated as follows.Firstly, we give each emotion a value e. The mapping we usein the experiment is: happy = +3, surprised = +2, angry = +1,scared = -1, disgusted = -2, and sad = -3. Then the temporalemotion energy N(k) for frame k is calculated as),()()(10iDiNkekNNi (3)where N is the length of the sliding window and)1()( iSxy eSiD (4)is a weighting function to give larger weights for recentemotion values. In (4), > 0 is a parameter to control thetemporal decay, xS and yS are the scale factors of descendingcurves on x-axis and y-axis, respectively. In our experiment,we use N = 75 (for NTSC video system), =1.05, xS =5/N,and yS = 0.7 threshold.F. The Alleviation TreatmentPositive words can bring positive thinking, uplift yourmood, and inspire your life. The magic mirror table isdesigned for doing this job. When the viewer has a negativeemotion, the magic mirror can give you positive words toalleviate the viewer’s negative emotions. In addition, we alsouse music for alleviation as the music therapy has beenclinically proven to be effective. In our system, a positive-words dictionary is built in advance. Then, the system canselect sentences from the dictionary according to the detectedemotion type and the viewer’s preference. In the meantime,the background music is also selected according to thedetected emotion type and the viewer’s favorite music style.IV. EXPERIMENTS AND RESULTSBefore conducting the experiments, we need to think how toevaluate the effect of alleviation. In the literature , bothsubjective measures (self-reports) and physiological data (skinconductance and heart rate) are used to assess affective responses.For physiological data, heart rate is a reliable index for valencechange, whereas skin conductance is associated with arousal. Inour experiment, we only use the change of heart rate as the meanto evaluate the alleviation effect. Positive emotion causes theheart rate to accelerate, whereas negative emotion causes theheart rate to slow down . When the heart rate accelerates, itmeans that the viewer recovers from a negative emotion.Following the convention of most psychological measures, wedivide the alleviation effect into five degrees based on the changeof the heart rate. For this reason, the heart rate of each homemember has been repeatedly measured to obtain the personaldistribution of heart rate before using the magic mirror. Thedistribution is used to map to the five levels of alleviation. In ourcase, a heart rate in the range between 0.5 and 1.0 standarddeviation above average is mapped to level one, between 1.0 and1.5 standard deviations is mapped to level two, and so on.In the experiment, we do not use self-report to evaluate thealleviation effect. The viewer’s report is only used to calculatethe correctness rate of the emotion recognition algorithm inthe proposed system. In the experiment, if the viewer has anegative emotion, the proposed system then delivers thealleviation treatment, and the degree of alleviation is evaluated.The experiments are conducted as follows. We select afamily of four people to test the proposed system. Thepersonal emotion profile for each person is measured inadvance. For this family, six emotions are tested 30 times. Thesuccess rate of emotion detection and the degree ofalleviations are recorded. The difficult part of the experimentis that the emotion cannot follow ones inclinations. If theemotion comes from acting, the results may not represent theactual situation in reality. In our case, we use the self-report todetermine the real mood of the viewer.Fig. 4. The facial features for social emotion classification.
130 IEEE Transactions on Consumer Electronics, Vol. 58, No. 1, February 2012The experimental results are shown in table I. The resultsshow that angry and sad emotions are much easier to expressand to detect, therefore higher success rates. The table alsoshows that the sad emotion can be effectively alleviated. Theresults also indicate that positive emotions are significantlyeasier to detect, particularly with the facial expression.Although the correct rate of our emotion recognitionalgorithm is not very high, the system nevertheless showssome positive results in emotion alleviation. The correct ratecan be further improved with more features and with a largertraining set, especially a training set containing historicalbehavioral data.In the experiment, we use heart rate to represent thealleviation effect. In the future development, the heart ratemay be replaced with other physical features such as muscleneural voltage, EGG, skin EKG, and so on. Certainly, moreevidences should be collected to support the relationshipbetween physical features and emotion.V. CONCLUSIONIn this paper, we propose a magic mirror system which isable to determine the viewer’s emotion through the analysis ofhis/her facial expressions. The results of scenario-basedevaluation confirm that the proposed mirror system is able toalleviate the viewer’s emotion when he/she is in sad mood.With the proposed system, we demonstrate the feasibility ofdeveloping a piece of socially-awarded furniture (smartfurniture). For the smart furniture, it may engage into a homemember’s social network and share his/her feeling.In the present design, the emotion of the viewer isdetermined solely by analyzing his/her facial expressions.However, social signals revealing the emotion of a personinclude facial expressions, vocal utterance, body gestures andpostures, and so on. In the further, the emotion awareness canbe extended into multi-modalities analysis to incorporate theabove mentioned social signals. Nevertheless, the proposedsystem is a meaningful starting point toward a moresophisticated human machine interaction (HCI) system.ACKNOWLEDGMENTS. D. Y. would like to thank Professor Wen-Chung Kao ofNational Taiwan Normal University, Taipei, Taiwan forvaluable discussions and suggestions.REFERENCES S. Davidoff, M.K. Lee, J. Zimmerman, and, A. Dey, “Socially-awarerequirements for a smart home,” In Proceedings of the InternationalSymposium on Intelligent Environments, p. 41-44, 2000. P.A. Cabarcos, R.S. Guerrero, F.A. Mendoza, D. Díaz-Sánchez, andA. Marín López, “FamTV: An architecture for presence-awarepersonalized television,” IEEE Transactions on ConsumerElectronics, vol.57, no. 1, pp.6-13, 2011. A. Vinciarelli, M. Pantic, and H. Bourlard. “Social signal processing:survey of an emerging domain,” Image and Vision ComputingJournal, vol. 27, no. 12, pp. 1743-1759, 2009. M. Pantic and I. Patras, “Dynamics of facial expression: recognition offacial actions and their temporal segments from face profile imagesequences,” IEEE Transactions on Systems, Man, and Cybernetics-Part B: Cybernetics, vol. 36, no. 2, pp. 433–449, 2006. J.F. Cohn, “Foundations of human computing: facial expression andemotion,” In Proceedings of the ACM International Conference onMultimodal Interfaces, pp. 233–238, 2006. M. Pantic and L.J.M. Rothkrantz, “Expert system for automaticanalysis of facial expression,” Image and Vision Computing Journal,vol. 18, no. 11, pp. 881-905, 2000. T. Balomenos, A. Raouzaiou, S. Ioannou, A. Drosopoulos, K.Karpouzis, and S. Kollias, “Emotion analysis in man-aachineinteraction systems,” Lecture Notes in Computer Science, vol. 3361,pp. 318 – 328, 2004. M. Pantic and L.J.M. Rothkrantz, “Automatic analysis of facialexpressions: the state of the art,” IEEE Transactions on PatternAnalysis and Machine Intelligence, vol.22, no.12, pp. 1424-1445,2000. H.-A. Kwang, and M. Chung, “Cognitive face analysis system forfuture interactive TV,” IEEE Transactions on Consumer Electronicsvol. 55, no. 4, pp. 2271 – 2279, 2009. V. Terzis, C.N., Moridis, and A.A. Economides, “Measuring instantemotions during a self-assessment test: the use of FaceReader,” InProceedings of the 7th International Conference on Methods andTechniques in Behavioral Research (MB 10), 2010. J. A. Russell, “Affective space is bipolar,” Journal of Personality andSocial Psychology, vol. 37, no. 3, pp. 345–356, 1979. P. Zhou, F.-F. Sui, A. Zhang, F. Wang, and G.-H. Li, “Musictherapy on heart rate variability,” 3rd International Conference onBiomedical Engineering and Informatics (BMEI), vol. 3, pp. 965 –968, 2010. T. Takahashi, S. Suzuki, K. Kasumatsu, K. Shoubu, and S.P.Ninomija, “Effectiveness measurement of familiar songs method bymeans of electroencephalography,” in Proceedings of the 22nd AnnualInternational Conference of the IEEE Engineering in Medicine andBiology Society, vol. 3, pp. 1887–1888, 2000. X.F. Teng, M.Y.M. Wong, and Y.T. Zhang, “The Effect of music onhypertensive patients,” in proceedings of the 29th AnnualInternational Conference of the IEEE Engineering in Medicine andBiology Society, pp. 4649 – 4651, 2007. H.W. Chiu, L.S. Lin, M.C. Kuo, H.S. Chiang, and C.Y. Hsu, "Usingheart rate variability analysis to assess the effect of music therapy onanxiety reduction of patients," in Computers in Cardiology, pp. 469-472, 2003. P. Cheng and R. Chen, “Study on therapeutic effect of music therapyon patients with dental anxiety of college Students,” in Proceedings ofInternational Conference on E-Health Networking, Digital Ecosystemsand Technologies (EDT2010), vol. 1, pp. 12 – 14, 2010. Y.-C. Yu, S. D. You, and D.-R. Tsai, “A calendar oriented service forsmart home,” in Proceedings of Sixth International Conference onNetworked Computing and Advanced Information Management(NCM), pp. 151 – 156, 2010. Q.-R Jiang, H.-l. Li, “Robust human face detection in complicatedcolor images,” in Proceedings of 2010 IEEE International Conferenceon Information Management and Engineering (ICIME), pp. 218 – 221,2010 Y. Sun, Z. Li, C. Tang, Y. Chen, and R. Jiang, “Facial expressionanalysis - a hybrid neural network based approach,” in Proceedings of2nd International Congress on Image and Signal Processing, pp. 1–5,2009.TABLE ISCENARIOS-BASED EVALUATIONSocial emotionSuccess rate ofEmotion detectionDegree of alleviationaHappy 22/30 ------bAngry 23/30 ✔ ✔Sad 12/30 ✔ ✔ ✔ ✔ ✔Surprised 14/30 ✔Scared 9/30 ✔ ✔Disgusted 10/30 ✔ ✔ ✔ ✔aFive levels of degree;bno alleviation needed
Y.-C. Yu et al.: Magic Mirror Table for Social-Emotion Alleviation in the Smart Home 131BIOGRAPHIESYuan-Chih Yu is an instructor in ComputerInformation Center, Chinese Culture University, andnow he is a PhD student in the Department of ComputerScience and Information Engineering, National TaipeiUniversity of Technology in Taipei, Taiwan. Hisresearch interests are in the areas of smart environment,image processing and software architectureShingchern D. You received the Ph.D. degree in ElectricalEngineering from the University of California, Davis, CA,USA in 1993. Currently, he is associate professor in theDepartment of Computer Science and InformationEngineering in the National Taipei University ofTechnology, Taipei, Taiwan. Dr. You’s research interestsinclude audio signal processing and recognition, applieddigital signal processing to communication systems, and intelligent systems.Dwen-Ren Tsai received the PhD degree in ComputerScience from City University of New York in 1990. Hewas an Information System Assistance Professor atStockton State College, New Jersey, USA, from 1990 to1993. He has been an Associate Professor of ComputerScience at Chinese Culture University in Taiwan, since1993. He has also been the Chair of Computer ScienceDepartment of CCU since 1998. His areas of interestinclude computer security, software engineering, management informationsystem, and digital archiving.