Affective Medicine

Summary

Background: Affective computing (AC) is concerned with emotional interactions performed with and through computers. It is defined as “computing that relates to, arises from, or deliberately influences emotions”.

AC enables investigation and understanding of the relation between human emotions and health as well as application of assistive and useful technologies in the medical domain.

Objectives: 1) To review the general state of the art in AC and its applications in medicine, and 2) to establish synergies between the research communities of AC and medical informatics.

Methods: Aspects related to the human affective state as a determinant of the human health are discussed, coupled with an illustration of significant AC research and related literature output. Moreover, affective communication channels are described and their range of application fields is explored through illustrative examples.

Results: The presented conferences, European research projects and research publications illustrate the recent increase of interest in the AC area by the medical community. Tele-home healthcare, Am I, ubiquitous monitoring, e-learning and virtual communities with emotionally expressive characters for elderly or impaired people are few areas where the potential of AC has been realized and applications have emerged.

Conclusions: A number of gaps can potentially be overcome through the synergy of AC and medical informatics. The application of AC technologies parallels the advancement of the existing state of the art and the introduction of new methods. The amount of work and projects reviewed in this paper witness an ambitious and optimistic synergetic future of the affective medicine field.

Keywords

Affective computing - emotions - health - medicine - human-computer interaction - learning - autism - biosensor networks

• References

• 1 Picard RW. Affective Computing. 1st ed. The MIT Press; 1997
• 2 Cacioppo J. Introduction: Emotion and Health. In: Handbook of Affective Stress. Oxford University Press; 2003
• 3 Picard RW. Affective medicine: Technology with emotional intelligence. Future of Health Technology 2001 pp 69-85.
• 4 Webster C. Medical affective computing: medical informatics meets affective computing. Stud Health Technol Inform 1998 p 52.
• 5 Smith RN, Frawley WJ. Affective Computing: Medical Applications. In: Proceedings of HCI International (the 8th International Conference on Human-Computer Interaction): Ergonomics and User Interfaces. L. Erlbaum Associates Inc.; 1999. pp 843-847.
• 6 Prendinger H, Ishizuka M. What Affective Computing and Life-Like Character Technology Can Do for Tele-Home Health Care. In: Workshop on HCI and Homecare: Connecting Families and Clinicians (Online Proceedings). in conj. with CHI; 2004
• 7 Bickmore TW, Picard RW. Establishing and maintaining long-term human-computer relationships. ACM Trans Comput-Hum Interact 2005; 12 (02) 293-327.
• 8 Glaser R, Pearl DK, Kiecolt-Glaser JK, Malarkey WB. Plasma cortisol levels and reactivation of latent Epstein-Barr virus in response to examination stress. Psychoneuroendocrinology 1994; 19 (08) 765-772.
• 9 Cohen S, Doyle WJ, Skoner DP, Rabin BS, Gwaltney JM. Social ties and susceptibility to the common cold. JAMA: The Journal of the American Medical Association 1997; 277 (24) 1940-1944.
• 10 Burg M. Stress, Behavior, and Heart Disease. In: Heart Book. Yale University School of Medicine 2002
• 11 Paulhaus D, Fridhandler B, Hayes S. Psychological Defense: Contemporary Theory and Research. In: Handbook of Personality Psychology. San Diego: Academic Press; 1997
• 12 Vaillant GE. Ego mechanisms of defense and personality psychopathology. J Abnorm Psychol 1994; 103 (01) 44-50.
• 13 O’Leary A. Self-efficacy and health: Behavioural and stress-physiological mediation. Cognitive Therapy and Research 1992; 16 (02) 229-245.
• 14 Scheier MF, Magovern G, Abbot RA, Matthews KA, Owens JF, Lefembre R, Carver CS. Dispositional optimism and recovery from coronary artery bypass surgery: the beneficial effects on physical and psychological well-being. Journal of personality and social psychology 1989; 57 (06) 1024-1040.
• 15 Wiebbe DJ, Williams PG. Hardiness and health: a social psychophysiological perspective on stress and adaptation. Journal of social and clinical psychology 1992; 11 (03) 238-262.
• 16 Stone AA, Cox DS, Valdimarsdottir H, Jandorf L, Neale JM. Evidence that secretory IgA antibody is associated with daily mood. J Pers Soc Psychol 1987; 52 (05) 988-993.
• 17 Middleton R, Byrd E. Psychosocial Factors and Hospital Readmission Status of Older Persons with Cardiovascular Disease. Journal of Applied Rehabilitation Counseling 1996 pp 273-210.
• 18 Martin LR, Friedman HS, Tucker JS, Tomlinson-Keasey C, Criqui MH, Schwartz JE. A life course perspective on childhood cheerfulness and its relation to mortality risk. Personality and Social Psychology Bulletin 2002; 28 (09) 1155.
• 19 Kimonides VG, Spillantini MG, Sofroniew MV, Fawcett JW, Herbert J. Dehydroepiandrosterone antagonizes the neurotoxic effects of corticosterone and translocation of stress-activated protein kinase 3 in hippocampal primary cultures. Neuroscience 1999; 89 (02) 429-436.
• 20 Eysenck MW. Anxiety and cognition: A unified theory. Hove, UK: Psychology Press; 1997
• 21 Fox E, Russo R, Bowles RJ, Dutton K. Do threatening stimuli draw or hold visual attention in sub-clinical anxiety?. Journal of Experimental Psychology: General, 2001; 130 (04) 681-700.
• 22 Pennebaker JW, Mehl MR, Niederhoffer KG. Psychological aspects of natural language use: Our words, our selves. Annual review of psychology 2003; 54 (01) 547-577.
• 23 Liu H, Lieberman H, Selker T. A model of textual affect sensing using real-world knowledge. In: Proceedings of the 8th international conference on intelligent user interfaces. Miami, Florida, USA: ACM; 2003. pp 125-132.
• 24 Fellbaum C. WordNet An electronic lexical database. Cambridge, MA: MIT Press; 1998
• 25 Valitutti A, Strapparava C, Stock O. Developing affective lexical resources. Psychology Journal 2004; 2 (01) 61-83.
• 26 Kim S, Hovy E. Identifying and analyzing judgment opinions. In: Proceedings of the main conference on Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics. New York, 2006. New York: Association for Computational Linguistics; 2006. pp 200-207.
• 27 Whissell C. The dictionary of affect in language. The Measurement of Emotions: Theory, Research and Experience. 1989 p 113.
• 28 Subasic P, Huettner A. Affect analysis of text using fuzzy semantic typing. IEEE Transactions on Fuzzy Systems 2001; 9 (04) 483-496.
• 29 Nasukawa T, Yi J. Sentiment analysis: capturing favorability using natural language processing. In: Proceedings of the 2nd International Conference on Knowledge Capture. 2003. Sanibel Island, FL, USA: 2003. pp 70-77.
• 30 Dellaert F. Recognizing emotion in speech. In: Proceedings of the ICSLP ’96. 1996
• 31 Petrushin VA. Emotion in speech: Recognition and application to call centers. In: Engr. 1999. pp 7-10.
• 32 Yu F, Chang E, Xu Y, Shum H. Emotion Detection from Speech to Enrich Multimedia Content. In: Advances in Multimedia Information Processing – PCM’01. 2001. pp 550-557.
• 33 Yacoub S, Simske S, Lin X, Burns J. Recognition of emotions in interactive voice response systems. In: Proc Eurospeech. Geneva: 2003. pp 1-4.
• 34 Neiberg D, Elenius K, Laskowski K. Emotion recognition in spontaneous speech using GMMs. In: Proc of Interspeech 2006. Pittsburgh: 2006
• 35 Roy D, Pentland A. Automatic spoken affect classification and analysis. In: Proceedings of the 2nd International Conference on Automatic Face and Gesture Recognition (FG ’96). IEEE Computer Society; 1996. p 363.
• 36 Montero JM, Gutierrez-Arriola JM, Palazuelos S, Enriquez E, Aguilera S, Pardo JM. Emotional speech synthesis: from speech database to TTS. In: Fifth International Conference on Spoken Language Processing. ISCA; 1998
• 37 Zovato E, Pacchiotti A, Quazza S, Sandri S. Towards emotional speech synthesis: a rule based approach. In: Fifth ISCA Workshop on Speech Synthesis. ISCA; 2004
• 38 Burkhardt F, Sendlmeier WF. Verification of acoustical correlates of emotional speech using formant-synthesis. In: ISCA Tutorial and Research Workshop (ITRW) on Speech and Emotion. ISCA; 2000
• 39 Edgington M. Investigating the limitations of concatenative synthesis. In: Fifth European Conference on Speech Communication and Technology. 1997
• 40 Heuft B, Portele T, Rauth M. Emotions in time domain synthesis. In: Proceedings of the Fourth International Conference on Spoken Language – ICSLP’96; 1996; 3: 1974-1977.
• 41 Reilly J, Klima ES, Bellugi U. Once more with feeling: affect and language in atypical populations. Development and Psychopathology 2008; 2 (04) 367-391.
• 42 Gillberg C. Asperger syndrome and high-functioning autism (review). British Journal of Psychiatry 1998; 172200-172209.
• 43 Dara C, Monetta L, Pell MD. Vocal emotion processing in Parkinson’s disease: reduced sensitivity to negative emotions. Brain Research 2008; 1188: 100-111.
• 44 Ekman P. Emotion in the human face. Cambridge University Press; 1982
• 45 Ekman P, Friesen WV. Facial Action Coding System. Consulting Psychologist Press; 1977
• 46 Lisetti CL, Schiano DJ. Automatic facial expression interpretation: where human-computer interaction, artificial intelligence and cognitive science intersect. Perspective 2000; 8 (01) 185-235.
• 47 Cohn JF, Kanade T. Use of automated facial image analysis for measurement of emotion expression. Handbook of Emotion Elicitation and Assessment; 2007
• 48 Kanade T, Cohn JF, Tian YL. Comprehensive database for facial expression analysis. In: Proceedings of the Fourth IEEE International Conference on Automatic Face and Gesture Recognition (FG’00). Grenoble, France: 2000. pp 46-53.
• 49 Schmidt K, Cohn JF, Tian YL. Signal characteristics of spontaneous facial expressions: Automatic movement in solitary and social smiles. Biological Psychology 2003; 65: 49-66.
• 50 Cohn JF, Schmidt KL. The timing of facial motion in posed and spontaneous smiles. International Journal of Wavelets, Multiresolution and Information Processing 2004; 2: 1-12.
• 51 Cootes TF, Edwards GJ, Taylor CJ. Active appearance models. IEEE Transactions on Pattern Analysis and Machine Intelligence; 2001 pp 681-685.
• 52 Wen Z, Huang TS. Capturing subtle facial motions in 3D face tracking. International Conference on Computer Vision, Nice, France; 2003 pp 1343-1350.
• 53 Valstar MF, Pantic M, Patras I. Motion history for facial action detection in video. In: Proceedings of the IEEE Conference on Systems, Man, and Cybernetics. the Hague, the Netherlands: 2004. pp 635-640.
• 54 Cassell J, Sullivan J, Prevost S, Churchill EF. Embodied conversational agents. Cambridge, MA: MIT Press; 2000
• 55 Bailenson J, Beall A, Blascovich J, Raimundo M, Weishbush M. Intelligent agents who wear your face: user’s reactions to the virtual self. Transfer Report. Center for the Virtual Environment and Behaviors Department of Psychology, University of California Santa Barbara: 2000
• 56 George P, McIllhagga M. The communication of meaningful emotional information for children interacting with virtual actors. In: Affective interactions: towards a new generation of computer interfaces. Springer-Verlag; 2000. pp 35-48.
• 57 Hone K. Empathic agents to reduce user frustration: The effects of varying agent characteristics. Interact Comput 2006; 18 (02) 227-245.
• 58 Moore D, Cheng Y, McGrath P, Powell NJ. Collaborative virtual environment technology for people with autism. Focus on Autism and Other Developmental Disabilities 2005; 20 (04) 231.
• 59 Luneski A, Konstantinidis E, Hitoglou-Antoniadou M, Bamidis PD. Affective computer-aided learning platform for autistic children. In: Proceedings of the 1st Workshop on Child, Computer and Interaction. Crete, Greece: 2008
• 60 Woodworth M, Waschbusch D. Emotional processing in children with conduct problems and callous/ unemotional traits. Child Care Health Dev 2008; 34 (02) 234-244.
• 61 Shiffrar M, Pinto J. Visual analysis of bodily motion. In: Common mechanisms in perception and action: Attention and Performance. Oxford: Oxford University Press; 2002
• 62 Burgoon JK, Jensen ML, Meservy TO, Kruse J, Nunamaker JF. Augmenting human identification of emotional states in video. In: Proceedings of the International Conference on Intelligent Data Analysis. 2005
• 63 Bianchi-Berthouze N, Kleinsmith A. A categorical approach to affective gesture recognition. Connection Science 2003; 15 (04) 259-269.
• 64 Kapur A, Kapur A, Virji-Babul N, Tzanekatis G, Driessen PF. Gesture-based affective computing on motion capture data. In: Affective computing and intelligent interaction – ACII 2005. Springer Verlag; 2005
• 65 Bernhardt D, Robinson P. Detecting Affect from Non-stylised Body Motions. In: Proceedings of the 2nd international conference on Affective Computing and Intelligent Interaction. Lisbon, Portugal: Springer-Verlag; 2007. pp 59-70.
• 66 Castellano G, Kessous L, Caridakis G. Emotion Recognition through Multiple Modalities: Face, Body Gesture, Speech. In: Affect and Emotion in Human-Computer Interaction. 2008. pp 92-103.
• 67 Balomenos T, Raouzaiou A, Ioannou S, Drosopoulos A, Karpouzis K, Kollias S. Emotion analysis in man-machine interaction systems. Machine Learning for Multimodal Interaction 2004 pp 318-328.
• 68 Norman DA. The Invisible Computer. Cambridge, MA: MIT Press; 1999
• 69 Healey JA, Picard RW. Detecting stress during real-world driving tasks using physiological sensors. IEEE Transactions on Intelligent Transportation Systems 2005; 6 (02) 156-166.
• 70 Papadelis C, Chen Z, Kourtidou-Papadeli C, Bamidis PD, Chouvarda I, Bekiaris E, Maglaveras N. Monitoring sleepiness with on-board electrophysiological recordings for preventing sleep- deprived traffic accidents. Clin Neurophysiol 2007; 118 (09) 1906-1922.
• 71 Skinner MJ, Simpson PA. Workload issues in military tactical airlift. The International Journal of Aviation Psychology 2002; 12 (01) 79-93.
• 72 Andreassi JL. Psychophysiology: Human Behavior and Physiological Response. 5th ed. Lawrence Erlbaum Associates; 2000
• 73 Haag A, Goronzy S, Schaich P, Williams J. Emotion recognition using bio-sensors: First steps towards an automatic system. Lecture Notes in Computer Science 2004; 306: 836-848.
• 74 Nasoz F, Alvarez K, Lisetti L, Finkelstein N. Emotion recognition from physiological signals using wireless sensors for presence technologies. Cogn Technol Work 2004; 6 (01) 4-14.
• 75 Picard RW, Vyzas E, Healey J. Toward machine emotional intelligence: analysis of affective physiological state. IEEE transactions on pattern analysis and machine intelligence 2001; 23 (10) 1175-1191.
• 76 Bamidis PD, Frantzidis C, Konstantinidis E, Luneski A. An Integrated Approach to Emotion Recognition for Advanced Emotional Intelligence. San Diego: 2009
• 77 Wagner J, Kim J, Andre E. From physiological signals to emotions: Implementing and comparing selected methods for feature extraction and classification. In: IEEE International Conference on Multimedia and Expo. 2005. ICME 2005. 2005. pp 940-943.
• 78 Lisetti C, Nasoz F, LeRouge C, Ozyer O, Alvarez K. Developing multimodal intelligent affective interfaces for tele-home health care. Int J Hum-Comput Stud 2003; 59 1–2 245-255.
• 79 Barreto A, Zhai J, Adjouadi M. Non-intrusive physiological monitoring for automated stress detection in human-computer interaction. In: Human-Computer Interaction. 2007. pp 29-38.
• 80 Paiva A, Picard R, Aylett R, Kolias S, Ortony A, Tao J, Prada R. Preface. In: Paiva A, Prada R, Picard R. editors. Proceedings of the 2nd International Conference on Affective Computing and Intelligent Interaction (ACII 2007). 2007. Berlin: Springer-Verlag; 2007
• 81 Van Bemmel J, Musen MA. editors. Handbook of medical informatics. Houten: Springer-Verlag; 1997
• 82 e-Health trends.. http://ec.europa.eu/information_society/ehealth. Last access May 10, 2009
• 83 Maojo V, Kulikowski CA. Bioinformatics and medical informatics: collaborations on the road to genomic medicine?. JAMIA 2003; 10 (06) 515-522.
• 84 Riva G. AmI in health care. CyberPsychology & Behavior 2003; 6 (03) 295-300.
• 85 Alcaniz M, Rey B. New technologies for Am I. In: AmI: The Evolution of Technology, Communication and Cognition towards the Future of HCI. 2005. pp 3-16.
• 86 Cearreta I, Lopez J, Hernandez C, Garay N, Grana M, Alvarez A. Affective computing as a component of Am I. In: Proceedings of the 10th Joint Conference on Information Sciences 2007. Salt Lake City, Utah, USA: 2007. pp 1580-1586.
• 87 Zhou J, Yu C, Riekki J, Karkkainen E. AmE framework: a model for emotion-aware Am I. In: Proceedings of the 2nd International Conference on Affective Computing and Intelligent Interaction (ACII 2007); 2007. Berlin: Springer-Verlag; 2007
• 88 Ducatel K, Bogdanowicz M, Scapolo F, Leijten J, Burgelman J. ISTAG Scenarios for AmI in 2010; 2001. http://cordis.europa.eu/ist/istag-reports.htm.
• 89 Riva G, Vatalaro F, Davide F, Alaniz M. AmI: The Evolution of Technology, Communication and Cognition towards the Future of HCI. IOS Press; Fairfax: 2005
• 90 Crist TM, Kaufman SB, Crampton KR. Home tele-medicine: a home health care agency strategy for maximizing resources. Home Health Care Management Practice 1996; 8 (04) 1-9.
• 91 Kenner AN. The effect of task differences, attention and personality on the frequency of body-focused hand movements. Journal of Nonverbal Behavior 1984; 8 (03) 159-171.
• 92 Alonso MB, Varkevisser M, Hekkert P, Keyson D. Exploring manipulative hand movements during a stressful condition. In: Proceedings of the 2nd Affective Computing and Intelligent Interaction conference (ACII 2007). 2007. pp 757-758.
• 93 Liu KKL. A personal, mobile system for understanding stress and interruptions. Cambridge: MIT; 2004
• 94 Healey J, Picard R. Smartcar: detecting driver stress. In: Proceedings of the 15th International Conference on Pattern Recognition. 2000
• 95 Metaxas D, Venkataraman S, Vogler C. Image-based stress recognition using a model-based dynamic face tracking system. In: Computational Science – ICCS 2004. 2004. pp 813-821.
• 96 Worgan S, Moore RK. Enabling reinforcement learning for open dialogue systems through speech stress detection. In: The Fourth International Workshop on Human-Computer Conversation. Bellagio, Italy: 2008
• 97 Hook C. A holistic approach to ubiquitous stress management. In: Proceedings of UbiComp. 2009. (in press).
• 98 Fabri M, Moore D. The use of emotionally expressive avatars in collaborative virtual environments. In: Proceeding of Symposium on Empathic Interaction with Synthetic Characters, at Artificial Intelligence and Social Behaviour Convention 2005. 2005
• 99 Murray D. Autism and information technology: therapy with computers. Autism and learning: a guide to good practice. 1997 pp 100-117.
• 100 Enyon A. Computer interaction: an update on the avatar program. Communication. 1997: 18.
• 101 Eddon G. Danny’s rooms. In: Proceedings of the Johns Hopkins National Search for Computing Applications to Assist Persons with Disabilities. IEEE Computing Society Press; 1992
• 102 Green SJ. Computer-Based Simulations in the Education and Assessment of Autistic Children. In: Rethinking the Roles of Technology in Education. Cambridge, MA: Massachusetts Institute of Technology; 1993. pp 334-336.
• 103 Luneski A, Konstantinidis E, Hitoglou-Antoniadou M, Bamidis PD. Affective Computer-Aided Learning Platform for Autistic Children. Chania, Crete, Greece: 2008
• 104 Kaliouby R, Picard R, Barron-Cohen S. Affective computing and autism. Annals of the New York Academy of Sciences. 2006; 1093 1 Progress in Convergence: Technologies for Human Well-being 228-248.
• 105 Blocher K, Picard RW. Affective social quest: emotion recognition therapy for autistic children. Socially Intelligent Agents: Creating Relationships with Computers and Robots. 133-140.
• 106 Frantzidis C, Bamidis PD. Description and future trends of ICT solutions offered towards independent living: the case of the LLM project. In: Proceedings of the 2nd ACM International Conference on Pervasive Technologies Related to Assistive Environments (PETRA 2009). Greece: 2009. In press.
• 107 Brain Fitness.. http://www.brainfitness.com. Last access May 12, 2009
• 108 Knaup P, Bott O, Kohl C, Lovis C, Garde S. Electronic patient records: moving from islands and bridges towards electronic health records for continuity of care. Meth Inf Med 2007; 46: 31-46.
• 109 Meystre S. Electronic patient records: some answers to the data representation and reuse challenges. Findings from the section on patient records. In: Geissbuhler A, Haux R, Kulikowski C. editors. IMIA Yearbook of Medical Informatics 2007. Methods Inf Med 2007; 46 (01) 47-49.
• 110 Bamidis PD, Papadelis C, Kourtidou-Papadeli C, Pappas C, Vivas A. Affective computing in the era of contemporary neurophysiology and health informatics. Interacting with Computers 2004; 16 (04) 715-721.