Emotional facial expressions

emotions and facial expressions pdf and emotional facial expressions the role of anxiety and awareness
Dr.NaveenBansal Profile Pic
Published Date:25-10-2017
Your Website URL(Optional)
Influence of Induced Mood on the Rating of Emotional Valence and Intensity of Facial Expressions Evgeniya Hristova and Maurice Grinberg Department of Cognitive Science and Psychology, Research Center for Cognitive Science New Bulgarian University, Sofia, Bulgaria ehristovacogs.nbu.bg, mgrinbergnbu.bg Abstract. The current study investigates the influence of mood (sad, happy, or neutral) on the valence/intensity ratings of facial expressions of sad, happy, and neutral emotions. The study uses video clips for mood induction and color pho- tographs of emotional facial expressions. Under these conditions, the results show that participants give more extreme ratings to the emotion displayed (hap- py or sad) when in happy or sad mood without mood-congruence effects. This effect supports the conclusion that arousal alone may play a role in emotion va- lence/intensity rating (in contrast to results showing mood congruence in other tasks like emotion recognition and detection of emotional expression change). The explanation proposed in the paper is that experienced arousal might guide judgments about the intensity of emotions expressed by other people – when in a more aroused state, a person tends judge that other people also experience more intense emotions. Keywords: emotions, mood induction, emotional facial expressions, evaluation of emotions. 1 Introduction In everyday life, we constantly judge the emotions experienced by other people. One of the most informative cues in such judgments is the emotional facial expression. There is a great interest in studying how the information about the emotions in facial expressions is acquired – which facial features are most informative, what is the pat- tern of looking at these features, etc. There is a lot of research demonstrating that mood influences human cognitive processing. Affective states have an effect on judgments, decision, perception, and thinking 1-4. This research has shown that affective states influence judgments by shifting them towards the experienced emotion (e.g. when participants feel happy, they give more favorable judgments). Positive mood also influences memory, reason- ing, and visual perception: people tend to use more global processing when in happy mood (global processing is demonstrated by greater reliance on heuristics, scripts, and holistic strategies). On the other hand, people use more local features when they are sad. Schmid et al. 5 investigated the influence of the mood in emotion recognition tasks using eye-movement recordings. For the purpose, they induced either happy or © Springer International Publishing Switzerland 2015 303 S. Bassis et al. (eds.), Recent Advances of Neural Networks Models and Applications, Smart Innovation, Systems and Technologies 37, DOI: 10.1007/978-3-319-18164-6_29 304 E. Hristova and M. Grinberg sad mood and after that presented photos for emotion recognition task (happiness, sadness, anger, and fear). Results showed that participants in happy mood used more global processing styles than participants in negative mood. There are also studies with a special focus on the influence of mood on emotion recognition. Some of them 6, 7 have found that depressed patients demonstrate im- paired emotion recognition and also tend to provide more negative ratings. Schmid & Mast 8 manipulated the mood of healthy participants and studied how that manipu- lation changes emotion recognition. They found that the induced mood impaired the recognition of the mood-incongruent facial expressions. Other research 9 has dem- onstrated that participants with induced sad mood tend to perceive more sadness and less happiness in sad and happy faces, respectively. Also, sad and happy participants were more sensitive to and detected earlier changes in sad and happy facial expres- sions, respectively, when they were gradually changed from sad and happy to neutral or from sad to happy and happy to sad 10, 11. In the present study, we wanted to explore the influence of induced mood on emotion recognition in a new task – emo- tion rating – in experimental settings similar to the previously used in this type of research. We used mood induction using short movie episodes following successful and well established procedures 12, 13. Next, the task of the participants was to rate the valence/intensity of expressed emotion of human faces taken from the FACES database 14. The induced mood and the selected facial expressions were happy, neutral, and sad. The human faces were presented in relatively natural settings (in color, with hair and background) to make the task as close as possible to real situa- tions. In our opinion, this is a promising way of addressing directly some issues of interest like mood-congruity in emotion rating and like stronger ecological validity related to the presence of non-relevant features like hair and background. As will be discussed below, our results give evidence that the influence of mood on emotion rating may be due to arousal and not to mood-congruence as research using other tasks (e.g. emotional expression change) has previously suggested. 2 Goals and Hypothesis In the present study, the influence of the mood experienced by participants on their valence/intensity ratings of facial emotional expressions is explored. In this field of research, as the discussion in the previous section has shown, emo- tion recognition is naturally a central task of interest due to its importance in social interactions and its evolutionary value. On the other hand, in our opinion, the evalua- tion of the expressed emotion valence/intensity cannot be underestimated as not only the perceived emotion but also its intensity is an important factor for action and deci- sion making in a larger social context. While 8 and 5 used black and white photos and only the faces were presented (the stimuli were oval-shaped with hair and background removed), in the current study, we used color photographs without removal of any information (hair or back- ground). This is done in order to explore the emotion recognition in more natural Influence of Induced Mood on the Rating of Emotional Valence and Intensity 305 settings. As emotion recognition is a process that takes place in everyday life and its study in natural settings seems important to us. Therefore, our main goal is to look for possible congruency effects of the induced mood and valence/intensity rating of facial emotional expressions in relatively natural settings. More precisely, the study aims at exploring how participants’ ratings of emo- tional valence/intensity are affected by their mood. The hypothesis, based on research on emotion recognition discussed in the preceding section, is that valence/intensity ratings will be shifted towards the experienced mood, e.g. neutral emotional expres- sions will be rated as more negative in negative mood compared to positive mood. The second goal is to study how the mood influences visual information acquisi- tion. We hypothesize that negative mood will provoke more elaborate processing than positive mood which will lead to longer observation times in sad induced mood com- pared to happy induced mood, based on the differences in processing information discussed previously. 3 Method 3.1 Stimuli In the current study, the mood of the participants was manipulated between-subjects on 3 levels – happy, neutral, and sad. Short video-clips were used for mood induction. Meta-reviews demonstrate that this is one of the most effective techniques for inducing emotional states 15, 16. Clips, shown to elicit the target moods, are used based on 12, 13. Video-clips from the following movies were selected: from ‘When Harry met Sally’ (duration 2’ 35’’) for happy mood induction; from ‘Return to me’ (3’36’’) for sad mood induction; and from ‘Hannah and her sisters’ (1’30’’) for neutral mood induction. All video-clips are taken from movies with actors in English with Bulgarian subtitles. Three types of emotional facial expressions are used – happy, neutral, and sad – taken from the FACES database 14. Photographs from 9 female (IDs 20, 48, 54, 63, 71, 90, 115, 152, and 182) and 9 male (IDs 8, 13, 37, 57, 109, 114, 127, 147, and 153) actors are used, each actor presenting each of the emotional expressions. The stimuli were presented in 3 pseudo-randomized sequences, each consisting of 18 photo- graphs, with an equal number of faces with happy, neutral, and sad emotional expres- sions. A photograph of a given actor was used only once in a given sequence. In order to get the participants accustomed to the experimental procedure, in the beginning of each list, 2 additional photographs (1 with a male and 1 with a female actor) with neutral expressions are included (IDs 10 and 89). Each of these 3 presentation se- quences is preceded by a happy, neutral, or sad mood induction resulting in a total of 9 presentation conditions. 306 E. Hristova and M. Grinberg 3.2 Design and Procedure The study employs a 3×3 factorial design with induced mood (‘happy’ vs ‘neutral’ vs ‘sad’) as a between-subjects factor and facial emotion expression (‘happy’ vs ‘neutral’ vs ‘sad’) as a within-subjects factor. The video-clips were presented with the instruction to watch them for subsequent rating of liking. After the end of the video-clip, the participants were asked to rate their mood on a 9-point Likert scale ranging from ‘−4’ = ‘extremely sad’ to ‘+4’ = ‘extremely happy’. In order to avoid reactivity in the subsequent mood induction rat- ings, no specific instruction with regard to the mood induction was given before watching the video-clips. After the mood induction phase, each participant was presented with color photo- graphs of human faces with sad, neutral, or happy expressions in a self-paced presen- tation. For each stimulus the participant had to rate the emotional expression of the face in the photo on a 7-point Likert scale ranging from ‘−3’ = ‘very sad’ to ‘+3’ = ‘very happy’. The duration of the presentation of the emotional expressions was self- paced. The observation time was taken to be the time during which participants looked at each photograph. It was measured from the beginning of the stimulus pres- entation to the mouse click after which a screen with the rating scale appeared. Thus, in order to answer the main questions addressed in the paper, namely wheth- er there are differences in valence/intensity ratings depending on the induced mood and how the induced mood influences the observation time needed for making the rating, the following metrics are used: • mean emotional valence/intensity rating for the facial expression; • mean observation time. 3.3 Participants 93 participants (34 male, 59 female) took part in the experiment: 32 in the happy mood condition; 30 in the neutral mood condition; and 31 in the sad mood condition. Participants’ age ranged from 18 to 44 years (average 24 years). The participants were university students taking part in the study for partial fulfillment of course require- ments or voluntarily. 4 Results 4.1 Mood Induction First, a manipulation check was performed. The results demonstrate that the mood manipulation was successful (see Table 1) the average ratings given by the partici- pants about their mood for happy, neutral, and sad moods differ significantly (F(2, 90) = 87.99, p .001). Bonferoni post-hoc test shows that all differences are statistically significant at .001 level. Influence of Induced Mood on the Rating of Emotional Valence and Intensity 307 Table 1. Mood rating after the manipulation on a 9-point Likert scale (from ‘−4’ = 'very sad' to ‘+4’ = 'very happy’) Target Mood Mean SD Happy 2.8 1.2 Neutral 0.5 1.0 Sad −1.1 1.3 4.2 Face Emotion Ratings In Fig. 1, the ratings of facial emotion expression intensity depending on the induced mood are shown. Mean valence/intensity ratings are analyzed in a repeated-measures ANOVA with facial emotional expression (‘happy’ vs ‘neutral’ vs ‘sad’) as a within- subjects factor and 'induced mood' (‘happy’ vs ‘neutral’ vs ‘sad’) as a between- subjects factors. Fig. 1. Face emotion ratings with respect to induced participants' mood. There is a main effect of the facial emotional expression (F(2, 180) = 699.94, p .001): the mean ratings are 1.96 for happy emotional expressions, −0.18 for neu- tral emotional expressions, and −2.05 for sad emotional expressions, respectively (post =-hoc tests show that all differences are significant at .001 level, Bonferoni cor- rection applied). There is no main effect of induced mood on valence/intensity ratings (p =.98). There is an interaction between induced mood and facial emotional expres- sion (F(4, 180) = 13.3, p .001). To explore the interaction found, additional analyses were performed separately for happy and for sad faces. For the ratings of happy faces, there is a main effect of induced mood (F(2, 90) = 19.19, p .001). Bonferoni post-hoc test shows that there are significant differences between happy and neutral (p .001) and neutral and sad (p .001) moods, 308 E. Hristova and M. Grinberg respectively. Happy faces are rated as more happy by the participants in happy or sad mood compared to the participants in neutral mood (see Fig. 1). Similarly, for the ratings of sad faces, there is a main effect of induced mood (F(2, 90) = 7.95, p .001) and the ratings in neutral mood are significantly higher (less negative) than the ratings in happy (p .001) or sad mood (p .004). Sad faces are rated as more sad by the participants in happy or sad mood compared to the participants in neutral mood. 4.3 Observation Times The stimuli observation time is measured from the time of appearance of the picture until its disappearance after participants had pressed the mouse left button. The data was not normally distributed and a logarithmic transformation was applied. Mean observation times were analyzed with repeated-measures ANOVA with fa- cial emotional expression (happy vs. neutral vs. sad) as a within-subjects factor and 'induced mood' (happy vs. neutral vs. sad) as a between-subjects factors. When needed, Bonferoni correction for multiple comparisons was applied. There is a main effect of the facial emotional expression (F(2, 180) = 10.92, p .001). The observation times for the happy faces are shorter than the ones for neutral and sad faces (p .001 and p = .002, respectively). It takes less time for the partici- pants to rate a happy facial expression. The means of the observation times for each facial expression are shown in Fig. 2. There are no significant differences between the observation times with respect to induced mood, nor any interaction between induced mood and the emotional facial expressions of the photographs. Thus, no conclusion about differences in processing depending on induced mood can be drawn for valence/intensity rating. Fig. 2. Comparison of observation times for happy, neutral, and sad faces. 5 Discussion and Conclusion In the present study, the influence of induced mood on valence/intensity ratings of facial emotional expressions is studied. Based on the previous studies on emotion Influence of Induced Mood on the Rating of Emotional Valence and Intensity 309 recognition, the expectation was that mood-congruency effects will be observed. However, such effects were not found. Instead, higher valence/intensity ratings are obtained for both sad and happy emotional expressions for participants in both happy and sad induced mood. This led us to conclude that when rating emotional valence/intensity, participants are influenced by the arousal associated to the induced mood without depending on its specific type – happy or sad. As this is a novel effect, not reported in the literature to our knowledge, it requires a deeper and more systematic analysis. It might be the case that the specific settings of our experiment have led to lower sensitivity to the specific mood or emotion expression. One possibility could be that the emotions expressed by the actors in the photographs are too extreme and do not allow for noticeable influ- ences from the mood of the perceivers. But, this cannot explain the difference be- tween the ratings in neutral vs non-neutral mood. In order to understand better the observed effects, stimuli with more graded emotional expression should be used. Moreover, the valence/intensity rating scale used should be split in the usual valence, arousal, and dominance scales in order to be able to identify and measure finer effects and their origin. However, there are also good reasons to consider that the effect found is due to the influence of the arousal experienced by the participants. As the now classical study of Schachter & Singer 17 showed, physiological arousal is interpreted in the light of the environmental context in order determine the experienced emotion by the person her- self. In our view, experienced arousal could influence the judgments not only about the emotions experienced by the subject, but also the judgments of emotions experienced by others. For instance, when in a more aroused state, a person tends to perceive and judge that other people also experience more intense emotions. This is a novel explanation, which has not been proposed previously and deserves further experimental efforts. Our expectations about observation times, related to previous research, were not met either. They were based on the hypothesis of selective influence of the induced mood on information processing relating information acquisition in a happy mood to more global (fast) processing and in a sad mood to more local feature based (slow) one, respectively. In our experiment, the observation times in all moods were not significantly different with respect to the mood induced. The only significant result was that happy face expressions require shorter observation time. To draw more gen- eral conclusions about the dependence of processing on mood, more data is needed. In summary, the current study contributes to the field of perception of facial ex- pressions of emotions by using a new task – valence/intensity rating. An effect of the induced mood is found, namely, an increase in valence/intensity ratings when the induced mood is different from neutral independent of the non-neutral mood type (sad or happy). This new experimental finding and its theoretical implications need to be explored in experiments manipulating the level of arousal and its dependence on a variety of tasks related to emotion recognition. 310 E. Hristova and M. Grinberg References 1. Gasper, K., Clore, G.L.: Attending to the Big Picture: Mood and global verasus local processing of visual information. Psychological Science 13, 34–40 (2002) 2. Gasper, K.: Do you see what I see? Affect and visual information processing. Cognition and Emotion 18, 405–421 (2004) 3. Schwarz, N., Bless, H.: Happy and Mindless, But Sad and Smart? The Impact of Affect States on analytic reasoning. In: Forgas, J. (ed.) Emotion and Social Judgement, pp. 55–71. Pergamon, Oxford (1991) 4. Schwarz, N., Clore, G.L.: Mood, Misattribution, and Judgments of Well-Being: Informa- tive and Directive Functions of Affective States. Journal of Personality and Social Psy- chology 45, 513–523 (1983) 5. Schmid, P.C., Schmid Mast, M., Bombari, D., Mast, F.W., Lobmaier, J.S.: How Mood States Affect Information Processing During Facial Emotion Recognition: An Eye Track- ing Study. Swiss Journal of Psychology 70, 223–231 (2011) 6. Hale III, W.W.: Judgment of Facial Expressions and Depression Persistence. Psychiatry Research 80, 265–274 (1998) 7. Persad, S.M., Polivy, J.: Differences Between Depressed and Nondepressed Individuals in the Recognition of and Response to Facial Emotional Cues. Journal of Abnormal Psychol- ogy 102, 358–368 (1993) 8. Schmid, P.C., Mast, M.S.: Mood Effects on Emotion Recognition. Motivation and Emo- tion 34, 288–292 (2010) 9. Bouhuys, A.L., Bloem, G.M., Groothuis, T.G.: Induction of Depressed and Elated Mood by Music Influences the Perception of Facial Emotional Expressions in Healthy Subjects. Journal of Affective Disorders 33, 215–226 (1995) 10. Niedenthal, P.M., Halberstadt, J.B., Margolin, J., Innes-Ker, Å.H.: Emotional State and the Detection of Change in Facial Expression of Emotion. European Journal of Social Psy- chology 30, 211–222 (2000) 11. Niedenthal, P.M., Brauer, M., Halberstadt, J.B., Innes-Ker, Å.H.: When Did Her Smile Drop? Facial Mimicry and the Influences of Emotional State on the Detection of Change in Emotional Expression. Cognition & Emotion 15, 853–864 (2001) 12. Rottenberg, J., Ray, R.R., Gross, J.J.: Emotion Elicitation Using Films. In: Coan, J.A., Al- len, J.B. (eds.) Handbook of Emotion Elicitation and Assessment. Oxford Univ. Press, New York (2007) 13. Hewig, J., Hagemann, D., Seifert, J., Gollwitzer, M., Naumann, E., Bartussek, D.: A Re- vised Film Set for the Induction of Basic Emotions. Cognition & Emotion 19, 1095–1109 (2005) 14. Ebner, N.C., Riediger, M., Lindenberger, U.: FACES—A Database of Facial Expressions in Young, Middle-Aged, and Older Women and Men: Development and Validation. Beha- vior Research Methods 42, 351–362 (2010) 15. Westermann, R., Spies, K., Stahl, G., Hesse, F.W.: Relative Effectiveness and Validity of Mood Induction Procedures: A Meta-Analysis. European Journal of Social Psychology 26, 557–580 (1996) 16. Gerrards-Hesse, A., Spies, K., Hesse, F.W.: Experimental Inductions of Emotional states and Their Effectiveness: A Review. British Journal of Psychology 85, 55–78 (1994) 17. Schachter, S., Singer, J.: Cognitive, Social, and Physiological Determinants of Emotional State. Psychological Review 69, 379–399 (1962) A Multimodal Approach for Parkinson Disease Analysis 1 1 2 Marcos Faundez-Zanuy , Antonio Satue-Villar , Jiri Mekyska , 3,4 3 1 3 3,4 Viridiana Arreola , Pilar Sanz , Carles Paul , Luis Guirao , Mateu Serra , 4 3,4 1 1 Laia Rofes , Pere Clavé , Enric Sesa-Nogueras , and Josep Roure 1 Fundació Tecnocampus, Avda. Ernest Lluch 32, 08302 Mataró, Spain faundez,satue,paul,sesa,rouretecnocampus.cat 2 Brno University of Technology, Brno, Czech Republic 3 Hospital de Mataró, Consorci Sanitari del Maresme, Spain msanz,mserracsdm.cat 4 Centro de Investigación Biomedica en Red de Enfermedades Hepaticas y Digestivas. Barcelona, Spain laia.rofes,pere.claveciberehd.org Abstract. Parkinson’s disease (PD) is the second most frequent neurodegenerative disease with prevalence among general population reaching 0.1-1 %, and an annual incidence between 1.3-2.0/10000 inhabitants. The mean age at diagnosis of PD is 55 and most patients are between 50 and 80 years old. The most obvious symptoms are movement-related; these include tremor, rigidity, slowness of movement and walking difficulties. Frequently these are the symptoms that lead to the PD diagnoses. Later, thinking and behavioral problems may arise, and other symptoms include cognitive impairment and sensory, sleep and emotional problems. In this paper we will present an ongoing project that will evaluate if voice and handwriting analysis can be reliable predictors/indicators of swallowing and balance impairments in PD. An important advantage of voice and handwritten analysis is its low intrusiveness and easy implementation in clinical practice. Thus, if a significant correlation between these simple analyses and the gold standard video-fluoroscopic analysis will imply simpler and less stressing diagnostic test for the patients as well as the use of cheaper analysis systems. Keywords: Speech analysis, dysphagia, Parkinson disease, database. 1 Introduction In this study we will focus on three kinds of signals, but the first step will befocused on speech signals and dysphagia. It is based on a collaboration be-tween an engineering faculty and a Hospital. 1.1 Voice Analysis In the PD patient, dysphagia is usually accompanied by other oro-bucal symptoms such as hypokinetic dysarthria. Some studies have reported that the presence of both © Springer International Publishing Switzerland 2015 311 S. Bassis et al. (eds.), Recent Advances of Neural Networks Models and Applications, Smart Innovation, Systems and Technologies 37, DOI: 10.1007/978-3-319-18164-6_30 312 M. Faundez-Zanuy et al. symptoms usually correlates and that voice disorders could be anticipatory of swallowing impairment 1. Other studies concluded that a clear post-swallow voice quality provides reasonable evidence that penetration-aspiration and dysphagia are absent 2. Voice analysis is a safe, non-invasive, and reliable screening procedure for patients with dysphagia which can detect patients at high risk of clinically significant aspiration 6. The volume-viscosity swallow test (V-VST) was developed at the Hospital de Mataró to identify clinical signs of impaired efficacy (labial seal, oral and pharyngeal residue, and piecemeal deglutition) and impaired safety of swallow (voice changes, cough and decrease in oxygen saturation ≥3 %) 4. The V-VST allows quick, safe and accurate screening for oropharyngeal dysphagia (OD) in hospitalized and independently living patients with multiple etiologies. The V-VST presents a sensitivity of 88.2 % and a specificity of 64.7 % to detect clinical signs of impaired safety of swallow (aspiration or penetration). The test takes 5-10 min to complete and is an excellent tool to screen patients for OD. It combines good psychometric properties, a detailed and easy protocol designed to protect safety of patients, and valid end points to evaluate safety and efficacy of swallowing and detect silent aspirations 3. However, nowadays voice assessment is usually done by subjective parameters and a more exhaustive and objective evaluation is needed to understand its relationship with dysphagia and aspiration, as well as the potential relevance of voice disorders as a prognostic factor and disease severity marker. Hypokinetic dysarthria is a speech disorder usually seen in PD which affects mainly respiration, phonation, articulation and prosody. Festination is the tendency to speed up during repetitive movements. It appears with gait in order for sufferers to avoid falling down and also in handwriting and speech. Oral festination shares the same pathophysiology as gait disorders 7. Voice analysis allows the assessment of all these parameters and has been used to evaluate the improvement of PD after treatment 8-11,17. Voice impairments appear in early stages of the disease and may be a marker of OD even when swallow disorders are not clinically evident, which would allow to establish early measures to prevent aspiration and respiratory complications. Oropharyngeal dysphagia is a common condition in PD patients. In a recent meta-analysis, the prevalence of PD patients who perceive difficulty in swallowing was estimated at 35% but when an objective swallowing assessment was performed, the estimated prevalence of OD reached 82% 20. This underreporting calls for a proactive clinical approach to dysphagia, particularly in light of the serious clinical consequences associated to OD in these patients. Dysphagia can produce two types of severe complications; a) alterations in the efficacy of deglutition that may cause malnutrition or dehydration which may occur in up to 24% of PD patients 21, and b) impaired safety of swallow, which may lead to aspiration pneumonia with high mortality rates (up to 50%) 22-23. Aspiration pneumonia remains the leading cause of death among PD patients. 1.2 Balance and Falls Postural instability is one of the cardinal signs in PD. It becomes more prevalent and worsens with disease progression and represents one of the most disabling symptoms A Multimodal Approach for Parkinson Disease Analysis 313 in the advanced stages of PD, as it is associated with falls and loss of independence 12. Balance impairments represent a major burden with high impact on individual’s functional capacity, mobility, quality of life and survival. Overall, more than half of patients with PD experience falls. Falls are a major milestone in the evolution of PD because their severe consequences such as bone fractures or head injuries leading to disability, institutionalization and death 13. Most falls occur during posture changes and are unrelated to extrinsic factors, but are dependent upon intrinsic deficits of balance control. However, pathophysiology of balance disorders and postural instability in PD is not well understood. Posturography allows an objective assessment of balance parameters and posturographic studies have contributed to significant advances in understanding the pathophysiology of postural instability in PD, but it still remains to be fully clarified, partially due to the difficulty to distinguish between the disease process and the compensatory mechanisms and also due to the lack of standardized techniques to measure balance. Dopaminergic treatments can provide improvements in postural instability in early- to mid-stage of PD but the effects tend to decrease with time consistent with spread of the disease process to non-dopaminergic pathways. 1.3 Handwriting Handwriting skill degradation appears in early stages of PD so handwriting analysis is also of interest in the assessment of the disease progression. Alterations of central dopaminergic neurotransmissions adversely affect movement execution during handwriting and automatic execution of well-learned movements. Drawing exercises in a digitalized tablet allows the accurate evaluation and quantification of size, velocity, acceleration, stroke duration and other parameters of handwriting 18-19. Although beneficial effects of dopaminergic treatments in kinematics of handwriting movements have been reported, PD patients do not reach an undisturbed level of performance, suggesting that dopamine medication results in partial restoration of automatic movement execution 14-15. Some authors have shown altered parameters in PD as well as a recovery to the skill of a healthy person after medication with apomorphine 16. Handwriting tests are useful for assessing the effect of medication and for determining the dosage of drugs for a specific patient. 2 Dysphagia and Speech Analysis This multimodal analysis has started on speech signals in the context of dysphagia test. A large number of group of people suffer dysphagia, as summarized in table 1, as well as their effect. The medical term for any difficulty or discomfort when swallowing is dysphagia. A normal swallow takes place in four stages, and involves 25 different muscles and five different nerves. Difficulties at different stages cause different problems and symptoms. The four stages of swallowing are the following ones: 1. The sight, smell, or taste of food and drink triggers the production of saliva, so that when you put food in your mouth (usually voluntarily) there is extra fluid to make the process of chewing easier. 314 M. Faundez-Zanuy et al. 2. When the food is chewed enough to make a soft bolus, your tongue flips it towards the back of the mouth to the top of the tube, which leads down to your stomach. This part of your throat is called the pharynx. This part of swallowing is also voluntary. 3. Once the bolus of food reaches your pharynx, the swallowing process becomes automatic. Your voice box (the larynx) closes to prevent any food or liquid getting into the upper airways and lungs, making the food bolus ready to pass down your throat (known as the oesophagus). 4. The oesophagus, which is a tube with muscular walls that contract automatically, then propels the food down to the stomach. Table 1. Groups affected by dysphagia and its sympthoms Group of people Effects Elderly people 45 % find some difficulty in swallowing; 65 % of those living in residential or nursing homes: chewing and swallowing muscles are weaker, loss of teeth and saliva production reduced. Stroke sufferers 40 % nerves, muscles and cognitive/brain function affected Multiple sclerosis or Nervous system and muscles affected Parkinson’s disease Alzheimer’s disease Cognitive/brain function affected sufferers and severe depressives Motor Neuron Disease Nervous system, nerves and muscles affected People with cancer of the Nerves and muscles damaged by disease and throat and/or mouth treatment People with head and neck Nerves and muscles damaged injuries Some signs of dysphagia are: 1 Swallow repeatedly. 2 Cough and splutter frequently. 3 Voice is unusually husky and you often need to clear your throat. 4 When you try to eat you dribble. Food and saliva escape from your mouth or even your nose. 5 Find it 6 eep old food in your mouth, particularly when you have not had a chance easier to eat slowly. 7 Quite often kto get rid of it unseen. 8 Feel tired and lose weight. A Multimodal Approach for Parkinson Disease Analysis 315 2.1 Gold Standard for Dysphagia Analysis The courrent approach for dysphagia analysis has been developed by some of the medical authors of this paper, and can be summarized in figure 1. Process for dysphagia analysis based on three liquids of different viscosity and three different volumes per liquid. After swallowing each liquid and volume a word is pronounced by the patient and a speech therapist evaluates the voice quality in a subjective way (just listening to the speech signal). In those cases where a possible dysphagia problem exists, a videofluoroscopic analysis is perfomed. This diagnose is more invasive as it implies radiation, but it is the procedure to have physical evidence of swal-lowing problems. Figure 2 shows Videofluoroscopic pictures and oropharyngeal swallow response during the ingestion of a 5 mL nectar bolus in: (a) a healthy individual; (b) an older patient with neurogenic dysphagia and aspiration associated with stroke. An increased total duration of the swallow response may be seen, as well as a delayed closure of the laryngeal vestibule and delayed aperture of the upper sphincter. The white dot indi- cates the time when contrast penetrates into the laryngeal vestibule, and the red dot indicates passage into the tracheobronchial tree (aspiration). GPJ = glossopalatal junction, VPJ = velopalatal junction, LV= laryngeal vestibule, UES = upper esophageal sphincter. The main goal of the first step of this study is to evaluate if an automatic tool based on speech analysis can be developed to support medical decision during the test depicted in figure 1. Fig. 1. Process for dysphagia analysis based on three liquids of different viscosity and three different volumes per liquid 316 M. Faundez-Zanuy et al. Fig. 2. Videofluorescence images for (a) a healthy individual; (b) an older patient with neurogenic dysphagia and aspiration associated with stroke 3 Database Acquisition and Future Lines At this moment a speech database is being acquired in the protocol depicted in figure 1, one sample after deglutition of each liquid and volume. Thus, a total of 9 realizations per patient are acquired. Figure 3 shows the acquisition scenario at Mataro’s Hospital. The acquisition setup is based on a capacitor microphone Rode NT2000 (positioned at a distance of approximately 20 cm from the speaker’s mouth) and external sampling card (M- AUDIO, FAST TRACK PRO Interface audio 4x4) operating at 48 kHz sampling rate, 16 bit per sample, monophonic recording. Currently we are acquiring 3 patients per week. The signal processing approach, after database collection will be based on: (a) Voiced/unvoiced classification and then to check the harmonic to noise ratio (HNR) on the vowels, jitter, shimmer, etc. (b) To align the sample before and after eating using Dynamic Time Warping. The higher the distance between both realizations, the higher the probability to have deglutition problems. (c) Some complexity measures 3.1 Gold Standard Videofluoroscopy (VFS) is the gold standard to study the oral and pharyngeal mechanisms of dysphagia. VFS is a dynamic exploration that evaluates the safety and efficacy of deglutition, characterizes the alterations of deglutition in terms of videofluoroscopic signs, and helps to select and assess specific therapeutic strategies. Since the hypopharynx is full of contrast when the patient inhales after swallowing. Thereafter, VFS can determine whether aspiration is associated with impaired glossopalatal seal (predeglutitive aspiration), a delay in triggering the pharyngeal A Multimodal Approach for Parkinson Disease Analysis 317 swallow or impaired deglutitive airway protection (laryngeal elevation, epiglottic descent, and closure of vocal folds during swallow response), or an ineffective pharyngeal clearance (post swallowing aspiration) 5. Fig. 3. Acquisition scenario at Mataro’s Hospital Acknowledgement. This work has been supported by FEDER and Ministerio de ciencia e Innovación, TEC2012-38630-C04-03. The described research was performed in laboratories supported by the SIX project; the registration number CZ.1.05/2.1.00/03.0072, the operational program Research and Development for Innovation. References 1. Perez-Lloreta, S., Negre-Pages, L., Ojero-Senarda, A., et al.: Oro-buccal symptoms (dysphagia, dysarthria, and sialorrhea) in patients with Parkinson’s disease: preliminary analysis from the French COPARK cohort. European Journal of Neurology 19, 28–37 (2012) 2. Waito, A., Bailey, G.L., Molfenter, S.M., et al.: Voice-quality abnormalities as a sign of dysphagia: validation against acoustic and videofluoroscopic data. Dysphagia 26(2), 125–134 (2011) 3. Rofes, L., Arreola, V., Clavé, P.: The volume-viscosity swallow test for clinicalscreening of dysphagia and aspiration. Nestle Nutr. Inst. Workshop Ser. 72, 33–42 (1998), doi:10.1159/000339979; Epub September 24, 2012, PubMed PMID: 23051998 4. Clavé, P., Arreola, V., Romea, M., Medina, L., Palomera, E., Serra-Prat, M.: Accuracy ofthe volume-viscosity swallow test for clinical screening of oropharyngeal dysphagia and aspiration. Clin. Nutr. 27(6), 806–815 (2008), doi:10.1016/j.clnu.2008.06.011; Epub September 11, 2008, PubMed PMID: 18789561 5. Rofes, L., Arreola, V., Almirall, J., Cabré, M., Campins, L., García-Peris, P., Speyer, R., Clavé, P.: Diagnosis and management of oropharyngeal Dysphagia and its nutritional and respiratory complications in the elderly. Gastroenterol Res. Pract. 2011, 818979 (2011), doi:10.1155/2011/818979; Epub August 3, 2010, PubMed PMID: 20811545; PubMed Central PMCID: PMC2929516 318 M. Faundez-Zanuy et al. 6. Ryu, J.S., Park, S.R., Choi, K.: Prediction of laryngeal aspiration using voice analysis. Am. J. Phys. Med. Rehabil. 83(10), 753–757 (2004) 7. Moreau, C., Ozsancak, C., Blatt, J.-L., Derambure, P., Destee, A., Defebvre, L.: Oral Festination in Parkinson’s Disease: Biomechanical Analysis and Correlation with Festination and Freezing of Gait. Movement Disorders 22(10), 1503–1506 (2007) 8. Nagulic, M., Davidovic, J., Nagulic, I.: Parkinsonian voice acoustic analysis in real-time after stereotactic thalamotomy. Stereotact. Funct. Neurosurg. 83(2-3), 115–121 (2005) 9. Gobermana, A.M., Coelho, C.: Acoustic analysis of Parkinsonian speech I: Speech characteristics and L-Dopa therapy. NeuroRehabilitation 17, 237–246 (2002) 10. Gobermana, A.M., Coelho, C.: Acoustic analysis of Parkinsonian speech II: L-Dopa related fluctuations and methodological issues. NeuroRehabilitation 17, 247–254 (2002) 11. Stewart, C., Winfield, L., Junt, A., Bressman, S.B., Fahn, S., Blitzer, A., Brin, M.F.: Speech dysfunction in early Parkinson’s disease. Movement Disorders 10(5), 562–565 (1995) 12. Kim, S.D., Allen, N.E., Canning, C.G., et al.: Postural instability in patients with Parkinson’s disease. Epidemiology, pathophysiology and management. CNS Drugs 27, 97–112 (2013) 13. Grabli, D.: Normal and pathological gait: what we learn from Parkinson’s Disease. J. Neurol. Neurosurg. Psychiatry 83 (2012), doi:10.1136/jnmp-2012-302263 14. Tucha, O., Mecklinger, L., Thome, J., Reiter, A., Alders, G.L., Sartor, H., Naumann, M., Lange, K.W.: Kinematic analysis of dopaminergic effects on skilled handwriting movements in Parkinson’s disease. J. Neural Transm. 113, 609–623 (2006) 15. Tucha, O., Mecklinger, L., Walitza, S., Lange, K.W.: The effect of caffeine on handwriting movements in skilled writers. Human Movement Science 25(4-5), 523–535 (2006) 16. Eichhorn, T.E., Gasser, T., Mai, N., Marquardt, C., Arnold, G., Schwarz, J., Oertel, W.H.: Computational analysis of open loop handwriting movements in Parkinson’s disease: A rapid method to detect dopamimetic effects. Movement Disorders 11(3), 289–297 (1996) 17. Eliasova, I., Mekyska, J., Kostalova, M., Marecek, R., Smekal, Z., Rektorova, I.: Acoustic evaluation of short-term effects of repetitive transcranial magnetic stimulation on motor aspects of speech in Parkinson’s disease. J. Neural Transm. 120(4), 597–605 (2013) 18. Drotar, P., Mekyska, J., Smekal, Z., Rektorova, I., Masarova, L., Faundez-Zanuy, M.: Prediction potential of different handwriting tasks for diagnosis of Parkinson’s. In: E- Health and Bioengineering Conference (EHB), pp. 1–4 (2013) 19. Drotar, P., Mekyska, J., Rektorova, I., Masarova, L., Smekal, Z., Faundez-Zanuy, M.: A new modality for quantitative evaluation of Parkinson’s disease: In-air movement. In: 2013 IEEE 13th International Conference on Bioinformatics and Bioengineering (BIBE), pp. 1–4 (2013) 20. Kalf, J.G., et al.: Prevalence of oropharyngeal dysphagia in Parkinson’s disease: a meta- anaysis. Parkinson and Related Disorders 1, 311–315 (2012) 21. Sheard, J.M., et al.: Prevalence of malnutrition in Parkinsom’s disease: a systematic review. Nutrition Reviews 69, 520–532 (2011) 22. Fernandez, H.H., Lapane, K.: Predictors of mortality among nursing home residents with a diagnosis of Parkinson’s disease. Med. Sci. Monit. 8, CR241–CR246 (2002) 23. Williams-Gray, C.H., Mason, S.L., Evans, J.R., et al.: The CamPaIGN study of Parkinson’s disease: 10-year outlook in an incident population-based cohort. J. Neurol. Neurosurg. Psychiatry 84, 1258–1264 (2013) Are Emotions Reliable Predictors of Future Behavior? The Case of Guilt and Other Post-action Emotions Olimpia Matarazzo and Ivana Baldassarre Department of Psychology, Second University of Naples, Italy olimpia.matarazzo,ivana.baldassarreunina2.it Abstract. This study had two goals: 1) to establish the relative importance of the violation of a moral norm and of the damage done to another person in the genesis of guilt and other post-action emotions; 2) to investigate if the post- action emotions are reliable predictors of future behavior in conditions similar to the ones that elicited them the first time. Through the scenario technique, four typical antecedents of guilt were built, in which the intentionality of the norm violation and of the damage to others were manipulated. In all scenarios the protagonist acted in such a way as to elicit guilt and the other emotions that participants were asked to assess. Thus, we presented a similar situation hap- pening a few months later in which he had to choose whether to behave in the same way as he had behaved previously or in the opposite way. We expected that: (1) moral emotions would stimulate a different behavior from the previous one, whereas selfish emotions should lead to repeat the same behavior; (2) emo- tions should influence future behavior in an indirect way, through the cognitive mediation of thoughts preceding the decision; (3) the norm violation would have analogous relevance in eliciting guilt to harming another person. On the whole, the results corroborated our predictions, with some exceptions that were discussed. Keywords: emotion, guilt, post-action emotions, future behavior. 1 Introduction The idea that emotion plays a central role in decision-making and, more generally, in orienting future behavior is largely accepted in psychological literature e.g. 1,2,3. In particular, the “feeling for doing” approach 3 emphasizes the motivational function of emotions and their instrumental value in pursuing goals. According to this ap- proach, the variety of emotions experienced in daily life, each of them with its own peculiar experiential content, has the essential function of guiding behavior, even though the debate about the ways by which emotion influences behavior is open 4. The most widely accepted theory posits that emotion directly causes behavior and that its function is to lead the organism to behave in such a way as to deal with the emo- tional event e.g. 5,6. The competing theory 4, based on a dual-process model Corresponding author. © Springer International Publishing Switzerland 2015 319 S. Bassis et al. (eds.), Recent Advances of Neural Networks Models and Applications, Smart Innovation, Systems and Technologies 37, DOI: 10.1007/978-3-319-18164-6_31 320 O. Matarazzo and I. Baldassarre distinguishing between “automatic affect” – simple, fast and often not conscious – and “conscious emotion” – a more complex phenomenon entailing the awareness of subjective experience – argues that only the former shapes behavior directly, while emotion affects behavior indirectly, as a feedback system. According to this perspec- tive, conscious emotion influences cognitive processes which in turn affect decision making and behavior regulation 4. The main function of emotion is to anticipate the emotional outcome of future behavior on the basis of the “affective residue” left by past actions. Thus, emotion is able to enhance behaviors leading to desired outcomes and modify the ones leading to undesired outcomes. The paradigmatic example of this view are moral emotions (e.g. guilt, shame, pride, embarrassment), i.e. the emotions that are generated by the conformity with, or the transgression of, the norms that regulate behavior and feelings of the members of a society e.g. 7,8. Moral emotions are typically post-action emotions, in the sense that their insurgence provides feedback on the adequacy or inadequacy of actual be- havior and prompts individuals to reflect on their conduct and eventually envisage counterfactual courses of actions (which constitutes the cognitive experience of such emotions). However, the influence that moral emotions exert on decision making and on future behavior comes from the anticipatory feedback of its emotional conse- quences 8,9. More precisely, it is the goal to avoid or repeat previous emotional experiences that motivates the choice between adopting or preempting a course of actions similar to the one that had generated those emotions. Perhaps the most prototypical moral emotion (at least in Western culture) is guilt, as evidenced by the considerable debate about its nature see for review 9, 10. Ac- cording to the intrapsychic approach e.g. 11,12, guilt is elicited by the violation of internalized moral norms and motivates self-punishing or reparative behavior. Ac- cording to the interpersonal approach e.g. 13-15, guilt is rooted in empathic concern for the well-being of others and is elicited by causing harm to another person, delibe- rately or unintentionally: thus, guilt-based behavior aims to restore the relationship with the damaged person. Along the empathic axis lies one of the main lines of demarcation between guilt and another typical moral emotion, shame. While guilt is usually seen as an adaptive emotion, aiming at re-establishing potentially broken relationships, shame is envi- saged as a maladaptive emotion tending to reduce rather than restore interpersonal relationships 8. So, guilt motivates pro-social and reparative behavior 8,16, while shame motivates withdrawal, even if some findings reveal that shame can also induce restorative behavior 17. It is noteworthy that empirical research has scarcely corro- borated the traditional distinctions between guilt and shame based on transgression of moral norms vs. social or communitarian values, and on private vs. public nature of transgression 8,18. The differences between the two emotions are rather envisaged in their phenomenological experience (guilt would be centered on the action, while shame would be centered on the entire self), in their level of pervasiveness (shame would be more pervasive than guilt) and in the behavior they motivate (other- or rela- tionship-preserving vs. self-preserving) 8,12. In our view, however, the literature on moral emotions, in particular on guilt, has emphasized the interpersonal nature of this emotion, based on the empathic concern for others, and has neglected the intrapsychic Are Emotions Reliable Predictors of Future Behavior? 321 point of view, according to which guilt arises from the transgression of a moral norm, regardless of the harm caused to others. To investigate the respective importance of interpersonal and intrapsychic component in the genesis of guilt (and other moral emotions) is one of the goals of this study, the other being to determine whether the emotions that arise after an action are good predictors of future behavior. 2 Overview of the Experiment This study aimed at pursuing two goals: 1) to compare the two main approaches about the intrapsychic vs. interpersonal nature of guilt; 2) to investigate whether and to what extent the emotions felt after an action can be reliable predictors of future behavior, in conformity with the hypothesis of the “feeling for doing” approach. As to the first goal, the way chosen to compare the two approaches was to assess the intensity of guilt elicited by the typical situational antecedents of this emotion. To this end, through the scenario technique, we built four antecedents of guilt by varying its struc- tural determinants: the norm violation and the harm to another. The third determinant we varied was the presence/absence of material advantage the male protagonist could gain from his actions. We created situations similar to those of daily life and capable of eliciting a wide range of emotions, other than guilt (relief, shame, anger toward oneself, satisfaction, sorrow), in order to assess their reciprocal relations. More specif- ically, since in three of our scenarios the protagonist had to face a dilemma concern- ing the choice between moral vs. selfish options and preferred the selfish one, we selected relief and satisfaction to assess the amount of positive emotional reaction generated by the decision to pursue his own material concerns even at the expense of moral values. Sorrow was chosen to gauge the broad range of emotional negative states deriving from hurting another person or from an incorrect action. Anger toward oneself was selected to evaluate the amount of aggressive auto-directed reaction asso- ciated with the awareness of having committed wrongdoing or unjust actions. Finally, shame was included in this study because it represents one of the two main moral emotions (the other being guilt): in order to contribute to the debate on the distinction between guilt and shame, we examined the extent to which these emotions were eli- cited by undiscovered breaches of moral norms and/or by damage caused to other people. We expected that the norm violation would have a similar relevance in elicit- ing guilt to harming another person, and that both antecedents would be more specific of guilt rather than shame (and other emotions). In order to pursue the second goal of this study, we subdivided the experiment into two parts: at time 1, we presented a situation in which the protagonist acted in a way capable of eliciting guilt and/or the other envisaged emotions; at time 2, we presented a situation similar to that of time 1 in which the protagonist had to choose whether to behave in the same way as he did the first time or in the opposite way. We expected that negative and moral emotions (anger toward oneself, sorrow, guilt, and shame) would stimulate a different behavior from the previous one, whereas positive selfish emotions (relief and satisfaction) should lead to repeat the same behavior. In confor- mity with Baumeister et al.’s 4 model, we expected that emotions should influence future behavior in an indirect way, through the cognitive mediation of thoughts preceding the decision. 322 O. Matarazzo and I. Baldassarre 3 Method Participants, Materials and Procedure. Two hundred undergraduates, aged from 18 to 30 years (Mean = 25.46; SD = 2.58), participated in this study, as unpaid volun- teers. No Psychology student was recruited so that prior knowledge about emotion would not influence the results. Each participant was randomly assigned to one of the 4 experimental conditions, except for gender for which they were paired. By means of the scenario technique, four typical antecedents of guilt were built, in which the intentionality of the norm violation and of the damage to others were mani- pulated. The third manipulated variable was the presence/absence of the material self- advantage. In the first scenario (henceforth S1) the protagonist deliberately violated a moral norm to gain an advantage for himself but without causing harm to others; in the second scenario (S2) the protagonist deliberately violated a moral norm and caused harm to another person to gain an advantage for himself; in the third scenario (S3) the protagonist deliberately violated a moral norm by causing harm to another person without the aim of gaining any material advantage for himself; in the fourth scenario (S4) the protagonist unintentionally caused harm to a friend: so, no moral norm was violated nor material advantage was pursued. While in the first three scena- rios the protagonist had to reach a decision through facing a moral dilemma, in S4 the damage was unintentional: so, no moral dilemma was posed. In all scenarios the na- ture and seriousness of the norm violation, of the damage to others, and of material self-advantage (if present) were kept constant so as to eliminate or reduce spurious effects due to these contingent variables. After reading the scenario, participants were asked to identify with the protagonist and to specify on a 9-point scale (1= not at all, 9 = extremely) the extent to which he felt the following 6 emotions: satisfaction, anger toward oneself, shame, relief, sorrow, guilt (Time 1). Participants were then asked to read a second scenario where, a few months later (Time 2), a situation similar to the initial one occurred. In the moral dilemmas, the protagonist had to decide whether to behave in a similar way to the previous one or in an opposite way. In the situation of unintentional damage, he has to decide whether to adopt precautionary measures to avoid the possibility of causing harm to another person or not. After the scenario, the protagonist’s two possible thoughts (moral/altruistic – henceforth moral - vs. selfish) and two choice options (moral vs. selfish) were presented. Participants were asked to evaluate, on a 9-point scale (1 = strongly disagrees, 9 = extremely agrees) the prota- gonist’s degree of agreement with each of the two thoughts and to predict his choice between two options: moral vs. selfish. The order of thoughts, types of choice, and 1 emotions was randomized. 4 Results In order to test the effect of the four scenarios on the six emotions, a mixed ANCOVA was performed, with the scenario as between variable and emotion as within variable; 1 The scenarios and the instructions were pre-tested by means of a pilot study with 32 undergraduates (8 for each experimental condition).