1. Introduction
Humans are social creatures who can hardly survive on their own. The everyday life
of
most people consists of a multitude of interactions with our fellow human beings.
These interactions do not only transfer factual information, but also reveal
information about the sender (self-revelation level), the relationship of the
interactors (relationship level), and information about what we are trying to
achieve from our communication partner (appeal level). Except for the factual
information level, sending and perceiving emotions is crucial for successful
communication. Emotion perception is helpful for contextual understanding of
information, interpretation of others’ actions and also their intentions.
Emotions can be conveyed in a variety of ways in human communication. For example,
speech or gestures can be used in addition to facial expressions. However, facial
expressions are by far the most commonly used form for human communication. The
correct interpretation of other people’s emotions from their facial
expressions can therefore lead to considerable advantages in social interaction
[1]
[2].
Over the past 50 years, great progress has been made in our understanding of mimic
emotion transmission. In contrast, our understanding of emotions themselves lags far
behind, and they remain one of the most fascinating and mysterious products of brain
function. This is evident in the difficulty of defining emotions.
In this article, we will first address the definition of facial expression,
communication, and emotion, and then discuss the voluntary and involuntary aspects
of our facial expression of emotion. We will discuss the societal benefits,
evolutionary origins, and current possibilities of automated emotion
recognition.
2. Facial expression, communication, and emotions
The term of facial expression describes visible movements of the facial surface,
which is conditioned by a specific pattern of muscle activity. Facial
expressions can be perceived and interpreted by people. It is a central element
of human expressive behavior and is particularly suitable for expressing
emotions and communicating them to the outside world. The interpretation of
emotions from facial expressions requires higher cerebral processing capacities,
which besides the evaluation of the current visual input also strongly depends
on contextual information and the availability of memory content [2]. In contrast to facial expressions and
communication, the concept of emotion is not clearly defined and there are still
controversial discussions about what an emotion is in the first place and
whether consciousness is a prerequisite for emotions [3]
[4]
[5]
[6]
[7]
[8]. Some authors believe
that emotions are conscious abstractions of one‘s internal state, which
are assembled cognitively. According to this definition, an emotion requires
awareness of one’s self, and thus emotion as a concept would no longer
be applicable to many other species. Other authors argue that consciousness is
merely a property of emotions and their processing, which is additive to the
basal emotion properties [9]
[10]
[11].
Emotions are often described by words (happy, sad, angry). In psychological
research, “valence” and “arousal” have become
accepted as a way of describing the concept numerically. Valence describes the
affective content of an emotion as a continuum from pleasant to unpleasant or
good to bad, while arousal describes the degree of internal activation by the
emotion.
In this paper, the latter definition will be used and thus emotions considered as
evolutionarily conserved functional states of the brain [7]
[12]
[13]. According to this
view, emotions emerged from evolutionary processes and primitive forms may still
be present in humans as conserved features. However, it is undisputed that
emotions trigger certain reactions on biochemical, physiological, and behavioral
levels [14].
To interpret an emotion from a person’s facial expression, we do not
always need the entire face; single parts are often sufficient (e.g., sad eyes
or happy mouth) [15]
[16]. The integration of a wide variety of
individual features across the entire face is a highly integrative performance
that is significantly more complex than just a summation of individual
properties [17]
[18].
The significance of individual facial regions for decoding emotions differs
between different emotions. For example, the eyes are particularly importance
for the perception of sadness [19], anger
[20]
[21], and fear [22]
[23]. Although other parts of the face also
provide cues for these emotions, it has been shown, for example, for fear that
emotion recognition form the upper part of the face is more accurate than from
the lower one [24]. In contrast, people
find it easier to recognize happy facial expressions from the lower part of the
face [24]. Accordingly, observers of happy
faces focus longer on the mouth region [22]
[23]
[25]. For the nose region, a particular
importance in decoding facial expressions of disgust could be demonstrated [19].
In addition to the interpretation of individual facial regions, facial expression
is also processed and interpreted holistically, i.e., there is still a
processing of facial expression that is not composed of the individual features
and is thus more than the sum of its parts [26]. The significance of this holistic facial interpretation for the
recognition of emotions differs between emotions. However, it is true for all
emotions that their interpretation is composed neither purely of the perception
and processing of individual features of subregions of the face nor exclusively
of this holistic face interpretation, but consists of a combination of these two
parallel processes [16]
[27].
3. Voluntary and involuntary facial expression, classification of facial
expression – microexpressions
Humans can control their facial expressions arbitrarily. This voluntary control
can be used to feign an emotional state. This deception mostly refers to a
communication partner, but can also be directed to oneself. The voluntary
control of facial expressions has a target state, which corresponds to an
internal model (what we want to express). In addition to the voluntary facial
movements, however, felt emotions are generally expressed involuntarily by
facial expressions and the corresponding muscles are involuntarily controlled.
They penetrate our conscious control and are reflected in our face. This
emotional leakage does not always have to be accompanied by large movements of
muscles, but can in part also be very discreet. These movements are summarized
under the term of microexpressions. Specifically, we understand microexpressions
as facial movements that are short-term, involuntary, and automatic. Certain
muscle groups contract or relax in fixed patterns. It is further assumed that
these patterns have a genetic basis and are closely coupled to the emotional
state of a person. The genetic basis has further led to the assumption that
there is a set of basal emotional states that are universal across cultures
[28], although this assumption
continues to be debated to this day [29].
Because these movements are involuntary, it is assumed that they allow
conclusions to be drawn about a person’s actual emotional state.
Of particular interest is the interaction of voluntary and involuntary facial
movements. Voluntary facial movement is often used to conceal the actual emotion
felt and thus to deceive the communication partner. The question is, whether
these voluntary facial movements are fast enough and, more importantly, complete
enough to suppress involuntary microexpressions that give a hint to genuine
emotion.
In a classic experiment, Paul Ekman and Wallace Friesen filmed a psychiatric
patient trying to convince the therapist that she was fine and could be
discharged from the psychiatric ward. At the end of the interview, the patient
broke down crying and confessed that she continued to be unwell. During the
phase of pretending to be well, various short-term microexpressions could be
detected, which clearly indicated deception [30]. Subsequently, there were still a large number of studies on this
topic that ultimately concluded that it is not always possible for the deceiver
to maintain the deception perfectly and that it is often involuntary that
emotions escape via facial expressions in the form of microexpressions. Due to
the involuntary nature of these microexpressions, it is further assumed that
they allow conclusions to be drawn about the actual emotional state of a
person.
4. Cultural similarities and differences in facial expression and
emotion
The dominance of the Western world in academia over the past century has resulted
in the majority of studies on the perception of facial expressions and emotions
being conducted on culturally westernized Caucasians. There have been few
studies in the last century that have examined the relationship between facial
expressions and emotions across cultures and ethnicities. The first significant
studies were conducted by the American psychologist Paul Ekman in the 1960s and
1970s and he concluded that facial expression of emotions was universal across
cultures [28]
[31]. This was thought to be true for at
least six basic emotions [32]. This view
remained unchallenged for a long time, but more recent studies now suggest
cultural differences [29]. This is
especially true when emotional expression is specifically examined according to
contextual and cultural influences. For example, people of Western and East
Asian cultures show common perceptions of pain, but differences in the mimic
expression of pleasure have also been found [29].
In our everyday life and communication with others, however, we assume as a
matter of course that our model of emotions and emotion perception is universal.
Seemingly contrary to this statement, however, there is also a racial bias that
influences the perception of the mimically expressed emotions of people from
other cultures [33].
5. Influence of diseases on facial expressions and emotions
Facial expressions – as well as the interpretation of facial expressions
of others – can be altered by various diseases. The classic example for
a disturbance of the mimic expression of emotions is Parkinson’s
syndrome. This is a neurodegenerative disease that is associated with a
reduction in facial movements [34]
[35]. In later stages, the ability to
modulate facial expressions is severely restricted so that even the term of
“mask face” is used [35].
In this stage, the patients do no longer show any emotionally triggered facial
expression. This diminished or, in some cases, nonexistent mimic expression has
a significant impact on the patients’ ability to communicate and on how
they are perceived by others [36]
[37]
[38]. Thus, observers perceive the corresponding patients as less
empathetic, bored, anxious, and cranky [36]
[37]
[38]. This leads to less interest in
relationships with these patients [36].
The consequences are increasing deterioration of social relationships, even with
close people, increasing social isolation, and a concomitant deterioration in
perceived quality of life [34]
[35]. These significant effects highlight
the importance of facial expressions for social interaction.
Reduced or absent facial expressions also have a direct impact on communication
behavior. In the context of medical conversations, for example, there are often
monologues of the doctors in which disease models are illustrated or further
therapy steps are explained. Although in such situations often only one person
speaks, the speaker receives a multitude of mimic messages, expressing e.g. the
degree of understanding of what is said, but also giving information about the
emotional state of the interlocutor. If the speaker becomes aware of these
messages, he/she then modulates the content according to the feedback
received. The complete absence of any mimic response in such a conversational
satiation is an impressive experience, since it gives the feeling of conducting
the conversation without orientation. The speaker has no indication of whether
the interlocutor has understood what has been said, nor does it become clear
whether the interlocutor reacts with annoyance, rejection, or acceptance. The
condition leads to uncertainty and discomfort for the speaker. There seems to be
an evolutionarily determined need for communicative feedback. The “still
face” experiments conducted with babies and toddlers by Edward Tronick
seem to point in this direction [39]. In
these experiments, the mothers suddenly stop any mimic expression. The children
then start various actions in order to restore interaction. They increasingly
show reactions of discomfort until they usually react with crying [39].
These experiments, but also the feelings as adults in similar situations do not
only emphasize the importance of facial expressions for communication, but also
force the interlocutors to change their communication behavior. Longer
monologues are no longer possible, since one has to rely on verbal feedback to
what is said. It also makes clear how strongly even “monologues”
depend on the constant non-verbal exchange of information.
The effect of reduced or absent facial expressions described in the example of
Parkinson’s syndrome is directly transferable to other diseases that are
accompanied by a reduced ability to express emotions facially, e.g., muscular
dystrophies, myasthenia gravis, or amyotrophic lateral sclerosis. The negative
effects on communication and perception of the affected person can also be found
in unilateral disorders of facial motor function such as peripheral facial nerve
palsy [40].
However, in addition to the reduced ability to express emotions mimically,
Parkinson patients also suffer from a reduced ability to perceive mimic
expressions of others [41]
[42]
[43] and also have fundamental difficulties with processing the
emotional content [44]
[45]
[46]. This is a characteristic of many neurodegenerative diseases. The
accompanying cerebral changes of the disease are often considered as cause.
However, the influence of one’s own restricted facial expressions has
also been discussed for some years as a cause of the limited ability to
interpret the emotional content of the facial expressions of others. Thus,
limitations in the interpretation of the emotional content of others’
facial expressions is also found in myasthenia gravis [47], myotonic dystrophy [48], Duchenne muscle dystrophy [49], and amyotrophic lateral sclerosis
[50]. Although all these diseases
primarily affect the patients’ motor function, they also have central
effects. Interestingly, however, changes in the interpretation of emotional
facial expressions of others can also be observed in purely peripheral disorders
such as peripheral facial nerve palsy [51]
[52], supporting the theory
that limitations in one’s own purely motor ability to express emotions
facially have a negative effect on the ability to interpret emotions in the
facial expressions of others. This theory is supported by the findings that, for
example, in amyotrophic lateral sclerosis, mimic emotion detection is reduced
even when other cognitive abilities are still preserved [50]. Also, Parkinson patients as well as
patients with Huntington’s disease show a direct correlation between the
ability to express themselves mimically and the ability to interpret the
emotions of others [53]
[54]
[55]. In summary, the main common feature of peripheral facial nerve
palsy with all these described neurodegenerative diseases is the reduction of
the ability to control facial expressions motorically. Nevertheless, all
diseases show common effects on emotion recognition and cerebral processing of
emotions. This suggests that even in the presence of possible alternative causes
for a reduced emotion recognition, the purely motor disorder may impair it.
These findings also align well with the currently dominant theory of embodied
recognition. This theory states that an important mechanism for perceiving
facial emotions is their sensorimotor simulation in one’s own brain.
This requires a mental replication of the perceived facial expression. However,
since one’s own facial expression is disturbed, also the mental
replicability is altered, which ultimately limits the ability to simulate and
thus recognize facial emotions [56]
[57].
However, in addition to these peripheral and/or central disorders
described above, there are also many central disorders that interfere with
emotion detection of others’ facial expressions. For example, deficits
are found in patients with schizophrenia [58], chronic alcoholism [59],
borderline personality disorder [60], and
multiple sclerosis [61]
[62]
[63].
6. Facial expression and emotions in the era of masks
Wearing face masks has become a commonplace in the era of the COVID pandemic.
Face masks represent the most cost-effective protective measure against the
SARS-CoV2 virus and allow continued personal social contact [64]. Nevertheless, considerable resistance
to the wearing of masks was shown in many countries. The main reason discussed
is the feeling of restriction of personal freedom [65]
[66]. The fact that wearing a face mask is perceived as such a strong
restriction of personal freedom despite the considerable advantages seems to be
related to the perceived strong restriction of physical and psychological
freedom by the face mask. Most probably, a piece of textile that would have to
be worn as a bandage on the upper arm, for example, would not have been
associated with the same feeling of restriction of personal freedom. Wearing a
face mask is certainly an inconvenience and can also lead to an increase in
headaches [67] or acne [68] in some individuals. However, these
side effects are rare. It is much more likely that the feeling of restriction of
freedom arises from the fact that the mask covers most of the very regions of
the body that makes us communicative and social creatures. However, our
acceptance of wearing the mask again affects how we are perceived by others.
Thus negative feelings when wearing the mask affect also the social perception
by other mask users [69]. Wearing masks
naturally also disturbs the recognition of emotions from the facial expressions
of our fellow human beings and makes social interactions more difficult [70]. Since the mimic emotion recognition is
not composed of the sum of facial subareas, but also involves as strong holistic
process, wearing masks worsens this ability more than would be expected from the
lack of information from the covered facial areas [71]. It is further argued that the
resulting deficits in nonverbal communication give people a feeling of
insecurity and discouragement and can also promote the emergence of
psychological disorders if there is a corresponding predisposition [72].
The effects described above are currently being reduced by the increasing
abolition of the obligation to wear face masks. However, it is still obligatory
in the health care system and thus a particular challenge for physicians
regarding empathy and conversation skills. It is crucial that both communication
partners are aware of the resulting perception deficits and that these deficits
are compensated by changing and, above all, increasing bidirectional
communication behavior.
7. Facial expressions and emotions in other species
In many species, the expression of emotion takes place on biochemical and
physiological levels similar to human responses. For example, responses to fear
are very similar with accelerated respiration, change in blood distribution,
hormone release, and pupil dilation. In facial expressions, there are
significantly stronger differences between species, which are due to the
different number of muscle groups in the face and the ability to control
them.
Already Darwin stated that animals can develop specific facial expressions
adapted to their species [73]. The
“Facial Action Coding System” developed by Ekman and Friesen for
the anatomy-based classification of emotions in humans [74] could also be successfully used in
seven other species including chimpanzees [75]
[76]
[77]. Even in rodents, mimic expressions
with at least an affective content could be detected [78]
[79].
However, it is undisputed that the possibilities of facial expression are reduced
with a lower level of development [75]. In
this context, the question of how much self-awareness is necessary to express
emotions mimically and to recognize them in others remains of great
interest.
8. Cerebral processing of mimic emotion perception
Functional imaging studies show that the processing of emotional faces is
associated with increased activity in visual, limbic, temporoparietal,
cerebellar, as well as prefrontal areas and in the putamen [80] ([Fig.
1]). Visual information first reaches the visual cortex. Here,
processing of emotional faces occurs in several subareas (inferior occipital
gyrus, lingual gyrus, fusiform gyrus). Processing in these regions appears to be
deep in the processing hierarchy and independent of the emotion processed or its
valence [81]
[82]. The cerebellum is also involved in
emotion processing. Although the cerebellar areas have strong anatomical
connections to the cortical association cortex and various limbic structures
[83], they, like the visual areas,
appear to be independent of the emotion itself being processed [84]
[85]
[86].
Fig. 1 Regions of increased neuronal activity in response to
happy, sad, angry, fearful, and disgusted human faces compared with
neutral faces (FG fusiform gyrus, ACC anterior cingulate cortex, LG
lingual gyrus, IOG inferior occipital gyrus, MFG medial frontal gyrus)
[80].
Fearful faces activate the medial frontal cortex, which plays a key role in
conscious experience and the regulation of emotion experience.
The amygdala is activated when looking at sad, fearful, or even happy faces,
whereas angry or disgusted faces do not cause a change in amygdala activity. For
the latter two emotions, particularly strong activation was found in the insular
region [80] that seems to respond more
strongly to disgusted faces than to angry ones. Limbic regions such as the
amygdala and hippocampus are thought to be more involved in processing external
positive emotional sensory information, while the insular region is activated
primarily for interoceptive stimuli and distressing external stimuli [87]. The cingulate cortex is particularly
activated by the perception of happy faces [87].
This distribution makes clear that the processing of emotional contents of faces
occurs in a complex network of brain regions. Different emotions involve
different brain regions to different degrees, but many brain regions are
involved in the processing of most emotions.
However, the cerebral activation pattern in the processing of emotional faces
seems to depend not only on the emotion presented but also on the attention paid
to it [88]
[89]
[90]. In general, there is
consensus that face processing shows right-brain dominance. This is a
long-standing finding for general processing of faces for identity recognition,
emotion, and attention [88]
[89]
[90]. This is also congruent to the finding that in patients with
parietal lesions, right-sided lesions lead to stronger neglect symptoms compared
to left-sided ones [91]. This right-sided
dominance for general processing of faces also seems to refer to the processing
of emotional faces. Thus, a bilateral presentation of fearful faces found
greater activation and connectivity of right visual cortex and right amygdala
compared with neutral faces [92]. Even
when little attention is paid to faces, right lateralized increased activity is
found in the amygdala and temporal pole [93].
The processing of emotional faces described above takes place explicitly, i.e.
with an awareness of the information. This is in contrast to the implicit, i.e.
unconscious, processing of emotional information from faces. Experimentally,
this unconscious perception can be generated, e.g., by temporally retroactive
masking with other stimuli or by interocular suppression. This implicit
processing is particularly interesting because it was shown early that patients
with structural lesions of the primary visual cortex could discriminate specific
characteristics of these stimuli [94].
Subsequent studies revealed that this ability relates particularly to the
perception of emotional faces [95]
– which is a phenomenon that has been called affective blindness.
Affected patients show the ability to correctly identify emotions from faces
with a disproportionately high frequency, although they cannot see them [96]
[97]. They show increased activity in the left amygdala, left
striatum, and right middle temporal gyrus [93]. Processing of unconsciously perceived faces is also dependent on
expressed emotions. In the processing of unconsciously perceived faces with
positive emotions, the temporal and parietal cortex are more involved than in
the unconscious processing of neutral faces. In contrast, unconsciously
perceived facial expressions with negative emotional content lead to stronger
activities in subcortical regions [93].
Overall, the parallel existence of a subcortical processing pathway to the right
amygdala via the midbrain and thalamus has been proposed for the processing of
emotional faces, while a parallel cortical pathway exists for conscious face
perception [93]
[98]
[99]
[100]. However, this view is
controversial, as some authors argue that the subconscious processing of
emotional faces is also cortical [101]
[102]
[103]. The existence of a subcortical
parallel pathway would also explain the fact that basal emotions in faces can be
perceived in prefrontal areas at the speed of up to 120 ms [104], which is surprisingly fast
considering that motion-independent foveal information reaches the occipital
cortex only after about 60 ms and the complex compound P100 component is
measurable only after 100 ms. Despite this considerable speed, however, a fast
cortical pathway cannot be ruled out here either. In addition to these fast
components, however, significantly later components are also measurable in the
prefrontal cortex. It appears that the interpretation of emotions occurs in a
temporally staggered process. The prefrontal cortex is also of particular
importance in this context because it modulates visual information processing in
other brain regions [104]. This iterative
information processing seems particularly plausible because it shows strong
similarities to the parallel organization in other sensory systems [105]
[106]. The existence of parallel very fast responses also indicates
the considerable evolutionary importance of emotion recognition, since this
additive processing pathway must have provided an evolutionary advantage due to
its speed.
9. Interpretation of emotions from facial expressions in their meaning in
society
It is generally agreed that the emotions in faces we look at have an effect on
us. Although we would not cognitively classify the nice smile of the hotel
employee at the counter as a sign of joy about our visit, we mostly perceive it
as pleasant or at least the absence of a smile as unpleasant. In the legal
systems of Western democracies, the interpretation of emotions by judges is also
seen as influencing the sentence. This fact presupposes the self-conception that
we believe we can correctly classify emotions. Thus, the display of grief, both
in facial expressions and in words, often has the effect of reducing the
sentence. In this regard, many people consider facial expressions to be more
decisive under the assumption that they allow a stronger insight into emotions
than words of regret read aloud. For example, a US Supreme Court Justice
(Anthony Kennedy) wrote in 1992 that reading the defendant’s emotions is
necessary to “know the heart and mind of the offender”.
Even if the interpretation of emotions and their assessment on sentencing is
considered fair in our legal system, the same is not true for emotion
interpretation in other areas of society. The American psychologist Paul Ekman
developed and implemented a training program for the US Transportation Security
Administration in 2007 that trained staff to observe passengers for suspicious
signs of stress, deception, or anxiety. The program is based on Paul
Ekman’s research from the late 1970s, during which Ekman published
several articles on identifying emotions from minimal changes in facial
expression [107]. It is further based on
the assumption that the attempt to hide or lie about emotions leads to the
leakage of emotions in the form of microexpressions in facial expressions.
However, the considerable inaccuracy and racial bias led to important criticism
by scientists, members of the US Congress, and various human rights
organizations [108]. Leaving aside ethical
and moral aspects, this program is highly interesting from a purely scientific
perspective, as it explores the hypothesis that it is possible to determine from
the mere observation of people whether they are cheating or planning hostile
acts. An idea of the size of the program comes from the number of people
involved. In 2015, 2,800 people were employed in this so-called SPOT program and
between 2007 and 2013, 900 million US dollars were invested. Of course, no
academic-driven research can reach this scale. The enormous data size and the
various technology deployments make this program particularly interesting.
The aforementioned hypothesis has found many supporters, and thus personnel have
been hired at London Heathrow Airport for behavioral observation, and the US
Department of Homeland Security is also pursuing a program for electronic
assessment of nonverbal behavior. In this context, the study of facial
expressions is a particularly important aspect. Critics argue that there is no
scientific evidence to support the assumption that conclusions about intent or
future behavior can be drawn from observations [109]. This assessment was shared by a review commission of the
program by the US Congress. In response to these concerns, the Transportation
Security Administration commissioned an independent study, the results of which
are currently pending.
Automated analysis and categorization of emotions from faces has been intensively
researched over the last 2 decades. Current algorithms are able to identify
emotions with more than 90% correctness from static pictures. In the
last 10 years, the analysis of pictures has taken a back seat in research, while
increasing research has been done on the analysis of emotions from video
sequences. Here, too, significant success has been achieved in the automated
analysis and categorization of emotions from faces due to the considerable
increase in computer power and, above all, to advances in the research field of
computer vision [110]
[111]
[112].
The analysis of videos is much more complex than the analysis of static images.
When analyzing videos, it is not a matter of analyzing each individual picture,
but rather these emotion analyses are dependent on the characteristics of the
face in the past and in the future. In particular, unlike static images,
voluntary and involuntary facial movements can be distinguished in videos. It is
precisely the “leakage” of genuine emotion through a feigned
emotion that can be analyzed. And it is precisely this interplay of voluntary
and involuntary movements and the associated detection of microexpressions that
are the subject of very active research [113].
The “leakage” of genuine emotions through an acted facial
expression lasts for a very short time and is often difficult to detect by the
human eye. The duration of microexpressions is reported to be in the range of
40–200 ms [30]
[114]. Another property of microexpressions
that makes them difficult to detect by the human eye is that they are often
fragmented. Only part of the muscle groups expressing a particular emotion are
activated during these microexpressions [115].
These characteristics of microexpressions make them an ideal application area for
computational image analysis. There are many applications, e.g., in business
negotiations, in psychiatric interviews, and in forensics [108]
[116]. A first approach was developed by Paul Ekman in 2002. He
trained people with the help of a tool to detect 7 different categories of
microexpressions [117]. The results were
however sobering, since it could be shown that students, who worked with this
tool in the recognition of microexpressions, were inferior to the staff of the
US Coast Guard that worked without assistance [118].
To date, the development of analysis methods has been significantly limited by
the paucity of available datasets. To create datasets with the richest possible
microexpressions, subjects are encouraged to watch videos with emotional content
and are simultaneously offered rewards if they do not show emotions. However,
labeling these data remains difficult and results in currently only a few
existing freely accessible datasets of limited size [113]
[119].
Additionally, analyses are complicated by head movements of the recorded person
but also by facial movements without emotional reference (e.g. blinking).
Besides the interpretation of the emotional content of a microexpression,
especially the identification of the phases of occurrence of this
microexpression represents a considerable problem [120]. For these partial problems, there has
been an increasing number of investigations and a significant improvement in the
quality of results, especially in the last 12 years. If the number of emotions
to be recognized is limited (e.g., to positive, negative, surprised, and
others), results with an accuracy of around 70% can be obtained [121]
[122]
[123].
On single databases (no performance across databases), current results show an
accuracy of 65% to more than 70% in the recognition, if the
characteristics that should be recognized are limited (e.g., positive, negative,
surprised, and others). Current methods also focus on the application of
so-called convolutional neural networks, which can achieve an accuracy of over
70% with different auxiliary methods [124] and this partly even across different datasets [125]. Despite this significant progress,
there is still potential for improvement in emotion detection from videos and
also in the detection and interpretation of microexpressions. The current
quality of results clearly limits its use in practice.
