A Web-based Game for Teaching Facial Expressions to Schizophrenic Patients

Keywords

Facial expression - video games - schizophrenia

1. Background and Significance

Facial expressions constitute an important component of non-verbal communication between people. Many psychological studies have shown that recognizing and understanding facial expressions is an important social skill [[1]–[3]]. In certain psychological disorders, such as autism and schizophrenia, the loss of these skills may complicate the patient’s daily life. For instance, patients may suffer from social difficulties and communication problems [[1]–[4]]. Prior research has shown that information technology may help to develop facial expression recognition skills through educational software and “serious” games [[5]–[7]]. The primary purpose of such serious games is training or educating the users rather than entertaining them [[8]].

There are many facial expressions and most of them are culture-specific. A facial expression specific to one culture may mean nothing or something quite different in another cultural environment [[9]]. However, it is widely accepted that there are seven universal facial expressions recognized in the same manner in every culture. The feelings these facial expressions represent are anger, fear, happiness, surprise, disgust, sadness, and neutrality [[10]]. Internet-based educational software using these expressions may be used in any culture and may help the patient to improve facial expression recognition skills. Frommann et al. designed a study to compare 16 ‘post-acute’ patients with schizophrenia and a control group. According to the study, a group of patients who were trained with software improved their performance in the identification of facial emotions [[5]]. Silver et al. explored the effect of emotion training exercises on the perception of facial emotional expressions. Twenty male chronic patients with schizophrenia underwent three training sessions using emotion training software, which was initially developed for autistic children and then adapted to the clinical setting. Patients were assessed before and after training with validated tests for the identification of facial emotions, differentiation of facial emotions, and working memory. They showed that brief emotion training could improve recognition of facial emotional expressions in patients with chronic schizophrenia [[6]]. Russel et al. investigated the effectiveness of the ‘micro-expressions training tool’ (METT), which was developed to improve emotion recognition skill. Twenty patients with schizophrenia and 20 healthy control participants were involved in the study. The patients with schizophrenia showed significant improvements in emotion recognition following the training with this tool [[7]].

2. Objectives

Although previous studies have shown that facial expression training software can help patients with schizophrenia, none of those were specially designed for this purpose. Disease-specific solutions can improve the outcomes, since issues in recognizing facial expressions exhibit different patterns for each psychiatric disease [[11]–[13]]. During development of a training environment for patients with schizophrenia, the general characteristics of the disease, such as impairment in social cognition, difficulties in working memory, long-term memory, attention, executive functioning, and speed of processing, should therefore be considered.

The aim of this study was to examine if a games website designed for teaching facial expressions could improve facial expression recognition skills of patients with schizophrenia.

3. Methods

3.1. Selection of Images

We designed a website that includes brief facial expression education and serious games for patients with schizophrenia. The first requirement was a basic facial expressions digital photography set. Even though the basic facial expressions could be universally recognized, we preferred to prepare a digital facial expressions photography set with the participation of 40 Turkish volunteers to decrease cultural deviations. The majority of these volunteers were amateur or professional theatre players. In contrast to the general acceptance of universal face expressions, several studies indicated that recognition of facial expressions could vary in different cultures [[9], [14]–[15]]. We collected photographs for each of the six basic facial expressions and one photograph with a neutral expression. A total 1001 photographs were evaluated according to three steps. At the first stage, all facial expression photographs were assessed by the research team (FI, KHG, BC, MKS and NZ) as a consequence, 561 photographs were accepted for the second stage. At the second stage, a web survey was prepared and 33 volunteers evaluated the 561 photographs. During this stage, we asked each participant to match images with the correct facial expression, and we used the images that were correctly recognized (187 photographs, consensus ≥97%). We eliminated the photographs (68 images) if they were poorly recognized (consensus < 58%) by the participants. At the final stage, the remaining 306 photographs were evaluated by an additional 396 volunteers, making a total of 427 voluntary participants. For the final evaluation, the survey was announced on the Facebook and medical informatics mail groups. All participants were scored according to their consensus with the other participants. For each photograph, the user’s response was noted and the number of other users with the same response was recorded. The ratio of this number and the total number of users was calculated as the consensus score for the photograph. For each user, an overall mean consensus score was calculated based upon the mean of the consensus scores for each photograph. Participants whose consensus score was lower than 0.65 were excluded from the study. Images that were recognized by the remaining participants with a high degree of consensus (≥85%) were retained in the final set of photographs. The one exception was that the threshold was decreased to 75% for the fear expression, because the consensus of volunteers was relatively low in this case. The details of developing the photography set were as described in a previous study [[16]]. The final set included 364 photographs which were selected from a collection of 1001 photographs, taken of 40 models. The photograph set has been shared for scientific use (http://yuzifadeleri.org/expressions.htm).

3.2. I am Learning Facial Expressions software (ILFE)

ILFE was designed as an internet-based educational tool in order to make it easily accessible (http://yuzifadeleri.org/). We used Microsoft Visual Studio 2010 Professional Edition, and Microsoft Access 2010 for developing the web site. Programming language was C#, and we also used HTML, JAVA Script and Jquery libraries The ILFE includes eight serious games, which were designed using the principles of errorless learning, repetition, feature abstraction, direct positive reinforcement, and self-instruction. Additionally, all the games were designed with a consideration for some of the common characteristics of schizophrenia-like deficiency in social cognition, difficulties in working and long-term memory, attention deficit, disturbance in executive functioning, and lower speed of processing. Decisions about the flow of the games were made by a psychiatrist (author BC).

In addition, the usability of ILFE was evaluated in two stages by using heuristic evaluation and then according the protocol analysis (PA), or the “think aloud” method.

Heuristic evaluation is an evaluation of an interface by one or more experts. Evaluators measure the usability, efficiency, and effectiveness of the interface based on ten usability heuristics originally defined by Nielsen [[17]]. According to Nielsen, the number of the evaluators is normally three to five, since one does not gain that much additional information by using larger numbers. For our study, seven “Medical Informatics” experts and one computer and education technologies expert completed the heuristic evaluation questionnaire; the feedback from evaluators was then used to ascertain ILFE’s design problems. Experts were given a scenario (task list) before they started the evaluation. Accordingly, they identified all major and minor problems.

Although a number of usability problems were identified in the assessments of the experts, studies like the current systems must also be evaluated by real users to increase the overall system quality and usability. We used PA with real patients as soon as we had completed the heuristic evaluation.

The PA method requires participants to verbalize their thoughts, feelings, and opinions during the test. One goal of this approach is to enable the tester to get a better understanding of the participant’s mental model during interaction with the interface. Previous studies have indicated that, for extensive usability evaluation methods such as PA, using a small number of subjects (for instance five) would be sufficient due to the long evaluation process [[18]]. Therefore, five patients with schizophrenia participated in the PA test. Prior to the assessment, the purpose of the research and PA was explained to patients and they were then given some tasks. After the PA, with the help of the data obtained from patients, we made the final alterations [[19]].

Training Module

Six basic face expression images and brief explanations about these expressions were present in the training module. The module was developed for patients who wanted to practice before playing the games or between the games.

Games Module

The games designed for the ILFE have difficulty levels ranging from easy to hard (►[Figure 1]). In order to proceed to the next screens, the patients have to answer the questions successfully. There is no playing limit for the games; they can keep trying until they find the correct answer. There are eight games, including a memory card game, and the games have been designed to be played using a mouse. None of the games requires advanced computer skills or motor abilities to do complex tasks on keyboards or other devices. The games have no time limitation except the seventh game. The psychiatrists in our development team (BC and SSK) suggested that the addition of sounds during the game might cause a distraction for the patients. Based on this suggestion, we used no sounds for the games except for applause that plays after a correct answer along with a ‘well done’ message. In the case of incorrect responses, a ‘try again’ message is immediately given. The patient cannot pass to the next game without giving a correct answer.

Fig. 1 Screen samples from the games (All text in the figures was translated into English).

Game 1, Name this expression

The user sees an image, and they have to find the correct facial expression text that matches to the image.

Game 2, Find the correct expression

The user sees an expression text and selects the correct expression from among several images. The game consists of three levels.

Game 3, Carry the correct image

In the first level, there are four images and one expression. The user must determine which image matches the expression, and drag the image over the expression with the help of the mouse. The number of the images increases in the second level. In the third level, the number of the images and the expressions are equal.

Game 4, Match the image and the expression

The user must find the written expression and drag it to under the appropriate image. The game starts with three images and in each level it increases by one until there are 10 images. In this game, if the user makes one or more incorrect match, they fall one level, if they make all of the matches correctly, they increase one level.

Game 5, Find the same expression

In this game, there is a sample image on the left of the screen, and there are four images on the right of the screen. These images belong to different people. The user must find the same expression in the sample image from among the images on the right. The user must find the same expression from among six images on the second level and eight images on the third level.

Game 6, Find the different expression

In the first level, there are three images of the same person. Expressions in two of the images are the same. The user has to find the different expression. In the second level, the images belong to different people.

Game 7, Balloons

In this game, balloons with facial expressions slide along the screen. Each balloon disappears within a pre-specified time. The user has to find and click on the balloon which has the correct facial expression. When they complete a mission, a congratulatory message appears and gives a new mission as: “Congratulations, you have found all the HAPPY faces, now your mission is to find SAD faces.” If the user makes nine mistakes, the game starts from the beginning. The game has seven difficulty levels.

Game 8. Memory cards game

This game needs an additional skill, memory. The user has to find the same expression in image pairs. In the beginning, all the images are reversed. The user clicks one of the boxes and sees the image then clicks another box to see the other image. If the expressions in both images match, the images stay open. If they are different, both close. Thus, the user has to remember the place of closed images to find the pairs. The number of the cards increases by level.

3.3. Patients and design

After completing written informed consent, 42 patients with schizophrenia took the pre-test. Four of those 42 patients recognized 20/21 or 21/21 expressions in the pre-test, thus they were excluded from this study. It is thought that the facial expression recognition skills of these patients were not affected by their disease, and no meaningful improvement would therefore be observed in these patients. Another six patients chose to leave the study; they were also excluded. The remaining sample consisted of 32 patients with schizophrenia (20 females, 12 males, mean age ± standard deviation; 37.3±9.2). All patients were diagnosed with schizophrenia according to the DSM-IV (Diagnostic and Statistical Manual of Mental Disorders, Sixth Edition) [[20]], and all were receiving pharmacological treatment with antipsychotics at the time of the study. Patients experiencing an acute exacerbation of illness were not accepted to the study. All patients were followed up by Akdeniz University, Department of Psychiatry.

For each participant, performance of facial expression recognition was evaluated by pre-test and post-test before and after training. The patients were randomly assigned to the study group (n=18) or to the control group (n=14). There was no statistically significant difference between the study and control groups in terms of the results of the psychiatric tests, gender, age, and educational level. The assessment of emotion recognition (pre-test, post-test) was carried out through the use of a computerized test of facial emotion recognition. The tests included 21 different photographs (three for each emotion and three neutral). The participants looked at each of the photographs one by one and decided on their answer without any time restriction. ►[Figure 2] shows a screen from the online test. Patients got one point for each correct answer. The pre-test and post-test scores of the patients were compared separately in two groups. Although the photographs of the pre-test were used in the games, the patients had never seen the post-test’s photographs prior to the post-test.

Fig. 2 Example of the Emotion Recognition Test: Participants were asked the question, “Which is the facial expression in this photograph?” Patients chose one of the seven expressions and pressed the “Next” button to move to the next question in both the pre-/post-tests (All text in the figures was translated into English).

3.4. Psychiatric Assessment Instruments

The psychopathological statuses of the patients were assessed by two psychiatrists (BC and SSK) according to the Scale for Assessment of Negative Symptoms (SANS) [[21]–[22]] the Scale for Assessment of Positive Symptoms (SAPS) [[23]–[24]] and the Brief Psychiatric Rating Scale (BPRS) [[25]–[26]]. Neuropsychological assessment tools included Serial Digit Learning Test (SDLT) [[27]–[28]], the Wisconsin Card Sorting Test (WCST) [[27]–[32]], and Porteus Labirynths [[33]]. Our intention was to document the relation between facial expression recognition skill and the clinical features of the patients by performing these tests.

3.5. Training of the Patients and Post-Test

Eighteen patients participated in the training sessions during a one month period. All patients in the training group were informed by an investigator (author FI) about how to play the games. They were requested to play the games at least twice a week, on each occasion for a 60-minutes period. Nine patients (50%) had no computer or internet access in their home, so they used a computer in the hospital while they were playing the games. Logs were recorded and checked each week. If patients had forgotten to play the games, they were reminded. At the end of the one-month period, one day after the training group’s last access to the system, their performance was assessed by the post-test. The control group also took a post-test one month after the pre-test. Both groups continued to take their medication with no change during the study period.

3.6. Statistical analysis

Normality was tested with the Shapiro Wilk test. Numeric variables were compared using the Mann Whitney U test and nominal variables were compared by chi-square tests. For pair-wise analysis, the Wilcoxon test was used. Correlations were examined by the Pearson or Spearman rho tests. All of the above tests were performed using the Statistical Package for Social Sciences 19.0. All tests were two-sided, and p<0.05 was considered to be significant.

4. Results

There was no significant difference between the training and control groups in terms of gender (p=0.574), education level (p=0.084), or age (p=0.413). Similarly no significant difference was observed between the groups regarding psychopathological and neuropsychological assessments. BPRS (p=0.143), SANS (p=0.764), SAPS (p=0.659), Porteus scores (p=0.593), SDLT (p=0.233) or WCST scales; WCST-Number of Trials (p=0.112), WCST-Number Correct (p=0.102)], WCST-Total Errors (p=0.983), WCST-Perseverative Responses (p=0.983), WCST-Nonperseverative Errors (p=0.722), WCST-Perseverative Errors (p=0.867), WCST-Categories (p=0.898), WCST-% Perseverative Errors (p=0.867), WCST-Trials to Complete First Category (p=0.437), WCST-% Conceptual Level Responses (p=0.834), or WCST-Failures to Maintain Set (p=0.112).

In the pre-test, the patients most successfully recognized happy faces (96.9%) followed by nearly a similar success rate for recognizing surprised faces (95.8%). Success in recognizing angry, sad, disgusted and neutral facial expressions was 82.3%, 78.1%, 71.0%, and 64.6% respectively. The most difficult facial expression for the patients to recognize was fear (52.1%). The patients selected “surprised” instead of a correct “feared” response in 45.8% of the questions.

There were statistical correlations between pre-test scores and neuropsychological tests. Porteus (p<0.001, r=0.695), IQ (p<0.001, r=0.692), WCST-number correct (p = 0.004, r = 0.512), WCST-total Errors, p = 0.004, r = –0.512), WCST- Nonperseverative Errors (p = 0.016, r = –0.435), WCST-% Conceptual Level Responses (p=0,019, r=0,578) and WCST-Trials to Complete First Category (p=0,002, r=0,550).

In one month period, the number of sessions for each game was determined from the logs. Six games were played median two times, however the Balloons game was played 2.5 times, while the Memory Game was played 4 times by each patient.

The users were asked which game(s) did they enjoyed during the sessions, and 10 of them expressed their preferences. Seven patients stated that they mostly liked the Balloons, two of them liked the Memory Game and one liked both the Balloons and the Memory game. Additionally, seven of the patients expressed the feeling that most of the games were too easy for them.

The median pre-test score was 16.5 in the study group (minimum-maximum: 8–19, mean±standard deviation: 15.6±2.8) and 17.5 in the control group (8–19, 16±3.2) on a 21-point scale (p=0.406). Median post-test scores were 20 (16–21, 19.7±1.2), and 18 (9–19, 16.5±3.1) in the study and control groups respectively (p<0.001, ►[Figure 3]). The patients’ post-test and pre-test scores were compared by pair-wise analysis for each group. There was a significant difference (p<0.001) in the training group patient’s scores whereas the change in the non-training group patient’s scores was marginally significant (p = 0.052). The mean difference of pre and post-tests in the study group was 4.1 whereas the mean difference was 0.5 in the control group (p<0.001).

Fig. 3 Boxplot graphic of the training and control groups’ pre and post-test scores. The horizontal lines within the boxes represent the median of all values. The top end of the box represents the upper quartile, while the bottom end of the box represents the lower quartile. The upper end of the whisker above the box plot represents maximum, and the lower end of the whisker below the box plot represents minimum. Means are shown by triangles.

5. Discussion

The benefits of serious games on health have been shown in various studies up to this point [[34]–[36]]. People who have psychiatric diseases, such as schizophrenia, Asperger syndrome or autism may have impaired recognition of facial expressions [[2]–[5]]. It has been shown by many studies that recognition levels can be increased with computer-based education software and games [[6]–[8], [37]]. However, none of these software programmes were specifically designed for patients with schizophrenia. This study describes a web-based education tool (ILFE), which was specifically developed for training patients with schizophrenia to recognize facial expressions.

We developed a website for this aim. A new photography set of Turkish people was prepared to minimize cultural deviations. Unlike past studies [[37]–[38]]; we did not want to idealize the photographs from voluntary models. As such, we did not determine clothing, make-up, jewellery or hairdress rules for the models; rather we asked the models to come to the studio in casual style of dress. The flow of the games was designed with consideration of the characteristics of this patient group. The games were developed in a web environment to enable the patients to have easi access at any time and any place where a computer and internet access were available. As in previous studies [[38]–[40]], we observed that recognition of happy expressions was the highest compared to other facial expressions, whereas the fear was the lowest. According to Biehl, the success in recognition of happy faces may be related to its frequency in real life [[39]]. Indeed, it is probable that people frequently see happy expressions in their life so they can easily recognize it. On the other hand, it is known that the mesial temporal lobe structures, implicated in fear processing, are affected in schizophrenia, a possible contribution to a defect in the recognition of fear [[41]]. However, difficulty in the recognition of fear expression is not specific to patients with schizophrenia: it is also seen in the normal population [[9], [42]]. Generally, happy expressions represent the only universally recognized positive emotion, in contrast to multiple negative emotions [[41]]. When we separate expressions as positive and negative, our findings show that the patients with schizophrenia are more likely to recognize positive expressions than the negative ones.

We do not have specific information about schizophrenic patients’ preferences for the properties of computer games in general. This study also sets out to describe the usage pattern of a game set which was designed for use in schizophrenic patients.

Peterson reports that loss of information from short-term memory begins immediately after the learning process ends. By three seconds after learning ends, 38% of the information is lost and by 18 seconds, 85% of information is lost [[43]]. Repetition is very important in ensuring the retention of learned knowledge. Possibly, repeating is helpful in preventing the loss of learned information. The most fundamental way of retaining such information is simple repetition: constantly repeating the information may transfer it from short-term memory to long-term memory [[44]]. Immediate testing after learning is the most effective way to retain knowledge [[45]]. With the help of these principles, we designed a series of eight games. These games were inspired by some educational games prepared for children. Six of the games were very simple: basic visual multiple choice questions or visual matches. These games were played in a very similar frequency by our patients; they were all played a median two times by the patients. The two other games, Balloons, and the Memory Game were more complex. In the Balloons game, the images were moving, and the users had to catch them in a limited time. The Memory Game needed another skill, remembering the images. The patients played these two games more, a median 2.5 times for Balloons and four times for the Memory Game. Interestingly, the patients expressed their preference for these two games. According to the results of this study, schizophrenic patients prefer more complex computer games in spite of their particular mental disadvantages. Schizophrenic patients should, therefore, be evaluated as intelligent adult individuals, albeit with certain special characteristics. Game designs for schizophrenic patients should be further investigated to add more information about this special patient group.

In spite of deficits in attention, executive function, processing speed and working and long-term memory, patients with schizophrenia are like adults without psychiatric disorders in preferring more complex computer games. Additional studies are needed to identify the optimal characteristics and complexity of games for individuals with schizophrenia.

In some studies, it is reported that a disorder of the recognition of facial expressions is positively correlated with a disorder of general cognition [[46]–[47]]. According to our findings, Porteus maze test scores, and WCST scores had a positive correlation with the recognition of facial expressions. In other words, if the patients with schizophrenia had higher visual-spatial perception scores, they were more successful in recognizing facial expressions. According to the results of this study, WCST-number correct, WCST-total Errors, WCST- Nonperseverative Errors, WCST- % Conceptual Level Responses, WCST-Trials to Complete First Category, Porteus, and IQ scores were correlated with the level of recognition of facial expressions in patients with schizophrenia. In contrast to Bryson’s study [[48]], we did not find any relation between the perseverative errors score in WCST and recognition of facial expressions. This situation may be due to the small number of patients involved in our study; this should be investigated with more patients. However, taken together, these findings suggest a possible association of a good cognitive performance with an improved skill in recognizing facial expressions.

After one month of training, the facial expression recognition ability of the patients was evaluated for comparison with the pre-test scores. In the study group, a statistically significant increase in the facial expression recognition score was observed. The difference between pre- and post-test scores in the control group was close to being statistically significant (p=0.052). The increase in score in the control group can be understandable, because the patients were receiving treatment. Nonetheless, the increase in the post-test scores of the study group was more prominent.

The present study was planned so as to minimize bias. The study and control groups were similar in composition, confirmed by both demographic data and psychiatric tests. Both groups were in their routine therapy protocol during the study. However, we did not plan to make another computer game for the control group to play to equalise the possible effect of playing computer games on the ability to recognise facial expressions. Playing computer games may have a positive effect on post-test scores. Half of the patients were playing the games in the hospital. Social interactions in the hospital may also have had positive effects on facial recognition ability. In future studies, researchers should plan to use another game for the control group. This is one limitation of the study. Additionally, the sample size is small, and thus results should be confirmed in future studies.

In this research, training was limited to two sessions a week, for a month. The effect of the duration of the training period may be studied in the future. Moreover, the evaluation of patient performance was conducted immediately after the end of the training period. The long term effects – and possible effects – of training on the daily life of a patient should also be investigated. Increases in facial expression recognition ability are positive, but the real aim of this training was to improve the clinical situation of the patients. A more reliable evaluation of the benefit of the games would be possible with the help of psychiatric assessment instruments. However, we do not expect a rapid improvement in patients’ ability to recognize facial expressions. This skill is required for social interactions, but developing social interactions needs time. It may have a cumulative effect on the clinical situation of the patients over the course of months or years. Future studies should, therefore, evaluate patients immediately after training and at several times after training has been completed.

6. Conclusions

It is commonly known that information technology can support individual health in a variety of situations. The results of this study are promising. Computer games may be used to educate people who have difficulty recognizing facial expressions.

Multiple Choice Question

According to results of this study, what type of games do patients with schizophrenia like?

• A) Relatively complex games

• B) Children’s games

• C) Role playing games

• D) Very simple games

Correct answer: A)

Explanation: Six of the games were very simple; basically just visual multiple choice questions or visual matches. These games were played at a very similar frequency by our patients (median two times). Two other games, Balloons, and the Memory Game were more complex. In the Balloons game, the images were moving, and the users had to catch them in a limited time. The Memory Game needed another skill: remembering the images. The patients played these two games more often, a median 2.5 times for the Balloons and four times for the Memory Game. According to the results of this study, schizophrenic patients prefer more complex computer games.

• References

• 1 Kohler CG, Walker JB, Martin EA, Healey KM, Moberg PJ. Facial emotion perception in schizophrenia: a meta-analytic review. Schizophr Bull 2010; 36: 1009-1019.
• 2 Laroi F, Fonteneau B, Mourad H. A Raballo. Basic emotion recognition and psychopathology in schizophrenia. J Nerv Ment Dis 2010; 198: 79-81.
• 3 Corbett BA, Carmean V, Ravizza S, Wendelken C, Henry ML, Carter C. SM Rivera.. A functional and structural study of emotion and face processing in children with autism. Psychiatry Res 2009; 173: 196-205.
• 4 Wolwer W, Frommann N, Halfmann S, Piaszek A, Streit M, Gaebel W. Remediation of impairments in facial affect recognition in schizophrenia: efficacy and specificity of a new training program. Schizophr Res 2005; 80: 295-303.
• 5 Frommann N, Streit M, Wolwer W. Remediation of facial affect recognition impairments in patients with schizophrenia: a new training program. Psychiatry Res 2003; 117: 281-284.
• 6 Silver H, Goodman C, Knoll G, Isakov V. Brief emotion training improves recognition of facial emotions in chronic schizophrenia. A pilot study. Psychiatry Res 2004; 128: 147-154.
• 7 Russell TA, Chu E, Phillips ML. A pilot study to investigate the effectiveness of emotion recognition remediation in schizophrenia using the micro-expression training tool. Br J Clin Psychol 2006; 45: 579-583.
• 8 Bergeron BP. Developing Serious Games. Hingham, USA: Charles River Media; 2006
• 9 Ekman P, Friesen WV, O’Sullivan M, Chan A, Diacoyanni-Tarlatzis I, Heider K, Krause R, LeCompte WA, Pitcairn T, Ricci-Bitti PE, Scherer K, Tomita M, Tzavaras A. Universals and cultural differences in the judgments of facial expressions of emotion. J Pers Soc Psychol 1987; 53: 712-717.
• 10 Ekman P, Friesen WV. Constants across cultures in the face and emotion. J Pers Soc Psychol 1971; 17: 124-129.
• 11 Gaebel W, Wolwer W. Facial expression and emotional face recognition in schizophrenia and depression. Eur Arch Psychiatry Clin Neurosci 1992; 242: 46-52.
• 12 Bach DR, Buxtorf K, Grandjean D, Strik WK. The influence of emotion clarity on emotional prosody identification in paranoid schizophrenia. Psychol Med 2009; 39: 927-938.
• 13 Addington J, Addington D. Facial affect recognition and information processing in schizophrenia and bipolar disorder. Schizophr Res 1998; 32: 171-181.
• 14 Huang Y, Tang S, Helmeste D, Shioiri T, Someya T. Differential judgement of static facial expressions of emotions in three cultures. Psychiatry Clin Neurosci 2001; 55: 479-483.
• 15 Ekman P, Friesen WV. Constants across cultures in the face and emotion. J Pers Soc Psychol 1971; 17: 124-129.
• 16 Gulkesen KH, Cinemre B, Isleyen F, Zayim N, Kemal M. What are the characteristics of people who are successful on recognizing human facial expressions on computer monitor?. Proceedings of the Biomedical Engineering and Informatics [BMEI], 4th International Conference. ; 15–17 October 2011; Shanghai, China: IEEE Computer Society; 2011; 4 (09) pp. 1901-1903.
• 17 Nielsen J, Mack RL. Usability inspection methods. New York, USA: Wiley; 1994
• 18 Hudson W. How many users does it take to change a Web site?. ACM SIGCHI Bulletin –a supplement to interactions. 2001 ; 2001 [May-June]: 6.
• 19 Isleyen F, Gulkesen KH, Cinemre B, Samur MK, Zayim N, Sen Kaya S. Evaluation of the usability of a serious game aiming to teach facial expressions to schizophrenic patients. Stud Health Technol Inform 2014; 205: 662-666.
• 20 Association AP. Schizophrenia; Diagnostic and statistical manual of mental disorders: DSM-IV. Washington, DC: American Psychiatric Publishing; 2000
• 21 Andreasen NC. The Scale for the Assessment of Negative Symptoms [SANS]: conceptual and theoretical foundations. Br J Psychiatry Suppl 1989: 49-58.
• 22 Erkoç, Arkonaç O. Ataklı C, Özmen E. Negatif semptomlan de[gbreve]erlendirme ölçe[gbreve]inin güvenilirli[gbreve]i ve geçerlili[gbreve]i. Dü ünen Adam 1991; 4 (02) 14-15.
• 23 Andreasen NC. Scale for the Assessment of Positive Symptoms [SANS]. Iowa City, USA: University of Iowa; 1984
• 24 Erkoç, Arkonaç O, Ataklı C, Özmen E. Pozitif Semptomları De[gbreve]erlendirme Ölçe[gbreve]inin Güvenilirli[gbreve]i ve Geçerlili[gbreve]i. Dü ünen Adam 1991; 4: 20-24.
• 25 Overall JE, Gorham DR. The Brief Psychiatric Rating Scale. Psychol Rep 1962; 10: 799-812.
• 26 Soykan Ç. Institutional differences and case typically related to diagnosis, symptom severity, prognosis and treatment. MSc, Middle East Technical University, Ankara. Turkey: 1990
• 27 Zangwill OL. Clinical Tests of Memory Impairment. Proc R Soc Med 1943; 36 (11) 576-580.
• 28 Karaka S. B[Idotaccent]LNOT Bataryası El Kitabı. Ankara, Turkey: Eryılmaz Offset Matbaacılık Gazetecilik Ltd. ti; 2006
• 29 Çitak S, Çakıcı ET, Çakıcı M. Neuropsychological assessment in schizophrenic patients treated with haloperidol or clozapine. BCP 2009; 19: 5-14.
• 30 Berg EA. A simple objective technique for measuring flexibility in thinking. J Gen Psychol. 1948; 1948: 15-22.
• 31 Heaton RK. The Wisconsin Card Sorting Test manual: Odessa, USA: Psychological Assessment Resources Inc.;. 1981
• 32 Karaka S, Kafadar H. izofrenideki Bili sel Süreçlerin De[gbreve]erlendirilmesinde Nöropsikolojik Testler: Bellek ve Dikkatin Ölçülmesi. izofreni Dizisi 1999; 4: 132-152.
• 33 Porteus SD. The Porteus maze test and intelligence. Palo Alto, USA: Pacific Books; 1950
• 34 Merry SN, Stasiak K, Shepherd M, Frampton C, Fleming T, Lucassen MF. The effectiveness of SPARX, a computerised self help intervention for adolescents seeking help for depression: randomised controlled non-inferiority trial. BMJ 2012; 344: e2598.
• 35 Walshe DG, Lewis EJ, Kim SI, O’Sullivan K, Wiederhold BK. Exploring the use of computer games and virtual reality in exposure therapy for fear of driving following a motor vehicle accident. Cyberpsychol Behav 2003; 6: 329-334.
• 36 Santamaría JJ, Soto A, Fernandez-Aranda F, Krug I, Forcano L, Gunnard K, Kalapanidas E, Lam T, Raguin T, Davarakis C, Menchon JM, Jimenez-Murcia S. Serious games as additional psychological support: A review of the literature. J Cyber Ther Rehabil 2011; 4: 469-476.
• 37 Silver M, Oakes P. Evaluation of a new computer intervention to teach people with autism or Asperger syndrome to recognize and predict emotions in others. Autism 2001; 5: 299-316.
• 38 Du S, Martinez AM. The resolution of facial expressions of emotion. J Vis 2011; 11: 24.
• 39 Biehl M, Matsumoto D, Ekman P, Hearn V, Heider K, Kudoh T, Ton V. Matsumoto and Ekman‘s Japanese and Caucasian Facial Expressions of Emotion [JACFEE]: Reliability Data and Cross-National Differences. J Nonverbal Behav 1997; 21: 3-21.
• 40 Palermo R, Coltheart M. Photographs of facial expression: accuracy, response times, and ratings of intensity. Behav Res Methods Instrum Comput 2004; 36: 634-638.
• 41 Kohler CG, Turner TH, Bilker WB, Brensinger CM, Siegel SJ, Kanes SJ, Gur RE, Gur RC. Facial emotion recognition in schizophrenia: intensity effects and error pattern. Am J Psychiatry 2003; 160: 1768-1774.
• 42 Elfenbein HA, Ambady N. Universals and cultural differences in recognizing emotions. Psychol Sci 2003; 12: 159-164.
• 43 Peterson LR, Peterson MJ. Short-term retention of individual verbal items. J Exp Psychol 1959; 58: 193-198.
• 44 Karpicke JD, III HLR. Repeated retrieval during learning is the key to long-term retention. J Mem Lang 2007; 57: 151-162.
• 45 Roediger HL, Karpicke JD. Test-enhanced learning: taking memory tests improves long-term retention. Psychol Sci 2006; 17: 249-255.
• 46 Sachs G, Steger-Wuchse D, Kryspin-Exner I, Gur RC, Katschnig H. Facial recognition deficits and cognition in schizophrenia. Schizophr Res 2004; 68: 27-35.
• 47 Bozikas VP, Kosmidis MH, Anezoulaki D, Giannakou M, Karavatos A. Relationship of affect recognition with psychopathology and cognitive performance in schizophrenia. J Int Neuropsychol Soc 2004; 10: 549-558.
• 48 Bryson G, Bell M, Lysaker P. Affect recognition in schizophrenia: a function of global impairment or a specific cognitive deficit. Psychiatry Res 1997; 71: 105-113.