Computer-Based Training Program to Facilitate Learning of the Relationship between Facial- Based and Situation-Based Emotions and Prosocial Behaviors

  • cc icon

    Individuals with autistic spectrum disorders (ASD) have difficulty inferring other people’s feelings from their facial expressions and/or from situational cues, and therefore, they are less able to respond with prosocial behavior. We developed a computer-based training program to help teach the connection between facial-based or situation-based emotions and prosocial behavioral responses. An 8-year-old male school child with ASD participated in the study. In this program, he was trained to identify persons in need of help and appropriate prosocial responses using novel photobased scenarios. When he misidentified emotions from photographs of another’s face, the program highlighted those parts of the face which effectively communicate emotion. To increase the likelihood that he would learn a generalized repertoire of emotional understanding, multiple examples of emotional expressions and situations were provided. When he misidentified persons expressing a need for help, or failed to identify appropriate helping behaviors, role playing was used to help him appreciate the state of mind of a person in need of help. The results of the training indicated increases in prosocial behaviors during a laboratory task that required collaborative work. His homeroom teacher, using a behavioral rating scale, reported that he now understood another’s emotion or situation better than before training. These findings indicate the effects of the training are not limited to the artificial experiment situation, but also carried over to his school life.


    Autistic Spectrum Disorders , Situation-Based Emotions , Facial-Based Emotions , Prosocial Behavior , Social Skill Training , Computer Based Training Program


    Prosocial behavior is a set of social skills that contributes to mutually beneficial human relationships. The term prosocial behavior describes voluntary acts that typically are helpful to others. Prosocial behavior often is motivated by empathic concern for the other person (Decety and Grezes, 2006). Recent brain imaging studies using functional magnetic resonance imaging (fMRI) have demonstrated that brain activity in response to the perceived pain of others can predict the actual amount of a charitable donation (Ma et al., 2011; Mathur et al., 2010). These results indicate that empathy for the pain of others can motivate prosocial behavior from the standpoint of brain science.

    Persons with an autistic spectrum disorder (ASD) have difficulty demonstrating empathic concern for others. Schulte-Ruther et al. (2011) reported that certain brain activity and corresponding empathic behavioral responses of participants with ASD were weaker than those of unaffected participants. Because of these brainbased behavioral differences, persons with ASD have trouble initiating prosocial behavior (Crozier and Tincani, 2007). Therefore, it is necessary to develop training programs regarding prosocial cognition and behavior for children with ASD as a strategy to support their social adjustment (Bosseler and Massaro, 2003).

    For this study, we developed computer-based social skill training (SST) programs to teach the relation between facial expressions of emotion and/or situational cues and appropriate prosocial responses that facilitate the goal attainment of others. We focused on the detection of another’s pain because children with ASD have trouble inferring painful emotions from facial expressions when interacting with others (Dapretto et al., 2006; Harms et al., 2010). The ability to detect another’s pain is a necessary precursor to responding prosocially, as painful emotion indicates a need for help. However, children with ASD have difficulty detecting another’s pain from not only facial expressions but also situational cues (Riby and Hancock, 2008). Therefore, the SST programs included two components that enhance the ability of children with ASD to infer another’s feeling state from facial expressions and situations. One was learning a correspondence between facial expressions and emotion, and the other was learning a correspondence between situations and prosocial responses. The SST programs also included role-playing in which the child with ASD was trained to infer another’s feelings by reproducing a scene where he caused someone difficulty, and then putting himself into the other’s position.

    Computer-based SST programs have the potential to reduce the demand on specially trained teachers and to transcend the limits of the school schedule. This study reports the ability of novel computer-based SST programs to teach prosocial behaviors to a child with ASD.

    2. METHOD

       2.1 Participant

    An 8-year-old male school child with ASD participated in this study. He attended an elementary school regular class and a conductive education class on the campus of a Fukui University "Tanpopo classroom." He had been evaluated and diagnosed with ASD at age 3 by a pediatrician (the fifth author of this study). His verbal IQ was 56, performance IQ was 87, and full scale IQ was 68 based on the Wechsler Intelligence Scale for Children-III (Wechsler, 1991).

       2.2 Study Duration and Settings

    The duration of the study was from September 2010 to January 2011, and it included three phases: baseline determination, training, and follow-up. There were three baseline sessions, six training sessions, and three post training, follow-up sessions. During the baseline and post sessions (one 30-minute session each per week), the participant’s helping or altruistic traits were measured based on the frequency of prosocial behaviors observed in laboratory settings before training. The frequency of prosocial behaviors was measured during a manual training activity (e.g., paper craft), a school-like activity which required collaborative work. During training sessions (two 60-minute sessions per week for three weeks), the participant was trained to identify the facial expression of persons who needed help, and appropriate prosocial responses, using computer-based SST programs and role-playing. Then the frequency of prosocial behaviors was compared between the baseline and post training sessions.

       2.3 Contents of Training

    The computer-based social skills training material was uniquely developed for this study; and the program consisted of two parts, assessment and training.

    2.3.1 Assessment

    The computer program asked the participant to choose the correct option in two types of quizzes by using a touch panel. The number of options was from four to six. The presentation of images, questions, answer options, and a record of the participant’s responses were controlled by Microsoft Visual Basic 2008.

    There were twenty quizzes about inferring another's feeling state from images of facial expressions, consisting of five people (two males and three females) × four kinds of expressions (joy, sadness, surprise, and disgust). Figure 1 shows an example. The question asked was "How does this person feel?". The participant was asked to select the most appropriate option which represented the emotion of the displayed facial expression.

    There also were 18 quizzes about identifying another’s need for help and appropriate expected prosocial behavior in a given situation. In these quizzes, images showed three people engaged in some sort of activity. Figure 2 shows one of the quizzes. In this case, two students are reading their text books. However, one of them has dropped her pencil case and her facial expression is intended to show distress. The quiz asked the following questions: (a) Who is unhappy? (b) Why is the person unhappy? (c) How should you act to help the person? To assess the ability of the participant to detect another’s pain accurately, we also prepared five images in which no one was unhappy. If the participant could detect whether someone is unhappy or not, he would be expected to select the "none" option to the question "Who is unhappy?" for those images.

    The number of correct responses to the quizzes was used to assess the effects of training. We set a criterion of 80% correct on each quiz to indicate mastery. When the participant achieved mastery regarding the facial expressions, the prosocial situation training began. When the participant achieved mastery regarding prosocial behavior, the training sessions were finished.

    2.3.2 Training

    When the participant made a mistake inferring another’s emotion from the depicted facial expressions, an image which highlighted specific parts of the face was immediately shown (Figure 3). Those parts of the facial expression that are critical to distinguish emotions were highlighted. Similarly, when the participant made mistakes in identifying another’s need for help and what the appropriate prosocial behavior would be, he was asked to role-play or imagine what the person in the image would be feeling. When the participant selected a correct option, a fanfare was played.

       2.4 Outcome Measures

    We determined the frequency of occurrence of the participant’s prosocial behaviors during a manual training activity in laboratory settings. The activity was videotaped and then two reviewers recorded whether the participant provided an appropriate prosocial behavioral response within 30 seconds of a request for help. Table 1 shows behaviors that were checked as correct prosocial responses. In addition, a behavioral survey was administered before (baseline) and after the training sessions to help determine skill acquisition and generalization to a "real world" setting. A teacher recorded the frequency of the participant’s in-classroom prosocial behaviors. The questionnaire used (Ueno and Okada, 2006) is composed of 48 items and is designed to assess social skills (collective behaviors, self-control skills, peer relationship skills, and communication skills). Collective

    behaviors are characterized by manners, situational comprehension, participation in groups, and role performance. Self-control skills include emotional control and behavioral control. Peer relationship skills are characterized by the establishment and maintenance of peer relationships. Communication skills include listening, speaking, assertion, and conversation skills. Each item was rated on a Likert-type scale from 0 to 3 according to how frequently the child demonstrated the behaviors.

    3. RESULTS

       3.1 Performance of Quizzes about Another’s Emotion and Prosocial Behavior

    Table 2 shows correct response rates for the quizzes about another’s feelings based on facial expression. On all training session days, correct rates for facial expression of joy and surprise were greater than or equal to 80%. However, correct rates for facial expressions of disgust (i.e., painful emotions) were less than 80% on the post-training first and second days. By the third day, correct rates for all facial expressions reached the mastery criteria after training.

    Table 3 shows correct response rates for quizzes regarding prosocial behavior. By the third day, correct rates reached the mastery criteria at both pre- and post-


       3.2 Occurrence of Prosocial Behaviors During Manual Training Activity

    Table 4 shows the occurrence of prosocial behaviors exhibited during a manual training activity in a laboratory setting. None of the prosocial behaviors were observed in the baseline sessions. However, three prosocial behaviors were observed in the training and followup sessions.

       3.3 Teacher Ratings of Classroom Social Skills

    Figure 4 shows the classroom teacher’s baseline and follow-up scores on the questionnaire about social skills. Scores regarding skills related to “situation comprehension” and “establishment of peer relationships” showed relatively large changes (more than 0.6 pt). While the scores for situation comprehension skills were higher

    after the training, scores for establishment of peer relationship skills decreased.


    The results of this study indicate increases in the frequency of prosocial behaviors by a child with ASD after training with the computer-based SST programs. Specifically, the participant who did not behave prosocially during the baseline sessions became able to show some prosocial behaviors during the manual training activity in the laboratory setting after training. In addition, the participant’s homeroom teacher indicated that he became better able to comprehend another’s emotion from situational cues after the training. These results indicate that the effects of the training were not limited to the experimental setting, but also carried over to his school setting. Therefore, it is possible that the computer- based SST programs can assist in the learning of prosocial behaviors.

    It is important to emphasize the value of learning to detect another person’s feelings from their facial expressions. In our computer-based SST programs, increases in prosocial behavior began after learning the correspondence between another’s emotion and facial expression. The ability to detect painful emotions from facial expressions is especially important because pained facial expressions can be seen as a request for prosocial behavior from another.

    It is also important that the method of feedback during the learning of facial expressions took into consideration the cognitive style of persons with ASD. Such persons are thought to use local, feature-based information processing, in contrast to the global, configuralbased strategy used by individuals with normal development (Harms et al., 2010; Spezio et al., 2007). In our computer-based SST programs, we highlighted the parts of the face (i.e., local features), anticipating that that method of feedback would be more effective for subjects with ASD.

    However, a side effect of our SST programs was also observed. The participant’s homeroom teacher reported that he became more passive regarding establishing peer relationships after the training. This could be interpreted as a result of the enhancement of the ability to better comprehend situations. As the participant became better able to comprehend another’s emotional state from situational cues, he may have erred on the side of caution regarding the establishment of a peer relationship. Further work is underway to assess the correctness of this interpretation. It also is necessary to verify further the effects of the training programs by increasing the number of participants.


    We developed a novel computer-based training program to help teach the connection between facialbased or situation-based emotions and prosocial behavioral responses for persons with ASD. An 8-year-old boy with ASD was trained to identify persons in need of help and appropriate prosocial responses using novel photo-based scenarios. His homeroom teacher estimated that he understood another’s emotion or situation better than before the training. This result indicates that the effects of the training were carried over to his school life. However, the single-participant design limits the generalizability of the training effects across the range of autistic children. Additional large scale, group design studies which include children of different ages and functioning levels are needed to replicate the results and to determine expected traing effects across a wider range of children with ASD.

  • 1. Bosseler A., Massaro D. W. (2003) Development and evaluation of a computer-animated tutor for vocabulary and language learning in children with autism [Journal of Autism and Developmental Disorders] Vol.33 P.653-672 google
  • 2. Crozier S., Tincani M. (2007) Effects of social stories on prosocial behavior of preschool children with autism spectrum disorders [Journal of Autism] Vol.37 P.1803-1814 google
  • 3. Dapretto M., Davies M. S., Pfeifer J. H., J. H. A. A., Sigman M., Bookheimer S. Y., Iacoboni M. (2006) Understanding emotions in others: mirror neuron dysfunction in children with autism spectrum disorders [Nature Neuroscience] Vol.9 P.28-30 google
  • 4. Decety J., Grezes J. (2006) The power of simulation: imagining one’s own and other’s behavior [Brain Research] Vol.1079 P.4-14 google
  • 5. Harms M. B., Martin A., Wallace G. L. (2010) Facial emotion recognition in autism spectrum disorders: a review of behavioral and neuroimaging studies [Facial emotion recognition in autism spectrum disorders:] Vol.20 P.290-322 google
  • 6. Ma Y., Wang C., Han S. (2011) Neural responses to perceived pain in others predict real-life monetary donation in different socioeconomic contexts [Neuroimage] Vol.57 P.1273-1280 google
  • 7. Mathur V. A., Harada T., Lipke T., Chiao J. Y. (2010) Neural basis of extraordinary empathy and altruistic motivation [Neuroimage] Vol.51 P.1468-1475 google
  • 8. Riby D. M., Hancock P. J. (2008) Viewing it differently: social scene perception in Williams syndrome and autism [Neuropsychologia] Vol.46 P.2855-2860 google
  • 9. Schulte-Ruther M., Greimel E., Markowitsch H. J., Kamp-Becker I., Remschmidt H., Fink G. R., Piefke M. (2011) Dysfunctions in brain networks supporting empathy: an fMRI study in adults with autism spectrum disroders [Social Neuroscience] Vol.6 P.1-21 google
  • 10. Spezio M. L., Adolphs R., Hurley R. S., Piven J. (2007) Abnormal use of facial information in highfunctioning autism [Journal of Autism and Developmental Disorders] Vol.37 P.929-939 google
  • 11. Ueno K., Okada S. (2006) Special Needs Education- Practical Social Skill Manual google
  • 12. Wechsler D. (1991) WISC-III (Wechsler Intelligence Scale for Children) Manual google
  • [Figure 1.] Quiz about other's emotion.
    Quiz about other's emotion.
  • [Figure 2.] Quiz about prosocial behavior.
    Quiz about prosocial behavior.
  • [Figure 3.] Instruction material.
    Instruction material.
  • [Table 1.] Prosocial behaviors during manual training activity
    Prosocial behaviors during manual training activity
  • [Table 2.] Correct rates of quizzes about other’s emotion
    Correct rates of quizzes about other’s emotion
  • [Table 3.] Correct rates of quizzes about prosocial behavior
    Correct rates of quizzes about prosocial behavior
  • [Table 4.] Occurrence of prosocial behaviors during manual training activity
    Occurrence of prosocial behaviors during manual training activity
  • [Figure 4.] Scores of the questionnaire about social skills.
    Scores of the questionnaire about social skills.