UBC Theses and Dissertations

UBC Theses Logo

UBC Theses and Dissertations

Judgment of feeling states from facial behavior: a bottom-up approach Snodgrass, Jacalyn D. 1992

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Item Metadata


831-ubc_1993_spring_phd_snodgrass_jacalyn.pdf [ 4.43MB ]
JSON: 831-1.0086368.json
JSON-LD: 831-1.0086368-ld.json
RDF/XML (Pretty): 831-1.0086368-rdf.xml
RDF/JSON: 831-1.0086368-rdf.json
Turtle: 831-1.0086368-turtle.txt
N-Triples: 831-1.0086368-rdf-ntriples.txt
Original Record: 831-1.0086368-source.json
Full Text

Full Text

JUDGMENT OF FEELING STATES FROM FACIAL BEHAVIOR A BOTTOM-UP APPROACH by JACALYN SNODGRASS B.A., North Carolina State University, 1980 M.A., University of British Columbia, 1985 A THESIS SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY in THE FACULTY OF GRADUATE STUDIES Department of Psychology  We accept this thesis as conforming  THE UNIVERSITY OF BRITISH COLUMBIA June 1992 © Jacalyn Dianne Snodgrass, 1992  In presenting this thesis in partial fulfilment of the requirements for an advanced degree at the University of British Columbia, I agree that the Library shall make it freely available for reference and study. I further agree that permission for extensive copying of this thesis for scholarly purposes may be granted by the head of my department or by his or her representatives. It is understood that copying or publication of this thesis for financial gain shall not be allowed without my written permission.  (Signature)  Department of  ^Ps  C_  c / on  The University of British Columbia Vancouver, Canada Date^3D  DE-6 (2/88)  C.; c 9' 2  Abstract A series of studies was conducted to examine the feasiblity of a bottom-up approach to the study of judgment of affective feeling states from facial behavior. Previous work on the judgment of emotion from facial expressions has taken a more top-down approach; observers judged the emotional meaning of a wholistic facial expression. Individual facial movements have sometimes then been identified within that complex expression, but the meaning of those individual movements has not been studied. A bottom-up approach begins by exploring the meaning of individual facial movements instead of complex facial expressions. In this approach the relationship between the emotional meaning of individual facial movements and complex facial expressions can be explored. It is argued that such an approach has the potential to explain judgment of not only a limited set of basic emotional expressions, but the full range of emotionally tinged feelings that individuals both experience in themselves and judge in others. Individual action units, as scored by Ekman and Friesen's (1978) Facial Action Coding System (FACS), and selected combinations of action units were presented to observers in three pairs of studies. Filmstrip sequences were used in the first pair of studies, and still photographs in the other two pairs. In the first study of each pair, observers judged the degree of pleasure and arousal expressed by the face. In the second study of each pair, observers rated how well each of a set of emotion terms described the feeling expressed by the face. Observers were found to reliably attribute meaning to individual action units on both scales. Additionally, pleasure and arousal judgments predicted emotion term ratings. The meaning attributed to combinations of action units was found to be related to the meanings of the individual action units occuring alone.  Resultant ratings were shown to be meaningful within a dimensional model of emotion space.  Table of Contents Abstract List of Tables List of Figures^  vi  Introduction^  1  Accuracy Studies^  6  Studies on How Judgments are Made^  13  A System for Describing Facial Movements ^  17  A Framework for Judgments of Feeling States^  20  A New Approach^  23  Overview of Six Studies^  26  Studies One and Two^  29  Studies Three and Four^  41  Studies Five and Six^  50  Conclusion^  57  References^  67  Appendix^  100  iv  List of Tables Table 1.^Action units scorable in the Facial Action Coding System. ^74 Table 2.^Combinations of action units predicted by Ekman and Friesen to express seven basic emotions. ^  75  Table 3.^Ekman filmstrip series action units used in Studies One and Two.^  76  Table 4.^Mean pleasure and arousal ratings of initial and ending poses in Ekman filmstrip sequences in Study One. ^  77  Table 5.^Calculated F ratios for sequence by time interaction in Study One.^  78  Table 6.^Changes in pleasure and arousal ratings of Ekman's filmstrip sequences in Study One.^  79  Table 7.^Mean category ratings of initial poses in filmstrip sequences in Study Two. ^  80  Table 8.^Mean category ratings of ending poses in filmstrip sequences in Study Two.^  80  Table 9.^Calculated F ratios for sequence by time interaction in Study Two.^  81  Table 10.^Beta weights for pleasure and arousal scores in Study One as predictors of category judgments in Study Two, and beta weights for pleasure and arousal scores from Study Three as predictors of category judgments in Study Four.^  82  Table 11.^Changes in category ratings of filmstrip sequences in Study Two.^  83  Table 12. ^Mean pleasure and arousal ratings of photographs in Study Three, in which action units were selected to vary along the arousal dimension.^  84  Table 13.^Mean emotion category ratings of photographs in Study Four.^  85  Table 14.^FACS scores of the photograph series used in Studies Five and Six, with predicted pleasure ratings.^  86  Table 15.^Mean pleasure and arousal ratings of photographs in Study Five.^  87  Table 16.^Mean category ratings of photographs in Study Six. ^  88  V  List of Figures Figure 1.^Circumplex model of emotion space. ^  89  Figure 2.^Mean pleasure and arousal scores of action unit sequences in Study One.^  90  Figure 3.^Mean pleasure and arousal ratings of ending single action unit sequences in Study One.^  91  Figure 4.^Vector plot of mean ratings of action unit combinations and components in Study One.^  92  Figure 5.^Mean pleasure and arousal ratings of photographs in Study Three. ^93 Figure 6.^Mean pleasure and arousal ratings of photographs in Study Five.^94 Figure 7.^Changes in pleasure ratings of photographs in Study Five plotted against intensity changes in action unit 12. ^  95  Figure 8.^Changes in pleasure ratings of photographs in Study Five plotted agianst intensity changes in action unit 15. ^  96  Figure 9.^Rating of descriptiveness of label "Excited" to photographs in Study Six plotted against pleasure ratings of same photographs from Study Five.^  97  Figure 10.^Rating of descriptiveness of label "Afraid" to photographs in Study Six plotted against pleasure ratings of same photographs from Study Five.^  98  Figure 11.^Rating of descriptiveness of label "Surprised" to photographs in Study Six plotted against pleasure ratings of same photographs from Study Five.^  99  vi  Introduction When psychology was first established as an independent discipline in the late 1800s the central task for many researchers was to discover how people come to know the world. Early psychologists such as Wilhelm Wundt tried to use scientific methods to describe the processes by which objective reality is  translated into subjective experience. Interpretation of facial expressions of emotion became one arena for this new science, as researchers attempted to discover the laws that linked facial movements to judgments of feeling states 1 (Boring & Titchner, 1923; Buzby, 1924; Feleky, 1914; Frois-Wittman, 1930; Gates, 1925; Jarden & Fernberger, 1926; Landis, 1924, 1929; Langfeld, 1918a, 1918b; Ruckmick, 1921; Sherman, 1927). Explaining how observers interpret facial expressions of emotion was a particularly challenging problem for the fledgling science since what was being studied was how observers arrive at subjective interpretations of another's subjective experience. Unlike the study of perception of color, sound, or temperature, or the judgment of weight, size or distance, there was no way of objectively measuring a corresponding physical dimension. Not only could actual emotions not be measured objectively, there was no clear understanding of the thing being judged. Although emotions had been written about since at least the time of Aristotle, there was as yet no agreement on such basic questions as the number of different emotions, or their relationship to each other.  1 "Feeling states" is used to refer to emotionally tinged conscious experieces (as opposed to nonemotional sensations and thoughts). This term is used instead of "emotion" since researchers don't agree on exactly which feeling states are emotions. "Feeling states" includes both those states which researchers agree are emotions and other related states such as moods.  1  Nevertheless, in the late 1800s and early 1900s it was generally accepted that facial behavior ja expressive of underlying internal states, and that observers, at least sometimes, recognize these expressions. This belief can be seen in the writing of the early researchers on emotion published in the psychological literature, as well as in previous literature in related fields. For example, in his work identifying how visible changes in the face were produced by contraction of the various facial muscles, Duchenne de Boulogne (1862/1990) electrically stimulated individual facial muscles and photographed the resulting expressions. So sure was he that emotional expressions were recognizable that he simply labeled the resulting photographs according to the feeling state that seemed lo him to be expressed in the face, occasionally verifying his judgments against the judgments of other observers. A similar attitude can be seen in the conclusion of Langfeld's (1918a) research on judgments of emotion from posed facial expressions, "It did not need this experiment to prove the well known fact that emotions and attitudes can be judged from pictures" (p. 183). In fact, according to the creationist view of emotional expression, humans were created with certain muscles specially adapted for the expression of feelings (e.g., Bell, 1806). Although Darwin (1872/1965) challenged the view that facial muscles were created to express feelings, he seemed to agree that facial behaviors were expressive. In The Expression of the Emotions in Man and Animals (1872/1965), Darwin wrote, "It seemed probable that the habit of  expressing our feelings by certain movements, though now rendered innate, had been in some manner gradually acquired" (p. 19). This belief that facial (and other) behavior offered a window into an organism's inner state is also apparent in Darwin's reference to "expressions and gestures involuntarily used  2  by man and lower animals, under the influence of various emotions and sensations" (p. 27). Darwin showed evidence that expressive behavior was similar in children and adults, in members of different cultures, and in individuals who were born blind, and concluded that the expression of emotions is both stereotypal and innate. He did note evidence, however, that at least some expressions could be suppressed and some could be produced at will in the absence of the associated feeling. Darwin also argued that the ability to recognize expressions was innate, but he offered little evidence on this point. It is not my purpose here to evaluate Darwin's work, but merely to note its influence on subsequent psychological research on the question of judging feeling states from facial behavior. There are a number of questions that could be asked about the communication of feeling states via facial behavior, including the following: What facial behaviors accompany various feelings? How accurate are observers in judging feelings from facial behavior? How do observers make judgments about feelings from facial behavior? The first question has received very little attention in the psychological literature, perhaps because it seemed settled (although the first researcher studying this question after Darwin concluded that there were a lot of individual differences in expression of emotion, Landis, 1924). The bulk of the psychological research on communication of feeling states via facial behavior has focused on the accuracy of observers. After nearly a century of research, it is still not clear exactly how accurate observers are in judging feeling states from facial behavior. This research is reviewed in a later section. Although the question of observer accuracy has dominated research on judgment of feeling states from facial behavior, some researchers have 3  simultaneously attempted to discover how observers are making their judgments (e.g., Langfeld, 1918a). Others have recognized that the question of how observers make inferences about feelings from facial behavior is an interesting question on its own, and can be addressed independently of questions, or assumptions, about the accuracy of such inferences (e.g., FroisWittman, 1930). It is the latter approach that is most similar to the present program of research. Even though the underlying question is how information about one person's feeling state is transmitted to another person via the face, it may be better answered by empirically separating the study of the relationship between judged feeling and facial behavior from any assumptions about the relationship between feelings and facial behavior. In other words, the focus here is on the receiver-message relationship, leaving aside for the time being the sender-message relationship. This approach is not new, but earlier attempts to relate judgments of feeling states to particular facial movements suffered from a lack of tools for measuring facial movement, as well as a lack of appropriate conceptual framework for judgments. Separate lines of research in the intervening years have now provided these tools. The present research is an integration of relatively new and sophisticated methods for measuring facial movement and a specific framework for describing judgments of feeling states. These tools help overcome the earlier obstacles to the exploration of the relationship between facial movement and judgment of emotion. In the following sections I will briefly review first the accuracy research, and then the few studies directed at discovering how judgments are made. I will then describe a method for objectively describing facial behavior, and a framework for judgments of feeling states. Finally I report new research that shows a systematic relationship  4  between individual facial movements and judgments of feeling states, interpretable within a structural model of emotion.  5  Accuracy Studies The belief in facial expressions as innate manifestations of underlying feeling states seems to have led researchers studying the perception of facial expressions in the first two decades of the twentieth century to assume that the face contained valid information about its owners feeling state, and so to focus on the accuracy of observers' judgments. Darwin (1872/1965) had begun this research, but his influence on researchers such as Feleky (1914) and Langfeld (1918a, 1918b) can only be inferred since neither cited his work. In fact, Feleky did not cite any prior theoretical or empirical work, and Langfeld cited only McDougal's (1908) descriptions of the eliciting stimuli for various emotions. Later researchers (e.g. Ekman, Friesen & Ellsworth, 1982) have tried to answer questions about the universality of facial expressions and the ability to interpret them simultaneously. A major problem with research on the accuracy of judgment of facial behavior is in what is held to be a correct judgment. Accuracy has been variously defined as producing or selecting the same or similar label as given by an actor posing an emotion, producing or selecting the same or similar label for facial behavior as its producer named for the experience, identifying the situation to which a photographed or filmed person was responding, or as producing or selecting the same or similar label as other observers viewing the same facial behavior. Early research seemed to show that observers are at least somewhat accurate in judging emotion from facial behavior (e.g., Darwin, 1872; Feleky, 1914; Langfeld, 1918a). Interestingly, the first of these studies used neither posed nor naturally occuring facial behavior, but photographs of a man's face in which expressions had been produced by electrically stimulating the muscles of the face. The photographs were produced by Duchenne de Boulogne 6  (1862/1990) as part of an innovative program of research aimed at discovering the relationship between the musculature of the face and visible facial behavior. Duchenne believed that facial behavior was expressive of emotion, and so labelled the resulting photographs according to the emotion he believed was expressed in each. In his studies of observer accuracy, Darwin showed some of the Duchenne photographs to "above twenty educated persons of various ages and both sexes" (Darwin, 1872/1965, p. 14). Darwin reported that, "Several of the expressions were instantly recognised by almost everyone .. . On the other hand, the most widely different judgments were pronounced in regard to some of them" (p.14). In Darwin's research a correct recognition consisted of giving the same or similar label to a photograph as Duchenne had given. Similar results showing both accuracy and inaccuracy were reported by Langfeld (1918a), who showed six subjects sketches of an actor posing emotions. While noting the tendency to interpret some emotions as related emotions (e.g., fear as anger), Langfeld concluded that the judgments were "uniformly good and consistent." Later researchers using spontaneous facial behavior concluded that observers are less accurate (e.g., Landis, 1929). Noting that most previous research had been done with posed facial behavior, Carney Landis (1924, 1929) studied whether observers could interpret spontaneous facial behavior. In a creative set of studies (which would probably not be approved by an ethics committee today) Landis photographed subjects as they were exposed to stimuli expected to elicit emotional reactions. The situations to which subjects were exposed in the laboratory included viewing explicit sexual photographs, being shocked, having a firecracker suddenly explode beneath their chair, having the experimenter read aloud their written descriptions of the meanest or 7  most embarrassing thing they had ever done, unknowingly putting their hands in a bucket of frogs or human brains, and cutting the head off a live white rat with a dull knife. Subjects were then asked to describe what they were feeling. Landis found there were large individual differences in facial response to the situations. These findings were the first published challenge to the assumption the facial expression of emotion is universal and innate. Landis also found that in attempts to later reproduce their original expressions subjects tended to produce more conventional responses. On this basis he suggested that, rather than having an innate ability to produce and recognize facial expressions, people may learn to associate meaning with facial expressions through cultural experience. The important role of culture in determining expression of emotion was later emphasized by anthropologists such as LaBarre (1947) and Birdwhistell (1970). If Landis's subjects learned the meanings of facial expressions through cultural experience, they apparently did not learn those meanings very well. Landis (1929) selected photographs from both the spontaneous and reproduced expressions that he believed were "very expressive," and measured observer accuracy in interpreting those expressions. On these attempts Landis reported, "The most outstanding fact brought out in this study is that observers cannot make a statement more accurate than one would expect by chance concerning the emotion which a subject was undergoing when photographed or concerning the probable situation which might give rise to such an expression." (Landis, 1929, p. 67) In a relatively early review of the above and related literature, Bruner and Tagiuri (1954) identified several inconsistencies in procedures used by various researchers that prevented a simple summation of their findings: 1) the number of facial expressions judged and their similarity or dissimilarity to one another, 8  2) the response format used, 3) constraints on judgment provided by the experimental context, and 4) the number of models of emotional expressions. On an observers ability to recognize emotion in the face they concluded, "We must come to the chastening conclusion that the literature is sufficiently haphazard to preclude a simple answer to this question. It depends on the difference in the emotions being expressed, upon the number and kinds of categories in terms of which judgment must take place, upon the amount of contextual information given the subject" (p. 638). In a later review of this same research, Ekman, Friesen and Ellsworth (1982) argued that the evidence is heavily weighted toward the conclusion that observers erg accurate. While Bruner and Tagiuri suggested that the inconsistent findings of the early research were due to widely different methods, Ekman et al. argued that only those studies showing accuracy used sound methods. For example, they discounted a number of studies because the researchers used sketches instead of photographs of faces. Although they argue against the use of posed expressions, Landis's (1924, 1929) work using photographs of subjects in emotion-inducing situations was discounted in part because still photographs were used instead of motion pictures, and thus the critical expression might have been missed. The differing conclusions of these authors seem to be due to differences in what is meant by saying observers are accurate in judging facial expressions. Bruner and Tagiuri (1954) seem to use the term "accuracy" to mean that observers judge faces as showing the same emotion as intended or felt by the person whose face is being judged. They concede that in some judgment situations accurate inferences about emotional state might be made from facial information, but point out that there are many situations in which the face alone,  9  while clearly providing information about emotion, is not sufficient for pinpointing an exact emotional state. Ekman and associates were primarily concerned with demonstrating that the face provides useable information about its owner's emotional state, as illustrated by their opening statement: "The question of whether the face can provide accurate information about emotion has been the central issue since the beginning of research on the face." Although demonstrating that the face does contain useable information about emotion is clearly the concern of Ekman et al., it is not the question addressed by all other researchers. The earliest researchers (e.g., Langfeld, 1918) simply assumed that the face was expressive of emotion, and set about trying to measure observers' accuracy in interpreting those expressions. Using the term "accurate" both for the ability of observers to make consistently correct judgments, and as a description of the information available in the face is confusing. The literature shows that observers' judgments about emotion from facial behavior sometimes coincide with the reports of the individual being observed and sometimes do not. In essence, Bruner and Tagiuri (1954) look for an answer to the question, "Are observers accurate?", and find in the literature the nonanswer—it depends. Ekman et al. (1982) look to the same literature for an answer to the question, "Does the face ever provide valid information about its owner's emotional state?", and find an unqualified yes. Once these two points of view are translated into the same language there really is no disagreement— except on the proper use of the term "accurate." The face does, at least sometimes, provide information that allows an observer to make valid inferences about another person's emotional state. Whether or not one takes the occasional valid inference as evidence of accuracy depends on whether  10  one is interested in how accurate observers are, or in the availability of information in the face. Perhaps because the former question proved intractable, researchers in recent years have abandoned it in favor of the latter. Effort has been focused on what has come to be called the universality question--that at least some emotions are communicated in all cultures by identical facial displays. An impressive number of studies have shown cross-cultural agreement on the interpretation of a small number of expressions of basic emotions (Boucher & Carlson, 1980; Ducci, Arcuri, Georgis & Sineshaw, 1982; Ekman, 1972; Ekman, Sorenson & Friesen, 1969; Ekman et al., 1987; Izard, 1971; McAndrew, 1986; Niit & Valsiner, 1977); although there is no consensus on exactly which emotions are basic, and the concept of basic emotions has itself been questioned (Ortony & Turner, 1990). Proponents of universality explain the failure of individuals to always show identical facial behavior in similar circumstances to cultural influences on both the appropriate emotions to feel in a given situation, and on the appropriate emotions to show in a given situation (Klineberg, 1940; Ekman, 1972). Culture is seen not as the source of facial expressions of feeling states, but as a mask that obscures the expressions all human beings have in common. The question of observer accuracy in judging feeling states from facial behavior remains unanswered. Sometimes observers are accurate and sometimes they are not, depending on the criteria for accuracy, on the specific judgment they are asked to make, and on the other information available. If the judgment task is restricted to matching a small number of labels to a few carefully chosen expressions, observers are quite accurate. Unfortunately this accuracy is demonstrated by eliminating expressions on which there is less  11  consensus, thus simultaneously eliminating the possibility of measuring the extent of observer accuracy.  12  Studies on How Judgments are Made A number of questions can be asked about the process of judging feeling states from facial behavior, including the following: What information in the face is used? What meaning do observers attribute to various facial behaviors? What other factors influence judgments in addition to facial behavior? What cognitive processes mediate judgments? Early attempts to explore the process of making judgments about emotion in facial expression (as opposed to the accuracy of those judgments) have taken two general forms, 1) asking subjects how they are making their judgments, and 2) attempting to relate variations in judgments to variations in the stimulus set. The introspective reports of Langfeld's subjects illustrate the first method (Langfeld, 1918a). Following is the account of one subject:  "Frequently the expression was visualized as appearing upon the face of some person of my acquaintance, and the effort was made to estimate the circumstance that would call forth that expression. Where the picture shown represented an expression of common occurrence or of not too detailed a character, this general idea was enough to indicate the estimate of the picture, the judgment being rendered very promptly. A few times I was conscious of a definite attempt to reproduce the expression in my own countenance in order that the resultant lines and sensations might aid in getting the meaning. Frequently the expression was analysed, the effort being made to determine the meaning of the eyes by themselves, then the mouth, etc. The result would sometimes be a unit, often a positive, complex description. The judgment was frequently made by trying to imagine what was before the eyes of the 13  man in the picture, that is, trying to imagine the circumstances which would be apt to produce such a facial reaction." While introspective reports are limited in the amount of insight they can give us into perceptual and cognitive processes that are often unconscious, this one is rich with hypotheses. This observer mentions more than once trying to imagine a situation that might bring forth such an expression. This may indeed be one of the later steps in interpreting facial movements, but as Langfeld points out, imagining situations in which the expression might be shown is not sufficient to explain judgment since an initial analysis is a necessary prerequisite to imagining an appropriate context. This initial judgment still needs explaining. An alternate possibility suggested by this observer is that mimicking the expression may somehow access the associated emotion. This suggestion is reminiscent of the James-Lange theory of emotion (James, 1890), or what has come to be known in later years as the facial feedback hypothesis (Tomkins, 1962, 1963; Laird, 1974). This view reverses the idea that facial behavior is expressive of emotional states, and argues that facial movements cause emotional feeling. In James's words "we feel sorry because we cry" (James, 1890). Although there is recent support for the notion that facial behavior influences feeling of emotion for some people (Laird, 1974; Zajonc, Murphy, & Inglehart, 1989), Osgood (1966) found that attempting to mimic facial expressions often resulted in less accurate judgments. A notable exception was the identification of physical pain, which seemed to be facilitated by mimicking facial behavior. Another interesting possibility suggested by Langfeld's subject is that individual facial movements have meaning that is somehow combined in the 14  final judgment. Langfeld's subject reports consciously analyzing individual facial movements, but this analysis might also operate at the preconscious level in much the same way that perceptions of objects are built out of preconscious calculations. For example, we are generally not aware of the cues we use to locate objects in space; we seem to just know where objects are in relation to each other by looking. But, by eliminating possible cues one by one, researchers have been able to study the aspects of the visual array that we use to judge distance (e.g., Ittelson, 1951, 1960). The lesson here for researchers interested in the study of perception of facial expression is that we may be unaware of some of the processes that contribute to our judgments. These processes can nevertheless be uncovered through systematic exploration of the correspondence between specific judgments and the elements of expression that are present. By systematically eliminating other possible cues to emotion, it may be possible to isolate the contribution of individual facial movements to judgments of emotion. The first attempts to investigate the elements of expression centered on discovering which area of the face is used in the judgment of emotion (Dunlap, 1927; Coleman, 1949; Hanawalt, 1944). These researchers had subjects view parts of the face (e.g., top half or bottom half of the face, or eyes or mouth only), and compared the judgments of partial faces with judgments made by observers viewing the whole face. There was no straightforward answer to the question of which area of the face produced judgments most similar to judgments made from the whole face. Rather it depended on the expression being identified. Summarizing his own and prior research on this question, Coleman (1949) concluded that the upper face seemed to give more clues for fear and surprise, and the mouth for happiness. The research suggests that  15  one area of the face is not a strong carrier of emotional messages in general, but that different features convey different kinds of information about emotion. A derivative approach focuses not just on general areas of the face, but on individual facial movements. In a study specifically designed to study judgment of facial behavior separately from any assumptions about its true meaning, Frois-Wittman (1930) tried to identify the facial components unique to judgments of one specific emotion, and those shared across emotion categories. Photographs of himself and drawings made from these photographs were scored for the presence or absence of a set of facial components. Observers chose one word from a list of 43 emotion words as most descriptive of the face. Frois-Wittman found that some emotion categories tended to overlap. For example, photographs for which disgust was the modal response were also frequently judged as showing contempt. Similarly, photographs judged most often as fear were also frequently judged as showing surprise. In addition he found that the number of facial components shared by two emotion categories was a good indicator of the amount of overlapping judgments by raters. Although Frois-Wittman's method was an improvement over earlier approaches examining only areas of the face, his scoring system was not comprehensive. Only a subset of facial movements were identified, and these were scored for presence or absence rather than degree of movement. What is needed to pursue this approach is a more complete description of facial movements, and a framework for judgments. Other researchers have since taken an approach similar to Frois-Wittman's with more detailed methods for describing facial movement (Birdwhistell, 1970; Ekman & Friesen, 1978; Ekman, Friesen, & Tomkins,1971; Izard, 1979). One of the resulting scoring systems, Ekman and Friesen's Facial Action Coding System, will be described in the next section. 16  A System for Describing Facial Movements Of the systems available for scoring facial behavior, Ekman and Friesen's (1978) Facial Action Coding System (FACS) was selected for the present research due to its comprehensiveness, its independence of assumptions about the meaning of the facial behavior, and the availability of detailed instructions for its use. Since FACS was used to score the facial behavior used as stimuli in the present research, the system is described here in some detail. Ekman and Friesen's system was intended to distinguish "all possible visually distinguishable facial movements," not only those associated with emotion (Ekman, 1982, p. 179). It identifies these minimally distinguishable facial movements, called action units, and outlines a complex set of rules for scoring their presence alone or in combinations. Ekman and Friesen's Facial Action Coding System was intended to be unconstrained by any particular set of expressions from which they might be derived. To accomplish this they began not with agreed upon expressions, but with the anatomy of the human face. From anatomy texts they identified the musculature of the human face, but reported being unable to uncover any systematic description of changes in the appearance of the face corresponding to facial muscle movement. They set about uncovering this relationship by practicing in front of a mirror until they learned to move each facial muscle voluntarily. They then photographed themselves producing these facial movements, and tried to pick out from the resultant set of photographs distinctive facial movements. If they could not easily distinguish one facial muscle movement from another in the photographs, the facial appearance was scored as a single action unit (regardless of which muscle movement produced it).  17  After identifying action units resulting from single muscle movements, they systematically examined all combinations of two action units. They report that most combinations were additive that is, in the resulting expression the two individual action units could still be identified. In some cases combinations of muscle movements produced a distinctive change in the appearance of the face qualitatively different from the action units associated with single muscle movements. In these cases the distinctive changes were scored as new action units. In some additional cases one action unit tended to mask, partially or completely, another action unit. In these cases a decision rule was added that a subordinate action unit cannot be scored in the presence of a dominant action unit (even if it is visible). The purpose of such decision rules was to increase the reliability of the scoring procedure. After examining all possible combinations of two action units, Ekman and Friesen went on to examine all possible combinations of three action units in the same way. They then examined selected combinations of four action units. Although the resulting scoring system has its basis in the musculature of the human face, the scoring rules are based on the visible changes in the face that observers can reliably distinguish. If two facial muscle movements produce the same visible change, the resulting change is scored as a single action unit. The system also leaves out "visible changes too subtle for reliable distinction," and "visible changes in muscle tonus that do not entail movement" (p. 181). Ekman and Friesen's FACS provides a reasonably complete description of the stimulus space--movements of the face. Table 1 lists the action units scorable with the system. Using this system, Ekman and his associates have identified patterns of facial movements which they argue correspond to seven basic emotions (see Table 2), but little other work has been done on the 18  relationship between the fine-grained facial movements scorable with FACS and judgments of emotion.  19  A Framework for Judgments of Feeling States Although proponents of universality argue that there exist discrete categories of emotional expression recognizable by all human beings (e.g., Ekman,1972), a recurrent finding in the research on judgment of emotion is that the judgments are related to each other. In 1918 Langfeld noted, "The intimate relation of anger and fear was brought out. Fear was seen when anger predominated and anger when fear was uppermost. Fear was also seen in amazement" (p. 183). Frois-Wittman (1930) found some categories of emotion tended to overlap in judgment, and that these overlaps were most common where several facial movements were shared. Attempts to systematically describe the relationship among judgments of emotion date back to Woodworth (1938). Woodworth noted that while there was not unanimous agreement on what labels should be used to describe a particular photograph of facial behavior, some emotion labels were more likely to be applied to a given photograph than others. He found that emotion categories could be arranged along a continuum such that the categories selected as descriptive of any photograph would be found alongside each other. Terms selected to describe a particular photograph were rarely more than one category away from each other. Schlosberg (1941) noticed that the categories on either end of the continuum were also likely to be selected as descriptive of the same photograph, and modified Woodworth's description of the emotion response space by suggesting a circular arrangement of emotion categories defined by two underlying dimensions. The suggestion that dimensions underlie judgments of emotion is traceable to Wundt (1924). Though the number of dimensions has varied, researchers since that time have consistently found that judgments of emotion are related, and that these relationships can be described by a small number of dimensions 20  (Ekman, 1965; Engen, Levy, & Schlosberg, 1962; Frijda & Philipzoon, 1963; Osgood, 1962, 1966; Plutchik, 1962). Two of these dimensions—pleasure and arousal 2—repeatedly emerge from studies using widely divergent methods, and account for the largest proportion of variance in judgment (Abelson & Sermat, 1962; Russell, 1978, 1979; Russell & Steiger, 1982; Watson & Tellegen, 1985). Russell (1978, 1979, 1980) has developed a geometric model of emotion, in which these two bipolar dimensions (pleasant-unpleasant and aroused-sleepy) underlie the circular arrangement of emotion terms that results from arranging most similar terms closest to each other (see Figure 1). The model represents emotion as a continuous two-dimensional space, with qualitatively different emotions located at different points on the circumference of the circle and intensity of emotion as distance from the center. Each point in the emotion space is uniquely defined by a particular combination of degree of pleasure and degree of arousal. Such a model is comprehensive in that it includes what have been called "basic emotions", "blends", and a variety of other terms that are used to describe feeling states. All of these terms are systematically related to each other in the geometric space defined by pleasure and arousal. The same circular arrangement has been found with judgment of emotion terms themselves (Russell, 1980), in English and four other languages (Russell, 1983), and with judgment of facial expressions (Russell & Bullock, 1986; Russell & Fehr, 1987).  2 The dimensions that emerge from multidimensional scaling solutions seem to be best described  as a pleasure dimension that captures the valence of the emotion on a continuous scale from intense displeasure to intense pleasure, and an arousal dimension that describes the subjective experience of activation on a continuous scale from extremely sleepy to extremely aroused. The arousal measured here is subjective experience of arousal rather than any physiological measure of arousal.  21  Russell's model offers a possible means for reconciling a categorical view of emotions with a dimensional view. Russell and Bullock (1986) found that pleasure and arousal ratings of expressions predicted similar locations in the two-dimensional space as found by judgments of category membership. They asked one group of observers to rate the degree of category membership of a set of facial expressions, and another group to rate the same expressions on pleasure and arousal scales. When the category ratings were multidimensionally scaled, a two dimensional circumplex resulted. Location of the photographs in the circumplex correlated very highly with the locations predicted by pleasure and arousal ratings.  A New Approach We are now in a position to return to the question of how observers judge feeling states from facial behavior. Ekman and Friesen (1978) have provided us with a comprehensive system for measuring facial behavior that can be used independently of any hypotheses about the meaning of that behavior. Judgments of feeling states can be measured in terms of emotion categories, dimensions of emotion, or as locations in a geometric space. Attention can now be turned to a systematic exploration of how judgments (the response space) map onto the various combinations of facial movements (the stimulus space). The relationships uncovered will provide missing pieces of the puzzle of how observers judge the feeling state of another person. It must be emphasized that facial behavior is only one piece of information used in judging the feeling state of another person. It is exceedingly rare that an observer judges another person's emotion solely from information provided in the face. Ordinarily the observer has other information, such as the situation in which the facial behavior was produced, cultural expectations about appropriate feelings and behavior in that situation, other behavior including verbal behavior, and perhaps information about the person's characteristic response to such situations. Additionally, it has been shown that the judgments of observers may be influenced by suggestion (Langfeld, 1918b, Jarden & Fernberger, 1926), by their own mood (Ruckmick, 1921), and by having previously judged another example of facial behavior (Russell & Fehr, 1987). My own initial research on judgments of emotion focused on the role of category accessibility in judgments of emotion (unpublished). Jim Russell and I had informally noticed what looked like a category accessibility effect in judging emotion from photographs of faces. Hearing the word sad, for example, while 23  looking at a photograph seemed to change the expression. A journey through the literature on the category accessiblity effect in social cognition revealed a number of factors influencing the effect of having a category primed. Using this information, I primed various emotion categories in various ways in a series of unpublished studies in an attempt to show that similar factors influenced judgments of emotion. After judgments had been collected from over 700 subjects in nine different experimental designs it became clear that the priming manipulations were having an effect on judgments, but in an unpredictable manner. I became convinced that understanding the influence of context on judgments of emotion from facial behavior requires an initial understanding of judgment of emotion from facial behavior in a more neutral context—without primes or other sources of information. Although context influences judgments, the face is not irrelevant in those judgments (Bruner & Taguiri, 1954) and thus it would be useful to know what meaning is attributed to facial movements when contextual influences are minimized. A thorough description of the process of judging emotion from facial  expression requires an understanding of how individual facial behaviors contribute to judgments, as well as an understanding of how context factors interact with facial behaviors to influence judgments. Russell and Bullock (1986) suggest that degree of pleasure and arousal are automatic first step processes in identifying facial expressions. This process locates the meaning of the expression in emotion space. Observers then use other contextual information to choose among possible labels near that point in emotion space. While this may be a viable model of judgments in some circumstances, it is not obvious how it would account for contrast effects in judgment (Russell & Fehr, 1987), or the effects of the observer's mood on judgment (Ruckmick, 1921). An alternative model of the judgment process is that hypotheses (not necessarily 24  conscious) about possible feeling states guide a visual search for facial behaviors believed to be related to such feeling states. In either case the meaning of individual facial movements, either in terms of pleasure and arousal or of emotion categories, is a valuable piece of missing information. A complete understanding of the meaning observers attribute to facial behavior requires a bottom-up approach. Rather than beginning with facial expressions and decomposing them into components as previous researchers have done (a top-down approach), one can begin with the components and systematically investigate the feeling states attributed to individual components and combinations of components. This bottom-up approach examines the meaning of indivdual facial movements, and explores how meaning is changed when individual facial movements are combined. Such an approach considerably broadens the scope of the study of interpretation of facial behavior to include the range of facial behavior we judge as converying information about feeling states on a daily basis, instead of just a few prototypical expressions.  25  Overview of Six Studies The purpose of this set of studies is to begin mapping out the correspondence between facial movements and judgments of feeling state. The underlying assumptions are that facial behavior is Dag source of og type of information about feeling states, and affective judgments arer judgment that can be made from facial behavior. Given the number of individual action units in Ekman and Friesen's (1978) system, the possible variations in intensity of each, and the staggeringly large number possible combinations, the complete mapping of response space onto stimulus space will be a lengthy process. In this set of studies I sought to first establish the feasibility of the approach. To that end I have asked some specific questions: Do observers ascribe meaning to individual action units? If so, is the meaning attributed to individual action units and combinations of action units interpretable within a geometrical model of emotion space? The present research combines the descriptive potential of Ekman and Friesen's FACS (1978) with a structural model of emotion space in order to explore the contribution of individual facial movements to judgments of feeling states. Virtually all the prior research on judgment of feeling states from facial behavior has used what I call a top-down approach. The attempt has been to create some fairly intense expressions, either posed, elicited, or sketched, and then look at how subjects label the facial behavior. These expressions have sometimes then been decomposed, and individual movements identified. In this way Ekman and Friesen (1978) have identified the prototypical facial movements in seven emotions (happiness, sadness, surprise, fear, anger, disgust, contempt) that they consider to be basic. There are no data on how individual action units are interpreted, or on how combinations of action units (other than the above seven) are interpreted. The research reported here takes 26  a bottom-up approach, beginning with individual facial movements and examining judgments of these movements alone and in combination. In other words I'm looking at how interpretable expressions are built out of the raw material of individual facial movements. This approach additionally covers judgments of feeling states that are not basic emotions, but may be found more frequently. Isolating the contribution of individual facial movements to judgments of feeling states based on facial information requires highly controlled conditions in which nothing else varies except individual facial movements. Such conditions are not conducive to ecological validity; judging emotion solely on the basis of a single facial movement is not representative of the range of information observers might use to make such judgments in nonlaboratory contexts. Still, systematically examining some of the components contributing to judgments of emotion can help us understand what is happening in more complex situations. As Mook (1983) has eloquently argued, "Even where findings cannot possibly generalize and are not supposed to, they can contribute to an understanding of the processes going on. . . It is that understanding that has external validity—not the findings themselves, much less the setting and the sample." (Mook, 1983, p. 382) By first establishing the meaning attributed to components of facial behavior in a neutral context, the interaction between those components and context may be more clearly seen. For example, if a particular facial movement is found to be judged as showing some amount of anger in neutral contexts, then in a situation where anger might be the expected response an observer might actively search for that movement, or other movements associated with anger. If those movements are found, the observer might be more confident of the judgment of anger in the non-neutral context. Or, a quick judgment in a non27  neutral context might be made using only the first facial feature that is fixated. Judgments might be made on the basis of whether or not the first AU seen is consistent with the expected feeling. To even begin testing hypotheses such as the above it is necessary to have some kind of dictionary of the general meaning attributed to individual action units. To begin the process of mapping out the relationship between facial behavior and judgment of feeling state we need to first ask whether individual action units have affective meaning, and, if so, how the meaning of individual action units is combined in judging the meaning of more complex facial behavior. The first two studies address these preliminary questions. Observers were asked to rate the feeling state of a person showing either single action units, combinations of two action units, or combinations of three action units. Secondly, we need to show that variations in intensity of individual action units and combinations of action units correspond to systematic variations along some dimension of the response space. In studies three and four action units were selected that were predicted (from studies one and two, as well as prior research) to influence primarily the arousal dimension of emotion, and ordered along this dimension. Finally, we need to show whether combining action units related to one dimension of emotion with those related to the other combine to locate a point in emotion space consistent with predictions from the model. In studies five and six predicted relations of arousal action units and pleasure action units were examined.  28  Studies One and Two To begin the study of the interpretation of individual action units we need a set of stimuli in which individual action units are presented with all other aspects (e.g., other facial movements, poser, context) held constant. Fortunately, Ekman has created a set of viewmaster filmstrips to demonstrate the Facial Action Coding System that meet most of these criteria. In these filmstrips Ekman himself poses individual action units, and some combinations of action units. For most action units, Ekman begins with a neutral facial expression (no scorable action units present) and gradually presents the action unit(s) in increasing amounts. Subjects in Study One rated these filmstrip sequences on pleasure and arousal, and subjects in Study Two rated the applicability of emotion category labels to the same sequences. With these data we can look at the meaning of individual action units along the dimensions of pleasure and arousal, the categorical meaning of the action units, and the relationship between dimensional and categorical judgments.  Study One Method Subjects. Subjects were 79 students attending the University of British Columbia.  Materials. Twenty of the Ekman sequences were used in this study. Eleven single action units were selected that occur in some of the basic emotions identified by Ekman and Friesen (since not all action units are related to emotion). Also selected were nine combinations of two or three of these action units where available. A few action units were included for simple expediency, since they intervened between action units of interest on the filmstrip. The 29  selected action units are listed in Table 3. All but one of the sequences (the combination of action units four and five) begin with a neutral facial expression, and the action units are presented in increasing intensity. Action units are at maximum intenstity at the end of the sequences for all sequences except 46. In this sequence Ekman begins with a neutral face, closes one eye and reopens it. Pleasure and arousal ratings were made on the Affect Grid (Russell, Weiss, & Mendelsohn, 1989). The Affect Grid is a rapid measure of judged pleasure and arousal in which observers simply place a mark on a nine-by-nine grid representing the two-dimensional emotion space. Judgments on the Affect Grid correlate highly with judgments made on more time-consuming pleasure and arousal scales (Russell, Weiss, & Mendelsohn, 1989). Pleasure and arousal scores are taken from the Affect Grid as the horizontal and vertical number of the square checked. The pleasure-displeasure dimension is numbered from left to right, with lower numbers indicating more displeasure and higher numbers indicating more pleasure. The arousal dimension is numbered from bottom to top, with lower numbers indicating lower levels of arousal and higher numbers indicating more arousal. For both dimensions a score of five indicates a neutral judgment on that dimension. An instruction booklet was used to train subjects in the use of the Affect Grid (see appendix).  Procedure. In total 20 action unit sequences were rated. For reasons of time, each subject rated only a subset of these sequences. Twenty-seven subjects rated the first seven sequences on the filmstrip, 26 rated the next seven, and 26 rated the remaining six. Subjects viewed the sequential facial movements by turning a crank on the viewmaster. In order to determine how the presentation of the action units influenced judgment of feeling, subjects were asked to judge the feeling present at the beginning and at the end of each sequence. Subjects were given the following instructions: 30  "Through the viewer you will see an actor attempting to convey certain messages, often a mood, feeling or emotion--or combination of moods, feelings or emotions. Whatever the message, your job is to describe the mood conveyed. "Each sequence begins with a code number written against a brown background. Please write that code number at the top of your answer sheet. Then turn the red handle clockwise to view the sequence. Stop when you reach the next code number. If you want to re-view the sequence, you can rewind by turning the handle counterclockwise. "To indicate your judgment about each sequence, you will use the Affect Grid you have already read about. Place an "S" somewhere on the grid to indicate how the actor feels at the START of the sequence. Place an "E" somewhere on the grid to indicate how the actor feels at the END of the sequence. If you see no change in feeling, then place "S" and "E" in the same appropriate square."  Results Table 4 lists the mean pleasure and arousal ratings at the beginning and end of each sequence. With the exception of the combination of action units 4 and 5 ( in which the initial pose is of action unit 4 and action unit 5 is added), all initial poses are scored as neutral by FAGS. Affect grid ratings were translated into 9point scales for both pleasure and arousal, with low scores indicating low ratings on that dimension and high scores indicating high ratings. A score of 5 indicates a score of neutral. Omitting the rating of that combination in which the initial pose was not scored as neutral, pleasure scores range from 4.26 to 5.74, and arousal ratings from 4.07 to 5.58--centering closely on the neutral point.  31  At the end of the sequences mean pleasure and arousal scores were considerably more spread out. Mean pleasure scores ranged from 1.67 to 6.58, and arousal from 3.78 to 8.46--suggesting that presentation of the various action units resulted in judgments of emotion that differed from neutral. Clearly the faces at the end of the sequences were judged as conveying much more varied affective messages than the faces at the beginning of the sequences. This interpretation is confirmed by sequence by time of rating (beginning and end) repeated measures analyses of variance, which showed significant sequence by time interactions for both pleasure and arousal for all three groups. Although the assumption of sphericity was violated, Huynh-Feldt corrected probabilities were still significant for all groups at the .00001 level (see Table 5). Pleasure and arousal were judged to change from the beginning to the end of the sequences, with the amount and direction of change determined by the action units presented. Figure 2 graphically illustrates the similarity of the initial ratings, and the diversity of the ending ratings. The ratings of the initial poses in the sequences are shown as open circles (clustering about the neutral point), and ratings of ending poses as solid circles. As can be seen, the ending poses were rated as farther away from the center neutral point--indicating that these poses were judged as conveying more varied information about emotional state. Table 6 summarizes the mean changes in pleasure and arousal by sequence. To get an idea of which changes were meaningful, a series of t-tests were calculated. To control for experimentwise error, the alpha level for each group was adjusted by the number of sequences judged by that group. Judgments of one sequence increased in pleasure (outer brow raise), some decreased (nose wrinkle, lower lip depress, upper lip raise, lip corner depress, brow lower, lids tight), and the rest were not significantly changed. Some action 32  units were judged as increasing arousal (upper lid raise, outer brow raise, nose wrinkle, upper lip raise), and others showed only minor changes. None of the sequences were judged as significantly less aroused at the end, presumably because no action units corresponding to decreases in arousal were presented.  Discussion  Are single action units interpretable? Figure 3 shows the mean pleasure and arousal ratings at the end of only those sequences in which only a single action unit is presented. Clearly faces showing different single action units are rated as showing different amounts of pleasure and arousal. Some action units, such as two and five, seem to influence mainly arousal judgments. Others, such as 1, 4, 15, and 7, seem to influence mainly pleasure. Still others, such as 9 and 10, seem to influence both pleasure and arousal. What happens when individual action units are combined? One prediction consistent with the geometrical model of emotion space is that averaging the vectors of the individual action unit ratings (a line running from neutral to the rating, representing both direction and magnitude of change) will produce the vector of the rating of their combination. This prediction can be examined by looking at Figure 4. The mean pleasure and arousal ratings of ending sequences of combinations and their components are plotted in Figure 4; this allows visual inspection of judgments of combinations of action units vs. judgments of the components. There are four combinations of two action units for which ratings of the individual action units were also obtained: five and seven, four and five, one and four, and one and two. Additionally, there are four combinations of three action units for which ratings of one single component and the other two components in combination were obtained: 1 + 2 and 5, 1 + 2 and 4, 9 and 16 + 33  25, 10 and 16 + 25. If vector averaging describes judgment of combinations, the vector of the combination should fall roughly half-way between the vectors of its components. Using this loose criterion, five of the eight combinations are consistent with vector averaging: 5 and 7, 1 and 4, 1 + 2 and 5, 10 and 16 + 25. For these combinations the ratings seem to be an average of the ratings of the components. Interestingly, the combination of action units one and two does not follow simple vector averaging. The influence of action unit one (inner brow raise) seems totally dominated by action unit two (outer brow raise) when they cooccur. Whether or not this is a special case of action units that are closely related remains to be seen. The other two violations of vector averaging are less extreme. An alternative to averaging is a weighted vector model. According to a weighted vector model, some action units are weighted more heavily than others when they co-occur. It may be that further research will reveal a set of consistent weights for individual action units that predict their meaning in combination. We now have a preliminary dictionary for the meaning attributed to action units, and some predictions about how individual action units combine. Table 6 can be used as a tentative dictionary, serving as a reference for further validation of the meaning attributed to the action units studied. Once all action units have been studied and validated with additional models and observers, a researcher could simply look up the meaning of particular action units in the dictionary. A researcher interested in, for example, action unit nine (nose wrinkle) can see that, in this study at least, action unit nine was judged as showing above neutral arousal and below neutral pleasure. Combinations of action units could be similarly referenced.  34  Study Two It was predicted that category ratings, as well as pleasure and arousal ratings, would reflect observers' ability to interpret single action units. It was additionally hypothesized that the pleasure and arousal ratings from Study One could be used to predict category ratings.  Method Subjects. Subjects were 60 students attending the University of British Columbia.  Materials. The stimuli for study two were the same three series of action unit sequences used in study one. Emotion category ratings were made on the Facial Judgment Scale (see appendix). This scale required subjects to use a one-to-eight rating scale to describe how accurately ten different emotion labels described the person at the beginning and the end of each sequence. Allowing subjects to rate how well emotion words describe the faces rather than selecting one word for each face is a more sensitive measure of the agreement among observers about the meaning of facial movements.  Procedure. The procedure was similar to study one except that subjects used the Facial Judgment Scale to make category judgments about the sequences. Twenty subjects made category judgments of each of the three series.  Results Table 7 lists the mean emotion category ratings at the beginning of each sequence. Since all but one of the sequences (action unit 4 plus action unit 5) 35  are neutral at the beginning according to FACS, it's useful to look at average category ratings across the set of neutral faces. Calm was the emotion category rated as most descriptive of the neutral poses, although its rating of 5.65 corresponds to a judgment only somewhere between slightly accurate and quite accurate. Surprised was judged the least descriptive of the emotion categories. Standard deviations of category ratings were all between .40 and .68, showing a fair amount of agreement on the ratings. Mean ratings for the ending pose of each sequence are quite different from both the initial ratings, and from each other (see Table 8). Sequence by time of rating ANOVAs were significant at at least the .05 level for all categories, with the exception of fear ratings for one group of subjects. The F ratios and significance levels are given in Table 9. The increased variance in ratings at the end of the sequences reflects the same pattern seen in study one, with initial poses being rated as very similar to each other, and the ending poses rated as indicating different degrees of different emotions. In other words, single action units are interpreted as conveying emotional meaning. The particular emotional meanings attributed to action units are related to their pleasure and arousal ratings. A regression analysis showed pleasure and arousal scores from Study One predicted all emotion category ratings except fear. The amount of variance in category ratings accounted for by pleasure and arousal ratings ranged from 59% for surprised to 83% for happy and sleepy. Table 10 summarizes the Beta weights and significance levels. As in Study One, whether the rating of a sequence increased or decreased from the beginning to the end depended on both the action units presented and the judgment being made. Table 11 summarizes the changes in ratings for each action unit sequence. Recall that calm was the category rated as most descriptive of the initial poses (see Table 8). With the presentation of the 36  various action units all of the 20 sequences were rated as less calm; 14 of these were statistically significant changes. Nine sequences were judged as significantly less bored at the end, three as significantly less sleepy, two as significantly less sad, and one each as significantly less angry and happy. Increases in ratings were scattered across the categories of angry, excited, afraid, disgusted, surprised, and happy.  Discussion As in Study One, the results of Study Two can be thought of as a preliminary dictionary of the meaning of action units, with the meaning given this time in terms of descriptiveness of emotion category labels. Additionally, the meaning found here for individual action units can be compared to Ekman and Freisen's (1978) predictions about the more complex expressions of which the AUs are components (see Table 1). For example, in Table 8 it can be seen that sadness is judged as the most descriptive emotion category for action units one, fifteen, and seventeen occurring alone, and for the combination of one and four, and the combination of one, two, and four. All of these action units except action unit two are predicted by Ekman and Friesen (1978) as components of what they call prototypical or major variants of expressions of sadness. What I have shown here is that the individual components themselves are interpreted as indicating some sadness. The ratings of sadness given to the components are not extremely high (between five and seven on an 8-point scale), but the results suggest that judgments of the meaning of individual components in a complex expression contribute to judgment of the larger expression. Anger was seen when action units four or seven were presented alone, and with the combinations of five and seven, 16 and 25, and 9, 16 and 25. Ekman 37  and Friesen give action units four, seven, and twenty-five as components of anger; nine and sixteen are not listed as components of either prototypes of anger or major variants. Again I have shown that anger can be seen in single components of the anger expression. Additionally it appears that other action units are interpreted as expressing anger. Surprise was seen with single action units two or five, and with the combinations of AUs one and two, four and five, and one, two and five. Ekman and Friesen give the combination of one, two and five as one of the major variants of the expression of surprise. Again we see that individual components are interpreted as showing emotion. Action unit six was interpreted as showing happiness; Ekman and Friesen give as happiness expressions either action unit 12 alone (not included in this study) or the combination of action units six and twelve. In a few cases, Ekman and Friesen give single AUs as prototypical expressions. For example, the single AUs nine and ten are both given as prototypical expressions of disgust. In this study, "disgust" was rated as the most descriptive term for both of these action units. Judgments of the meaning of action units were found to be related to judgments of the complex expressions of which they are components. Does this mean that both complex expressions and component AUs are judged as showing one and only one emotion? The category headings in Table 8 are arranged as they fall on the circumplex, if we cut the circle between calm and sleepy. Reading across the rows of the table, it can be seen that action units rated as indicating a high degree of one emotion were also sometimes rated as high on neighboring categories. Rarely was a sequence judged high on one category and low on all the others. For example, action unit 4 was rated as showing a high degree of anger (6.60) and 38  also a high degree of disgust (6.55). The combination of action units one, two and five was rated as showing a high degree of surprise (7.35), but also relatively high degrees of excitement (6.60) and fear (5.65). There is a tendency for emotion categories that are close to each other on the circumplex to receive more similar ratings for a given pose than categories farther from each other. The results of studies one and two strongly suggest that single action units are interpretable, and interpretable within the described emotion space in a  graded fashion. For example, at the end of the filmstrip sequence in which single action unit 5 is presented, arousal was rated very high, and pleasure was rated slightly below neutral. "Surprise" was judged the most descriptive emotion term (mean = 6.60). "Afraid" and "excited" were also judged as somewhat descriptive (means = 5.90 and 5.60, respectively). The category judged least descriptive was "sleep" (mean = 1.35), followed by "bored" (mean = 1.45) and "calm" (mean = 1.75). Mean judgment changes from the start of the sequence to the end of the sequence were largest for these categories. Similar results were found for action units 1 + 2, and the combination of 1 + 2 + 5. Both of these combinations were rated high on arousal and relatively neutral on pleasure. "Surprise" was again judged as the most descriptive emotion term for 1 + 2 (mean = 7.00) and for 1 + 2 + 5 (mean = 7.35). "Excited" was judged the second most descriptive emotion term for both of these combinations (4.85 for 1 + 2 and 6.60 for 1 + 2 + 5). "Fear" was again the third most descriptive emotion term for 1 + 2 + 5 (mean = 5.65), but "happy" was third for 1 + 2 (mean = 3.90). A trend is discernable in that as action units judged as showing increases in arousal are combined, categories high on the arousal dimension are judged  39  increasingly more descriptive of the face and categories low on the arousal dimension are judged less descriptive. One might argue that the method of asking for judgments at the beginning and end of each sequence encouraged subjects to exaggerate change. However, the important thing to note about the results is that for each action unit or combination of action units, judgments of certain emotions categories increase while judgments of others decrease or are unaffected. Moreover, there is a pattern to the relationship between the categories judged more descriptive of a face with a single action unit and those categories judged as less descriptive. This pattern is predicted by the pleasure and arousal ratings made by separate groups of subjects.  40  Studies Three and Four Studies One and Two were primarily descriptive. The results showed that some individual action units are interpreted as conveying information about feeling state, and that judgments of emotion categories were related to independent judgments of pleasure and arousal. Several action units were identified which were judged primarily as showing increased arousal, and were similarly rated as high on those categories of emotion that are high on the arousal dimension in Russell's model. A tendency for sequences showing combinations of action units individually rated as showing high arousal to be rated even higher on arousal was noted. Studies Three and Four were designed to test some specific hypotheses based on the results of Studies One and Two. The focus in Studies Three and Four was on the arousal dimension of the feeling space. Historically, arousal has been seen as a central dimension in the physiology of emotion (Lindsley, 1951), perception of facial expressions (Schlosberg, 1954; Abelson & Sermat, 1962), and in the experience of emotion (Schachter & Singer, 1962). Arousal can be thought of as a continuum. At one end is sleep. As arousal increases, the person wakes slowly, reaches moderate alertness, then heightened attention, and finally frantic excitement (provided that "frantic excitement" is not thought of as either positive or negative). The question is how these changes would be manifest in the face. Common sense suggests that at the low end, muscles are relaxed, eyes closed or droopy. In the middle of the continuum, muscle tension is normal, gaze is steady. Then, as arousal increases further, the eye lids rise, the eyebrows rise, the lips part. At the "frantic excitement" end, the eyebrows are completely raised (1 + 2), the eyes are wide open (5), and the mouth is open (26). Darwin (1872/1965) described 41  these changes in the face as a person moves from attention to surprise, "Attention is shown by the eyebrows being slightly raised; and as this state increases into surprise, they are raised to a much greater extent, with the eyes and mouth widely open" (p. 278). For Studies Three and Four action units were chosen that were either previously shown to be judged as high on arousal or expected to be related to arousal as outlined above. Action units one, two, and five were included since they were shown to be related to judgments of increased arousal in Study One. Actions units 25 (lips part) and 26 (jaw drop) were hypothesized to also increase judgments of arousal. Several action units were also included that were expected to decrease judgments of arousal, since this area of the circumplex was underrepresented in Studies One and Two. Action units selected for this purpose were 41, 42 &43 (eyes droop to eyes closed), 64 (eyes down), and 54 (head down). A series of photographs was created and scored by a FACS trained person in which the above AUs varied incrementally. FACS provides an intensity code for most AUs. Here 1%. 2i, y., and z represent least to maximum intensity. Subjects judged a set of 19 photographs in which the above action units were systematically varied. In Study Three subjects rated the photographs on the Affect Grid, and in Study Four subjects rated the same photographs on the Facial Judgment Scale. It was predicted that ratings of the series would show gradual increases in arousal as the number and intensity of associated action units increased, and that emotion categories high on the arousal dimension would be judged increasingly more likely.  42  Study Three  In Study One action unit five was rated as indicating high arousal when presented individually at high intensity. It was therefore predicted that varying the intensity of this single action unit would produce corresponding variations in judged arousal. The highest mean arousal rating in Study One was given for the combination of action units one, two and five, though five alone and the combination of one and two were both rated high on arousal. It was therefore predicted that the combination of these action units would be rated more highly on arousal than the individual components presented alone. It was further hypothesized that increasing the intensity of combinations of action units presented together would also increase judged arousal. Finally it was hypothesized that presenting action units 41, 42, 43, 64 and 54 would decrease judged arousal.  Method Subjects. Subjects were twenty-one students at the University of British  Columbia.  Stimuli. The stimuli were 19 photographs of a female trained in FACS in  which action units were varied incrementally. Action unit combinations expected to increase ratings of arousal were presented in four increments. These combinations included: 5 (upper lid raise), 1 + 2 (inner and outer brow raise), 1 + 2 + 5 + 25/26 (inner and outer brow raise, upper lid raise, and lips part or jaw drop). Action unit combinations expected to decrease ratings of arousal were: 41 + 64 (lids droop and eyes down), 42 + 64 (eyes slit and down), 43 (eyes closed), and 43 + 54 (eyes closed and head down). A neutral  43  photograph (no action units present) was included that was expected to fall between those action units increasing arousal and those decreasing arousal. The photographs were scored by a rater trained in the use of the Facial Action Coding System, and numbered according to predicted location on the arousal dimension. Thus, photograph number one was expected to receive the lowest arousal rating, and photographs 18 and 19 the highest. Ordering of the photographs was done using the predictions described above. These predictions allowed easy ordering of the photographs expected to be rated below neutral in arousal: photograph number one shows the model with her eyes closed and head down (43, 54y), number two with her eyes closed (43), number three with her eyes slit and looking down (42, 64), and number four and five with her eyes drooping and looking down (41, 64). Photographs four and five contained the same action units at the same intensities, so numbers were arbitrarily assigned. Photograph six contained no action units. Within the photographs predicted to be rated as higher than neutral in arousal, only some of the ordering could be based on clear predictions. Within single action units and within combinations, increasing intensity was predicted to increase judgments of arousal as in Study One. The combination of action units 1z, 2z, 5z, and 26 was predicted to be highest in arousal. As with photographs four and five there were two photographs containing this combination of action units, so they also were numbered arbitrarily. At similar intensities action unit 5 had been predicted to be rated as higher in arousal than the combination of one and two. There was no real basis for ordering the different combinations at different levels of intensities, so the one and twos were ordered by intensity, followed by the fives, followed by the 1, 2, 5, and 26s.  44  Procedure. Subjects were asked to rate the19 photographs by using the Affect Grid. The experimenter spread the photographs in front of the subject so that (s)he was provided with a brief overview of the entire set. The experimenter gathered the photographs, shuffled them and then placed them face down. The subject was presented with an instruction booklet on the use of the Affect Grid. After reading the instructions, the subject was asked if there were any questions concerning the use of the Affect Grid. Any queries were answered. The subject was instructed to rate each photograph individually and then to turn each photograph and each questionnaire face down. This procedure was to be repeated until the entire set of photographs had been rated. The duration of the experiment was approximately 30 minutes.  Results Figure 5 shows the mean placement of each photograph on the affect grid. Most of the variation is along the arousal dimension, suggesting the action units selected for this study do influence mainly that dimension of emotion. A repeated measures analysis of variance showed a highly significant main effect of photograph on arousal ratings,  E 18/360 = 54.72, 12 <.00001. As in  Study One the assumption of sphericity was violated; however, the Huyhn-Feldt correction made no noticeable difference in the significance level. Table 12 shows the pleasure and arousal scores rank ordered by arousal; the photograph number gives the predicted order. The Spearman rank order correlation coefficient for predicted order with arousal rating was .94. A series of Bonferroni comparisons were made to test the specific hypotheses made. Since the circularity assumption was not satisfied, Bonferroni is were calculated using contrast-specific residual mean squares as outlined in Kirk (1988). Four contrasts were made, holding experimentwise error at .05. 45  The first contrast tested the prediction that increasing the intensity of a single action unit associated with arousal would increase rated arousal. Action unit 5, which was rated as indicating a high degree of arousal in Study One, was presented in four increments of intensity in this study. Comparison of the arousal ratings of the two photographs marking the ends of this set, photograph 11 (eyes slightly wide) and photograph 14 (eyes very widely opened), showed that the higher intensity photograph was reliably judged as indicating higher arousal (120 = 4.86). The second contrast tested a similar prediction for the low arousal action units. It was predicted that as the eyes went from drooping to closed (action units 41, 42 and 43) arousal would be rated as lower. Arousal ratings of photographs 14 and 5 (both showing eyelids drooping and eyes down) were compared with ratings for photograph 1 (eyes closed and head down). This contrast failed to reach acceptable levels of significance, although it was in the predicted direction (120 = 1.90). The first contrast discussed above showed that variations in intensity of a single action unit produced reliable judgments of changing intensity of arousal. Contrast three tested the prediction that changes in intensity of action unit combinations also produce changes in judged arousal. Rated arousal of photograph 15, in which action units 1, 2, and 5 are presented in minimally scoreable amounts along with action unit 25 (lips part), was compared with ratings of photographs 18 and 19, in which action units 1, 2, and 5 are presented in maximum amounts along with action unit 26 (jaw drop). This contrast supported the prediction that observers reliably rate higher intensity combinations of these action units as indicating higher levels of arousal (12o = 3.15).  46  The fourth and final contrast tested the prediction that combinations of arousal action units are judged as showing more arousal than the component action units presented alone. Ratings of photograph 10 (1z, 2z) and photograph 14 (5z) were compared with ratings of photographs 18 and 19 (1z, 2z, 5z, 26). Results showed that photographs five and six were reliably judged as showing higher arousal than photographs one and ten (t20 = 12.30).  Discussion  As individual action units or groups of action units increase in intensity, rated arousal increased. There is only one reversal in this pattern--photograph 15 is rated as higher in arousal than photograph 16. We would not, of course, expect perfect correspondence since we know that other factors besides the face being rated influence judgments of facial expression (e.g., Russell & Fehr,1987 showed contrast effects with previously rated faces). As in Study One, action unit five generally seems to contribute more to arousal ratings than action units one and two. Photographs in which all three action units are present in maximum amounts are rated as highest in arousal. In the other direction, as the eyes move from drooping to closed arousal decreases. There was one reversal in predicted ordering of low arousal action units--photograph two was rated as lower in arousal than photograph one.  47  Study Four Method Twenty-one subjects were asked to rate the same 19 photographs using the Facial Judgment Scale. The procedure was identical to the procedure above except for the rating instrument used. Again subjects completed the task in approximately 30 minutes.  Results Table 13 lists the mean category ratings by photograph. The photographs are ordered by the arousal ratings obtained in Study Three, and the emotion categories are ordered by predicted arousal from left to right. The ordering of categories was derived from Russell and Mehrabian (1977). The highest category rating for each photograph is shown in boldface type. In general, as arousal increases ratings of low arousal categories (sleepy, bored, calm and sad) decrease, while ratings of high arousal categories (fear, surprise,and excitement) increase. No clear pattern is apparent for the other emotion categories. Table 13 also gives the correlation coefficient between the mean category ratings and the mean arousal rating across photographs. Strong negative correlations are shown between arousal ratings and ratings of "sleepy," "bored," "calm," and "sad." Strong positive correlations are shown between rated arousal and ratings of "afraid," "surprised," and "excited." Weaker correlations are shown for "disgusted," "happy," and "angry." Again we see that nearby categories are often rated as similarly descriptive of a given photograph. For those photographs below neutral on arousal the categories of sleepy, calm and bored all tend to be rated as descriptive. For those photographs high in arousal, ratings tend to be high on the cluster of surprised, afraid and excited, or the cluster of angry and disgusted. 48  A regression analysis again showed that pleasure and arousal ratings (from Study Three) predicted category ratings. Table 10 summarizes the Beta weights and significance levels. Arousal weights are significant for excited, calm, afraid, sad, sleepy, surprised, happy, and bored (all but angry and disgusted). Pleasure weights are significant for angry, disgusted, excited, sad, surprised, and happy. The variance in category judgments accounted for by pleasure and arousal ratings ranged from 71 % for angry and happy to 93% for excited.  Discussion  As AUs expected to be related to arousal increased in number or intensity, rated calmness, sleepiness and boredom decreased, while rated fear, surprise and excitement increased. For both high and low arousal photographs, category ratings peaked at pleasure-neutral categories—sleepy for low arousal and surprised for high arousal. This is exactly what is predicted if arousal action units have been isolated. It is important to remember that related categories were also rated as descriptive of the expressions, only less so. In the next two studies an attempt is made to shift these judgments to peak at either fear or excitement by adding pleasure-related action units (either one increasing or decreasing pleasure) to the high arousal action unit combination.  49  Studies Five and Six Studies Five and Six focused on the pleasure-displeasure continuum, and were designed to examine the effects of combining action units related to arousal with action units related to pleasure or displeasure. A set of photographs was created in which arousal was held constant (very approximately) while pleasure-related action units were varied. Arousal action units used were 1 + 2 (inner & outer brow raise), five (upper lid raise), and 26 (jaw drop); thus arousal was high. Pleasure action units were selected that were predicted by Ekman and Friesen (1978) to be present in either positive or negative emotions, but not both (see Table 2). Action unit 15 (lip corner depress), is present in sadness, and was shown in Study One to decrease pleasure ratings. Action unit 12 (lip corner pull), is present in happiness, and was expected to increase pleasure ratings. Action unit four (brow lower) is present in anger and fear, and in Study One it also increased arousal.  Study Five Method Subjects . Subjects were 30 students enrolled in introductory psychology at  Okanagan College, who participated for course credit.  Stimuli. The stimuli were 24 photographs of a female posing various  combinations of action units 1, 2, 5, 25, 26, 4, 12, and 15. We began with the combination shown in Study Three to show the highest degree of arousal. We tried to hold the combination of 1z, 2z, 5z and 26 (high arousal) constant, while adding one of the pleasure-related action units (4, 12, 15) in varying intensities. The model was trained in Ekman and Friesen's FAGS, yet, despite repeated and frustrating attempts, we were not entirely successful in producing the 50  desired combinations of action units. In part this was due to technical problems in maintaining sufficiently bright levels of lighting under which it was difficult for the model to maintain high intensities of action units one, two and five, and in part to some combinations that are not part of our natural repetoire of expressions being extremely difficult to produce. Two neutral poses were also included. The resulting set of photographs was scored by an independent scorer trained in FAGS. Table 14 lists the scored action units in the final set of photographs, and their predicted ratings on the pleasure and arousal dimensions.  Procedure. On arrival subjects were given the following instructions: "We're interested in how people judge what another person is feeling from his or her facial expression. You'll be looking at a set of photographs of a person's face, and rating what you think the person was feeling in each photograph." Subjects were then introduced to the Affect Grid, and any questions about its use were answered. The photographs were then spread out in front of the subject briefly, collected and shuffled, and placed face down in front of the subject. "You'll be rating each photograph on this grid. Turn over one photograph at a time, write the number of the photograph on the grid at the point that you think best describes what the person is feeling in that photograph. Then turn over the photograph and go on to the next. Continue until you have rated all the photographs."  51  Results Figure 6 shows the location of each photograph in emotion space, and Table 15 lists the mean pleasure and arousal ratings. All photographs predicted to be rated high in pleasure were rated above neutral, and all those predicted to be rated low in pleasure were rated below neutral. In order to examine the prediction that variations in intensity of action units 4, 12 and 15 produce graded judgments along the pleasure dimension, each photograph was assigned a rank according to the intensity of the action units expected to influence pleasure, disregarding the variations in arousal action units. For photographs with AUs expected to decrease pleasure, a rank of one was assigned to the maximum intensity (z), a rank of two to the next intensity (y), and a rank of three for the weakest intensity (x). Photographs without any action units predicted to influence pleasure were assigned the rank of 4, photographs with action unit 12x were ranked 5, and photographs with action unit 12y were ranked 6. The correlation coefficient calculated between these ranks and pleasure judgments was .93. Increasing the intensity of action units predicted to decrease pleasure (15, 7 and 4) resulted in progressively lower pleasure ratings, while increasing the intensity of action unit 12 (predicted to increase pleasure) had the opposite effect.  Discussion  ,  The results are supportive of predictions, although the data do not allow a direct test of the hypotheses since we were unable to hold the arousal action units at a constant intensity while simultaneously varying the intensity of action units 12, 15 and 4. Nevertheless, pleasure ratings did vary according to the action units present as predicted. Even though arousal varied somewhat, the correlation between pleasure ratings and predicted order was quite strong. 52  Figures 7 and 8 illustrate the change in pleasure ratings corresponding to changes in the intensity of action units 12 and 15.  53  Study Six The general approach taken here is that specific AU combinations predict category ratings via pleasure and arousal. This thesis was examined for the 24 photographs of Study Five. In Study Four high arousal categories were judged as increasingly descriptive of photographs increasing in arousal. "Surprise" was judged most descriptive of photographs with arousal action units present in strong intensities. If pleasure and arousal action units combine to locate a point in emotion space, the addition of an action unit associated with high pleasure to our high arousal expression should result in facial behavior rated as showing increasing excitement. Similarly, adding an action unit associated with declining pleasure should create a face rated as showing increasing fear. Category ratings of the photographs from Study Five were obtained in a between-subject design. Each subject looked at a single photograph and rated how well three emotion category terms ("excited," "surprised," "afraid") described how the person in the photograph was feeling.  Method subjects. Subjects were 347 students at Fraser Valley College who were approached in the hallway and asked to volunteer two minutes of their time to a psychology study.  Stimuli. The stimuli were the same 24 photographs used in Study Five.  Procedure. Thirty-five researchers were equipped with a photograph and response sheet and, after practicing, asked to obtain responses from ten volunteers each. Each researcher used to following verbatim instructions: 54  "My psychology class is doing a study on the judgment of facial expressions. I need people to look at a photograph and answer three short questions. It will take no more than two minutes of your time. Are you willing to participate?" If the individual agreed, the researcher proceeded with the following instructions: "Please look carefully at this photograph. Look for the feeling that is expressed there. Using a scale from 1 to 8, where 1 is extremely inaccurate and 8 is extremely accurate, how well does excited describe the feeling you see in the photograph? surprised? afraid?" The rating scale was presented to the subject along with the photograph to be rated.  Results  Table 16 summarizes the category ratings. The previous studies showed that increasing action units associated with arousal increases the probability that the face will be rated high on emotion categories associated with high arousal— excited, surprised, afraid—with the pleasure-neutral "surprised" rated the highest. In this study we tried to move judgments in a positive or negative direction by adding action units expected to influence pleasure ratings. It was hypothesized that photographs containing action unit 12 would be rated as more "excited," those containing action unit 15 or action unit 4 as more "afraid," and those without either action unit as "surprised." An analysis of variance was performed to compare category ratings of subjects viewing photographs with low pleasure action units (group one) to those viewing photographs with high pleasure action units (group three) and those viewing photographs without any pleasure action units (group two). This analysis showed a significant effect of group for all categories (excited F 2/317 = 67.21, p < .0001; surprised F 2/317 = 55  30.51, p < .0001; afraid F 2/ 317 = 44.18, p < .0001). Tukey pairwise comparisons showed that the high pleasure photographs were rated as significantly more "excited" than the other two groups (ps < .0001), the low pleasure photographs were rated as significantly less "surprised" than the other two groups (ps < .0001), and the low pleasure photographs were rated as significantly more "afraid" than the neutral pleasure photographs (p < .005) which were rated as significantly more "afraid" than the high pleasure photographs (p < .0001). Figures 9, 10, and 11 illustrate the relationship between rated pleasure and category ratings. As pleasure increases rated "excitement" increases (r = .77) and rated "fear" decreases (r = -.76). There is a weaker positive relationship between rated pleasure and rated "surprise" (r = .55).  Discussion  Again, the rated descriptiveness of emotion terms was related to pleasure and arousal ratings. Higher pleasure ratings were related to higher ratings of the descriptiveness of the term "excited," and lower pleasure ratings were related to higher ratings of the descriptiveness of the term "afraid." Ratings of both pleasure and emotion categories were mediated by the action units predicted to influence the pleasure dimension of feeling states.  Conclusion There are several advantages to a bottom-up approach in the study of judgment of feeling states from facial behavior. The major advantage is that it allows a comprehensive study of the affective meaning observers attribute to facial behavior. In this approach the study of judgment of feeling states is not restricted to a few expressions that are believed to be basic. Individual facial movements can be isolated for study, and the meaning of complex expressions can be examined in relationship to the meaning of their components. With this approach observer agreement both within and between cultures on both individual facial movements and complex expressions can be examined. Furthermore, this approach may give researchers insight into how observers are making judgments about complex facial expressions by showing how the meanings of their components are combined. One possibility consistent with the results of the present research is that individual facial movements have meaning in terms of dimensions, such as pleasure and arousal, that help to locate a point in a geometric emotion space. The meaning of combinations is then some weighted combination of the meaning of the individual movements. How individual action units are weighted when they occur in combination is an important question that remains to be answered. It may be that certain facial movements less under voluntary control are given relatively more weight than other more voluntary movements (see Rinn, 1984 for a description of voluntary versus involuntary control of facial movements). It may be that action units consistent with the feeling state observers expect to see are given relatively more weight than inconsistent ones. Or there may be individual differences in the relative weighting of action units. Complicating matters further, we may weight one set of action units in judging the feeling state of one person with whom we are well acquainted, while finding an altogether different set of action 57  units most telling about the state of another. Discovering how individual action units are weighted awaits a systematic exploration of judgments of the possible combinations. Although the present research focused on the dimensions of pleasure and arousal, there may also be other meanings attributed to particular facial movements that go beyond these two dimensions. Pleasure and arousal account for by far the largest amount of variance in judgments of affective quality, but this does not mean that these dimensions capture all the meaning present in facial expressions. Certain action units may carry additional meaning related to situations likely to give rise to that facial behavior, or other behavior that is likely to follow. The idea that individual facial movements have meaning is not new. Although Darwin (1872/1965) described the expressions of specific emotions, such as joy and anger, he did so by focusing on the individual components of those expressions. It was these individual components, not the wholistic expression of emotion, that Darwin argued became associated with emotions through an evolutionary process. For example, Darwin argued that the contraction of the muscles around the eyes is "a fundamental element in several of our most important expressions" (p. 159). Although this movement came to be associated with various emotional states, Darwin argued that its original purpose was to "protect the eyes during violent expiration." Emotional states with some common physiological or behavioral responses would thus contain similar elements of expression. A similar componential approach to the study of both the experience and the expression of emotion has recently been advocated by Ortony and Turner (1990), and somewhat earlier by Scherer (1984).  58  In fact, Ortony and Turner (1990) have argued that a componential approach to the study of emotion is more useful than approaches based on the notion of basic emotion categories. They argue that there is no good evidence that basic emotions exist. One type of evidence that has been used to support the claim for biologically determined basic emotions has been the cross-cultural agreement on the meaning of some facial expressions (Darwin, 1872/1965; Ekman, 1982; Izard, 1971). The present research offers an alternate interpretation of those findings that is consistent with Ortony and Turner's argument—since the individual facial movements themselves have been shown to be interpreted as conveying emotional meaning, it may be the components of emotion that are biologically given rather than a set of discrete emotions. This possibility can be explored by expanding cross-cultural studies to include judgments of individual facial movements. Taking a bottom-up approach to the study of how observers judge feeling states from facial behavior allows examination of the vast variation in both facial behavior and feeling states. This examination can be made without any restrictive assumptions about which feeling states can be recognized, or which states constitute basic emotions. The results of research undertaken with this approach can nevertheless be compared with prior theory and research that does make such assumptions. It has been shown in this series of studies that observers attribute affective meaning to individual facial movements, and that the meaning attributed to individual facial movements is consistent with theoretically derived predictions about the facial expressions accompanying some emotions considered to be basic. The components of complex expressions were frequently judged as showing the same qualitative emotional state as predicted for the larger expression, though to a lesser extent. The affective meanings of both individual facial movements and complex 59  expressions can be located within a two-dimensional space, capturing the continuous nature of feeling states. The results of these six studies support the idea that a meaningful relationship can be mapped out between individual movements of the face and judgment of feeling states. Additionally, although there clearly remains much to be done, these studies begin the mapping process. The next step is to complete the mapping of the meaning of action units, and to verify the meaning across stimulus persons and presentation modalities. Once reliable meanings have been mapped out for one culture, cross-cultural comparisons could be made to examine which action units, if any, have similar meanings across cultures. Action units were identified that influence primarily judgments of degree of arousal (1 + 2, 5, 26, 41, 42, 43), and others were identified that influence primarily judgments of pleasure (12, 15). For both dimensions, different action units were identified that were reliably rated as high or low on the dimension. Variations in intensity of these action units were shown to influence ratings on the related dimension for both pleasure and arousal. Action unit 12 was rated as showing above average pleasure, with increasing intensity of the action unit rated as showing increasing pleasure. Action unit 15 was rated as showing displeasure, again with degree of displeasure increasing with the intensity of the action unit. Action units 1 + 2, 5, and 26 were all rated as showing above average arousal, and increasing the intensity of these action units increased rated arousal. Conversely, action units 41, 42, and 43 were rated as showing below average arousal. Arousal action units were shown to be at least somewhat additive—the combination of action units one, two and five were rated as showing more arousal than the individual action units presented alone. Combinations of pleasure action units and arousal action units were shown to 60  locate a point in a two-dimensional space consistent with ratings of pleasure and arousal of the individual action units. Pleasure and arousal ratings predicted which terms were rated as most descriptive of faces. How reliable are these findings? The similarities across experiments using different stimulus persons as well as different presentation modes and different subjects are particularly striking. Action unit five (eyes wide) was presented alone in maximum intensity in Study One by a male stimulus person in a filmstrip sequence, where its mean arousal rating was 8.08 (on a 1-to-9 scale). The same action unit was presented at the same intensity in a still photograph of a female stimulus person in Study Three, where its mean arousal rating was 7.86. The combination of action units one, two and five at maximum intensity received a mean arousal rating of 8.46 in Study One, and mean arousal ratings of 8.24 and 8.43 in Study Three. Action unit 15 (lip corner depress) presented at maximum intensity by a male stimulus person in a filmstrip sequence in Study One was rated as 2.04 on the pleasure dimension. The same action unit at the same intensity presented (in combination with several arousal action units) in photographs of a female stimulus person in Study Five was rated as 1.5 and 1.9 on the pleasure dimension. With different sets of stimuli, pleasure and arousal ratings consistently predicted emotion category ratings. The beta weights are similar for both sets of stimuli, and mostly consistent with predictions from Russell's model. An exception is the apparent reversal of locations of "excited" and "surprised" from predicted locations in both Study Two and Study Four. In the circumplex model, "excited" has generally been located as high on both pleasure and arousal, while "surprised" is high on arousal and more neutral on pleasure. In the present studies "surprised" was generally judged as more descriptive of stimuli rated higher on pleasure than was "excited." 61  These studies have demonstrated that it is feasible to map out the judged meaning of individual action units and combinations of action units in a geometrical response space. The results further suggest that the meanings of complex facial expressions are related to the meanings of the individual components of those expressions. The judgments obtained in this set of studies can be thought of as a preliminary dictionary giving the meaning of individual facial movements. These meanings should be treated cautiously, however, due to the necessary limitations of such exploratory research. Although three different models of facial behavior were used in these studies, adequate sampling requires still more models. Only a subset of action units were studied, and this restriction may have implications beyond limiting the entries in the dictionary. Five of the six studies conducted were within-subject designs, and thus observers' ratings may have been influenced by the particular set of stimuli they were asked to judge. Future studies should include more models of facial behavior, more action units, and more between-subject designs. An intriguing possibility for future research on individual action units is the use of computer generated facial expressions. Technology is rapidly advancing to the point where we should soon be able to create composite facial expressions electronically. Such a procedure would allow the production of action unit combinations that are difficult to produce voluntarily. Additionally, this would allow greater control over extraneous variables such as lighting levels, shadows produced by head movements, and differences in clothing or hair. Obviously we would want to supplement such work with research using live models, but the increase in control makes this an appealing possiblity for laying much of the groundwork.  62  This high degree of control will be especially important in the study of how individual action units combine to signal affective meaning. Strong conclusions about the meaning of combinations of action units were not possible in this set of studies due to difficulties in maintaining constant levels of a set of action units while varying the intensity of other action units. Computer generated faces would allow systematic exploration of large numbers of such combinations. The more interesting combinations could then be verified with live models. Much work remains to finish the mapping process and to establish the reliability of judgments with different stimulus persons and presentation modalities. Once this is done, however, the mapping will serve as a useful frame of reference for understanding influences on judgment of feeling states in real life situations. The work that has been outlined so far is laboratory research, removed from real life situations, but necessarily so in order to collect the fundamental data needed at this level of research. The purpose here has been to isolate the contribution of individual facial movements to judgments about the emotion expressed in faces. This information supplies one piece of the puzzle of how observers judge what another person is feeling. The understanding gained from such laboratory research can be thought of as a tool to be used in the larger challenge of discovering how judgments of emotion are made in more complex social situations. In non-laboratory situations the observer often has various sources of information in addition to facial behavior from which to make inferences about another's feeling state. Potentially useful information includes knowledge of the person's characteristic responses, knowledge of the situation, what the person says, body movements or tone of voice. Previous research attempting to discover which source of information dominates judgment have had mixed results. The first generation of such 63  studies used what has been called the cue opposition paradigm (Walker, 1977). In this paradigm observers are presented with incongruent facial and verbal messages (e.g., a negative verbal message accompanied by a smile). Researchers using this method overwhelmingly found that nonverbal information had significantly more impact than verbal information (Argyle, Alkema & Gilmour, 1972; Argyle, Salter, Nicholson, Williams & Burgess, 1970; Burns & Beier, 1973; Graves & Robinson, 1976; Mehrabian & Ferris, 1967; Mehrabian & Weiner, 1967; Walker, 1977). If this conclusion generalizes to real world situations, it suggests that the individual AUs studied here might have a relatively large impact on judgment. A second paradigm used consistent rather than inconsistent messages, and compared judgments of observers presented with audio or visual information to the judgments of observers presented with both. Researchers using this method found that what was said seemed to be relatively more important than other sources of information (Furnham, Trevethan & Gaskell, 1981; Krauss, Apple, Morency, Wenzel & Winton, 1981; Strahan & Zytowski, 1976). Other investigators have found that the relative weight given to facial behavior versus other sources of information depends on such variables as the expressor, the perceiver, the messages in each channel, the context, and previous experience (Bugental, Kaswan, Love, & Fox, 1970; Bugental, Love & Gianetto, 1971; Bugental, Love, & Kaswan, 1971; Ekman, Friesen, O'Sullivan & Scherer, 1980). Some studies have specifically compared the relative influence of facial information versus contextual information. Although the methods of early studies have been criticized (Ekman, Friesen, & Ellsworth, 1982), some more recent studies have found that facial information dominates over contextual information (Nakamura, Buck, & Kenny, 1990; Watson, 1972).  Disentangling the relative weights of different sources of information is complicated by the fact that in any given judgment task the observer is also subject to a number of biasing influences such as mood and previous judgments. Having a dictionary of the meanings generally attributed to facial movements offers another method for investigating the relative importance of facial behavior in judgments of feeling states. Rather than contriving combinations of incongruent verbal and nonverbal cues, researchers could compare the meaning given to facial behavior in context to the general (dictionary) meaning of that facial behavior. Another advantage to a bottom-up approach is that it allows researchers to examine real world facial expressions as they occur. We can look at the meaning of individual action units in a given context instead of waiting for, or creating, a prototypical emotional expression that may not naturally occur with any great frequency. Although it can be argued that the initial work of establishing the meanings of individual action units lacks ecological validity, the use of this information in later real world research offers a unique opportunity to  increase ecological validity by allowing the use of spontaneous facial behavior. Researchers will no longer be limited to the few expressions that have so far been precisely described. There is no evidence that such expressions occur with any great frequency, and it is my guess that individual AUs occur, and are used in making judgments, more freqently than the prototypical facial expressions that have been described. By focusing on individual AUs, any facial behavior can be studied in any context and its meaning in that particular context compared to its meaning in our more neutral laboratory context. Such an approach offers hope for the tremendous task of disentangling the complex influences on judgment.  65  In this paper I have outlined a new approach to the study of judgment of feeling states from facial behavior that promises to fill some existing gaps in our knowledge. A bottom-up approach may help researchers study the full richness of the information communicated by the face. Here I have focused on pleasure and arousal, but there may be additional dimensions of information conveyed by the face (Ortony & Turner, 1990). The data I have presented support the feasibility of a bottom-up approach, and suggest several directions for further research.  References Abelson, R.P. & Sermat, V. (1962). Multidimensional scaling of facial expressions. Journal of Experimental Psychology, 52, 546-554. Argyle, M., Alkema, F., & Gilmour, R. (1972). The communication of friendly and hostile attitudes by verbal and non-verbal signals. European Journal of Social Psychology, 1, 385-400. Argyle, M., Salter, V., Nicholso, H., Williams, M., & Burgess, P. (1970). The communication of inferior and superior attitudes by verbal and non-verbal signals. British Journal of Social and Clinical Psychology, 2, 222-231. Birdwhistell, R.L. (1970). Kinesics and context. Philadelphia: University of Pennsylvania Press. Block, J. (1957). Studies in the phenomenology of emotions. Journal of Abnormal and Social Psychology, 5A, 358-363. Boring, E.G., & Titchner, E.B. (1923). A model for the demonstration of facial expression. American Journal of Psychology, 24, 471-485. Boucher, J.D., & Carlson, G.E. (1980). Recognition of facial expression in three cultures. Journal of Cross-Cultrual Psychology, 11, 263-280. Bruner, J.S., & Tagiuri, R. (1954). The perception of people. In G. Lindzey (Ed.), Handbook of social psychology (Vol. 2, pp. 634-654). Cambridge, Ma: Addison-Wesley. Bugental, D. E., Kaswan, J., Love, L., & Fox, M. (1970). Child versus adult perception of evaluative messages in verbal, vocal, and visual channels. Developmental Psychology, 2, 367-375. Bugental, D. E., Love, L., & Gianetto, R. (1971). Perfidious feminine faces. Journal of Personality and Social Psychology, 17, 314-318. Bugental, D. E., Love, L., & Kaswan, J. (1971). Verbal-nonverbal conflict in parental messages to normal and disturbed children. Journal of Abnormal Psychology, fl, 6-10. Burns, K. L., & Beier, E. G. (1973). Significance of vocal and visual channels in the decoding of emotional meaning. Journal of Communication, 22, 118130. Buzby, D.E. (1924). The interpretation of facial expression. American Journal of Psychology, 35, 602-604.  67  Coleman, J.C. (1949). Facial expressions of emotion. Psychological Monograph, Ea. Darwin, C. (1965). The expression of emotions in man and animals. Chicago: University of Chicago Press. (Original work published 1872) Ducci, L., Arcuri, L., W/Georgis, T., & Sineshaw, T. (1982). Emotion recognition in Ethiopia. Journal of Cross-Cultural Psychology, la, 340-351. Duchenne de Bologne, G.B. (1990). The mechanism of human facial expression (R.A. Cuthbertson, Trans.). Cambridge: Cambridge University Press. (Original work published 1862) Dunlap, K. (1927). The role of eye-muscles and mouth-muscles in the expression of the emotions. Genetic Psychology Monograph, 2, 199-233. Ekman, P. (1972). Universal and cultural differences in facial expressions of emotions. In J. K. Cole (Ed.), Nebraska Symposium on Motivation. 1971 (pp. 207-283). San Diego, CA: Academic. Ekman, P., & Friesen, W.V. (1978). The Facial Action Coding System: A technique for the measurement of facial movement. Palo Alto, Calif.: Consulting Psychologists Press. Ekman, P., Friesen, W.V., & Ellsworth, P. (1982). Does the face provide accurate information? In P. Ekman (Ed.), Emotion in the human face (2nd ed.). New York: Cambridge University Press. Ekman, P., Friesen, W. V., O'Sullivan, M., & Scherer, K. (1980). Relative importance of face, body and speech in judgments of personality and affect. Journal of Personality and Social Psychology, ati , 270-277. Ekman, P., Friesen, W.V., O'Sullivan, M., Chan, A., Diacoyanni-Tarlatzis, I., Heider, K., Krause, R., LeCompte, W.A., Pitcairn, T., Ricci-Bitti, P.E., Scherer, K., Tomita, M., & Tzavaras, A. (1987). Universals and cultural differences in the judgments of facial expressions of emotion. Journal of Personality and Social Psychology, 11, 712-717. Ekman, P., Friesen, W.V., & Tomkins, S.S. (1971). Facial Affect Scoring Technique (FAST): A first validity study. Semiotica, 2, 37-58. Ekman, P., Sorenson, E. R., & Friesen, W. V. (1969). Pan-cultural elements in the facial displays of emotions. Science, 114, 86-88. Engen, T., Levy, N., & Schlosberg, H. (1958). The dimensional analysis of a new series of facial expressions. Journal of Experimental Psycholoay , 55, 454-458.  68  Feleky, A. M. (1914). The expression of the emotions. Psychological Review, 21, 33-41. Fridlund, A. J., Ekman, P., & Oster, H. (1987). Facial expressions of emotions: Review of literature 1970 - 1983. In A. W. Siegman & S. Feldstein (Eds.) Nonverbal Behavior and Communication. Hillsdale, N. J.: Lawrence Erlbaum Associates. Frijda, N.H. (1969). Recognition of emotion. In L. Berkowitz (Ed.) Advances in experimental social psychology (Vol. 4, pp. 167-223). New York: Academic Press. Frijda, N.H., & Phiilipszoon, E. (1963). Dimension of recognition of emotion. Journal of Abnormal Social Psychology, Da, 45-51. Frois-Wittman, J. (1930). The judgment of facial expression. Journal of Experimental Psychology, la, 113-151. Furnham, A., Trevethan, R., & Gaskell, G. (1981). The relative contribution of verbal, vocal, and visual channels to person perception: Experiment and critique. Semiotica, 31, 39-57. Gates, G.S. (1925). A test for ability to interpret facial expressions. Psychological Bulletin, 22, 120. Gladstones, W.H. (1962). A multidensional study of facial expression of emotion. Australian Journal of Psychology, 14, 95-100. Graves, J. R., & Robinson, J. D. (1976). Proxemic behavior as a function of inconsistent verbal and nonverbal messages. Journal of Counseling Psychology, za, 333-338. Hanawalt, N.G. (1944). The role of the upper and the lower parts of the face as the basis for judging facial expressions: I. In posed expressions and "candid camera" pictures. Journal of General Psychology, a, 23-36. Hasdorf, A.H., Osgood, D.E., and Ono, H. (1966). The semantics of facial expressions and the prediction of the meanings of stereoscopically fused facial expressions. Scandinavian Journal of Psycholoay, 7, 179-188. Ittelson, W.H. (1951). Size as a cue to distance: Static localization. Americal Journal of Psychology, E4, 54-67. Ittelson, W.H. (1960). Visual space perception. Berlin & New York: SpringerVe rlag . Izard, C. E. (1971). The face of emotion. New York: Appleton Century Crofts. James, W. (1890). The principles of psychology. New York: Holt. 69  Jarden, E., & Fernberger, S.W. (1926). The effect of suggestion on the judgment of facial expression of emotion. American Journal of Psychology, aZ, 565-570. Kirk, R. E. (1988). Experimental design: Procedures for the Behavioral Sciences, (2nd ed.). Monterey, CA.: Brooks/Cole. Krauss, R. M., Apple, W., Morency, W., Wenzel, C., & Winton, W. (1981). Verbal, vocal, and visible factors in judgments of another's affect. Journal of Personality and Social Psychology, AQ, 4312-4320. Laird, J.D. (1974). Self-attribution of emotion: The effects of expressive behavior on the quality of emotional experience. Journal of Personality and Social Psychology, 24, 475-486. Landis, C. (1924). Studies of emotional reactions. I. A preliminary study of facial expression. Journal of Experimental Psychology, Z, 325-341. Landis, C. (1929). The interpretation of facial expression in emotion. Journal of General Psychology, 2, 59-71. Langfeld, H.S. (1918a). The judgment of emotions from facial expressions. Journal of Abnormal Psychology, la, 172-184. Langfeld, H. S. (1918b). Judgments of facial expression and suggestion. Psychological Review, 25, 488-494. Lindsley, D.B. (1951). Emotion. In S.S. Stevens (Ed.), Handbook of experimental psychology (pp. 473-516). New York: Wiley. McAndrew, F. T. (1986). A cross-cultural study of recognition thresholds for facial expression of emotion. Journal of Cross-Cultural Psychology, 17, 211224. McDougal, W. (1908). An introduction to social psychology. London: Methuen. Mehrabian, A., & Ferris, S. (1967). Inference of attitudes from nonverbal communication in two channels. Journal of Consulting Psychology, a, 248252. Mehrabian, A., & Weiner, M. (1967). Decoding inconsistent communication. Journal of Personality and Social Psychology, fi, 109-114. Mook, D. G. (1983). In defense of external invalidity. American Psychologist, $1, 379-387.  70  Nakamura, M., Buck, R., & Kenny, D. A. (1990). Relative contribution of expressive behavior and contextual information to the judgment of the emotional state of another. Journal of Personality and Social Psycholoay, 52, 1032-1039. Niit, T., & Valsiner, J. (1977). Recognition of facial expressions: An experimental investigation of Ekman's model. Acta et Commentationes Universitatis Tarvensis, 422, 85-107. Ortony, A., & Turner, T.J. (1990). What's basic about basic emotions? Psychological Review, 97, 315-331. Osgood, C.E. (1962). Studies on the generality of affective meaning systems. American Psychologist, 1Z, 10-28. Osgood, C.E. (1966). Dimensionality of the semantic space for communication via facial expressions. Scandinavian Journal of Psychology, 7, 1-30. Plutchik, R. (1962). The emotions: Facts. theories. and a new model. New York: Random House. Rinn, W. E. (1984). The neuropsychology of facial expression: A review of the neurological and psychological mechanisms for producing facial expressions. Psychological Bulletin, 95, 52-77. Ruckmick,C.A. (1921). A preliminary study of the emotions. Psychological Monographs, $1, 30-35. Russell, J. A. (1978). Evidence of convergent validity on the dimensions of affect. Journal of Personality and Social Psychology, aa, 1152-1168. Russell, J.A. (1979). Affective space is bipolar. Journal of Personality and Social Psychology, 31, 345-356. Russell, J.A. (1980). A circumplex model of affect. Journal of Personality and Social Psychology, 32, 1161-1178. Russell, J.A. (1983). Pancultural aspects of the human conceptual orgaiztion of emotion. Journal of Personality and Social Psycholoay, 15, 1281-1288. Russell, J.A., & Bullock, M. (1986). Fuzzy concepts and the perception of emotion in facial expressions. Social Cognition, 4, 309-341. Russell, J. A., & Fehr, B. (1987). Relativity in the perception of emotion in facial expressions. Journal of Experimental Pschyology: General, 116, 223-237. Russell, J.A., & Steiger, J.H. (1982). The structure in persons' implicit taxonomy of emotions. Journal of Research in Personality, la, 447-469.  Russell, J.A., & Mehrabian, A. (1977). Evidence for a three-factor theory of emotions. Journal of Research in Personality, fl, 273-294. Russell, J.A., Weiss, A., & Mendelsohn, G.A. (1989). Affect grid: A single-item scale of pleasure and arousal. Journal of Personality and Social Psychology, 51, 493-502. Schachter, S., & Singer, J.E. (1962). Cognitive, social, and physiological determinants of emotional state. Psychological Review, 51 379-399. Scherer, K. (1984). On the nature and function of emotions: A component process approach. In K. Scherer & P. Ekman (Eds.), Approaches to emotion (pp. 293-317). Hillsdale, NJ: Erlbaum. Schlosberg, H. (1941). A scale for the judgment of facial expressions. Journal of Experimental Psychology, 22, 497-510. Schlosberg, H. (1954). Three dimensions of emotion. Psychological Review, 51, 81-88. Sherman, M. (1927). The differentiation of emotional responses in infants: Judgments of emotional responses from motion picture views and from actual observation. Journal of Comparative Psycholoay, 7, 265-284. Strahan, C., & Zytowski, D. G. (1976). Impact of visual, vocal, and lexical cues on judgments of counselor qualities. Journal of Counseling Psychology, 22, 387-393. Stringer, P. (1973). Do dimensions have face validity? In M. von Cranach & I. Vine (Eds.), Social communication and movement. New York: Academic Press. Tomkins, S.S. (1962, 1963). Affect. imagery. consciousness. (Vols. 1-2). New York: Springer. Walker, M. B. (1977). The relative importace of verbal and non-verbal cues in the expression of confidence. Australian Journal of Psychology, 22, 45-57. Watson, S. G. (1972). Judgment of emotion from facial and contextual cue combinations. Journal of Personality and Social Psychology, 24, 334-342. Watson, D., & Tellegen, A. (1985). Toward a consensual structure of mood. Psychological Bulletin, afi, 219-235. Woodworth, R.S. (1928). How emotions are identified and classified. In M.L. Reymert (Ed.), Feelings and emotions: The Wittenberg Symposium. Worcester, Mass.: Clark University Press.  72  Woodworth, R.S. (1938). Experimental psychology. New York: Holt, Rinehart and Winston. Wundt, W. (1924). An introduction to psychology (R. Pinter, Trans.). London: Allen & Unwin. (Original work published 1912) Zajonc, R.B., Murphy, S., & Inglehart, M. (1989). Feeling and facial efference: Implications of the vascular theory of emotion. Psychological Review, 95, 395-416.  73  Table 1 Action units scorable in the Facial Action Coding System. Li Description  AU Description  1^inner brow raise 2^outer brow raise 4^brow lower 5^upper lid raise 6^cheek raise 7^lids tights 8^lips toward 9^nose wrinkle 10^upper lip raise 11^nasolabial deepen 12^lip corner pull 13^cheek puff 14^dimpler 15^lip corner depress 16^lower lip depress 17^chin raise 18^lip pucker 19^tongue show 20^lip stretch 21^neck tighten 22^lip funnel 23^lip tight 24^lip press 25^lips part 26 jaw drop 27^mouth stretch 28^lip suck 29^jaw thrust 30 jaw to sideways  31^jaw clench 32^bite 33^blow 34^puff 35 cheek suck 36^tongue bulge 37^lip wipe 38^nostril dilate 39^nostril compress 41^lids droop 42^eyes slit 43^eyes closed 44^squint 45^blink 46^wink 51^head left 52^head right 53 head up 54 head down 55^head tilt left 56^head tilt right 57^head forward 58 head back 61^eyes left 62^eyes right 63 eyes up 64 eyes down 65^walleye 66^crosseye  74  Table 2  Combinations of action units predicted by Ekman and Friesen to express seven basic emotions. Emotion^Prototypes  Major variants  Surprise^1 + 2 + 5x + 26 1 + 2 + 5x + 27  1 + 2 + 5x 1 +2+26 1 +2+27 5x + 26 5x + 27  Fear^1+2 +4 +5 +20+25, 26, or 27^1 +2 +4+5 +LorR 20 +25, 26,or27 1+2 +4 +5 +25, 26, or 27^1+2 +4+5 1 + 2 + 5z with or without 25, 26, or 27 5 + 20 with or without 25, 26, or 27 Happy^6 + 12 12y Sadness^1 + 4 + 11 + 15x with or without 54 + 64 1 + 4 + 11 with or without 54 + 64 1 + 4 + 15 with or without 54 + 64^1 + 4 + 15x with or without 54 + 64 6 + 15 with or without 54 + 64^1 + 4 + 15x + 17 with or without 54 + 64 11 + 15x with or without 54 + 64 11 + 17 25 or 26 may occur with all prototypes or major variants Disgust^9 9 + 16 + 15, 26 9+17 10 10 + 16 + 25, 26 10 + 17 Anger^4+5 + 7 +10 +22+23 +25, 26 4 +5 + 7+ 10 +23 +25,26 4 + 5 + 7 + 23 + 25, 26 4 +5 + 7+ 17 +23 4+5 + 7 +17 +24 4 +5 + 7+ 23  75  Any of the prototypes without any one of the following: 4, 5, 7, or 10  Table 3  Ekman filmstrip series action units used in Studies One and Two.  Single action units action unit number^action unit description 1^ inner brow raise 2^ outer brow raise 4^ brow lower 5^ upper lid raise 6^ cheek raise 7^ lids tight 9^ nose wrinkle 10^ upper lip raise 15^ lip corner depress 17^ chin raise 46^ wink Combinations action unit number 1+4 1+2 1+2+4 1+2+5 4+5 5+7 16 + 25 9 + 16 + 25 10 + 16 + 25  action unit description inner brow raise & brow lower inner brow raise & outer brow raise inner & outer brow raise & brow lower inner & outer brow raise & upper lid raise brow lower & upper lid raise upper lid raise & lids tight lower lip depress & lips part nose wrinkle, lower lip depress & lips part upper lip raise, lower lip depress & lips part  •  76  Table 4 Mean pleasure and arousal ratings of initial and ending poses in Ekman filmstrip sequences in Study One.  AUs 1 2 5 4 6 7 9 10 15 17 46* 1+4 1+2 1+2+4 1+2+5 4 + 5* 5+7 16 + 25 9 +16 +25 10 + 16 + 25  Initial^ratings Pleasure^Arousal 5.04 4.65 4.88 4.38 4.42 4.31 4.26 5.48 4.30 4.93 4.35 4.96 5.73 4.81 5.35 3.04 5.27 4.48 5.74 4.85  5.04 5.58 5.04 4.92 4.92 4.50 4.41 4.22 4.41 4.07 4.77 4.19 4.92 5.04 4.88 6.19 4.23 4.78 4.59 4.93  Ending ratings Pleasure^Arousal 3.12 6.50 3.69 2.38 5.88 2.46 1.93 1.81 2.04 4.00 5.54 2.38 6.58 3.42 5.62 2.38 3.23 2.33 1.67 1.93  4.19 7.69 8.08 6.15 5.65 5.08 6.74 6.19 3.78 4.67 5.31 4.31 7.77 5.65 8.46 7.35 6.96 7.04 7.93 6.48  * The sequence in which action unit 46 (wink) was presented began with a neutral expression and returned to a neutral expression. For the combination of action units four (brow lower) and five (upper lid raise) the sequence began with four present, and five was added.  77  Table 5  Calculated F ratios for sequence by time interaction in Study One. Probabilities are calculated with the Huynh-Feldt correction for violation of sphericity. Group 1 Pleasure E 7.83  6/156  clf  prob. Arousal E dl .00001 10.88 6/156  prob. .00001  Group 2 Pleasure F 27.45  di 6/150  prob. Arousal E ji .00001 11.73 6/150  prob. .00001  Group 3 Pleasure F di prob. Arousal F di prob. 13.83 5/125 .00001 17.80 5/125 .00001 *Individual analyses were computed since different groups of subjects rated different sequences. Neither group size nor number of sequences rated was identical between groups.  Table 6 Changes in pleasure and arousal ratings of Ekman's filmstrip sequences in Study One. Positive numbers indicate an increase from the beginning to the end of the sequence; negative numbers indicate a decrease. AUs 1 2 4 5 6 7 9 10 15 17 46 1+2 1+4 4+5 5+7 16 + 25 1+2+4 1+2+5 9 + 16 + 25 10 + 16 + 25  Pleasure *-1.92 *1.85 *-2.00 -1.19 1.46 *-1.85 *-2.33 *-3.67 *-2.26 -.93 1.19 .85 *-2.58 -.66 °-2.04 *-2.15 -1.39 .27 *-4.07 *-2.92  Arousal -.85 *2.11 1.23 *3.04 .73 .58 *2.33 *1.97 -.63 .6 .54 *2.85 .12 1.16 *2.73 *2.26 .61 *3.58 *3.34 °1.55  *significant at the .001 level °significant at the .01 level  79  Table 7  Mean category ratings of initial poses in filmstrip sequences in Study Two. sleepy  bored  1 2  3.05 2.30  5.45 3.55  5 6 7 9 10 15 17 46 1 +2 1+4 4 + 5"  3.25 2.85 4.00 2.90 3.25 3.25 3.50 3.55 3.45 4.45 1.75  4.80 4.35 5.00 4.70 5.55 4.95 4.50 4.50 4.50 5.25 3.85  Action units  4  5+7  16+25 1 +2+4 1 +2+5 9+16+25 10+16+25  Mean s. d.  2.45  3.90  5.10  5.40  disgust^anger  Li^surprise excited  happy  JIm  4.20 3.65  3.30^2.55 4.35 3.60  2.70^2.60^2.15 3.10 3.00 2.50  2.75 2.60  6.15 5.30  3.55 3.10 4.70 3.70 3.40 4.20 5.20 3.65 3.80 5.10 3.15  3.10 3.65 3.50 3.35 2.40 3.15 3.10 2.95 2.20 2.55 4.90  2.30 2.60 1.90 1.85 2.15 2.35 1.85 2.30 3.00 1.90 2.90  2.55 3.10 2.35 2.20 3.10 1.80 2.10 2.50 4.15 3.25 1.65  5.65 5.55 5.95 5.80 6.05 4.70 5.75 6.00 6.10 6.05 4.10  3.55  3.80  x,24  4Q  4.10 3.85 3.10 3.60 3.50  3.32 .53  4.72 .63  3.88 .59  2.85 3.95 3.45 3.45  3.20 5.50 4.40 4.20  3.60  3.35  2.35  2.20  2.55 2.90 2.85 2.90 2.65 2.98 .45  3.20 3.90 2.90 3.30 2.45 4.25 2.15 3.05 2.35 2.10 5.00  2.40  2.20  2.25  2.05  2.05  2.75  2.20 2.20 2.25 2.00 2.30 2.65 2.60 2.10 2.60 2.50 2.90  2.30 3.50 2.80 2.55 2Q  3.70 2.70 2.40 2.60  2  2.96 .68  2.49 .38  2.15 2.10 1.80 1.70 2.40 1.75 1.90 2.15 2.75 2.10 2.45  2.60  3.15  5.75  6.65  2^2Q  2.75 2.85 2.85 2.25 2.20  2.25 3.45 4.50 2.55 2.45  3.70 5.75 5.90 5.45 5.10  2.27 .52  2.39 .40  2.81 .68  5.65 .64  3.95 2.35 2.05 2.00  *Ratings for 4 + 5 were excluded in calculating means across photographs since the initial pose was not neutral.  Table 8 Mean category ratings of ending poses in filmstrip sequences in Study Two.  Categories with the highest rating for each sequence are shown in boldface  type.  Action units 1 2 4 5 6 7 9 10  15  17 46 1 +2 1 +4 4+5 5+7 16+25 1 +2+4 1 +2+5 9+16+25 10+16+25  sleepy  bored  3.15 1.75 1.70 1.35 2.55 2.80 1.70 1.40  3.85 2.05 2.25 1.45 2.40 2.85 2.55 2.35  2.95 3.00 2.25 3.30 1.30 1.75 1.80 3.45 1.10 1.25 1.70  4.25 4.55 3.25 3.95 1.65 2.55 1.85 4.00 1.50 1.15 1.55  3.00  3.05  disgust  anger  fit surprise excited haooy  calm  5.85 2.25 2.75 2.25 2.35 2.75 2.35 2.25  4.05 2.65 6.55 5.10 3.20 5.65 6.85 7.15  2.10 1.55 6.60 4.85 2.15 6.15 5.90 6.25  5.20 2.40 2.55 5.90 2.70 2.25 2.10 2.65  4.50^2.70 7.55 5.90 2..65 3.85 5.60 6.60 4.95 3.60 3.80 2.60 3.40 2.05 2.50 3.50  1.95 5.30 1.40 1.90 6.25 1.95 1.05 1.15  3.30 2.90 2.40 1.75 3.10 2.70 2.00 1.95  5.95 2.85 2.65 6.60 2.50 2.30 3.45 5.25 2.25 2.30 2.75  2.85 2.95 2.60 3.55 3.80 4.85 4.20 4.65 3.25 5.70 6.10  2.50 1.90 2.95 4.05 4.95 3.35 4.45 3.55 5.65 2.40 3.95  2.40 2.50 7.00 3.40 6.00 3.35 2.95 4.20 7.35 1.85 3.00  2.15 3.75 3.90 1.80 2.10 1.85 1.15 2.45 3.35 1.05 1.15  3.95 5.70 3.60 3.25 2.10 2.10 1.55 3.90 1.90 1.10 1.40  6.50  3.75  80  3.20  2.45 2.65 2.05 2.55 4.95 6.50 5.10 3.50 2.50 7.55 5.70  3.15  1.95  1.75  1.90 2.80 4.85 3.50 5.90 5.75 4.50 3.65 6.60 5.65 4.15  1.55  3.15  Table 9 Calculated F ratios for sequence by time interaction in Study Two. Probabilities are calculated with the Huynh-Feldt correction for violation of sphericity where required. Group 1 2 3 1 2 3 1 2 3 1 2 3 1 2 3 1 2 3 1 2 3 1 2 3 1 2 3 1 2 3  Category sleepy sleepy sleepy bored bored bored sad sad sad disgusted disgusted disgusted angry angry angry afraid afraid afraid surprised surprised surprised excited excited excited happy happy happy calm calm calm  sit  6/114 6/114 5/95 6/114 6/114 5/95 6/114 6/114 5/95 6/114 6/114 5/95 6/114 6/114 5/95 6/114 6/114 5/95 6/114 6/114 5/95 6/114 6/114 5/95 6/114 6/114 5/95 6/114 6/114 5/95  4.36 2.58 4.07 6.64 5.82 3.17 10.82 7.91 11.30 16.27 7.99 6.65 22.28 28.49 17.15 1.27 18.67 5.48 3.04 19.95 13.18 12.23 6.81 6.39 6.59 22.53 3.45 9.26 7.07 6.15  =h. .001  .05 .002 .0000 .0000 .01 .0000 .0000 .0000 .0000 .0000 .0000 .0000 .0000 .0000 .28 .0000 .0002 .01 .0000 .0000 .0000 .0000 .0002 .0000 .0000 .007 .0000 .0000 .0001  Table 10 Beta weights for pleasure and arousal scores in Study One as predictors of category judgments in Study Two. Nonsignificant weights are excluded, though the model tested included both pleasure and arousal as category predictors.  Category angry excited calm afraid disgusted sad sleepy surprised happy bored  pleasure -.86 • .69 . -.81 • .34 .54 .93 .41  arousal .48 .85 -.73 • • -.76 -.93 .43 • -.85  132  .76 .77 .75 • .63 .62 .83 .59 .83 .71  E  26.64 27.65 25.73 . 14.26 13.88 40.05 12.15 40.91 20.40  izslt.  .0001 .0001 .0001 • .0002 .0003 .0001 .0005 .0001 .0001  Beta weights for pleasure and arousal scores from Study Three as predictors of category judgments in Study Four. Category  Pleasure  angry excited calm afraid disgusted sad sleepy surprised happy bored  -.85 .17 . • -.93 -.29 • .39 .80 •  arousal • 1.04 -.96 .93 • -.99 -.93 1.01 .62 -.89  82  R2  F  =L.  .71 .93 .90 .79 .80 .89 .90 .92 .71 .72  19.61 103.43 72.92 29.79 31.62 62.11 73.22 88.45 • 19.52 20.34  .0001 .0001 .0001 .0001 .0001 .0001 .0001 .0001 .0001 .0001  Table 11 Changes in category ratings of filmstrip sequences in Study Two. Positive numbers indicate an increase in the rated accuracy of the label from the beginning to the end of the sequence; negative numbers indicate a decrease. Action units  sleepy ladatd  1 2 4 5 6 7 9 10 15 17 46 1 +2 1 +4 4+5 5+7 16 + 25 1+2+ 5 1+2+ 4 10+ 16 + 25 9+16+ 25  .10 -.55 -.75 -1.90 -.30 -1.20 -1.20 -1.85* -.25 -.55 -.55 -1.20 -1.15 -.45 -2.15 -1.05 -2.35* -.50 -1.50 -2.20*  -1.60 -1.50 -2.85* -3.35* -1.95 -2.15* -2.15 -3.20* -1.90 -.25 .05 -1.25 -1.30 -2.20* -2.85* -1.35 -2.90* -1.50 -3.15* -3.05*  lad 1.65 -1.40 -.80 -1.30 -.75 -1.95* -1.35 -1.15 2.30 .75 -.80 -1.15* 1.50 -.65 -1.50 -.65 -.85 1.40 -.75 -1.30  disgust  =e  .75^-.45 -2.80* -.95 3.25* 2.95* 2.00 1.65 -.45 -1.75 2.15* 3.25* 3.50* 2.60* 4.75* 3.80* -1.05 .60 .30 -.25 -.40 0.00 -.30 .40 1.00 .45 -1.10 -.05 2.50* 4.30* 1.65 2.80* -.30 .40 0.00 1.75 3.45* 2.80* 2.80* 5.00*  *significant with experimentwise error set at .001  83  feat surprise excited 2.50* -.10 .15 3.70* .50 0.00 .10 .35 .50 -.10 -.20 .35 1.55 2.05 1.30 .75 3.25* .85 1.10 -.20  1.90^.55 4.55* 2.80* 1.60 .45 3.30* 4.45* 2.35* 1.50 1.90* .80 1.55 .35 1.35 .10 -.60 .20 .05 .50 .50 .35 1.85* 4.25* 1.30 1.60 3.00* 3.55* 3.00* 1.30 -1.00 1.75 5.30* 3.75* 1.85 .80 .80 1.95 -.15 3.40*  = calm -.80 2.70* -1.20 -.65 3.15* -.40 -1.15 -1.95* -.25 .05 1.25 -.25 -1.45 .45 -1.30 -1.10 -1.15 -1.00 -1.30 -1.50  -2.85* -2.40 -3.35* -3.90* -2.45* -3.25* -3.80* -4.10* -1.55 -1.80 -.30 -2.50 -2.80* -2.00 -4.55* -2.15* -4.00* -1.85 -3.70* -4.35*  Table 12 Mean pleasure and arousal ratings of photographs in Study Three, in which action units were selected to vary along the arousal dimension. Photographs are ordered by arousal ratings. photo 2 1 3 4 5 6 11 7 8 9 16 10 15 12 13 14 17 18 19  Pleasure 6.00 5.71 4.29 3.86 3.57 4.14 3.86 4.76 4.05 4.67 3.57 4.95 5.10 2.48 3.24 2.71 4.71 5.22 4.62  al 43 43, 54y 42, 64 41, 64 41, 64 neutral 5w 1w + 2w 1x+2x 1y+2y 1x + 2x, 5x, 25 1z+2z 1w + 2w, 5w, 25 5x 5y 5z 1y + 2y, 5y, 25 1z + 2z, 5z, 26 1z + 2z, 5z, 26  84  Arousal 1.38 2.10 2.57 3.10 3.62 3.90 4.48 4.76 4.86 5.76 5.81 5.86 6.48 6.80 7.33 7.86 7.95 8.24 8.43  Table 13 Mean emotion category ratings of photographs in Study Four. The highest category rating for each photograph is shown in boldface type. #^Alb  sleepy  bored  calm  sad  disgust happy  angry  afraid surprise  excited  2^43 1^43, 54y 3^42, 64 4^41, 64 5^41, 64 6^neutral 11^5w 7^1w+ 2w 8^1x + 2x 9^1y + 2y 16^lx + 2x, 5x, 25 10^1z + 2z 15^1w + 2w, 5w, 25 12^5x 13^5y 14^5z 17^ly + 2y, 5y, 25 18^1z + 2z, 5z, 26 19^1z + 2z, 5z, 26  7.10 7.14 6.24 5.33 5.05 3.48 3.48 3.00 2.71 1.95 2.57 2.62 2.29 2.00 1.86 1.52 1.52 1.29 1.29  3.76 4.43 5.62 5.38 5.19 5.19 5.81 5.05 4.90 4.14 3.62 3.95 2.86 2.52 2.14 2.10 1.71 1.52 1.52  6.38 6.29 6.05 5.52 5.48 5.57 5.76 6.00 5.14 4.43 4.29 4.33 3.62 2.86 2.67 2.38 2.10 1.52 1.52  3.95 4.90 4.67 5.24 4.86 3.95 3.90 3.62 3.90 3.14 3.76 3.14 3.33 3.24 2.81 2.48 2.05 1.95 2.05  3.14 2.33 3.90 3.90 4.38 4.81 4.57 3.43 3.71 3.67 3.86 3.62 3.19 5.14 4.76 5.62 2.90 2.81 3.05  2.48 1.95 2.14 2.01 2.00 2.05 2.14 2.43 2.57 2.71 2.52 2.81 3.43 1.76 1.90 1.76 2.90 3.14 2.81  2.71 1.48 3.67 3.24 3.62 5.14 4.05 3.71 3.90 3.19 3.19 3.05 2.90 6.24 5.81 6.62 2.86 2.05 2.67  2.38 1.81 2.67 2.67 2.86 2.71 2.52 3.10 3.24 2.90 4.29 3.19 4.57 3.52 4.67 4.14 5.05 5.38 6.76  1.95 1.24 1.62 1.81 1.86 1.71 2.33 2.67 3.05 4.67 5.48 5.10 5.71 3.57 4.76 4.33 7.24 7.71 7.48  1.95 1.33 1.76 2.05 1.95 2.24 2.48 2.76 3.10 3.67 2.95 3.48 4.62 4.43 4.05 5.33 5.76 5.62 6.10  correlation with arousal  -.95  -.82  -.95  -.90  .12  .36  .25  .87  .89  .95  85  Table 14 FAGS scores of the photograph series used in Studies Five and Six, with predicted pleasure ratings. Photo  FAGS score  12-6 12-5 12-4 12-3 12-2 12-1  1x + 2x + 5y + 12y + 26 1w + 2w + 12y + 26 1w + 2w + 5x + 12y + 26 1w+2w+12y+ 26 1x+ 2x+ 5y+ 12x+ 26 1x + 2x + 5w + 12x + 26  highest highest high high high high  high high high high high high  N-1 N-2  neutral neutral  neutral neutral  medium medium  A-1 A-2 A-3  1x+2x+26 1x + 2x + 26 1y+2y+5y+26  neutral neutral neutral  high high high  4-1 4-2 4-3 4-4 4-5  1x+2x+4x+5y+26 1y+2y+4x+5x+26 1z+2z+4y+26 1z+2z+4y+5z+26 1z+2z+4y+5z+26  ow ow ow ow ow  high high high high high  15-1 15-2 15-3 15-4 15-5 15-6 15-7  1y+2y+5x+15x+26 1y+2y+5z+15x+26 1y+2y+5x+15x+26 1y+2y+5x+15y+26 1z+2z+5y+15y+26 1z+2z+5y+15z+26 1z+2z+5z+15z+26  ow ow ow ow ow ow ow  high high high high high high high  7-1  1x+2x+5w+7x+26  ow  high  Expected pleasure  86  Expected arousal  Table 15 Mean pleasure and arousal ratings of photographs in Study Five. Predicted as high Photo pleasure 12-6 8.2 12-5 8.1 12-4 7.2 12-3 7.0 12-2 6.6 12-1 6.4  in pleasure as! arousal .77 8.1 .73 6.7 .89 6.5 5.9 1.02 6.4 1.40 1.16 5.5  asL  .80 1.63 1.38 1.20 1.40 1.43  Predicted as neutral in pleasure and arousal (no AUs) Photo pleasure Ste. 5,sL arousal N-1 3.2 4.1 1.05 1.50 N-2 3.3 1.41 1.41 4.5 Predicted as neutral in pleasure (arousal AUs only) Photo pleasure arousal asi 5.d A-1^4.1 1.16 5.1 1.95 A-2^4.1 1.45 5.3 1.39 A-3^4.1 1.52 6.6 1.45 Predicted as low in pleasure Photo^pleasure 5,_d arousal 4-1 3.9 1.33 5.6 4-2 3.5 1.46 5.2 4-3 3.0 1.21 4.9 4-4 3.2 1.38 4.9 4-5 2.9 1.58 6.0 15-1 3.3 1.30 5.8 15-2 2.7 .92 7.4 15-3 2.7 1.21 7.1 15-4 2.4 .97 7.2 15-5 1.8 .82 8.0 15-6 1.9 .92 8.4 15-7 1.5 .73 8.4 7-1 4.0 1.16 5.3  87  5.d. 1.54 1.68 1.49 2.19 1.68 1.61 1.27 1.68 1.14 .95 .89 1.00 1.56  Table 16 Mean category ratings of photographs in Study Six. Categories with the highest rating for each photograph (except neutral ones) are shown in boldface type. Photo  IY.  No action units  N-1 N-2  10 17  Arousal AUs only  A-1^10 A-2 20 A-3 20  Ed  Surprised  Ed  Airaid  ad  1.70 1.06  0.68 0.24  1.50 2.53  0.71 1.46  1.90 5.29  0.88 1.53  3.10 2.30 2.60  1.20 1.56 1.70  6.10 6.80 5.95  0.74 1.01 1.57  5.80 4.90 5.05  1.03 1.71 1.91  1.27 0.00 1.53 2.04 2.07  5.80 3.20 6.40 5.30 6.00  1.20 2.10 1.27 1.34 1.70  4.65 7.80 4.55 6.50 7.60  2.16 0.42 1.93 1.08 0.70  1.67 1.27 1.35 0.82 1.26 1.78 1.66  5.10 5.00 6.10 3.40 3.65 3.95 4.15  1.74 1.94 0.88 2.12 2.30 1.64 1.27  6.10 6.80 6.40 5.70 5.20 6.70 6.75  1.74 0.92 1.27 1.49 1.85 1.84 1.33  4.65  1.66  5.90  1.41  7.00 6.60 5.90 4.80 6.55 7.50  1.56 0.70 1.60 1.32 1.28 0.97  2.70 4.90 2.70 2.30. 2.10 1.40  2.06 1.66 1.34 1.57 1.59 0.52  Excited  Low pleasure AUs (4)  4-1 4-2 4-3 4-4 4-5  20 10 20 10 10  2.35 1.00 2.70 3.80 3.50  Low pleasure AUs (15)  15-1^20^2.20 10 15-2 1.60 10 15-3 2.40 15-4 10 1.70 15-5 20 1.70 15-6 20 3.00 15-7 20 3.30  Low pleasure AUs (7--unintended)  7-1^20^3.40^1.05 High pleasure AUs (12)  12-1^10^3.50 12-2 10 4.00 12-3 10 5.00 12-4 10 3.80 12-5 20 6.75 12-6 10 6.30  2.12 1.76 1.16 0.79 0.85 0.82  Figure 1  Circumplex model of emotion space. Multidimensional solution of judgments of 14 facial expressions (from Russell & Bullock, 1986).  M Surprise  •  N Surprise •  L Fear •  A Excitement •  K Anger •  B Happiness ^•  J  Disgust  C Happiness • • 0 Calmness  • I Sadness •  E Calmness •  H Sadness •  F Sleepiness • G Sleepiness •  89  Figure 2 Mean pleasure and arousal scores of action unit sequences in Study One.  9 8  110'  7  Arousal 5  ...  4  ....^.  • • •  ........  ..  3  .,  2  1  ^  3^4^5^6 Pleasure  O Initial judgments--no AUs present O Initial judgment of 4 + 5 (AU 4 present) •  Ending judgments--various AUS present  O Ending judgment of 46 (no AUs present)  90  ^  8^9  Figure 3  Mean pleasure and arousal ratings of single action unit ending sequences in Study One.  t^I 8 7  4  2•  ,  6 Arousal 5  ......... - ..... . ... L .. . ..... _  ...  60  ..... ....... 1 7.111 .... . ......... . ......... 1711 1  15 4  •  460  ... ^  ...... ow.... immomm. ..... ;11...........•..■  ■  3  - ...... ..--- •-•  2  1 i i  1  ,  i i  1^2^3^4^5^6 Pleasure  91  ^  8^9  ^ ^  Figure 4 Vector plot of ratings of combinations of AUs and component AUs in Study One.  9  I^I^ 1^1 i .. 1+2 +5 5  9 +16 + 25;  8  F  ..^;  4+5 16 125  7  : 5 +7  .  9  10 +16+ 25 10  6  ... ..  Arousal 5 1:+4 —. —. ..  4 ^a ^.  ^I  1  i ....... __ ii ....... .... . , . . . i^i r _ ....... .... I^I^I^ .  ....^  1  ..._.._..4  ^t^i^ ^.^.^ • .^'.^ .^' .^. •  3  — —..... ... s ..... —..—.  2  ..--.  ......  —  i  .--. .. ..—  I^I 1^1  .^. . .^• .^•  I.. .^. I ■^. . ^  .. I. ..---..i. ...... —. i.—. .. .. I I^I I^I^I 8^•^s .^•^• .^.^. .^• I^ i i i^I I ^i^ I --.• • ., ..... ..^ ...^ t^ ..... ....... .1 -. -1....... .. ..I, ^ -^ 1^ '. 1 I^I^I I I I i —^ e^I^I^ I^I^I •^• • .^.^.^  .^.^. •.^• •^•  ^I^I^I ^I^I^I^I 1 1  ^  2^3^4  ^  5^6  Pleasure  92  ^  7^8^9  ^  Figure 5 Mean pleasure and arousal ratings of photographs in Study Three.  9 8  I^I^I^I^I^I^I d z 1+ 2z + 54 + 26 i .^.^. .^.^.^; 0 1z + 2z+ 5z + ;26 - " " "--:• "SiO • :- " " " " :^0 .q •+• - Zi -+• tjf- + • N - T • - " • " - • 7 - . ''''' — 5 g0  7 6 Arousal 5  ^i  .^. .............................. . t + x t  3  1  NI+  5w +  :.  _^....... ..41.t. L4.0 .. ^ .^; 42+ 640  2  01w +  1 y. + 2y .^.^. . .^.^.^..^a• _ ......./ ....... ... t..i.).( ..; .2).(0. . ^ w + 2w Q1 5w 0 . .^..^.^ .^. .^ .^.^ ••--••-••---•----------------------OHENAT-----,41 -4 640 .^  4  •- -- • -• - - • •  :5x^  -.^  •  •  .^'^.^. .^•.^..^.  ••. •  •^•  ^I^I 1^2^3^4^5^6  Pleasure  93  7  ^ ^ 8 9  Figure 6  Mean pleasure and arousal ratings of photographs in Study Five.  9  ..... . ............ .......... .01 2  8 7  .....  ...4 .  a  12-i  6  e112-4  012-5 ....... • -  •  Arousal 5  ...  141 7_,G 4-1 A2  .......^4-4 ; 0 N-1  4  ^0.tU . .... ^ .....^.... ^ .  3  ......^......... ..... ^ .........^...... .  ........  2 1  1^2^3^4^5  ^  Pleasure O  No action units  O  Arousal AUs only  O  AU 12  (31 AU 15 O  AU 4  •  AU 7  34  6^7^8  ^  9  Figure 7  Changes in pleasure ratings of photographs in Study Five plotted against rank ordered intensity of action unit 12.  9 12y• 12y• 8 7  12y • 12y0 12x • 12x •  6  Pleasure 5 4  3 2 1  1^2^3^4^5^6  Rank order of AU  95  Figure 8 Changes in pleasure ratings of photographs in Study Five plotted against rank order of intensity of action unit 15.  9 8 7 6 Pleasure 5 4 15x • 15x• 15x1 15y•  2  15y. 15z• 15z•  1  1^2^3^4^5^6^7 Rank ordered Intensity of AU  96  Figure 9  Ratings of descriptiveness of label "Excited" to photographs in Study Six plotted against pleasure ratings of same photographs from Study Five.  1  ^ ^ ^ ^ ^ ^ ^ 2 3 5 6 7 8 9 Pleasure  •  High pleasure AUs  O  No pleasure AUs  •  Low pleasure AUs  97  .  Figure 10  Ratings of descriptiveness of label "Afraid" to photographs in Study Six plotted against pleasure ratings of same photographs from Study Five.  8  7  0 0 0  6  e  CD  Afraid 5  4  3  0 0 0  2  1 1^2^3^4^5^6^7^8^9 Pleasure  O  High pleasure AUs  •  No pleasure AUs  O  Low pleasure AUs  98  Figure 11  Ratings of descriptiveness of label "Surprised" to photographs in Study Six plotted against pleasure ratings of same photographs from Study Five.  8  7  0  6  Surprised 5  4  3  2  1 1^2^3^4^5^6^7^8^9 Pleasure  O  High pleasure AUs  O  No pleasure AUs  O  Low pleasure AUs  99  Appendix Description of Faces  Please examine the face in the photograph carefully. Look for the feeling that is expressed there. Below is a list of words that can be used to describe feelings. We would like you to rate how accurately each word below describes the feeling you see in the photograph. Please use the following 1-8 rating scale for your answer. (You are not restricted in any way in how you use the scale. You may assign every word the same number or every word a different number--whatever you think is most appropriate). 1 2 3 4  = = = =  5 6 7 8  extremely inaccurate very inaccurate quite inaccurate slightly inaccurate  = = = =  slightly accurate quite accurate very accurate extremely accurate  Feeling at START of Sequence ^  angry^  ^  excited  ^  calm  ^  afraid  ^  disgusted  sad sleepy  surprised happy bored  Feeling at END of sequence ^  angry  ^  excited calm  sad sleepy surprised  ^  afraid  ^  disgusted  loo  happy bored  The Affect Grid You use the "affect grid" to describe feelings. It is in the form of a square -- a kind of map for feelings. The center of the square (marked by X in the grid below) represents a neutral, average, everyday feeling. It is neither positive nor negative. •  The right half of the grid represents pleasant feelings. The farther to the right the more pleasant. The left half represents unpleasant feelings. The farther to the left, the more unpleasant.  EXTREMELY  EXTREMELY  UNPLEASANT  PLEASANT  PEELINGS  FEELINGS  101  The vertical dimension of the map represents Cegree of arousal. Arousal has to do with how wide awake, alert, or activated a person feels -- independent of whether the feeling is positive or negative. The top half is for feelings that are above average in arousal. The lower half for feeling below average. The bottom represents sleep, and the higher you go, the more awake a person feels. So, the next step up from the bottom would be half awake/half asleep. At the top of the square is maximum arousal. If you imagine a state we might call frantic excitement (remembering that it could be either positive or negative), then this feeling would define the top of the grid. EXTREMELY HIGH AROUSAL  EXTREME SLEEPINESS If the "frantic excitement" was positive it would, of course, fall on the right half of the grid. The more positive, the farther to the right. If the "frantic excitement" was negative, it would fall on the left half of the grid. The more negative, the farther to the left. If the "frantic excitement" was neither positive nor negative, then it would fall in the middle square of the top row, as shown below. i  ■  , r i  -----  i  102  Other areas of the grid can be labeled as well. Up and to the right are  feelings of ecstasy, excitement, joy. Opposite these, down and to the left, are feelings of depression, melancholy, 15tadness, and gloom. Up and to the left are feelings of stress and tension. Opposite these, down and to the right, are feelings of calm, relaxation, serenity. EXCITEMENT  STRESS^  1111  111  1  1  1  ,  111 ■M 1  RELAXATION  DEPRESSION^  103  Feelings are complex. They come ia all shades and degrees. The labels we have given are merely landmarks to help you understand the affect grid. When actually using the grid, put an X anywhere in the grid to indicate the exact shade and  intensity of feeling. Please look over the entire grid to get a feel for the meaning of the various area s .  Example: Suppose that you were just surprised. Suppose further that the surprise was neither pleasant nor unpleasant. Probably you would feel more aroused than average. You might put your mark as shown to the right.  Example: Suppose, instead, that you were only mildly surprised but that the surprise was a very pleasant one. You might put your mark as shown to the right.  


Citation Scheme:


Citations by CSL (citeproc-js)

Usage Statistics



Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            async >
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:


Related Items