Figure 1.
The setting for the data collection (from Gironzetti, 2017a)
34
Figure 2.
Schematic representation of the setting for the data collection (from Gironzetti, 2017a)
34
Figure 3.
Example of humor coding in ELAN (humorous utterance highlighted in dark gray)
38
Figure 4.
Sample smiling intensity coding in ELAN with a dedicated tier for each participant
38
Figure 5.
Sample line graph of joint smiling behavior with explanatory labels
39
Figure 6.
Predicted smiling intensity in the presence and absence of humor
41
Figure 7.
Individual smiling intensity: Participants’ smiling baseline and smiling with humor
41
Figure 8.
Most frequent smiling behaviors for baseline (left, SIS 0, no smiling) and humor (right, SIS 3, wide open mouth smiling) conditions across conversations
42
Figure 9.
Joint framing smiling pattern (segment JYH8, Jabline)
43
Figure 10.
Smiling accommodation gesture after the humorous event (segment DAH8, Jabline)
44
Figure 11.
Smiling synchronicity for humorous and non-humorous segments (significant differences are marked with an asterisk)
46
Figure 12.
Percentage of time participants looked at the interlocutor’s eyes
49
Figure 13.
Percentage of time participants looked at the interlocutor’s mouth
50
Figure 14.
Side-by-side view of facial muscles and smiling facial display [First image by J. Sobotta, Illustration: K. Hajek and A. Schmitson – Sobotta’s Atlas and Textbook of Human Anatomy, 1909, Public Domain,
https://commons.wikimedia.org/w/index.php?curid=29817225]
62
Figure 15.
The Smiling Intensity Scale (from Gironzetti, Pickering, Huang, Zhang, Menjo, and Attardo, 2016b)
65
Figure 16.
The Smiling Intensity Scale as displayed by amy in face-to-face conversation
65
Figure 17.
Relationship between Duchenne display and humor in the conversation between Ann (left) and Kate (right)
68
Figure 18.
Duchenne display and smiling during non-humorous segment TJNH4: Tony SIS (light gray line), John SIS (dark gray line), and John Duchenne display (diamond marks)
69
Figure 19.
Duchenne display and smiling during humorous segment TJH7: Tony SIS (light gray line), John SIS (dark gray line), and John Duchenne display (diamond marks)
69
Figure 20.
Distribution of true and false positives and negatives in the identification of smiling voice
72
Figure 21.
Individual smiling intensity distribution with humor
74
Figure 22.
Individual smiling intensity distribution without humor (baseline)
75
Figure 23.
Andy facial display at SIS 0 (left) and SIS 1 (right) during the non-humorous segment AHNH1
76
Figure 24.
Andy facial display at SIS 2 (left) and SIS 3 (right) during the humorous segment AHH1
77
Figure 25.
Box-plot comparison of SIS with and without humor across language groups
78
Figure 26.
Distribution of smiling and non-smiling behaviors with and without humor across language groups
79
Figure 27.
Box-plot comparison of SIS with and without humor across gender groups
80
Figure 28.
Distribution of smiling and non-smiling behaviors with and without humor across gender groups
80
Figure 29.
Smiling intensity across types of humor
81
Figure 30.
Linear visualizations of smiling intensity (SIS on the vertical axis) during humorous segments per participant across conversational roles
83
Figure 31.
Examples of each joint smiling behavior category
95
Figure 32.
Example of a single framing smiling pattern (Ann and Kate, segment AKH1)
98
Figure 33.
Example of a joint framing smiling pattern (Emma and Jane, segment EJH2)
99
Figure 34.
Example of joint framing smiling pattern with an accommodation gesture (Hans and Andy, segment HAH5)
99
Figure 35.
Deadpan delivery of a punchline (Ann and Kate, segment AKH2)
100
Figure 36.
Deadpan delivery of an ironic utterance (Dani and Amy, segment DAH9)
101
Figure 37.
Deadpan delivery of a jabline (Tony and John, segment DAH7)
102
Figure 38.
Deadpan delivery of a jabline (Jake and Yoan, segment JYH3)
102
Figure 39.
Peak smiling pattern with an ironic utterance (John and Tony, segment JTH9)
103
Figure 40.
Peak smiling pattern with an ironic utterance (John and Tony, segment JTH10)
104
Figure 41.
Types of smiling synchronic behaviors displayed (in milliseconds) by dyads with and without humor
106
Figure 42.
Jake (dark gray line) contribution to JT Type 0 behavior with humor (Yoan and Jake, segment YJH1, punchline)
106
Figure 43.
Jake (dark gray line) contribution to JT Type 0 behavior with humor (Yoan and Jake, segment YJH6, irony)
107
Figure 44.
Smiling synchronicity for humorous and non-humorous segments (significant differences marked with *)
109
Figure 45.
Triangular scanning path of face processing
118
Figure 46.
Percentage of time participants looked at their interlocutor smiling facial areas (mouth and eyes together) during humorous and non-humorous segments of conversations
130
Figure 47.
Gaze (in msec) to the interlocutor’s mouth with and without humor
131
Figure 48.
Gaze (in msec) to the interlocutor’s eyes with and without humor
132
Figure 49.
Gaze to the interlocutor’s mouth with humor by conversational role
133
Figure 50.
No significant effect of interlocutor SIS on participants’ gaze to interlocutor’s smiling expression with irony
138
Figure 51.
Effect of interlocutor SIS on participants’ average fixation duration
140
Figure 52.
Misalignment during the delivery of a punchline (Ann, dark gray line, and Kate, light gray line, segment AKH2)
157
Figure 53.
Ann (left) and Kate (right) facial expression and fixations (gray dots) signaling failed humor understanding
157
Figure 54.
Kate’s scanning path on Ann’s mouth area after Ann delivered the punchline
158
Figure 55.
Smiling misalignment during a humorous utterance (Dani and Amy, segment DAH11)
159
Figure 56.
Amy and Dani facial expressions and gaze behaviors
161
Figure 57.
Emma (dark gray line) and Jane (light gray line) smiling behavior during Excerpt EJH2
163
Figure 58.
Jane facial expressions at frames 10, 15, and 30 after the delivery of the punchline
163
Figure 59.
Emma facial expressions at frames 10, 15, and 30 after the delivery of the punchline
163
Figure A1.
The Tobii X Configuration Tool for Calibration with a Scene Camera and a Physical Grid
211
Figure A2.
Fixations and saccades over a sample dynamic area of interest (Eyes) (from Gironzetti, 2020)
213
Figure B1.
Baseline and Humor Individual Smiling Intensity: Conversation 1, John and Tony
215
Figure B2.
Baseline and Humor Individual Smiling Intensity: Conversation 2, Yoan and Jake
216
Figure B3.
Baseline and Humor Individual Smiling Intensity: Conversation 3, Emma and Jane
216
Figure B4.
Baseline and Humor Individual Smiling Intensity: Conversation 4, Andy and Hand
217
Figure B5.
Baseline and Humor Individual Smiling Intensity: Conversation 5, Ann and Kate
217
Figure B6.
Baseline and Humor Individual Smiling Intensity: Conversation 6, Amy and Dani
222