Article published In:
GestureVol. 19:2/3 (2020) ► pp.196–222
Social hyperscanning with fNIRS
Intra-brain and inter-brain connectivity for social, affective, and informative gestures reproduction
The interest of neuroscience has been aimed at the investigation of the neural bases underlying gestural communication. This research explored the intra- and inter-brain connectivity between encoder and decoder. Specifically, adopting a “hyperscanning paradigm” with the functional Near-infrared Spectroscopy (fNIRS) cerebral connectivity in oxygenated (O2Hb) and deoxygenated (HHb) hemoglobin levels were revealed during the reproduction of affective, social, and informative gestures of different valence. Results showed an increase of intra- and inter-brain connectivity in dorsolateral prefrontal cortex for affective gestures, in superior frontal gyrus for social gestures and in frontal eyes field for informative gestures. Moreover, encoder showed a higher intra-brain connectivity in posterior parietal areas more than decoder. Finally, an increasing of inter-brain connectivity more than intra-brain (ConIndex) was observed in left regions for positive gestures. The present research has explored how the individuals neural tuning mechanisms turn out to be strongly influenced by the nature of specific gestures.
Keywords: inter-brain connectivity, encoder, decoder, fNIRS, hyperscanning, gestures
Article outline
- Introduction
- Materials and methods
- Subjects
- Procedure
- Stimuli
- fNIRS recording and signal processing
- Data analysis
- Results
- Intra-brain connectivity
- Inter-brain connectivity
- ConIndex
- Discussion
- Conflict of interest
- Data availability statement
-
References
References
Alibali, Martha W., Kita, Sotaro, & Young, Amanda J.
(2000) Gesture and the process of speech production: We think, therefore we gesture. Language and Cognitive Processes, 15 (6), 593–613. 
Baeken, Chris, Van Schuerbeek, Peter, De Raedt, Rudi, De Mey, Johan, Vanderhasselt, Marie A., Bossuyt, Axel, & Luypaert, Robert
(2011) The effect of one left-sided dorsolateral prefrontal sham-controlled HF-rTMS session on approach and withdrawal related emotional neuronal processes. Clinical Neurophysiology, 122 (11), 2217–2226.
Baker, Joseph M., Liu, Ning, Cui, Xu, Vrticka, Pascal, Saggar, Manish, Hosseini, Hadi S. M., & Reiss, Allan L.
(2016) Sex differences in neural and behavioral signatures of cooperation revealed by fNIRS hyperscanning. Scientific Reports, 61, 26492.
Balconi, Michela
(2010) Neuropsychology of communication. Springer Science & Business Media.
Balconi, Michela & Canavesio, Ylenia
(2013) High-frequency rTMS improves facial mimicry and detection responses in an empathic emotional task. Neuroscience, 2361, 12–20.
Balconi, Michela & Fronda, Giulia
(2020a) The use of hyperscanning to investigate the role of social, affective, and informative gestures in non-verbal communication. Electrophysiological (EEG) and inter-brain connectivity evidence. Brain Sciences, 10 (1), 29.
(2020b) Gesture in hyperscanning during observation. Inter-brain connectivity. Neuropsychological Trends, 281, 59–81.
(2021) Intra-brain connectivity vs. inter-brain connectivity in gestures reproduction: What relationship?. Brain Sciences, 11 (5), 577.
Balconi, Michela, Fronda, Giulia, & Bartolo, Angela
(2020) Affective, social, and informative gestures reproduction in human interaction: Hyperscanning and brain connectivity. Journal of Motor Behavior, 53 (3), 296–315.
Balconi, Michela, Grippa, Elisabetta, & Vanutelli, Maria Elide
(2015) Resting lateralized activity predicts the cortical response and appraisal of emotions: an fNIRS study. Social Cognitive and Affective Neuroscience, 10 (12), 1607–1614.
Balconi, Michela & Vanutelli, Maria Elide
(2017) Cooperation and competition with hyperscanning methods: review and future application to emotion domain. Frontiers in Computational Neuroscience, 111, 86.
Bhat, Anjana N., Hoffman, Michael D., Trost, Susanna L., Culotta, McKenzie L., Eilbott, Jeffrey, Tsuzuki, Daisuke, & Pelphrey, Kevin A.
(2017) Cortical activation during action observation, action execution, and interpersonal synchrony in adults: a functional near-infrared spectroscopy (fNIRS) study. Frontiers in Human Neuroscience, 111, 431.
Bianchi, Marta, Cosseddu, Maura, Cotelli, Maria, Manenti, Rosa, Brambilla, Michela, Rizzetti, Cristina M., […], & Borroni, Barbara
(2015) Left parietal cortex transcranial direct current stimulation enhances gesture processing in corticobasal syndrome. European Journal of Neurology, 22 (9), 1317–1322.
Bressem, Jana & Müller, Cornelia
(2017) The “Negative-Assessment-Construction” – A multimodal pattern based on a recurrent gesture? Linguist Vanguard, 3 (s1).
Cassell, Justine, Sullivan, Joseph, Prevost, Scott, & Churchill, Elizabeth F.
(2001) Embodied dialog systems. Cambridge, MA: MIT Press.
Centelles, Laurie, Assaiante, Christine, Nazarian, Bruno, Anton, Jean L., & Schmitz, Christina
(2011) Recruitment of both the mirror and the mentalizing networks when observing social interactions depicted by point-lights: A neuroimaging study. PLoS One, 61: e15749.
Chaudhary, Ujwal, Hall, Michael, DeCerce, Joe, Rey, Gustavo, & Godavarty, Anuradha
(2011) Frontal activation and connectivity using near-infrared spectroscopy: Verbal fluency language study. Brain Research Bulletin, 84 (3), 197–205.
Cui, Xu, Bryant, Daniel M., & Reiss, Allan L.
(2012) NIRS-based hyperscanning reveals increased interpersonal coherence in superior frontal cortex during cooperation. Neuroimage, 59 (3), 2430–2437.
Davidson, Richard J.
(1992) Emotion and affective style: Hemispheric substrates. Psychological Science, 3 (1), 39–43.
Enfield, Nick J., Kita, Sotaro, & de Ruiter, Jan P.
(2007) Primary and secondary pragmatic functions of pointing gestures. Journal of Pragmatics, 39 (10), 1722–1741.
Friston, Karl J.
(2011) Functional and effective connectivity: a review. Brain Connectivity, 1 (1), 13–36.
Fronda, Giulia & Balconi, Michela
(2020) The effect of interbrain synchronization in gesture observation: A fNIRS study. Brain and Behavior, e01663.
Gallagher, Helen L. & Frith, Christopher D.
(2004) Dissociable neural pathways for the perception and recognition of expressive and instrumental gestures. Neuropsychologia, 42 (13), 1725–1736.
Gentilucci, Maurizio, Bernardis, Paolo, Crisi, Girolamo, & Volta, Riccardo D.
(2006) Repetitive transcranial magnetic stimulation of Broca’s area affects verbal responses to gesture observation. Journal of Cognitive Neuroscience, 18 (7), 1059–1074.
Harmon-Jones, Eddie
(2003) Clarifying the emotive functions of asymmetrical frontal cortical activity. Psychophysiology, 40 (6), 838–848.
Hasson, Uri, Ghazanfar, Asif A., Galantucci, Bruno, Garrod, Simon, & Keysers, Christian
(2012) Brain-to-brain coupling: A mechanism for creating and sharing a social world. Trends in Cognitive Sciences, 16 (2), 114–121.
Holle, Henning, Gunter, Thomas C., Rüschemeyer, Shirley-Ann, Hennenlotter, Andreas, & Iacoboni, Marco
(2008) Neural correlates of the processing of co-speech gestures. Neuroimage, 39 (4), 2010–2024.
Holle, Henning, Obleser, Jonas, Rueschemeyer, Shirley-Ann, & Gunter, Thomas C.
(2010) Integration of iconic gestures and speech in left superior temporal areas boosts speech comprehension under adverse listening conditions. Neuroimage, 49 (1), 875–884.
Holler, Judith, Shovelton, Heather, & Beattie, Geoffrey
(2009) Do iconic hand gestures really contribute to the communication of semantic information in a face-to-face context?. Journal of Nonverbal Behavior, 33 (2), 73–88.
Holper, Lisa, Scholkmann, Felix, & Wolf, Martin
(2014) The relationship between sympathetic nervous activity and cerebral hemodynamics and oxygenation: a study using skin conductance measurement and functional near-infrared spectroscopy. Behavioural Brain Research, 2701, 95–107.
Honk, Jack V. & Schutter, Dennis J. L. G.
(2006) From affective valence to motivational direction: the frontal asymmetry of emotion revised. Psychological Science, 17 (11), 963–965.
Huxham, Chris, Dick, Bob, & Stringer, Ernie
(2009) Theory in action research. Action Research, 71, 5–12.
Johnson, David W. & Johnson, Roger T.
(2005) New developments in social interdependence theory. Genetic, Social, and General Psychology Monographs, 131 (4), 285–358.
Kalbe, Elke, Schlegel, Marius, Sack, Alexander T., Nowak, Dennis A., Dafotakis, Manuel, Bangard, Christopher, […], & Kessler, Josef
(2010) Dissociating cognitive from affective theory of mind: a TMS study. Cortex, 46 (6), 769–780.
Kendon, Adam
(2017) Reflections on the “gesture-first” hypothesis of language origins. Psychonomic Bulletin & Review, 24 (1), 163–170.
Kita, Sotaro
(2000) How representational gestures help speaking. In David McNeill. (Ed.), Language and gesture (pp. 162–185). Cambridge, UK: Cambridge University Press.
(2009) Cross-cultural variation of speech-accompanying gesture: A review. Language and Cognitive Processes, 24 (2), 145–167.
Knoblich, Gunther, Butterfill, Stephen, & Sebanz, Natalie
(2011) Psychological research on joint action: Theory and data. In B. Ross. (Ed.), The psychology of learning and motivation (Vol. 541, pp. 59–101). Burlington, MA: Academic Press.
Koehler, Saskia, Egetemeir, Johanna, Stenneken, Prisca, Koch, Stefan P., Pauli, Paul, Fallgatter, Andreas J., & Herrmann, Martin J.
(2012) The human execution/observation matching system investigated with a complex everyday task: a functional near-infrared spectroscopy (fNIRS) study. Neuroscience Letters, 508 (2), 73–77.
Krauss, Robert M., Chen, Yihsiu, & Chawla, Purnima
(1996) Nonverbal behavior and nonverbal communication: What do conversational hand gestures tell us?. In M. P. Zanna. (Ed.), Advances in experimental social psychology (Vol. 281, pp. 389–450). Academic Press.
Kron, Assaf, Pilkiw, Maryna, Banaei, Jasmin, Goldstein, Ariel, & Anderson, Adam K.
(2015) Are valence and arousal separable in emotional experience? Emotion, 15 (1), 35–44.
Lindenberger, Ulman, Li, Shu C., Gruber, Walter, & Müller, Viktor
(2009) Brains swinging in concert: cortical phase synchronization while playing guitar. BMC Neuroscience, 10 (1), 22.
Lindquist, Kristen A., Satpute, Ajay B., & Gendron, Maria
(2015) Does language do more than communicate emotion? Current Directions in Psychological Science, 24 (2), 99–108.
Liu, Tao, Saito, Hirofumi, & Oi, Misato
(2015) Role of the right inferior frontal gyrus in turn-based cooperation and competition: a near-infrared spectroscopy study. Brain and Cognition, 991, 17–23.
Lotze, Martin, Heymans, Ulrike, Birbaumer, Niels, Veit, Ralf, Erb, Michael, Flor, Herta, & Halsband, Ulrike
(2006) Differential cerebral activation during observation of expressive gestures and motor acts. Neuropsychologia, 44 (10), 1787–1795.
Marcos, Luis R.
(1979) Nonverbal behavior and thought processing. Archives of General Psychiatry, 36 (9), 940–943.
Matsuda, Goh & Hiraki, Kazuo
(2006) Sustained decrease in oxygenated hemoglobin during video games in the dorsal prefrontal cortex: A NIRS study of children. Neuroimage, 29 (3), 706–711.
Molenberghs, Pascal, Cunnington, Ross, & Mattingley, Jason B.
(2009) Is the mirror neuron system involved in imitation? A short review and meta-analysis. Neuroscience & Biobehavioral Reviews, 33 (7), 975–980.
Molenberghs, Pascla, Cunnington, Ross, & Mattingley, Jason B.
(2012) Brain regions with mirror properties: a meta-analysis of 125 human fMRI studies. Neuroscience & Biobehavioral Reviews, 36 (1), 341–349.
Montague, Read P., Berns, Gregory S., Cohen, Jonathan D., McClure, Samuel M., Pagnoni, Giuseppe, Dhamala, Mukesh, […], & Fisher, Ronald E.
(2002) Hyperscanning: Simultaneous fMRI during linked social interactions. Neuroimage, 16 (4), 1159–1164.
Montero-Hernandez, Samuel, Orihuela-Espina, Felipe, Sucar, Luis E., Pinti, Paola, Hamilton, Antonia, Burgess, Paul, & Tachtsidis, Ilias
(2018) Estimating functional connectivity symmetry between oxy-and deoxy-haemoglobin: implications for fNIRS connectivity analysis. Algorithms, 11 (5), 70.
Morrel-Samuels, Palmer & Krauss, Robert M.
(1992) Word familiarity predicts temporal asynchrony of hand gestures and speech. Journal of Experimental Psychology: Learning Memory and Cognition, 18 (3), 615–622.
Müller, Viktor & Lindenberger, Ulman
(2011) Cardiac and respiratory patterns synchronize between persons during choir singing. PLoS One, 61: e24893.
Nagpal, Jaya, Nicoladis, Elena, & Marentette, Paula
(2011) Predicting individual differences in L2 speakers’ gestures. International Journal of Bilingualism, 15 (2), 205–214.
Nicoladis, Elena, Pika, Simone, Yin, Hui U. I., & Marentette, Paula
(2007) Gesture use in story recall by Chinese-English bilinguals. Applied Psycholinguistics, 28 (4), 721–735.
Oi, Misato, Saito, Hirofumi, Li, Zongfeng, & Zhao, Wenjun
(2013) Co-speech gesture production in an animation–narration task by bilinguals: A near-infrared spectroscopy study. Brain and Language, 125 (1), 77–81.
Pelphrey, Kevin A., Morris, James P., & Mccarthy, Gregory
(2004) Grasping the intentions of others: the perceived intentionality of an action influences activity in the superior temporal sul- cus during social perception. Journal of Cognitive Neuroscience, 16 (10), 1706–1716.
Péran, Patrice, Démonet, Jean F., Cherubini, Andrea, Carbebat, Dominique, Caltagirone, Carlo, & Sabatini, Umberto
(2010) Mental representations of action: the neural correlates of the verbal and motor components. Brain Research, 13281, 89–103.
Pinti, Paola, Aichelburg, Clarisse, Lind, Frida, Power, Sarah, Swingler, Elizabeth, Merla, Arcangelo, […], & Tachtsidis, Ilias
(2015) Using fiberless, wear- able fNIRS to monitor brain activity in real-world cognitive tasks. Journal of Visualized Experiments, 106, e53336.
Pinti, Paola, Scholkmann, Felix, Hamilton, Antonia, Burgess, Paul, & Tachtsidis, Ilias
(2019) Current status and issues regarding pre-processing of fNIRS neuroimaging data: An investigation of diverse signal filtering methods within a general linear model framework. Frontiers in Human Neuroscience, 121, 505.
Rauscher, Frances H., Krauss, Robert M., & Chen, Yihsiu S.
(1996) Gesture, speech, and lexical access: The role of lexical movements in speech production. Psychological Science, 7 (4), 226–231.
Ruby, Perrine & Decety, Jean
(2004) How would you feel versus how do you think she would feel? A neuroimaging study of perspective-taking with social emotions. Journal of Cognitive Neuroscience, 16 (6), 988–999.
Sadato, Norihiro, Campbell, Gregory, Ibanez, Vicente, Deiber, Marie P., & Hallett, Mark
(1996) Complexity affects regional cerebral blood flow change during sequential finger movements. Journal of Neuroscience, 16 (8), 2693–2700.
Sato, Wataru, Kochiyama, Takanori, Uono, Shota, & Yoshikawa, Sakiko
(2009) Commonalities in the neural mechanisms underlying auto- matic attentional shifts by gaze, gestures, and symbols. Neuroimage, 45 (3), 984–992.
Schmidt, Louis A. & Trainor, Laurel J.
(2001) Frontal brain electrical activity (EEG) distinguishes valence and intensity of musical emotions. Cognition & Emotion, 15 (4), 48–500.
Scholkmann, Felix, Kleiser, Stefan, Metz, Andreas J., Zimmermann, Raphael, Pavia, Juan M., Wolf, Ursula, & Wolf, Martin
(2014) A review on continuous wave functional near-infrared spectroscopy and imaging instrumentation and methodology. Neuroimage, 851, 6–27.
Seo, Ju-Hwan, Yang, Jeong-Yean, & Kwon, Dong-Soo
(2015) Learning and reproduction of valence-related communicative gesture. In 2015 IEEE-RAS 15th International Conference on Humanoid Robots (Humanoids) (pp. 237–242).
Shafir, Tal, Tsachor, Rachelle P., & Welch, Kathleen B.
(2016) Emotion regulation through movement: unique sets of movement characteristics are associated with and enhance basic emotions. Frontiers in Psychology, 61, 2030.
Shima, Keisetsu & Tanji, Jun
(2000) Neuronal activity in the supplementary and presupplementary motor areas for temporal organization of multiple movements. Journal of Neurophysiology, 84 (4), 2148–2160.
Skipper, Jeremy I., Goldin-Meadow, Susan, Nusbaum, Howard C., & Small, Steven L.
(2007) Speech-associated gestures, Broca’s area, and the human mirror system. Brain and Language, 101 (3), 260–277.
(2009) Gestures orchestrate brain networks for language understanding. Current Biology, 19 (8), 661–667.
Szymański, Jędrzej, Janikiewicz, Justyna, Michalska, Bernadeta, Patalas-Krawczyk, Paulina, Perrone, Mariasole, Ziółkowski, Wieslaw, […], & Więckowski, Mariusz R.
(2017) Interaction of mitochondria with the endoplasmic reticulum and plasma membrane in calcium homeostasis, lipid trafficking and mitochondrial structure. International Journal of Molecular Sciences, 18 (7), 1576.
Tomasello, Michael, Carpenter, Malinda, Call, Josep, Behne, Tanya, & Moll, Henrike
(2005) Understanding and sharing intentions: The origins of cultural cognition. Behavioral and Brain Sciences, 28 (5), 675–691.
Toppi, Jlenia, Borghini, Gianluca, Petti, Manuela, He, Eric J., De Giusti, Vittorio, He, Bin, […], & Babiloni, Fabio
(2016) Investigating cooperative behavior in ecological settings: An EEG hyperscanning study. PLoS One, 111, e0154236.
Tsalamlal, Mohamed Y., Amorim, Michel A., Martin, Jean C., & Ammi, Mehdi
(2016) Combining facial expression and touch for perceiving emotional valence. IEEE Transactions on Affective Computing, 9 (4), 437–449.
Walker, Robin, Techawachirakul, Puncharat, & Haggard, Patrick
(2009) Frontal eye field stimulation modulates the balance of salience between target and distractors. Brain Research, 12701, 54–63.
Wang, Feng, Xu, Jie, Wang, Xin, & Cui, Shuguang
(2017) Joint offloading and computing optimization in wireless powered mobile-edge computing systems. IEEE Transactions on Wireless Communications, 17 (3), 1784–1797.
Wheaton, Kylie J., Thompson, James C., Syngeniotis, Ari, Abbott, David F., & Puce, Aina
(2004) Viewing the motion of human body parts activates different regions of premotor, temporal, and parietal cortex. Neuroimage, 22 (1), 277–288.
Wheland, David, Joshi, Anand, McMahon, Katie L., Hansell, Narelle, Martin, Nicholas, Wright, Margaret, […], & Leahy, Richard
(2012) Robust identification of partial-correlation based networks with applications to cortical thickness data. In 2012 9th IEEE International Symposium on Biomedical Imaging (ISBI) (pp. 1551–1554).
Wildgruber, Dirk, Hertrich, Ingo, Riecker, Axel, Erb, Michael, Anders, Simon, Grodd, Walfgang, & Ackermann, Hermann
(2004) Distinct frontal regions subserve evaluation of linguistic and emotional aspects of speech intonation. Cerebral Cortex, 14(12), 1384–1389.
Willems, Roel M., Özyürek, Aslı, & Hagoort, Peter
(2007) When language meets action: The neural integration of gesture and speech. Cerebral Cortex, 17 (10), 2322–2333.
Zhao, Xiaohu, Xi, Qian, Wang, Peijun, Li, Chunbo, & He, Hongjian
(2014) Altered activity and functional connectivity of superior temporal gyri in anxiety disorders: A functional magnetic resonance imaging study. Korean Journal of Radiology, 15 (4), 523–529.