Article published In:
Interaction StudiesVol. 23:1 (2022) ► pp.116–142
Interacting with an embodied interface
Effects of embodied agent and voice command on smart TV interface
Despite their potential for facilitating interaction between a user and computer, an embodied agent and voice command have not been examined enough for their matching effects. The current study proposes that an embodied agent and voice command generate positive evaluative outcomes, particularly when they are accompanied by each other. To test this prediction, we conducted a 2 (visual output: embodied agent vs. geometric figure) × 2 (input modality: voice command vs. remote controller) between-subjects experiment (N = 52), and examined whether visual output and input modality jointly influence participants’ social attribution (i.e., anthropomorphism, animacy, likability, and perceived intelligence), social presence, and satisfaction. Results show that voice command does facilitate users’ social attribution and social presence, but only when an embodied agent was presented. Also, the effects of voice command on social presence and satisfaction were mediated by anthropomorphism and perceived intelligence respectively, but only when the interface displayed an embodied agent. The present study evidences the holistic nature of human-computer interaction, revealing the importance of matches in the input and output interface.
Keywords: smart TV, embodied agent, voice control, social attribution, social presence, satisfaction
Article outline
- 1.Introduction
- 2.Literature review
- 2.1Effects of embodied agents on a computer interface
- 2.2Input modality of a smart TV
- 2.3Contingent effect of visual output and input modality
- 3.Method
- 3.1Design and participants
- 3.2Procedure
- 3.3Material
- 3.4Manipulation
- 3.5Measures
- Attribution of social characters
- Social presence
- Satisfaction
- 4.Results
- 4.1Effects on social attribution
- 4.2Effects on social presence and satisfaction
- 4.3Conditional indirect effect of input modality
- 5.Discussion
- 5.1Summary of findings
- 5.2Implications
- 5.3Limitations and directions for further research
- Disclosure statement
-
References
References
Allison, F., Luger, E., & Hofmann, K.
(2017) Spontaneous
Interactions with a Virtually Embodied Intelligent Assistant in Minecraft. Proceedings of the
2017 CHI Conference Extended Abstracts on Human Factors in Computing
Systems, 2337–2344. 
Bartneck, C., Kulić, D., Croft, E., & Zoghbi, S.
(2009) Measurement instruments for the anthropomorphism, animacy, likeability, perceived intelligence, and perceived safety of robots. International Journal of Social Robotics, 1(1), 71–81.
Baylor, A. L., & Ryu, J.
(2003) The
effects of image and animation in enhancing pedagogical agent persona. Journal of Educational
Computing
Research,
28
1, 373–394.
Brave, S., Nass, C., & Hutchinson, K.
(2005) Computers
that care: Investigating the effects of orientation of emotion exhibited by an embodied computer
agent. International Journal of Human-Computer
Studies,
62
1, 161–178.
Brennan, S. E., & Hulteen, E. A.
(1995) Interaction
and feedback in a spoken language system: A theoretical framework. Knowledge-Based
Systems,
8
1, 143–151.
Clark, H. H.
(1999) How
do real people communicate with virtual partners. Proceedings of 1999 AAAI Fall Symposium,
Psychological Models of Communication in Collaborative
Systems, 43–47. North Falmouth, MA.
Cohen, P. R., & Oviatt, S. L.
(1995) The
role of voice input for human-machine communication. Proceedings of the National Academy of
Sciences,
92
1, 9921–9927. Retrieved
from [URL].
Cordasco, G., Esposito, M., Masucci, F., Riviello, M. T., Esposito, A., Chollet, G., … Pelosi, G.
(2014) Assessing
voice user interfaces: The vAssist system prototype. 5th IEEE Conference on Cognitive
Infocommunications, 91–96.
Dehn, D. M., & Van Mulken, S.
(2000) The
impact of animated interface agents: A review of empirical research. International Journal of
Human-Computer
Studies,
52
1, 1–22.
Dworkin, M., Chakraborty, A., Lee, S., Monahan, C., Hightow-Weidman, L., Garofalo, R., … Jimenez, A.
(2018) A
realistic talking human embodied agent mobile phone intervention to promote hiv medication adherence and retention in care in
young HIV-positive african american men who have sex with men: Qualitative study. JMIR MHealth
and UHealth,
6
1, e10211.
Fogg, B., & Nass, C.
(1997) How
users reciprocate to computers: An experiment that demonstrates behavior change. Proceedings of
the CHI ’97 Extended Abstracts on Human Factors in Computing
Systems, 331–332.
Hasegawa, D., Cassell, J., & Araki, K.
(2010) The
role of embodiment and perspective in direction-giving systems. 2010 AAAI Fall
Symposium. Retrieved from [URL]
Hayes, A. F.
(2013) Introduction
to mediation, moderation, and conditional process analysis: A regression-based approach. New York, NY, US: Guilford Press.
Hong, W., Hess, T. J., & Hardin, A.
(2013) When
filling the wait makes it feel longer: A paradigm shift perspective for managing online
delay. MIS
Quarterly,
37
1, 383–406. Retrieved
from [URL].
Hutchins, E. L., Hollan, J. D., & Norman, D. A.
(1985) Direct
manipulation interfaces. Human–Computer
Interaction,
1
1, 311–338.
Jeong, J.-W., & Lee, D.-H.
(2014) Inferring
search intents from remote control movement patterns: A new content search method for smart
TV. IEEE Transactions on Consumer
Electronics,
60
1, 92–98.
Kätsyri, J., Förger, K., Mäkäräinen, M., & Takala, T.
(2015) A
review of empirical evidence on different uncanny valley hypotheses: Support for perceptual mismatch as one road to the valley
of eeriness. Frontiers in
Psychology,
6
1, 390.
Kim, Yanghee, Thayne, J., & Wei, Q.
(2017) An
embodied agent helps anxious students in mathematics learning. Educational Technology Research
and
Development,
65
1, 219–235.
Kim, Youjeong, & Sundar, S. S.
(2012) Anthropomorphism
of computers: Is it mindful or mindless? Computers in Human
Behavior,
28
1, 241–250.
Koda, T., & Maes, P.
(1996) Agents
with faces: The effect of personification. Proceedings of 5th International Workshop on Robot
and Human Communication, 189–194.
Lee, B., Isenberg, P., Riche, N. H., & Carpendale, S.
(2012) Beyond
Mouse and Keyboard: Expanding Design Considerations for Information Visualization
Interactions. IEEE Transactions on Visualization and Computer
Graphics,
18
1, 2689–2698.
Lee, D., Sah, Y. J., & Lee, S.
(2019) Improving
usability perception of error-prone ai speakers: Elaborated feedback mitigates negative consequences of
errors. International Journal of Human–Computer
Interaction,
35
1, 1645–1652.
Lee, K. M.
(2004) Presence,
explicated. Communication
Theory,
14
(1), 27–50.
Lee, K. M., & Nass, C.
(2005) Social-psychological
origins of feelings of presence: Creating social presence with machine-generated voices. Media
Psychology,
7
1, 31–45.
Lee, K. M., Peng, W., Jin, S.-A., & Yan, C.
(2006) Can
robots manifest personality?: An empirical test of personality recognition, social responses, and social presence in
human–robot interaction. Journal of
Communication,
56
1, 754–772.
Mitchell, W. J., Szerszen, K. A., Lu, A. S., Schermerhorn, P. W., Scheutz, M., & MacDorman, K. F.
(2011) A
mismatch in the human realism of face and voice produces an uncanny
valley. I-Perception,
2
1, 10–12.
Mori, M.
(1970) The
uncanny
valley. Energy,
7
(4), 33–35.
Nass, C., & Gong, L.
(2000) Speech
interfaces from an evolutionary perspective. Communications of the
ACM,
43
(9), 36–43.
Nass, C., & Moon, Y.
(2000) Machines
and mindlessness: Social responses to computers. Journal of Social
Issues,
56
1, 81–103.
Nass, C., Moon, Y., & Carney, P.
(1999) Are
people polite to computers? Responses to computer-based interviewing systems. Journal of
Applied Social
Psychology,
29
1, 1093–1109.
Nass, C., Moon, Y., & Green, N.
(1997) Are
machines gender neutral? Gender-stereotypic responses to computers with voices. Journal of
Applied Social
Psychology,
27
1, 864–876.
Nass, C., Steuer, J., & Tauber, E. R.
(1994) Computers
are social actors. Proceedings of the SIGCHI Conference on Human Factors in Computing
Systems, 72–78.
Pyae, A., & Joelsson, T. N.
(2018) Investigating
the usability and user experiences of voice user interface: A case of google home smart
speaker. Proceedings of the 20th International Conference on Human-Computer Interaction with
Mobile Devices and Services Adjunct, 127–131.
Rhee, E., Shin, I., Jung, Y., & Lee, H.
(2013) Cloud-based
3D gaming system for smart TV. Proceedings of the 2013 International Conference on ICT
Convergence (ICTC), 714–715.
Rosenberg-Kima, R. B., Baylor, A. L., Plant, E. A., & Doerr, C. E.
(2007) The
importance of interface agent visual presence: Voice alone is less effective in impacting young women’s attitudes toward
engineering. In Y. de Kort, W. IJsselsteijn, C. Midden, B. Eggen, & B. J. Fogg (Eds.), Persuasive
Technology (pp. 214–222). Berlin: Springer.
Sah, Y. J., & Peng, W.
(2015) Effects
of visual and linguistic anthropomorphic cues on social perception, self-awareness, and information disclosure in a health
website. Computers in Human
Behavior,
45
1, 392–401.
Seyama, J., & Nagayama, R. S.
(2007) The
uncanny valley: Effect of realism on the impression of artificial human faces. Presence:
Teleoperators and Virtual
Environments,
16
1, 337–351.
Shamekhi, A., Liao, Q. V., Wang, D., Bellamy, R. K. E., & Erickson, T.
(2018) Face
value? Exploring the effects of embodiment for a group facilitation agent. Proceedings of the
2018 CHI Conference on Human Factors in Computing
Systems, 1–13.
Silvia, P. J., & Duval, T. S.
(2001) Objective
self-awareness theory: Recent progress and enduring problems. Personality and Social Psychology
Review,
5
1, 230–241.
Sproull, L., Subramani, M., Kiesler, S., Walker, J. H., & Waters, K.
(1996) When
the interface is a face. Human–Computer
Interaction,
11
1, 97–124.
Stafford, R. Q., MacDonald, B. A., Jayawardena, C., Wegner, D. M., & Broadbent, E.
(2014) Does
the robot have a mind? Mind perception and attitudes towards robots predict use of an eldercare
robot. International Journal of Social
Robotics,
6
1, 17–32.
Walther, J. B.
(1996) Computer-mediated
communication: Impersonal, interpersonal, and hyperpersonal interaction. Communication
Research,
23
1, 3–43.
(2011) Visual
cues in computer-mediated communication: Sometimes less is
more. In A. Kappas & N. C. Krämer (Eds.), Face-to-Face
Communication over the
Internet (pp. 17–38). Cambridge, UK: Cambridge University Press.
Walther, J. B., Slovacek, C. L., & Tidwell, L. C.
(2001) Is
a picture worth a thousand words? Photographic images in long-term and short-term computer-mediated
communication. Communication
Research,
28
(1), 105–134.
Yee, N., Bailenson, J. N., & Rickertsen, K.
(2007) A
meta-analysis of the impact of the inclusion and realism of human-like faces on user experiences in
interfaces. Proceedings of the SIGCHI Conference on Human Factors in Computing
Systems, 1–10.
Yi, J. S., Kang, Y. a, & Stasko, J.
(2007) Toward
a Deeper Understanding of the Role of Interaction in Information Visualization. IEEE
Transactions on Visualization and Computer
Graphics,
13
1, 1224–1231.
Yu, E., Hong, A., & Hwang, J.
(2016) A
socio-technical analysis of factors affecting the adoption of smart TV in Korea. Computers in
Human
Behavior,
61
1, 89–102.