Cyberpsychology, Behavior, and Social NetworkingVol. 22, No. 6
Abstract
This study explores the independent and joint effects of immersion and real-world distractions (a ringing cell phone) on cognitive (i.e., recognition and recall), affective valence, and interpersonal outcomes (i.e., interpersonal liking and communication satisfaction) as well as general feelings of presence (social presence and telepresence) during a virtual experience. Participants interacted with a virtual agent in an immersive virtual environment or nonimmersive virtual environment under three different levels of real-world distractions (i.e., no distraction, passively being exposed to the sound of a ringing cell phone, and actively responding to ringing cell phone). Increased immersion had a positive effect on telepresence, but a negative effect on recognition and recall; immersion did not have a significant effect on social presence. Real-world distractions had a negative effect on recognition, recall, and social presence, but did not affect telepresence or affective valence. Participants who were actively distracted performed more poorly on the recall measure and reported lower levels of social presence than their passively distracted counterparts. These findings suggest that (a) increased immersion will not uniformly improve social virtual reality experiences and (b) more research is needed on whether and how real-world events should be integrated into virtual environments.
Introduction
With the increasing availability of virtual reality (VR) technology, there has been great optimism regarding the potential of VR as an everyday communication tool.1 Multiple companies have launched social VR platforms (e.g., Facebook Spaces and VRChat), which allow users to connect with other people in immersive virtual environments (IVEs). Perhaps with the goal to make the experience as seamless as possible, there have also been efforts to enable media multitasking2 within VR. For example, one commercial VR headset (HTC Vive) allows users to receive and respond to notifications from their phone through Bluetooth technology.3
Despite the amount of resources going into developing social VR platforms, few research studies have explored how the introduction of immersive VR technology affects one’s communication experience (exceptions include4,5). In addition, considering that distractions from the physical world hinder one’s sense of presence (i.e., the subjective feeling of “being there”) in VR,6 it is unclear what, if any, the benefits of enabling media multitasking within VR would be. Despite the significance of this issue, only a handful of studies systematically explored the impact of real-world distractions on one’s VR experience.6,7 In addition, the majority of this research used passive distractions (e.g., overhearing the experimenter8), whereas media multitasking typically consists of active distractions9 where people actively engage in various streams of information (e.g., talking on the phone while responding to an e-mail). This study aims to bridge this gap in the literature by examining the separate and joint effects of a common real-world distraction—a ringing cell phone—and the degree of immersion on recognition and recall, affective valence, interpersonal liking, communication satisfaction, and general feelings of presence.
The aforementioned variables have been used in previous studies on interactions. For instance, studies on recall asked participants to report as much as they can remember about the conversation content (“free recall”), whereas recognition was assessed by whether participants could determine whether specific information was mentioned during an interaction.10As both are measures of cognitive processing, recall and recognition are referred to as “cognitive outcomes” from hereon. Liking and communication satisfaction are interpersonal evaluations of how socially desirable the interaction partner was and how enjoyable interaction itself was, respectively, as defined by previous research.11,12 These measures will be referred to as “interpersonal outcomes” from hereon. Affective valence can be assessed by the valence of words that participants use to describe an experience. For example, more positive words (e.g., happy, exciting, and stimulating) indicate that they felt more positively about the experience.13 Finally, telepresence describes the psychological state of feeling physically present in a virtual environment,14 and social presence refers to the sense of being together in the virtual environment with a real person15; we address both dimensions of presence, in line with extant research.16,17
Immersion and virtual interactions
Although immersion and presence are sometimes used interchangeably, immersion refers to the computer system’s technological capacity to deliver a vivid experience that removes the user from physical reality, and is measured by assessing the technical affordances provided by the system. In contrast, presence is a subjective psychological experience.14 Our study focused on two dimensions of presence, namely telepresence (the feeling of actually “being there”14) and social presence (the “sense of being with another”18). Although studies generally show a positive association between immersion and telepresence,19 research that explored the implications of increased immersion for social presence have yielded mixed results.20 In addition, although there is evidence of a positive association between subjective feelings of social presence and interpersonal liking,21 less is known about the relationship between immersion and interpersonal measures.
Virtual environments and breaks in presence
In IVEs, individuals are exposed to two sources of sensory data: one from the real world and another from the virtual environment.22 Breaks in presence (BIPs) occur when users start responding to the sensory data from the real world instead of that from the virtual world,22 such as the experimenter’s voice8 or technical failures.6 Given that BIPs negatively affect one’s sense of presence, one can predict that the real-world distraction of a ringing cell phone will also decrease feelings of telepresence and social presence. Furthermore, as BIPs are contingent on one’s engagement with the real-world stimuli,23 it is likely that people who actively respond to a distraction will experience lower levels of presence than those who are passively exposed to it. Moreover, experiencing BIPs may also induce negative affective responses due to the interrupted virtual experience. Past research demonstrating that interrupted media experiences lead to negative emotions24 lends support to this hypothesis.
Limited capacity model and media multitasking
The limited capacity model of motivated mediated message processing (LC4MP25) posits that individuals have a limited cognitive capacity, and that these resources are both automatically and constantly allocated to data encoding, storage, and retrieval during media use.26 When taking into account the highly vivid and sensory nature of IVEs,27 this model suggests that IVEs may negatively impact cognitive outcomes due to the additional cues that need to be attended to.
Similarly, real-world distractions during a virtual interaction may lead to poorer recall and recognition as cognitive resources will need to be allocated—subconsciously and/or consciously—to process external stimuli. Past research has shown that engaging in different media hurts cognitive performance.28 Some studies also suggest that chronic media multitasking may lead to worse social well-being and higher levels of depression.29,30 However, few studies31,32 have explored how engaging in different sensory information influences short-term interpersonal outcomes. Therefore, based on the literature on immersion, BIPs, media multitasking, and the LC4MP model, we systematically varied the levels of immersion and distraction to examine how these variables influence cognitive, affective, and interpersonal outcomes, as well as feelings of presence during a social virtual experience.
Method
Participants
A total of 129 participants (64 men, 65 women; age M = 21.05, SD = 2.37) were recruited from a medium-sized western university. Owing to technical issues (n = 8), manipulation check failure (n = 4), and withdrawal (n = 1), 13 participants were removed from the final analyses, leaving 116 participants. All procedures and materials were approved by the Ethical Committee of the Institutional Review Board at Stanford University.
Study procedure
The study adopted a two immersion (immersive and nonimmersive) × three real-world distraction (absent, passive, and active) between-subjects design. This design was used to avoid carryover effects across conditions and to prevent participant exhaustion.a Upon giving consent to participate in the experiment, participants were told that they would be performing a “Getting to Know You” (GTKY) task with a virtual human and that their goal was to get to know their partner as best as they could. The GTKY task was a semistructured interview wherein participants asked and answered questions adapted from previous research33 (see Appendix 1 for details). To maximize internal validity, answers given by the participant’s virtual partner (a computerized agent) were prerecorded and identical for all participants. To maximize external validity, we asked the questions used in the GTKY task to six undergraduate students before the actual study. The virtual partner’s responses were scripted based on these responses. All participants interacted with a virtual agent that matched their biological gender (Fig. 1).
FIG. 1. Representations of the virtual environment with female (left) and male (right) virtual agents.
After the experimenter described the task, participants in the “immersive” condition were shown how to wear the VR headset (Oculus Rift CV1). Once they put on the headset, participants were able to see their partner in a virtual room. There was a teleprompter on the wall behind the virtual agent that displayed instructions (Fig. 1). Participants in the “nonimmersive” condition carried out the same task, but the virtual world was displayed on a laptop computer (MSI GT73VR). Participants were given earphones so they could hear the virtual agent.
Participants in the two real-world distraction conditions (passive or active) heard a cell phone ringing for 20 seconds on three separate occasions during the conversation. The cell phone was placed inside a bag that was hung on the wall behind the participant. Thus, the cell phone was not visible to the participant. Participants in the “passive” distraction condition were not given any instructions regarding the cell phone. Participants in the “active” distraction condition were asked to keep their thumb on a game controller button (the “A” button on Oculus Touch) and press it each time they heard the phone, requiring active engagement. Participants who interacted with the virtual human in the “absent” distraction condition completed the study without any distractions. After the interaction, participants answered a postquestionnaire, which assessed telepresence, social presence, recognition, recall, affect, interpersonal liking, and communication satisfaction.b
Telepresence (three items, Cronbach’s α = 0.90), social presence (nine items, Cronbach’s α = 0.85), interpersonal liking (three items, Cronbach’s α = 0.80), and communication satisfaction (four items, Cronbach’s α = 0.89) were measured with a questionnaire using 7-point Likert-type scales (1 = strongly disagree, 7 = strongly agree). Recall was measured with an open-ended question that asked participants to spontaneously recall their partner’s answers to the seven questions they asked during the GTKY task. Two coders blind to experimental condition counted the number of correct responses (Cohen’s κ = 0.89, 95% CI [0.83–0.96]). Recognition was measured using a series of true or false questions, which asked whether or not the participant’s virtual partner made specific statements during the GTKY task. Responses were dummy coded (0 = incorrect, 1 = correct) and summed to form a composite recognition score. Affective valence was measured by conducting a linguistic inquiry word count (LIWC34) analysis on an open-ended question that asked participants to describe their communication experience. More specifically, we adopted Pennebaker’s differential emotion model35 and subtracted the percentage of negative emotion words from the percentage of positive emotion words. A full list of the questions used to measure these dependent variables is given in Appendix 2.
Three manipulation check items were administered at the end of the postquestionnaire. Participants were asked whether they heard any sounds coming from outside of the virtual environment during their conversation. We also asked participants (a) whether they heard a phone ring during the conversation and, for those who reported that they had heard a phone ring, (b) whether they heard the sound coming from within or outside of the virtual environment.
Statistical analysis
Our data met the assumptions of homogenous variance and independence of residuals, but violated the assumption of normally distributed residuals. Thus, we conducted a series of 2 × 3 aligned rank transform analyses of variance36 using the ARTool package in R. Although there is an ongoing debate regarding the robustness of analysis of variance in cases of normality assumption violations and Likert-type scales,37–39 we report the results of the nonparametric aligned rank transform analyses of variance, as it is the more conservative method.
To test the hypothesized difference between the no distraction condition and the two distraction conditions and the difference between the two distraction conditions, planned orthogonal contrasts40 were used. The orthogonal matrix that was used is included in Table 1.
| Distraction | Contrast 1 | Contrast 2 |
|---|---|---|
| None | 1.0 | 0 |
| Passive | −0.5 | 1 |
| Active | −0.5 | −1 |
Results
The participant distribution and condition-specific means and standard deviations of the dependent variables are given in Table 2. The correlations between the dependent variables are given in Table 3.
| Desktop | Desktop | Desktop | IVE | IVE | IVE | |
|---|---|---|---|---|---|---|
| distraction—none | distraction—passive | distraction—engage | distraction—none | distraction—passive | distraction—engage | |
| M (SD) | M (SD) | M (SD) | M (SD) | M (SD) | M (SD) | |
| Telepresence | 2.57 (1.25) | 2.67 (1.23) | 2.53 (1.37) | 4.68 (1.10) | 4.63 (1.41) | 4.16 (1.56) |
| Social presence | 4.29 (0.96) | 4.58 (0.84) | 3.93 (1.14) | 5.04 (0.91) | 4.44 (1.13) | 4.20 (1.15) |
| Recall | 5.70 (1.25) | 5.24 (1.58) | 4.18 (1.71) | 4.71 (1.41) | 4.42 (1.74) | 3.97 (1.42) |
| Recognition | 5.65 (0.75) | 5.74 (0.56) | 5.58 (0.61) | 5.58 (0.51) | 5.35 (0.81) | 5.26 (0.73) |
| Affective valence | 2.90 (3.27) | 2.01 (2.59) | .54 (5.82) | 2.14 (2.73) | 1.08 (1.77) | 1.01 (2.79) |
| Interpersonal liking | 4.97 (0.80) | 4.84 (1.40) | 4.46 (1.33) | 4.98 (1.03) | 4.83 (0.93) | 4.68 (1.08) |
| Communication satisfaction | 4.61 (1.15) | 4.89 (1.27) | 4.25 (1.72) | 4.67 (1.25) | 4.33 (1.37) | 4.17 (1.23) |
| No. of participants | 20 | 19 | 19 | 19 | 19 | 20 |
| Telepresence | Social presence | Recall | Recognition | Affective valence | Liking | Satisfaction | |
|---|---|---|---|---|---|---|---|
| Telepresence | — | 0.58* | −0.13 | −0.10 | 0.11 | 0.33* | 0.31** |
| Social presence | — | — | 0.29** | 0.07 | 0.24*** | 0.67* | 0.73* |
| Recall | — | — | — | 0.43* | −0.08 | 0.33* | 0.29** |
| Recognition | — | — | — | — | −0.08 | 0.12 | 0.01 |
| Affective valence | — | — | — | — | — | 0.23*** | 0.28** |
| Liking | — | — | — | — | — | — | 0.73* |
As expected, we found a significant effect of immersion on telepresence (F(1, 110) = 61.58, ηp2 = 0.36, p < 0.0001), such that participants in the immersive condition reported higher levels of telepresence than those in the nonimmersive condition (b = 19.83, p < 0.0001). Contrary to our hypothesis, distraction did not have a significant effect on telepresence (F(2, 110) = 0.76, p = 0.47). Inversely, although immersion did not significantly impact social presence (F(1, 110) = 1.64, p = 0.20), distraction significantly impacted social presence (F(2, 110) = 3.50, ηp2 = 0.06, p = 0.03). Interaction terms were nonsignificant for all outcome measures (all p‘s > 0.15). Planned orthogonal contrasts showed that participants who were distracted by a ringing cell phone reported lower levels of social presence than those who were not, although the results were only marginally significant (b = −8.21, p = 0.06). Furthermore, participants who actively engaged in the distraction felt marginally lower levels of social presence than their passively distracted counterparts (b = −6.71, p = 0.08) (Fig. 2).
FIG. 2. Effects of immersion on telepresence (top) and distraction on social presence (bottom). ***Positive at p < 0.001; †positive at p < 0.10. IVE, immersive virtual environment.
Immersion had a significantly negative effect on recall (F(1, 110) = 6.11, ηp2 = 0.05, p = 0.01) and a marginally significant effect on recognition (F(1, 110) = 3.50, ηp2 = 0.03, p = 0.06), such that participants in the immersive condition recalled and recognized less information about their virtual partner than their counterparts (b = −7.82, p = 0.01; b = −6.18, p = 0.02). As expected, distraction had a significant impact on recall (F(2, 110) = 6.12, ηp2 = 0.10, p = 0.003). Analyses showed that distracted participants recalled less information about their partner (b = −9.90, p = 0.02) and that participants in the active distraction condition recalled less information about their partner compared with those in the passive distraction condition (b = −9.06, p = 0.01). However, distraction did not significantly affect recognition (F(2, 110) = 1.23, p = 0.30) (Fig. 3).
FIG. 3. Effects of immersion and distraction on recall (top) and immersion on recognition (bottom). *Positive at p < 0.05; †positive at p < 0.10.
Neither distraction nor immersion had a significant effect on any of the interpersonal measures or affective valence (all p‘s > 0.14).
Discussion
This study is the first to systematically explore the relative effects of immersion and real-world distractions (i.e., ringing cell phone) on experiences within IVEs and non-IVEs. As predicted, enhanced immersion increased feelings of telepresence, but reduced recognition and recall. Given that high levels of telepresence are considered to be a desirable goal for IVEs,19 this negative impact of immersion on memory may seem counterintuitive. There are two potential explanations for the negative effect of immersion on memory. First, when adopting the LC4MP,25 the immersive qualities of the IVE may have drained the participant’s cognitive resources, limiting their ability to recall and recognize information. Bailey et al.27 similarly employed this framework to explain the negative correlation they found between telepresence and cued recall. Second, these results may be partially attributed to a novelty effect, such that participants in the IVE condition were more distracted by the features available in the virtual environment. For instance, McMahan et al.41 found that participants tended to exhibit better performance when the technology leveraged familiar experiences. Another exploratory study found that participants tended to show poorer retrieval memory for an augmented/virtual reality-based application than a trifold brochure.42
As expected, distractions from the real world had a negative impact on recall, and these effects were strongest when the individual was actively engaging in the distraction. These results point to the potentially harmful effects of real-world distractions during social virtual interactions. Given that this negative effect was not found for recognition, it is possible that the distractions exerted a stronger influence on the encoding, rather than retrieval of information. However, more research is needed before we can conclude that real-world distractions impact recall and recognition differently.
We also found that although immersion was a strong predictor of telepresence, it did not have a significant influence on social presence, which is consistent with previous research.20 Inversely, only social presence was influenced by distractions. Theoretically, our findings underscore that social presence is highly sensitive to contextual and individual boundary conditions, whereas immersion impacts telepresence in a more robust manner.19 Our results suggest that introducing additional immersive features without consideration for the context or user traits may not necessarily be beneficial in enhancing social presence.
Contrary to expectations, distraction did not have a significant effect on affective valence. As the distraction employed in our study (i.e., a ringing cell phone) is a relatively common occurrence, it is possible that participants were not greatly disturbed by the experience. However, considering that communicating while simultaneously engaging in a different task has been associated with feelings of frustration and/or dissatisfaction,42 it is possible that employing a distraction with a higher cognitive load may negatively influence affective valence. Future studies could benefit from exploring how different levels of cognitive load influence affective outcomes.
Surprisingly, we did not find any evidence supporting our hypotheses that immersion and distraction would influence interpersonal outcomes. Taken together, these results suggest that although immersion and distraction did impact the overall virtual experience (i.e., cognitive outcomes, telepresence, and social presence), participants did not misattribute their judgments toward the interaction experience to the target virtual human. This may be due to the personal nature of the GTKY task, which was designed to foster interpersonal closeness.33 As such, participants may have been more favorably inclined toward their virtual partner. Considering previous studies on how individuals tend to misattribute negative affect to others when the source of their discomfort is unclear,43 it is possible that immersion and distraction would have influenced virtual interactions if participants had interacted with a more neutral or hostile virtual partner. Future studies should try to replicate these findings using different types of tasks (e.g., goal-oriented task instead of GTKY) and/or having participants interact with virtual avatar rather than agent.
The goal of this study was to determine the effects of immersion and real-world distractions on presence, and cognitive, affective, as well as interpersonal outcomes. Contrary to current industry trends that favor the development of social VR systems with increased immersion and multitasking capacity, we found that although immersion does increase telepresence, it harms cognitive outcomes and does not significantly influence social presence, affective valence, or interpersonal outcomes. These results suggest that although VR systems can be helpful for experiences that benefit from perceptions of physically engaging with the virtual world (e.g., simulations, discussions in spaces where people can walk and move around freely), they may not outperform nonimmersive platforms in contexts wherein movement is less salient (e.g., one-on-one seated conversations), and can even have costs for memory.
Notes
| a. | The duration of the study was ∼30 minutes. As such, there was a risk that conducting the same study multiple times would lead to participant exhaustion. | ||||
| b. | We also included a behavioral measure to assess helping behavior. However, the majority of participants expressed skepticism regarding the validity of the task. Owing to deception failure and space constraints, these results are not reported. | ||||
Author Disclosure Statement
No competing financial interests exist.
Appendix 1. “Getting to Know You” Task
During the first half of the conversation, participants asked the virtual agent the following seven questions, which were displayed on a teleprompter in the virtual world. For the second half of the conversation, the virtual agent asked the same questions, and the participants responded to them.A1
| 1. | Before making a telephone call, do you ever rehearse what you are going to say, and why? | ||||
| 2. | For what in your life do you feel most grateful? | ||||
| 3. | If you could wake up tomorrow having gained one quality or ability, what would it be? | ||||
| 4. | What do you value most in a friendship? | ||||
| 5. | What is your most treasured memory? | ||||
| 6. | Given the choice of anyone in the world, whom would you want as a dinner guest? | ||||
| 7. | Your house, containing everything you own, catches on fire. After saving your loved ones and pets, you have time to safely make a final dash to save any one item. What would it be, and why? | ||||
Appendix 2. Full Wording for Questionnaire Measures
Presence Measures
Telepresence
How strongly do you agree or disagree with the following statements about the virtual environment? (1 = strongly disagree, 7 = strongly agree)A2,A3
| 1. | I felt like I was really there inside the virtual environment. | ||||
| 2. | I felt as if I could reach out and touch the objects or people in the virtual environment. | ||||
| 3. | The virtual environment seemed like the real world. | ||||
Social presence
How strongly do you agree or disagree with the following statements about your partner? (1 = strongly disagree, 7 = strongly agree)A4
| 1. | I felt like I was face-to-face with my partner. | ||||
| 2. | I felt like I was in the same room as my partner. | ||||
| 3. | I felt that my partner was watching me and aware of my presence. | ||||
| 4. | I remained focused on my partner throughout our interaction. | ||||
| 5. | My partner remained focused on me throughout our interaction. | ||||
| 6. | It was easy to understand my partner. | ||||
| 7. | My partner found it easy to understand me. | ||||
| 8. | I could tell how my partner felt. | ||||
| 9. | My partner could tell how I felt. | ||||
Interpersonal Outcome Measures
Liking
How strongly do you agree or disagree with the following statements about your partner? (1 = strongly disagree, 7 = strongly agree)A5
| 1. | I like my partner. | ||||
| 2. | I would like to get to know my partner better. | ||||
| 3. | I think my partner would be popular with his or her friends. | ||||
Communication satisfaction
How strongly do you agree or disagree with the following statements? (1 = strongly disagree, 7 = strongly agree)A6
| 1. | I would like to have another conversation like this one. | ||||
| 2. | I was satisfied with the conversation. | ||||
| 3. | I enjoyed the conversation. | ||||
| 4. | The conversation flowed smoothly. | ||||
Cognitive Outcome Measures
Recall (open-ended response)
You asked your partner seven questions. What were your partner’s answers to these questions? Please be as detailed as possible.
Recognition
Are the following statements about your partner true or false?
| 1. | My partner is most grateful for his/her friends. | ||||
| 2. | My partner sometimes rehearses what he/she is going to say before a telephone call. | ||||
| 3. | My partner is a great singer. | ||||
| 4. | My partner values loyalty the most in a friendship. | ||||
| 5. | My partner’s most treasured memory is New Year’s day with his/her grandmother. | ||||
| 6. | If my partner could save one item in a fire, it would be his/her laptop. | ||||
Affective Valence Measure
Please describe your experience talking with your partner. Focus on how you felt and what you thought during the conversation. Please be as detailed as possible (at least five sentences).A7
Appendix References
A1. Bailey J, Bailenson JN, Won AS, et al. (2012) Presence and memory: Immersive virtual reality effects on cued recall. In Proceedings of the 14th Annual International Workshop on Presence. Oct. 24–26; Philadelphia, PA: Temple University.
A2. Lombard M, Ditton TB, Weinstein L. (2009) Measuring presence: the temple presence inventory. In Proceedings of the 12th Annual International Workshop on Presence. 2009 Nov. 11–13; Los Angeles, CA: University of Southern California.
A3. Aymerich-Franch L, Karutz C, Bailenson JN. (2012) Effects of facial and voice similarity on presence in a public speaking virtual environment. In Proceedings of the 14th Annual International Workshop on Presence. Oct. 24–26; Philadelphia, PA: Temple University.
A4. Biocca F, Harms C, Gregg J. (2001) The networked minds measure of social presence: Pilot test of the factor structure and concurrent validity. In Proceedings of the 4th Annual International Workshop on Presence. May 21–23; Philadelphia, PA: Temple University.
A5. Rudman LA, Fairchild K. Reactions to counterstereotypic behavior: the role of backlash in cultural stereotype maintenance. Journal of Personality and Social Psychology 2004; 87:157–176.
A6. Hecht ML. The conceptualization and measurement of interpersonal communication satisfaction. Human Communication Research 1978; 4:253–264.
A7. Oh SY, Bailenson J, Krämer N, et al. Let the avatar brighten your smile: effects of enhancing facial expressions in virtual environments. PLoS One 2016; 11:e0161794.
Quelle:
https://www.liebertpub.com/doi/10.1089/cyber.2018.0404
