The Future Of Zoom: Meet The Founders Of Spatial In VR

The same tech that powers Pokémon Go might make virtual meetings more like real life.

Now six months into the pandemic, it’s not unusual to have a work meeting, a doctor’s appointment, and a happy hour without leaving your desk. And our new Zoom-centric lifestyle isn’t going away anytime soon. With cold weather around the corner, you can count on spending more hours in video chats and a lot less time seeing people in real life. A small startup called Spatial thinks this is an opportunity to transform the way we interact in digital spaces.

Spatial’s co-founders are incredibly excited about the future of augmented reality. You may have encountered AR, which is a technology that superimposes digital images onto the real world, during the Pokémon Go craze four years ago. But instead of making it look like Pikachu is in your living room, Spatial makes it look like your coworkers are there — or at least realistic avatars of them are. Spatial also works with virtual reality headsets like the Oculus Quest, which put you in completely immersive digital environments, but the company’s co-founders seem particularly bullish about AR and a future in which we’ll all wear lightweight glasses that blur the line between the real world and a computer interface.

This sort of thing isn’t just some sci-fi fantasy anymore. The pandemic is showing us how, if the technology becomes more accessible, AR and VR experiences can fill some of the vacuum of human connection that remote work has created. Spatial is providing a glimpse at how useful this technology can be: It can make working and learning remotely feel more like being in an office or a classroom, rather than just being a box in a Zoom call grid. But to get there, the tech needs better connectivity — namely 5G, which is rolling out across the United States and other parts of the world, and could vastly expand what we can do with AR, VR, and countless other technologies.

“I think this pandemic has accelerated not only the pace of development but also has opened up brand new or expanded business opportunities for new applications or need for the existing applications to be recognized,” Babak Beheshti, dean of the College of Engineering and Computing Sciences at the New York Institute of Technology, told me. Within five years, he added, he believes that technology like lightweight AR headsets — let’s call them smart glasses — will replace the smartphone for many people.

A world where people interact through headsets is something that’s been anticipated and even feared for decades. While VR technology has fully arrived, it’s mainly been adopted by gamers. Meanwhile, AR technology seems to be stuck in arrested development. This has limited the possibilities of mixed reality, which combines elements of VR and AR and anchors virtual objects to the real world so you can interact with them in new ways. Here’s an example of what a mixed reality might look like through smart glasses:

But ahead of that dystopian future, AR is the next frontier when it comes to new ways of combining our digital and physical worlds. While Apple has been investing in adding AR capabilities to iPhones, and Pokémon Go turned AR into a fad four years ago, the technology hasn’t had its killer app. Spatial wants to make it.

Spatial’s co-founders say they want to build the Google Docs of augmented reality. The concept is wonderfully simple: Spatial provides a virtual space where groups of people can collaborate on projects. (While I tried Spatial on an Oculus Quest VR headset, the app also works on Microsoft’s HoloLens and the Magic Leap One, which are the two leading AR headsets. There are iOS and Android versions in beta.) When you fire up the Spatial app, you create an avatar by uploading a single photo of your face, and then machine learning turns it into a digital, 3D version of you. The avatars represent you from the waist up, so you can move your arms and make gestures, thanks to hand-tracking technology in the Oculus headset. Spatial also uses AI to animate the avatar’s faces based on what you’re saying.

Once you’ve created an avatar, you can join rooms, kind of like you would in Zoom, and you can interact with the avatars of your friends or coworkers. You can talk to them and see versions of their real faces, as well as move around the room, watch videos together, and give high-fives.

I recently spent a couple of hours in Spatial’s virtual workspace, interacting with 3D models and talking to the avatars of Anand Agarawala and Jinha Lee, the Spatial co-founders. This very 21st-century experience was what I needed in the early days of the pandemic: an escape into a world that felt more real than a Zoom call but less whimsical than a video game.

The experience did feel awkward at first, but the feeling faded after a couple of minutes. Although I was sitting alone in my apartment wearing a VR headset, I found myself having fun talking to these slightly uncanny avatars in a pixelated realm. As Jacob Loewenstein, the head of business at Spatial, explained how companies like Mattel and Ford are using Spatial to design new products, I found myself marveling at the simple pleasure of looking around the room at the different people in it, everything virtual but not cartoonish. It’s been so long since I’ve had a meeting in real life that something I never would have imagined has happened: I’ve started pining for IRL meetings.

At one point, someone pulled a 3D model of a new Home Depot store layout into the room, and we got to look at it from all angles, as if we were Home Depot executives deciding how best to display the power tools in a store. The Google Docs analogy made some sense for what we were doing, but frankly, I want to hang out with my friends in Spatial. It would be a heck of a lot better than another Zoom happy hour.

Agarawala and Lee later explained that this was just one version of their vision for Spatial. As cool as it is to have virtual meetings on a virtual reality headset, they mostly talked about what their technology can do in augmented reality. They raved about the HoloLens version of their software, which overlays digital objects onto the real world — a more primitive version of the holographic future they look forward to. They described how their tech can host meetings where the avatar makes a client look like they’re sitting in a chair in their living room, and thanks to 3D audio, it sounds like they’re sitting right there too. I didn’t see this demo, but I can see how that client would find this story enticing.

“Most computers we use, whether it’s a laptop or a phone, you kind of turn your back to the world when you use them, because you’re lurched over this thing,” Agarawala told me. “AR is inherently collaborative. It’s like, ‘Hey, the whole room is our monitor, and we’re working on it together.’”

For the whole world to become our computer screen is an alluring concept. Smart glasses and 5G connectivity stand to make our user experiences with this tech even more immersive and realistic. These innovations are closer than you might think, too. Computers are increasingly passive, intuitive, and tiny. What once filled a room now fits on your wrist and knows when your heart beats. A host of companies, including Apple, Google, Microsoft, and Facebook, are working on a new class of wearables that put a screen right in front of your eyes, connect to the internet, and use a lidar sensor to map the world around you in real time.

Apple is the one to watch here. The company reportedly has 1,000 engineers working on lightweight smart glasses that it hopes will someday be as ubiquitous as the iPhone. But we’ve heard these sorts of ambitions before. Google made a lot of noise in 2013 when it released Glass, a camera-equipped wearable that places a small transparent display in front of your eye. A startup called North struggled to release some basic smart glasses last year, and then got bought by Google.

For now, the $3,500 Microsoft HoloLens 2, which resembles the visor on a fighter pilot helmet and overlays digital images with the real world, is the closest thing we’ve got to a portable AR headset. However, it is not for the average person. The HoloLens is designed for industrial applications — for example, so a worker can see instructions for how to build a car while they’re on the factory floor. This would explain why Spatial can count Ford as one of its clients.

Still, experiencing Spatial on a $400 Oculus Quest headset brought me slightly closer to the concept of hanging out with holographic friends while wearing smart glasses. What’s still missing, aside from the actual glasses, is fast-enough internet connectivity to make this kind of tech work. 5G networks would change that.

Beheshti, who is also a senior member of IEEE (the Institute of Electrical and Electronics Engineers), explained to me that 5G technology is key for augmented reality technology to take off because it offers high-bandwidth connections with super-low latency. Whereas 4G LTE speed tops out at about 50 megabits per second, 5G can offer 2 gigabits per second. While older wireless technology can’t really get below 20 milliseconds of latency, 5G promises 1 millisecond. This means that interacting with 3D avatars can be ultra-high resolution, free of glitches or jitters. As you probably know too well, existing networks sometimes struggle to handle 2D Zoom calls.

These features will enable a more lifelike AR experience, but they will also help miniaturize hardware, like smart glasses. Better connectivity means devices can depend on the cloud for the heavy processing power needed to render 3D images. So instead of wearing an AR device that feels almost as bulky as a helmet — this is how I felt when wearing a HoloLens for the first time — future headsets will be as light as eyeglasses. A gadget like this would be truly revolutionary.

“It will be your communications device; you will make your phone calls or video calls,” said Beheshti. “It will have the technology embedded in it to allow for these AR experiences pretty much on a normal basis.”

This all sounds exciting — and expensive. The introduction of any new technology also stands to carry with it unanticipated consequences. Smartphones introduced all kinds of anxieties over how much time we spend looking at screens, and there’s every reason to believe that a technology as immersive as augmented reality will have psychological effects as well. Creating new digital spaces also allows for more exclusion. Think of how the digital divide has led to vastly different experiences during the pandemic, as many with internet access are enjoying the benefits of remote work and learning while those without this access, or the option to go remote, are struggling to keep up.

Then there are concerns around what the technology itself can do. If you assume that smart glasses will be equipped with cameras and other sensors, it’s likely that they’ll introduce myriad privacy concerns. Todd Richmond, who is an IEEE member and director of the Tech & Narrative Lab at the Pardee RAND Graduate School for Public Policy, suggested to me that advanced facial recognition technology might work with smart glasses, so users could feasibly start scanning the faces of passersby and accessing details about their identities in real time. So while it might seem remarkable to have a meeting full of holograms, the same technology that powers those experiences could have potentially harmful applications elsewhere.

“We’re in a time where the world is struggling, and we have to be coming up with technological solutions that are equitable and sustainable,” said Richmond. “And that’s a hard thing to do, because it’s hard enough just to make the technology work.”