Spatial computing already brings us virtual home assistants and ride-hailing apps. It lets gamers summon ghouls into their living rooms, and shoppers try on clothes in digital changing rooms. Next, imagine working, shopping and socialising as avatars, in a rich,
three-dimensional digital world that overlays our own.
The idea of a 'metaverse', built with spatial computing, is no longer confined to science fiction.
Facebook has dedicated an entire division to its development, and has gone as far rebranding the company as Meta. Microsoft envisages what its CEO, Satya Nadella, calls an 'enterprise metaverse'.1 Jensen Huang, chief executive of Nvidia, an American chipmaker, wants to create “a virtual world that is a digital twin of ours.”2,3
Their vision of a visible alternate realm, always “on”, and always interacting with the real one, is still a long way from reality. Yet, fuelled by technologies such as AR, VR and mixed reality (MR), the spatial computing revolution is well under way.
What is spatial computing?
“Spatial computing integrates technologies like virtual reality and augmented reality with the real world, so we can move about and interact with both the virtual and the physical world at the same time,” explains Corinna Lathan, an inventor who started a US-based research and development company, AnthroTronix. “You can think of AR and VR as
technologies. Spatial computing is a way of being in the world.”
For years, clunky headsets, laggy connectivity and a lack of decent content held back the rise of those technologies. Advances in 5G mobile broadband and smartphones are changing that. “AR came out some time in the mid-noughties, and it didn’t really get anywhere because smartphones weren’t a thing. Then, suddenly, once smartphones came out, it all made sense,” says Alex Jenkins, creative director of the interactive arts
division of Nexus Studios, a film and creative technology studio based in London and LA.
Today’s mobiles are kitted out with cameras and GPS, which make it possible to blend the real and digital. Their processing power and capabilities are increasing. Newer iPhones are equipped with LiDAR (light detection and ranging), a remote sensing tool which Jenkins describes as “a big game-changer for spatial computing.”
It is in gaming that the two worlds feel closest to converging. US software developer Niantic harnessed spatial computing with the hugely popular “Pokémon Go” game in 2016. It is now building a worldwide AR platform for scalable user interactions. Elsewhere, gamers can project increasingly sophisticated content into their homes.
Take Illumix, an American AR entertainment, technology and gaming company, which has applied its technology to a popular horror game, “Five Nights at Freddy’s”, allowing gamers to populate their rooms with savage animatronics. “It’s different to any kind of horror game on PC or mobile,” explains Kirin Sinha, the company’s founder and chief
executive. “It gets closer to those physical, immersive, Halloween horror-type experiences, but it is adaptive to your space.”
For a taste of what the metaverse might become, analysts point to “Fortnite”, the brainchild of Epic Games.4 It started life as a video game but has evolved into something resembling a social universe, where users chat, brands plug their products, and
musicians perform live concerts to millions of viewers via digital avatars. In April, Epic Games announced a USD1 billion funding round to support its visions for the metaverse.5
Pieces of a metaverse
Beyond the tech space, brands are only just beginning to consider the possibilities of spatial computing.
Spurred on by the pandemic, labels including Macy’s and Adidas6 have started harnessing AR and VR to create virtual fitting rooms. Others have created 3D digital iterations of their stores. In 2019 the Dallas Cowboys, the world’s most valuable sports franchise, partnered with Nexus Studios to project giant avatars of their players across their stadium. Spectators could conjure them, as well as real-time performance stats, through 5G phones.
To those advancing the technologies, examples like that are proof of concept. “There's a lot of interest, a lot of exploration for what people could do with spatial computing, and I do see a shift from just random experimentation to wanting to get something more out of it,” says Jenkins. At Illumix, meanwhile, Sinha foresees “a democratisation of
what used to be very expensive, very specialised hardware-based technology”. She explains: “With spatial computing, you take that, and put it on every phone and every browser in the world.”
Healed by a hologram
There are applications beyond retail and entertainment. Spatial computing promises to make factories more efficient and improve worker productivity. It could help us “meet” with colleagues who are miles away. And it is bleeding into heavily regulated healthcare.
“That technology is now spilling into so many applications. In the next few years, it’s going to increase in an exponential way,” says Nassir Navab, who leads research on computer-aided medical procedures at both the Technical University of Munich and Johns Hopkins University.
Hospital researchers are experimenting with everything from remote surgeries to ways of projecting X-rays or scans directly on to patients. There are hopes that spatial computing could overhaul medical training too. Navab’s team is building a tool that allows consultants to generate virtual avatars, to guide colleagues remotely. The German government has asked them to hone the technology for the treatment of Covid-19
The “holy grail”, he says, would be to create a 3D digital duplicate of an operating room and everyone in it, giving a perspective even from inside the patient’s body. “With virtual and augmented reality, you don’t need to go physically into the operating room; you can put on a head-mounted display and see five surgeries in a night. And the best
surgeon can expose his work to thousands of others,” Navab explains.
If spatial computing is already enhancing our lives, that is just the beginning. “What we’re going to start to see with the metaverse is a growth of the vision and ubiquity of applications. Even prior to everyone wearing contact lenses or glasses and looking at the world around them, digital and physical being will become truly intertwined in our daily lives, and all through existing platforms,” says Sinha. As technologies advance and 5G brings faster connectivity, real and virtual will inch closer than ever before.