Virtual reality
by Chris Woodford. Last updated: August 29, 2021.
You'll probably never go to Mars, swim with dolphins, run an
Olympic 100 meters, or sing onstage with the Rolling Stones. But if
virtual reality ever lives up to its promise, you might be able to do
all these things—and many more—without even leaving your home.
Unlike real reality (the actual world in which we live),
virtual reality means simulating bits of our world (or completely
imaginary worlds) using high-performance computers and sensory
equipment, like headsets and gloves. Apart from games and
entertainment, it's long been used for training airline pilots and
surgeons and for helping scientists to figure out complex problems
such as the structure of protein molecules. How does it work? Let's take a
closer look!
Photo: Virtual pilot. This US Air Force student is learning to fly a giant C-17 Globemaster plane using a virtual reality simulator. Picture by Trenton Jancze courtesy of US Air Force.
What is virtual reality?
Virtual reality (VR) means experiencing things through our
computers that don't really exist. From that simple definition, the
idea doesn't sound especially new. When you look at an amazing
Canaletto painting, for example, you're experiencing the sites and
sounds of Italy as it was about 250 years ago—so that's a kind of
virtual reality. In the same way, if you listen to ambient
instrumental or classical music with your eyes closed, and start
dreaming about things, isn't that an example of virtual reality—an
experience of a world that doesn't really exist? What about losing
yourself in a book or a movie? Surely that's a kind of virtual
reality?
If we're going to understand why books, movies, paintings, and
pieces of music aren't the same thing as virtual reality, we need to
define VR fairly clearly. For the purposes of this simple, introductory article,
I'm going to define it as:
A believable, interactive 3D computer-created world that you
can explore so you feel you really are there, both mentally and physically.
Putting it another way, virtual reality is essentially:
- Believable: You really need to feel like you're in your virtual world (on Mars, or wherever) and to
keep believing that, or the illusion of virtual reality will
disappear.
- Interactive: As you move around, the VR world needs to move
with you. You can watch a 3D movie and be transported up to the Moon
or down to the seabed—but it's not interactive in any sense.
- Computer-generated: Why is that important? Because only powerful
machines, with realistic 3D computer graphics, are fast enough to make
believable, interactive, alternative worlds that change in real-time as we move around them.
- Explorable: A VR world needs to be big and detailed enough for
you to explore. However realistic a painting is, it shows
only one scene, from one perspective. A book can describe a vast and complex
"virtual world," but you can only really explore it in a linear
way, exactly as the author describes it.
- Immersive: To be both believable and interactive, VR
needs to engage both your body and your mind. Paintings by war
artists can give us glimpses of conflict, but they can never fully
convey the sight, sound, smell, taste, and feel of battle. You can
play a flight simulator game on your home PC and be lost in a very
realistic, interactive experience for hours (the landscape will
constantly change as your plane flies through it), but it's not like
using a real flight simulator (where you sit in a hydraulically
operated mockup of a real cockpit and feel actual forces as it tips
and tilts), and even less like flying a plane.
Artwork: This Canaletto painting of Venice, Italy is believable and in some sense explorable (you can move your eyes around and think about different parts of the picture), but it's not interactive, computer-generated, or immersive, so it doesn't meet our definition of virtual reality: looking at this picture is not like being there. There's nothing to stop us making an explorable equivalent in VR, but we need CGI—not oil paints—to do it. Picture courtesy of Wikimedia Commons.
We can see from this why reading a book, looking at a painting,
listening to a classical symphony, or watching a movie don't qualify
as virtual reality. All of them offer partial glimpses of
another reality, but none are interactive, explorable, or fully
believable. If you're sitting in a movie theater looking at a giant
picture of Mars on the screen, and you suddenly turn your head too
far, you'll see and remember that you're actually on Earth and the
illusion will disappear. If you see something interesting on the
screen, you can't reach out and touch it or walk towards it; again,
the illusion will simply disappear. So these forms of entertainment
are essentially passive: however plausible they might be, they
don't actively engage you in any way.
VR is quite different. It makes you think you are actually living
inside a completely believable virtual world (one in which, to use
the technical jargon, you are partly or fully immersed). It is
two-way interactive: as you respond to what you see, what you see
responds to you: if you turn your head around, what you see or hear
in VR changes to match your new perspective.
Types of virtual reality
"Virtual reality" has often been used as a marketing buzzword
for compelling, interactive video games or even 3D movies and
television programs, none of which really count as VR because they don't immerse
you either fully or partially in a virtual world. Search for "virtual
reality" in your cellphone app store and you'll find hundreds of
hits, even though a tiny cellphone screen could never get anywhere
near producing the convincing experience of VR. Nevertheless, things
like interactive games and computer simulations would certainly meet
parts of our definition up above, so there's clearly more than
one approach to building virtual worlds—and more than one flavor of
virtual reality. Here are a few of the bigger variations:
Fully immersive
For the complete VR experience, we need three things. First, a
plausible, and richly detailed virtual world to explore; a computer model
or simulation, in other words. Second, a powerful computer that
can detect what we're going and adjust our experience accordingly, in
real time (so what we see or hear changes as fast as we move—just
like in real reality). Third, hardware linked to the computer that
fully immerses us in the virtual world as we roam around. Usually,
we'd need to put on what's called a head-mounted display (HMD) with
two screens and stereo sound, and wear one or more sensory gloves.
Alternatively, we could move around inside a room, fitted out with
surround-sound loudspeakers, onto which changing images are projected
from outside. We'll explore VR equipment in more detail in a moment.
Non-immersive
A highly realistic flight simulator on a home PC might qualify as
nonimmersive virtual reality, especially if it uses a very wide
screen, with headphones or surround sound, and a realistic joystick
and other controls. Not everyone wants or needs to be fully immersed
in an alternative reality. An architect might build a detailed 3D
model of a new building to show to clients that can be explored on a
desktop computer by moving a mouse. Most people would classify that
as a kind of virtual reality, even if it doesn't fully immerse you.
In the same way, computer archaeologists often create engaging 3D
reconstructions of long-lost settlements that you can move around and
explore. They don't take you back hundreds or thousands of years or
create the sounds, smells, and tastes of prehistory, but they give a
much richer experience than a few pastel drawings or even an animated
movie.
Collaborative
What about "virtual world" games like Second Life and Minecraft? Do they
count as virtual reality? Although they meet the first four of our
criteria (believable, interactive, computer-created and explorable),
they don't really meet the fifth: they don't fully immerse you. But
one thing they do offer that cutting-edge VR typically doesn't is
collaboration: the idea of sharing an experience in a virtual
world with other people, often in real time or something very close
to it. Collaboration and sharing are likely to become increasingly
important features of VR in future.
Web-based
Virtual reality was one of the hottest, fastest-growing
technologies in the late 1980s and early 1990s, but the rapid rise of
the World Wide Web largely killed off interest after that. Even
though computer scientists developed a way of building virtual worlds
on the Web (using a technology analogous to HTML called Virtual
Reality Markup Language, VRML), ordinary people were much more
interested in the way the Web gave them new ways to access real
reality—new ways to find and publish information, shop, and share
thoughts, ideas, and experiences with friends through social media.
With Facebook's growing interest in the technology, the future of VR
seems likely to be both Web-based and collaborative.
Augmented reality
Mobile devices like smartphones and tablets have put what used to
be supercomputer power in our hands and pockets.
If we're wandering round the world, maybe visiting a heritage site like the pyramids or a fascinating
foreign city we've never been to before, what we want is typically
not virtual reality but an enhanced experience of the exciting
reality we can see in front of us. That's spawned the idea of
augmented reality (AR), where,
for example, you point your smartphone at a
landmark or a striking building and interesting information about it
pops up automatically. Augmented reality is all about connecting the
real world we experience to the vast virtual world of information
that we've collectively created on the Web. Neither of these worlds
is virtual, but the idea of exploring and navigating the two
simultaneously does, nevertheless, have things in common with virtual
reality. For example, how can a mobile device figure out its precise
location in the world? How do the things you see on the screen of
your tablet change as you wander round a city? Technically, these
problems are similar to the ones developers of VR systems have to
solve—so there are close links between AR and VR.
Photo: Augmented reality: A heads-up display, like this one used by the US Air Force,
superimposes useful, computer-based information on top of the things you see with your own eyes.
Picture by Major Chad E. Gibson courtesy of US Air Force.
What equipment do we need for virtual reality?
Photo: Virtual reality means blocking yourself off from the real world and substituting a computer-generated alternative. Often, it involves wearing a wraparound headset called a head-mounted display, clamping stereo headphones over your ears, and touching or feeling your way around your imaginary home using datagloves (gloves with built-in sensors). Picture by Wade Sisler courtesy of NASA Ames Research Center.
Close your eyes and think of virtual reality and you probably
picture something like our top photo: a geek wearing a wraparound
headset (HMD) and datagloves, wired into a powerful workstation or
supercomputer. What differentiates VR from an ordinary computer
experience (using your PC to write an essay or play games) is the
nature of the input and output. Where an ordinary computer uses
things like a keyboard,
mouse, or (more exotically)
speech recognition for input, VR uses sensors that detect how your body is
moving. And where a PC displays output on a screen (or a printer), VR
uses two screens (one for each eye), stereo or surround-sound
speakers, and maybe some forms of haptic (touch and body perception)
feedback as well. Let's take a quick tour through some of the more
common VR input and output devices.
Head-mounted displays (HMDs)
Photo: The view from inside. A typical HMD has two tiny screens
that show different pictures to each of your eyes, so your brain produces a combined
3D (stereoscopic) image. Picture by courtesy of US Air Force.
There are two big differences between VR and looking at an
ordinary computer screen: in VR, you see a 3D image that changes
smoothly, in real-time, as you move your head. That's made possible
by wearing a head-mounted display, which looks like a giant motorbike
helmet or welding visor, but consists of two small screens (one in
front of each eye), a blackout blindfold that blocks out all other
light (eliminating distractions from the real world), and stereo
headphones. The two screens display slightly different, stereoscopic
images, creating a realistic 3D perspective of the virtual world.
HMDs usually also have built-in
accelerometers or position sensors
so they can detect exactly how your head and body are moving (both
position and orientation—which way they're tilting or pointing) and
adjust the picture accordingly. The trouble with HMDs is that they're
quite heavy, so they can be tiring to wear for long
periods; some of the really heavy ones are even mounted on stands
with counterweights. But HMDs don't have to be so elaborate
and sophisticated: at the opposite end of the spectrum, Google
has developed an affordable, low-cost pair of cardboard goggles
with built-in lenses that convert an ordinary smartphone into a crude HMD.
Immersive rooms
An alternative to putting on an HMD is to sit or stand inside a
room onto whose walls changing images are projected from outside. As you
move in the room, the images change accordingly. Flight simulators
use this technique, often with images of landscapes, cities, and
airport approaches projected onto large screens positioned just
outside a mockup of a cockpit. A famous 1990s VR experiment called
CAVE (Cave Automatic Virtual Environment), developed at the
University of Illinois by Thomas de Fanti, also worked this way.
People moved around inside a large cube-shaped room with
semi-transparent walls onto which stereo images were back-projected
from outside. Although they didn't have to wear HMDs, they did need
stereo glasses to experience full 3D perception.
Datagloves
Photos: EXOS datagloves produced by NASA in the 1990s had very intricate external sensors to detect finger movements with high precision. Picture courtesy of NASA Marshall Space Flight Center (NASA-MSFC).
Right/below: This more elaborate EXOS glove had separate sensors on each finger segment, wired up to a single ribbon
cable connected up to the main VR computer. Picture by Wade Sisler courtesy of NASA Ames Research Center.
See something amazing and your natural instinct is to reach out
and touch it—even babies do that. So giving people the ability to
handle virtual objects has always been a big part of VR. Usually,
this is done using datagloves, which are ordinary gloves with sensors
wired to the outside to detect hand and figure motions. One technical
method of doing this uses fiber-optic cables stretched the length of
each finger. Each cable has tiny cuts in it so, as you flex your
fingers back and forth, more or less light escapes. A photocell at
the end of the cable measures how much light reaches it and the
computer uses this to figure out exactly what your fingers are doing.
Other gloves use strain gauges, piezoelectric sensors, or
electromechanical devices (such as potentiometers) to measure finger
movements.
Photo: This more elaborate EXOS glove had separate sensors on each finger segment, wired up to a single ribbon cable connected up to the main VR computer. Picture by Wade Sisler courtesy of NASA Ames Research Center.
Artwork: How a fiber-optic dataglove works. Each finger has a fiber-optic cable stretched along its length. (1) At one end of the finger, a light-emitting diode (LED) shines light into the cable. (2) Light rays shoot down the cable, bouncing off the sides. (3) There are tiny abrasions in the top of each fiber through which some of the rays escape. The more you flex your fingers, the more light escapes. (4) The amount of light arriving at a photocell at the end gives a rough indication of how much you're flexing your finger. (5) A cable carries this signal off to the VR computer. This is a simplified version of the kind of dataglove VPL patented in 1992, and you'll find the idea described in much more detail in US Patent 5,097,252.
Wands
Even simpler than a dataglove, a wand is a stick you can use to
touch, point to, or otherwise interact with a virtual world.
It has position or motion sensors (such as accelerometers)
built in, along with mouse-like buttons or scroll wheels. Originally,
wands were clumsily wired into the main VR computer; increasingly,
they're wireless.
Photo: A typical handheld virtual reality controller (complete with elastic bands), looking not so different from a video game controller. Photo courtesy of NASA Ames Research Center.
Applications of virtual reality
VR has always suffered from the perception that it's little more
than a glorified arcade game—literally a "dreamy escape" from
reality. In that sense, "virtual reality" can be an unhelpful
misnomer; "alternative reality," "artificial reality," or
"computer simulation" might be better terms. The
key thing to remember about VR is that it really isn't a fad or
fantasy waiting in the wings to whistle people off to alternative
worlds; it's a hard-edged practical technology that's been routinely
used by scientists, doctors, dentists, engineers, architects,
archaeologists, and the military for about the last 30 years. What
sorts of things can we do with it?
Education
Difficult and dangerous jobs are hard to train for. How can you
safely practice taking a trip to space, landing a jumbo jet, making a
parachute jump, or carrying out brain surgery? All these things are
obvious candidates for virtual reality applications. As we've seen
already, flight cockpit simulators were among the earliest VR
applications; they can trace their history back to mechanical
simulators developed by Edwin Link in the 1920s.
Just like pilots, surgeons are now routinely trained using VR. In a
2008 study of
735 surgical trainees from 28 different countries, 68 percent said
the opportunity to train with VR was "good" or "excellent"
for them and only 2 percent rated it useless or unsuitable.
Photo: Flight training is a classic application of virtual reality, though it doesn't use HMDs or datagloves. Instead, you sit in a pretend cockpit with changing images projected onto giant screens to give an impression of the view you'd see from your plane. The cockpit is a meticulous replica of the one in a real airplane with exactly the same instruments and controls. Photo by Javier Garcia courtesy of
US Air Force.
Scientific visualization
Anything that happens at the atomic or molecular scale is
effectively invisible unless you're prepared to sit with your eyes
glued to an electron microscope. But suppose you want to design new
materials or drugs and you want to experiment with the molecular
equivalent of LEGO. That's another obvious application for virtual
reality. Instead of wrestling with numbers, equations, or
two-dimensional drawings of molecular structures, you can snap
complex molecules together right before your eyes. This kind of work
began in the 1960s at the University of North Carolina at Chapel
Hill, where Frederick Brooks launched
GROPE, a project to develop a VR system for exploring the interactions between protein molecules
and drugs.
Photo: If you're heading to Mars, a trip in virtual reality could help you visualize what you'll find when you get there. Picture courtesy of NASA Ames Research Center.
Medicine
Apart from its use in things like surgical training and drug design,
virtual reality also makes possible telemedicine (monitoring,
examining, or operating on patients remotely). A logical extension of
this has a surgeon in one location hooked up to a virtual reality
control panel and a robot in another location (maybe an entire
continent away) wielding the knife. The best-known
example of this is the daVinci surgical robot, released in 2009, of
which several thousand have now been installed in hospitals
worldwide. Introduce collaboration and there's the possibility of a
whole group of the world's best surgeons working together on a
particularly difficult operation—a kind of WikiSurgery, if you
like!
Although it's still early days, VR has already been tested as a treatment
for various kinds of psychiatric disorder (such as
schizophrenia,
agoraphobia,
and phantom-limb pain),
and in rehabilitation for
stroke patients and
those suffering degenerative diseases such as
multiple sclerosis.
Industrial design and architecture
Architects used to build models out of card and paper; now they're
much more likely to build virtual reality computer models you can
walk through and explore. By the same token, it's generally much
cheaper to design cars, airplanes, and other complex, expensive
vehicles on a computer screen than to model them in
wood, plastic, or
other real-world materials. This is an area where virtual reality
overlaps with computer modeling: instead of simply making an
immersive 3D visual model for people to inspect and explore, you're
creating a mathematical model that can be tested for its aerodynamic,
safety, or other qualities.
Games and entertainment
From flight simulators to race-car games, VR has long hovered on
the edges of the gaming world—never quite good enough to
revolutionize the experience of gamers, largely due to computers
being too slow, displays lacking full 3D, and the lack of decent HMDs
and datagloves. All that may be about to change with the development
of affordable new peripherals like the Oculus Rift.
Photo: The US military continues to experiment with virtual reality as a battle-training aid. Here are some cutting-edge VR HaptX datagloves under test at Dyess Air Force Base, Texas. Picture by Colin Hollowell courtesy of
US Air Force.
Pros and cons of virtual reality
Like any technology, virtual reality has both good and bad points.
How many of us would rather have a complex brain operation carried
out by a surgeon trained in VR, compared to someone who has merely
read books or watched over the shoulders of their peers? How many of
us would rather practice our driving on a car simulator before we set
foot on the road? Or sit back and relax in a Jumbo Jet, confident in
the knowledge that our pilot practiced landing at this very airport,
dozens of times, in a VR simulator before she ever set foot in a real
cockpit?
Critics always raise the risk that people may be seduced by
alternative realities to the point of neglecting their real-world
lives—but that criticism has been leveled at everything from radio
and TV to computer games and the Internet. And, at some point, it
becomes a philosophical and ethical question: What is real anyway?
And who is to say which is the better way to pass your time? Like
many technologies, VR takes little or nothing away from the real
world: you don't have to use it if you don't want to.
The promise of VR has loomed large over the world of computing for
at least the last quarter century—but remains unfulfilled.
While science, architecture, medicine, and the military all rely on
VR technology in different ways, mainstream adoption remains
virtually nonexistent; we're not routinely using VR the way we use
computers, smartphones, or the Internet. The 2014 acquisition of
VR company Oculus, by Facebook, greatly renewed interest in the area
and, for a little while, looked like it would change everything.
Facebook's basic idea is to let people share things with their friends using the Internet and the Web. What
if you could share not simply a photo or a link to a Web article but
an entire experience? Instead of sharing photos of your wedding with
your Facebook friends, what if you could make it possible for people
to attend your wedding remotely, in virtual reality, in perpetuity?
What if we could record historical events in such a way that people
could experience them again and again, forever more? These are the
sorts of social, collaborative virtual reality sharing that (we might
guess) Facebook is thinking about exploring right now.
If that does eventually happen, the future of virtual reality looks very bright indeed.
Unfortunately, the huge wave of enthusiasm that arrived with Oculus has largely
dissipated and disappeared once again. Google and the BBC both pulled the plug on major
VR projects in 2019, for example. And while it sounds impressive that Sony has sold several million VR headsets,
if you bear in mind that cellphones have sold by the billion, it's clear that the
technology is nowhere near as pervasive or transformative as it sounds.
For the time being, it's going to remain a niche interest, albeit one with
increasingly imaginative applications in science, architecture, medicine,
and elsewhere.
A brief history of virtual reality
So much for the future, but what of the past. Virtual reality has
a long and very rich history. Here are a few of the more interesting
highlights...
Artwork: The first virtual reality machine? Morton Heilig's 1962 Sensorama. Picture courtesy US Patent and Trademark Office.
- 1890s: Thomas Edison and his assistant William Dickson pioneer the
Kinetograph (a camera for recording pictures) and Kinetoscope (a
projector for playing them back)—in effect, the first one-person
"movie experience."
- 1895: French brothers Auguste and Louis Lumière open the first
movie theater in Paris, France. Legend has it that one of their movie
shorts, Arrival of a Train at La Ciotat, is such a convincing
depiction of reality that people in the audience scream and run to
the back of the room.
- 1929: Edwin Link develops the Link Trainer (also called the Pilot
Maker), a mechanical airplane simulator. His work pioneers the field of flight simulation.
- 1950: US Air Force psychologist James J. Gibson publishes an
influential book, The Perception of the Visual World,
describing how people see and experience things as an "optic flow"
as they move through the world. These ideas, and those of
contemporaries such as Adelbert Ames, help to form the foundations of
the 20th century psychology of visual perception, which feeds into
academic studies of computer vision and virtual reality.
- 1956: Cinematographer Morton Heilig begins developing machines
that can produce artificial sensory experiences. In 1957, he develops a
pioneering 3D head-mounted display.
In 1962, he's granted a patent for a machine called the
Sensorama (a kind of updated, highly sophisticated Kinetoscope) that can immerse its user in artificial vision, sound, smell, and
vibration. Many people regard Heilig as the true father of virtual
reality, though he has seldom been acknowledged as such.
- 1961: C. Comeau and J. Bryan make Headsight, the first real
head-mounted display.
- 1962: Ivan Sutherland, pioneer of human-computer interaction,
develops the lightpen and a program called Sketchpad that allows
people to draw on a computer screen—opening the way for the kind of
computer graphics later used in virtual reality.
- 1965: Sutherland develops Ultimate Display, a sophisticated HMD.
- 1968. Sutherland produces a HMD with stereo (3D) vision.
- 1970s: Computer scientist Myron Krueger opens Videoplace, a
pioneering VR laboratory.
- 1975-1976: Programmer Will Crowther develops Adventure
(also called Colossal Cave Adventure), a highly influential,
text-based computer game in which players explore a virtual world,
solving problems through a question and answer dialog.
- 1977: Dan Sandin, Richard Sayre, and Thomas Defanti produce the
first dataglove.
- 1980s: Fast 3D graphical workstations, notably those developed by
Silicon Graphics, accelerate the development of scientific
visualization, visual computer modeling, CGI movies, and VR.
- 1982: Tron, a pioneering CGI movie, tells the story of a
software engineer (played by actor Jeff Bridges) who ventures into
the software of a mainframe computer.
- 1983: Computer scientist Myron Krueger coins the term "artificial
reality."
- 1983: Writer William Gibson invents the related term "cyberspace."
- 1989: Computer scientist and musician Jaron Lanier: coins the now preferred term "virtual reality." His company, VPL Research,
garners huge media attention and develops pioneering VR peripherals,
including a HMD and dataglove. Lanier has been popularly referred to
as the "father of virtual reality" ever since, though, as this
timeline demonstrates, he was quite a late entrant to the field!
- 1992: The Lawnmower Man, another influential VR film, based
on a Stephen King story, is partly inspired by the story of VPL.
- 1993: Brothers Robyn and Rand Miller create Myst, an
extremely successful graphical computer game, in which players
explore an island in non-immersive virtual reality.
- 1994: Dave Raggett, an influential English computer scientist who
has played a key role in the development of the World Wide Web,
coins the term VRML (Virtual Reality Markup Language).
- 1999: The Matrix, a movie starring Keanu Reaves based on
virtual reality, grosses over $450 million at the box office.
- 2011: Palmer Luckey develops the Oculus Rift, an inexpensive
homemade HMD, in his parents' garage.
- 2014: Facebook announces its acquisition of Oculus in a deal worth
$2 billion.
- 2016: Oculus begins shipping its Rift headsets to customers to generally positive reviews.
Meanwhile, a variety of smartphone makers (including Samsung and HTC) produce rival VR systems,
VR also appears for PlayStation, and Google announces it has shipped over 5 million cardboard head-mounted displays for smartphones.
- 2017: Sony reveals that it sold almost a million PlayStation VR headsets in their first four months on the market.
- 2018: Neurable announces a "brain-scanning" headband with built-in electrodes that gives hands-free control of virtual reality.
- 2019: Google scraps its Daydream VR project citing a lack of broad, consumer interest in VR.
- 2021: Facebook introduces "Horizon Workrooms," a way of meeting other people in virtual conference rooms.
