Augmented reality (AR)
by Chris Woodford. Last updated: May 2, 2021.
Not so long ago, future-minded people enjoyed making a sharp
distinction between "reality" (the often tiresome, problematic
world around us) and "cyberspace" (the promising new "virtual"
world inside computers). Back in 1996, when the Web was still a novelty and relatively few people were online, John Perry Barlow
famously declared that cyberspace is "a civilization of the Mind...
both everywhere and nowhere, but it is not where bodies live."
Some years down the line, it's clear that black-and-white distinctions
between real and virtual were incredibly wide of the mark. What
most people actually want from computers is much more pragmatic:
intuitively easy-to-use technology that will enhance their busy,
everyday, real lives—to help them find a decent coffee shop,
to make shopping less of a chore, to meet new friends who share their
interests, or whatever it might be. Now computers are smaller and
more portable than ever, and you can go online almost anywhere on the
planet, using online information to enhance ("augment") real life
is where the smart money is heading. Augmented
reality, as it's known, is something we'll be hearing an awful
lot more about in the next few years. Let's take a closer look at how
it works!
Photo: Augmented reality: What's that plant? Just point your tablet computer at the
pesky thing so the digital camera can take a photo, press a button, and the plant-finder app automatically identifies what you're looking at.
In seconds, it's called up the Wikipedia entry (left) and a selection of photos of the species on Flickr (right) so you can
double-check. This is only one of the possible, future applications of augmented reality,
which means blending computer-generated information with the world you can see in front of you. (Simulated image.)
What is augmented reality?
Augmented reality sounds horribly theoretical and abstract, but
it's actually very simple: real life plus topical, relevant,
background information equals something massively more useful. A few
quick examples will make the idea clearer:
- You're watching a tennis grand-slam on TV and there's a
controversial call from one of the line judges. Was that serve in or
out? The TV station runs an instant replay with a computer animation
showing the exact trajectory of the ball and where it landed—just
outside the line. Then a little table comes up on the screen showing
how many serves have been in or out for each player and how the
figures have changed over the course of the match.
- You're walking the streets of London, England and you
suddenly come across an amazing bit of architecture. What is this
fantastic building? Who was the architect? Is that really titanium?
You're dying to find out more, but the building is closed and
there's no information about it at all. So you hold your cellphone
up and take a quick photo. The phone uses its built-in GPS
(satellite navigation) system to figure out roughly where you are,
then quickly searches Google Images to find similar photos taken in
the same neighborhood. In a couple of seconds, it's identified the
building and brought up a Wikipedia page telling you all about it.
- You're a fighter pilot flying over a warzone with
anti-aircraft fire shooting up at you. You really have to
concentrate and looking down at all the gauges on your instrument
panel is a distraction you can do without. Fortunately, you're
wearing what's called a heads-up display (HUD), a set of goggles
with built-in, miniaturized computers that automatically
project instrument readings so they "float" in front of your eyes.
You can find out everything you need to know without taking your
eyes off the sky.
- You're driving down the freeway sometime in 2020 and you
start to feel hungry. Wink your right eye twice and a computer
display overlays your windshield with a list of eating places in
nearby towns. Wink your eye to select the one that looks most
promising and your sat-nav system reads out directions for how to
get there. You can email an order in advance so it'll be ready when
you arrive.
You can see that augmented reality is actually a mixture of real
life and virtual reality, somewhere in between the two, so it's often
referred to as mixed reality. The key point is that the extra
information it gives you is highly topical and relevant to what you
want to do or know in a certain place and time.
Photos: 1) A relatively cumbersome, prototype helmet-mounted display developed by NASA in the early 1990s: computer generated images are blended with what the pilot sees through his eyes. Photo courtesy of NASA Langley Research Center (NASA-LaRC).
2) A more recent prototype developed for the US Air Force in 2008. Here the augmented reality display is incorporated into an ordinary pair of wraparound eyeglasses. Photo by Jonathan Snyder courtesy of US Air Force.
How does AR tracking work?
If you're out and about in the real world with your laptop,
netbook, or cellphone, it's easy enough to get information: just
bring up Google and type in some words. In the brave new world of
augmented reality, it's even easier: you get the extra information
automatically. That means your portable computing device needs some
automatic way of finding out where you are or what you're looking
at—a problem known as tracking.
The simplest form of tracking is for the device to use GPS (or some
other satellite navigation system) to figure out your position
automatically, which is fine if you want broad, background
information about a place you're visiting (a local street map, a list
of nearby coffee bars, directions to the nearest hotel, or whatever).
It's relatively easy to use tracking information from Wi-Fi hotspots
as well. But what if you're somewhere like an art gallery or museum
and what you actually want is information about each picture or
exhibit automatically coming up on your cellphone as you walk through
the building? GPS isn't (yet) precise enough for an application like
that, so what could we do instead? Broadly speaking, there are two
different solutions known as marker-based and markerless tracking.
- Markerless tracking: You could point your phone at each
picture or exhibit and have some kind of pattern recognition or
feature-detection system try to identify it. That's how our own
perceptual systems work, after all: our eyes see things and our
brains figure out what we're looking at, then "call up"
background information. Our brains are amazingly good at this and
make it seem very easy, but it's a much harder problem for a
computer to tackle, not least because the best computer vision
systems are only a fraction as good as our own.
- Marker-based tracking: A simpler option would be for the
gallery or museum to print small, two-dimensional barcodes (also
called data-matrix codes) next to each item on display. Then you'd
simply point your phone's camera at one of them, your phone would
turn the barcode into a web address, and its browser would call up
an appropriate web page with further information. AR systems can be
designed to read all kinds of other markers (or
fiducial markers)
as these "added reference points" are called.
Marker-based tracking is currently proving to be the more popular option,
largely because it's so simple to implement. But in
the longer term fully automatic, markerless tracking seems certain
to win out because that's how our own visual systems work—and it's
what most users prefer. After all, we recognize our friends
automatically without them having to walk around with
barcodes printed on their foreheads!
Photo: Marker-based tracking: The Inkhunter app (described below) demonstrates what a tattoo would look like anywhere on your body. Before it can do that, you need to help it figure out where to do its stuff by drawing a "square smile" on your skin. This is a simple example of marker-based tracking.
Displaying augmented reality
Virtual reality has taken off in a much more limited way than many
people supposed. One reason for this is the difficulty of "immersing"
people in a computer-generated world: ideally, you need to wear a
sophisticated headset that completely blocks out the real world and
replaces it with a computer-drawn equivalent. With augmented reality,
the problem is different: what we need is a way of overlaying
computer-generated information on what we can see already. When
people first started talking about augmented reality, there was a
feeling we'd all end up walking around in heads-up displays, like
fighter pilots. Now Net-connected cellphones with cameras are
commonplace, they've become the focus for AR—although that may
change in future. Already, researchers are working on prototype
contact lenses with built-in computer displays, which would
superimpose things like Web pages onto our field of vision. Another
possibility is wearing eyeglasses with miniature, forward-facing
cameras built into the frames and transparent screens added to the
lenses.
What kind of AR applications already exist?
Links in this section take you mostly to YouTube demos of the various apps, which I think
is the best way of giving you a quick taste of what's available.
When the creators of the James Bond films came up with the movie title "The World is Not
Enough," they might have been talking about augmented reality. Lots
of things we do everyday rely on us simultaneously interacting with
the world outside our bodies and sneaking a glance at the digital
world inside our computers; enhancing the real world is where
augmented reality comes into its own. So what sort of applications
are already available?
If you want to change your world but you're not sure how to go about it, augmented
reality's happy to help. Take home remodeling for example. It's
often quite hard to imagine what your rooms will look like painted a
different shade of red, blue, yellow, or gold—which will look best,
and what if you make a mistake? Augmented reality home decorating
apps like the
Dulux Visualizer allow you to look through your
smartphone camera and change the colors of your walls at will. Also
designed with home makeover buffs in mind,
IKEA's Catalog app lets
you try out pieces of virtual furniture in your real-world home. If
it's your body you want to change, rather than your home, how about
one of the many clothes and shoe samplers? They let you see what a
new pair of sneakers (or shorts) would look like without going
anywhere near a shop. In much the same vein,
Inkhunter is
a tattoo app that work similar magic on your skin, allowing you to
check what a new design would look like on any part of your body
before going anywhere near a needle!
Photo: Decorating with AR: What would your home look like painted purple, green, or yellow? Decorating apps like Dulux Visualizer, shown here, give you an instant impression.
If all this sounds a little bit frivolous, it's worth remembering that augmented reality
has lots of serious applications in everything from medical education
to parcel transportation. Doctors use AR both for teaching (medical
students can use AR apps to
explore organs in 3D, then project them
onto their own bodies) and for "previewing" to patients how scary
treatments might affect them; a company called Orca MD makes several
"decide" apps (Eyedecide, Knee Decide, and Foot Decide to name
just three) that allow people to explore what bits of their body look
like from the inside when they're suffering a variety of medical
conditions. In 2020, doctors from Johns Hopkins University used
augmented reality to help them perform spinal surgery more effectively.
We're not all doctors and rocket scientists in the
making, but AR can help all of us, in one way or another. DHL, the
delivery company, has trialled augmented reality headsets that help
its warehouse staff to find and pack parcels more quickly. And
where heads-up displays used to be the preserve of fighter pilots,
now they're heading for the mainstream in car windscreens and
motorcycle visors (though the long-anticipated, crowd-funded
Skully
AR motorbike helmet failed to make it into production).
Classroom education's another area where reality is well worth augmenting. If
you want to teach people about the stars and planets, for example,
what better way than to use a smartphone or tablet that can help them
label the things they can see in the sky above their heads?
Sun Seeker and
Star Walk are
just two of many AR applications giving students an inspiring new way to find their way around the world.
Satellites and passing flights can be tracked the same way; check out
the app version of Flight Radar for example.
Augmented reality is endlessly entertaining—and fun too. Who could ever forget the
global buzz of excitement when the most popular AR game to date,
Pokémon Go, hit the streets in 2016.
By 2019, it had been downloaded over a billion times, making it the most popular and accessible
introduction to augmented reality so far. Slightly less exciting,
but no less useful, there are apps like
Layar that add digital information to existing real-world stuff: you could scan a magazine to see the adverts leap to life, for example, or point your smartphone at a textbook to watch some handy, related videos.
If you're interested in developing your own AR apps, you don't necessarily have to start
from scratch. Wikitude is probably
the best known AR-creating software for mobile apps, with built-in object recognition and tracking.
