Augmented reality (AR)

Last updated: February 1, 2010.

Not so long ago, future-minded people enjoyed making a sharp distinction between "reality" (the often timesome, problematic world around us) and "cyberspace" (the promising new "virtual" world inside computers). Back in 1996, when the Web was still a novelty and relatively few people were online, John Perry Barlow famously declared that cyberspace is "a civilization of the Mind... both everywhere and nowhere, but it is not where bodies live."

Some years down the line, it's clear that black-and-white distinctions between real and virtual were incredibly wide of the mark. What most people actually want from computers is much more pragmatic: intuitively easy-to-use technology that will enhance their busy, everyday, real lives—to help them find a decent coffee shop, to make shopping less of a chore, to meet new friends who share their interests, or whatever it might be. Now computers are smaller and more portable than ever, and you can go online almost anywhere on the planet, using online information to enhance ("augment") real life is where the smart money is heading. Augmented reality, as it's known, is something we'll be hearing an awful lot more about in the next few years. Let's take a closer look at how it works!

Photo: Augmented reality: Who wants to stare at a computer when you can get Google in your goggles? Imagine being able to see maps right in front of your eyes or read emails as you walk along. Imagine looking at a famous building and being able to call up its Wikipedia entry, in your glasses, right then and there. These are some of the possible, future applications of augmented reality, which means blending computer-generated information with the world you can see in front of you. (Simulated image!)

What is augmented reality?

Augmented reality sounds horribly theoretical and abstract, but it's actually very simple: real life plus topical, relevant, background information equals something massively more useful. A few quick examples will make the idea clearer:

• You're watching a tennis grand-slam on TV and there's a controversial call from one of the line judges. Was that serve in or out? The TV station runs an instant replay with a computer animation showing the exact trajectory of the ball and where it landed—just outside the line. Then a little table comes up on the screen showing how many serves have been in or out for each player and how the figures have changed over the course of the match.
• You're walking the streets of London, England and you suddenly come across an amazing bit of architecture. What is this fantastic building? Who was the architect? Is that really titanium? You're dying to find out more, but the building is closed and there's no information about it at all. So you hold your cellphone up and take a quick photo. The phone uses its built-in GPS (satellite navigation) system to figure out roughly where you are, then quickly searches Google Images to find similar photos taken in the same neighborhood. In a couple of seconds, it's identified the building and brought up a Wikipedia page telling you all about it.
• You're a fighter pilot flying over a warzone with anti-aircraft fire shooting up at you. You really have to concentrate and looking down at all the gauges on your instrument panel is a distracton you can do without. Fortunately, you're wearing what's called a heads-up display (HUD), a set of goggles with built-in, miniaturized computers that automatically project instrument readings so they "float" in front of your eyes. You can find out everything you need to know without taking your eyes off the sky.
• You're driving down the freeway sometime in 2020 and you start to feel hungry. Wink your right eye twice and a computer display overlays your windshield with a list of eating places in nearby towns. Wink your eye to select the one that looks most promising and your sat-nav system reads out directions for how to get there. You can email an order in advance so it'll be ready when you arrive.

Photo: Left: A relatively cumbersome, prototype helmet-mounted display developed by NASA in the early 1990s: computer generated images are blended with what the pilot sees through his eyes. Photo courtesy of NASA Langley Research Center (NASA-LaRC). Right: A more recent prototype developed for the US Air Force in 2008. Here the augmented reality display is incorporated into an ordinary pair of wraparound eyeglasses. Photo by Jonathan Snyder courtesy of US Air Force and Defense Imagery.

You can see that augmented reality is actually a mixture of real life and virtual reality, somewhere in between the two, so it's often referred to as mixed reality. The key point is that the extra information it gives you is highly topical and relevant to what you want to do or know in a certain place and time.

How does AR tracking work?

If you're out and about in the real world with your laptop, netbook, or cellphone, it's easy enough to get information: just bring up Google and type in some words. In the brave new world of augmented reality, it's even easier: you get the extra information automatically. That means your portable computing device needs some automatic way of finding out where you are or what you're looking at—a problem known as tracking.

The simplest form of tracking is for the device to use GPS (or some other satellite navigation system) to figure out your position automatically, which is fine if you want broad, background information about a place your visiting (a local street map, a list of nearby coffee bars, directions to the nearest hotel, or whatever). It's relatively easy to use tracking information from Wi-Fi hotspots as well. But what if you're somewhere like an art gallery or museum and what you actually want is information about each picture or exhibit automatically coming up on your cellphone as you walk through the building? GPS isn't (yet) precise enough for an application like that, so what could we do instead? Broadly speaking, there are two different solutions known as marker-based and markerless tracking.

• Markerless tracking: You could point your phone at each picture or exhibit and have some kind of pattern recognition or feature-detection system try to identify it. That's how our own perceptual systems work, after all: our eyes see things and our brains figure out what we're looking at, then "call up" background information. Our brains are amazingly good at this and make it seem very easy, but it's a much harder problem for a computer to tackle, not least because the best computer vision systems are only a fraction as good as our own.

• Marker-based tracking: A simpler option would be for the gallery or museum to print small, two-dimensional barcodes (also called data-matrix codes) next to each item on display. Then you'd simply point your phone's camera at one of them, your phone would turn the barcode into a web address, and its browser would call up an appropriate web page with further information. AR systems can be designed to read all kinds of other markers (or fiduciary markers) as these "added reference points" are called.

Marker-based tracking will probably be popular initially, but in the longer term fully automatic, markerless tracking seems certain to win out because that's how our own visual systems work—and it's what most users prefer. After all, we recognize our friends automatically without them having to walk around with barcodes printed on their foreheads!

Photo: Marker-based tracking. This advertisement features a QR code® (a popular type of data-matrix code). You're supposed to see the ad and then point your phone camera at it for more information. Your phone will decode the black-and-white pattern into a URL (website address) that its browser will then load automatically. Photo by courtesy of Gaku, published on Flickr in 2007 under a Creative Commons Licence.

Displaying augmented reality

Virtual reality has taken off in a much more limited way than many people supposed. One reason for this is the difficulty of "immersing" people in a computer-generated world: ideally, you need to wear a sophisticated headset that completely blocks out the real world and replaces it with a computer-drawn equivalent. With augmented reality, the problem is different: what we need is a way of overlaying computer-generated information on what we can see already. When people first started talking about augmented reality, there was a feeling we'd all end up walking around in heads-up displays, like fighter pilots. Now Net-connected cellphones with cameras are commonplace, they've become the focus for AR—although that may change in future. Already, researchers are working on prototype contact lenses with built-in computer displays, which would superimpose things like Web pages onto our field of vision. Another possibility is wearing eyeglasses with miniature, forward-facing cameras built into the frames and transparent screens added to the lenses.

Photo: Goggles like these are fine for virtual reality applications, such as parachute training sessions run by the US Air Force, but they're too cumbersome for most of us to want to wear routinely. Picture by courtesy of US Air Force.

What kind of AR applications already exist?

There's plenty of online information and no shortage of reality—the trick is bringing the two things together. Some exciting strides are already being made in this area by cellphone applications such as Layar, Wikitude, and Yelp. With Layer, you simply look through your cellphone camera at the world in front of you and see layers of extra information, like transparent webpages, added on top. Wikitude offers an application called World Browser, which overlays useful information about landmarks and other points of interest seen through a camera phone. Yelp gives you instant access to reviews of shops, restaurants, hotels and so on in American cities such as San Francisco. Expect to see lots more AR applications along these lines in the next few years!