Augmented reality (AR)
by Chris Woodford. Last updated: February 2, 2017.
Not so long ago, future-minded people enjoyed making a sharp distinction between "reality" (the often tiresome, problematic world around us) and "cyberspace" (the promising new "virtual" world inside computers). Back in 1996, when the Web was still a novelty and relatively few people were online, John Perry Barlow famously declared that cyberspace is "a civilization of the Mind... both everywhere and nowhere, but it is not where bodies live."
Some years down the line, it's clear that black-and-white distinctions between real and virtual were incredibly wide of the mark. What most people actually want from computers is much more pragmatic: intuitively easy-to-use technology that will enhance their busy, everyday, real lives—to help them find a decent coffee shop, to make shopping less of a chore, to meet new friends who share their interests, or whatever it might be. Now computers are smaller and more portable than ever, and you can go online almost anywhere on the planet, using online information to enhance ("augment") real life is where the smart money is heading. Augmented reality, as it's known, is something we'll be hearing an awful lot more about in the next few years. Let's take a closer look at how it works!
Photo: Augmented reality: What's that plant? Just point your tablet computer at the pesky thing so the digital camera can take a photo, press a button, and the plant-finder app automatically identifies what you're looking at. In seconds, it's called up the Wikipedia entry (left) and a selection of photos of the species on Flickr (right) so you can double-check. This is only one of the possible, future applications of augmented reality, which means blending computer-generated information with the world you can see in front of you. (Simulated image.)
What is augmented reality?
Augmented reality sounds horribly theoretical and abstract, but it's actually very simple: real life plus topical, relevant, background information equals something massively more useful. A few quick examples will make the idea clearer:
- You're watching a tennis grand-slam on TV and there's a controversial call from one of the line judges. Was that serve in or out? The TV station runs an instant replay with a computer animation showing the exact trajectory of the ball and where it landed—just outside the line. Then a little table comes up on the screen showing how many serves have been in or out for each player and how the figures have changed over the course of the match.
- You're walking the streets of London, England and you suddenly come across an amazing bit of architecture. What is this fantastic building? Who was the architect? Is that really titanium? You're dying to find out more, but the building is closed and there's no information about it at all. So you hold your cellphone up and take a quick photo. The phone uses its built-in GPS (satellite navigation) system to figure out roughly where you are, then quickly searches Google Images to find similar photos taken in the same neighborhood. In a couple of seconds, it's identified the building and brought up a Wikipedia page telling you all about it.
- You're a fighter pilot flying over a warzone with anti-aircraft fire shooting up at you. You really have to concentrate and looking down at all the gauges on your instrument panel is a distraction you can do without. Fortunately, you're wearing what's called a heads-up display (HUD), a set of goggles with built-in, miniaturized computers that automatically project instrument readings so they "float" in front of your eyes. You can find out everything you need to know without taking your eyes off the sky.
- You're driving down the freeway sometime in 2020 and you start to feel hungry. Wink your right eye twice and a computer display overlays your windshield with a list of eating places in nearby towns. Wink your eye to select the one that looks most promising and your sat-nav system reads out directions for how to get there. You can email an order in advance so it'll be ready when you arrive.
You can see that augmented reality is actually a mixture of real life and virtual reality, somewhere in between the two, so it's often referred to as mixed reality. The key point is that the extra information it gives you is highly topical and relevant to what you want to do or know in a certain place and time.
Photos: 1) A relatively cumbersome, prototype helmet-mounted display developed by NASA in the early 1990s: computer generated images are blended with what the pilot sees through his eyes. Photo courtesy of NASA Langley Research Center (NASA-LaRC). 2) A more recent prototype developed for the US Air Force in 2008. Here the augmented reality display is incorporated into an ordinary pair of wraparound eyeglasses. Photo by Jonathan Snyder courtesy of US Air Force.
How does AR tracking work?
If you're out and about in the real world with your laptop, netbook, or cellphone, it's easy enough to get information: just bring up Google and type in some words. In the brave new world of augmented reality, it's even easier: you get the extra information automatically. That means your portable computing device needs some automatic way of finding out where you are or what you're looking at—a problem known as tracking.
The simplest form of tracking is for the device to use GPS (or some other satellite navigation system) to figure out your position automatically, which is fine if you want broad, background information about a place you're visiting (a local street map, a list of nearby coffee bars, directions to the nearest hotel, or whatever). It's relatively easy to use tracking information from Wi-Fi hotspots as well. But what if you're somewhere like an art gallery or museum and what you actually want is information about each picture or exhibit automatically coming up on your cellphone as you walk through the building? GPS isn't (yet) precise enough for an application like that, so what could we do instead? Broadly speaking, there are two different solutions known as marker-based and markerless tracking.
- Markerless tracking: You could point your phone at each picture or exhibit and have some kind of pattern recognition or feature-detection system try to identify it. That's how our own perceptual systems work, after all: our eyes see things and our brains figure out what we're looking at, then "call up" background information. Our brains are amazingly good at this and make it seem very easy, but it's a much harder problem for a computer to tackle, not least because the best computer vision systems are only a fraction as good as our own.
- Marker-based tracking: A simpler option would be for the gallery or museum to print small, two-dimensional barcodes (also called data-matrix codes) next to each item on display. Then you'd simply point your phone's camera at one of them, your phone would turn the barcode into a web address, and its browser would call up an appropriate web page with further information. AR systems can be designed to read all kinds of other markers (or fiducial markers) as these "added reference points" are called.
Marker-based tracking is currently proving to be the more popular option, largely because it's so simple to implement. But in the longer term fully automatic, markerless tracking seems certain to win out because that's how our own visual systems work—and it's what most users prefer. After all, we recognize our friends automatically without them having to walk around with barcodes printed on their foreheads!
Photo: Marker-based tracking: The Inkhunter app (described below) demonstrates what a tattoo would look like anywhere on your body. Before it can do that, you need to help it figure out where to do its stuff by drawing a "square smile" on your skin. This is a simple example of marker-based tracking.
Displaying augmented reality
Virtual reality has taken off in a much more limited way than many people supposed. One reason for this is the difficulty of "immersing" people in a computer-generated world: ideally, you need to wear a sophisticated headset that completely blocks out the real world and replaces it with a computer-drawn equivalent. With augmented reality, the problem is different: what we need is a way of overlaying computer-generated information on what we can see already. When people first started talking about augmented reality, there was a feeling we'd all end up walking around in heads-up displays, like fighter pilots. Now Net-connected cellphones with cameras are commonplace, they've become the focus for AR—although that may change in future. Already, researchers are working on prototype contact lenses with built-in computer displays, which would superimpose things like Web pages onto our field of vision. Another possibility is wearing eyeglasses with miniature, forward-facing cameras built into the frames and transparent screens added to the lenses.
What kind of AR applications already exist?
Links in this section take you mostly to YouTube demos of the various apps, which I think is the best way of giving you a quick taste of what's available.
When the creators of the James Bond films came up with the movie title "The World is Not Enough," they might have been talking about augmented reality. Lots of things we do everyday rely on us simultaneously interacting with the world outside our bodies and sneaking a glance at the digital world inside our computers; enhancing the real world is where augmented reality comes into its own. So what sort of applications are already available?
If you want to change your world but you're not sure how to go about it, augmented reality's happy to help. Take home remodeling for example. It's often quite hard to imagine what your rooms will look like painted a different shade of red, blue, yellow, or gold—which will look best, and what if you make a mistake? Augmented reality home decorating apps like the Dulux Visualizer allow you to look through your smartphone camera and change the colors of your walls at will. Also designed with home makeover buffs in mind, IKEA's Catalog app lets you try out pieces of virtual furniture in your real-world home. If it's your body you want to change, rather than your home, how about one of the many clothes and shoe samplers? They let you see what a new pair of sneakers (or shorts) would look like without going anywhere near a shop. In much the same vein, Inkhunter is a tattoo app that work similar magic on your skin, allowing you to check what a new design would look like on any part of your body before going anywhere near a needle!
Photo: Decorating with AR: What would your home look like painted purple, green, or yellow? Decorating apps like Dulux Visualizer, shown here, give you an instant impression.
If all this sounds a little bit frivolous, it's worth remembering that augmented reality has lots of serious applications in everything from medical education to parcel transportation. Doctors use AR both for teaching (medical students can use AR apps to explore organs in 3D, then project them onto their own bodies) and for "previewing" to patients how scary treatments might affect them; a company called Orca MD makes several "decide" apps (Eyedecide, Knee Decide, and Foot Decide to name just three) that allow people to explore what bits of their body look like from the inside when they're suffering a variety of medical conditions. We're not all doctors and rocket scientists in the making, but AR can help all of us, in one way or another. DHL, the delivery company, has trialled augmented reality headsets that help its warehouse staff to find and pack parcels more quickly. And where heads-up displays used to be the preserve of fighter pilots, now they're heading for the mainstream in car windscreens and motorcycle visors (though the long-anticipated, crowd-funded Skully AR motorbike helmet failed to make it into production).
Classroom education's another area where reality is well worth augmenting. If you want to teach people about the stars and planets, for example, what better way than to use a smartphone or tablet that can help them label the things they can see in the sky above their heads? Sun Seeker and Star Walk are just two of many AR applications giving students an inspiring new way to find their way around the world. Satellites and passing flights can be tracked the same way; check out the app version of Flight Radar for example.
Augmented reality is endlessly entertaining—and fun too. Who could ever forget the global buzz of excitement when the most popular AR game to date, Pokémon Go, hit the streets in 2016. Already, it's been download well over a half-billion times, making it the most popular and accessible introduction to augmented reality so far. Slightly less exciting, but no less useful, there are apps like Layar that add digital information to existing real-world stuff: you could scan a magazine to see the adverts leap to life, for example, or point your smartphone at a textbook to watch some handy, related videos.
If you're interested in developing your own AR apps, you don't necessarily have to start from scratch. Wikitude is probably the best known AR-creating software for mobile apps, with built-in object recognition and tracking.
Find out more
On this website
- Understanding Augmented Reality: Concepts and Applications by Alan B. Craig. Newnes, 2013. A clear and accessible introduction.
- Professional Augmented Reality Browsers for Smartphones: Programming for Junaio, Layar and Wikitude by Lester Madden. John Wiley and Sons, 2011.
- Emerging Technologies of Augmented Reality: Interfaces and Design by Michael Haller, Mark Billinghurst, and Bruce H. Thomas. Idea Group Inc (IGI), 2007.
- Augmented Reality: Placing Artificial Objects in Real Scenes by Reinhold Behringer et al (eds). A K Peters, 1999. Proceedings of the First International Workshop on Augmented Reality.
- Leap Motion's Augmented-Reality Computing Looks Stupid Cool by Kyle VanHemert. Wired, 7 July 2015. Are 3D augmented workspaces the future of desktop computing?
- How AR could change our view of the world by Spencer Kelly, BBC Click, 24 June 2011. A short video demonstration of cutting-edge augmented reality on tablet computers.
- Augmented Reality Comes Closer to Reality by John Markoff. The New York Times, 7 April 2011. Will augmented reality take us beyond conventional computer interfaces?
- Augmented reality: it's like real life, but better by Charles Arthur, The Guardian/Observer, 21 March 2010. Argues that AR could revolutionize sport, advertising, and much more.
- Wired: Augmented reality Read recent posts about AR from Wired magazine.
If you'd like some technical detail about how AR systems actually work, these patent documents are worth check out:
- US Patent Application Publication: Augmented reality shopper routing by Robert Nicholson, IBM Corporation, February 25 2011. Explains how augmented reality could be used to guide shoppers through a store, quickly guiding them to the products they actually want to buy.
- US Patent Application Publication: Markerless augmented reality system and method using projective invariant by Suk June Yoo et al, Samsung Electronics, 21 April 2011. A good technical description of how a markerless tracking system can extract features from a real-world image and then accurately project augmented graphical images on top.
- US Patent Application Publication: Systems and Methods for Augmented Reality by Mark J. Roebke et al, 16 June 2011. This describes how something like a smartphone can use data from its GPS, compass, accelerometers, and other sensors to identify a location, then access a database of information, and overlay appropriate augmented reality data on top of an image captured by the phone's camera.