Tech n Gadgets Blog: Augmented reality in action

AR WEEK: Will we ever need a mechanic again?
Augmented reality in action - maintenance and repair. Science, Augmented Reality, Features, Interviews, Steve Feiner, AR Week, 0

Picture the scene. It’s late. You’re driving down the motorway on a snowy Christmas Eve when disaster strikes and that ratting somewhere in your engine reaches breaking point. Your car slows to a stop and with it your dream of getting home in time for turkey. Or so you think, but that's only if you're looking at this scene as someone from 2011.

This is the future, and a not too distant one at that. So, you get out of your car with no automotive repair knowledge whatsoever and open the bonnet to see what mystery awaits. As you prop up the metal lid, a series of tiny projectors flicker on like the lights of a kitchen fridge and, instructed by your car’s internal system, they display virtual, computer generated directions of exactly what the problem is and, more importantly, what you need to do to fix it. This is the goal of Steve Feiner, Professor of Computer Science at Columbia University and augmented reality pioneer.

“Well, there are a few problems with using fixed projectors for AR,” he explains to Pocket-lint grounding our dream down a crackly line from his office in New York.

“First, they’re limited in the volume that they can project and, second, not all surfaces are good for projecting onto. You could probably do it onto the nice, shiny metal parts of a car not long after you’ve driven it out of the showroom but once it’s covered in grease and whatever else, I’m not so sure.”

Instead, Feiner and his team on the ARMAR project (Augmented Reailty for Maintainence and Repair) have been focusing on the use of head-mounted displays (HMDs) since inception in 2009. Funded and partnered by the US Marine Corps, they’ve managed to build a set a virtual set of instructions and guidance system for soldiers out in the field to perform repairs and maintenance tasks to the LAV-25A1 armored personnel carrier, what Feiner describes as essentially “a smallish tank”.

The system works by overlaying the computer generated graphical assistance over the soldier's real view of the inside of the vehicle as supplied by a camera attached to their headwear. Their movements are tracked such that a red arrow from over the shoulder can point them in the right direction for the next part of the task and, once in field of view, a simple animation of the missing part and where it should go leads into the correct mechanical move.

“We got to test the system out with at the Marines’s Aberdeen training ground using people who had enlisted and just taken the course to become mechanics. These were kids who grew up fiddling with cars and taking radios apart to see what’s inside, and they all really liked the AR system. Some would even hang out while we were setting up the tests. These were all people who’d grown up with 3D and computer games and they were really interested in what we were doing.”

The full ARMAR set up was tested against a group just working from a normal set of PDF instructions on an LCD screen inside the vehicles as well a third set of subjects with an HMD with virtual instructions that just hung in the air, i.e. not with the animations over the parts where they should be. As had been hoped, even with trained mechanics, those using the full AR set up performed the task significantly better.

Feiner and his team actually tracked the movements of the subjects’ heads to see how efficient their work time was. Those using the ARMAR system used 37 per cent less head travel than those with the PDF instructions whom themselves ended up spending much of the task with their heads bobbing back and forth between the LCD screen and parts of the vehicle they were fixing.

“If you looked at the head movements of someone who knew exactly what to do without instructions at all, in the case of AR, it was very close to that; not perfect, but very close.”

A highly successful piece of work then. So, have the US Marines adopted AR in their armoured vehicles as standard? Not quite yet.

“It’s not really ready yet. It needs to be counted on, so the software needs to be amazing,” explains Feiner detailing the ongoing work on the project at Columbia.

“Unlike mobile AR, which if it doesn’t work 100 per cent of the time, it doesn’t matter, this has to be relied upon absolutely and, at the time, what we had come up with wasn’t bulletproof.”

The hardware itself hadn’t quite been what Feiner had wanted either. It turned out that the top grade HMDs were too big to allow the mechanics to turn their heads properly in the small turret space they had to work inside the vehicles. Two years of ironing out the creases later, though, and the puzzle is nearly complete

“We’re approaching that point where the stuff is ready to go. We’re just trying to problem solve the issues.”

“We have computer power, bandwidth is good enough to get information through and the tracking is good as well but there are still problems when we need really small details - such as getting down to the level of one small screw among a very small area of hundreds - but that’s coming because there are lots of people out there working on that.”

Metaio and Qualcomm are two of such companies sponsoring just that kind of research. The ARMAR set up is a pre-mapped environment which the computer in charge has a copy of. To get the real and the virtual to line up, it’s a case of knowing the starting point of the mechanic’s head and accurately tracking its movements through the space from there. The other option is for the software to recognise real world objects through the camera and to adjust the computer generated components in real time.

The likes of a smartphone can already track physical edges as well as faces, of course, and Feiner is confident that a detailed enough system is getting near. Proof of such advancement was one of his own student’s apps, FARMAR, which reached the latter stages of Qualcomm’s augmented reality developer challenge. One area where the system doesn’t need to be perfect, though, is the graphics themselves, in fact, it’s of paramount importance that they’re not.

“It’s no good if we develop the animations to the point where they look like the real thing. The mechanics need to be able to differentiate between the real and the virtual otherwise it becomes impossible to know which bits to fix. But, if you think about it, all manuals and instructions have used caricature-type drawings, so what we have right now fits just right.”

The big question, of course, is should this system become “bulletproof” as Feiner puts it, if the US Marines do use this as standard in their vehicles, does that mean that we, the consumer, will ever get a taste of AR in our maintenance and repairs as well?

“Oh, absolutely. I believe this is for end users and not just professionals; folks who might know a few low level things - how to use a screwdriver and a wrench - but not in what order to do stuff. It should show you what to do next and how to do it as well. So it’s not just ‘this screw here’ but ‘tighten this one carefully but not too tight’ or ‘be careful with this task because this part here will pop up’.

“Most of all though, to be really successful, this stuff has to be fun and not burdensome or no one will use it.”

If that means cutting down on garage bills, mending your own boiler and never having to call Ikea because the manual’s not clear enough, then we’re pretty sure that’ll make AR enjoyable enough on its own.

Tech n Gadgets Blog

Thursday, March 3, 2011

Augmented reality in action - maintenance and repair

No comments:

Post a Comment