Research

Evan Wells, a Los Alamos National Laboratory employee and former Marine, models the augmented reality headset that helps train the military’s explosive ordnance disposal technicians. Photo Courtesy LANL

In an article published on August 1, 2019, the organization Action on Armed Violence reported that the previous day, a roadside improvised explosive device—commonly known as an IED—detonated under a bus in Afghanistan’s Farah province. This IED killed at least 32 passengers, critically injuring an additional 15. Although no group has claimed responsibility for this attack, it is known that the Taliban and ISIS (Islamic State of Iraq and Syria) operate in this part of the world and frequently use IEDs.

Just what is an IED? According to the Department of Homeland Security, the term was coined in 2003 during the Iraq War. An IED is a “homemade” bomb or destructive device, one designed to destroy, incapacitate, harass, or distract an enemy.

The practice of placing explosive charges as a means of harming the enemy is not a new invention. Before IEDs there were various types of just-under-the-ground explosive devices, such as surface mines and underwater depth charges, designed to destroy military vehicles and personnel. The insidious nature of such weapons led to development of the 1997 Mine Ban Treaty, which specifically outlaws the production, use, stockpiling, and transfer of antipersonnel mines. More than 164 nations have signed this treaty to achieve a mine-free world.

Enter the IED, which is typically made by civilians using conventional materials and commercially available chemicals. Such improvised explosive devices can be used by criminals, vandals, terrorists, suicide bombers, and insurgent organizations. The goal of those who use IEDs is not only to cause damage and loss of life but also to call attention to their organization or to a radical cause. One of the particularly harrowing characteristics of the IED is its adaptability. Almost any organization can create and use IEDs to cause harm to targets of its choice, including civilians and first responders such as firefighters, medical personnel, and the police.

Every year, IED attacks injure or kill more people than any other type of weapon, save firearms. According to Action on Armed Violence, a review of selected international media reports from 2011 to 2015 revealed more than 6,300 IED explosions, which killed more than 105,000 individuals. According to a letter from the United Nations Security Council, about half of the world’s countries have suffered from IED attacks.

Training for the chaos

One of the principal problems associated with IEDs is their improvised nature. An IED can be made with various combinations of materials and chemicals. To cause more damage than just the blast of an explosion, some IEDs include screws and nails, scrap metal, and other materials that add fragments to the explosion. These fragments are launched when the IED goes off, damaging buildings, injuring personnel, and destroying vehicles. IEDs can be of any size, from small pipe bombs and vest bombs (used by suicide bombers) to more sophisticated devices planted in vehicles to be used in destroying part or all of a building.

Such IED variability makes training designed to locate, identify, and ultimately render safe such devices a great challenge. The core idea behind training responders such as police and bomb squads is to make the training as realistic as possible. As with all such hazard-laden training, however, safety must always be of utmost importance. At first, bringing the inherent dangers associated with reality-based training together with comprehensive safety may sound impossible, but scientists at Los Alamos National Laboratory feel that they have found an answer. That answer is to employ augmented reality in such training.

It’s not virtual, it’s augmented

A small LANL Alamos team led by David Mascareñas of the National Security Education Center has developed a software suite that, in conjunction with the hardware associated with augmented reality, can be used to train first responders in locating, identifying, and rendering IEDs safe under a variety of scenarios.

Mascareñas explains that augmented reality is basically an offshoot of its more recognizable cousin, virtual reality. “Virtual reality is designed to completely replace an individual’s real-world environment,” he says, “whereas augmented reality alters the perception of a real-world environment.”

Mascareñas points to the recent Pokémon Go fad as a classical example of augmented reality. Released in 2016, the Pokémon Go game enabled smartphone users to locate, battle, and train virtual creatures, known as Pokémon. These virtual creatures could be found in a user’s real-world location using a smartphone. Pokémon Go has gone on to become a worldwide hit, with more than 1 billion downloads and earnings in excess of $3 billion.

The immersive nature of augmented reality makes it a huge draw when it comes to gaming, as players can find themselves in outrageous situations in an otherwise real environment. Mascareñas, among others, has discovered, however, that it is also possible to use augmented reality to train first responders in countering dangerous devices with no chance of exposing the trainees to actual physical danger.

After all, the danger is virtual—it exists only in the realm of the digital—thus enabling trainees to face real-world dangers in real-world environments without worrying about getting hurt. Imagine firefighting trainees learning how to battle fires before they train under more real-life scenarios. Picture bomb squads locating and disarming an IED using virtual tools and explosives before moving on to train under more reality-driven scenarios. Augmented reality can be used to enhance the extensive training regimes of our emergency response community without the need to add the time and expense of hands-on training.

“We have developed a prototype of this technology specifically designed to help first responders train in countering IEDs,” Mascareñas says. “So, what we’ve done is take the various components that make up an IED and use augmented technology to overlay these components onto the real world.”

A key component of augmented reality is a headset that typically comes with transparent lenses, headphones, and a microphone. This headset is linked to a computer that runs a software program. Think about the interface Neo used in the 1999 science-fiction film The Matrix, only in this instance the software enables the user to blend objects into the real world rather than a user entering a virtual computer world.

“I like to think of it as someone wearing a set of cool, oversized glasses,” Mascareñas says. “The software then projects what amounts to overlaid holograms onto a real-world environment, to where it is possible for the user to manipulate these holograms. These glasses enable you to see everything in the real world, but you can also see and hear holograms and virtual objects in this environment. As such, I would say that augmented reality is much more useful than virtual reality when it comes to military and industrial applications.”

Although Mascareñas imagines many uses for this technology, his current focus is on IED awareness and training.

“Our software enables users to bring up information about specific IEDs to identify exactly what they are dealing with,” Mascareñas continues. “The software can also simulate various types of sensors, such as an infrared sensor that once tripped, can trigger an IED to explode. If a trainee gets too close to such a sensor, for example, it can set off the IED, thus ending the training exercise.”

Although Mascareñas and his team had experience in augmented reality, they did not have great knowledge or experience when it came to IEDs and explosives in general.

“We reached out and collaborated with explosives experts here at the Lab,” Mascareñas says. “Working with such experts, we were able to develop virtual mockups of IEDs. For the prototype, we then hid an IED in a trash can. Now, there are all kinds of scenarios when it comes to IED placement—the trash can simulation is the first of many. We then designed a virtual infrared sensor, which pretty much works like a garage door opener. We even designed a defusing device, a water-canon disruptor, which the trainees must place in the proper place for it to defuse the IED. Otherwise, the IED will go off.”

Mascareñas says future work includes emulating the different types of sensors used to interrogate IEDs to determine what’s in them, developing a greater suite of IED types (each of which comes with its own set of problems) and incorporating different virtual objects where an IED can be hidden to achieve maximum damage (a pipe bomb in a suitcase or an IED inside the camper of a large truck, for example).

“We’ve actually started work on virtual tools to counter IEDs,” Mascareñas says. “For example, we’re working on simulating a bomb-disposal robot with virtual controls. Thus, you can simulate not only an IED and the object it’s hidden in but also the tools necessary to defuse the IED.”

Getting into augmented reality

Mascareñas and his team have been exploring the applications of augmented reality for about three years.

“Before getting into augmented reality, my team and I worked on a project that used virtual reality to address problems related to nuclear criticality safety,” Mascareñas says. “The project focused on the safety of a technician operating a glovebox. We then changed the approach, using augmented reality instead of virtual reality. We found that augmented reality worked much better when it came to training—we also found that trainees learned faster and retained more knowledge when we used augmented reality.”

As Mascareñas and his team expanded their understanding of augmented reality, they started to find various applications for it. “We’ve done some work on global security, such as visualizing satellite data. We’ve also worked with the United States Department of State in ways related to treaty verification.”

Augmented reality also played a minor role in developing a structural health monitoring tool known as ViDeoMAgic, which in 2018 earned the Laboratory an R&D 100 Award from R&D Magazine. Dubbed the “Oscars of Invention,” the R&D 100 Awards honor pioneers and their revolutionary ideas in science and technology.

ViDeoMAgic was invented by a Los Alamos team that included Yongchao Yang and Mascareñas. ViDeoMAgic (with the capital letters standing for Video-based Dynamic Measurement & Analysis) takes a video of a vibrating structure and from it extracts high-spatial-resolution (pixel-level) structural vibration/dynamics information. ViDeoMAgic applications cover the gamut of the imagination, from assessing aircraft and vehicle safety to detecting damage in large-scale infrastructure (buildings, bridges, and dams), providing noncontact analysis of sensitive machines such as wind turbines, performing quality control for manufacturing, and one day even helping physicians diagnose diseases such as cancer.

The future of augmented reality

Enhanced safety is just one benefit of using augmented reality for training. “Another benefit is expense,” Mascareñas says. “Real-world training exercises cost a lot to pull off, particularly when it comes to building custom environments and using props designed to enhance an actual scenario but also ensure safety. It doesn’t cost much, on the other hand, to create a three-dimensional hologram. Even the most primitive of these holograms get the point across, particularly when the environment itself is a real one.”

Another option is to use real-world props on which you could overlay holograms to make the scenario much more elaborate. For example, it would be possible to place a suitcase in the trunk of a car. The response team would have to find the most effective means of opening the trunk, removing the suitcase, and then opening it to find a virtual set of pipe bombs, all linked to a timer but also with infrared sensors for additional security. The vehicle and the suitcase are real, but the pipe bombs and the sensors are holograms.

Mascareñas notes that the original software prototype and the commercially available hardware are now ready, with the team actually capturing video of scenarios that show a real environment with simulated objects.

“This prototype software demonstrates that it is possible to use this technology for training,” Mascareñas says, “and that we’re pretty good at it.”

Although impressive today, the future of augmented reality begins to skirt the realm of science fiction.

“Five or ten years down the road, the holograms and virtual objects created by augmented reality will make the objects today look primitive,” Mascareñas says. “The technology is getting better every day, and we at Los Alamos are getting better with it every day, so the results 10 years from now will be mind-blowing.”