One of the most difficult VR haptic problems at the moment is the development of an uncomplicated and easily deployable full-body tracking solution for end users. Facebook is trying a new approach using electromagnetic sensors.
Virtual reality is already being put into multiple applications such as in remote presence, communication, collaboration, entertainment, healthcare and medicine among others. Various forms of virtual reality interactions have been developed over the years. One aspect of an immersive AR/VR experience that is yet to gain traction is the development and deployment of a solution for accurately reconstructing the user’s full-body pose.
Consumer cameras already feature pose estimation with varying levels of sophistication. However, the requirement for external cameras has an inherent limitation in the user’s mobility: if you move out of the camera’s field of view, you are no longer tracked. On the other hand, cameras built into virtual reality headsets are not an optimal solution for full-body tracking as the body often blocks itself or might not even be visible in the first place.
A team of researchers from Facebook Reality Labs and ETH Zurich are therefore trying out the use of electromagnetic sensors in full-body tracking. These sensors do not require visual contact. Such body-worn sensors provide the promise of overcoming the limitations of the external cameras and built-in cameras in the headset.
Full-Body Tracking: The Fewer the Sensors the Better
The use of electromagnetic (EM)-based sensing requires a base station that generates an electromagnetic reference field as well as a number of electromagnetic (EM) sensors whose distance from the base station and orientation can be measured precisely. For the research work, the team developed its own tracking system using light and compact EM sensors.
The tests were performed once with six sensors and once with twelve sensors. Fewer sensors means ease of use but this has a bearing on the accuracy of the full-body tracking which decreases with fewer sensors.
The researchers state that developing your own electromagnetic tracking system is a challenge and deriving the poses from the measurement data is also another serious challenge. If you only use six EM sensors, the Artificial Intelligence (AI) model has to close the gaps in measurement data. The spatial position of the forearms, for instance, is then no longer measured.
Little Deviation with Six Sensors
To overcome this, the team built a two-part solution. First, the pose is derived on the basis of the training data then the accuracy is optimized.
The researchers are satisfied with the results. With the use of 12 sensors, the accuracy of the EM sensors deviates by a maximum of 31.8mm and 13.3 degrees. With six sensors, it deviates only slightly more by 35.4mm and 14.9 degrees.
The video below shows full body electromagnetic tracking in action. In the right image section, the avatar, including the derived poses was placed directly over the test subject so that the real position and movement can be compared with the reconstructed one.
EM-Based Tracking
EM-based tracking is not a new concept and has been around for decades. There were military applications of EM sensing as early as in the 1960s. EM sensing entails the use of a source that emits an electromagnetic field where the sensor determines its orientation and position relative to the source.
There are a number of existing EM-based tracking systems currently in the market. They differ based on factors such as the update rate, supported tracking range as well as in the form factor of the hardware.
For its specific user case (which is in full body tracking), the team found that the current solutions aren’t well suited because they either feature large sensors or they use tethered sensors that limit mobility. The custom EM tracking system that the team developed utilizes up to 12 small, wireless EM sensors tat have been tuned to work in the 0.3-to-1.0-meter range around the source. This enabled the research team to mount a total of 12 sensors on the human body as shown in the images above.
For a typical usage scenario, the accuracy of a single sensor is in the range of 1cm positional and 2-3 degrees angular error in contrast to optical marker-based tracking. However, in contrast to optical tracking, the EM-based sensors do not feature any line-of-sight constraints.
Body-worn sensors can overcome this limitation but they also have their own sets of challenges. Facebook Reality Labs and ETH Zurich’s project tested the use of EM-based sensing using 6-12 customized wireless EM sensors without any line-of-sight constraints.
Quelle:
