Earlier this month Facebook Reality Labs (FRL) published a paper, with an accompanying video, about the real-time tracking of human hands. Hand tracking for VR is expected to make a major difference to the quality of immersion when it becomes suitably adept. However, at this time users are better of using the specially designed controllers rather than suffering with hand tracking hardware that is limited in its ability which thus lessens the VR experience.
With Facebook's interest in VR though Oculus and so on it isn't surprising to see it working on better VR environment to human interaction. The efforts detailed today focus on the solving of a big problem with previous hand tracing tech – when your hands overlap, touch, or fingers are self-occluded. The FRL team has leve raged recent advances in vision-based tracking and physically based animation during its research and development. Now FRL claims that it has developed "the first algorithm capable of tracking high-fidelity hand deformations through highly self-contacting and self-occluding hand gestures, for both single hands and two hands".
Common hand interactions introduce many self-interactions and massive self occlusions, says the research team. However, it managed to create a robust algorithm by "constraining a vision-based tracking algorithm with a physically based deformable model". The moel has been tested against some of the most complicated interaction of human hands, and you can see the tracking quality and accuracy in the embedded video.
Click to zoom image
So, can we expect an Oculus branded camera setup to deliver this tech to VR-adventurers in the near future? From checking through the research paper it doesn't look like this level of hand tracking tech will be available for consumers very soon. The dedicated hand tracking hardware setup sounds rather beefy to be reduced to the consumer space anytime soon as, for accurate 30fps tracking, it needed; 124 cameras, six threads on a 2.2GHz Intel E5-2698 Xeon processor and an Nvidia Tesla V100 GPU. I note that various camera arrays were tested, from two upwards, and a 43 camera setup also fared rather well.
