Transformer-Based Model Enables Real-Time Hand Gesture Recognition in OpenXR

Summary

The study develops a real-time hand gesture recognition system for human-computer interaction using hand-tracking data captured through the OpenXR standard in Unity. It trains a custom Transformer-based machine-learning model on positional data from hand joints and wrist rotation angles. By modeling short sequences of gesture data, the Transformer captures temporal dependencies and classifies gestures across different hand orientations and sizes. The reported results show a significant improvement in gesture classification accuracy. The work is relevant to applications including gaming, virtual reality, robotics, and other intuitive interfaces. As future work, the authors propose detecting the flow of movement and transitions between individual gestures, extending recognition beyond isolated gesture classes.