Interaction Detection with Depth Sensing and Body Tracking Cameras in Physical Rehabilitation

Summary

Introduction: This article is part of the Focus Theme of Methods of Information in Medicine on “Methodologies, Models and Algorithms for Patients Rehabilitation”. Objectives: This paper presents a camera based method for identifying the patient and detecting interactions between the patient and the therapist during therapy. Detecting interactions helps to discriminate between active and passive motion of the patient as well as to estimate the accuracy of the skeletal data. Methods: Continuous face recognition is used to detect, recognize and track the patient with other people in the scene (e.g. the therapist, or a clinician). We use a method based on local binary patterns (LBP). After identifying users in the scene we identify interactions between the patient and other people. We use a depth map/point cloud for estimating the distance between two people. Our method uses the association of depth regions to user identities and computes the minimal distance between the regions. Results: Our results show state-of-the-art performance of real-time face recognition using low-resolution images that is sufficient to use in adaptive systems. Our proposed approach for detecting interactions shows 91.9% overall recognition accuracy what is sufficient for applications in the context of serious games. We also discuss limitations of the proposed method as well as general limitations of using depth cameras for serious games. Conclusions: We introduced a new method for frame-by-frame automated identification of the patient and labeling reliable sequences of the patient’s data recorded during rehabilitation (games). Our method improves automated rehabilitation systems by detecting the identity of the patient as well as of the therapist and by detecting the distance between both over time.

Keywords

User identification - interaction detection - rehabilitation - face recognition

• References

• 1 Chang CY, Lange B, Zhang M, Koenig S, Requejo P, Somboon N. et al. Towards pervasive physical rehabilitation using Microsoft Kinect. Pervasive Computing Technologies for Healthcare (PervasiveHealth), 2012. 6th International Conference on 2012 pp 159-162.
• 2 Chang YJ, Chen SF, Huang JD. A Kinect-based system for physical rehabilitation: A pilot study for young adults with motor disabilities. Res Dev Disabil 2011; 32 (Suppl. 06) 2566-2570.
• 3 Clark RA, Pua YH, Bryant AL, Hunt MA. Validity of the Microsoft Kinect for providing lateral trunk lean feedback during gait retraining. Gait Posture (Internet) 2013; 38 (Suppl. 04) 1064-1066 Available from: http://www.sciencedirect.com/science/article/pii/S0966636213001884
• 4 Van den Audenaeren L, Vanden Abeele V, Geurts L, Husson J, Annema J-H, Desmet S. Boneshaker. Proceedings of the 2012 ACM annual conference extended abstracts on Human Factors in Computing Systems Extended Abstracts – CHI EA ’12 (Internet). New York, NY, USA: ACM Press; 2012. (cited 2014 Oct 23). p 2447 Available from: http://dl.acm.org/citation.cfm?id=2212776.2223817
• 5 Ahonen T, Hadid A, Pietikäinen M. Face recognition with local binary patterns. Pajdla T, Matas J. editors. Comput Vision-ECCV 2004 (Internet). Springer; Berlin/Heidelberg: 2004. (cited 2010 Nov 28); 3021 469-481. Available from: http://www.springerlink.com/index/P5D9XP9GFKEX5GK9.pdf
• 6 Turk M, Pentland A. Eigenfaces for recognition. J Cogn Neurosci (Internet) 1991; (cited 2010 Sep 25); 3 (Suppl. 01) 71-86. Available from: http://portal.acm.org/citation.cfm?id=1326894#
• 7 Grother PJ, Quinn GW, Phillips PJ. Report on the Evaluation of 2D Still-Image Face Recognition Algorithms. NIST Interagency Report 7709, Multiple-Biometric Evaluation (MBE) 2010. 2010
• 8 Zhu Z, Luo P, Wang X, Tang X. Deep Learning Identity-Preserving Face Space. Computer Vision (ICCV), 2013 IEEE International Conference on (Internet). 2013 (cited 2014 Sep 15). pp 113-120. Available from: http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=6751123
• 9 Chen D, Cao X, Wen F, Sun J. Blessing of dimensionality: High-dimensional feature and its efficient compression for face verification. CVPR ’13 Proceedings of the 2013 IEEE Conference on Computer Vision and Pattern Recognition Pages (Internet). 2013 (cited 2014 Sep 15). pp 3025-3032. Available from: http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=6619233
• 10 Poppe R. A survey on vision-based human action recognition. Image Vis Comput 2010; 28 (Suppl. 06) 976-990.
• 11 Yun K, Honorio J, Chattopadhyay D, Berg TL, Samaras D. Two-person interaction detection using body-pose features and multiple instance learning. 2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (Internet). IEEE; 2012 (cited 2014 Jul 25). pp 28-35. Available from: http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=6239234
• 12 Ban J, Pavlovicova J, Feder M, Omelina L, Oravec M. Face recognition methods for multimodal interface. Wireless and Mobile Networking Conference (WMNC), 2012 5th Joint IFIP 2012 pp 110-113.
• 13 Oravec M, Pavlovičová J, Mazanec J, Omelina Ľ, Féder M, Ban J. Efficiency of Recognition Methods for Single Sample per Person Based Face Recognition. Reviews, Refinements and New Ideas in Face Recognition. Rijeka: InTech; 2011. pp 181-206.
• 14 Ban J, Feder M, Jirka V, Loderer M, Omelina L, Oravec M. et al. An Automatic Training Process Using Clustering Algorithms for Face Recognition System. Proceedings ELMAR-2013: 55th International Symposium Zadar Croatia: 2013 pp 15-18.
• 15 Omelina L, Jansen B. Serious games for physical rehabilitation: designing highly configurable and adaptable games. Proc 9th Intl Conf Disability, Virtual Reality & Associated Technologies (Internet). Laval, France: 2012 (cited 2014 Sep 9). pp 195-201. Available from: http://www.icdvrat.reading.ac.uk/2012/papers/ICDVRAT2012_S06N5_Omelina_etal.pdf
• 16 Bonnechère B, Jansen B, Omelina L, Da Silva L, Mouraux D, Rooze M. et al. Patient follow-up using Serious Games. A feasibility study on low back pain patients. Proceedings of the 3rd European conference on gaming and playful interaction in health care 2013 pp 185-195.