Use of Key Points and Transfer Learning Techniques in Recognition of Handedness Indian Sign Language
Article Sidebar
Main Article Content
Abstract
The most expressive way of communication for individuals who have trouble speaking or hearing is sign language. Normal people are unable to comprehend sign language. As a result, communication barriers are put up. Majority of people are right-handed. Statistics say that, an average population of left-handed person in the world is about 10%, where they use left hand as their dominating hand. In case of hand written text recognition, if the text is written by left-handed or right-handed person, then there would not be any problem in recognition neither for human and nor for computer. But same thing is not true for sign language and its detection using computer. When the detection is performed using computer vision and if it falls into the category of detection by appearance, then it might not detect correctly. In machine and deep learning, if the model is trained using just one dominating hand, let’s say right hand, then the predictions can go wrong if same sign is performed by left-handed person. This paper addresses this issue. It takes into account the signs performed by any type of signer: left-handed, right-handed or ambidexter. In proposed work is on Indian Sign Language (ISL). Two models are trained: Model I, is trained on one dominating hand and Model II, is trained on both the hands. Model II gives correct predictions regardless of any type of signer. It recognizes alphabets and numbers in ISL. We used the concept of Key points and Transfer Learning techniques for implementation. Using this approach, models get trained quickly and we could achieve validation accuracy of 99%.
Article Details
References
Aiswarya V, Naren Raju N, Johanan Joy Singh S, Nagarajan T, Vijayalakshmi P “Hidden Markov model-based Sign Language to Speech Conversion System in TAMIL”, I2018 IEEE, 2018 4th ICBSII, 22-24 March 2018, Chennai.
L. Fernandes, P. Dalvi, A. Junnarkar, and M. Bansode, “Convolutional neural network based bidirectional sign language translation system,” in Proceedings of the 3rd, ICSSIT 2020, IEEE, Aug. 2020, pp. 769–775. doi: 10.1109/ICSSIT48917.2020.9214272.
S Yarisha Heera1, Madhuri K Murthy1, Sravanti V S1, Sanket Salvi2 “Talking Hands – An Indian Sign Language to Speech Translating Gloves” (ICIMIA 2017) 978-1-5090-5960-7/17/2017 IEEE
L. Quesada, G. López, and L. Guerrero, “Automatic recognition of the American sign language fingerspelling alphabet to assist people living with speech or hearing impairments,” J Ambient Intell Humaniz Comput, vol. 8, no. 4, pp. 625–635, Aug. 2017, doi: 10.1007/s12652-017-0475-7.
S. Sharma, R. Gupta, and A. Kumar, “Trbaggboost: an ensemble-based transfer learning method applied to Indian Sign Language recognition,” J Ambient Intell Humaniz Comput, vol. 13, no. 7, pp. 3527–3537, Jul. 2022, doi: 10.1007/s12652-020-01979-z.
Kshitij Bantupalli, Ying Xie “American Sign Language Recognition using Deep Learning and Computer Vision” , N. Abe, Institute of Electrical and Electronics Engineers, and IEEE Computer Society, 2018 IEEE International Conference on Big Data?: proceedings?: Dec 10 - Dec 13, 2018, Seattle, WA, USA.
Jiangbin Zheng, Yidong Chen, Chong Wu, Xiaodong Shi, Suhail Muhammad Kamal “Enhancing Neural Sign Language Translation by highlighting the facial expression information” Elsevier, Science Direct Neurocomputing 464 (2021) 462–472, Available online 20 August 2021
Y. Du, P. Xie, M. Wang, X. Hu, Z. Zhao, and J. Liu, “Full transformer network with masking future for word-level sign language recognition,” Neurocomputing, vol. 500, pp. 115–123, Aug. 2022, doi: 10.1016/j.neucom.2022.05.051.
L. Gao, H. Li, Z. Liu, Z. Liu, L. Wan, and W. Feng, “RNN-Transducer based Chinese Sign Language Recognition,” Neurocomputing, vol. 434, pp. 45–54, Apr. 2021, doi: 10.1016/j.neucom.2020.12.006.
Weizhe WANG2, Hongwu YANG, “Towards Realizing Sign Language to Emotional Speech Conversion by Deep Learning”, 2021 12th (ISCSLP) | 978-1-7281-6994-1/20/$31.00 ©2021 IEEE | DOI: 10.1109/ISCSLP49672.2021.9362060
Umang Patel, Aarti G. Ambekar “Moment Based Sign Language Recognition For Indian Languages” 2017 Third ICCUBEA, 978-1-5386-4008-1/17/2017 IEEE
W. Li, H. Pu, and R. Wang, “Sign Language Recognition Based on Computer Vision,” in 2021 IEEE, ICAICA 2021, , pp. 919–922. doi: 10.1109/ICAICA52286.2021.9498024.
Suharjito, Herman Gunawan, Narada Thiracitta, Ariadi Nugroho “Sign Language Recognition Using Modified Convolutional Neural Network Model”, The 1st 2018 INAPR, 978-1-5386-9422-0/18/2018 IEEE
R. Kumar, N. Bhardwaj, and N. S. Kumar, “An Optimum Approach to Indian Sign Language Recognition using Efficient Convolution Neural Networks,” in Proceedings - 2022 4th International Conference on Advances in Computing, Communication Control and Networking, ICAC3N 2022, Institute of Electrical and Electronics Engineers Inc., 2022, pp. 1563–1567. doi: 10.1109/ICAC3N56670.2022.10074173.
Tuan Linh Dang , Sy Dat Tran , Thuy Hang Nguyen , Suntae Kim , Nicolas Monet ,“An improved hand gesture recognition system using keypoints and hand bounding boxes”,Elsevier, Array 16 (2022) 100251
Mohidul Alam Laskara, Amlan Jyoti Dasb, Anjan Kumar Talukdarc, Kandarpa Kumar Sarma, “ Stereo Vision-based Hand Gesture Recognition under 3D Environment” ScienceDirect, Procedia Computer Science 58 ( 2015 ) 194 – 201, VisionNet 15
F. Pravin, D. Rajiv, ““HASTA MUDRA” An Interpretation of Indian Sign Hand Gestures”, 3rd International conference on Electronics Computer technology, vol. 2, pp. 377-380, 2011
Purva C. Badhe, Vaishali Kulkarni “Indian Sign Language Translator Using Gesture Recognition Algorithm”, 2015 IEEE International Conference on CGVIS
Malli Mahesh Chandra, Rajkumar S, Lakshmi Sutha Kumar “Sign Languages to Speech Conversion Prototype using the SVM Classifier”, 978-1-7281-1895-6/19/2019 IEEE
Aishwarya Sharma, Dr. Siba Panda, Prof. Saurav Verma “Sign Language to Speech Translation”, 11th ICCCNT 2020 July 1-3, 2020 - IIT – Kharagpur
V. Adithya and R. Rajesh, “Hand gestures for emergency situations: A video dataset based on words from Indian sign language,” Data Brief, vol. 31, Aug. 2020, doi: 10.1016/j.dib.2020.106016.
“Real-time Indian Sign Language (ISL) Recognition”, IEEE – 43488, 9th ICCCNT 2018 July 10-12, 2018, IISC, Bengaluru , India
Shravani K1, Sree Lakshmi A2, Sri GeethikaM3, Dr.Sapna B Kulkarni4 “ Indian Sign Language Character Recognition”, IOSR-JCE e-ISSN: 2278-0661,p-ISSN: 2278-8727, Volume 22, Issue 3, Ser. I (May - June 2020), PP 14-19
“Indian Sign Language Recognition System”, International Journal of Engineering and Technology · July 2017, DOI: 10.21817/ijet/2017/v9i3/170903S030 ISSN (Online) : 0975-4024 Yogeshwar I. Rokade et al. / International Journal of Engineering and Technology (IJET)
S. Katoch, V. Singh, and U. S. Tiwary, “Indian Sign Language recognition system using SURF with SVM and CNN,” Array, vol. 14, Jul. 2022, doi: 10.1016/j.array.2022.100141.
P. K. Athira, C. J. Sruthi, and A. Lijiya, “A Signer Independent Sign Language Recognition with Co-articulation Elimination from Live Videos: An Indian Scenario,” Journal of King Saud University - Computer and Information Sciences, vol. 34, no. 3, pp. 771–781, Mar. 2022, doi: 10.1016/j.jksuci.2019.05.002.
Purva Chaitanya Badhe, Vaishali Kulkarn ,“Artificial Neural Network based Indian Sign Language Recognition using hand crafted features”, IEEE – 49239, 11th ICCCNT 2020 July 1-3, 2020 - IIT – Kharagpur
K. Shenoy, T. Dastane, V. Rao, and D. Vyavaharkar, “Real-time Indian Sign Language (ISL) Recognition.” IEEE – 43488, 9th ICCCNT 2018 July 10-12, 2018, IISC, Bengaluru Bengaluru, India
Shahnaj Fatima, Ambuj Agarwal, Ambuj Agarwal” Different Approaches to Convert Speech into Sign Language” 978-9-3805-4421-2/16/ 2016 IEEE
A. Ardiansyah, B. Hitoyoshi, M. Halim, N. Hanafiah, and A. Wibisurya, “Systematic Literature Review: American Sign Language Translator,” in Procedia Computer Science, Elsevier B.V., 2021, pp. 541–549. doi: 10.1016/j.procs.2021.01.038.
I.A.Adeyanju , O.O. Bello , M.A. Adegboye, “ Machine learning methods for sign language recognition: A critical review and analysis”, Elsevier, ScienceDirect Intelligent Systems with Applications 12 (2021) 20 0 056
Yuraj Grover, Riya Aggaraval, Deepak Sharma, Prashant K. Gupta “Sign Language Translation systems for Hearing/Speech Impaired People: A review” 2021 International Conference on Innovative Practices in Technology and Management (ICIPTM) | 978-1-6654-2530-8/21/2021 IEEE | DOI: 10.1109/ICIPTM52218.2021.9388330