Sign Language Recognition Utilizing LSTM & Media pipe for Dynamic Gestures of ISL
##plugins.themes.bootstrap3.article.main##
Abstract
Humans, in general, are social creatures who communicate themselves through an assortment of spoken languages. Deaf and Mute individuals converse in a manner that's comparable, however many others areignorantoftheirsignlanguage. Asaresult, thereisaneedtodevelopasystemthatfacilitatescommunication among the hearing and hard-of-hearing communities. This research offers a real-time Indian Sign Language (ISL) recognition system for 24 dynamic signals using the Mediapipe framework and an LSTM network. The method proposed in the study involves training a LSTM to differentiate between different signs using a dataset created of 24 dynamic gesture signs.To accomplish dataset creation, a pre-trained Holistic model of the Mediapipe framework is used as a feature extractor. Theresultsof thestudy demonstrate that theaboveapproachachieves 97%testaccuracy.
##plugins.themes.bootstrap3.article.details##
Section
Articles
References
[1]. D. S, K. H. K B, A. M, S. M, D. S and K. V, "An Efficient Approach for Interpretation of IndianSign Language using Machine Learning," 2021 3rd International Conference on Signal ProcessingandCommunication(ICPSC),Coimbatore,India,2021,pp.130-133,doi:10.1109/ICSPC51351.2021.9451692.
[2]. K.Shenoy,T.Dastane,V.RaoandD.Vyavaharkar,"Real-timeIndianSignLanguage(ISL)Recognition," 2018 9th International Conference on Computing, Communication and Networking Technologies(ICCCNT), Bengaluru, India,2018, pp.1-9,doi:10.1109/ICCCNT.2018.8493808.
[3]. P. C. Badhe and V. Kulkarni, "Artificial Neural Network based Indian Sign Language Recognitionusinghandcraftedfeatures,"202011thInternationalConferenceonComputing,CommunicationandNetworkingTechnologies(ICCCNT),Kharagpur,India,2020,pp.1-6,doi:10.1109/ICCCNT49239.2020.9225294.
[4]. A. Wadhwan, and P. Kumar, “Sign Language Recognition Systems: A Decade Systematic LiteratureReview”,ArchivesofComputationalMethodsinEngineering,Springer,2019,DOI:https://doi.org/10.1007/s11831-019-09384-2
[5]. D. R. Kothadiya, C. M. Bhatt, T. Saba, A. Rehman and S. A. Bahaj, "SIGNFORMER: DeepVisionTransformer for Sign Language Recognition," in IEEE Access, vol. 11, pp. 4730-4739, 2023, doi:10.1109/ACCESS.2022.3231130.
[6]. K. Han, Y. Wang, H. Chen, X. Chen, J. Guo, Z. Liu, Y. Tang, A. Xiao, C. Xu, Y. Xu, Z. Yang, Y.Zhang, and D. Tao, ‘‘A survey on vision transformer,’’ IEEE Trans. Pattern Anal. Mach. Intell., vol.45,no. 1, pp. 87–110, Jan. 2023.
[7]. M. Al-Qurishi, T. Khalid and R. Souissi, "Deep Learning for Sign Language Recognition: CurrentTechniques, Benchmarks, and Open Issues," in IEEE Access, vol. 9, pp. 126917-126951, 2021, doi:10.1109/ACCESS.2021.3110912.
[8]. S.Stoll, N.C.Camgoz, S.Hadfield, and R.Bowden,‘‘ Text 2sign:Towardssignlanguageproduction using neural machine translation and generative adversarial networks,’’ Int. J. Comput.Vis., vol. 128, no. 4, pp. 1–18, 2020.
[9]. Jebali, M., Dakhli, A.& Jemni, M. “Vision-basedcontinuous sign language recognition usingmultimodalsensorfusion,”EvolvingSystems,vol12,1031–1044(2021).https://doi.org/10.1007/s12530-020-09365-y
[10]. Wenwen Yang, Jinxu Tao, Zhongfu Ye, “Continuous sign language recognition using level buildingbased on fast hidden Markov model, Pattern Recognition Letters, vol 78, pp 28–35, ISSN 0167-8655,https://doi.org/10.1016/j.patrec.2016.03.030.
[11]. I. Papastratis, K.Dimitropoulos, D.Konstantinidis and P.Daras," Continuous Sign Language Recognition Through Cross-Modal Alignment of Video and Text Embeddings in a Joint-LatentSpace,"inIEEEAccess,vol. 8,pp. 91170-91180,2020, doi:10.1109/ACCESS.2020.2993650.
[12]. O. Koller, J.Forster, and H.Ney,‘‘ Continuous sign languagerecognition:Towardslargevocabularystatisticalrecognitionsystemshandlingmultiplesigners,’’Comput.Vis.ImageUnderstand.,vol. 141, pp. 108–125, Dec. 2015.
[13]. E.Rajalakshmietal.," Multi-Semantic Discriminative Feature LearningforSignGestureRecognition Using Hybrid Deep Neural Architecture," in IEEE Access, vol. 11, pp. 2226-2238,2023,doi: 10.1109/ACCESS.2022.3233671.
[14]. X. Jiang, M. Lu, and S.-H. Wang, ‘‘An eight-layer convolutional neural network with stochasticpooling, batch normalization and dropout for fingerspelling recognition of Chinese sign language,’’MultimediaTools Appl.,vol.79, nos. 21–22, pp. 15697–15715, Jun. 2020.
[15]. Z. Wangetal.,"HearSignLanguage:AReal-TimeEnd-to-EndSignLanguageRecognitionSystem," in IEEE Transactions on Mobile Computing, vol. 21, no. 7, pp. 2398-2410, 1 July 2022,doi:10.1109/TMC.2020.3038303.
[16]. G. Marin, F. Dominio, and P. Zanuttigh, “Hand gesture recognition with leap motion and kinectdevices,”in Proc.IEEEInt.Conf. ImageProcess.,2014,pp. 1565–1569.
[17]. E. Esco be do, L.Ramirez and G. Camara," Dynamic SignLanguageRecognitionBasedonConvolutionalNeuralNetworksandTextureMaps,"201932ndSIBGRAPIConferenceonGraphics,PatternsandImages(SIBGRAPI),RiodeJaneiro,Brazil,2019,pp.265-272,doi:10.1109/SIBGRAPI.2019.00043.
[18]. H. Bilen, B. Fernando, E. Gavves, A. Vedaldi, and S. Gould, “Dynamic image networks for actionrecognition,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,2016, pp.3034–3042.
[19]. N. Vasani, P. Autee, S. Kalyani and R. Karani, "Generation of Indian sign language by sentenceprocessing and generative adversarial networks," 2020 3rd International Conference on IntelligentSustainableSystems(ICISS),Thoothukudi,India,2020,pp.1250-1255,doi:10.1109/ICISS49785.2020.9315979.
[20]. Stoll, S., Camgöz, N. C., Hadfield, S., & Bowden, R. (2018, September). “Sign language productionusing neural machine translation and generative adversarial networks.” In Proceedings of the 29thBritishMachine Vision Conference(BMVC 2018). BritishMachine Vision Association.
[21]. P. V. V. Kishore, A. S. C. S. Sastry and A. Kartheek, "Visual-verbal machine interpreter for signlanguage recognition under versatile video backgrounds," 2014 First International Conference onNetworks&SoftComputing(ICNSC2014),Guntur,India,2014,pp.135-140,doi:10.1109/CNSC.2014.6906696.
[22]. A.K.Talukdar and M.K.Bhuyan," Vision-Based Continuous Sign LanguageSpottingUsingGaussian Hidden Markov Model," in IEEE Sensors Letters, vol. 6, no. 7, pp. 1-4, July 2022, Art no.6002304,doi: 10.1109/LSENS.2022.3185181.
[23]. P.Xie, M.Zhaoand X.Hu," PiSLTRc: Position-Informed Sign Language Transformer With Content Aware Convolution," in IEEE Transactions on Multimedia, vol. 24, pp. 3908-3919, 2022,doi:10.1109/TMM.2021.3109665.
[24]. J. Zhang, W. Zhou,X. Chao, J. Pu, and H. Li, “Chinese sign language recognition with adaptiveHMM,”in Proc.IEEEInt.Conf. MultimediaExpo., 2016, pp.1–6.
[25]. J. Zhao, W. Qi, W. Zhou, N. Duan, M. Zhou and H. Li, "Conditional Sentence Generation andCross-Modal Reranking for Sign Language Translation," in IEEE Transactions on Multimedia, vol.24,pp. 2662-2672, 2022, doi: 10.1109/TMM.2021.3087006.
[26]. F. Obaid, A. Babadi, and A. Yoosofan, “Hand Gesture Recognition in Video Sequences Using DeepConvolutionalandRecurrentNeuralNetworks,”Appl.Comput.Syst.,vol.25,no.1,pp.57–61,May2020, doi: 10.2478/acss-2020-0007.
[2]. K.Shenoy,T.Dastane,V.RaoandD.Vyavaharkar,"Real-timeIndianSignLanguage(ISL)Recognition," 2018 9th International Conference on Computing, Communication and Networking Technologies(ICCCNT), Bengaluru, India,2018, pp.1-9,doi:10.1109/ICCCNT.2018.8493808.
[3]. P. C. Badhe and V. Kulkarni, "Artificial Neural Network based Indian Sign Language Recognitionusinghandcraftedfeatures,"202011thInternationalConferenceonComputing,CommunicationandNetworkingTechnologies(ICCCNT),Kharagpur,India,2020,pp.1-6,doi:10.1109/ICCCNT49239.2020.9225294.
[4]. A. Wadhwan, and P. Kumar, “Sign Language Recognition Systems: A Decade Systematic LiteratureReview”,ArchivesofComputationalMethodsinEngineering,Springer,2019,DOI:https://doi.org/10.1007/s11831-019-09384-2
[5]. D. R. Kothadiya, C. M. Bhatt, T. Saba, A. Rehman and S. A. Bahaj, "SIGNFORMER: DeepVisionTransformer for Sign Language Recognition," in IEEE Access, vol. 11, pp. 4730-4739, 2023, doi:10.1109/ACCESS.2022.3231130.
[6]. K. Han, Y. Wang, H. Chen, X. Chen, J. Guo, Z. Liu, Y. Tang, A. Xiao, C. Xu, Y. Xu, Z. Yang, Y.Zhang, and D. Tao, ‘‘A survey on vision transformer,’’ IEEE Trans. Pattern Anal. Mach. Intell., vol.45,no. 1, pp. 87–110, Jan. 2023.
[7]. M. Al-Qurishi, T. Khalid and R. Souissi, "Deep Learning for Sign Language Recognition: CurrentTechniques, Benchmarks, and Open Issues," in IEEE Access, vol. 9, pp. 126917-126951, 2021, doi:10.1109/ACCESS.2021.3110912.
[8]. S.Stoll, N.C.Camgoz, S.Hadfield, and R.Bowden,‘‘ Text 2sign:Towardssignlanguageproduction using neural machine translation and generative adversarial networks,’’ Int. J. Comput.Vis., vol. 128, no. 4, pp. 1–18, 2020.
[9]. Jebali, M., Dakhli, A.& Jemni, M. “Vision-basedcontinuous sign language recognition usingmultimodalsensorfusion,”EvolvingSystems,vol12,1031–1044(2021).https://doi.org/10.1007/s12530-020-09365-y
[10]. Wenwen Yang, Jinxu Tao, Zhongfu Ye, “Continuous sign language recognition using level buildingbased on fast hidden Markov model, Pattern Recognition Letters, vol 78, pp 28–35, ISSN 0167-8655,https://doi.org/10.1016/j.patrec.2016.03.030.
[11]. I. Papastratis, K.Dimitropoulos, D.Konstantinidis and P.Daras," Continuous Sign Language Recognition Through Cross-Modal Alignment of Video and Text Embeddings in a Joint-LatentSpace,"inIEEEAccess,vol. 8,pp. 91170-91180,2020, doi:10.1109/ACCESS.2020.2993650.
[12]. O. Koller, J.Forster, and H.Ney,‘‘ Continuous sign languagerecognition:Towardslargevocabularystatisticalrecognitionsystemshandlingmultiplesigners,’’Comput.Vis.ImageUnderstand.,vol. 141, pp. 108–125, Dec. 2015.
[13]. E.Rajalakshmietal.," Multi-Semantic Discriminative Feature LearningforSignGestureRecognition Using Hybrid Deep Neural Architecture," in IEEE Access, vol. 11, pp. 2226-2238,2023,doi: 10.1109/ACCESS.2022.3233671.
[14]. X. Jiang, M. Lu, and S.-H. Wang, ‘‘An eight-layer convolutional neural network with stochasticpooling, batch normalization and dropout for fingerspelling recognition of Chinese sign language,’’MultimediaTools Appl.,vol.79, nos. 21–22, pp. 15697–15715, Jun. 2020.
[15]. Z. Wangetal.,"HearSignLanguage:AReal-TimeEnd-to-EndSignLanguageRecognitionSystem," in IEEE Transactions on Mobile Computing, vol. 21, no. 7, pp. 2398-2410, 1 July 2022,doi:10.1109/TMC.2020.3038303.
[16]. G. Marin, F. Dominio, and P. Zanuttigh, “Hand gesture recognition with leap motion and kinectdevices,”in Proc.IEEEInt.Conf. ImageProcess.,2014,pp. 1565–1569.
[17]. E. Esco be do, L.Ramirez and G. Camara," Dynamic SignLanguageRecognitionBasedonConvolutionalNeuralNetworksandTextureMaps,"201932ndSIBGRAPIConferenceonGraphics,PatternsandImages(SIBGRAPI),RiodeJaneiro,Brazil,2019,pp.265-272,doi:10.1109/SIBGRAPI.2019.00043.
[18]. H. Bilen, B. Fernando, E. Gavves, A. Vedaldi, and S. Gould, “Dynamic image networks for actionrecognition,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,2016, pp.3034–3042.
[19]. N. Vasani, P. Autee, S. Kalyani and R. Karani, "Generation of Indian sign language by sentenceprocessing and generative adversarial networks," 2020 3rd International Conference on IntelligentSustainableSystems(ICISS),Thoothukudi,India,2020,pp.1250-1255,doi:10.1109/ICISS49785.2020.9315979.
[20]. Stoll, S., Camgöz, N. C., Hadfield, S., & Bowden, R. (2018, September). “Sign language productionusing neural machine translation and generative adversarial networks.” In Proceedings of the 29thBritishMachine Vision Conference(BMVC 2018). BritishMachine Vision Association.
[21]. P. V. V. Kishore, A. S. C. S. Sastry and A. Kartheek, "Visual-verbal machine interpreter for signlanguage recognition under versatile video backgrounds," 2014 First International Conference onNetworks&SoftComputing(ICNSC2014),Guntur,India,2014,pp.135-140,doi:10.1109/CNSC.2014.6906696.
[22]. A.K.Talukdar and M.K.Bhuyan," Vision-Based Continuous Sign LanguageSpottingUsingGaussian Hidden Markov Model," in IEEE Sensors Letters, vol. 6, no. 7, pp. 1-4, July 2022, Art no.6002304,doi: 10.1109/LSENS.2022.3185181.
[23]. P.Xie, M.Zhaoand X.Hu," PiSLTRc: Position-Informed Sign Language Transformer With Content Aware Convolution," in IEEE Transactions on Multimedia, vol. 24, pp. 3908-3919, 2022,doi:10.1109/TMM.2021.3109665.
[24]. J. Zhang, W. Zhou,X. Chao, J. Pu, and H. Li, “Chinese sign language recognition with adaptiveHMM,”in Proc.IEEEInt.Conf. MultimediaExpo., 2016, pp.1–6.
[25]. J. Zhao, W. Qi, W. Zhou, N. Duan, M. Zhou and H. Li, "Conditional Sentence Generation andCross-Modal Reranking for Sign Language Translation," in IEEE Transactions on Multimedia, vol.24,pp. 2662-2672, 2022, doi: 10.1109/TMM.2021.3087006.
[26]. F. Obaid, A. Babadi, and A. Yoosofan, “Hand Gesture Recognition in Video Sequences Using DeepConvolutionalandRecurrentNeuralNetworks,”Appl.Comput.Syst.,vol.25,no.1,pp.57–61,May2020, doi: 10.2478/acss-2020-0007.