A Deep Learning Approach for Gloss Sign Language Translation using Transformer

Mohamed, Ammar; Hefny, Hesham; Amin, Mohamed

doi:10.21608/jocc.2022.254979

A Deep Learning Approach for Gloss Sign Language Translation using Transformer

Document Type : Original Article

Authors

¹ Faculty of Graduate Studies for Statistical Research Cairo University

² Department of Computer Science, Faculty of Graduate Studies for Statistical Research, Cairo University

³ Department of Computer Science, Faculty of Graduate Studies for Statistical Research, Cairo University, Cairo, Egypt

10.21608/jocc.2022.254979

Abstract

One of the most recent applications of machine learning is to translate sign language into natural language. Many studies have attempted to classify sign language based on whether it is gesture or facial expression. These efforts, however, ignore genuine sentences' linguistic structure and context. The quality of traditional translation methods is poor, and their underlying models are not salable. They also take a long time to complete. The contribution of this paper is that it suggests utilizing a transformer to perform bidirectional translation using a deep learning approach. The proposed models experiment on the ASLG-PC12 corpus. The experimental results reveal that the proposed models outperform other approaches to the same corpus in both directions of translation, with ROUGE and BLEU scores of 98.78% and 96.89%, respectively, when translating from text to gloss. Additionally, the results indicate that the model with two layers achieves the best result with ROUGE and BLEU scores of 96.90% and 84.82% when translating from gloss to text.

Keywords