Automatic Sign Language Recognition Using Colour Image Representation of Dense Optical Flow and Multi-Cluster Optical Flow Map
Published in PhD thesis, School of Engineering, University of Warwick, 2024
Supervisor: Prof. Tardi Tjahjadi
Abstract: Sign language is the natural language of deaf people. There are many sign languages around the world because, just like spoken languages, each region has its own sign language. Automatic Sign Language Recognition (ASLR) is a system for automatically translating the sign into its understandable meaning. This thesis aims to develop an ASLR method that uses standard RGB video as an input. Being able to recognise sign language from a standard RGB camera makes ASLR easier to use in a real world setting over the other modalities like depth camera that is less available or wearable device that needs user cooperation. However, it has more challenges, particularly in extracting hand motion features that convey most sign language information. The thesis is divided into two main parts. The first part is the development of a hand tracking algorithm, and the second part is the development of an ASLR system. The contributions of this thesis lie in the proposed hand feature extraction methods and feature representation for hand tracking and ASLR system from the combination of motion and colour data from standard video camera. In addition, the proposed hand tracking algorithm performs multi-object tracking which tracks both of the signer’s hands simultaneously. The proposed ASLR algorithm performs recognition through a novel spatio-temporal feature that is constructed from a multicluster optical flow map. The hand tracking results are used to enhance the spatiotemporal feature used in the ASLR system. The proposed methods are tested using publicly available sign language datasets.
The full thesis is available from the Warwick Research Archive Portal (WRAP).
Recommended citation: Jawas, N. (2024). "Automatic Sign Language Recognition Using Colour Image Representation of Dense Optical Flow and Multi-Cluster Optical Flow Map." PhD thesis, University of Warwick. https://wrap.warwick.ac.uk/id/eprint/190559/
Download Paper