Automatic Sign Language Interpreter in Court
Ref. No. SSTCRC018
Project acronym: ASLIC
Abstract:
Perceptions and needs on Assistive Technologies ( Ats) vary according to the type of disability. This variety of perceptions and needs on ATs affects not only concrete technological solutions and fields of applications but also specific measures for a more effective use of devices. Furthermore, the labelling of ATs as medical or mainstream device is still contested. This has consequences not only in terms of the perceived ‘value’ of a device but also in terms of regulative measures and accessibility to them.
Out of its keen interest in materializing the Saudi Vision 2030, especially with regard to the deaf, the Ministry of Health (MOH) has prompted its staff to uphold ‘We Are with You’ Initiative, meant to support this group throughout the Kingdom. To that end, the Ministry has raised its staff’s awareness of the deaf community, as well as of the ways to deal with them, and trained them on sign language rules and basics. This is an important issue given the large number of deaf people. In Saudi Arabia the number of people with Hearing Impairment is 289.355.
Since interpret can counterfeit and translate subjectively deafness person sayings, the need of automatic interpretation system is very important. In this project we propose to develop an automatic sign language interpreter that be used in court to have reliable translation of deafness person sayings and testimonies. This system helps, considerably, deafness and auditory impairment to get their rights.
Keywords: Sign Language, Automatic interpreter, Court Interpreter, 2030 Vision, Assistive Technologies, Deep Learning, AI.
What problem or which question is to be solved/answered? Which idea or vision is to be tested
or further developed? What is the hypothesis?
The purpose of the project:
The goal of this project is to Develop an automatic sign language interpreter that be used in court to have reliable translation of deafness person sayings and testimonies. Since interpret can counterfeit and translate subjectively deafness person sayings, the need of automatic interpretation system is very important. This system helps, considerably, deafness and auditory impairment to get their rights.
Consequently, our project focuses on family and legal thematic sign language. The proposed system can also be used by parents in order to interact with their children at home at an early age before going to school.
There are two types of systems:
1. Device based system which uses sensors, gloves…
2. Vision based system using AI.
The second type is easier to use.
There are three types of sign language recognition:
1. Alphabet sign language recognition: based on static hand/head gestures (images)
2. Isolated word sign language recognition: based on dynamic gestures using hand and head
3. Sentence sign language: based on continuous gestures
We propose to develop the second type which is more practical and interesting. So, the proposed system consists on capturing video of deafness people and tracking hands and face sign in order to translate/recognize them to words.
Rothley the project context is developing assistive technologies and modern technologies for people with disabilities and studying the effect of their use in various aspects (such as education, training, and employment).
Background information:
This is an important issue given the large number of deaf people. According to the world Health Organization (WHO), the deaf and hard-hearing community forms around 6.1% of world’s population in 2018. In Saudi Arabia the number of people with Hearing Impairment is 289.355
Only to compare with other countries, California has one sign language interpreter for every 46 hearing impaired people however, Saudi Arabia has one for approximately every 93,000 (statistics in 2014).
In many other countries and thanks to today’s technology, anyone with an iPhone, Android smartphone or tablet can learn SL basics or practice its techniques. There is a list of popular applications. But they all use foreign sign language.
Analysis of the actual situation shows that:
· Little research exists about Saudi deaf education and Saudi Deaf culture. SASL still is not recognized as a true language.
· Saudi deaf individuals face many challenges in their life, such as language deprivation, poor quality education, and a poor work environment.
· Today, there are at least 15 deaf schools in Saudi Arabia; none of those schools have deaf teachers, administrators, and staff, which explicitly show the absence of deaf professionals and deaf role models in Saudi deaf education. During the first 20 years of teaching deaf students, the Saudi Ministry of Education hired teachers from other Arab countries. The hearing teachers from Jordan and Egypt created significant impacts on the nature of SASL and Saudi Deaf culture by using their country’s sign language and culture
· Unlike Wolsey, who investigated the necessary components of a successful deaf and hearing research partnership in the US, there are few deaf researchers in Saudi Arabia who can conduct research from a Deaf-centric perspective because Saudi universities did not accept deaf students into their undergraduate and graduate programs.
Case of study: deafness and auditory impairment assistance
The interpreter is bound by confidentiality and the interpreter’s sole task is to provide a correct interpretation, both linguistically and culturally.
The sign language interpreter must be qualified to work in legal settings. This can be verified through the national registry of legal interpreters of the country, or another similar national system qualifying the interpreters.
Many countries have national legislation stating that any deaf, hard of hearing and deafblind persons who use sign language are entitled to a professional and qualified sign language interpreter in criminal proceedings.
For example, In France, a litigant suffering from deafness has the possibility to appear at the hearing assisted by a person who know the sign language allowing communication with deaf people or equipped with a technical device ensuring this communication. However, in Saudi Arabia, they invite deaf people to come accompanied by a person of their choice able to provide the translation. In fact, there are no efforts and interests in preserving the rights of people with disabilities. So, they can’t lead dignified lives due to poor services provided to them and absence of basic law of governance.
Existing works in Saudi:
Most of previous studies propose methods in order to recognize alphabet, numbers or a limit number of words corresponding only on static gesture (like the word me, school, welcome, left, right).
Most of works use Arabic sign language datasets; others propose a system for generating a SASL corpus containing a limited number of words.
There is a growing body of literature that examines sign language interpreting provision and practices in legal contexts in various countries. The common theme in the results of all these studies is the limitations faced by deaf sign language users in gaining access to justice, either through inadequate interpreting provision, poor quality interpreting services, or lack of training, accreditation and standards for legal SLIs.
Requirements/ contributions:
1. Saudi sign language dataset (family and legal thematic sign language).
The ArSL2018 is a new comprehensive fully labelled dataset of Arabic Sign Language images launched in Prince Mohammad Bin Fahd University, Al Khobar, Saudi Arabia to be made available for researchers in the field of Machine Learning and Deep Learning. This data set contain only static gestures described by images, however the most word sign language are presented by video.
Arabic Sign Language (ArSL) is one of the languages that is used in Arab countries. This language is the unified language of several sign languages that exist in Arabic countries. It was proposed in 1999 by the League of Arab States (LAS) and the Arab League Educational, Cultural and Scientific Organization (ALECSO) and a dictionary consisting of 3200 sign words was published in two parts in 2000 and 2006.
Since, ArSL et SASL are different, we must create our dataset basing on Saoudi Dictionary that was published by Saudi association for hearing disability three years ago. It contains 28 fields such as social, medical, working...
2. AI system based on machine/deep learning:
Sign language complexity makes this task very hard especially when treating dynamic task such as IWSL recognition.
Many difficulties must be solved taking into consideration: speed gesture changeability, sign interchangeability, environmental conditions, geometric variations…
The proposed system must be efficient in terms of accuracy, precision, convergence speed and complexity to be embedded.
3. Continual learning:
We must propose a method for continual learning which studies the problem of learning from an infinite stream of data, with the goal of gradually extending acquired knowledge and using it for future learning. We therefore adapt the artificial intelligence system cross different sign language variations and new sign language.
Expected project results:
1. An application for automatic sign language interpretation
2. Novel products (Automatic interpreter) which will make deaf person feel more integrated and more understood
3. Safeguarding the rights of deaf sign language users to access quality interpreting services in criminal proceedings.
4. Deep CNN for sign language recognition
5. Huge Arabic database in legal and family context.
Work plan including Materials and Methods
The project comprises 7 work packages:
WP1 Project Management
WP2: Literature review and criticism
WP3: Database collection (family and legal thematic sign language)
WP4: Developing of CNN model for word recognition
WP5: Evaluation of the proposed model
WP6: Embedded the proposed solution in portable application
WP7: Test the application with real cases
The research plan is in the GanttChart table.
