This project will create a mobile app to improve Deaf / hearing communication by developing technology to automatically translate sign languages into spoken or written language and vice versa. It will not be limited to a single spoken/signed language pair, but will be able to translate among multiple signed and spoken languages within the EU domain. The design of the app will be user-driven, while exhaustive evaluation procedures will be carried out by different stakeholders.

Using the app, Deaf users will be able to use their web cams to sign to a hearing user. Hearing users will be able to read the translation of the signed communication as text or hear the translation as speech. Hearing users will be able to speak or type into the app and Deaf users will be able to see the translation signed by an avatar that goes significantly beyond the state-of-the-art by incorporating a number of innovative grammatical and pragmatic features that convey more of the communicative information of the translated message than was previously possible.

For the purposes of the app development, state-of-the-art machine translation technologies will exploit a significant amount of structured European sign language resources along with unstructured big data of sign language videos from less resourced sign languages. Moreover, in addition to the mobile app, other project outcomes will be a set of computer-assisted (human) interpretation tools, computer vision based language analysis tools and corpora building tools. These last two will assist users of under-resourced signed languages to develop additional corpora for use in the sign language technology pipeline that is at the core of the mobile app.