Poles have developed a revolutionary communication system for people with profound intellectual disabilities. The European Union hopes the project will be successful
With the Internet of Things (IoT) and artificial intelligence, Poles from an international consortium have been trying to help people with profound mental disabilities to communicate with the world.
They have created a system which basically uses cameras to observe people with disabilities and sends the materials they record to a database that is then used by researchers to train artificial intelligence algorithms. The objective of the AI is to help the disabled to communicate with other people.
Youtube movie URL: https://www.youtube.com/watch?v=G8s8U8lxMcw
Source: PCSS Poznańskie Centrum Superkomputerowo-Sieciowe (Poznan Supercomputing and Networking Center) / YouTube
The project is coordinated by the Poznan Supercomputing and Networking Center. Another important member of the consortium includes the Poznan-based “Na Tak” Association, which supports people with the so-called Profound Intellectual and Multiple Disabilities (PIMD) in kindergartens, schools and facilities for adults. The project involves persons under care of the association who cooperate with researchers, engineers and therapists.
Behaviors as decisions
The project is called Insension. The European Union is pinning hopes on it and has decided to subsidize it with almost EUR 2.3 million under the Horizon 2020 program over three years. The project is led by Michał Kosiedowski from the Poznan Supercomputing and Networking Center, who is also a co-developer of the SelFind electronic system, which, supported by an application, helps people with mild and moderate intellectual disability.
The Insension platform is designed to recognize non-symbolic behaviors (i.e. human behaviors which do not refer to symbols) of PIMD persons and to interpret them as decisions of a profoundly disabled person. The solution will make it possible for the people with PIMD to directly influence their own personal situation. The crucial issue will be smart support applications which will make it much easier for the disabled to communicate their needs. The system will be able to distinguish particular gestures, sounds and facial expressions of persons with PIMD and learn how to interpret them to facilitate the interaction of such persons with the surrounding environment.
Six partners, one tool
The Insension project was launched in 2018. Michał Kosiedowski points out to the fact that the consortium consists of six partners from Poland, Germany, Spain and Slovenia.
“The technological platform is developed by IT experts from PCSS, the CTIC technology center from Spain and the Jozef Stefan Institute from Slovenia,” he explains. “We also collaborate with specialists from the Heidelberg University of Education, who study the interactions of people with disabilities and who share their knowledge in the field of special education. Both them and the therapists from the “Na Tak” association help us understand how people with profound intellectual disabilities communicate with the world and how a tool facilitating the communication should be developed.
In general, the system uses cameras to observe people with disabilities and sends the materials they record to a database that is then used by caregivers to train artificial intelligence
The last partner of the consortium includes Harpo, a Poznan-based company, which is a distributor and producer of technologies supporting people with disabilities. Harpo is responsible for finding possibilities to commercialize the system.
“The special education experts from the Heidelberg University of Education explain to us how people with profound intellectual and multiple disabilities communicate. At the end of this year German researchers are going to analyze the impact of the Insension project on the communication of such people. We have also been supported by professor Andreas Fröhlich, a retired world-class specialist in profound disabilities, who appeared in the video promoting the project.
Gestures, facial expressions, vocalizations
Gestures, facial expressions and vocalizations of the persons under care of the “Na Tak” association compiled as audio and video data are analyzed by the caregivers and therapists in collaboration with the researchers from Heidelberg. This helps to determine which signals recognized by the platform are important and signify one’s needs or reactions.
“When interacting with other people, persons with very profound intellectual disabilities do not use symbols. Their reactions to the world around them are limited to gestures, facial expressions and vocalizations. A reaction can either be positive, in which case there is no need to take any action, or negative, which means that a disabled person wants to change something in a particular situation,” explains Kosiedowski.
Youtube movie URL: https://www.youtube.com/watch?v=qKXbGOKrwzQ
Source: Digital Single Market / YouTube
The Insension system will be personalized for every user. The software “learns” how to interpret signals sent by users, which allows to generate the information regarding a specific need. Obviously, artificial intelligence will never fully replace the parents of the disabled but we mustn’t forget that the parents are not always around and that one day they will eventually pass away.
Three key applications
People with PIMD resort to different non-symbolic behaviors to show their reactions to a given situation. The task of the system is to individually interpret them by correlating them with the surroundings, persons, objects and sounds. This way, persons once fully relying on others can now become more independent.
“The Insension platform interprets the behaviors of the disabled as their intentions to take actions. This can be achieved with the so-called support applications. We have been developing three applications of that sort. The first one will make it possible for a person to communicate their needs in a way that is understandable for other people, for example: ‘I’m glad you’re here’. The second one is a multimedia player that can play the favorite song to a person in distress, e.g. because of being alone in a room. The last one is an intelligent room application. It will help a disabled person to switch on a device that has a soothing effect, for example a fan,” explains Michał Kosiedowski.
The platform is scheduled to be tested this fall.