Case overview
Visionear is a MVP concept that aims to cure deafness through technology. The application helps the deaf by convert sound into visual form through Cutting Edge Technology and present it to the deaf. The text or other visuals being near the speaker face and in real-time so the deaf can easily read and understand it. This cures the effects of deafness such as social isolation.
The Brief
Visionear involved a problem of curing deafness through the latest Mixed Reality and Artificial Intelligence Technology. In order to achieve the goals, there were many attempts of finding a technology which was portable, usable. feasible both technically and financially. After looking into many frameworks such as Unity, Mods of Unity, many Mobile AR Frameworks & Hybrid approach of Unity & Mobile we finally found something that had the criteria for the system.
Our Approach
Flutter Framework was used to create the mobile app since it is much more native as compared to some of the other frameworks and a technology which is new with good reviews. Using flutter, we accessed the camera module and implemented 2D text manipulation to move the text. The person face is detected through AI models and the coordinates are given to the 2D algorithm to track and follow the face. The text is generated through NLP AI Models, and it is refreshed every time.
There are other features to aid the deaf such as nearby sound alerts, sentiment analysis and visual customization options.
The Results
By the end of the designated timeframe a MVP was formed with the the following features:
- Speech to Text in a mixed reality settings using a phone without internet in real-time.
- Sentiment Analysis with the generated text in real-time.
- Text Customization Options
- AR & VR modes that work with Cardboard AR & VR.
There were some features that didn’t make it in the final build which were proposed by us initially:
- Nearby Sound Analysis did not have the best accuracy however we were nearly there.
- Personality detection was not working accurately at all, and it was almost random.
- More confidential features that await future ventures.
