Hey GUI is a chatbot that can be used to find images or textual information...
Hey GUI is a chatbot that can be used to find images or textual information about apps and their user interfaces using natural language conversations.
Source: Aalto University

Chatbot explains apps and how they access data

Chatbots have already become a part of our everyday lives with their quick and intuitive way to complete tasks like scheduling and finding information using natural language conversations. Researchers at Aalto University have harnessed the power of chatbots to help designers and developers develop new apps and allow end users to find information on the apps on their devices.

The chatbot — ‘Hey GUI’ (pronounced goo-ee), short for Graphical User Interface can answer questions by showing images and screenshots of apps, or through simple text phrases.

“Hey GUI eliminates the need for coding skills or technical expertise. Anyone can use it to search for information on, for example, which one of my apps is asking for permission to use the camera, or where is the search button on this screen”, says postdoctoral researcher Kashyap Todi of Aalto University.

Important findings for chatbot developers

Different kinds of applications and user interfaces contain vast amounts of information, which can be used for different needs and tasks. In their study, the researchers set out to find the most desirable chatbot features and how users would have conversations with them.

Three target groups were considered in the experiment: designers, software developers, and general users. Over 100 designers, developers, and ordinary users responded to the survey. “Our goal was to get a good understanding of what capabilities and functions different kinds of users would find most beneficial when using this new type of chatbot, as well as how they would formulate questions — especially what kinds of words and phrases they would use. This is an important first step towards developing chatbots that can help users find information about apps using simple conversations”, says Kashyap.

As a result of the survey, 21 key features and more than 1300 of the most common questions put forward by users were collected and analysed to create Hey GUI, allowing users to find information within user interfaces using simple conversations, based on the most frequent expressions collected in the survey.

“Designers can get inspiration, for instance, by asking Hey GUI to find good online shop designs. The chatbot responds with several screenshots from existing apps in the catalogue. Software developers could ask a chatbot what typical colour codes are used for login buttons, and the chatbot responds with several options”, Kashyap adds.

In questions from ordinary users, there was a special emphasis on matters related to privacy and accessibility information, or getting help about their apps, such as asking which apps display maps.

Subscribe to our newsletter

Related articles

Can chatbots help fill the empathy gap?

Can chatbots help fill the empathy gap?

Stressed out? Need to talk? Turning to a chatbot for emotional support might help, research from Michigan State University shows.

Dementia: Chatbots for patients and caregivers need more work

Dementia: Chatbots for patients and caregivers need more work

Chatbots hold promise for dementia patient or caregiver support, but are still in their infancy, new research finds. None of the interactive digital apps tested performed well on all testing criteria.

Can an app change your personality?

Can an app change your personality?

Research has shown that daily use of a smartphone app can lead to desired personality changes within three months.

Opening the ‘black box’ of AI

Opening the ‘black box’ of AI

AI is growing ever more powerful and entering people’s daily lives, yet often we don’t know what goes on inside these systems.

Chatbots help decrease opioid use after surgery

Chatbots help decrease opioid use after surgery

A study showed that patients receiving messages from a chatbot used fewer opioids after fracture surgery, and their overall pain level fell, too.

App for autism screening yields useful data

App for autism screening yields useful data

A study of an iPhone app to screen young children for signs of autism has found that the app is easy to use, welcomed by caregivers and good at producing reliable scientific data.

Augmented reality app may aid patients with Parkinson’s

Augmented reality app may aid patients with Parkinson’s

Bioengineering students program smartphone to guide patients who ‘freeze’ while walking.

The gait lab for the pocket – app-based fall prevention

The gait lab for the pocket – app-based fall prevention

Every day, elderly people fall – be it at home or in care facilities. Lindera aims to reduce the risk of falling with the help of artificial intelligence.

AR apps are designed mainly for youngsters

AR apps are designed mainly for youngsters

A study exploring the use of augmented reality to support older adults finds the user interface is sometimes confusing for those aged 50+.

Popular articles

Subscribe to Newsletter