ӰAPP

ūDzԲ

Feature

Researchers developing augmented reality game to help stroke patients recover

27 March 2023
The puzzle game strengthens the body, stimulates the brain and encourages social connection – and about all patients need is a smartphone and wooden blocks.

For patients recovering from stroke, the leading cause of adult disability in the developed world, the rehabilitation process is monotonous and solitary.

Researchers at Waipapa Taumata Rau, University of ӰAPPare working to change that with an augmented reality game that strengthens the body, stimulates the brain and encourages social connection.

Associate Professor , a human-computer interaction expert with the School of Computer Science, along with colleagues including neurorehabilitation experts Professor , Professor and Research Fellow , wanted to develop a rehab tool that would motivate patients to keep going.

With collaborators including Amber Taylor (Ngāti Whātua, Ngāpuhi, Te Rarawa, Tainui, Ngāti Mutunga, Te Ātiawa), CEO of , an award-winning Māori-owned game development company, honours student Maya Gibson (Ngāti Apa) and hospital physiotherapist Jon Bagnall, the researchers are prototyping an augmented reality game for stroke rehab patients.

“There’s a massive problem in that people cut their rehab too early and never finish their programme,” says Taylor. “We’re really hoping the gameplay we’ve come up with encourages daily use and keeps up engagement levels so patients keep improving their movement and coordination.”

photo (Danielle)
Danielle Lottridge

The complexity of simplicity

photo L

The team’s first prototype had patients don virtual reality headsets to do puzzles in space. With more research and input, however, the team determined that manipulating real objects is better for rehabilitation purposes. What’s more, VR headsets are expensive and unfamiliar to many.

At the other end of the technological scale are wooden blocks. They’re a standard part of stroke rehab for good reasons – manipulating them has a lot in common with everyday tasks such as picking up a pill bottle or twisting a lid. They’re cheap and readily available. They are, however, boring.

After many design iterations, Lottridge and her colleagues are now focusing on an activity that uses high-tech augmented reality to turn low-tech wooden blocks into an infinite series of puzzles.

When you see the prototype, it seems like a simple idea. That’s exactly the point.

“The goal is for it to be self-explanatory,” says Lottridge. “With human-computer interaction, though, the simpler the interface, the harder it is to design.”

Co-designing with stroke patients

As part of the process, Lottridge and her colleagues interviewed Māori, Pacific, South Asian and Pākehā stroke patients in their homes. They asked about their experiences of healing from stroke, how they use technology and what they find motivating. From these sessions came the message that stroke survivors need the therapy and tech to fit into their everyday lives.

“For me as a researcher in computer science, I sometimes feel pushed to use the latest gadgets and technologies,” says Lottridge. “Co-design pushes me to look at what technology works best for people.”

“For me as a researcher in computer science, I sometimes feel pushed to use the latest gadgets and technologies. Co-design pushes me to look at what technology works best for people.”
Danielle Lottridge

Another clear message from participants was that they experienced health and healing as multidimensional, comprising physical and mental health as well as social connection and, for some, a spiritual dimension. This fit with the te ao Māori concept of te whare tapa whā – that there are four cornerstones to health: physical, spiritual, family and mental.

While the game is informed by te whare tapa whā, it’s designed to help anybody recovering from a stroke.

“The idea is to create a multiplayer environment so people can rehab with whānau,” says Taylor. “Whānau is important to everyone, so it really contributes to rehab if you can interact with your mokopuna even if they’re not in the same house as you. That connection has significant health and wellbeing benefits.”

“The idea is to create a multiplayer environment so people can rehab with whānau. Whānau is important to everyone, so it really contributes to rehab if you can interact with your mokopuna even if they’re not in the same house as you. That connection has significant health and wellbeing benefits.”
Amber Taylor

How the game works

photo L

The game combines a smartphone app, blocks with markers resembling QR codes, a flexible clamp for a smartphone and a magnetic whiteboard that allows the game to be played on a tilted surface, which was inspired by a co-design participant who had to keep her leg elevated.

Other than the smartphone – something most people already have – the equipment required is low-tech and low-cost.

When viewed through the phone’s camera, the blocks look like cut-up pieces of an image. The blocks have to be turned the right way and brought together. The image can be anything – a family photo, scenery, a work of art, a funny meme or text such as a whakatauki.

The plan is for friends and relatives to download the app and use it to send the patient images and interact with them.

“We’re still prototyping, but we’d like it to be similar to how you post a picture to Facebook – you hit ‘post’ and the pictures cue up for your relative or friend,” says Lottridge.

“Once the patient has done the puzzle, they can add virtual stickers or overlays and send the image back to the friend who sent it to them, so it comes full circle and encourages the friend to send the next picture.”

The app does remind users to practice so they can stay on track with their physiotherapy goals, but the researchers hope the game itself is motivating enough to keep patients coming back.

“Being creative, being social and having moments of respite and enjoyment all rolled into physical therapy will really help people heal,” says Lottridge.

photo

Next steps

The team is now seeking funding to do a feasibility study in a hospital ward, with patients receiving initial instruction on the game in hospital and given a kit to take home. With participant permission, their gameplay would be tracked and their upper limb function assessed at regular intervals. Through questionnaires and interviews, participants could provide feedback about the game as well as their mood and social support levels.

After that would come randomised control trials, the gold standard for establishing effectiveness.

Eventually, Lottridge says, she’d like the tool to be available to patients anywhere, as a low-cost intervention, with kits shipped around the world.

“This is very different from other interventions that are out there, so it can make a real difference.”

Interested in connecting with the University’s medtech researchers?
Contact Hope Harding