Sparking Memories on Big Life Fix

I recently met Aman as part of our BBC Big Life Fix episode for Children in Need. Over 3 months we worked together to create some digital solutions to address her challenges around memory loss.

At age 10, Aman is a bubbly funny amazing young girl. But about 3 years ago she was involved in a terrible car crash while travelling with her family in India and suffered severe brain injury as a result.

When she first woke up from the accident, she couldn’t walk or eat, as her mum said “it was like she was a newborn baby again.”

Aman has come a long way since then, she’s up and about, running around with her friends. She’s mastering the use of her left-hand as the accident has left her with uncontrolled spasms on her right side. She’s in school and starting Year 6!

At the same time, Aman continues to suffer from her brain injury. She has what people think of as “short term memory loss” – which in her day to day means she has trouble really remembering what was said 5 to 10 minutes ago and generally struggles with forming longer term memories.

This impacts two big facets of Aman’s life: School & Family.

At school Aman struggles in class to remember the teacher’s instructions, after school with revising the lesson.

At home Aman sometimes struggles to remember family memories, outings, stories – the underpinnings of her childhood memories.

I started on this journey without any knowledge of the depths and tricks of memory. Diving deeper into Aman’s case, it became apparent that ‘short term’ memory is not what we think it is.

Short Term Memory is technically only what’s happened in the last 1-2 minutes. Any memories from earlier than that is dealt with by Long Term Memory.

Here’s a very simplified view of Short Term and Long Term Memory.

In Long Term Memory we have 2 big aspects of memory: Episodic and Semantic

Episodic Memory deals with memories of events, sort of like the timeline of your life. These are moments in your life.

Semantic Memory is more like a Wiki, where you keep general knowledge but they aren’t necessarily associated with an event. For example, you might have some general knowledge about ‘Ancient Egypt’ – you don’t have to remember where you learned this, you’ve just built up your Semantic knowledge over time.

Traditionally, researchers have regarded how memories are formed to be a sequential process, going from Episodic Memory to Semantic.

For example, you experience an event, like a class on Ancient Egypt and then what you learned goes into Semantic Memory through some process – for example, while you are sleeping.

But researchers doing longer term studies have discovered that people are able to acquire general knowledge even without episodic memory. Somehow general knowledge can skip episodic memory and be retained. Kids with this kind of injury are often able to progress through school and university, without the ability to remember events.

Aman seems to have this kind of ability. For example she said when she learns something in school, she really doesn’t remember it after class or the next day, but over the course of the school year the knowledge she acquired does come back to her. And she doesn’t really know why.

Aman has decent short term memory, but very poor episodic memory. We think her semantic memory is better than her episodic memory.

So what is the challenge here? Because memory problems pervade her life and we can’t give her back her memory, how can we create technology fixes that help Aman in those moments when she really needs her memory?

Aman’s life, as a 10 year old, is pretty simple, she has her family routines, she goes to school, and she has experiences with her family like holidays and outings.

In order to tackle this, I decided to focus on some very specific pain points in her life.

In School, she has trouble getting through a class, she can’t recall what the teacher instructed and therefore cannot perform activities in the class. She has a full-time teaching assistant funded by the state, but this may disappear when she goes to secondary school. She also needs to be able to review classes after the fact, so she can study for exams.

With her family, Aman has trouble recalling special moments like ‘the picnic they went on last weekend’. These are moments she can’t share with her family and she gets increasingly frustrated, sometimes going up to her room and slamming the door.

There’s been some research into ‘diarycams’ – for example with the Sensecam project. These are wearable cameras that take pictures at regular intervals.

I tried one out for a day. Would this be able to replace Aman’s memory?

For one thing, the images are blurry and not that interesting. There’s a lot of detail, but not much emotion.

What Aman really needs is not to be bogged down by the Details of her day, but to recall those special distinct moments with her family. Maybe for adults the need will be different because you do need to know if you’ve just taken some pills or locked the door when you go out.

Focusing in on the challenge, I wanted to create technology fixes that will support Aman in her transition to being a young adult.

For school, this means something to help retain and build her semantic knowledge, so she can finish school.

For home, this is perhaps something that can help Aman relive the emotional moments with her family. Not necessarily the details of a holiday, but those special moments with emotional resonance.

I took Aman on a punting trip in Cambridge, with a polaroid camera to help take a record of the experience. Instead of taking photos all the time, we would naturally curate the images by taking pictures of interesting things.

What was really interesting was that during the boat ride Aman was really excited about our Punter, who reminded her of Niall from One Direction, she kept giggling and taking photos of him.

Afterwards, Aman looked over the photos and she could recall about 60% of them.

She looked at the photo of Niall and didn’t remember him at all.

It was then that I got an insight into something special, Aman’s mum started reminding her about Niall, telling her how she felt. And Aman’s eyes lit up – she started giggling and remembering Niall. In that moment I really felt like she was reliving an emotional moment, that she could remember how it felt to be on the boat with Niall.

Perhaps it’s about the combination of images and audio stories that can evoke these emotional memories for Aman.

I worked with my amazing colleague Nic Villar and our summer intern Joshua Moody to start prototyping some of these experiences.

Our first fix is called Memory Sparks.

It’s a very simple idea – it’s so simple it might even be invisible.

Memory Sparks combines family photos and audio stories, so family members can upload photos for Aman and record stories and recollections to go along with them. It’s a shared family space. Aman can then sit with her family to reminisce over these moments or be on her own and listen to the stories her family has recorded.

I really think this will mean a lot for Aman to be able to recall some special moments in her past, but also to have these to keep throughout her life.

  • Memory Sparks are Photos + Audio Stories
  • Aman’s family members can upload photos and record audio stories and share these memories with the whole family
  • The photos can then be played back like a Movie
  • We are using machine learning to analyse the images and speech recognition for intelligent searching, order to allow Aman to deeply search her memories as they build up

 

 

Study Sparks is a more pragmatic tool – it records the audio of Aman’s lesson and allows her to live rewind the class, bookmark moments to review later.

This way she has a record of her classes and can refer to the important parts of the lesson.

  • Study Sparks provides a Live Rewind of the class
  • It helps her get through the immediate struggles in class of recalling 5-15 mins in the past
  • Aman can bookmark of important points in time to review these later
  • Speech recognition provides a transcript of the class, automatically giving Aman lesson notes

 

To see a summary of Aman’s reaction to these fixes, watch the BBC clip here – http://www.bbc.co.uk/news/av/stories-41835115/the-invention-helping-a-little-girl-find-her-lost-memories

You can also see the full Big Life Fix episode on iPlayer!!

And finally, I’m raising money for BBC’s Children in Need – your donations will be 100% matched by my company, Microsoft, so donate now on my JustGiving page!

 

8 Comments

  1. Dear Haiyan

    Like others I am so impressed and excited by your innovations and their potential. I am a Palliative Care physician and Medical Director at a UK hospice and wondered if there was any chance of a conversation around Memory Sparks and its use in our patient group? My email is cecily.wright@thameshospice.org.uk

    Thank you!

  2. Catherine Powell says:

    Hi Haiyan, your imagination and skill is helping Aman is truly inspirational.

    I wonder if there are even wider applications for this software.

    For example, I have been creating a “red book” for each of my parents. It contains photos and annotation about each.

    As my parents age and have friends who are struggling with memory loss, sometime just “old age” some times dementia related I hope that by collecting these photos and annotations I will be able to use them in a similar way to Memory Sparks.

    Have you consider trialling memory sparks with dementia patients? Family and friends could record messages in the same way as you did for Aman.

    I would definitely be interested in converting my “red books” into Memory Sparks files and would be very interested in obtaining a copy of the software

  3. Haiyan,

    You are a real inspiration. I am using your work to inspire my Computer Science students. Using technology to make a difference for good.

    Wonderful stuff.

    Thank you.

  4. Sarah Rudebeck says:

    Hi Haiyan,

    I am a Paediatric Neuropsychologist working at the Evelina Children’s Hospital. We often work with children with episodic memory problems and would be really interested in seeing if the Memory Spark Ap would benefit them. If this would be possible please email me at: sarah.rudebeck@gstt.nhs.uk

    Thanks,

    Sarah

  5. Nicky Robledo says:

    Hi Haiyan, your inventions are just astounding! Such amazing work you & your colleagues do, changing people’s lives! Amazing! I was wondering if “Memory Sparks” would be available to buy? I think my father, who’s memory is failing, would really benefit from this. Many thanks, Nicky.

  6. Hi Haiyan – hope you’re well.

    This is inspirational.

    I know that other modes of senses can really help evoke memories too. I wonder if audio and video recollections could somehow be combined with other senses, such as smell, to help develop the longer-term memories?

  7. Vanie Nagashima says:

    Haiyan, I am so proud of your involvement with these beautiful individuals. With Aman-you’ve helped to alleviate her ABI (Aquired Brain Injury) issues/symptoms through your amazing & innovative tech tools. Your team Big Life Fix stories are so inspiring!! The transformations to their lives REAL. It is a beautiful gift to give these individuals back control & self-dignity. Bravo!

  8. Vanie Nagashima says:

    Haiyan, I am so proud of your involvement with these beautiful individuals. With Aman-you’ve helped to alleviate her ABI (Aquired Brain Injury)issues/symptoms through your amazing & innovative tech tools. Your team Big Life Fix stories are so inspiring!! The transformations to their lives os REAL. It is a beautiful gift to give these individuals back control & self-dignity. Bravo!

Leave a Reply

Your email address will not be published. Required fields are marked *

*