Playful Toothbrush: Ubicomp Technology for Teaching Tooth Brushing to Kindergarten Children CHI 2008, Florence Italy Yu-Chen Chang, Chao-Ju Huang, Hao Chu , Peggy Chi National Taiwan University Jin-Ling Lo, Nan-Yi Hsu, Hisn-Yen Wang, Ya-Lin Hsieh National Taiwan University Hospital

Child Behavior Training Child behavior training as important but challenging parental responsibility Potty training, self-dressing, cleaning room, self-feeding, tooth brushing (this work).

Child behavior training as important but challenging parental responsibility

Potty training, self-dressing, cleaning room, self-feeding, tooth brushing (this work).

Persuade Children to Change Behaviors The most common form of parental persuasion is verbal persuasion. “ If you don’t brush your teeth properly and thoroughly, you are not allowed to go to sleep”. Not effective, why not? Verbal persuasion alone lacks proper incentive to motivate children

The most common form of parental persuasion is verbal persuasion.

“ If you don’t brush your teeth properly and thoroughly, you are not allowed to go to sleep”.

Not effective, why not?

Verbal persuasion alone lacks proper incentive to motivate children

A Case Study: Child Tooth Brushing Target kindergarten children (5-6Y) Learn to care for their own oral hygiene Average 5Y children brush only 1/4 of teeth (Rugg-Gunn) A common scenario: candies -> improper brushing -> cavities -> dental clinic

Target kindergarten children (5-6Y)

Learn to care for their own oral hygiene

Average 5Y children brush only 1/4 of teeth (Rugg-Gunn)

A common scenario: candies -> improper brushing -> cavities -> dental clinic

Play-based Occupational Therapy Pediatric occupational therapy (O.T.) Leverage the desire of children to play to induce their behavior change willingly. Children may not like to brush teeth, but like to play. Add playfulness (game) into the brushing activity Effective – full of toys in pediatric O.T. clinic.

Pediatric occupational therapy (O.T.)

Leverage the desire of children to play to induce their behavior change willingly.

Children may not like to brush teeth, but like to play.

Add playfulness (game) into the brushing activity

Effective – full of toys in pediatric O.T. clinic.

Play-Based O.T. Limitations (1) Children treated in clinics during regular office hours (NTU Hospital) Many child behavior problems not observable to therapists Eating (dinner time), brushing (before bedtime), etc. Effective treatment is difficult. (2) Train general performance skill rather than specific functional skill Hand dexterity skill vs. tooth brushing skill Improvement in general performance skills may not translate into the target functional skills

(1) Children treated in clinics during regular office hours (NTU Hospital)

Many child behavior problems not observable to therapists

Eating (dinner time), brushing (before bedtime), etc.

Effective treatment is difficult.

(2) Train general performance skill rather than specific functional skill

Hand dexterity skill vs. tooth brushing skill

Improvement in general performance skills may not translate into the target functional skills

HCI/UBICOMP Opportunities Embed digital technology into a child naturalistic living environment Sensing to observe child behavior anytime, anywhere Game playing to influence child behavior anytime, anywhere Occupational therapist perspective: From treatment clinic To the child actual living environment (functional behaviors) HCI/UBICOMP perspective: From sense and track behaviors To engage children to change behaviors Also called Persuasive technology (by Fogg, King, and others)

Embed digital technology into a child naturalistic living environment

Sensing to observe child behavior anytime, anywhere

Game playing to influence child behavior anytime, anywhere

Occupational therapist perspective:

From treatment clinic

To the child actual living environment (functional behaviors)

HCI/UBICOMP perspective:

From sense and track behaviors

To engage children to change behaviors

Also called Persuasive technology (by Fogg, King, and others)

Playful Toothbrush Goals (Child) Dislike brushing Make brushing more attractive (Child) Habitual brushing but not properly Teach proper brushing skill Not replace adults’ supervision Extend their effectiveness Used together can overcome young children’s limited physical and cognitive abilities, such that they can successfully learn proper brushing

(Child) Dislike brushing

Make brushing more attractive

(Child) Habitual brushing but not properly

Teach proper brushing skill

Not replace adults’ supervision

Extend their effectiveness

Used together can overcome young children’s limited physical and cognitive abilities, such that they can successfully learn proper brushing

Sensing Camera vision to detect children’s brushing actions Brushing actions become game inputs Playing An interactive game of brushing teeth Start with a mirror image of dirty teeth. Brushing own teeth maps to cleaning the same virtual teeth in the mirror image. Demo video Playful Toothbrush Brush extension Web camera for tracking brushing motions Camera #2 to record videos for later human analysis Brushing game

Sensing

Camera vision to detect children’s brushing actions

Brushing actions become game inputs

Playing

An interactive game of brushing teeth

Start with a mirror image of dirty teeth.

Brushing own teeth maps to cleaning the same virtual teeth in the mirror image.

Demo video

Play-based Occupational Therapy Model Playful toothbrush is a treatment program / play activity (three steps). Volition Performance Habituation Bring enough enjoyment to attract a child to participate in the target activity. Ensure a child will have a successful experience. Set appropriate level of difficulty. Apply reinforcement to reward good performance, so increase change of repeating desirable behavior. Enough times to become a habit.

Playful toothbrush is a treatment program / play activity (three steps).

Design Considerations Enjoyment Bring enough fun to attract children to brushing Associate brushing with game playing and having fun Engagement Simple Interaction (Not all young children can learn to operate complex devices) Use their natural brushing actions as game inputs Automation Help children learn tooth brushing skill and internalize as habit In brushing case, internalizing means doing without much thinking) Advice adults not to interrupt children and their own motor planning, necessary for internalization of brushing motor skill

Enjoyment

Bring enough fun to attract children to brushing

Associate brushing with game playing and having fun

Engagement

Simple Interaction (Not all young children can learn to operate complex devices)

Use their natural brushing actions as game inputs

Automation

Help children learn tooth brushing skill and internalize as habit

In brushing case, internalizing means doing without much thinking)

Advice adults not to interrupt children and their own motor planning, necessary for internalization of brushing motor skill

Playful Toothbrush Prototype (two main components) Vision-based brushing tracker Brushing game

Vision-based Brushing Tracker Use one camera to detect brushing Recognize brushing of 24 different teeth surface areas (or granularity of 2 adjacent teeth surfaces) Mouth closed during brushing Bristle-teeth contact area not visible to an external camera Brush extension Each of four faces has unique LED-pattern Used as marker to assist camera-vision recognition Children told not to block brush extension from the camera

Use one camera to detect brushing

Recognize brushing of 24 different teeth surface areas (or granularity of 2 adjacent teeth surfaces)

Mouth closed during brushing

Bristle-teeth contact area not visible to an external camera

Brush extension

Each of four faces has unique LED-pattern

Used as marker to assist camera-vision recognition

Children told not to block brush

extension from the camera

How to recognize a brushing stroke on a specific teeth surface area? Take an example (brushing the outer surface of frontal teeth) Three features determines this brushing stroke Bristle rotated toward the face Brush oriented parallel to the face Back-and-forward motion vector (parallel to the face)

Take an example (brushing the outer surface of frontal teeth)

Three features determines this brushing stroke

Bristle rotated toward the face

Brush oriented parallel to the face

Back-and-forward motion vector

(parallel to the face)

Computer Vision Technique Use the brush extension marker to reconstruct Bristle rotation angle (x-axis) Brush orientation angle (z-axis) Brush back-n-forward motion vector Infer the brushing stroke and its target teeth area x (θ roll ) z (θ yaw ) y brush extension

Use the brush extension marker to reconstruct

Bristle rotation angle (x-axis)

Brush orientation angle (z-axis)

Brush back-n-forward motion vector

Infer the brushing stroke and its target teeth area

Accuracy Test (vision-based brushing tracker) 13 kindergarten children (72-81 months) Recorded 48 minutes of their brushing videos Compare human-read brushing strokes (ground-truth) and machine-recognized brushing strokes Machine-recognized accuracy 90% Need not be perfect. Occasional error in games has little effect.

13 kindergarten children (72-81 months)

Recorded 48 minutes of their brushing videos

Compare human-read brushing strokes (ground-truth) and machine-recognized brushing strokes

Machine-recognized accuracy 90%

Need not be perfect. Occasional error in games has little effect.

8.4% 6.8% Average 12.0% 41/343 3.7% 241/6493 13 0% 0/168 2.4% 118/4974 12 17.6% 111/630 10.2% 695/6847 11 8.6% 115/1298 0.9% 32/3532 10 3.3% 23/688 3.4% 239/7104 9 12.2% 206/1288 13.5% 579/4302 8 0% 0/196 5.3% 290/5520 7 0% 0/80 2% 90/4551 6 9.5% 60/634 4.3% 209/4813 5 5.3% 20/380 11.7% 617/5270 4 16% 209/1304 6.5% 331/5130 3 14.8% 49/332 14.8% 880/5958 2 9.7% 47/483 9.2% 602/6521 1 (%) false positive rate (frame) (%) false negative rate (frame) Children

Brushing Game Start with a virtual mirror image of the children’s own uncleaned teeth Grouped into 24 teeth surface areas “ 1” appears next to the first cleaning target 7 layers of plaques were drawn, requiring 7 brushing strokes Each brushing stroke removes one layer of plaque Combine audio feedback: Do-Re-Mi-Fa .. No game response for brushing other teeth areas

Start with a virtual mirror image of the children’s own uncleaned teeth

Grouped into 24 teeth surface areas

“ 1” appears next to the first cleaning target

7 layers of plaques were drawn, requiring 7 brushing strokes

Each brushing stroke removes one layer of plaque

Combine audio feedback: Do-Re-Mi-Fa ..

No game response for brushing other teeth areas

Brushing Game After 1st area is complete, “2” appears next to the 2 nd target area. Applause at the game completion Visual-audio feedbacks provide a sense of accomplishment Enforce a brushing sequence (Stillman’s brushing method) Upper right, upper left, the lower right, and lower left Two benefits of sequencing Ensure brushing all teeth Repeat this sequence enough time -> “do without thinking” 2 3 2

After 1st area is complete, “2” appears next to the 2 nd target area.

Applause at the game completion

Visual-audio feedbacks provide a sense of accomplishment

Enforce a brushing sequence (Stillman’s brushing method)

Upper right, upper left, the lower right, and lower left

Two benefits of sequencing

Ensure brushing all teeth

Repeat this sequence enough time -> “do without thinking”

User Study Two questions guided our user study How effective is the Playful Toothbrush in improving the brushing skills of kindergarten children? What aspects of brushing behaviors were affected by Playful Toothbrush?

Two questions guided our user study

How effective is the Playful Toothbrush in improving the brushing skills of kindergarten children?

What aspects of brushing behaviors were affected by Playful Toothbrush?

User Study Test Subjects 13 young children (72 – 81 months) from a NTUH-run kindergarten class 5 girls and 8 boys Written informed consent from parents Tooth brushing is required after meals or snacks Setup Install our system at the kindergarten’s restroom A video camcorder to record children’s brushing sessions

13 young children (72 – 81 months) from a NTUH-run kindergarten class

5 girls and 8 boys

Written informed consent from parents

Tooth brushing is required after meals or snacks

Setup

Install our system at the kindergarten’s restroom

A video camcorder to record children’s brushing sessions

User Study Procedure (11 days spanned over 3 weeks) Pre-trial (one day) Familiarize with the therapists and our system Pretest (two days) Brushed with their own toothbrushes Established a baseline behavior Training (five days) Brushed with our playful toothbrush A trained therapist helps children understand the brushing game Posttest (two days) Brushed with their own toothbrushes Measured behavior improvement from pretest One week follow up (one day) Brushed with their own toothbrushes Measured behavior retention after a week

Pre-trial (one day)

Familiarize with the therapists and our system

Pretest (two days)

Brushed with their own toothbrushes

Established a baseline behavior

Training (five days)

Brushed with our playful toothbrush

A trained therapist helps children understand the brushing game

Posttest (two days)

Brushed with their own toothbrushes

Measured behavior improvement from pretest

One week follow up (one day)

Brushed with their own toothbrushes

Measured behavior retention after a week

User Study: Brushing Effect Teeth cleaning effectiveness Red plaque disclosing dye before each brushing session (dye attached the plaque) Oral exam (counted teeth surfaces with dye before and after brushing) Plaque index number of teeth surfaces with plaque / total number of teeth surfaces Cleaning effect Reduction of Plaque Index from before to after brushing

Teeth cleaning effectiveness

Red plaque disclosing dye before each brushing session (dye attached the plaque)

Oral exam (counted teeth surfaces with dye before and after brushing)

Plaque index

number of teeth surfaces with plaque / total number of teeth surfaces

Cleaning effect

Reduction of Plaque Index from before to after brushing

User Study: Brushing Behavior By analyzing/coding brushing videos, measured 3 aspects of brushing behaviors : Length of brushing time Number of brushing strokes on each of 24 teeth areas Total number of brushing strokes

By analyzing/coding brushing videos, measured 3 aspects of brushing behaviors :

Length of brushing time

Number of brushing strokes on each of 24 teeth areas

Total number of brushing strokes

User Study: Cleaning Effect Results 0 .70( 0 .14) 0 .18( 0 .10) 0 .88( 0 .14) Day 11 Follow-up 0.66(0.28) 0.16(0.07) 0.85(0.10) Average 0 .70( 0 .15) 0 .15( 0 .08) 0 .86( 0 .16) Day 10 0 .67( 0 .13) 0 .16( 0 .10) 0 .83( 0 .11) Day 9 Post - test 0.66(0.17) 0.08(0.06) 0.74(0.19) Average 0 .68( 0 .17) 0 .11( 0 .10) 0 .79( 0 .18) Day 8 0 .64( 0 .26) 0 .04( 0 .05) 0 .68( 0 .28) Day 6 0 .67( 0 .15) 0 .09( 0 .10) 0 .76( 0 .14) Day 4 Training 0 .32( 0 .21) 0 .37( 0 .18) 0 .69( 0 .25) Day 3 Pre-test Cleaning effect M ean ( SD ) After brushing M ean ( SD ) Before brushing M ean ( SD )

User Study: Brushing Behavior Results 120(36) 8.31(5.07) 239.62(107.48) Day 11 Follow-up 118(30) 8.46(4.82) 248.00(87.12) Average 99(31) 9.46(5.12) 214.31( 71.91) D ay 10 137(41) 7.46(4.89) 281.69(120.66) D ay 9 Posttest Average Training 76(45) 12.39(4.75) 190.46(138.38) Average 67(46) 13.46(5.35) 168.62(157.03) Day 3 84(53) 11.31(5.25) 212.31(137.77) Day 2 Pretest Brushing time M ean (SD) (sec) Number of unbrushed teeth areas M ean (SD) Number of brushing strokes M ean (SD)

Result Summary (Pretest) Child subject failed to clean 37% of their teeth surfaces After 5 training days, improvements in Teeth cleaning effectiveness Number of brushing strokes Length of brushing time Coverage of brushed teeth areas

(Pretest) Child subject failed to clean 37% of their teeth surfaces

After 5 training days, improvements in

Teeth cleaning effectiveness

Number of brushing strokes

Length of brushing time

Coverage of brushed teeth areas

Conclusion Demonstrate a case study of applying HCI/UBICOMP technology in play-based occupational therapy For teaching young children proper tooth brushing, user study results were encouraging Other similar repetitive development tasks for young children (self-feeding, potty training, cleaning room, etc.) This technique can make training attractive and simple for adults/children.

Demonstrate a case study of applying HCI/UBICOMP technology in play-based occupational therapy

For teaching young children proper tooth brushing, user study results were encouraging

Other similar repetitive development tasks for young children (self-feeding, potty training, cleaning room, etc.)

This technique can make training attractive and simple for adults/children.

Examples of Other Persuasive Technologies

Sense to recognize behavior Weight sensor underneath the tray to sense eating actions Eating actions as game input Play to engage behavior change Interactive games: coloring cartoon character or penguin fishing Case Study: Playful Tray Encourage good eating habit in young children

Sense to recognize behavior

Weight sensor underneath the tray to sense eating actions

Eating actions as game input

Play to engage behavior change

Interactive games: coloring cartoon character or penguin fishing

Sense to recognize behavior Tilt sensor to detect drinking actions Drinking actions are game inputs Play to engage behavior change Game metaphor: hydrating/dehydrating body -> watering/drying a tree Case Study: Mug-Tree Encourage healthy habit of drinking fluid regularly

Sense to recognize behavior

Tilt sensor to detect drinking actions

Drinking actions are game inputs

Play to engage behavior change

Game metaphor: hydrating/dehydrating body -> watering/drying a tree

Sense to recognize dress & dressing color Camera and Computer Vision Play to engage behavior change Playful explore & experiment with how different clothing color look on people CHI 2008 poster on Tuesday Case Study: ChroMirror Persuade people to explore more colorful dressing

Sense to recognize dress & dressing color

Camera and Computer Vision

Play to engage behavior change

Playful explore & experiment with how different clothing color look on people

CHI 2008 poster on Tuesday

Thanks Q & A

Lesson #4 Mediate, not automate Mediation: not to replace children’s efforts but make the experience of performing more enjoyable for children Mediation works better than automation in this case.

Mediate, not automate

Mediation: not to replace children’s efforts but make the experience of performing more enjoyable for children

Mediation works better than automation in this case.

Lesson #1 Unpredictable behaviors from young children Make activity recognition difficult Randomly switch from left to right hands, swing brush wildly, head movement brushing the brush … Observe child behaviors carefully before designing and programming the system

Unpredictable behaviors from young children

Make activity recognition difficult

Randomly switch from left to right hands, swing brush wildly, head movement brushing the brush …

Observe child behaviors carefully before designing and programming the system

Lesson #2 Activity grading Children have different physical and cognitive capabilities Some learned fast (not challenging enough); some learned slowly (frustration) Different levels of challenges fitting to each child’s ability “ Play-based O.T. is about the experience of performance, or the fitness between the level of challenge in an activity and a child’s physical and cognitive capabilities.”

Activity grading

Children have different physical and cognitive capabilities

Some learned fast (not challenging enough); some learned slowly (frustration)

Different levels of challenges fitting to each child’s ability

“ Play-based O.T. is about the experience of performance, or the fitness between the level of challenge in an activity and a child’s physical and cognitive capabilities.”

Lesson #3 Personalization & customization Easily personalized and customized to environmental or human factors (preferences of children, changing performance of a child, different deployment environments). Left hand vs. right hand Lighting condition Child height Preferences about game characters

Personalization & customization

Easily personalized and customized to environmental or human factors (preferences of children, changing performance of a child, different deployment environments).

Left hand vs. right hand

Lighting condition

Child height

Preferences about game characters

Implementation Details Toothbrush extension weight Not too heavy for children to cause inconvenience in brushing Prevent child from holding the brush extension Block the brush extension from the overhead camera Solution: a protruding cap as separator Camera placement Not too high such that camera cannot see the brush extension clearly. Not too low such that children can switch the brush extension outside camera view

Toothbrush extension weight

Not too heavy for children to cause inconvenience in brushing

Prevent child from holding the brush extension

Block the brush extension from the overhead camera

Solution: a protruding cap as separator

Camera placement

Not too high such that camera cannot see the brush extension clearly.

Not too low such that children can switch the brush extension outside camera view