Personal Speech Analytics Bill Jarrold 29 May 2011 Quantified Self 2011

1. Personalized Speech Analytics
Bill Jarrold
29 May 2011
Contents
I xeed P
IFI xeed qu—nti(—˜le me—sures of ourX F F F F F F F F F F F F F F F P
IFP qo ˜eyond selfEreport F F F F F F F F F F F F F F F F F F F F F F Q
IFQ €redi™t yut™omes F F F F F F F F F F F F F F F F F F F F F F F F F Q
IFR essist with hi—gnosis F F F F F F F F F F F F F F F F F F F F F F F Q
P ‡h—t is spee™h —n—lyti™sc Q
PFI winiEhes™ription F F F F F F F F F F F F F F F F F F F F F F F F F Q
PFP uinds of uestions it g—n enswer F F F F F F F F F F F F F F F Q
Q ‡h—t pro˜lems h—s it su™™essfully solvedD if —nyc Q
QFI sntro F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F Q
QFP ‚e™ording iveryd—y vife PRˆU he˜ ‚oy F F F F F F F F F F F F R
QFPFI ‚e™ord ƒon9s vifeX Q ye—rs QHH ter—˜ytes F F F F F F F F R
QFPFP „ih „—lk 4„he firth of — ‡ord4 F F F F F F F F F F F F R
QFQ row st ‡orks F F F F F F F F F F F F F F F F F F F F F F F F F F F R
QFR ‡ork ˜y my ™olle—uges —nd s F F F F F F F F F F F F F F F F F F R
QFRFI xeurodegener—tive disorders —nd spee™h F F F F F F F F R
QFRFP righ ™ognitive imp—irment versus typi™—ls F F F F F F F R
QFRFQ €re elzheimer9s hise—se —nd spee™h F F F F F F F F F F F R
QFRFR hepression —nd spee™h F F F F F F F F F F F F F F F F F F R
QFS vixe ‚esour™esD h—s t—ught us these thingsX F F F F F F F F F S
R ‡h—t —re tod—y9s frontiersc S
RFI xot the ™ost of stor—ge3 F F F F F F F F F F F F F F F F F F F F F S
RFIFI —udio re™ord your whole life for £ 6UDHHH F F F F F F F S
RFP qetting enough tr—ining d—t— F F F F F F F F F F F F F F F F F T
I

2. RFPFI need —udio s—mples t—gged with 4l—˜els4 or out™ome
me—sureF F F F F F F F F F F F F F F F F F F F F F F F F F F T
RFPFP gomputer s™ien™e does the rest F F F F F F F F F F F F F T
RFPFQ ix—mples of 4v—˜els4 F F F F F F F F F F F F F F F F F F T
RFQ qetting phone d—t— re™orded F F F F F F F F F F F F F F F F F F T
RFR eutom—ti™ ƒpee™h ‚e™ognition @eƒ‚A e™™ur—™y F F F F F F F F T
RFS en—lysis —nd snterpret—tion of h—t— F F F F F F F F F F F F F F F T
RFT u—nti(—˜le €sy™hodyn—mi™s F F F F F F F F F F F F F F F F F F U
S row might selfEtr—™kers p—rti™ip—tec U
SFI golle™t „ext h—t— F F F F F F F F F F F F F F F F F F F F F F F F F U
SFP golle™t ƒpee™h d—t— F F F F F F F F F F F F F F F F F F F F F F F F U
SFQ sdentify 4yut™omes4 to €redi™t F F F F F F F F F F F F F F F F F U
SFR r—™kers ‡—nted F F F F F F F F F F F F F F F F F F F F F F F F F F U
SFS porm — hs‰ vixe …ser9s qroup F F F F F F F F F F F F F F F F U
T ‡h—t —re some good linksF U
TFI vs‡g F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F U
TFP vixe F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F U
TFQ €geh F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F U
TFR he˜ ‚oy „—lk 4„he firth of — ‡ord4 F F F F F F F F F F F F F U
EBEorgEBE
1 Need
1.1 Need quantiable measures of our:
ˆ emotions
ˆ mood
ˆ rel—tionships
ˆ developmentF
sf you me—sure it you ™—n m—n—geGimproveG™ontrol itF
P

3. 1.2 Go beyond self-report
1.3 Predict Outcomes
elzheimer9s dise—se —nd moreF
1.4 Assist with Diagnosis
reg—rding neuropsy™hi—tri™ ™onditions su™h —s depressionD su˜types of frontoE
tempor—l lo˜—r degener—tion
2 What is speech analytics?
2.1 Mini-Description
e ™olle™tion of —utom—ti™ ™omput—tion—l methods for extr—™ting useful qu—nE
titi—tive inform—tion —˜out spe—ker ™h—r—™teristi™s or spee™h ™ontent G emoE
tionF
2.2 Kinds of Questions it Can Answer
ss the spe—ker X
ˆ depressed nonEdepressed or —t high risk of sui™idec
ˆ lying or truthfulc
ˆ m—le of fem—lec
ˆ —t high risk of elzheimer9s or not @despite ˜eing ™urrently ™ognitively
norm—lA
3 What problems has it successfully solved, if any?
3.1 Intro
ˆ te™hnologies still in their inf—n™yF
ˆ rese—r™h —nd development in progressF
ˆ progress likely f—stF
Q

4. 3.2 Recording Everyday Life 24X7 Deb Roy
QFPFI ‚e™ord ƒon9s vifeX Q ye—rs QHH ter—˜ytes
QFPFP „ih „—lk 4„he firth of — ‡ord4
httpXGGwwwFtedF™omGt—lksGde˜•roy•the•˜irth•of•—•wordFhtml
ƒ„e‚„ €ve‰sxqX R min U se™onds in ƒ„y€ €ve‰sxqX T min PI se™E
onds or so
3.3 How It Works
ƒee di—gr—m on white˜o—rdF
3.4 Work by my colleauges and I
QFRFI xeurodegener—tive disorders —nd spee™h
€eintner fD t—rrold ‡D †ergyriy hD ‚i™hey gD „empini wvD yg—r tF ve—rning
di—gnosti™ models using spee™h —nd l—ngu—ge me—suresF gonf €ro™ siii
ing wed fiol ƒo™F PHHVYPHHVXRTRVESIF
QFRFP righ ™ognitive imp—irment versus typi™—ls
t—rroldD ‡FvFD €eintnerD fFD ‰eh iFD ur—snowD ‚FD t—vitzD rFƒFD ƒw—nD qFiF
@PHIHA v—ngu—ge en—lyti™s for essessing fr—in re—lthX gognitive smp—irE
mentD hepression —nd €reEƒymptom—ti™ elzheimer9s hise—seD to —ppe—r in
€ro™eedings fr—in snform—ti™s PHIH
QFRFQ €re elzheimer9s hise—se —nd spee™h
ƒee fr—in snform—ti™s p—per immedi—tely —˜oveF
efter —pplying m—™hine le—rning to ™omputerE˜—sed lexi™—l —n—lysis of
the tr—ns™ripts we were —˜le to predi™t whi™h individu—ls went on to develop
eh with —n —™™ur—™y of UQ 7@™omp—red to st—nd—rd ˜en™hm—rk of n—ive
le—rner ˜—seE line —™™ur—™y of SV7AF
QFRFR hepression —nd spee™h
ƒee fr—in snform—ti™s p—per immedi—tely —˜oveF
po™using on —nswer to question row do you feel —˜out wh—t you h—ve
—™™omplished in your work or ™—reerF
R

5. …se —ll wordElevel fe—tures in m—™hine le—rning —lgorithms —™™ur—™y a
@UV@RFUA 7AF
…se (rst person word frequen™y only me—n@sdAaWUFT@RFWIA 7
3.5 LENA Resources, has taught us these things:
4 What are today's frontiers?
4.1 Not the cost of storage!
RFIFI —udio re™ord your whole life for £ 6UDHHH
g—n you improve upon this ™ost estim—tec
ˆ I meg—˜ypte per minutec
httpXGGwikiF—nswersF™omGGrow•m—ny•gig—˜ytes•is•R•hours•of•—udio•˜ook
sf you rip —udio to IPVk˜Gs stereo mpQD you get —˜out I minute per
meg—˜yteF
R hrs ˆ TH minutes in —n hour a PRH minutes a —˜out PRH meg—˜ytes
a less th—n FPS of — gig—˜yteF
iven less if you rip —t — lower ˜itr—te —nd mono @sound qu—lity is not
usu—lly — hugely import—nt issue when it ™omes to —udio ˜ooksA
xow if you rip —s Fw—v (leD whi™h is un™ompressedD you ™—n multiply
those (gures ˜y —˜out IH
ˆ PIU meg per hour
prom this (le there is this quoteFF
I hr is IQ gigs in mpegP
„hus I hr per IQ gigs aa I hr per IQHHH meg aa @G IQHHH THFHA a
PITFTT
ˆ gost of ƒtor—geX I „f d—t— ™osts 6IRH @pro˜—˜ly lessA
ƒee httpXGGwikiF—nswersF™omGGrow•mu™h•does•—•ter—˜yte•™ost
ˆ I „f a IDHHHDHHH weg
I „f a IHHH qig I qig a IHHH meg therefore I „f a IDHHHDHHH weg
ˆ IHH ye—rs a SPDSTHDHHH minutes
life of IHH ye—rsF F F @IHH ye—rsA @QTS d—ys per ye—rA @PR hours per
d—yA@TH min per hourA a @B IHH QTS PR THA a SPSTHHHH a SPDSTHDHHH
S

6. ˆ €ut it togetherX IHH ye—rs of —udio ™osts 6UPVH
@G @SPSTHHHH min per lifeA@IHHHHHH min per terr—˜yteAA a SP terr—˜ytes
needed per life @B SP @6IRH per terr—˜yte driveAA a 6UPVH
4.2 Getting enough training data
RFPFI need —udio s—mples t—gged with 4l—˜els4 or out™ome me—E
sureF
RFPFP gomputer s™ien™e does the rest
‡e then —pply ™omput—tion—l te™hniques su™h —s fe—ture sele™tion —nd m—E
™hine le—rning to le—rn to predi™t the out™ome l—˜lesw on new —udio s—mplesF
RFPFQ ix—mples of 4v—˜els4
por ex—mpleX
ˆ mood
ˆ level of depression
ˆ ™ognitive s™ore
ˆ money spent th—t d—y
ˆ wh—t elsec ‰our ide—s here
4.3 Getting phone data recorded
v—rge w—ll ˜etween phone —nd €he —ppli™—tion devi™eF
veg—l issuesX illeg—l to re™ord telephone ™onvers—tionF
ƒkypec qooglevoi™ec endroidc
4.4 Automatic Speech Recognition (ASR) Accuracy
‡ord irror ‚—te E possi˜ly not so import—nt
himension—lity ‚edu™tion implies ro˜ustness to error
4.5 Analysis and Interpretation of Data
‡h—t do the p—tterns me—nc
pe—ture sele™tion
w—™hine ve—rning
T

7. 4.6 Quantiable Psychodynamics
por ex—mpleD does selfEfo™us ™—use depressionc
‡h—t —re norm—lD lowD high levels of selfEfo™usc
5 How might self-trackers participate?
5.1 Collect Text Data
„ext wess—ges E €hone†iew
5.2 Collect Speech data
yther9s †oi™em—il E €hone†iew ‚e™ord ‰ourself r—™k endroidD qoogle†oi™eD
ƒkype
5.3 Identify Outcomes to Predict
5.4 Hackers Wanted
5.5 Form a DIY LENA User's Group
6 What are some good links.
6.1 LIWC
httpXGGwwwF—n—lyzewordsF™omG httpXGGliw™FnetG
6.2 LENA
httpXGGwwwFlen—found—tionForgG
6.3 PCAD
httpXGGwwwFg˜Esoftw—reF™omG
6.4 Deb Roy Talk The Birth of a Word
httpXGGwwwFtedF™omGt—lksGde˜•roy•the•˜irth•of•—•wordFhtml
U