M1gp Shimizu - Darwin phone

464 views

Published on

Published in: Technology, Education
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
464
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
5
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide
  • .
  • M1gp Shimizu - Darwin phone

    1. 1. Darwin Phones: the Evolution ofSensing and Inferenceon Mobile Phones<br />EmilianoMiluzzo, Cory T. Cornelius, AshwinRamaswamy,TanzeemChoudhury, ZhigangLiu, Andrew T. Campbell<br />Mobisys 2010<br />Presenter: Kazuto SHIMIZU SezakiLab, Dept. of IST, Univ. of Tokyo<br />
    2. 2. Introduction<br />Fortunately, <br />the presentation the author used at Mobisys 2010 is available on the web site.<br />http://www.cs.dartmouth.edu/~miluzzo/publications.html<br />
    3. 3. Introduction<br />Fortunately, <br />the presentation the author used at Mobisys 2010 is available on the web site.<br />http://www.cs.dartmouth.edu/~miluzzo/publications.html<br />So,<br />
    4. 4. Introduction<br />Fortunately, <br />the presentation the author used at Mobisys 2010 is available on the web site.<br />http://www.cs.dartmouth.edu/~miluzzo/publications.html<br />Experience Top Conference Quality from Now!!<br />
    5. 5. Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones<br />Emiliano Miluzzo*, Cory T. Cornelius*, AshwinRamaswamy*, TanzeemChoudhury*, Zhigang Liu**, <br />Andrew T. Campbell*<br />* CS Department – Dartmouth College<br />** Nokia Research Center – Palo Alto<br />
    6. 6. miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    7. 7. evolution of sensing and inference<br />on mobile phones <br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    8. 8. PR time<br />Emiliano Miluzzo<br />miluzzo@cs.dartmouth.edu<br />
    9. 9. miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    10. 10. miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    11. 11. miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    12. 12. miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    13. 13. miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    14. 14. ok… so what ??<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    15. 15. density<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    16. 16. sensing<br />accelerometer<br />….<br />digital compass<br />microphone<br />light sensor/camera<br />GPS<br />WiFi/bluetooth<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    17. 17. sensing<br />….<br />accelerometer<br />air quality /<br />pollution sensor<br />digital compass<br />gyroscope<br />microphone<br />light sensor/camera<br />GPS<br />WiFi/bluetooth<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    18. 18. programmability<br /><ul><li> free SDK</li></ul>- multitasking<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    19. 19. hardware<br />- 600 MHz CPU<br />- up to 1GB <br />application memory <br />computation capability is increasing<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    20. 20. application distribution<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    21. 21. application distribution<br />deploy apps onto millions of phones at the blink of an eye<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    22. 22. application distribution<br />deploy apps onto millions of phones at the blink of an eye<br />collect huge amount of data for research purposes<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    23. 23. cloud infrastructure<br />cloud - backend support<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    24. 24. cloud infrastructure<br />cloud - backend support<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    25. 25. cloud infrastructure<br />cloud - backend support<br />we want to push intelligence to the phone<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    26. 26. cloud infrastructure<br />cloud - backend support<br />preserve the phone user experience<br />(battery lifetime, ability to make calls, etc.)<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    27. 27. cloud infrastructure<br />cloud - backend support<br /><ul><li> sensing
    28. 28. run machine learning algorithms locally (feature extraction + inference)</li></ul>miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    29. 29. cloud infrastructure<br />cloud - backend support<br />run machine learning<br />algorithms <br />(learning)<br /><ul><li> sensing
    30. 30. run machine learning algorithms locally (feature extraction + inference)</li></ul>miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    31. 31. cloud infrastructure<br />cloud - backend support<br />store and crunch <br />big data<br />(fusion)<br />run machine learning<br />algorithms <br />(learning)<br /><ul><li> sensing
    32. 32. run machine learning algorithms locally (feature extraction + inference)</li></ul>miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    33. 33. cloud infrastructure<br />cloud - backend support<br />store and crunch <br />big data<br />(fusion)<br />run machine learning<br />algorithms <br />(learning)<br />3 to 5 years from now our phones will be as powerful as a<br /><ul><li> sensing
    34. 34. run machine learning algorithms locally (feature extraction + inference)</li></ul>miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    35. 35. cloud infrastructure<br />cloud - backend support<br />store and crunch <br />big data<br />(fusion)<br />run machine learning<br />algorithms <br />(learning)<br />3 to 5 years from now our phones will be as powerful as a<br /><ul><li> sensing
    36. 36. run machine learning algorithms locally (feature extraction + inference)</li></ul>miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    37. 37. cloud infrastructure<br />cloud - backend support<br />store and crunch <br />big data<br />(fusion)<br />run machine learning<br />algorithms <br />(learning)<br />3 to 5 years from now our phones will be as powerful as a<br /><ul><li> sensing
    38. 38. run machine learning algorithms locally (feature extraction + inference)</li></ul>miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    39. 39. cloud infrastructure<br />cloud - backend support<br />store and crunch <br />big data<br />(fusion)<br />run machine learning<br />algorithms <br />(learning)<br />3 to 5 years from now our phones will be as powerful as a<br /><ul><li> Sensing
    40. 40. run machine learning algorithms locally</li></ul>(feature extraction + learning + inference)<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    41. 41. programmability<br />sensing<br /> cloud infrastructure<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    42. 42. programmability<br />sensing<br />??<br /> cloud infrastructure<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    43. 43. societal scale sensing<br /> reality mining using mobile phones <br /> will play a big role in the future<br /> global mobilesensor network<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    44. 44. end of PR – now darwin<br />Emiliano Miluzzo<br />miluzzo@cs.dartmouth.edu<br />
    45. 45. a small building block towards the big vision<br />Emiliano Miluzzo<br />miluzzo@cs.dartmouth.edu<br />
    46. 46. from motes to mobile phones<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    47. 47. evolution of sensing and inference<br />on mobile phones <br />from motes to mobile phones<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    48. 48. evolution of sensing and inference<br />on mobile phones <br />from motes to mobile phones<br /><ul><li> classification model</li></ul> evolution<br />darwin<br /><ul><li> classification model</li></ul> pooling<br /><ul><li> collaborative </li></ul> inference<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    49. 49. darwin<br />sensing<br />apps<br />social context<br />microphone<br />camera<br />audio / pollution / RF<br /> fingerprinting<br />GPS/WiFi/<br />cellular<br />air quality <br />pollution<br /> image / video<br /> manipulation<br /><ul><li> classification model</li></ul> evolution<br />darwin applies distributed computing and collaborative inference concepts to <br />mobile sensing systems <br /><ul><li> classification model</li></ul> pooling<br /><ul><li> collaborative </li></ul> inference<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    50. 50. why darwin?<br />mobile phone sensing today<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    51. 51. why darwin?<br />mobile phone sensing today<br /> train classification <br />model X in the lab<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    52. 52. why darwin?<br />mobile phone sensing today<br /> train classification <br />model X in the lab<br />deploy classifier X<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    53. 53. why darwin?<br />mobile phone sensing today<br /> train classification <br />model X in the lab<br />deploy classifier X<br />train classification <br />model X’ in the lab<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    54. 54. why darwin?<br />mobile phone sensing today<br /> train classification <br />model X in the lab<br />deploy classifier X<br />train classification <br />model X’ in the lab<br />deploy classifier X’<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    55. 55. why darwin?<br />mobile phone sensing today<br />a fully supervised approach doesn’t scale!<br /> train classification <br />model X in the lab<br />deploy classifier X<br />train classification <br />model X’ in the lab<br />deploy classifier X’<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    56. 56. why darwin?<br /> a same classifier does not scale to multiple environments (e.g., quiet and noisy env)<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    57. 57. why darwin?<br /> a same classifier does not scale to multiple environments (e.g., quiet and noisy env)<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    58. 58. why darwin?<br /> a same classifier does not scale to multiple environments (e.g., quiet and noisy env)<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    59. 59. why darwin?<br /> a same classifier does not scale to multiple environments (e.g., quiet and noisy env)<br />darwin creates new classification models transparently from the user<br />(classification model evolution)<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    60. 60. why darwin?<br />ability for an application to<br />rapidly scale to many devices<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    61. 61. why darwin?<br />darwin re-uses classification models when possible<br />(classification model pooling)<br />ability for an application to<br />rapidly scale to many devices<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    62. 62. why darwin?<br />leverage the large<br /> ensemble of in-situ resources <br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    63. 63. why darwin?<br />darwin exploits spatial diversity and co-operate to alleviate the “sensing context” problem<br />(collaborative inference)<br />leverage the large<br /> ensemble of in-situ resources <br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    64. 64. darwin design<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    65. 65. speaker recognition<br /> (subject to audio noise, sensing context, etc.) <br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    66. 66. darwin phases<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    67. 67. darwin phases<br />supervised<br />initial training (derive model seed)<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    68. 68. darwin phases<br />supervised<br />initial training (derive model seed)<br />unsupervised<br />classification model evolution<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    69. 69. darwin phases<br />supervised<br />initial training (derive model seed)<br />unsupervised<br />classification model evolution<br />classification model pooling<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    70. 70. darwin phases<br />supervised<br />initial training (derive model seed)<br />unsupervised<br />classification model evolution<br />classification model pooling<br />collaborative inference<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    71. 71. classification model training<br />sensed event<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    72. 72. classification model training<br />filtering (silence suppression + voicing)<br />sensed event<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    73. 73. classification model training<br />feature<br />extraction<br />(MFCC)<br />filtering (silence suppression + voicing)<br />sensed event<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    74. 74. classification model training<br />send MFCC to<br />backend to train the model<br />send model + <br />baseline back to phone<br />model<br />model<br />training<br />(GMM)<br />feature<br />extraction<br />(MFCC)<br />filtering (silence suppression + voicing)<br />baseline<br />sensed event<br /> backend<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    75. 75. classification model training<br />phone: feature extraction<br /> (low computation)<br />backend: model training<br /> (high computation)<br /> backend<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    76. 76. classification model evolution<br />phone: determines when to evolve<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    77. 77. classification model evolution<br />training<br />phone: determines when to evolve<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    78. 78. classification model evolution<br />training<br />sampled<br />phone: determines when to evolve<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    79. 79. classification model evolution<br />match?<br />YES<br />phone: determines when to evolve<br />do not evolve<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    80. 80. classification model evolution<br />match?<br />NO<br />phone: determines when to evolve<br />evolve<br />(train new model using backend as before)<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    81. 81. classification model evolution<br />training<br /> new speaker voice model<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    82. 82. classification model evolution<br />training<br /> new speaker voice model<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    83. 83. classification model evolution<br />training<br /> new speaker voice model<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    84. 84. classification model pooling<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    85. 85. classification model pooling<br />Phone B<br />Phone A<br />Speaker A’s model<br />Speaker B’s model<br />Speaker B’s model<br />Speaker C’s model<br />Speaker C’s model<br /> Phone C<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    86. 86. classification model pooling<br />Phone B<br />Phone A<br />we have two options<br />Speaker A’s model<br />1. train a new classifier for each speaker (costly for power, inference delay)<br />Speaker B’s model<br />Speaker B’s model<br />Speaker C’s model<br />Speaker C’s model<br /> Phone C<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    87. 87. classification model pooling<br />Phone B<br />Phone A<br />we have two options<br />Speaker A’s model<br />1. train a new classifier for each speaker (costly for power, inference delay)<br />Speaker B’s model<br />Speaker B’s model<br />2. re-use already available classifiers<br />Speaker C’s model<br />Speaker C’s model<br /> Phone C<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    88. 88. classification model pooling<br />Phone B<br />Phone A<br />we have two options<br />Speaker A’s model<br />1. train a new classifier for each speaker (costly for power, inference delay)<br />Speaker B’s model<br />Speaker B’s model<br />2. re-use already available classifiers<br />Speaker C’s model<br />Speaker C’s model<br /> Phone C<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    89. 89. classification model pooling<br />Phone B<br />Phone A<br />Speaker A’s model<br />Speaker B’s model<br />Speaker B’s model<br />Speaker C’s model<br />Speaker C’s model<br /> Phone C<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    90. 90. classification model pooling<br />Phone B<br />Phone A<br />Speaker A’s model<br />Speaker A’s model<br />Speaker B’s model<br />Speaker B’s model<br />Speaker C’s model<br />Speaker C’s model<br />Speaker C’s model<br /> Phone C<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    91. 91. classification model pooling<br />Phone B<br />Phone A<br />Speaker A’s model<br />Speaker A’s model<br />Speaker B’s model<br />Speaker B’s model<br />Speaker B’s model<br />Speaker C’s model<br />Speaker A’s model<br />Speaker C’s model<br />Speaker C’s model<br /> Phone C<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    92. 92. classification model pooling<br />Phone B<br />Phone A<br />Speaker A’s model<br />Speaker B’s model<br />Speaker B’s model<br />Speaker C’s model<br />Speaker A’s model<br />Speaker C’s model<br />Speaker C’s model<br />Speaker A’s model<br /> Phone C<br />Speaker B’s model<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    93. 93. classification model pooling<br />ready to run the collaborative inference algorithm<br />- local inference first<br />- final inference later<br />Phone B<br />Phone A<br />Speaker A’s model<br />Speaker B’s model<br />Speaker B’s model<br />Speaker C’s model<br />Speaker A’s model<br />Speaker C’s model<br />Speaker C’s model<br />Speaker A’s model<br /> Phone C<br />Speaker B’s model<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    94. 94. collaborative inference<br />two phases<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    95. 95. collaborative inference<br />two phases<br />1. local inference(running independently in parallel on each mobile phone)<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    96. 96. collaborative inference<br />two phases<br />1. local inference(running independently in parallel on each mobile phone)<br />2. final inference(after collecting Local Inference results, to get better confidence about the final classification result)<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    97. 97. collaborative inference<br />local inference (LI)<br />Phone B<br />Phone A<br /> Phone C<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    98. 98. collaborative inference<br />local inference (LI)<br />speaker A speaking!!!<br />Phone B<br />Phone A<br /> Phone C<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    99. 99. collaborative inference<br />local inference (LI)<br />speaker A speaking!!!<br />Phone B<br />Phone A<br />B’s LI results:<br />Prob(A speaking) = 0.79<br />Prob(B speaking) = 0.11<br />Prob(C speaking) = 0.10<br />A’s LI results:<br />Prob(A speaking) = 0.65<br />Prob(B speaking) = 0.25<br />Prob(C speaking) = 0.10<br />C’s LI results:<br />Prob(A speaking) = 0.30<br />Prob(B speaking) = 0.67<br />Prob(C speaking) = 0.03<br /> Phone C<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    100. 100. collaborative inference<br />local inference (LI)<br />speaker A speaking!!!<br />Phone B<br />Phone A<br />B’s LI results:<br />Prob(A speaking) = 0.79<br />Prob(B speaking) = 0.11<br />Prob(C speaking) = 0.10<br />A’s LI results:<br />Prob(A speaking) = 0.65<br />Prob(B speaking) = 0.25<br />Prob(C speaking) = 0.10<br />C’s LI results:<br />Prob(A speaking) = 0.30<br />Prob(B speaking) = 0.67<br />Prob(C speaking) = 0.03<br /> Phone C<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    101. 101. collaborative inference<br />local inference (LI)<br />speaker A speaking!!!<br />Phone B<br />Phone A<br />B’s LI results:<br />Prob(A speaking) = 0.79<br />Prob(B speaking) = 0.11<br />Prob(C speaking) = 0.10<br />A’s LI results:<br />Prob(A speaking) = 0.65<br />Prob(B speaking) = 0.25<br />Prob(C speaking) = 0.10<br />C’s LI results:<br />Prob(A speaking) = 0.30<br />Prob(B speaking) = 0.67<br />Prob(C speaking) = 0.03<br /> Phone C<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    102. 102. collaborative inference<br />local inference (LI)<br />speaker A speaking!!!<br />Phone B<br />Phone A<br />B’s LI results:<br />Prob(A speaking) = 0.79<br />Prob(B speaking) = 0.11<br />Prob(C speaking) = 0.10<br />A’s LI results:<br />Prob(A speaking) = 0.65<br />Prob(B speaking) = 0.25<br />Prob(C speaking) = 0.10<br />C’s LI results:<br />Prob(A speaking) = 0.30<br />Prob(B speaking) = 0.67<br />Prob(C speaking) = 0.03<br /> Phone C<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    103. 103. collaborative inference<br />local inference (LI)<br />speaker A speaking!!!<br />Phone B<br />Phone A<br />B’s LI results:<br />Prob(A speaking) = 0.79<br />Prob(B speaking) = 0.11<br />Prob(C speaking) = 0.10<br />A’s LI results:<br />Prob(A speaking) = 0.65<br />Prob(B speaking) = 0.25<br />Prob(C speaking) = 0.10<br />C’s LI results:<br />Prob(A speaking) = 0.30<br />Prob(B speaking) = 0.67<br />Prob(C speaking) = 0.03<br /> Phone C<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    104. 104. collaborative inference<br />local inference (LI)<br />speaker A speaking!!!<br />individual classification can be misleading!<br />Phone B<br />Phone A<br />B’s LI results:<br />Prob(A speaking) = 0.79<br />Prob(B speaking) = 0.11<br />Prob(C speaking) = 0.10<br />A’s LI results:<br />Prob(A speaking) = 0.65<br />Prob(B speaking) = 0.25<br />Prob(C speaking) = 0.10<br />C’s LI results:<br />Prob(A speaking) = 0.30<br />Prob(B speaking) = 0.67<br />Prob(C speaking) = 0.03<br /> Phone C<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    105. 105. collaborative inference<br />final inference (FI)<br />each phone gathers LI results<br />Phone B<br />Phone A<br />B’s LI results<br />B’s LI results<br />B’s LI results<br />A’s LI results<br />A’s LI results<br />A’s LI results<br />C’s LI results<br />C’s LI results<br />C’s LI results<br /> Phone C<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    106. 106. collaborative inference<br />final inference (FI)<br />on each phone<br />A’s LI results:<br />Prob(A speaking) = 0.65<br />Prob(B speaking) = 0.25<br />Prob(C speaking) = 0.10<br />C’s LI results:<br />Prob(A speaking) = 0.30<br />Prob(B speaking) = 0.67<br />Prob(C speaking) = 0.03<br />B’s LI results:<br />Prob(A speaking) = 0.79<br />Prob(B speaking) = 0.11<br />Prob(C speaking) = 0.10<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    107. 107. collaborative inference<br />final inference (FI)<br />on each phone<br />A’s LI results:<br />Prob(A speaking) = 0.65<br />Prob(B speaking) = 0.25<br />Prob(C speaking) = 0.10<br />C’s LI results:<br />Prob(A speaking) = 0.30<br />Prob(B speaking) = 0.67<br />Prob(C speaking) = 0.03<br />B’s LI results:<br />Prob(A speaking) = 0.79<br />Prob(B speaking) = 0.11<br />Prob(C speaking) = 0.10<br />x<br />x<br />x<br />x<br />x<br />x<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    108. 108. collaborative inference<br />final inference (FI)<br />on each phone<br />A’s LI results:<br />Prob(A speaking) = 0.65<br />Prob(B speaking) = 0.25<br />Prob(C speaking) = 0.10<br />C’s LI results:<br />Prob(A speaking) = 0.30<br />Prob(B speaking) = 0.67<br />Prob(C speaking) = 0.03<br />B’s LI results:<br />Prob(A speaking) = 0.79<br />Prob(B speaking) = 0.11<br />Prob(C speaking) = 0.10<br />x<br />x<br />x<br />x<br />x<br />x<br />=<br />FI results (normalized):<br /> Confidence (A speaking) = 1 <br /> Confidence (B speaking) = 0.12<br /> Confidence (C speaking) = 0.002<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    109. 109. collaborative inference<br />final inference (FI)<br />on each phone<br />A’s LI results:<br />Prob(A speaking) = 0.65<br />Prob(B speaking) = 0.25<br />Prob(C speaking) = 0.10<br />C’s LI results:<br />Prob(A speaking) = 0.30<br />Prob(B speaking) = 0.67<br />Prob(C speaking) = 0.03<br />B’s LI results:<br />Prob(A speaking) = 0.79<br />Prob(B speaking) = 0.11<br />Prob(C speaking) = 0.10<br />x<br />x<br />x<br />x<br />x<br />x<br />=<br />FI results (normalized):<br /> Confidence (A speaking) = 1 <br /> Confidence (B speaking) = 0.12<br /> Confidence (C speaking) = 0.002<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    110. 110. collaborative inference<br />final inference (FI)<br />on each phone<br />A’s LI results:<br />Prob(A speaking) = 0.65<br />Prob(B speaking) = 0.25<br />Prob(C speaking) = 0.10<br />C’s LI results:<br />Prob(A speaking) = 0.30<br />Prob(B speaking) = 0.67<br />Prob(C speaking) = 0.03<br />B’s LI results:<br />Prob(A speaking) = 0.79<br />Prob(B speaking) = 0.11<br />Prob(C speaking) = 0.10<br />collaborative inference compensates the inaccuracies of individual inferences <br />x<br />x<br />x<br />x<br />x<br />x<br />=<br />FI results (normalized):<br /> Confidence (A speaking) = 1 <br /> Confidence (B speaking) = 0.12<br /> Confidence (C speaking) = 0.002<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    111. 111. evaluation<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    112. 112. evaluation<br /> C/C++ & <br />implemented on Nokia N97 and<br />iPhone in support of a speaker recognition app<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    113. 113. evaluation<br /> C/C++ & <br />implemented on Nokia N97 and<br />iPhone in support of a speaker recognition app<br />unix server<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    114. 114. evaluation<br /> C/C++ & <br />implemented on Nokia N97 and<br />iPhone in support of a speaker recognition app<br />lightweight reliable protocol to transfer models from the server and between phones<br />unix server<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    115. 115. evaluation<br /> C/C++ & <br />implemented on Nokia N97 and<br />iPhone in support of a speaker recognition app<br />UDP multicast protocol to distribute<br />local inference results between phones<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    116. 116. experimental scenarios<br />up to eight people in conversation in three different scenarios (quiet indoor, down the street, in a restaurant)<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    117. 117. some numerical results<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    118. 118. need for evolution<br /> train indoor, evaluate outdoor<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    119. 119. need for evolution<br />accuracy<br /> accuracy improvement after evolution<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    120. 120. indoor quiet scenario<br /> 8 people talking around a table<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    121. 121. indoor quiet scenario<br /> 8 people talking around a table<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    122. 122. indoor quiet scenario<br /> 8 people talking around a table<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    123. 123. indoor quiet scenario<br /> 8 people talking around a table<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    124. 124. indoor quiet scenario<br /> 8 people talking around a table<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    125. 125. indoor quiet scenario<br /> 8 people talking around a table<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    126. 126. indoor quiet scenario<br />collaborative inference + <br />classification model evolution <br />boost the performance of a mobile sensing app <br /> 8 people talking around a table<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    127. 127. impact of the number of mobile phones<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    128. 128. impact of the number of mobile phones<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    129. 129. impact of the number of mobile phones<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    130. 130. impact of the number of mobile phones<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    131. 131. impact of the number of mobile phones<br />the larger the number of mobile phones collaborating, the better the final inference result<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    132. 132. battery lifetime Vs inference responsiveness<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    133. 133. battery lifetime Vs inference responsiveness<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    134. 134. battery lifetime Vs inference responsiveness<br />high responsiveness<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    135. 135. battery lifetime Vs inference responsiveness<br />short battery life<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    136. 136. battery lifetime Vs inference responsiveness<br />longer battery duration<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    137. 137. battery lifetime Vs inference responsiveness<br />low responsiveness<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    138. 138. battery lifetime Vs inference responsiveness<br />smart duty-cycling techniques and machine learning algorithms with better performance in terms of energy usage on mobile phones need to be identified<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    139. 139. a quick recap<br />smartphone’s are everywhere, let’s exploit their collective sensing and computation capabilities<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    140. 140. a quick recap<br />smartphone’s are everywhere – let’s exploit their collective sensing and computation capabilities<br />smartphone sensing opens up new frontiers: applications can be spread and big data collected at unprecedented scale enabling endless research opportunities<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    141. 141. a quick recap<br />smartphone’s are everywhere – let’s exploit their collective sensing and computation capabilities<br />smartphone sensing opens up new frontiers: applications can be spread and big data collected at unprecedented scale enabling endless research opportunities<br />continuous sensing is still challenging; efficient mobile sensing requires to preserve the phone user experience (need for energy efficient ML algorithms and smart duty-cycling techniques) <br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    142. 142. a quick recap<br />smartphone’s are everywhere – let’s exploit their collective sensing and computation capabilities<br />smartphone sensing opens up new frontiers: applications can be spread and big data collected at unprecedented scale enabling endless research opportunities<br />continuous sensing is still challenging; efficient mobile sensing requires to preserve the phone user experience (need for energy efficient ML algorithms and smart duty-cycling techniques) <br />ML algorithms should perform reliably in the wild <br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    143. 143. a quick recap<br />smartphone’s are everywhere – let’s exploit their collective sensing and computation capabilities<br />ok I think I’m done…<br />smartphone sensing opens up new frontiers: applications can be spread and big data collected at unprecedented scale enabling endless research opportunities<br />continuous sensing is still challenging; efficient mobile sensing requires to preserve the phone user experience (need for energy efficient ML algorithms and smart duty-cycling techniques) <br />ML algorithms should perform reliably in the wild <br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    144. 144. a quick recap<br />smartphone’s are everywhere – let’s exploit their collective sensing and computation capabilities<br />but please bear in mind…<br />smartphone sensing opens up new frontiers: applications can be spread and big data collected at unprecedented scale enabling endless research opportunities<br />continuous sensing is still challenging; efficient mobile sensing requires to preserve the phone user experience (need for energy efficient ML algorithms and smart duty-cycling techniques) <br />ML algorithms should perform reliably in the wild <br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    145. 145. Mobile Phone Sensing is the NextBig Thing!<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    146. 146. Thank you!! <br />Mobile Sensing Group<br />http://sensorlab.cs.dartmouth.edu<br />miluzzo@cs.dartmouth.edu<br />Emiliano Miluzzo<br />
    147. 147. Personal Opinion<br />Contribution<br />-Implemented the modality of unsupervised labeling<br />-Built & implemented concept of collaborative sensing<br />Merit<br />-Drastic improve of accuracy <br />-Shorten learning time<br />Future work<br />-Energy management<br />-Machine resource<br />
    148. 148. Thank you<br />REFERENCE<br />EmilianoMiluzzo, Cory T. Cornelius, AshwinRamaswamy, TanzeemChoudhury, Zhigang Liu, Andrew T. Campbell.<br />“Darwin Phone:the Evolution of Sensing and Inference on Mobile Phones,” <br />http://www.cs.dartmouth.edu/~miluzzo/publications.html<br />Talk(ppt), pdf, video, press available<br />
    149. 149. Appendix<br />
    150. 150. EmilianoMiluzzo (Ph.D) <br />Andrew T. Campbell (Professor) etc…<br />Mobile Sensing Group, Dartmouth College, Hanover, NH, USA<br />http://sensorlab.cs.dartmouth.edu/index.html<br />Author Background<br />
    151. 151. Related Work<br />
    152. 152. Machine performance<br />
    153. 153. Machine performance<br />
    154. 154. Machine performance<br />
    155. 155. Sample Application<br />Speaker Model Computation<br />->MFCC feature extraction   (Mel Frequency CepstramCoefficient,  メル周波数ケプストラム係数)<br />Leading approach for speech feature extraction [16,17,42]<br />Emphasize the part human use<br />Machine learning algorithm<br />->GMM (Gaussian Mixture Model)<br />Common to unsupervised machine learning<br />
    156. 156. Privacy & Security<br />- Store and share not raw data but model & feature (of course protected)<br />- User can opt in and out anytime<br />- Darwin meets<br />Run on trusted device<br />Subscribe to trusted system<br />Run on trusted application i.e. pre-installed or downloaded from trusted third party.<br />
    157. 157. Collaborative Inference<br />Individual Inference<br />LI = {Speaker1, Speaker2, .. ,Speaker_k}<br />Final Inference<br />
    158. 158. Evaluation Setting<br />Situation<br />5 phones<br />8 people used<br />Several hours a day<br />2 weeks <br />Voice chunk<br />Manually labeled to compare<br />

    ×