Your SlideShare is downloading. ×
M1gp Shimizu - Darwin phone
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Introducing the official SlideShare app

Stunning, full-screen experience for iPhone and Android

Text the download link to your phone

Standard text messaging rates apply

M1gp Shimizu - Darwin phone

284
views

Published on

Published in: Technology, Education

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
284
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
3
Comments
0
Likes
1
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide
  • .
  • Transcript

    • 1. Darwin Phones: the Evolution ofSensing and Inferenceon Mobile Phones
      EmilianoMiluzzo, Cory T. Cornelius, AshwinRamaswamy,TanzeemChoudhury, ZhigangLiu, Andrew T. Campbell
      Mobisys 2010
      Presenter: Kazuto SHIMIZU SezakiLab, Dept. of IST, Univ. of Tokyo
    • 2. Introduction
      Fortunately,
      the presentation the author used at Mobisys 2010 is available on the web site.
      http://www.cs.dartmouth.edu/~miluzzo/publications.html
    • 3. Introduction
      Fortunately,
      the presentation the author used at Mobisys 2010 is available on the web site.
      http://www.cs.dartmouth.edu/~miluzzo/publications.html
      So,
    • 4. Introduction
      Fortunately,
      the presentation the author used at Mobisys 2010 is available on the web site.
      http://www.cs.dartmouth.edu/~miluzzo/publications.html
      Experience Top Conference Quality from Now!!
    • 5. Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones
      Emiliano Miluzzo*, Cory T. Cornelius*, AshwinRamaswamy*, TanzeemChoudhury*, Zhigang Liu**,
      Andrew T. Campbell*
      * CS Department – Dartmouth College
      ** Nokia Research Center – Palo Alto
    • 6. miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 7. evolution of sensing and inference
      on mobile phones
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 8. PR time
      Emiliano Miluzzo
      miluzzo@cs.dartmouth.edu
    • 9. miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 10. miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 11. miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 12. miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 13. miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 14. ok… so what ??
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 15. density
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 16. sensing
      accelerometer
      ….
      digital compass
      microphone
      light sensor/camera
      GPS
      WiFi/bluetooth
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 17. sensing
      ….
      accelerometer
      air quality /
      pollution sensor
      digital compass
      gyroscope
      microphone
      light sensor/camera
      GPS
      WiFi/bluetooth
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 18. programmability
      • free SDK
      - multitasking
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 19. hardware
      - 600 MHz CPU
      - up to 1GB
      application memory
      computation capability is increasing
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 20. application distribution
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 21. application distribution
      deploy apps onto millions of phones at the blink of an eye
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 22. application distribution
      deploy apps onto millions of phones at the blink of an eye
      collect huge amount of data for research purposes
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 23. cloud infrastructure
      cloud - backend support
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 24. cloud infrastructure
      cloud - backend support
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 25. cloud infrastructure
      cloud - backend support
      we want to push intelligence to the phone
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 26. cloud infrastructure
      cloud - backend support
      preserve the phone user experience
      (battery lifetime, ability to make calls, etc.)
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 27. cloud infrastructure
      cloud - backend support
      • sensing
      • 28. run machine learning algorithms locally (feature extraction + inference)
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 29. cloud infrastructure
      cloud - backend support
      run machine learning
      algorithms
      (learning)
      • sensing
      • 30. run machine learning algorithms locally (feature extraction + inference)
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 31. cloud infrastructure
      cloud - backend support
      store and crunch
      big data
      (fusion)
      run machine learning
      algorithms
      (learning)
      • sensing
      • 32. run machine learning algorithms locally (feature extraction + inference)
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 33. cloud infrastructure
      cloud - backend support
      store and crunch
      big data
      (fusion)
      run machine learning
      algorithms
      (learning)
      3 to 5 years from now our phones will be as powerful as a
      • sensing
      • 34. run machine learning algorithms locally (feature extraction + inference)
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 35. cloud infrastructure
      cloud - backend support
      store and crunch
      big data
      (fusion)
      run machine learning
      algorithms
      (learning)
      3 to 5 years from now our phones will be as powerful as a
      • sensing
      • 36. run machine learning algorithms locally (feature extraction + inference)
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 37. cloud infrastructure
      cloud - backend support
      store and crunch
      big data
      (fusion)
      run machine learning
      algorithms
      (learning)
      3 to 5 years from now our phones will be as powerful as a
      • sensing
      • 38. run machine learning algorithms locally (feature extraction + inference)
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 39. cloud infrastructure
      cloud - backend support
      store and crunch
      big data
      (fusion)
      run machine learning
      algorithms
      (learning)
      3 to 5 years from now our phones will be as powerful as a
      • Sensing
      • 40. run machine learning algorithms locally
      (feature extraction + learning + inference)
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 41. programmability
      sensing
      cloud infrastructure
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 42. programmability
      sensing
      ??
      cloud infrastructure
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 43. societal scale sensing
      reality mining using mobile phones
      will play a big role in the future
      global mobilesensor network
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 44. end of PR – now darwin
      Emiliano Miluzzo
      miluzzo@cs.dartmouth.edu
    • 45. a small building block towards the big vision
      Emiliano Miluzzo
      miluzzo@cs.dartmouth.edu
    • 46. from motes to mobile phones
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 47. evolution of sensing and inference
      on mobile phones
      from motes to mobile phones
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 48. evolution of sensing and inference
      on mobile phones
      from motes to mobile phones
      • classification model
      evolution
      darwin
      • classification model
      pooling
      • collaborative
      inference
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 49. darwin
      sensing
      apps
      social context
      microphone
      camera
      audio / pollution / RF
      fingerprinting
      GPS/WiFi/
      cellular
      air quality
      pollution
      image / video
      manipulation
      • classification model
      evolution
      darwin applies distributed computing and collaborative inference concepts to
      mobile sensing systems
      • classification model
      pooling
      • collaborative
      inference
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 50. why darwin?
      mobile phone sensing today
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 51. why darwin?
      mobile phone sensing today
      train classification
      model X in the lab
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 52. why darwin?
      mobile phone sensing today
      train classification
      model X in the lab
      deploy classifier X
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 53. why darwin?
      mobile phone sensing today
      train classification
      model X in the lab
      deploy classifier X
      train classification
      model X’ in the lab
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 54. why darwin?
      mobile phone sensing today
      train classification
      model X in the lab
      deploy classifier X
      train classification
      model X’ in the lab
      deploy classifier X’
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 55. why darwin?
      mobile phone sensing today
      a fully supervised approach doesn’t scale!
      train classification
      model X in the lab
      deploy classifier X
      train classification
      model X’ in the lab
      deploy classifier X’
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 56. why darwin?
      a same classifier does not scale to multiple environments (e.g., quiet and noisy env)
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 57. why darwin?
      a same classifier does not scale to multiple environments (e.g., quiet and noisy env)
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 58. why darwin?
      a same classifier does not scale to multiple environments (e.g., quiet and noisy env)
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 59. why darwin?
      a same classifier does not scale to multiple environments (e.g., quiet and noisy env)
      darwin creates new classification models transparently from the user
      (classification model evolution)
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 60. why darwin?
      ability for an application to
      rapidly scale to many devices
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 61. why darwin?
      darwin re-uses classification models when possible
      (classification model pooling)
      ability for an application to
      rapidly scale to many devices
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 62. why darwin?
      leverage the large
      ensemble of in-situ resources
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 63. why darwin?
      darwin exploits spatial diversity and co-operate to alleviate the “sensing context” problem
      (collaborative inference)
      leverage the large
      ensemble of in-situ resources
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 64. darwin design
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 65. speaker recognition
      (subject to audio noise, sensing context, etc.)
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 66. darwin phases
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 67. darwin phases
      supervised
      initial training (derive model seed)
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 68. darwin phases
      supervised
      initial training (derive model seed)
      unsupervised
      classification model evolution
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 69. darwin phases
      supervised
      initial training (derive model seed)
      unsupervised
      classification model evolution
      classification model pooling
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 70. darwin phases
      supervised
      initial training (derive model seed)
      unsupervised
      classification model evolution
      classification model pooling
      collaborative inference
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 71. classification model training
      sensed event
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 72. classification model training
      filtering (silence suppression + voicing)
      sensed event
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 73. classification model training
      feature
      extraction
      (MFCC)
      filtering (silence suppression + voicing)
      sensed event
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 74. classification model training
      send MFCC to
      backend to train the model
      send model +
      baseline back to phone
      model
      model
      training
      (GMM)
      feature
      extraction
      (MFCC)
      filtering (silence suppression + voicing)
      baseline
      sensed event
      backend
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 75. classification model training
      phone: feature extraction
      (low computation)
      backend: model training
      (high computation)
      backend
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 76. classification model evolution
      phone: determines when to evolve
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 77. classification model evolution
      training
      phone: determines when to evolve
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 78. classification model evolution
      training
      sampled
      phone: determines when to evolve
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 79. classification model evolution
      match?
      YES
      phone: determines when to evolve
      do not evolve
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 80. classification model evolution
      match?
      NO
      phone: determines when to evolve
      evolve
      (train new model using backend as before)
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 81. classification model evolution
      training
      new speaker voice model
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 82. classification model evolution
      training
      new speaker voice model
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 83. classification model evolution
      training
      new speaker voice model
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 84. classification model pooling
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 85. classification model pooling
      Phone B
      Phone A
      Speaker A’s model
      Speaker B’s model
      Speaker B’s model
      Speaker C’s model
      Speaker C’s model
      Phone C
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 86. classification model pooling
      Phone B
      Phone A
      we have two options
      Speaker A’s model
      1. train a new classifier for each speaker (costly for power, inference delay)
      Speaker B’s model
      Speaker B’s model
      Speaker C’s model
      Speaker C’s model
      Phone C
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 87. classification model pooling
      Phone B
      Phone A
      we have two options
      Speaker A’s model
      1. train a new classifier for each speaker (costly for power, inference delay)
      Speaker B’s model
      Speaker B’s model
      2. re-use already available classifiers
      Speaker C’s model
      Speaker C’s model
      Phone C
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 88. classification model pooling
      Phone B
      Phone A
      we have two options
      Speaker A’s model
      1. train a new classifier for each speaker (costly for power, inference delay)
      Speaker B’s model
      Speaker B’s model
      2. re-use already available classifiers
      Speaker C’s model
      Speaker C’s model
      Phone C
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 89. classification model pooling
      Phone B
      Phone A
      Speaker A’s model
      Speaker B’s model
      Speaker B’s model
      Speaker C’s model
      Speaker C’s model
      Phone C
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 90. classification model pooling
      Phone B
      Phone A
      Speaker A’s model
      Speaker A’s model
      Speaker B’s model
      Speaker B’s model
      Speaker C’s model
      Speaker C’s model
      Speaker C’s model
      Phone C
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 91. classification model pooling
      Phone B
      Phone A
      Speaker A’s model
      Speaker A’s model
      Speaker B’s model
      Speaker B’s model
      Speaker B’s model
      Speaker C’s model
      Speaker A’s model
      Speaker C’s model
      Speaker C’s model
      Phone C
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 92. classification model pooling
      Phone B
      Phone A
      Speaker A’s model
      Speaker B’s model
      Speaker B’s model
      Speaker C’s model
      Speaker A’s model
      Speaker C’s model
      Speaker C’s model
      Speaker A’s model
      Phone C
      Speaker B’s model
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 93. classification model pooling
      ready to run the collaborative inference algorithm
      - local inference first
      - final inference later
      Phone B
      Phone A
      Speaker A’s model
      Speaker B’s model
      Speaker B’s model
      Speaker C’s model
      Speaker A’s model
      Speaker C’s model
      Speaker C’s model
      Speaker A’s model
      Phone C
      Speaker B’s model
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 94. collaborative inference
      two phases
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 95. collaborative inference
      two phases
      1. local inference(running independently in parallel on each mobile phone)
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 96. collaborative inference
      two phases
      1. local inference(running independently in parallel on each mobile phone)
      2. final inference(after collecting Local Inference results, to get better confidence about the final classification result)
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 97. collaborative inference
      local inference (LI)
      Phone B
      Phone A
      Phone C
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 98. collaborative inference
      local inference (LI)
      speaker A speaking!!!
      Phone B
      Phone A
      Phone C
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 99. collaborative inference
      local inference (LI)
      speaker A speaking!!!
      Phone B
      Phone A
      B’s LI results:
      Prob(A speaking) = 0.79
      Prob(B speaking) = 0.11
      Prob(C speaking) = 0.10
      A’s LI results:
      Prob(A speaking) = 0.65
      Prob(B speaking) = 0.25
      Prob(C speaking) = 0.10
      C’s LI results:
      Prob(A speaking) = 0.30
      Prob(B speaking) = 0.67
      Prob(C speaking) = 0.03
      Phone C
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 100. collaborative inference
      local inference (LI)
      speaker A speaking!!!
      Phone B
      Phone A
      B’s LI results:
      Prob(A speaking) = 0.79
      Prob(B speaking) = 0.11
      Prob(C speaking) = 0.10
      A’s LI results:
      Prob(A speaking) = 0.65
      Prob(B speaking) = 0.25
      Prob(C speaking) = 0.10
      C’s LI results:
      Prob(A speaking) = 0.30
      Prob(B speaking) = 0.67
      Prob(C speaking) = 0.03
      Phone C
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 101. collaborative inference
      local inference (LI)
      speaker A speaking!!!
      Phone B
      Phone A
      B’s LI results:
      Prob(A speaking) = 0.79
      Prob(B speaking) = 0.11
      Prob(C speaking) = 0.10
      A’s LI results:
      Prob(A speaking) = 0.65
      Prob(B speaking) = 0.25
      Prob(C speaking) = 0.10
      C’s LI results:
      Prob(A speaking) = 0.30
      Prob(B speaking) = 0.67
      Prob(C speaking) = 0.03
      Phone C
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 102. collaborative inference
      local inference (LI)
      speaker A speaking!!!
      Phone B
      Phone A
      B’s LI results:
      Prob(A speaking) = 0.79
      Prob(B speaking) = 0.11
      Prob(C speaking) = 0.10
      A’s LI results:
      Prob(A speaking) = 0.65
      Prob(B speaking) = 0.25
      Prob(C speaking) = 0.10
      C’s LI results:
      Prob(A speaking) = 0.30
      Prob(B speaking) = 0.67
      Prob(C speaking) = 0.03
      Phone C
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 103. collaborative inference
      local inference (LI)
      speaker A speaking!!!
      Phone B
      Phone A
      B’s LI results:
      Prob(A speaking) = 0.79
      Prob(B speaking) = 0.11
      Prob(C speaking) = 0.10
      A’s LI results:
      Prob(A speaking) = 0.65
      Prob(B speaking) = 0.25
      Prob(C speaking) = 0.10
      C’s LI results:
      Prob(A speaking) = 0.30
      Prob(B speaking) = 0.67
      Prob(C speaking) = 0.03
      Phone C
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 104. collaborative inference
      local inference (LI)
      speaker A speaking!!!
      individual classification can be misleading!
      Phone B
      Phone A
      B’s LI results:
      Prob(A speaking) = 0.79
      Prob(B speaking) = 0.11
      Prob(C speaking) = 0.10
      A’s LI results:
      Prob(A speaking) = 0.65
      Prob(B speaking) = 0.25
      Prob(C speaking) = 0.10
      C’s LI results:
      Prob(A speaking) = 0.30
      Prob(B speaking) = 0.67
      Prob(C speaking) = 0.03
      Phone C
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 105. collaborative inference
      final inference (FI)
      each phone gathers LI results
      Phone B
      Phone A
      B’s LI results
      B’s LI results
      B’s LI results
      A’s LI results
      A’s LI results
      A’s LI results
      C’s LI results
      C’s LI results
      C’s LI results
      Phone C
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 106. collaborative inference
      final inference (FI)
      on each phone
      A’s LI results:
      Prob(A speaking) = 0.65
      Prob(B speaking) = 0.25
      Prob(C speaking) = 0.10
      C’s LI results:
      Prob(A speaking) = 0.30
      Prob(B speaking) = 0.67
      Prob(C speaking) = 0.03
      B’s LI results:
      Prob(A speaking) = 0.79
      Prob(B speaking) = 0.11
      Prob(C speaking) = 0.10
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 107. collaborative inference
      final inference (FI)
      on each phone
      A’s LI results:
      Prob(A speaking) = 0.65
      Prob(B speaking) = 0.25
      Prob(C speaking) = 0.10
      C’s LI results:
      Prob(A speaking) = 0.30
      Prob(B speaking) = 0.67
      Prob(C speaking) = 0.03
      B’s LI results:
      Prob(A speaking) = 0.79
      Prob(B speaking) = 0.11
      Prob(C speaking) = 0.10
      x
      x
      x
      x
      x
      x
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 108. collaborative inference
      final inference (FI)
      on each phone
      A’s LI results:
      Prob(A speaking) = 0.65
      Prob(B speaking) = 0.25
      Prob(C speaking) = 0.10
      C’s LI results:
      Prob(A speaking) = 0.30
      Prob(B speaking) = 0.67
      Prob(C speaking) = 0.03
      B’s LI results:
      Prob(A speaking) = 0.79
      Prob(B speaking) = 0.11
      Prob(C speaking) = 0.10
      x
      x
      x
      x
      x
      x
      =
      FI results (normalized):
      Confidence (A speaking) = 1
      Confidence (B speaking) = 0.12
      Confidence (C speaking) = 0.002
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 109. collaborative inference
      final inference (FI)
      on each phone
      A’s LI results:
      Prob(A speaking) = 0.65
      Prob(B speaking) = 0.25
      Prob(C speaking) = 0.10
      C’s LI results:
      Prob(A speaking) = 0.30
      Prob(B speaking) = 0.67
      Prob(C speaking) = 0.03
      B’s LI results:
      Prob(A speaking) = 0.79
      Prob(B speaking) = 0.11
      Prob(C speaking) = 0.10
      x
      x
      x
      x
      x
      x
      =
      FI results (normalized):
      Confidence (A speaking) = 1
      Confidence (B speaking) = 0.12
      Confidence (C speaking) = 0.002
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 110. collaborative inference
      final inference (FI)
      on each phone
      A’s LI results:
      Prob(A speaking) = 0.65
      Prob(B speaking) = 0.25
      Prob(C speaking) = 0.10
      C’s LI results:
      Prob(A speaking) = 0.30
      Prob(B speaking) = 0.67
      Prob(C speaking) = 0.03
      B’s LI results:
      Prob(A speaking) = 0.79
      Prob(B speaking) = 0.11
      Prob(C speaking) = 0.10
      collaborative inference compensates the inaccuracies of individual inferences
      x
      x
      x
      x
      x
      x
      =
      FI results (normalized):
      Confidence (A speaking) = 1
      Confidence (B speaking) = 0.12
      Confidence (C speaking) = 0.002
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 111. evaluation
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 112. evaluation
      C/C++ &
      implemented on Nokia N97 and
      iPhone in support of a speaker recognition app
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 113. evaluation
      C/C++ &
      implemented on Nokia N97 and
      iPhone in support of a speaker recognition app
      unix server
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 114. evaluation
      C/C++ &
      implemented on Nokia N97 and
      iPhone in support of a speaker recognition app
      lightweight reliable protocol to transfer models from the server and between phones
      unix server
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 115. evaluation
      C/C++ &
      implemented on Nokia N97 and
      iPhone in support of a speaker recognition app
      UDP multicast protocol to distribute
      local inference results between phones
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 116. experimental scenarios
      up to eight people in conversation in three different scenarios (quiet indoor, down the street, in a restaurant)
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 117. some numerical results
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 118. need for evolution
      train indoor, evaluate outdoor
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 119. need for evolution
      accuracy
      accuracy improvement after evolution
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 120. indoor quiet scenario
      8 people talking around a table
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 121. indoor quiet scenario
      8 people talking around a table
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 122. indoor quiet scenario
      8 people talking around a table
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 123. indoor quiet scenario
      8 people talking around a table
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 124. indoor quiet scenario
      8 people talking around a table
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 125. indoor quiet scenario
      8 people talking around a table
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 126. indoor quiet scenario
      collaborative inference +
      classification model evolution
      boost the performance of a mobile sensing app
      8 people talking around a table
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 127. impact of the number of mobile phones
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 128. impact of the number of mobile phones
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 129. impact of the number of mobile phones
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 130. impact of the number of mobile phones
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 131. impact of the number of mobile phones
      the larger the number of mobile phones collaborating, the better the final inference result
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 132. battery lifetime Vs inference responsiveness
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 133. battery lifetime Vs inference responsiveness
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 134. battery lifetime Vs inference responsiveness
      high responsiveness
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 135. battery lifetime Vs inference responsiveness
      short battery life
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 136. battery lifetime Vs inference responsiveness
      longer battery duration
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 137. battery lifetime Vs inference responsiveness
      low responsiveness
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 138. battery lifetime Vs inference responsiveness
      smart duty-cycling techniques and machine learning algorithms with better performance in terms of energy usage on mobile phones need to be identified
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 139. a quick recap
      smartphone’s are everywhere, let’s exploit their collective sensing and computation capabilities
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 140. a quick recap
      smartphone’s are everywhere – let’s exploit their collective sensing and computation capabilities
      smartphone sensing opens up new frontiers: applications can be spread and big data collected at unprecedented scale enabling endless research opportunities
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 141. a quick recap
      smartphone’s are everywhere – let’s exploit their collective sensing and computation capabilities
      smartphone sensing opens up new frontiers: applications can be spread and big data collected at unprecedented scale enabling endless research opportunities
      continuous sensing is still challenging; efficient mobile sensing requires to preserve the phone user experience (need for energy efficient ML algorithms and smart duty-cycling techniques)
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 142. a quick recap
      smartphone’s are everywhere – let’s exploit their collective sensing and computation capabilities
      smartphone sensing opens up new frontiers: applications can be spread and big data collected at unprecedented scale enabling endless research opportunities
      continuous sensing is still challenging; efficient mobile sensing requires to preserve the phone user experience (need for energy efficient ML algorithms and smart duty-cycling techniques)
      ML algorithms should perform reliably in the wild
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 143. a quick recap
      smartphone’s are everywhere – let’s exploit their collective sensing and computation capabilities
      ok I think I’m done…
      smartphone sensing opens up new frontiers: applications can be spread and big data collected at unprecedented scale enabling endless research opportunities
      continuous sensing is still challenging; efficient mobile sensing requires to preserve the phone user experience (need for energy efficient ML algorithms and smart duty-cycling techniques)
      ML algorithms should perform reliably in the wild
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 144. a quick recap
      smartphone’s are everywhere – let’s exploit their collective sensing and computation capabilities
      but please bear in mind…
      smartphone sensing opens up new frontiers: applications can be spread and big data collected at unprecedented scale enabling endless research opportunities
      continuous sensing is still challenging; efficient mobile sensing requires to preserve the phone user experience (need for energy efficient ML algorithms and smart duty-cycling techniques)
      ML algorithms should perform reliably in the wild
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 145. Mobile Phone Sensing is the NextBig Thing!
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 146. Thank you!!
      Mobile Sensing Group
      http://sensorlab.cs.dartmouth.edu
      miluzzo@cs.dartmouth.edu
      Emiliano Miluzzo
    • 147. Personal Opinion
      Contribution
      -Implemented the modality of unsupervised labeling
      -Built & implemented concept of collaborative sensing
      Merit
      -Drastic improve of accuracy
      -Shorten learning time
      Future work
      -Energy management
      -Machine resource
    • 148. Thank you
      REFERENCE
      EmilianoMiluzzo, Cory T. Cornelius, AshwinRamaswamy, TanzeemChoudhury, Zhigang Liu, Andrew T. Campbell.
      “Darwin Phone:the Evolution of Sensing and Inference on Mobile Phones,”
      http://www.cs.dartmouth.edu/~miluzzo/publications.html
      Talk(ppt), pdf, video, press available
    • 149. Appendix
    • 150. EmilianoMiluzzo (Ph.D)
      Andrew T. Campbell (Professor) etc…
      Mobile Sensing Group, Dartmouth College, Hanover, NH, USA
      http://sensorlab.cs.dartmouth.edu/index.html
      Author Background
    • 151. Related Work
    • 152. Machine performance
    • 153. Machine performance
    • 154. Machine performance
    • 155. Sample Application
      Speaker Model Computation
      ->MFCC feature extraction   (Mel Frequency CepstramCoefficient,  メル周波数ケプストラム係数)
      Leading approach for speech feature extraction [16,17,42]
      Emphasize the part human use
      Machine learning algorithm
      ->GMM (Gaussian Mixture Model)
      Common to unsupervised machine learning
    • 156. Privacy & Security
      - Store and share not raw data but model & feature (of course protected)
      - User can opt in and out anytime
      - Darwin meets
      Run on trusted device
      Subscribe to trusted system
      Run on trusted application i.e. pre-installed or downloaded from trusted third party.
    • 157. Collaborative Inference
      Individual Inference
      LI = {Speaker1, Speaker2, .. ,Speaker_k}
      Final Inference
    • 158. Evaluation Setting
      Situation
      5 phones
      8 people used
      Several hours a day
      2 weeks
      Voice chunk
      Manually labeled to compare