Presented by Audio Analytic's Director of R&D, Sacha Krstulovic, at 2017's SANE workshop at Google's offices in New York. The recognition of audio events is emerging as a relatively new field of research compared to speech and music recognition. Whereas it has started from known recipes from the latter fields, 24/7 sound recognition actually defines a new range of research problems which are distinct from speech and music. After reviewing the constraints related to running sound recognition successfully on real world consumer products deployed across thousands of homes, the talk discusses the nature of some of sound recognition’s distinctive problems, such as open set recognition or the modelling of interrupted sequences. This is put in context with the most recent advances in the field, supported in the public domain, e.g., by competitive evaluations such as the DCASE challenge, to assess which of sound recognition’s distinctive problems are being currently addressed by state-of-the-art methods, and which ones could deserve more attention.

1. © Audio Analytic Ltd, 2017
Sacha Krstulović
Sound is not Speech
SANE Workshop, New York - October 19th 2017
Director of AALabs – Audio Analytic Ltd.

2. © Audio Analytic Ltd, 2017
The missing piece of the AI puzzle
2

3. © Audio Analytic Ltd, 2017
The missing piece of the AI puzzle
3
Speech AI:
• Speech recognition / synthesis:
natural speech interaction, dialogue
• Biometric voice recognition:
identity, personalisation
• Machine translation
Image AI:
• Face recognition:
identity, security, personalisation
• Video processing:
activity, security, presence
Music AI:
• Fingerprinting:
entertainment, information
• Query by humming:
entertainment, information
Sound AI?
• Sound recognition:
context, attention, presence,
security, entertainment
• Scene recognition:
context, activity

4. © Audio Analytic Ltd, 2017
For the first time, devices can
intelligently respond to sound.
• We do sound recognition software and
algorithms
• Founded in 2010
• Based in Cambridge, UK and Palo Alto, USA
• Over 40 people
• Experts…
• in machine listening
• in sound recognition
• in software engineering
• Venture-backed company
Audio Analytic:

5. © Audio Analytic Ltd, 2017
“An AI start-up like no other… like a
Shazam for real-world sounds”
Bloomberg

6. © Audio Analytic Ltd, 2017© Audio Analytic Ltd, 2017
• Smart speakers and smart home devices
• Support the wellbeing of family, loved ones and
possessions by recognising and responding to:
Active market focus
• Window glass break
• Smoke and CO alarm
• Baby cry
• Dog bark
• Anomaly
• Voice presence
• All technology will become context aware and
intelligent
• New opportunities to enhance wellness,
entertainment and social interaction
• Hearables, wearables, VR, gaming, automotive,
smart home, buildings, mobile and more…
Expanding market focus

7. © Audio Analytic Ltd, 2017
Sound is not speech.

8. © Audio Analytic Ltd, 2017
Is there a language of sounds?
• Speech is bounded by language.
• Language is predictable and enumerable:
English language:171,476 words current use,
47,156 obsolete words, 9,500 derivative words
Wall Street Journal
“Hello [???]”
• e.g., end-to-end speech recognition:
CTC networks address the direct prediction
of labels from audio
𝑝 𝑙 𝑥 =
𝜋∈𝐵−1(𝑙)
𝑝(𝜋|𝑥)
(Alex Graves & al, 2006)
• What about environmental sounds?
Is their occurrence predictable?
“What sound will I make now?”
• Nonetheless, sounds tell us something:
• Intentionally communicative
e.g., smoke alarm, alert sounds
• Incidental cues
e.g., glass break, vacuum cleaner
• Environmental
e.g., aircon, babble; wind, rain
• And the notion of generative model remains
somewhat valid.
𝑝 smoke alarm beep )
• Oh, wait a minute:
𝑝( smoke alarm | beep and lots of garbage )
8

9. © Audio Analytic Ltd, 2017
The variety of production processes…
9

10. © Audio Analytic Ltd, 2017
… leads to a variety of acoustic features.
Beeps Harmonic Sounds
Crash/Bangs Shaped noise

11. © Audio Analytic Ltd, 2017
Should acoustic features be hand-crafted or learned?
11
• Most standard acoustic features were invented by looking at spectrogram images.
• If image recognition can infer features, then why not… => Convolutive Neural Networks.

12. © Audio Analytic Ltd, 2017
Are acoustic features enough?
• Temporal modelling might help.
12

13. © Audio Analytic Ltd, 2017
DCASE 2017 data challenge - Task 2 results
• DCASE: comparative evaluation.
Task 2: detection of rare events.
• Baby cry, glass break, gun shots
in background noise
• Convolutive Recursive
Neural Networks (CRNNs)
stole the show!
• (But has anybody actually
tried anything else?)
13
DCASE 2017 T2: H. Lim, J. Park and Y. Han E. Cakir and T. Virtanen

14. © Audio Analytic Ltd, 2017
Interrupted sequences
• T3 pattern:
14
0.5s
beep
0.5s
of silence
1s
of silence

15. © Audio Analytic Ltd, 2017
1s
of ANYTHING!
0.5s
of ANYTHING!
Interrupted sequences
• T3 pattern:
Out of the 4 second sequence, 62.5% of the acoustics
do not predict anything about the smoke alarm!
Perhaps need to model some attention mechanism?
15
0.5s
beep
“Attention and Localization Based on a Deep Convolutional
Recurrent Model for Weakly Supervised Audio Tagging”
Y. Xu, Q. Kong, Q. Huang, W. Wang and M. D. Plumbley
Interspeech 2017
DCASE 2016 Task4

16. © Audio Analytic Ltd, 2017
• The variability of the non-target set is very large.
• Data balance: by nature, the non-target set is much larger than the target set.
• Open set: model a ball around the target data and some measure of “outlierism”?
Early results in vision and forensics: “The open set recognition problem”, A. Rocha and W. Scheirer, ICIP 2016
24/7 sound recognition: perhaps an open-set problem?
16
Target Non-target
Confusion
matrix:
1
∞

17. © Audio Analytic Ltd, 2017
Research is evolving: paradigm shift
• From the explicit modelling of phenomena
• MFCCs related to audition, generative models,
Markov chains, factorization, sparsity etc.
• To heavily data-driven DNN models
• Bottleneck features, posteriors from FFDNNs
• To higher level functions achieved in principle by DNNs
• Feature extraction => CNNs, Temporal modelling => RNNs,
Attention networks, CTC etc.
• Largely evaluation driven
• “Assuming that the network architecture does X,
evaluation shows that it improves the rates by Y%.”
• Data is a parameter.
• Some attempts at interpretation.
But this could all be a massive horse!
17

18. © Audio Analytic Ltd, 2017
The “real world”…

19. © Audio Analytic Ltd, 2017
Channel variability
19

20. © Audio Analytic Ltd, 2017
Running on the edge
• Speech may use Cloud Computing.
What about sound recognition
for consumer products?
• Running on the edge has lots of value
• Distributed computing
• Privacy concerns
• Reliability
• Real time
• etc.
• => Running on embedded systems requires
to optimise the computational cost.
20
(Image: Pubnub.com)

21. © Audio Analytic Ltd, 2017
Which machine yields the best bang for MIPS?
Sigtia & al., “Automatic Environmental Sound Recognition: Performance versus
Computational Cost”, IEEE/ACM Trans. ASLP, Vol.24 Issue 11, Nov. 2016, pp.2096-2107
21

22. © Audio Analytic Ltd, 2017
Summary: why sound is not speech
It is not exactly the same problem as speech or music:
• Not bounded by language or musical theory.
• Diversity of production processes and acoustic features.
• Temporal structure matters, but involves interruption.
• One against many: open-set recognition.
Additional topics for industrial impact:
• Robustness to channel and room responses is crucial.
• Running on the edge matters, computational cost
is a tangible question.
This is a new type of AI in its own right, and a new research community is forming around it.
22

23. © Audio Analytic Ltd, 2017
We are hiring!
https://www.AudioAnalytic.com/careers/
23

24. UK headquarters
2 Quayside
Cambridge
CB5 8AB, UK
info@audioanalytic.com
audioanalytic.com
US office
3505 El Camino Real
Palo Alto
CA 94306, USA
© Audio Analytic Ltd, 2017
v1
Thank you

25. © Audio Analytic Ltd, 2017