Designing high impact acoustics apps for cross-species communication research, principally Android apps for underwater whistles emission and recognition with free dolphins

1. Designing High Impact Acoustics
Apps for Cross-Species
Communication Research
Serge Masse
Montreal
2018
!1

2. Data Example - Using an Android App
!2

3. Valtamer/Samsung Android Tablet
!3

4. Are you…
Looking to improve the world with your apps?
Looking for a field to disrupt with your work?
Looking for very difficult apps to develop?
Looking for great scientists to help?
Looking for good reasons to go to sea and travel around the world?
Don't know what you're looking for?
!4

5. About Me
• Informaticien – s/w engineer – since ~1973
• claiming a few world firsts (ex.: in trains, telco micro-economics, 747 autopilot flight testing, stock market
buy&sell, dolphins)
• one app deemed impossible (ex.: an app that writes other apps in production for complex telco switches)
• some life-critical (ex.: trains, airplanes, pesticides)
• invented and used my own tools (ex.: event-based modelling, knowledge maps)
• designed multiple expert systems in production (gofai)
• said “no, sorry” to G twice
• developing open-source library for acoustic communication apps, on GitLab
• 2 apps on G Play, a third one coming soon
!5

6. Do you know the species?
!6
Species?

7. Gamers for Orcas – Trendy?
https://tiltify.com/@gamersfororcas/official-livestream/about
April 2018 – the streaming has now ended
!7

8. Designing X-Species Comm Apps
• Part 1: Markets, main players, potential users, partners, competitors,
antagonists, and a bit of history
• Part 2: Android – app design and technical issues – AI/aNN
• Part 3: Features – Required & Recommended
• Part 4: Usage issues
!8

9. Your potential user/competitor: Dr Denise Herzing
et al.
• Wild Dolphin Project – WDP
• http://www.wilddolphinproject.org/membership/the-chat-society/
• Dr. Thad Starner – Google Glass Tech Lead
• Two-way device, with some AI
• The Bahamas: > 30 years, same families of dolphins
!9

10. Your newish competitors/partners/users:
• Fabrice Schnöller: www.darewin.org – UN, VR, videos
• www.gavagai.se – AI, Davos 2018, with Gov of Sweden
• http://monacoexplorations.org/ – Prince of Monaco
• https://www.dolphinembassy.org/ – Tenerife, mostly Europeans, esoteric,
films ($), sail boats
• www.kintocetaceans.org
• cetalingua.com – AI?, $ for lessons? maybe a kind of scam?
!10

11. Your potential user/partner: Scott Veirs et al.
• https://www.orcasound.net/ “Orcasound connects your headphones to live
hydrophones”
• https://www.google.com/maps/d/u/0/viewer?
mid=1F2tUPCjqmYNQhMS-767YRipppIE&ll=48.26009155791273%2C-123.00161468982
697&z=8
• Dr. Paul Spong (orcas) – whale sanctuary project – OrcaLab with Helena Symonds – ~50
years
• http://www.orcasound.net/2018/03/29/shortcuts-for-listening-to-the-canadian-pacific/
• Working on an app, crowd-sourced
• passive (no emission) only listening – one-way – limited
!11

12. Fixed Hydrophone
!12

13. Your IT helper: me
• 2 Free Android apps, in the Play store:
• Dolphin Communicator: https://play.google.com/store/apps/details?
id=com.simplecode.leafy.seadragon.forandroid.free&hl=en
• Spectrogram: https://play.google.com/store/apps/details?
id=sm.app.spectro
• Open-sourced on GitLab – access controlled
• Considering the Alltab®, underwater tablet by Valtamer: https://youtu.be/
gxOlF_6XDFM (not partners)
!13

14. Your potential user: Dr. David Rothenberg
• Analog, music (clarinet)
• Book: https://www.amazon.com/Thousand-Mile-Song-Whale-Music/dp/
0465071287/ref=sr_1_1?
ie=UTF8&qid=1521134677&sr=8-1&keywords=thousand+mile+song
• www.thousandmilesong.com – diagram
!14

15. Your potential IT partner: aguasonic.com
• Mark Fischer – San Francisco
• Advanced acoustic analysis using proprietary wavelet software
• Algo, Matlab, C++, D, Java, AI/NN,
• Beaked whales identification for mitigation
• https://www.sciencephoto.com/search?
subtype=contributors&searchstring=AAC&media_type=images&per_page=96 – black
holes merging
• https://get.google.com/albumarchive/113099471692993588785?source=pwa
• https://news.cnrs.fr/articles/joseph-fourier-is-still-transforming-science
!15

16. Your potential IT partners:
Wolfram & son, Alpha programming language
• http://wolfram.com/
• Arrival movie:
• http://blog.wolfram.com/2017/01/31/analyzing-and-translating-an-alien-
language-arrival-logograms-and-the-wolfram-language/
• seti.org
• https://www.sciencefriday.com/segments/how-to-talk-with-aliens/
• Hawkins was opposed
!16

17. Your clients/competitors/antagonists:
Mil.-Indus. Robots – UAV
• Example:
• https://www.liquid-robotics.com/ – Boeing, mil.
• James Gosling, father of Java, now at AMZ
!17

18. Your potential legacy users/competitors: DCP
• Dolphin Communication Project – DCP
• www.dolphincommunicationproject.org
• Terramar – terramarresearch.org
• www.whalestewardship.org
!18

19. DCP
• Dr Kathleen Dudzinski
• “You can buy trips and participate in research”
• Dr Toni Frohoff - Terramar - Whale Stewardship – whale sanctuary project
• https://wearesonar.org/2017/07/21/conversing-with-dolphins-holy-grail-
interspecies-communication/
• Catherine Kinsman
• USA, Nova Scotia, The Bahamas, Central America, other places…
!19

20. Your legacy potential users/antagonists:
• Dr. Lori Marino – whale sanctuary project
• Dr. Diana Reiss
• Nora Bateson (father was Dr J C Lilly’s best friend) – Okeanos Foundation
• Dr. Vergara, Vancouver Aquarium, St. Lawrence belugas
• Richard Sears, et al., Îles Mingan, QC, blue
• Dr. Lusseau, U. Aberdeen, “culture”
• Dr. Peter Tyack, U. St. Andrews
• Dr. Whitehead (Nova Scotia) – whale sanctuary project https://whalesanctuaryproject.org/our-team/
• Dr. Au
• Dr. John Ford: orca dialects, since ~1977, BC, CA gov. employee?
!20

21. Other important legacy players
• Dr. Roger Payne: humpback songs, a first, with 2 others, risked career),
now ecology, RV The Odyssey, Gloucester, US
• Dr. Xanadu Halkias, acoustics analysis
• The pirate, Captain Paul Watson ☠
• Sea Shepherd Conservation Society https://seashepherd.org
• My advice: Avoid violence
!21

22. Other potential clients/antagonists
• $$$ oil and gas – https://www.edgewiseenvironmental.com/ – Canada
• http://www.soundandmarinelife.org/ $$$
• PAM passive acoustic monitoring s/w – Mellinger et al., Univ. St. Andrews
• www.pamguard.org
• paid jobs
• open-source; Java (desktop, laptop); api; Matlab; only listening
• resources links; community; “big players”
• https://www.youtube.com/watch?v=exGJpYmws-M – 20 min. delay - Monterey, Calif. USA
!22

23. Other markets
• Non-radio comm – ultrasound, no animals – sensors
• Ultrasonic security: https://ubeacsec.org/
• Bird Id: commercial app, db, $$
• Parrots, Dr. Irene Pepperberg; Alex; current human language recog s/w (NL)
• Elephants (some infra-sound), cats $, dogs $$$
• GUI: Chimps, orangutans, gorillas; not much audio at first
• 80+ species of cetaceans; primary Tt, Sf, Beluga, Minke…
• http://www.ukogorter.com/merchandise/whales-of-world-poster.html
!23

24. Brief History – Slide 1 of 2
• Dr Lou Herman, et al., U. Hawaii, 1980’s, visual, first artificial
sentences
• e.g., “get the blue frisbee and put it at the left corner of pool”
• Dr. Roger Payne et al., 1970’s, first analysis of whale songs, Nat Geo
• R. Cheney, USCG, 1960’s, audio, analog, two-way
• sonars http://www.navy.mil/viewGallery.asp?id=103&page=7&r=4
!24

25. Brief History – Slide 2 of 2
• Dr J. C. Lilly, 1950s-1980s, Janus Project, audio, digital, two-way;
completed?; US Gov; Gregory Bateson, George Lucas
• https://en.wikipedia.org/wiki/John_C._Lilly
• Dr Dwight (Wayne) Batteau, ~1964, first automated system, analog,
emitter (two-way?), US Navy; daily ocean swimmer but drowned on day of
making the device public
!25

26. App Design Issues
• Generic
• Android/technical
• aNN = artificial Neural Net = s/w dev tech = limited AI
• Features/Requirements
• Usage
!26

27. Input Data Example
!27
on Android sm Spectrogram app

28. DC - Dolphin Comm
!28
Android app showing
recognition chat in yellow

29. Android Technical – DC Threads
!29

30. Android Audio
• PCM – Pulse Code Modulation
• 16-bit integer or Float? A: Float! recent, not all devices
• Q: Java or C?
• Q: C with OpenSL ES or AAudio?
• A: Java because Java Audio is not the limiting factor, the graphics is
(canvas)
• https://developer.android.com/ndk/guides/audio/audio-latency
!30

31. Android OK?
!31
• Pros:
• fast math (DSP, FFT)
• inexpensive, widely available
• underwater +$
• AI TensorFlow
• two-way
• acceptable sampling rate:
• 48,000 sps input => ~11kHz output
• Cons:
• Slow graphics
• 40 slices (frames) per sec.,
due to graphics
• lack of legacy peripheral devices
• connectors – Bluetooth is very
limiting for audio
• versions incompatibilities

32. Feasibility with Android
• Proof-of-concept device feasible when starting with artificial whistles
• Example: DC – Dolphin Communicator app
• Sufficient device memory for acoustic data if using appropriate data
structures
!32

33. !33
DC aNN - An Example

34. aNN
• Pre-model data collection & analysis
• Existing recordings
• Ex.: https://swfsc.noaa.gov/
textblock.aspxDivision=PRD&ParentMenuId=148&id=5776
• https://soundcloud.com/user-361819329/180211-sperm-whales-clicks
!34

35. aNN
• Post-session data analysis, audio, recognition, model training (ML)
• NNAPI: https://developer.android.com/ndk/guides/neuralnetworks/
• Android 8.1 = API Level 27
• Longer term: Discover higher-level patterns, some possibly not in human
languages
!35

36. aNN
• Consider using models from other contexts – “TransferLearning” –
Inception V3 – TBD
• Cheat sheet to pick ML:
• https://blogs.sas.com/content/subconsciousmusings/2017/04/12/
machine-learning-algorithm-use/
!36

37. aNN
• Training aNN model with two humans communicating with artificial
whistles using one device each (in air)
• Sources for artificial whistles:
• Your own whistles
• Your researcher’s whistles
• Inspired by existing human whistle languages
• https://en.wikipedia.org/wiki/Whistled_language
!37

38. aNN
• Is only one of many components of such a system
• Is not absolutely required for proof-of-concept
• May be advantageous after data accumulates from sessions
!38

39. Feature – Two-Way
• Two-way communication – just listening is not enough
• Recognition & emission by same device
• requires specific acoustic data structures
!39

40. Feature – Off-the-Shelf
• Lower costs, easy access, more users, more progress
!40

41. Feature – Non-IT Users
• Users are scientists, not IT:
• can be very limited
• human UI issues
• the app must automate all possible tasks
!41

42. Feature – Audio Commands
• Audio UI for non-human participants:
• some whistles are commands to system
• DC includes some
!42

43. Feature – Replication
• Experiment Replication – required by science
• Data Exchange between teams of scientists/citizens
• Data in and out automatically, ex.: relational database, xml/json files
!43

44. Feature – No Internet
• Assume no internet during session
• underwater
• radio waves are absorbed
• cables are cumbersome – but a possibility in some cases
• And no significant internet even on boat when at sea
• satellite links are slow and expensive
!44

45. Feature – Artificial Whistles
• First step: artificial whistles, dolphins learn from humans, at sea
!45

46. Feature – Fast Edit
• Second step: humans learn from dolphins, change vocab, repeat, improve
system…additional species…
• Difficulty: humans learning and changing vocab on-the-fly for keeping
dolphins interested
!46

47. Usage Issues – Medical
• Potential future usage:
• medical
• species protection
• survival (ex.: hurricane)
• science
• unknown usage today
• Human-to-human underwater communication: diver to diver, diver to vessel
!47

48. Usage Issues – WWW
• Possible future world-wide network of devices:
• underwater and on surface, at sea, far or near shore
!48

49. Usage Issues – Legality
• Attention: Underwater emission can be illegal without a “Take” license
in some jurisdictions (e.g., USA, maybe Canada, other locations)
• It seems that all US citizens are not allowed to emit underwater sound
near cetaceans without a Take license, anywhere in the world
!49

50. Usage Issues – My Advice
• No Captivity
• S/W not to be used in captivity, and not for hunting, not for harassment
• Maybe open source but controlled distribution
• Maybe special S/W license
!50

51. Usage Issues – My Advice
• No violence, harassment, etc., towards cetaceans ... or humans
• Do not support violent activists or activities, directly or indirectly, and take
care on social media
• Some pro-animal, some anti-whaling, some vegan, etc., can be violent
!51

52. Related
• The legal “person” definition: can cetaceans be persons, legally?
• https://en.wikipedia.org/wiki/Personhood#Non-human_animals
!52

53. Last slide
• Mind-meld
!53