The document discusses using web APIs to interact with Bluetooth devices. It provides an overview of the Bluetooth protocol and describes how to use the Web Bluetooth and Web Audio APIs to connect to Bluetooth devices, read audio data, detect vowels in speech, and send commands to control devices. Code examples are given for connecting to a device, reading frequency data from the audio stream, analyzing it to determine vowels, and writing messages to a Bluetooth characteristic to trigger actions like vibration. The goal is to enable creative uses of web APIs to experiment with Bluetooth and audio.

1. @fischaelameergeildanke.com #webrebels
Using Web APIs
For Your Own PleasurePleasure

2. What? Why?
Created by Ben Davis from the Noun Project, Created by Viktor Vorobyev from the Noun Project

3. What? Why?
Created by Ben Davis from the Noun Project, Created by Viktor Vorobyev from the Noun Project

4. Created by Ben Davis from the Noun Project, Created by Viktor Vorobyev from the Noun Project
What? Why?

5. Created by Ben Davis from the Noun Project, Created by Viktor Vorobyev from the Noun Project
What? Why?

6. Created by Ben Davis from the Noun Project, Created by Viktor Vorobyev from the Noun Project
What? Why?

7. Ahhh
Created by Ben Davis from the Noun Project, Created by Viktor Vorobyev from the Noun Project
What? Why?

10. Web Bluetooth API Support
Devices with
Bluetooth 4.0 or higher
Chrome, Chrome on Android,
Samsung Internet, Opera
Created by Ben Davis from the Noun Project, Created by Viktor Vorobyev from the Noun Project

11. Three Steps to Happiness
Write Data
via Bluetooth API
Connect
Bluetooth Device
Request Connection
Read
Audio Data
Expose Vowels
Uncover Bluetooth Commands

12. Web Bluetooth API
Connect to and interact with
Bluetooth Low Energy devices.

13. Bluetooth Low Energy
Bluetooth LE, BLE, Bluetooth smart

14. Bluetooth Low Energy
Support
iOS 5+ Android 4.3+ OS X 10.6+ Windows 8+

15. Harald "Blåtand"
Gormsson
~958-986
King of Denmark and Norway
means Bluetooth

16. Haglaz
Old Norse: Hagall
Berkanan
Old Norse: Bjarkan

17. Advertising and Discovery
Peripheral Device Central Device
Frequency Hopping Spread Spectrum (FHSS)
short-wavelength radio waves
~2.4GHz

18. Advertising and Discovery
Peripheral
Device
Central
Device
scan
response
request
advertising
data
advertising
data
scan
response
data

19. Advertising as Broadcasting
Peripheral
Device
Central Device
Central Device
Central Device
Central Device
Central DeviceCentral Device
Advertising Data

20. GATT
Generic Attribute Profile
Defines the way two Bluetooth LE devices
transfer data back and forth
using Characteristics and Services.

21. GATT
Server
GATT
Client
Communication via GATT
sends
response
sends
request
sends
response
sends
request
sends
reques

22. Profile
collection of
related services
Service
collection of
related
characteristics
Parts of GATT
Characteristic
Characteristic
Characteristic
read (RX) and
write (TX) data

23. Connect Bluetooth Device
const LUSH_SERVICE = '6e400001-b5a3-f393-e0a9-e50e24dcca9e';
button.addEventListener('click', () => {
navigator.bluetooth.requestDevice({
filters: [{ services: [LUSH_SERVICE] }]
}).then((device) => {
console.log('Device found. Device: ', device);
return device.gatt.connect();
}).then((server) => {
console.log('Device connected. Server ', server);
});
});

24. Connect Bluetooth Device
const LUSH_SERVICE = '6e400001-b5a3-f393-e0a9-e50e24dcca9e';
button.addEventListener('click', () => {
navigator.bluetooth.requestDevice({
filters: [{ services: [LUSH_SERVICE] }]
}).then((device) => {
console.log('Device found. Device: ', device);
return device.gatt.connect();
}).then((server) => {
console.log('Device connected. Server ', server);
});
});

25. const LUSH_SERVICE = '6e400001-b5a3-f393-e0a9-e50e24dcca9e';
button.addEventListener('click', () => {
navigator.bluetooth.requestDevice({
filters: [{ services: [LUSH_SERVICE] }]
}).then((device) => {
console.log('Device found. Device: ', device);
return device.gatt.connect();
}).then((server) => {
console.log('Device connected. Server ', server);
});
});
Connect Bluetooth Device

26. const LUSH_SERVICE = '6e400001-b5a3-f393-e0a9-e50e24dcca9e';
button.addEventListener('click', () => {
navigator.bluetooth.requestDevice({
filters: [{ services: [LUSH_SERVICE] }]
}).then((device) => {
console.log('Device found. Device: ', device);
return device.gatt.connect();
}).then((server) => {
console.log('Device connected. Server ', server);
});
});
Connect Bluetooth Device

27. nRF Connect

28. nRF Connect
Nordic UART
Service

29. const LUSH_SERVICE = '6e400001-b5a3-f393-e0a9-e50e24dcca9e';
button.addEventListener('click', () => {
navigator.bluetooth.requestDevice({
filters: [{ services: [LUSH_SERVICE] }]
}).then((device) => {
console.log('Device found. Device: ', device);
return device.gatt.connect();
}).then((server) => {
console.log('Device connected. Server ', server);
});
});
Connect Bluetooth Device

30. const LUSH_SERVICE = '6e400001-b5a3-f393-e0a9-e50e24dcca9e';
button.addEventListener('click', () => {
navigator.bluetooth.requestDevice({
filters: [{ services: [LUSH_SERVICE] }]
}).then((device) => {
console.log('Device found. Device: ', device);
return device.gatt.connect();
}).then((server) => {
console.log('Device connected. Server ', server);
});
});
Connect Bluetooth Device

32. Web Audio API
Control audio on the web inside an Audio Context
with linked together Audio Nodes that for an
Audio Routing Graph.

33. Audio Context
Audio Routing Graph
Source Node
e.g. <audio>
output
Audio Node
Processing Node
e.g. reverb
input
Audio Node
output
Destination Node
e.g. speakers
Audio Node
input

34. UltrasonicSubsonic
20Hz
Sound Frequencies
heard by human ears
20kHz
What is Sound?

35. What is Sound?
https://en.wikipedia.org/wiki/Sound#/media/File:Sine_waves_different_frequencies.svg

36. What is Sound?
https://de.wikipedia.org/wiki/Frequenzspektrum

37. How to detect Vowels
formants:
f1 = ~800hz
f2 = ~1600hz
https://de.wikipedia.org/wiki/Formant

38. Vowel Detection within the Web
AudioContext MediaStreamAudioSourceNode AnalyserNode
Create Context and Nodes
Connect Nodes
MediaStreamAudioSourceNode AnalyserNode Destination

39. Vowel Detection within the Web
let audioContext = new window.AudioContext();
let stream = document.getElementById('video').captureStream();
let audioSource = audioContext.createMediaStreamSource(stream);
let analyser = audioContext.createAnalyser();
audioSource.connect(analyser);
analyser.connect(audioContext.destination);

40. Vowel Detection within the Web
let audioContext = new window.AudioContext();
let stream = document.getElementById('video').captureStream();
let audioSource = audioContext.createMediaStreamSource(stream);
let analyser = audioContext.createAnalyser();
audioSource.connect(analyser);
analyser.connect(audioContext.destination);

41. AnalyserNode
Input Node Anaylser Node Output Node
UnchangedFast Fourier
Transformation
Frequency Data

42. AnalyserNode
analyser.fftSize = 2048;
console.log(analyser.frequencyBinCount); //1024
let dataArray = new Uint8Array(analyser.frequencyBinCount);
analyser.getByteFrequencyData(dataArray);

43. AnalyserNode
analyser.fftSize = 2048;
console.log(analyser.frequencyBinCount); //1024
let dataArray = new Uint8Array(analyser.frequencyBinCount);
analyser.getByteFrequencyData(dataArray);

44. AnalyserNode
analyser.fftSize = 2048;
console.log(analyser.frequencyBinCount); //1024
let dataArray = new Uint8Array(analyser.frequencyBinCount);
analyser.getByteFrequencyData(dataArray);

45. AnalyserNode
analyser.fftSize = 2048;
console.log(analyser.frequencyBinCount); //1024
let dataArray = new Uint8Array(analyser.frequencyBinCount);
analyser.getByteFrequencyData(dataArray);

46. Visualize and Analyse Audio
let render = function () {
analyser.getByteFrequencyData(dataArray);
formants = getFormants(dataArray);
vowel = formantsToAh(formants[0], formants[1]);
window.requestAnimationFrame(render);
};
window.requestAnimationFrame(render);

47. Frequency Data
decibel = value / 256;Get Decibel
frequency = index * 44100 / 2048;Get Frequency
audioContext.sampleRate
FFT Size

48. Get Formants
let getFormants = function (data) {
let f1, f2, maxF1 = -Infinity, maxF2 = -Infinity;
data.forEach(function (date, i) {
let frequency = i * 44100 / 2048, decibel = date / 256;
if (frequency < 1000) {
if (decibel > maxF1) { maxF1 = decibel; f1 = frequency; }
} else if (frequency < 5000) {
if (decibel > maxF2) { maxF2 = decibel; f2 = frequency; }
}
});
return [{ db: maxF1, freq: f1}, { db: maxF2, freq: f2 }];
};

49. Get Formants
let getFormants = function (data) {
let f1, f2, maxF1 = -Infinity, maxF2 = -Infinity;
data.forEach(function (date, i) {
let frequency = i * 44100 / 2048, decibel = date / 256;
if (frequency < 1000) {
if (decibel > maxF1) { maxF1 = decibel; f1 = frequency; }
} else if (frequency < 5000) {
if (decibel > maxF2) { maxF2 = decibel; f2 = frequency; }
}
});
return [{ db: maxF1, freq: f1}, { db: maxF2, freq: f2 }];
};

50. Get Formants
let getFormants = function (data) {
let f1, f2, maxF1 = -Infinity, maxF2 = -Infinity;
data.forEach(function (date, i) {
let frequency = i * 44100 / 2048, decibel = date / 256;
if (frequency < 1000) {
if (decibel > maxF1) { maxF1 = decibel; f1 = frequency; }
} else if (frequency < 5000) {
if (decibel > maxF2) { maxF2 = decibel; f2 = frequency; }
}
});
return [{ db: maxF1, freq: f1}, { db: maxF2, freq: f2 }];
};

51. Get Formants
let getFormants = function (data) {
let f1, f2, maxF1 = -Infinity, maxF2 = -Infinity;
data.forEach(function (date, i) {
let frequency = i * 44100 / 2048, decibel = date / 256;
if (frequency < 1000) {
if (decibel > maxF1) { maxF1 = decibel; f1 = frequency; }
} else if (frequency < 5000) {
if (decibel > maxF2) { maxF2 = decibel; f2 = frequency; }
}
});
return [{ db: maxF1, freq: f1}, { db: maxF2, freq: f2 }];
};

52. Get Formants
let getFormants = function (data) {
let f1, f2, maxF1 = -Infinity, maxF2 = -Infinity;
data.forEach(function (date, i) {
let frequency = i * 44100 / 2048, decibel = date / 256;
if (frequency < 1000) {
if (decibel > maxF1) { maxF1 = decibel; f1 = frequency; }
} else if (frequency < 5000) {
if (decibel > maxF2) { maxF2 = decibel; f2 = frequency; }
}
});
return [{ db: maxF1, freq: f1}, { db: maxF2, freq: f2 }];
};

53. Get Vowels
let formantsToAh = function (f1, f2) {
if (f1.freq > 700 && f1.freq < 1000 && f1.db > 0.4 &&
f2.freq > 1200 && f2.freq < 1800 && f2.db > 0.2) {
return 'a';
}
return null;
};

54. Real-time Vowel Detection
let render = function () {
analyser.getByteFrequencyData(dataArray);
formants = getFormants(dataArray);
vowel = formantsToAh(formants[0], formants[1]);
if (vowel === 'a') {
console.log('Ah!');
}
window.requestAnimationFrame(render);
};
Next video: https://www.youtube.com/watch?v=-74Tyi-fvi8

56. Identify Bluetooth Commands

57. Vibrate:4;
Identify Bluetooth Commands
56 69 62 72 61 74 65 3a 34 3b

58. Write to a Bluetooth Characteristic
const LUSH_TX_CHARACTERISTIC = '6e400002-b5a3-f393-e0a9-e50e24dcca9e';
let txCharacteristic = null;
button.addEventListener('click', () => {
/* … */
}).then((server) => {
return server.getPrimaryService(LUSH_SERVICE);
}).then((service) => {
return service.getCharacteristic(LUSH_TX_CHARACTERISTIC);
}).then((characteristic) => { txCharacteristic = characteristic; });
});

59. const LUSH_TX_CHARACTERISTIC = '6e400002-b5a3-f393-e0a9-e50e24dcca9e';
let txCharacteristic = null;
button.addEventListener('click', () => {
/* … */
}).then((server) => {
return server.getPrimaryService(LUSH_SERVICE);
}).then((service) => {
return service.getCharacteristic(LUSH_TX_CHARACTERISTIC);
}).then((characteristic) => { txCharacteristic = characteristic; });
});
Write to a Bluetooth Characteristic

61. const LUSH_TX_CHARACTERISTIC = '6e400002-b5a3-f393-e0a9-e50e24dcca9e';
let txCharacteristic = null;
button.addEventListener('click', () => {
/* … */
}).then((server) => {
return server.getPrimaryService(LUSH_SERVICE);
}).then((service) => {
return service.getCharacteristic(LUSH_TX_CHARACTERISTIC);
}).then((characteristic) => { txCharacteristic = characteristic; });
});
Write to a Bluetooth Characteristic

62. if (vowel === 'a') {
let message = new TextEncoder('ASCII').encode('Vibrate:10;');
txCharacteristic.writeValue(message);
} else {
let message = new TextEncoder('ASCII').encode('Vibrate:0;');
txCharacteristic.writeValue(message);
}
Write to a Bluetooth Characteristic

63. if (vowel === 'a') {
let message = new TextEncoder('ASCII').encode('Vibrate:10;');
txCharacteristic.writeValue(message);
} else {
let message = new TextEncoder('ASCII').encode('Vibrate:0;');
txCharacteristic.writeValue(message);
}
Write to a Bluetooth Characteristic
Next video: https://www.youtube.com/watch?v=-74Tyi-fvi8

65. @fischaelameergeildanke.com #webrebels
Be Creative!
Use More Web APIs, Experiment, Give Feedback

66. @fischaelameergeildanke.com #webrebels
More Information
https://developers.google.com/web/updates/2015/07/interact-with-ble-devices-on-the-web
http://nilhcem.com/iot/reverse-engineering-simple-bluetooth-devices
https://developer.mozilla.org/en-US/docs/Web/API/Web_Bluetooth_API
https://www.bluetooth.com/specifications/gatt/services
Thanks to:
Stefan Judis
https://www.metafetish.com/
https://github.com/gre/zpeech