Downloaded 94 times
Uploaded byST_World
Track 3 session 8 - st dev con 2016 - music and voice over ble
This document provides an overview of Roberto Sannino's presentation on BlueVoice, STMicroelectronics' solution for playing high-quality audio over Bluetooth Low Energy. BlueVoice uses advanced audio processing and compression techniques like ADPCM to stream compressed audio data over BLE at rates up to 64 kbps, enabling applications like voice-controlled devices, wireless audio, and the internet of audio things. The document describes BlueVoice's architecture, software libraries, development kits, and example applications to demonstrate voice streaming to mobile and cloud services using STM32 and BlueNRG hardware.
More Related Content
Similar to Track 3 session 8 - st dev con 2016 - music and voice over ble
Track 3 session 8 - st dev con 2016 - music and voice over ble
- 1. October 4, 2016 SantaClara Convention Center Mission City Ballroom Solutions for playing high-quality music and voice over BLE Roberto Sannino
- 2. Audio Over BluetoothSmart: Scenarios 2 Voice controlled TV remoteAudio Entertainment & Gaming Smart Home Wearables
- 3. Voice Terminal • Audiocapture & render • Signal processing • Low power • Constained geometry Voice & data Gateway Seamless connectivity BlueVoice: IoT Audio Augmented 3 Cloud • Natural Language Processing • Dialogue Management • Services Play Music Control Lighting, heating, … News, sport, traffic, weather, … Answer questions, create to-do lists, shopping lists, … Place orders online, use other online services: taxi, pizza, …
- 4. A BlueVoice toCloud Architecture Integrated Terminal Audio Front End Signal Processing Communication & 3rd Party SW libs Cloud-based ServicesGateway 4
- 5. BlueVoice at aGlance 5 MEMS Microphone Microcontroller BlueNRG RF connectivity The ST HW and SW solution for Ultra low-power Voice streaming over Bluetooth Low-Energy STM32 Signal Processing and Application Firmware & Software BLUEVOICELINK BLE and microphone reference application based on STM32Cube OSXBLUEVOICE Voice-over-BLE vendor-specific profile library for STM32 and BlueNRG BLUEMICROSYSTEM Voice and Sensor data over a BLE link to an Android - iOS Smartphone App BlueMS, BlueST-SDK Bluetooth Smart and Sensors Technology Application for Android and iOS
- 6. BlueVoice Mapping overStandard Bluetooth 4.0 protocol stack 6 BlueVoice Vendor Specific Profile Audio processing Generic Access Profile (GAP) Generic Attribute Profile (GATT) Attribute Protocol Security Manager Logical Link Control and Adaptation Protocol Host-Controller Interface Link Layer Direct Test Mode Physical Layer Bluetooth Low Energy Stack GAP configuration GATT configuration Application: BlueVoice Profile (vendor specific) Central Unit (Master) Peripheral Unit (Slave) Client Server ClientServer Service Characteristic Characteristic Descriptor Audio is exported as a Service Service Characteristic Characteristic Descriptor
- 7. BlueVoice Server Attributes CharactersticValue Compressed audio data Audio Characteristic Service Declaration • BVS UUID • Type: primary Characteristic Declaration • AudioData UUID • Char properties: • Notification • Maximum value length: 20 B Characteristic Declaraton Characteristic Value Characterstic Value Synchronization data Sync Characteristic Characteristic Declaration • SyncData UUID • Char properties: • Notification • Maximum value length: 6 B Characteristic Declaraton Characteristic Value Reference chart BlueVoice Service Service Characteristic Characteristic Descriptor Server 7
- 8. BlueVoice Packet ExchangeSequence Advertise Connect Request Notification Notification Notification Advertise Advertisement ModeDiscovery Mode Central Unit (Master) Peripheral Unit (Slave) Connection created Notification Notification Notification Peripheral-to- Central communication Central-to-Peripheral communication ...... No response is required for Notifications: • OK for Audio streaming • Optimal Bandwidth exploitation 8
- 9. BlueVoice Architecture: Audio processingand transmission PDM to PCM conversion PCMPDM Audio Compression Raw Data Server - TX Audio Acquisition symmetrical architecture for bi-directional communication Audio Decompression Serial Audio Out USB, I2S, … Client - RX 9
- 10. BlueVoice Development Kit BlueVoiceon open.Audio 10 Voice Gateway Cloud BlueVoice Terminal • Microphone Acquisition • PDM to PCM conversion: 8 and 16 KHz • ADPCM compression: 4 bit per sample • BTLE streaming: 32 and 64 kbps SensorTile BlueCoin BlueMS app
- 11. Software IP andST Eco-system Open Software Design Environment 11 Algorithms and system demonstrators for the Internet of Things. Unleashing the power of embedded software Bring your ideas to now! BlueMicroSystem STM32 ODE STM32 Nucleo development boards STM32 Nucleo expansion boards STM32 Cube software STM32 Cube expansion software Software libraries BlueVoiceLink SmartAcoustics BlueMS Android & iOS Example Projects
- 12. Open.AUDIO OSXBlueVoice Voice overBLE Vendor Specific Profile library 12 PDM to PCM conversion PCMPDM ADPCM Audio Compression ADPCM Data Server - TX Microphone Acquisition ADPCM Audio Decompression Serial Audio Out USB, I2S, … Client - RX 16k or 8k sample/s 4 bit/sample 16bit/sample @16kHz or @ 8kHz OSXBlueVoice Speech Tx library Open Software Expansion Open.AUDIO OSXBlueVoice The BlueVoice Library OSXBlueVoice Speech Rx library ADPCM Data 16k or 8k sample/s 4 bit/sample
- 13. Audio BLE Streaming13 Connection Event 10ms t Audio Data Packet, 20 bytes 160ms Connection interval Connection Event … ADPCM Synch interval Sync Data Packet, 6 bytes … ……… Data streaming is implemented as periodical data exchange • Connection Event: data packet exchange sequence • Connection Interval: the time between two consecutive Connection Events Radio Idle Connection Event Connection Event Connection Event Low-power state between connection events Voice Streaming • Audio Format: PCM 16 bit @ 16 kHz • ADPCM Compression @ 64 Kbps: Low latency and low complexity • Side Information at low data rate ehnances error resilience
- 14. OSXBLUEVOICE v2.0.0 Library14 OSXBLUEVOICE 2.0.0
- 15. Working mode: • TRANSMITTER •RECEIVER • HALF-DUPLEX Configurable: • Audio Sampling Frequency: 8 or 16 kHz • Audio input channel select Audio input buffer: 1, 2, 5 or 10 ms of PCM audio data. BLE service and characteristic modularity Compatible with Android and iOS app: STBlueMS osxBlueVoice 2.0.0 Library 15 osxBlueVoice 2.0.0 Features Compiled for different ARM Cortex core: • M0+ • M4 STM32 Example code for: • STM32F4 • STM32L4 • STM32L0 ST development Platforms: • STM32 Nucleo • SensorTile • BlueCoin Hardware CompatibilitySTM32 application project
- 16. osxBlueVoice 2.0.0 LibraryArchitecture 16 Input ch1 ch2 ch1 ch2 ch1 ch2 ch1 ch2…… PCM buffer: - 1, 2, 5 or 10 ms audio - One or more audio channel - Audio sampling frequency: 8 or 16 kHz. - Total audio channels acquired. - Audio Channel chosen: one of the acquired channels (1, 2 or … n). - Service and characteristics configuration Configuration osxBlueVoice Library 2.0.0 Service and characteristics creation/configuration BLE packetization and sending BLE packet parsing and ADPCM decompression Audio parameters configuration BLE packets: - Audio data 20 Bytes - Sync data 6 Bytes Output Audio send application TX Control Audio receive application RX Control SendData ready GATT Notification BLE Event ADPCM compression Internal double buffering Data Control Input Output
- 17.
- 18. BLUEVOICELINK1 Half-Duplex communication basedon STM32 Open development Environment Audio configuration 8 kHz,16 kHz 4:1 ADPCM USB USB BlueVoiceLink1 2.0.0 - Peripheral osxBlueVoice 2.0.0 BlueVoiceLink1 2.0.0 - Central osxBlueVoice 2.0.0 Source code reference application based on osxBlueVoice 2.0.0 NUCLEO-F401RE NUCLEO-L476RG Microphone Bluetooth Low Energy X-NUCLEO-CCA02M1 X-NUCLEO-IDB04A1 STM32 Nucleo 18
- 19. ST BlueMS +BlueVoiceLink1 Simplex voice over BLE communication from STM32 ODE to Android/iOS 19 Cloud-based ASR Service In configuration 8 kHz ADPCM Available on Google Play and App Store NUCLEO-F401RE NUCLEO-L476RG NUCLEO-L053R8 Microphone BLE X-NUCLEO-CCA02M1 X-NUCLEO-IDB04A1 STM32 Nucleo BlueVoiceLink1 2.0.0 - Peripheral osxBlueVoice 2.0.0
- 20. ST BlueMS +Bluemicrosystem2 BlueVoice Service Integration in BlueMicrosystem environment Cloud-based ASR Service ST BlueMS available on Google Play and App Store 8 kHz ADPCM 20 BlueVoiceLink1 2.0.0 - Peripheral osxBlueVoice 2.0.0
- 21. BlueVoice Applications ADPCM @ 8kHz ADPCM @8 or 16 kHz ST BlueMS 3.0.0 BlueMicrosystem2.1.0 BlueVoiceLink1 2.0.0 Peripheral BlueVoiceLink1 2.0.0 Central data rates ≤ 32kbps to maximize compatibility From version 4.4 KITKAT API 19 R&D From version 8 NUCLEO F401 NUCLEO L476 SensorTile NUCLEO F401 NUCLEO L476 NUCLEO L053 BlueCoin osxBlueVoice 2.0.0 osxBlueVoice 2.0.0 osxBlueVoice 2.0.0 NUCLEO F401 NUCLEO L476 Cloud-based ASR Service 21
- 22.
- 23. SensorTile Integrated Development Platform CoreSystem Cradle eXpansion Cradle 23
- 24. SensorTile Software platform 24 Starter ProjectsBlueMicrosystem BlueVoice Function Pack Core System
- 25. SensorTile Native Supportfor BlueVoice 25 SensorTile Terminal Platform Signals Comm Interface Cloud-based Services Mobile Device «Natural Language»
- 26. BlueVoice over BlueNRG-1 BLEApplication Processor 26 Audio Sampling frequency: 8 kHz Compression algorithm: ADPCM Bitrate 32kbps Receiver can be any STM32 BlueVoice client or an Android/iOS device running ST BlueMS app STEVAL-IDB007V1 BlueNRG-1 evaluation platform Demo application:
- 27. BlueVoice R&D 27 R&D R&D BlueVoice FullDuplexSpeech BlueVoice FullBand Music
- 28. BlueVoice R&D –FullBand Music 28 Compression algorithm: Opus • Highly-flexible speech and audio codec • Bit-rates from 6 kb/s to 510 kb/s • Narrowband (8 kHz) to fullband (48 kHz) • Frame sizes from 2.5 ms to 60 ms • Speech and Music support • Mono and Stereo • Flexible rate control • Flexible complexity • All changeable dynamically • Completely free • Royalty-free licensing • Open-source implementation • IETF RFC 6716 Stereo Full Band Music @ up to 48 kHz
- 29. Full Band Audio Streaming USB I2S Stereo Notifications BlueVoiceControl Data Central/Client - RX USB /I2S 16bit@48 kHz Compressed AudioPCM BLE 48kbps PCM Compressed Audio BLE Peripheral/Server - TX 16bit@48kHz USB .wav 48kbps Full band music @ 48kbps Notifications Write without resp Full Band Audio Streaming • BlueNRG-MS • STM32F446, STM32F7 • Audio Format: PCM @ 48 kHz stereo • Opus Compression @ 48kbps BlueVoice FullBand Demo Overview 29
- 30. BlueVoice R&D –FullDuplex Speech 30 Bidirectional (Full-Duplex) audio streaming Input: PCM @ 16 kHz Opus Compression @ 10 kbps or higher (Variable Bit Rate)
- 31. BlueVoice R&D –FullDuplex Speech Central Data Packet Peripheral Data Packet Radio Idle t … Connection Event Connection Event t …… Radio Idle Connection Event Connection Event Connection Event Connection Event Central GATT Interval Peripheral GATT Interval Different Central-Peripheral interleaving schemas may be implemented 31
- 32. BlueVoice Solutions Half-duplex Highquality musicFull-duplex Full band music OpusADPCM 32/64 kbps 8 - 16 kbps SBC <64 kbps <64 kbps >64 kbps 8 - 16 kbps 32/64 kbps 32 - 64 kbps 8 - 10 kbps Compression schema R&D Demo R&D Demo Simplex R&D 32/64 kbps Demo <64 kbps R&D 32/64 kbps 32/64 kbps R&D Demo R&D 32
- 33. BlueVoice Solutions Advanced features Ultra-Low Power BlueVoice-L •STM32L4 or BlueNRG-1 • Features • ADPCM @8kHz • Hw PCM filtering • Low power mic BlueVoice+ • Advanced audio processing • Beamforming • ASR BlueVoiceLink • STM32F4/STM32L4/STM32L0 • Features • Half Duplex Speech • ADPCM @16/8kHz • ST BlueMS app Android/iOS Basic functonalities High Performance BlueVoice Full Band • Opus Compression • High Quality Stereo Music • 48+ KHz sampling rate BlueVoice-Full Duplex • Full Duplex Speech • Opus @ 16 kHz • Low Latency BlueVoice-Android • Half Duplex Speech • ADPCM @ 8/16kHz • SBC/Opus @ 16 kHz 33
- 34.