This document discusses the author's approach to classifying radio signal modulations using deep neural networks in the 2018 Army Signal Classification Challenge. It summarizes the author's 6th place solution using two deep learning models: a ResNet architecture and a CLDNN architecture. Key details provided include the dataset and evaluation metrics, model architectures and implementations, training procedures, performance on validation and test sets, and challenges faced in the competition.

1. Radio Signal Classification with Deep
Neural Networks
2018 Army Signal Classification Challenge
6th place solution
Kachi Odoemene
29 Aug 2018

2. Competition
• Army Rapid Capabilities Office (RCO)
– April 30th – August 13th 2018
– $150,000 in prizes
• Radio signal modulation classification
– Automatically identify modulation type of
received radio signal
– 24 modulation classes, including noise
class

3. Competition
• Finished 6th place (of 161 participants)

4. Radio Signal Modulation Schemes
Binary Frequency Shift Keying (BFSK)

5. Quadrature Signals
• I/Q: In-phase and Quadrature components
• 90° shift between identical periodic signals
– e.g. sine and cosine wave
Q
I

6. I/Q Modulation
• Summation of I/Q pairs results in any modulation
(frequency, amplitude, phase, etc)
• Transmitted & received radio signal represented
as I/Q components
Modulated
RF Signal
I channel, I(t)
Q channel, Q(t)
Inputs
(Modulating waves)
Carrier
wave

7. I/Q Time Domain Examples – high SNR
PSK: phase shift keying
FSK: frequency shift keying
ASK: amplitude shift keying
MSK: Minimum (frequency) shift keying
QAM: quadrature amplitude modulation
CP: continuous phase
OQ: offset quadrature
SNR: Signal to Noise

8. I/Q Time Domain Examples – low SNR

9. Dataset
• > 30 GB
– 4.32 million I/Q instances
• Dimension: 2 x 1024
– Short time window, real world conditions
– 24 modulation classes
– White noise added to signals
• 6 signal-to-noise (SNR) levels
• Test datasets (public)
– 2 sets of 100000 I/Q instances (unlabeled)

10. Evaluation Metric
• Derived from multiclass log loss
Score =

11. Goal
• Automatically identify the modulation type
of the received radio signal
– What modulation format was employed?
– Eg. Military applications:
• Spectrum surveillance, electronic warfare, and
threat analysis
– Identify modulation type of intercepted enemy
communication

12. Traditional approach
• Hand-crafted features (feature engineering)
– Higher order statistics
– Autocorrelation and spectral correlation functions
– Measures derived from instantaneous frequency,
amplitude, phase
• Mean, standard deviation, kurtosis, etc
– Small number of features (28-32)
• SVM, decision trees, ensemble, neural
networks

13. Deep Neural Networks for Radio Modulation
Recognition
(2016)
Convolutional networks outperform
expert feature-based classifiers

14. • Proposed 3 additional architectures for
modulation recognition:
– Inception
– ResNet
– Hybrid of Convolutional, Long short term memory
(LSTM), and Fully Connected (FC) Deep Neural
Network (CLDNN)
• No source code provided
• Sparse details on architecture hyperparameters
(2017)

15. Residual Unit
Layer Input
Model 1: ResNet (original)
• Image classification
• CNNs with skip (residual or
shortcut) connections
– Feed previous representations
(activations) into downstream layers
– Prevents information loss
• Enables training of deeper
networks
– 100s to 1000s of layers
He et al Deep Residual Learning for Image Recognition (ArXiv 2015)
Skipconnection

16. Input
(2x1024)
Conv
(1x128)
D-Conv
(2x1)
Avg. Pool
“ResBlock” x4
Global Avg
Pool
FC (128) x2
Softmax (24)
(Temporal)
(Spatial)
Model 1: ResNet (modified)
• Temporal convolution on each
IQ channel separately
• Depth-wise convolution (D-
Conv)
• Batch normalization after
convolutional and fully
connected (FC) layers
• Multiple residual units within
ResBlock
• # Parameters: 255,944
He et al Deep Residual Learning for Image Recognition (ArXiv 2015)

17. Model 2: CLDNN (original)
• Speech recognition
• Unified model: CNN, LSTM, FC
• CNN: reduce spectral variations of
input data
• LSTM: learn temporal structure
• FC: transform LSTM features into
output easy to classify
Sainath et al Convolutional, Long Short-Term Memory, fully
connected Deep Neural Networks (IEEE, 2015)
Convolutional
layers
Linear
layer
LSTM
layers
Fully
Connected
layers
Output targets
(1)
(2)
C
C
L
L
D
D
Dim.
red.Xt
[Xt-l,..,Xt,…,Xt+r]

18. Model 2: CLDNN (modified)
• Temporal convolution on each IQ
channel separately
• Depth-wise convolution (D-Conv)
• Batch normalization after each
convolutional and fully connected
(FC) layers
• “ConvBlock”: Conv + BatchNorm +
ReLU
• Dropout between FC layers
• # Parameters: 147,480
Sainath et al Convolutional, Long Short-Term Memory, fully
connected Deep Neural Networks (IEEE, 2015)
Input
(2x1024)
Conv
(1x128)
D-Conv
(2x1)
Avg. Pool
ConvBlock x2
Concatenate
LSTM (48)
x2FC (128)
Softmax (24)
(Temporal)
(Spatial)

19. Data preparation
• Raw data, no preprocessing
• Trained from scratch
• Data split
– Train: 80%, Valid: 13.33%, Holdout: 6.67%

20. Implementation details
• Hyper-parameter selection
– Temporal filter kernel size
– Number of FC units
• Maximum of 25 epochs, early stopping
• Adam optimizer
• Learning rate: 1e-3
• Keras (Tensorflow)
• Hardware (Personal)
– GTX 1080Ti (11 GB) GPU
– 16 GB RAM

21. Loss curves

22. Accuracy per modulation class: all SNRs

23. Accuracy per modulation class: high SNR

24. Accuracy per SNR: pooled across classes

25. Accuracy per SNR: individual classes
PSK: phase shift keying
FSK: frequency shift keying
ASK: amplitude shift keying
MSK: Minimum (frequency) shift keying
QAM: quadrature amplitude modulation
CP: continuous phase
OQ: offset quadrature

26. Leaderboard performance
Model TestSet1 Score TestSet2 Score Final Score
CLDNN 42.09 54.22 50.58
ResNet-18 33.27 58.27 50.77
Combined 42.09 58.27 53.41
Final Score = 0.3*TestSet1 + 0.7* TestSet2

27. Ensembling

28. Ensembling
• Trained meta-learner to combine
predictions from multiple models

29. Leaderboard performance: Ensembling
Model TestSet1 Score TestSet2 Score Final Score
CLDNN 42.09 54.22 50.58
ResNet-18 33.27 58.27 50.77
Combined 42.09 58.27 53.41
Logistic
Regression
45.17 63.25 57.83
Light GBM 46.40 64.54 59.10
Final Score = 0.3*TestSet1 + 0.7* TestSet2

30. Challenges
• Structural (competition organization)
– Multiple changes to scoring procedure and test set,
submission site shutdown, etc
• Time constraint
– 1 submission/day (10am) & leaderboard update
(5pm)
– On Kaggle: up to 5 submissions/day, immediate LB
update
• Technical
– Hardware failure final week of competition
• Backup: Google Colab

31. Future Efforts
• Incorporate additional features:
– Amplitude
– Phase difference
– Magnitude of Fourier transform
– Spectrogram
• Explore other architectures & ensembling methods
– Inception-like architecture: process and combine multiple
frequency scales
• Model interpretability
– DeepLIFT (Deep Learning Important FeaTures)
– LIME (Local Interpretable Model-agnostic Explanations)
– Ablation studies

32. Acknowledgements

33. Thank You!
Questions?