This document provides an overview of signal processing techniques used in 4G smartphones. It discusses the evolution of mobile communications networks and hardware and software architectures of smartphones. It then focuses on various signal processing applications in smartphones for communications, speech processing, image and video processing, and location services. Key signal processing challenges for 4G such as high data rates, interference, and power consumption are also summarized.

1. +
Signal Processing in Smartphones –
4G Perspective
Ramesh Prasad

2. +
About Me
n M.E. in Electronics & Telecommunications Engineering
n 15+Years of Technology Experience
n Senior Member IEEE
n Entrepreneur/Startup Enthusiast/Technologist/Evangelist/
Author/Speaker
n United States Patent 7,522,774
n http://www.rpthegeek.com/

3. +
Agenda
n Introduction
n Evolution of Mobile Communications
n Smartphone Anatomy
n Smartphone Hardware Architecture
n Smartphone Software Architecture
n Signal Processing In Smartphones
n Communications Signal Processing
n Speech Processing (VoIP)

4. +
Agenda
n Camera Image And Video Processing
n Display Image Processing
n Audio Post Processing
n Signal Processing For Multimedia Communication
n Geolocation Techniques & GPS Signal Processing
n Signal Processing For Apps
n Voice Enabled Services
n Scanner and BarCode Reader
n Augmented Reality
n References

5. +
Introduction

6. +
Scanner & Bar
Code Reader
Media
Player
Internet
Email
Chat
Navigation
Games
TV
Phone
Credit
Card
AppsPersonal
Assistant
Social Video
Conf.
Bluetooth/
WiFi/NFC
Camera

7. +
Technology Enablers
Signal Processing
Communications
VLSI
Mobile
Computing

8. +
Modern Communications Theory

9. +Evolution of Mobile
Communications

10. +
Wireless Technology Generations

11. +
Wireless Technology Migration

12. +
Introduction to 4G (LTE)
n Standard for wireless communication of high-speed data for
mobile phones and data terminals
n Evolution of 3G systems
n Developed by the 3GPP (3rd Generation Partnership Project)
n OFDM (Orthogonal Frequency Division Multiplexing) to
avoid ISI
n MIMO (Multiple-Input Multiple-Output) to boost Data Rates
n All-IP flat architecture supporting QoS

13. +
Drivers For 4G
n Creation and development of new services for mobile
devices
n Advancement of the Signal Processing and Communications
technologies for mobile systems
n Advancement of the VLSI technology
n Competition between mobile operators
n Challenges from other mobile technologies
n New regulation of spectrum use

14. +
Features Of 4G
n Provides a global ecosystem with inherent mobility
n Offers easier access and use with greater security and
privacy
n Dramatically improves speed and latency
n Delivers enhanced real-time video and multimedia for a
better overall experience
n Enables high-performance mobile computing
n Supports real-time applications due to its low latency
n Creates a platform upon which to build and deploy the
products and services of today and those of tomorrow
n Reduces cost per bit through improved spectral efficiency

15. +
Benefits Of 4G
n High peak speeds:
n 100 Mbps downlink (20 MHz, 2x2 MIMO)—both indoors
and outdoors
n 50 Mbps uplink (20 MHz, 1x2)
n At least 200 active voice users in every 5 MHz (i.e., can
support up to 200 active phone calls)
n Low latency:
n < 5 ms user plane latency for small IP packets
n < 100 ms camped to active
n < 50 ms dormant to active

16. +
Benefits Of 4G
n Scalable bandwidth:
n The 4G channel offers four times more bandwidth than
current 3G systems and is scalable. So, while 20 MHz
channels may not be available everywhere, 4G systems will
offer channel sizes down to 5 MHz, in increments of 1.5
MHz.
n Improved spectrum efficiency:
n Spectrum efficiency refers to how limited bandwidth is
used by the access layer of a wireless network. Improved
spectrum efficiency allows more information to be
transmitted in a given bandwidth, while increasing the
number of users and services the network can support.
n Two to four times more information can be transmitted
versus the previous benchmark, HSPA Release 6.

17. +
Benefits Of 4G
n Improved cell edge data rates:
n Not only does spectral efficiency improve near cell towers, it also
improves at the coverage area or cell edge.
n Data rates improve two to three times at the cell edge over the
previous benchmark, HSPA Release 6.
n Packet domain only
n Enhanced support for end-to-end quality of service:
n Reducing handover latency and packet loss is key to delivering a
quality service.
n This reduction is considerably more challenging with mobile
broadband than with fixed-line broadband.
n The time variability and unpredictability of the channel become more
acute.
n Additional complications arise from the need to hand over sessions
from one cell to another as users cross coverage boundaries.These
handover sessions require seamless coordination of radio resources
across multiple cells.

18. +
Technical Specifications &
Attributes

19. +
3G Limitations
n The maximum bit rates still are factor of 20 and more behind
the current state of the systems like 802.11n and 802.16e/m.
n The latency of user plane traffic (UMTS: >30 ms) and of
resource assignment procedures (UMTS: >100 ms) is too big
to handle traffic with high bit rate variance efficiently.
n The terminal complexity for WCDMA or MC‐CDMA systems
is quite high, making equipment expensive, resulting in poor
performing implementations of receivers and inhibiting the
implementation of other performance enhancements.

20. +
3G vs 4G

21. +Challenges in 4G Mobile
System Design

22. +
Challenges in 4G Mobile System
Design
n The goal of 4G has been made possible by sophisticated
signal processing algorithms
n Signal processing for 4G communications-
n Antenna
n RF
n Modulation
n Baseband
n Source coding
n Channel coding
n Signal Processing for Services and Apps

23. +
Challenges in 4G Mobile System
Design

24. +
Signal Processing Challenges in
4G
Performance
Cost Power

25. +
Signal Processing Challenges in
4G
n Data rate - Unfortunately, the growth of data rates is not
matched by advanced in semiconductor structures, terminal
power consumption improvements or battery lifetime
improvements. New processing architectures and algorithms
are needed to cope with these data rates.
n High performance MIMO receivers such as maximum-
likelihood-like receivers, sphere decoders or interference
rejection combiners offer substantial performance gains but
impose an implementation challenge, especially when the
high peak data rates are targeted.

26. +
Signal Processing Challenges in
4G
n LTE utilizes precoding, which requires accurate channel
estimation. Advanced methods like iterative decision
directed channel estimation offer performance
improvements, but pose again a complexity challenge.
n LTE has a large “toolkit” of MIMO schemes and adaptive
methods.The selection and combination of the right method
in a cell with heterogeneous devices, channel conditions and
bursty data services is a challenge.

27. +
Signal Processing Challenges in
4G
n LTE roll-out will be gradual in most cases– interworking with
other standards such as GSM or HSPA will be required for a
long time.This imposes not only a cost and complexity issue.
One of the reasons many early 3G terminals had poor power
consumption was the need for 2G cell search and handover
in addition to normal 3G operation. Reduced talk-time for
dual-mode devices is not acceptable.
n “Dirty RF” challenges. OFDM systems cause problems with
power amplifier nonlinearity, and are sensitive to frequency
errors and phase noise. Digital compensation techniques are
proposed for the transmitter and receiver, but innovation is
needed to make them reality in low cost devices.

28. +
Smartphone Anatomy

29. +
Smartphone Anatomy

30. +
Smartphone Anatomy

31. +
Smartphone Anatomy

32. +
Smartphone Anatomy

33. +Smartphone Hardware
Architecture

34. +
Hardware Architecture

35. +
Single Chip Solution - SoC

36. +
SoC – Functional Blocks

37. +Smartphone Software
Architecture

38. +
Software Architecture

39. +
Software Architecture - Telephony
RF Driver
RF Modules
Protocol Stack Interface
Protocol Stack
Telephony Middleware
Telephony Driver
Telephony Apps
Telephony Middleware Interface

40. +
Application Stack - Android

41. +Challenges & Solutions For
Wireless Communications

42. +
Challenges in Wireless
Communications
n Signal Propagation
n Signal Attenuation and Path Loss
n Multipath Effect
n Signal Spread
n Interference
n Noise

43. +
Signal Propagation Effect

44. +
Signal Attenuation and Path Loss

45. +
Multipath Effect

46. +
Delay & Doppler Spread

47. +
Interference

48. +
Noise
n Thermal Noise
n White Noise
n Flicker Noise
n Phase Noise
n Burst Noise
n Shot Noise
n Avalanche Noise

49. +
Overcoming Challenges in
Wireless Communications
n Diversity – Time, Frequency, Space
n Higher Order Modulation
n Channel Coding
n Interleaving
n Channel Estimation & Equalization
n Multi Carrier Transmission

50. +
Fundamental Limits on Data Rates

51. +
Fundamental Limits on Data Rates
n To provide high data rates as efficiently as possible, the
transmission bandwidth should be at least of the same order
as the data rates to be provided
n For provisioning of higher data rates with good coverage,
wider transmission bandwidth is required
n Challenges
n Spectrum is a scarce and expensive resource
n The use of wider transmission and reception bandwidths has an
impact on the complexity of the radio equipment, both at the base
station and at the terminal.
n increased corruption of the transmitted signal due to time
dispersion on the radio channel

52. +
Multi-Carrier Transmission
M symbols transmitted in parallel, so rate R becomes R/N

53. +
Principles of OFDM

54. +
OFDM Transmitter Receiver

55. +
Simplified Realization of OFDM

56. +
OFDMA

57. +
SC - FDMA

58. +
SC-FDMA Transmitter Receiver

59. +
MIMO

60. +Signal Processing For
Communications

61. +
Protocol Stack in 4G Handset

62. +
Protocol Stack - PHY
n To support the high data rate, exceptionally large amounts of
processing power are needed, particularly in the baseband,
where all the error handling and signal processing occurs.
n The LTE PHY employs orthogonal frequency division
multiplexing (OFDM) and multiple input multiple output
(MIMO) data transmission.
n The LTE PHY uses orthogonal frequency division multiple
access (OFDMA) on the downlink (DL) and single carrier –
frequency division multiple access (SC-FDMA) on the uplink
(UL).

63. +
Signal Processing in PHY
Downlink Uplink

64. +
PHY Layer in 4G Handset

65. +
Speech Processing (VoIP)

66. +
Speech Coder Attributes
n Bitrate
n Quality
n Complexity
n Delay
n Robustness

67. +
VoIP Technology Barriers
n End-to-End Delay and Jitter
n Packet Loss
n Throughput
n Internet Availability and Reliability
n Security and Confidentiality

68. +
VoIP Over Wireless
n Channel Quality and Adaptive Operation
n Background Noise
n Tandeming
n Voice Activity Detection
n Unequal Error Protection
n Frame Erasures

69. +
AMR Wideband
n Adaptive Multi-Rate Wideband (AMR-WB) is
a patented wideband speech coding standard developed based
on Adaptive Multi-Rate encoding, using similar methodology
as Algebraic Code Excited Linear Prediction (ACELP).
n AMR-WB provides improved speech quality due to a wider speech
bandwidth of 50–7000 Hz compared to narrowband speech coders
which in general are optimized for POTS wireline quality of 300–
3400 Hz.
n AMR-WB codec has the following parameter:
n Bit Rate – 16 Kbps
n Delay frame size: 20 ms
n Look ahead: 5ms
n Complexity: 38 WMOPS, RAM 5.3KWords
n Voice activity detection, Discontinuous Transmission, Comfort Noise Generator
n Fixed point: Bit-exact C

70. +
AMR Wideband Encoder

71. +
AMR Wideband Decoder

72. +Camera Image And Video
Processing

73. +
Smartphone Camera Features
n Video Camcorder
n Still Image Capture
n Auto Focus & Metering
n Low Light Photography
n Zero Shutter Lag Image Capture & Video Snapshot
n Face Detection & Face Tracking
n Imaging Responsiveness
n Video Editing
n Image Post Processing

74. +
Image Processing Pipeline

75. +
Black-Level Adjustment
n This stage in the pipeline adjusts for dark current from the
sensor and for lens flare, which can lead to the whitening of
an image’s darker regions. In other words, sensor black is not
the same as image black.The most common method for
calculating this adjustment is to take a picture of a
completely black field (typically accomplished by leaving
the lens cap on), resulting in three base offsets to be
subtracted from the raw sensor data. Failure to adjust the
black level will result in an undesirable loss of contrast.

76. +
Noise Filtering
n There are numerous sources of noise that can distort image
data – optical, electrical, digital and power
n The actual noise level present in an image, however, plays a
critical role in determining how strong the noise filter must
be since the use of a strong filter on a clean image will
actually distort and blur the image rather than clear it up.
n Noise reduction is achieved by averaging similar
neighboring pixels.Through the use of an Optical Electrical
Conversion Function (OECF) chart and a uniform lighting
source, the noise level can be characterized for different
intensities.

77. +
Noise Filtering
n If the noise level is high for a particular intensity, then more
weight is given to the average pixel value of similar
neighbors.
n On the other hand, if the noise level is low, more weight is
given to the original pixel value.
n The OECF chart is comprised of 12 uniform gray patches and
produces 12 corresponding power levels based on the noise
standard deviation at the mean value for each intensity/
luminance level.These 12 power levels are then used to
reduce noise across an image using either a linear or square-
root model, depending on the sensor and gain (or ISO) level.

78. +
Noise Filtering

79. +
White Balance
n Different types of lighting – such as incandescent,
fluorescent, natural light sources, LED flash – have a
pronounced effect on color.
n The most difficult to tune is in mixed-light conditions.
n White balance automatically compensates for color
differences based on lighting so white actually appears
white.

80. +
White Balance

81. +
White Balance
n Fine tuning white balance begins by measuring the average
RGB values across the six gray patches on a ColorChecker
chart

82. +
White Balance
n Using mean square error minimization, the appropriate gains
for each color can be calculated.
The resulting gains are applied to each image pixel:

83. +
CFA Interpolation
n Typically, digital cameras employ only a single sensor to capture
an image, so the camera can only obtain a single color
component for each pixel even though three components are
necessary to represent RGB color.
n CFA interpolation is the process of interpolating two missing
color components for each pixel based on the available
component and neighboring pixels.
n CFA interpolation is primarily a transform function that does not
vary based on sensor or lighting conditions, and therefore no
tuning of this image-processing pipeline stage is required.
n However, it is still one of the most complex algorithms in the
image-processing pipeline

84. +
CFA Interpolation

85. +
RGB Blending
n Different sensors produce different RGB values for the same
color.
n Tuning this pipeline stage involves creating a blending
matrix to convert the sensor RGB color space to a standard
RGB color space such as the Rec709 RGB color space.

86. +
RGB Blending
n The blending matrix is calculated by starting with a
ColorChecker chart and obtaining average RGB values for 18
different color patches (the top three rows of the chart) that
have already been white balanced.
n Next, inverse Gamma correction is applied to the reference
RGB values.The blending matrix is then constructed using
constrained minimization.

87. +
RGB Blending
n The blending matrix is applied as follows:
The final result is consistent color between cameras using different sensors.

88. +
Gamma Correction
n Gamma correction compensates for the nonlinearity of
relative intensity as the frame buffer value changes in output
displays.
n Typically, displays are calibrated using a standard gamma
correction such as Rec709 or SMPTE240M.
n Calibrating the image-processing pipeline to the same
standards ensures optimal image quality across the majority
of displays.

89. +
RGB to YCC Conversion
n Images also need to be adjusted for the human eye, which is
more sensitive to luminance (Y) than color (Cb, Cr)
information.This pipeline stage separates luminance from
color for different processing using different precisions.

90. +
Edge Enhancement
n Edge enhancement affects the sharpness of an image.
n In low-light conditions where images have a lower signal-to-
noise ratio (SNR), edge enhancement will boost noise making
it more visible to users.

91. +
Contrast Enhancement
n Contrast enhancement is comprised of two parameters:
contrast and brightness.
n Since the optimal contrast and brightness vary based on the
particular lighting conditions, as well as upon user
preference, these parameters are often implemented so that
they can be dynamically selected by the user.

92. +
Contrast Enhancement

93. +
False Chroma Suppression
n The final stage in the image-processing pipeline corrects
various color artifacts.
n Chroma suppression is a color correction that neutralizes the
green or blue colored light that often bounces off a green
screen or blue screen background and tints the edges of a
subject during a shoot.

94. +
Display Image Processing

95. +

96. +
Audio Post Processing

97. +Signal Processing For
Multimedia Communication

98. +
Applications
n Video-on-demand.
n Distance learning and training.
n Interactive gaming.
n Remote shopping.
n Online media services,such as
news reports.
n Videotelephony.
n Videoconferencing.
n Telemedicine for remote
consultation and diagnosis.
n Telesurveillance.
n Remote consultation or scene-
of-crime work.
n Collaborative working and
telepresence.

99. +
Challenges
n Higher coding efficiency
n Reduced computational complexity
n Improved error resilience

100. +
MultiMedia Communications

101. +
Challenges
n Bandwidth limitations of communication channels.
n Real-time processing requirements.
n Inter-media synchronization.
n Intra-media continuity.
n End-to-end delays and delay jitters.
n Multimedia indexing and retrieval.

102. +
Audio/Video Streaming
n Increase in demand for fast and location
independent multimedia access
n Driving force behind growth of Mobile
Internet
n “Killer App” needed for success of 3G/4G
systems
n Broad range of Applications – from
Entertainment to Telemedicine

103. +
What is Streaming
n Is it File download?
n Real Time consumption of data
n Advantages
n Low initial delay
n Media protection
n Lower Memory usage

104. +
Challenges in Video Streaming
n Bit-rate
n Packet Errors/Losses
n Buffer Management
n Low Delay
n Low Jitter
n No Re-transmission

105. +
What if Network is Wireless ?
n Higher Attenuation
n Shadowing
n Fading
n Multi-user interference
n Mobility/Handoff
n Low Processing power and Memory

106. +
Solution
n Low Bit-rate Codec
n Error Resiliency
n Error Concealment
n Prevent Spatio-Temporal Error Propagation
n Flexibility of Bit-stream
n Feedback from Transport layer

107. +
Network Architecture

108. +
Protocol Architecture
RTSP RTCP RTP
TCP UDP
IP
SESSION SETUP &
CONTROL QUALITY FEEDBACK
MEDIA DELIVERY
(AUDIO & VIDEO)

109. +
RTCP-Real Time Control Protocol
n Adaptive encoders and streaming servers
can utilize the feedback information for
adjusting the stream to match the current
transport quality
n Feedback is delivered in RTCP sender and
receiver reports
n periodic transmission of control packets to
all participants in the session

110. +
Video Coding – H.264
n Massive Quality, Minimal Files
n Scalable from 3G to HD and Beyond
n The New Industry Standard
n Latest Innovations in Video Technology
n Outperforms all preceding standards

111. +
H.264 - Innovations
n multi-frame motion-compensated prediction
n adaptive block size for motion compensation
n generalized B-Pictures
n quarter-pel motion accuracy
n intra coding utilizing prediction in the spatial
domain
n in-loop de-blocking filters
n efficient entropy coding methods

112. +
H.264 – Block Diagram

113. +
H.264 - Performance
Use Scenario Resolution & Frame Rate Example Data Rates
Mobile Content 176x144, 10-15 fps 50-60 Kbps
Internet/Standard
Definition 640x480, 24 fps 1-2 Mbps
High Definition 1280x720, 24fps 5-6 Mbps
Full High Definition 1920x1080, 24fps 7-8 Mbps

114. +
H.264 – Bandwidth Adaptation
n Send one of several pre-encoded versions
of the same content, based on the current
channel bit-rate
n Frame dropping of non-reference frames, if
the channel rate fluctuates only in a small
range
n Instantaneous decoder refresh (IDR)
pictures to compensate large scale
variations of the channel rate

115. +
H.264 – Error Probability Reduction
n Slice-structured coding – Slices are
independently coded
n Short slice/packets reduce the amount of
lost information
n Probability of a bit-error hitting a short
packet is generally lower than for large
packets
n Short packets reduce the amount of lost
information and, hence, the error is limited

116. +
H.264 – Error Resilience
n Flexible MB Ordering (FMO)
n Data partitioning - unequal error protection

117. +
H.264 – Error concealment
n Intra coded MBs
n Multiple Reference Frames
n Redundant Slices

118. +H.264 – Intra Frame Error Concealment

119. +H.264 – Inter Frame Error Concealment

120. +Geolocation Techniques &
GPS Signal Processing

121. +
Geolocation Techniques
n For wireless communication networks, an inherently suitable
approach for wireless geolocation is known as radiolocation,
in which the parameters that are used for location estimation
are obtained fromradio signal measurements between a
target and one or more fixed stations.
n GPS is often inoperable in areas where satellites are blocked,
such as in buildings and built-up urban areas.
n Further, the time-to-first-fix (TTFF) for a conventional GPS
receiver from a “cold” start can take several minutes.
n Additionally, adding GPS functionality to a handset can be
costly, bulky, and drain battery power at an unacceptable rate

122. +
Geolocation Methods
n The parameters that are often measured and used for
location include-
n angles of arrival (AOAs),
n signal strength, times of arrival (TOAs), and
n time differences of arrival (TDOAs).
n The fundamental methods of radiolocation that use these
parameters can be grouped into three categories:
n direction finding,
n ranging, and
n range differencing.

123. +
Direction Finding

124. +
Ranging

125. +
Range Differencing

126. +
Algorithms For Geolocation
n Geometrical Techniques
n When there are errors in the measured AOAs, ranges, or
range differences, statistical solutions are more justifiable in
the presence of measurement errors
n Least Squares Estimation - estimating parameters by
minimizing the squared discrepancies between observed
data, on the one hand, and their expected values on the other

127. +
GPS

128. +
GPS Signals

129. +
GPS Receiver
Adaptive Space Time Array Receiver

130. +
Signal Processing For Apps

131. +
Voice Enabled Services
n Google Voice Search Video

132. +
Distributed Speech Recognition

133. +
Scanner & Bar Code Reader

134. +
2D Bar Code
n Defining of threshold as pre-processing,
n Detecting a black bar using spiral search method, and
n Finding the sampled scanning line which is perpendicular to
the detected bar in phase

135. +
QR Code
n Pre-processing The gray level histogram calculation is
adopted,
n Corner marks detection Three marked corners are detected
using the finder pattern,
n Fourth corner estimation The fourth corner is detected using
the special algorithm,
n Inverse perspective transformation Inverse transformation is
adopted based on the obtained corner geometry positions to
normalize the size of the code, and
n Scanning of code Sample the inside of code and output the
normalized bi-level code data to host CPU.

136. +
Visiting Card Scanner

137. +
Augmented Reality
n Augmented Reality Video

138. +
Image Matching
Query Image
Prefetched Data
Database Images

139. +
System Architecture
Geo-Tagged
Images
Group Images
by Loxel
Extract
Features
Camera
Image
Extract
Features
Compute
Feature Matches
Geometric
Consistency Check
Match
Images
Geometric
Consistency Check
Display Info for
Top Ranked Image
Device
Location
Feature
Cache
ANN
Loxel-Based
Feature Store
Server
Network
Cluster
Features
Prune
Features
Compress
Descriptors

140. +
References

141. +
References
n Mobile Handset Design – Sajal Das
n Signal Processing for Mobile Communications Handbook –
Mohamed Ibnkahla
n 4G LTE/LTE-Advanced for Mobile Broadband - Erik Dahlman