Signal Processing in Smartphones –
n M.E. in Electronics & Telecommunications Engineering
n 15+Years of Technology Experience
n Senior Member IEEE
n Entrepreneur/Startup Enthusiast/Technologist/Evangelist/
n United States Patent 7,522,774
n Evolution of Mobile Communications
n Smartphone Anatomy
n Smartphone Hardware Architecture
n Smartphone Software Architecture
n Signal Processing In Smartphones
n Communications Signal Processing
n Speech Processing (VoIP)
n Camera Image And Video Processing
n Display Image Processing
n Audio Post Processing
n Signal Processing For Multimedia Communication
n Geolocation Techniques & GPS Signal Processing
n Signal Processing For Apps
n Voice Enabled Services
n Scanner and BarCode Reader
n Augmented Reality
Introduction to 4G (LTE)
n Standard for wireless communication of high-speed data for
mobile phones and data terminals
n Evolution of 3G systems
n Developed by the 3GPP (3rd Generation Partnership Project)
n OFDM (Orthogonal Frequency Division Multiplexing) to
n MIMO (Multiple-Input Multiple-Output) to boost Data Rates
n All-IP flat architecture supporting QoS
Drivers For 4G
n Creation and development of new services for mobile
n Advancement of the Signal Processing and Communications
technologies for mobile systems
n Advancement of the VLSI technology
n Competition between mobile operators
n Challenges from other mobile technologies
n New regulation of spectrum use
Features Of 4G
n Provides a global ecosystem with inherent mobility
n Offers easier access and use with greater security and
n Dramatically improves speed and latency
n Delivers enhanced real-time video and multimedia for a
better overall experience
n Enables high-performance mobile computing
n Supports real-time applications due to its low latency
n Creates a platform upon which to build and deploy the
products and services of today and those of tomorrow
n Reduces cost per bit through improved spectral efficiency
Benefits Of 4G
n High peak speeds:
n 100 Mbps downlink (20 MHz, 2x2 MIMO)—both indoors
n 50 Mbps uplink (20 MHz, 1x2)
n At least 200 active voice users in every 5 MHz (i.e., can
support up to 200 active phone calls)
n Low latency:
n < 5 ms user plane latency for small IP packets
n < 100 ms camped to active
n < 50 ms dormant to active
Benefits Of 4G
n Scalable bandwidth:
n The 4G channel offers four times more bandwidth than
current 3G systems and is scalable. So, while 20 MHz
channels may not be available everywhere, 4G systems will
offer channel sizes down to 5 MHz, in increments of 1.5
n Improved spectrum efficiency:
n Spectrum efficiency refers to how limited bandwidth is
used by the access layer of a wireless network. Improved
spectrum efficiency allows more information to be
transmitted in a given bandwidth, while increasing the
number of users and services the network can support.
n Two to four times more information can be transmitted
versus the previous benchmark, HSPA Release 6.
Benefits Of 4G
n Improved cell edge data rates:
n Not only does spectral efficiency improve near cell towers, it also
improves at the coverage area or cell edge.
n Data rates improve two to three times at the cell edge over the
previous benchmark, HSPA Release 6.
n Packet domain only
n Enhanced support for end-to-end quality of service:
n Reducing handover latency and packet loss is key to delivering a
n This reduction is considerably more challenging with mobile
broadband than with fixed-line broadband.
n The time variability and unpredictability of the channel become more
n Additional complications arise from the need to hand over sessions
from one cell to another as users cross coverage boundaries.These
handover sessions require seamless coordination of radio resources
across multiple cells.
n The maximum bit rates still are factor of 20 and more behind
the current state of the systems like 802.11n and 802.16e/m.
n The latency of user plane traffic (UMTS: >30 ms) and of
resource assignment procedures (UMTS: >100 ms) is too big
to handle traffic with high bit rate variance efficiently.
n The terminal complexity for WCDMA or MC‐CDMA systems
is quite high, making equipment expensive, resulting in poor
performing implementations of receivers and inhibiting the
implementation of other performance enhancements.
Challenges in 4G Mobile System
n The goal of 4G has been made possible by sophisticated
signal processing algorithms
n Signal processing for 4G communications-
n Source coding
n Channel coding
n Signal Processing for Services and Apps
Signal Processing Challenges in
Signal Processing Challenges in
n Data rate - Unfortunately, the growth of data rates is not
matched by advanced in semiconductor structures, terminal
power consumption improvements or battery lifetime
improvements. New processing architectures and algorithms
are needed to cope with these data rates.
n High performance MIMO receivers such as maximum-
likelihood-like receivers, sphere decoders or interference
rejection combiners offer substantial performance gains but
impose an implementation challenge, especially when the
high peak data rates are targeted.
Signal Processing Challenges in
n LTE utilizes precoding, which requires accurate channel
estimation. Advanced methods like iterative decision
directed channel estimation offer performance
improvements, but pose again a complexity challenge.
n LTE has a large “toolkit” of MIMO schemes and adaptive
methods.The selection and combination of the right method
in a cell with heterogeneous devices, channel conditions and
bursty data services is a challenge.
Signal Processing Challenges in
n LTE roll-out will be gradual in most cases– interworking with
other standards such as GSM or HSPA will be required for a
long time.This imposes not only a cost and complexity issue.
One of the reasons many early 3G terminals had poor power
consumption was the need for 2G cell search and handover
in addition to normal 3G operation. Reduced talk-time for
dual-mode devices is not acceptable.
n “Dirty RF” challenges. OFDM systems cause problems with
power amplifier nonlinearity, and are sensitive to frequency
errors and phase noise. Digital compensation techniques are
proposed for the transmitter and receiver, but innovation is
needed to make them reality in low cost devices.
Fundamental Limits on Data Rates
n To provide high data rates as efficiently as possible, the
transmission bandwidth should be at least of the same order
as the data rates to be provided
n For provisioning of higher data rates with good coverage,
wider transmission bandwidth is required
n Spectrum is a scarce and expensive resource
n The use of wider transmission and reception bandwidths has an
impact on the complexity of the radio equipment, both at the base
station and at the terminal.
n increased corruption of the transmitted signal due to time
dispersion on the radio channel
M symbols transmitted in parallel, so rate R becomes R/N
Protocol Stack - PHY
n To support the high data rate, exceptionally large amounts of
processing power are needed, particularly in the baseband,
where all the error handling and signal processing occurs.
n The LTE PHY employs orthogonal frequency division
multiplexing (OFDM) and multiple input multiple output
(MIMO) data transmission.
n The LTE PHY uses orthogonal frequency division multiple
access (OFDMA) on the downlink (DL) and single carrier –
frequency division multiple access (SC-FDMA) on the uplink
VoIP Technology Barriers
n End-to-End Delay and Jitter
n Packet Loss
n Internet Availability and Reliability
n Security and Confidentiality
VoIP Over Wireless
n Channel Quality and Adaptive Operation
n Background Noise
n Voice Activity Detection
n Unequal Error Protection
n Frame Erasures
n Adaptive Multi-Rate Wideband (AMR-WB) is
a patented wideband speech coding standard developed based
on Adaptive Multi-Rate encoding, using similar methodology
as Algebraic Code Excited Linear Prediction (ACELP).
n AMR-WB provides improved speech quality due to a wider speech
bandwidth of 50–7000 Hz compared to narrowband speech coders
which in general are optimized for POTS wireline quality of 300–
n AMR-WB codec has the following parameter:
n Bit Rate – 16 Kbps
n Delay frame size: 20 ms
n Look ahead: 5ms
n Complexity: 38 WMOPS, RAM 5.3KWords
n Voice activity detection, Discontinuous Transmission, Comfort Noise Generator
n Fixed point: Bit-exact C
Smartphone Camera Features
n Video Camcorder
n Still Image Capture
n Auto Focus & Metering
n Low Light Photography
n Zero Shutter Lag Image Capture & Video Snapshot
n Face Detection & Face Tracking
n Imaging Responsiveness
n Video Editing
n Image Post Processing
n This stage in the pipeline adjusts for dark current from the
sensor and for lens flare, which can lead to the whitening of
an image’s darker regions. In other words, sensor black is not
the same as image black.The most common method for
calculating this adjustment is to take a picture of a
completely black field (typically accomplished by leaving
the lens cap on), resulting in three base offsets to be
subtracted from the raw sensor data. Failure to adjust the
black level will result in an undesirable loss of contrast.
n There are numerous sources of noise that can distort image
data – optical, electrical, digital and power
n The actual noise level present in an image, however, plays a
critical role in determining how strong the noise filter must
be since the use of a strong filter on a clean image will
actually distort and blur the image rather than clear it up.
n Noise reduction is achieved by averaging similar
neighboring pixels.Through the use of an Optical Electrical
Conversion Function (OECF) chart and a uniform lighting
source, the noise level can be characterized for different
n If the noise level is high for a particular intensity, then more
weight is given to the average pixel value of similar
n On the other hand, if the noise level is low, more weight is
given to the original pixel value.
n The OECF chart is comprised of 12 uniform gray patches and
produces 12 corresponding power levels based on the noise
standard deviation at the mean value for each intensity/
luminance level.These 12 power levels are then used to
reduce noise across an image using either a linear or square-
root model, depending on the sensor and gain (or ISO) level.
n Different types of lighting – such as incandescent,
fluorescent, natural light sources, LED flash – have a
pronounced effect on color.
n The most difficult to tune is in mixed-light conditions.
n White balance automatically compensates for color
differences based on lighting so white actually appears
n Fine tuning white balance begins by measuring the average
RGB values across the six gray patches on a ColorChecker
n Using mean square error minimization, the appropriate gains
for each color can be calculated.
The resulting gains are applied to each image pixel:
n Typically, digital cameras employ only a single sensor to capture
an image, so the camera can only obtain a single color
component for each pixel even though three components are
necessary to represent RGB color.
n CFA interpolation is the process of interpolating two missing
color components for each pixel based on the available
component and neighboring pixels.
n CFA interpolation is primarily a transform function that does not
vary based on sensor or lighting conditions, and therefore no
tuning of this image-processing pipeline stage is required.
n However, it is still one of the most complex algorithms in the
n Different sensors produce different RGB values for the same
n Tuning this pipeline stage involves creating a blending
matrix to convert the sensor RGB color space to a standard
RGB color space such as the Rec709 RGB color space.
n The blending matrix is calculated by starting with a
ColorChecker chart and obtaining average RGB values for 18
different color patches (the top three rows of the chart) that
have already been white balanced.
n Next, inverse Gamma correction is applied to the reference
RGB values.The blending matrix is then constructed using
n The blending matrix is applied as follows:
The final result is consistent color between cameras using different sensors.
n Gamma correction compensates for the nonlinearity of
relative intensity as the frame buffer value changes in output
n Typically, displays are calibrated using a standard gamma
correction such as Rec709 or SMPTE240M.
n Calibrating the image-processing pipeline to the same
standards ensures optimal image quality across the majority
RGB to YCC Conversion
n Images also need to be adjusted for the human eye, which is
more sensitive to luminance (Y) than color (Cb, Cr)
information.This pipeline stage separates luminance from
color for different processing using different precisions.
n Edge enhancement affects the sharpness of an image.
n In low-light conditions where images have a lower signal-to-
noise ratio (SNR), edge enhancement will boost noise making
it more visible to users.
n Contrast enhancement is comprised of two parameters:
contrast and brightness.
n Since the optimal contrast and brightness vary based on the
particular lighting conditions, as well as upon user
preference, these parameters are often implemented so that
they can be dynamically selected by the user.
False Chroma Suppression
n The final stage in the image-processing pipeline corrects
various color artifacts.
n Chroma suppression is a color correction that neutralizes the
green or blue colored light that often bounces off a green
screen or blue screen background and tints the edges of a
subject during a shoot.
n Bandwidth limitations of communication channels.
n Real-time processing requirements.
n Inter-media synchronization.
n Intra-media continuity.
n End-to-end delays and delay jitters.
n Multimedia indexing and retrieval.
n Increase in demand for fast and location
independent multimedia access
n Driving force behind growth of Mobile
n “Killer App” needed for success of 3G/4G
n Broad range of Applications – from
Entertainment to Telemedicine
What is Streaming
n Is it File download?
n Real Time consumption of data
n Low initial delay
n Media protection
n Lower Memory usage
Challenges in Video Streaming
n Packet Errors/Losses
n Buffer Management
n Low Delay
n Low Jitter
n No Re-transmission
What if Network is Wireless ?
n Higher Attenuation
n Multi-user interference
n Low Processing power and Memory
n Low Bit-rate Codec
n Error Resiliency
n Error Concealment
n Prevent Spatio-Temporal Error Propagation
n Flexibility of Bit-stream
n Feedback from Transport layer
RTSP RTCP RTP
SESSION SETUP &
CONTROL QUALITY FEEDBACK
(AUDIO & VIDEO)
RTCP-Real Time Control Protocol
n Adaptive encoders and streaming servers
can utilize the feedback information for
adjusting the stream to match the current
n Feedback is delivered in RTCP sender and
n periodic transmission of control packets to
all participants in the session
Video Coding – H.264
n Massive Quality, Minimal Files
n Scalable from 3G to HD and Beyond
n The New Industry Standard
n Latest Innovations in Video Technology
n Outperforms all preceding standards
H.264 - Performance
Use Scenario Resolution & Frame Rate Example Data Rates
Mobile Content 176x144, 10-15 fps 50-60 Kbps
Definition 640x480, 24 fps 1-2 Mbps
High Definition 1280x720, 24fps 5-6 Mbps
Full High Definition 1920x1080, 24fps 7-8 Mbps
H.264 – Bandwidth Adaptation
n Send one of several pre-encoded versions
of the same content, based on the current
n Frame dropping of non-reference frames, if
the channel rate fluctuates only in a small
n Instantaneous decoder refresh (IDR)
pictures to compensate large scale
variations of the channel rate
H.264 – Error Probability Reduction
n Slice-structured coding – Slices are
n Short slice/packets reduce the amount of
n Probability of a bit-error hitting a short
packet is generally lower than for large
n Short packets reduce the amount of lost
information and, hence, the error is limited
+Geolocation Techniques &
GPS Signal Processing
n For wireless communication networks, an inherently suitable
approach for wireless geolocation is known as radiolocation,
in which the parameters that are used for location estimation
are obtained fromradio signal measurements between a
target and one or more fixed stations.
n GPS is often inoperable in areas where satellites are blocked,
such as in buildings and built-up urban areas.
n Further, the time-to-first-fix (TTFF) for a conventional GPS
receiver from a “cold” start can take several minutes.
n Additionally, adding GPS functionality to a handset can be
costly, bulky, and drain battery power at an unacceptable rate
n The parameters that are often measured and used for
n angles of arrival (AOAs),
n signal strength, times of arrival (TOAs), and
n time differences of arrival (TDOAs).
n The fundamental methods of radiolocation that use these
parameters can be grouped into three categories:
n direction finding,
n ranging, and
n range differencing.
Algorithms For Geolocation
n Geometrical Techniques
n When there are errors in the measured AOAs, ranges, or
range differences, statistical solutions are more justifiable in
the presence of measurement errors
n Least Squares Estimation - estimating parameters by
minimizing the squared discrepancies between observed
data, on the one hand, and their expected values on the other
2D Bar Code
n Defining of threshold as pre-processing,
n Detecting a black bar using spiral search method, and
n Finding the sampled scanning line which is perpendicular to
the detected bar in phase
n Pre-processing The gray level histogram calculation is
n Corner marks detection Three marked corners are detected
using the finder pattern,
n Fourth corner estimation The fourth corner is detected using
the special algorithm,
n Inverse perspective transformation Inverse transformation is
adopted based on the obtained corner geometry positions to
normalize the size of the code, and
n Scanning of code Sample the inside of code and output the
normalized bi-level code data to host CPU.
n Augmented Reality Video
Display Info for
Top Ranked Image