Signal processing in smartphones - 4G perspective


Published on

Published in: Technology, Business
1 Comment
1 Like
No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Signal processing in smartphones - 4G perspective

  1. 1. + Signal Processing in Smartphones – 4G Perspective Ramesh Prasad
  2. 2. + About Me n  M.E. in Electronics & Telecommunications Engineering n  15+Years of Technology Experience n  Senior Member IEEE n  Entrepreneur/Startup Enthusiast/Technologist/Evangelist/ Author/Speaker n  United States Patent 7,522,774 n
  3. 3. + Agenda n  Introduction n  Evolution of Mobile Communications n  Smartphone Anatomy n  Smartphone Hardware Architecture n  Smartphone Software Architecture n  Signal Processing In Smartphones n  Communications Signal Processing n  Speech Processing (VoIP)
  4. 4. + Agenda n  Camera Image And Video Processing n  Display Image Processing n  Audio Post Processing n  Signal Processing For Multimedia Communication n  Geolocation Techniques & GPS Signal Processing n  Signal Processing For Apps n  Voice Enabled Services n  Scanner and BarCode Reader n  Augmented Reality n  References
  5. 5. + Introduction
  6. 6. + Scanner & Bar Code Reader Media Player Internet Email Chat Navigation Games TV Phone Credit Card AppsPersonal Assistant Social Video Conf. Bluetooth/ WiFi/NFC Camera
  7. 7. + Technology Enablers Signal Processing Communications VLSI Mobile Computing
  8. 8. + Modern Communications Theory
  9. 9. +Evolution of Mobile Communications
  10. 10. + Wireless Technology Generations
  11. 11. + Wireless Technology Migration
  12. 12. + Introduction to 4G (LTE) n  Standard for wireless communication of high-speed data for mobile phones and data terminals n  Evolution of 3G systems n  Developed by the 3GPP (3rd Generation Partnership Project) n  OFDM (Orthogonal Frequency Division Multiplexing) to avoid ISI n  MIMO (Multiple-Input Multiple-Output) to boost Data Rates n  All-IP flat architecture supporting QoS
  13. 13. + Drivers For 4G n  Creation and development of new services for mobile devices n  Advancement of the Signal Processing and Communications technologies for mobile systems n  Advancement of the VLSI technology n  Competition between mobile operators n  Challenges from other mobile technologies n  New regulation of spectrum use
  14. 14. + Features Of 4G n  Provides a global ecosystem with inherent mobility n  Offers easier access and use with greater security and privacy n  Dramatically improves speed and latency n  Delivers enhanced real-time video and multimedia for a better overall experience n  Enables high-performance mobile computing n  Supports real-time applications due to its low latency n  Creates a platform upon which to build and deploy the products and services of today and those of tomorrow n  Reduces cost per bit through improved spectral efficiency
  15. 15. + Benefits Of 4G n  High peak speeds: n  100 Mbps downlink (20 MHz, 2x2 MIMO)—both indoors and outdoors n  50 Mbps uplink (20 MHz, 1x2) n  At least 200 active voice users in every 5 MHz (i.e., can support up to 200 active phone calls) n  Low latency: n  < 5 ms user plane latency for small IP packets n  < 100 ms camped to active n  < 50 ms dormant to active
  16. 16. + Benefits Of 4G n  Scalable bandwidth: n  The 4G channel offers four times more bandwidth than current 3G systems and is scalable. So, while 20 MHz channels may not be available everywhere, 4G systems will offer channel sizes down to 5 MHz, in increments of 1.5 MHz. n  Improved spectrum efficiency: n  Spectrum efficiency refers to how limited bandwidth is used by the access layer of a wireless network. Improved spectrum efficiency allows more information to be transmitted in a given bandwidth, while increasing the number of users and services the network can support. n  Two to four times more information can be transmitted versus the previous benchmark, HSPA Release 6.
  17. 17. + Benefits Of 4G n  Improved cell edge data rates: n  Not only does spectral efficiency improve near cell towers, it also improves at the coverage area or cell edge. n  Data rates improve two to three times at the cell edge over the previous benchmark, HSPA Release 6. n  Packet domain only n  Enhanced support for end-to-end quality of service: n  Reducing handover latency and packet loss is key to delivering a quality service. n  This reduction is considerably more challenging with mobile broadband than with fixed-line broadband. n  The time variability and unpredictability of the channel become more acute. n  Additional complications arise from the need to hand over sessions from one cell to another as users cross coverage boundaries.These handover sessions require seamless coordination of radio resources across multiple cells.
  18. 18. + Technical Specifications & Attributes
  19. 19. + 3G Limitations n  The maximum bit rates still are factor of 20 and more behind the current state of the systems like 802.11n and 802.16e/m. n  The latency of user plane traffic (UMTS: >30 ms) and of resource assignment procedures (UMTS: >100 ms) is too big to handle traffic with high bit rate variance efficiently. n  The terminal complexity for WCDMA or MC‐CDMA systems is quite high, making equipment expensive, resulting in poor performing implementations of receivers and inhibiting the implementation of other performance enhancements.
  20. 20. + 3G vs 4G
  21. 21. +Challenges in 4G Mobile System Design
  22. 22. + Challenges in 4G Mobile System Design n  The goal of 4G has been made possible by sophisticated signal processing algorithms n  Signal processing for 4G communications- n  Antenna n  RF n  Modulation n  Baseband n  Source coding n  Channel coding n  Signal Processing for Services and Apps
  23. 23. + Challenges in 4G Mobile System Design
  24. 24. + Signal Processing Challenges in 4G Performance Cost Power
  25. 25. + Signal Processing Challenges in 4G n  Data rate - Unfortunately, the growth of data rates is not matched by advanced in semiconductor structures, terminal power consumption improvements or battery lifetime improvements. New processing architectures and algorithms are needed to cope with these data rates. n  High performance MIMO receivers such as maximum- likelihood-like receivers, sphere decoders or interference rejection combiners offer substantial performance gains but impose an implementation challenge, especially when the high peak data rates are targeted.
  26. 26. + Signal Processing Challenges in 4G n  LTE utilizes precoding, which requires accurate channel estimation. Advanced methods like iterative decision directed channel estimation offer performance improvements, but pose again a complexity challenge. n  LTE has a large “toolkit” of MIMO schemes and adaptive methods.The selection and combination of the right method in a cell with heterogeneous devices, channel conditions and bursty data services is a challenge.
  27. 27. + Signal Processing Challenges in 4G n  LTE roll-out will be gradual in most cases– interworking with other standards such as GSM or HSPA will be required for a long time.This imposes not only a cost and complexity issue. One of the reasons many early 3G terminals had poor power consumption was the need for 2G cell search and handover in addition to normal 3G operation. Reduced talk-time for dual-mode devices is not acceptable. n  “Dirty RF” challenges. OFDM systems cause problems with power amplifier nonlinearity, and are sensitive to frequency errors and phase noise. Digital compensation techniques are proposed for the transmitter and receiver, but innovation is needed to make them reality in low cost devices.
  28. 28. + Smartphone Anatomy
  29. 29. + Smartphone Anatomy
  30. 30. + Smartphone Anatomy
  31. 31. + Smartphone Anatomy
  32. 32. + Smartphone Anatomy
  33. 33. +Smartphone Hardware Architecture
  34. 34. + Hardware Architecture
  35. 35. + Single Chip Solution - SoC
  36. 36. + SoC – Functional Blocks
  37. 37. +Smartphone Software Architecture
  38. 38. + Software Architecture
  39. 39. + Software Architecture - Telephony RF Driver RF Modules Protocol Stack Interface Protocol Stack Telephony Middleware Telephony Driver Telephony Apps Telephony Middleware Interface
  40. 40. + Application Stack - Android
  41. 41. +Challenges & Solutions For Wireless Communications
  42. 42. + Challenges in Wireless Communications n  Signal Propagation n  Signal Attenuation and Path Loss n  Multipath Effect n  Signal Spread n  Interference n  Noise
  43. 43. + Signal Propagation Effect
  44. 44. + Signal Attenuation and Path Loss
  45. 45. + Multipath Effect
  46. 46. + Delay & Doppler Spread
  47. 47. + Interference
  48. 48. + Noise n  Thermal Noise n  White Noise n  Flicker Noise n  Phase Noise n  Burst Noise n  Shot Noise n  Avalanche Noise
  49. 49. + Overcoming Challenges in Wireless Communications n  Diversity – Time, Frequency, Space n  Higher Order Modulation n  Channel Coding n  Interleaving n  Channel Estimation & Equalization n  Multi Carrier Transmission
  50. 50. + Fundamental Limits on Data Rates
  51. 51. + Fundamental Limits on Data Rates n  To provide high data rates as efficiently as possible, the transmission bandwidth should be at least of the same order as the data rates to be provided n  For provisioning of higher data rates with good coverage, wider transmission bandwidth is required n  Challenges n  Spectrum is a scarce and expensive resource n  The use of wider transmission and reception bandwidths has an impact on the complexity of the radio equipment, both at the base station and at the terminal. n  increased corruption of the transmitted signal due to time dispersion on the radio channel
  52. 52. + Multi-Carrier Transmission M symbols transmitted in parallel, so rate R becomes R/N
  53. 53. + Principles of OFDM
  54. 54. + OFDM Transmitter Receiver
  55. 55. + Simplified Realization of OFDM
  56. 56. + OFDMA
  57. 57. + SC - FDMA
  58. 58. + SC-FDMA Transmitter Receiver
  59. 59. + MIMO
  60. 60. +Signal Processing For Communications
  61. 61. + Protocol Stack in 4G Handset
  62. 62. + Protocol Stack - PHY n  To support the high data rate, exceptionally large amounts of processing power are needed, particularly in the baseband, where all the error handling and signal processing occurs. n  The LTE PHY employs orthogonal frequency division multiplexing (OFDM) and multiple input multiple output (MIMO) data transmission. n  The LTE PHY uses orthogonal frequency division multiple access (OFDMA) on the downlink (DL) and single carrier – frequency division multiple access (SC-FDMA) on the uplink (UL).
  63. 63. + Signal Processing in PHY Downlink Uplink
  64. 64. + PHY Layer in 4G Handset
  65. 65. + Speech Processing (VoIP)
  66. 66. + Speech Coder Attributes n  Bitrate n  Quality n  Complexity n  Delay n  Robustness
  67. 67. + VoIP Technology Barriers n  End-to-End Delay and Jitter n  Packet Loss n  Throughput n  Internet Availability and Reliability n  Security and Confidentiality
  68. 68. + VoIP Over Wireless n  Channel Quality and Adaptive Operation n  Background Noise n  Tandeming n  Voice Activity Detection n  Unequal Error Protection n  Frame Erasures
  69. 69. + AMR Wideband n  Adaptive Multi-Rate Wideband (AMR-WB) is a patented wideband speech coding standard developed based on Adaptive Multi-Rate encoding, using similar methodology as Algebraic Code Excited Linear Prediction (ACELP). n  AMR-WB provides improved speech quality due to a wider speech bandwidth of 50–7000 Hz compared to narrowband speech coders which in general are optimized for POTS wireline quality of 300– 3400 Hz. n  AMR-WB codec has the following parameter: n  Bit Rate – 16 Kbps n  Delay frame size: 20 ms n  Look ahead: 5ms n  Complexity: 38 WMOPS, RAM 5.3KWords n  Voice activity detection, Discontinuous Transmission, Comfort Noise Generator n  Fixed point: Bit-exact C
  70. 70. + AMR Wideband Encoder
  71. 71. + AMR Wideband Decoder
  72. 72. +Camera Image And Video Processing
  73. 73. + Smartphone Camera Features n  Video Camcorder n  Still Image Capture n  Auto Focus & Metering n  Low Light Photography n  Zero Shutter Lag Image Capture & Video Snapshot n  Face Detection & Face Tracking n  Imaging Responsiveness n  Video Editing n  Image Post Processing
  74. 74. + Image Processing Pipeline
  75. 75. + Black-Level Adjustment n  This stage in the pipeline adjusts for dark current from the sensor and for lens flare, which can lead to the whitening of an image’s darker regions. In other words, sensor black is not the same as image black.The most common method for calculating this adjustment is to take a picture of a completely black field (typically accomplished by leaving the lens cap on), resulting in three base offsets to be subtracted from the raw sensor data. Failure to adjust the black level will result in an undesirable loss of contrast.
  76. 76. + Noise Filtering n  There are numerous sources of noise that can distort image data – optical, electrical, digital and power n  The actual noise level present in an image, however, plays a critical role in determining how strong the noise filter must be since the use of a strong filter on a clean image will actually distort and blur the image rather than clear it up. n  Noise reduction is achieved by averaging similar neighboring pixels.Through the use of an Optical Electrical Conversion Function (OECF) chart and a uniform lighting source, the noise level can be characterized for different intensities.
  77. 77. + Noise Filtering n  If the noise level is high for a particular intensity, then more weight is given to the average pixel value of similar neighbors. n  On the other hand, if the noise level is low, more weight is given to the original pixel value. n  The OECF chart is comprised of 12 uniform gray patches and produces 12 corresponding power levels based on the noise standard deviation at the mean value for each intensity/ luminance level.These 12 power levels are then used to reduce noise across an image using either a linear or square- root model, depending on the sensor and gain (or ISO) level.
  78. 78. + Noise Filtering
  79. 79. + White Balance n  Different types of lighting – such as incandescent, fluorescent, natural light sources, LED flash – have a pronounced effect on color. n  The most difficult to tune is in mixed-light conditions. n  White balance automatically compensates for color differences based on lighting so white actually appears white.
  80. 80. + White Balance
  81. 81. + White Balance n  Fine tuning white balance begins by measuring the average RGB values across the six gray patches on a ColorChecker chart
  82. 82. + White Balance n  Using mean square error minimization, the appropriate gains for each color can be calculated. The resulting gains are applied to each image pixel:
  83. 83. + CFA Interpolation n  Typically, digital cameras employ only a single sensor to capture an image, so the camera can only obtain a single color component for each pixel even though three components are necessary to represent RGB color. n  CFA interpolation is the process of interpolating two missing color components for each pixel based on the available component and neighboring pixels. n  CFA interpolation is primarily a transform function that does not vary based on sensor or lighting conditions, and therefore no tuning of this image-processing pipeline stage is required. n  However, it is still one of the most complex algorithms in the image-processing pipeline
  84. 84. + CFA Interpolation
  85. 85. + RGB Blending n  Different sensors produce different RGB values for the same color. n  Tuning this pipeline stage involves creating a blending matrix to convert the sensor RGB color space to a standard RGB color space such as the Rec709 RGB color space.
  86. 86. + RGB Blending n  The blending matrix is calculated by starting with a ColorChecker chart and obtaining average RGB values for 18 different color patches (the top three rows of the chart) that have already been white balanced. n  Next, inverse Gamma correction is applied to the reference RGB values.The blending matrix is then constructed using constrained minimization.
  87. 87. + RGB Blending n  The blending matrix is applied as follows: The final result is consistent color between cameras using different sensors.
  88. 88. + Gamma Correction n  Gamma correction compensates for the nonlinearity of relative intensity as the frame buffer value changes in output displays. n  Typically, displays are calibrated using a standard gamma correction such as Rec709 or SMPTE240M. n  Calibrating the image-processing pipeline to the same standards ensures optimal image quality across the majority of displays.
  89. 89. + RGB to YCC Conversion n  Images also need to be adjusted for the human eye, which is more sensitive to luminance (Y) than color (Cb, Cr) information.This pipeline stage separates luminance from color for different processing using different precisions.
  90. 90. + Edge Enhancement n  Edge enhancement affects the sharpness of an image. n  In low-light conditions where images have a lower signal-to- noise ratio (SNR), edge enhancement will boost noise making it more visible to users.
  91. 91. + Contrast Enhancement n  Contrast enhancement is comprised of two parameters: contrast and brightness. n  Since the optimal contrast and brightness vary based on the particular lighting conditions, as well as upon user preference, these parameters are often implemented so that they can be dynamically selected by the user.
  92. 92. + Contrast Enhancement
  93. 93. + False Chroma Suppression n  The final stage in the image-processing pipeline corrects various color artifacts. n  Chroma suppression is a color correction that neutralizes the green or blue colored light that often bounces off a green screen or blue screen background and tints the edges of a subject during a shoot. 
  94. 94. + Display Image Processing
  95. 95. +
  96. 96. + Audio Post Processing
  97. 97. +Signal Processing For Multimedia Communication
  98. 98. + Applications n  Video-on-demand. n  Distance learning and training. n  Interactive gaming. n  Remote shopping. n  Online media services,such as news reports. n  Videotelephony. n  Videoconferencing. n  Telemedicine for remote consultation and diagnosis. n  Telesurveillance. n  Remote consultation or scene- of-crime work. n  Collaborative working and telepresence.
  99. 99. + Challenges n  Higher coding efficiency n  Reduced computational complexity n  Improved error resilience
  100. 100. + MultiMedia Communications
  101. 101. + Challenges n  Bandwidth limitations of communication channels. n  Real-time processing requirements. n  Inter-media synchronization. n  Intra-media continuity. n  End-to-end delays and delay jitters. n  Multimedia indexing and retrieval.
  102. 102. + Audio/Video Streaming n Increase in demand for fast and location independent multimedia access n Driving force behind growth of Mobile Internet n “Killer App” needed for success of 3G/4G systems n Broad range of Applications – from Entertainment to Telemedicine
  103. 103. + What is Streaming n  Is it File download? n  Real Time consumption of data n  Advantages n  Low initial delay n  Media protection n  Lower Memory usage
  104. 104. + Challenges in Video Streaming n  Bit-rate n  Packet Errors/Losses n  Buffer Management n  Low Delay n  Low Jitter n  No Re-transmission
  105. 105. + What if Network is Wireless ? n  Higher Attenuation n  Shadowing n  Fading n  Multi-user interference n  Mobility/Handoff n  Low Processing power and Memory
  106. 106. + Solution n  Low Bit-rate Codec n  Error Resiliency n  Error Concealment n  Prevent Spatio-Temporal Error Propagation n  Flexibility of Bit-stream n  Feedback from Transport layer
  107. 107. + Network Architecture
  109. 109. + RTCP-Real Time Control Protocol n Adaptive encoders and streaming servers can utilize the feedback information for adjusting the stream to match the current transport quality n Feedback is delivered in RTCP sender and receiver reports n periodic transmission of control packets to all participants in the session
  110. 110. + Video Coding – H.264 n  Massive Quality, Minimal Files n  Scalable from 3G to HD and Beyond n  The New Industry Standard n  Latest Innovations in Video Technology n  Outperforms all preceding standards
  111. 111. + H.264 - Innovations n multi-frame motion-compensated prediction n  adaptive block size for motion compensation n generalized B-Pictures n quarter-pel motion accuracy n intra coding utilizing prediction in the spatial domain n in-loop de-blocking filters n efficient entropy coding methods
  112. 112. + H.264 – Block Diagram
  113. 113. + H.264 - Performance Use Scenario Resolution & Frame Rate Example Data Rates Mobile Content 176x144, 10-15 fps 50-60 Kbps Internet/Standard Definition 640x480, 24 fps 1-2 Mbps High Definition 1280x720, 24fps 5-6 Mbps Full High Definition 1920x1080, 24fps 7-8 Mbps
  114. 114. + H.264 – Bandwidth Adaptation n Send one of several pre-encoded versions of the same content, based on the current channel bit-rate n Frame dropping of non-reference frames, if the channel rate fluctuates only in a small range n Instantaneous decoder refresh (IDR) pictures to compensate large scale variations of the channel rate
  115. 115. + H.264 – Error Probability Reduction n Slice-structured coding – Slices are independently coded n Short slice/packets reduce the amount of lost information n Probability of a bit-error hitting a short packet is generally lower than for large packets n Short packets reduce the amount of lost information and, hence, the error is limited
  116. 116. + H.264 – Error Resilience n  Flexible MB Ordering (FMO) n  Data partitioning - unequal error protection
  117. 117. + H.264 – Error concealment n  Intra coded MBs n  Multiple Reference Frames n  Redundant Slices
  118. 118. +H.264 – Intra Frame Error Concealment
  119. 119. +H.264 – Inter Frame Error Concealment
  120. 120. +Geolocation Techniques & GPS Signal Processing
  121. 121. + Geolocation Techniques n  For wireless communication networks, an inherently suitable approach for wireless geolocation is known as radiolocation, in which the parameters that are used for location estimation are obtained fromradio signal measurements between a target and one or more fixed stations. n  GPS is often inoperable in areas where satellites are blocked, such as in buildings and built-up urban areas. n  Further, the time-to-first-fix (TTFF) for a conventional GPS receiver from a “cold” start can take several minutes. n  Additionally, adding GPS functionality to a handset can be costly, bulky, and drain battery power at an unacceptable rate
  122. 122. + Geolocation Methods n  The parameters that are often measured and used for location include- n  angles of arrival (AOAs), n  signal strength, times of arrival (TOAs), and n  time differences of arrival (TDOAs). n  The fundamental methods of radiolocation that use these parameters can be grouped into three categories: n  direction finding, n  ranging, and n  range differencing.
  123. 123. + Direction Finding
  124. 124. + Ranging
  125. 125. + Range Differencing
  126. 126. + Algorithms For Geolocation n  Geometrical Techniques n  When there are errors in the measured AOAs, ranges, or range differences, statistical solutions are more justifiable in the presence of measurement errors n  Least Squares Estimation - estimating parameters by minimizing the squared discrepancies between observed data, on the one hand, and their expected values on the other
  127. 127. + GPS
  128. 128. + GPS Signals
  129. 129. + GPS Receiver Adaptive Space Time Array Receiver
  130. 130. + Signal Processing For Apps
  131. 131. + Voice Enabled Services n  Google Voice Search Video
  132. 132. + Distributed Speech Recognition
  133. 133. + Scanner & Bar Code Reader
  134. 134. + 2D Bar Code n  Defining of threshold as pre-processing, n  Detecting a black bar using spiral search method, and n  Finding the sampled scanning line which is perpendicular to the detected bar in phase
  135. 135. + QR Code n  Pre-processing The gray level histogram calculation is adopted, n  Corner marks detection Three marked corners are detected using the finder pattern, n  Fourth corner estimation The fourth corner is detected using the special algorithm, n  Inverse perspective transformation Inverse transformation is adopted based on the obtained corner geometry positions to normalize the size of the code, and n  Scanning of code Sample the inside of code and output the normalized bi-level code data to host CPU.
  136. 136. + Visiting Card Scanner
  137. 137. + Augmented Reality n  Augmented Reality Video
  138. 138. + Image Matching Query Image Prefetched Data Database Images
  139. 139. + System Architecture Geo-Tagged Images Group Images by Loxel Extract Features Camera Image Extract Features Compute Feature Matches Geometric Consistency Check Match Images Geometric Consistency Check Display Info for Top Ranked Image Device Location Feature Cache ANN Loxel-Based Feature Store Server Network Cluster Features Prune Features Compress Descriptors
  140. 140. + References
  141. 141. + References n  Mobile Handset Design – Sajal Das n  Signal Processing for Mobile Communications Handbook – Mohamed Ibnkahla n  4G LTE/LTE-Advanced for Mobile Broadband - Erik Dahlman