Sumit& archit osi nov-2011-displays-in-mobile-devices

Displays in Mobile Devices OSI Days 2011, Bangalore Archit Taneja (archit@ti.com) Sumit Semwal (sumit.semwal@ti.com) 1 Linux Development Centre

Agenda• Historical look at mobile devices – Non-phone mobile devices• Types of display panels• Display Subsystems – In PC World – SoC World – PC vs SoC – Example Data Flow• Mobile Display Interface Standards• Interesting Display Use Cases• Example of SoC DSS: – OMAP2+ Display Subsystem – Current software design – DSS2 framework• Linux Kernel Display Frameworks• What Next?• Q&A 2 Linux Development Centre

Under CC License from http://xkcd.com/732/ Linux Development Centre

A Historical Look at Mobile Displays LG Optimus 3D 3D Stereoscopic LCD; 800x480; 4.3” [4] Nokia 9000 Communicator: First business phone Google Galaxy Nexus Monochrome display, Wide Super AMOLED plus 640x200: [3] 720x1280; 4.65”Nokia 1011First mass One of the firstproduced color screenGSM phone phonesMonochrome Sony Ericsson T68i[1] 101x80, 256color [2] Apple iPhone 4 Google Nexus One “Retina” display AMOLED: 480x800, 3.7” 960 x 640; 3.5” [5] All trademarks and copyrights are properties of and attributed to their respective companies Sources: in references

non-phone Mobile Displays Device Positioning Screen Resolution Technology size Samsung Tablet 5.3” 1280 x 800 HD sAMOLED Galaxy Note Smartphone Dell Phone 5” 800 x 480 TFT LCD Tablet 7” 800 x 480 Capacitative 10.1” 1366 x 768 TFT Amazon eBook Reader 6” 600 x 800 eInk, 16 level Kindle 3 grayscale B&N Nook eBook Reader 6” 800 x 600 eInk, 16 level grayscale Archos PMP 7” 800 x 480 TFT Tablet 8” 1024 x 768 Tablet 10.1” 1024 x 600 Apple iPad Tablet 9.7” 1024 x 768 IPS LED Samsung Tablet 7” 1024 x 600 TFT Galaxy Tab 8.9” 1280 x 800 10.1” 1280 x 800 Only an indicative list; certainly not exhaustive!All trademarks and copyrights are properties of and attributed to their respective companies LinuxSources: in references Development Centre

Types of Display Panels• Retina – Apple iPhone 4 – In Plane Switching (IPS) LCD – Highest density: 326 ppi (3.5” screen, 960x640)• AMOLED (Active Matrix Organic LED) – No Backlight needed • Higher contrast ratios, more power efficient. – Pen Tile arrangement – RGBG arrangement of sub-pixels, where G sub- pixel is shared by 2 pixels. So the display has only 2 sub-pixels per real pixel.• Super AMOLED – Same as AMOLED, but touchscreen is integrated onto the same glass, thus increasing visibility on the screen.All trademarks and copyrights are properties of and attributed to their respective companies LinuxSources: in references Development Centre

Types of Display Panels• Super AMOLED Plus – Integrated touchscreen, and no Pen tile arrangement – much brighter, thinner and more efficient. – regular RGB, so 50% more subtiles, hence lesser resolution.• eInk Panel – Based on actual electronic ink that is coated on some ultra-thin plastic. – eInk contains microcapsules filled with ink, and pigmented, charged chips. – Each microcapsule acts as a pixel; when an electric charge is applied, either the light or dark pigmented chip will come on top, thus switching the pixel on or off.All trademarks and copyrights are properties of and attributed to their respective companies LinuxSources: in references Development Centre

Display Subsystems: PC World 2D/3D HDMI Accelerators Pixel Gen 1 Display Port Port Video Pixel Gen 2 Decoders Interconnect Display(s) LVDSScene information Overlay / Pixel Gen M or Analog Blending Encoded frames Video GPU Display I/F Display Subsystem Shared/Dedicated Memory Linux Development Centre

Generic SoC Hardware Blocks Video Display Sub MPU GPU Accelerator System Display(s) interconnect EMIF DSP Imaging Other Peripherals Memory Only display related blocks are shown; there are many other blocks too, which are out of scope for this discussion. Linux Development Centre

Display Subsystem in SoCs Serial Plane 1 Compositor 1 Interface & (MIPI DSI) Pixel Gen Plane 2 Parallel Interface (MIPI DPI) Compositor 2 Plane 3 & Interconnect Remote Display(s) Pixel Gen framebuffer Interface (MIPI DBI)GPU composed frames orDecoded Video frames Compositor M HDMI/ & Display Port Plane N Pixel Gen Overlay Display I/F Display Subsystem Linux Development Centre

Display Subsystems: PC vs SoC• Differences: – PC: GPUs are generally responsible for composing different windows/layers to get one single frame. Display Subsystem only fetches and display this composed frame . – SoC: Display Subsystem is also responsible for composing layers generated by the GPU and Video accelerators. – PC: Display Subsystem may only fetch frame data from the dedicated memory for the graphics card/controller. – SoC: Display Subsystem fetches frame data from the main memory connected to the SoC.• Reasons: – Legacy: GPUs and Video Accelerators have existed in SoCs as separate entities for a while. Making it important for the Display Subsystems to compose the output of each into one frame. – Area: SoCs (and mobile devices) cant afford dedicated memory for GPUs. – IP reuse: Different Vendors of GPUs and Display Subsytems in SoCs, easier to interface with a standard interconnect. – Power: Overlaying different layers is more optimal for SoCs in some cases. Linux Development Centre

Example data flow 3. Display frame on panel 1. Video frames to decode 1. Commands to draw UI 2.Done Video Display Sub MPU GPU Accelerator System 2. Done interconnect 2. Soc 2. 3. Display Control path Data path MemoryOnly display related blocks are shown; there are many other blocks too, which are out of scope Linuxfor this discussion. Development Centre

Mobile Display Interface StandardsMIPI- Display Working GroupPrimary (fixed) panels: - Display Serial Interface (DSI): - serializes all pixel data, commands, and events. - low power(SLVS) - Works with Smart / Dumb Panels. - Display Parallel Interface (DPI): - Legacy parallel interface. - Lots of lanes needed for pixel data and control events. - used with Dumb Panels. - Display Bus Interface (DBI): - Parallel Interface for smart panels.Pluggable panels:- HDMI- Display Port- SDTV / Composite- DVIAll trademarks and copyrights are properties of and attributed to their respective companies LinuxSources: in references Development Centre

Interesting Display Use cases• Device Dock• Dynamic configurability, parallel usage – Switch output paths, using multiple paths simultaneously• Handle audio interfacing additionally – HDMI support – DSI support in future• Support for virtual pipelines – when we run out of video pipelines – Co-use of 2D/3D graphics accelerator.• Multi-display configurations – Clone, Virtual.● Display security 14 Linux Development Centre

OMAP2+ DSS hardware: Overview OMAP2,3,4 Display OMAP3,4 Panels OMAP4 DISPC DSI1 encoderGFX ● Gfx, vid1-vid3 are Primary DSI overlays, which color- convert, scale each frameVID1 LCD1 RFBI mgr RFBI ● Managers can compose,VID2 overlay multiple streams LCD2 DSI2 encoder into one.VID3 mgr Secondary DSI ● Not all paths are shown: DPI All overlays connect to all TV overlay managers. mgr DPI HDMI encoder VENC HDMI Display Interfaces SDTV DSS hardware 15 Linux Development Centre

Current software design – DSS2 framework UI // Graphics UI Graphics Video playback Video playback Userspace app Userspace app Control Userspace app Userspace appUserspace Framebuffer APIs V4L2 APIs /dev/fb0 /dev/video1 sysfs /dev/fb1 /dev/video2 Panel omapfb driver drivers omap_vout driverKernelspace gfx vid1 vid2 vid3 DSS2 DSS2 lcd1-mgr lcd2-mgr tv-mgr OMAP DSS hardwareDisplayhardware HDMI TV LCD panels 16 Linux Development Centre

Linux Kernel Display Frameworks• Frame Buffer – Oldest of them all - first public release in 1999. – Linux Kernel Abstraction for graphics hardware – represents framebuffer of some video hardware, so the software doesnt need to know the low-level (hardware) stuff. – allows emulate text console without hardware text-mode support. – is a possible graphic output method for X server; independent of video adapter hardware and drivers. – allows direct graphics programs without X; popular in embedded world [GTK+, Qt Extended are some examples] – No inherent window or buffer management• DirectFB [6] – Specifically designed for embedded systems. – Is a thin library providing hardware graphics acceleration. – Integrated windowing system with support for translucent windows and multiple display layers. – Input device handling. – Is a complete hardware abstraction layer, with software fallbacks to every graphics operation not supported by an underlying hardware. – Provides for X-less graphical systems: gtk+, LinuxTV, HP WebOS to name a few. Linux Development Centre

Linux Kernel Display Frameworks• Video For Linux (V4L) – Kernel interface for streaming capture and output drivers. – Provides framework for buffer management and streaming – very relevant for streaming kind of use-cases [video record, playback, TV, even analog radio! :)] – Buffers are tied to each device, and allocated before streaming starts.• Direct Rendering Manager (DRM) – The modern Linux (and *BSD) display driver framework – The kernel component of DRI (Direct Rendering Infrastructure) – But not just DRI: • Hotplug • XRandR/KMS • EDID parsing – Kernel Mode Setting (KMS) • Setting screen resolution/timings • Hotplug, and retrieving EDID • Multi-display configurations – Buffers are independent of overlays – Security inherent Linux Development Centre

What Next?- HDMI 3D- eInk Color (Triton) [7] Linux Development Centre

Q and A? 20 Linux Development Centre

References 3430 public Technical Reference Manual 4430 public Technical Reference Manual[1]: http://en.wikipedia.org/wiki/List_of_Nokia_products[2]: http://www.mobiles.co.uk/mobile-phone-history-uk.html[3]: http://royal.pingdom.com/2010/02/08/15-fantastic-firsts-on-the-internet/[4]: http://www.lg.com/in/mobile-phones/all-phones/LG-smart-phones-P920.jsp[5]: http://gigaom.com[6]: http://directfb.org[7]: http://www.eink.com/display_products_triton.html Linux Development Centre

Thank You! 22 Linux Development Centre

Backup 23 Linux Development Centre

Acronyms• SoC: System on Chip• DSI: Display Serial Interface: a MIPI standard interface for serial display panels.• DPI: Display Pixel Interface: a MIPI standard interface for parallel display panels.• RFBI: Remote Frame Buffer Interface: a MIPI standard interface for parallel display panels supporting remote frame buffer mechanism.• HDMI: High Definition Multimedia Interface: a standard compact AV interface to transmit uncompressed digital data.• VENC: Video ENCoder: standard interface to display content on standard definition TVs, using TV formats like NTSC / PAL. 24 Linux Development Centre

Features of OMAP2+ DSS Feature OMAP2430 OMAP3430 OMAP4430 GFX overlay 1 1 1 VID overlay 2 2 3 WB overlay - - 1 LCD overlay 1 1 2 manager TV overlay 1 1 1 manager Rotation Via VRFB* Via VRFB* Via TILER* (per overlay)OMAP3/4 support ARGB color formats.OMAP4 additionally has native support for NV12, configurable Z-order.Overlay perspectiveGFX overlay supports CLUT/gamma table, anti-flicker, replication logic – primarily meant for UI.VID overlays support Multiple color formats(RGB, YUV, NV12 based on OMAP version), colorspace conversion, independent horizontaland vertical polyphase filter scaling, and VC-1 range mapping.WB overlay, in mem-to-mem mode: colorspace conversion, rescaling, compositing processes. Capture mode: allows composited encoded pixel data to be stored back into memory.Overlay perspectiveEach overlay supports: overlaying [fixed-order for OMAP2/3, z-order per pipeline for OMAP4], transparency color-keys [source anddestination] and global and pixel alpha blending [OMAP3/4], Gamma correction, Sync buffers.LCD overlays additionally support Active / Passive matrix dithering, color phase rotation. 25* Read more about Virtual Rotated Frame Buffer (VRFB) and TILER in the 3430 public TRM and 4430 public TRM respectively Linux Development Centre

Sumit& archit osi nov-2011-displays-in-mobile-devices

by OpenSourceIndia on Dec 15, 2011

Accessibility

Categories

Upload Details

Usage Rights

Statistics

Sumit& archit osi nov-2011-displays-in-mobile-devices — Presentation Transcript