SlideShare a Scribd company logo
1 of 18
Download to read offline
Infrared simulation and processing
On
Nvidia Platforms
Neuronics
• Developing projects in
• Computer vision & AI
• 3D generation and simulation
• Nvidia Edge and server platforms and SDKs
• Defense / Military
• UAV / Autonomous vehicles
• Autonomous vassals
• robots
• Medical
COMPUTER
VISION & AI
NVIDIA
PLATFORMS
3D
GENERATION
&
SIMULATION
Agenda
•Solve IR camera processing challenges
IR Camera chalanges
Noise!
Camera connection
•Camera Link -> Capture card
•GMSL – Fiber ( for distanced camera)
•USB
•GIGE
•HDMI
•RAW Transfer
Noise
•DSNU - Dark Signal Non-Uniformity ~1% bias
•PRNU - Photo-Response Non-Uniformity ~1% gain
variability
•Thermal noise
•Sensor
•Gain
•Temp
Data problems
•Capture & Label Data – expensive / not possible
•Augmentation - Sensitive
•Just changing brightness like in RGB images wont work!
•Simulation
•3D World simulation (Real, Blender, Omniverse, Unity)
•Thermal assignment (simple)
• Glitter simulation / Ray tracing
•Atmospheric simulation (Frequency, distance, angle)
Simulation – Basic
3D World
simulation
Temperature
Map
Sensor
Simulation
Distance
Map
Glitter Map
Shared
config
Sensor Simulation – Advance
Temperature
to electrons
Lens Shading
PSF
) Point spread function(
Motion Blur
PRNU /
DSNU
NUC Simulation
Shot Noise
Electrons to
sensor out
Temperature
map
Distance Map
Glitter Map
Per frame
exposure Distance Map
Wave Gliter
to sensor
out
The simulation is not standalone since exposure is
dynamic and changed by the processing pipe
Frame Processor
Inverse PSFs
Fix NUC
Exposure estimation To
sensor simulation
Target Acquisition
Double Filter
(Hadar)
Morph
operations
Target to track
association
Kalman
filtering
Main target
selection +
reporting
IR Recording and transfer
•We DON’T compress IR sensor output and pixel targets
•Store in raw format 8/10/12 bit!!!
•Sometimes we have problem in streaming it
•Lossless JPEG, Lossless AV1
•Process at the edge - > Jetsons!!!
•Stream Raw over fiber using NVIDIA Rivermax
Nvidia RiverMax
•Nvidia architecture for stream 10s-100s of GB direct to GPU
•Used to stream RAW 8/10/12bit video 4K
•Broadcasting quality
NVIDIA Video Analytics flow
Sensors Actions
Capture
and Decode
Pre-processing &
Batching
AI Inference Tracking Composition
Business Rules
and Analytics
…
…
H.265
Processing Pipe requirements
• Process stream/s of video (and audio/other data synchronously)
• Block based for each unit/function
• Easy connection between camera / socket / encoder / decoder / file / parser
• Ability to reuse and enhance existing modules (“Inheritance” + open source)
• Fix all the threads / buffers on its own
• Runs on Linux
Edge Processing Architecture
•DeepStream Nvidia Standard processing architecture – Not used
•Python CPP Processing
•Basic filtering using OpenCV/CUDA , RAPIDs
• Double Filter – DoG
• Inverse PSF
• Neural Networks
• Segmentation
• Binary Operations
•Tracking – MTT with Kalman/EKF/UKF
10 Min
questions
time
Yossi Cohen
Thank You

More Related Content

Similar to Infrared simulation and processing on Nvidia platforms

Presentation power point (Advertising Upgrade))
Presentation power point (Advertising Upgrade))Presentation power point (Advertising Upgrade))
Presentation power point (Advertising Upgrade))
andrew maybir
 
Advertising System Upgrade
Advertising System UpgradeAdvertising System Upgrade
Advertising System Upgrade
andrew maybir
 
Granvista plus gpv 280 panorama network camera briefing 2012 r1.1(2)
Granvista plus gpv 280 panorama network camera briefing 2012 r1.1(2)Granvista plus gpv 280 panorama network camera briefing 2012 r1.1(2)
Granvista plus gpv 280 panorama network camera briefing 2012 r1.1(2)
Web Technology
 

Similar to Infrared simulation and processing on Nvidia platforms (20)

Imaging on embedded GPUs
Imaging on embedded GPUsImaging on embedded GPUs
Imaging on embedded GPUs
 
Gpu with cuda architecture
Gpu with cuda architectureGpu with cuda architecture
Gpu with cuda architecture
 
OMAP
OMAPOMAP
OMAP
 
MIPI DevCon 2016: How to Use the VESA Display Stream Compression (DSC) Standa...
MIPI DevCon 2016: How to Use the VESA Display Stream Compression (DSC) Standa...MIPI DevCon 2016: How to Use the VESA Display Stream Compression (DSC) Standa...
MIPI DevCon 2016: How to Use the VESA Display Stream Compression (DSC) Standa...
 
Radiumboards- Complete Solution for Digital Security and Surveillance (DSS) I...
Radiumboards- Complete Solution for Digital Security and Surveillance (DSS) I...Radiumboards- Complete Solution for Digital Security and Surveillance (DSS) I...
Radiumboards- Complete Solution for Digital Security and Surveillance (DSS) I...
 
“Tensilica Processor Cores Enable Sensor Fusion for Robust Perception,” a Pre...
“Tensilica Processor Cores Enable Sensor Fusion for Robust Perception,” a Pre...“Tensilica Processor Cores Enable Sensor Fusion for Robust Perception,” a Pre...
“Tensilica Processor Cores Enable Sensor Fusion for Robust Perception,” a Pre...
 
A2 e overview
A2 e overviewA2 e overview
A2 e overview
 
Klony Lieberman (Sixdof Space): Ultra Fast 6DOF Tracking
Klony Lieberman (Sixdof Space): Ultra Fast 6DOF TrackingKlony Lieberman (Sixdof Space): Ultra Fast 6DOF Tracking
Klony Lieberman (Sixdof Space): Ultra Fast 6DOF Tracking
 
Presentation power point (Advertising Upgrade))
Presentation power point (Advertising Upgrade))Presentation power point (Advertising Upgrade))
Presentation power point (Advertising Upgrade))
 
Advertising System Upgrade
Advertising System UpgradeAdvertising System Upgrade
Advertising System Upgrade
 
Fixed-point Multi-Core DSP Application Examples
Fixed-point Multi-Core DSP Application ExamplesFixed-point Multi-Core DSP Application Examples
Fixed-point Multi-Core DSP Application Examples
 
7 reasons why video conferencing world will never
7 reasons why video conferencing world will never7 reasons why video conferencing world will never
7 reasons why video conferencing world will never
 
HD CCTV -Arecont Exacq Pivot3.ppt
HD CCTV -Arecont Exacq Pivot3.pptHD CCTV -Arecont Exacq Pivot3.ppt
HD CCTV -Arecont Exacq Pivot3.ppt
 
Hai Tao at AI Frontiers: Deep Learning For Embedded Vision System
Hai Tao at AI Frontiers: Deep Learning For Embedded Vision SystemHai Tao at AI Frontiers: Deep Learning For Embedded Vision System
Hai Tao at AI Frontiers: Deep Learning For Embedded Vision System
 
MIPI DevCon 2016: MIPI CSI-2 Application for Vision and Sensor Fusion Systems
MIPI DevCon 2016: MIPI CSI-2 Application for Vision and Sensor Fusion SystemsMIPI DevCon 2016: MIPI CSI-2 Application for Vision and Sensor Fusion Systems
MIPI DevCon 2016: MIPI CSI-2 Application for Vision and Sensor Fusion Systems
 
"Making Computer Vision Software Run Fast on Your Embedded Platform," a Prese...
"Making Computer Vision Software Run Fast on Your Embedded Platform," a Prese..."Making Computer Vision Software Run Fast on Your Embedded Platform," a Prese...
"Making Computer Vision Software Run Fast on Your Embedded Platform," a Prese...
 
TARDEC Presentation 2
TARDEC Presentation 2TARDEC Presentation 2
TARDEC Presentation 2
 
Ximea - the pc camera, 90 gflps smart camera
Ximea  - the pc camera, 90 gflps smart cameraXimea  - the pc camera, 90 gflps smart camera
Ximea - the pc camera, 90 gflps smart camera
 
Extending Hadoop for Fun & Profit
Extending Hadoop for Fun & ProfitExtending Hadoop for Fun & Profit
Extending Hadoop for Fun & Profit
 
Granvista plus gpv 280 panorama network camera briefing 2012 r1.1(2)
Granvista plus gpv 280 panorama network camera briefing 2012 r1.1(2)Granvista plus gpv 280 panorama network camera briefing 2012 r1.1(2)
Granvista plus gpv 280 panorama network camera briefing 2012 r1.1(2)
 

More from Yoss Cohen

Product wise computer vision development
Product wise computer vision developmentProduct wise computer vision development
Product wise computer vision development
Yoss Cohen
 

More from Yoss Cohen (20)

open platform for swarm training
open platform for swarm training open platform for swarm training
open platform for swarm training
 
Deep Learning - system view
Deep Learning - system viewDeep Learning - system view
Deep Learning - system view
 
Dspip deep learning syllabus
Dspip deep learning syllabusDspip deep learning syllabus
Dspip deep learning syllabus
 
IoT evolution
IoT evolutionIoT evolution
IoT evolution
 
Nvidia jetson nano bringup
Nvidia jetson nano bringupNvidia jetson nano bringup
Nvidia jetson nano bringup
 
Autonomous car teleportation architecture
Autonomous car teleportation architectureAutonomous car teleportation architecture
Autonomous car teleportation architecture
 
Motion estimation overview
Motion estimation overviewMotion estimation overview
Motion estimation overview
 
Computer Vision - Image Filters
Computer Vision - Image FiltersComputer Vision - Image Filters
Computer Vision - Image Filters
 
Intro to machine learning with scikit learn
Intro to machine learning with scikit learnIntro to machine learning with scikit learn
Intro to machine learning with scikit learn
 
DASH and HTTP2.0
DASH and HTTP2.0DASH and HTTP2.0
DASH and HTTP2.0
 
HEVC Definitions and high-level syntax
HEVC Definitions and high-level syntaxHEVC Definitions and high-level syntax
HEVC Definitions and high-level syntax
 
Introduction to HEVC
Introduction to HEVCIntroduction to HEVC
Introduction to HEVC
 
FFMPEG on android
FFMPEG on androidFFMPEG on android
FFMPEG on android
 
Hands-on Video Course - "RAW Video"
Hands-on Video Course - "RAW Video" Hands-on Video Course - "RAW Video"
Hands-on Video Course - "RAW Video"
 
Video quality testing
Video quality testingVideo quality testing
Video quality testing
 
HEVC / H265 Hands-On course
HEVC / H265 Hands-On courseHEVC / H265 Hands-On course
HEVC / H265 Hands-On course
 
Web video standards
Web video standardsWeb video standards
Web video standards
 
Product wise computer vision development
Product wise computer vision developmentProduct wise computer vision development
Product wise computer vision development
 
3D Video Programming for Android
3D Video Programming for Android3D Video Programming for Android
3D Video Programming for Android
 
Analog Video
Analog Video Analog Video
Analog Video
 

Recently uploaded

Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptxHarnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
FIDO Alliance
 
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
panagenda
 

Recently uploaded (20)

Event-Driven Architecture Masterclass: Challenges in Stream Processing
Event-Driven Architecture Masterclass: Challenges in Stream ProcessingEvent-Driven Architecture Masterclass: Challenges in Stream Processing
Event-Driven Architecture Masterclass: Challenges in Stream Processing
 
Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...
Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...
Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...
 
Vector Search @ sw2con for slideshare.pptx
Vector Search @ sw2con for slideshare.pptxVector Search @ sw2con for slideshare.pptx
Vector Search @ sw2con for slideshare.pptx
 
JavaScript Usage Statistics 2024 - The Ultimate Guide
JavaScript Usage Statistics 2024 - The Ultimate GuideJavaScript Usage Statistics 2024 - The Ultimate Guide
JavaScript Usage Statistics 2024 - The Ultimate Guide
 
2024 May Patch Tuesday
2024 May Patch Tuesday2024 May Patch Tuesday
2024 May Patch Tuesday
 
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptxHarnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
 
Introduction to FIDO Authentication and Passkeys.pptx
Introduction to FIDO Authentication and Passkeys.pptxIntroduction to FIDO Authentication and Passkeys.pptx
Introduction to FIDO Authentication and Passkeys.pptx
 
UiPath manufacturing technology benefits and AI overview
UiPath manufacturing technology benefits and AI overviewUiPath manufacturing technology benefits and AI overview
UiPath manufacturing technology benefits and AI overview
 
Continuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
Continuing Bonds Through AI: A Hermeneutic Reflection on ThanabotsContinuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
Continuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
 
Intro to Passkeys and the State of Passwordless.pptx
Intro to Passkeys and the State of Passwordless.pptxIntro to Passkeys and the State of Passwordless.pptx
Intro to Passkeys and the State of Passwordless.pptx
 
Where to Learn More About FDO _ Richard at FIDO Alliance.pdf
Where to Learn More About FDO _ Richard at FIDO Alliance.pdfWhere to Learn More About FDO _ Richard at FIDO Alliance.pdf
Where to Learn More About FDO _ Richard at FIDO Alliance.pdf
 
WebRTC and SIP not just audio and video @ OpenSIPS 2024
WebRTC and SIP not just audio and video @ OpenSIPS 2024WebRTC and SIP not just audio and video @ OpenSIPS 2024
WebRTC and SIP not just audio and video @ OpenSIPS 2024
 
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
 
AI mind or machine power point presentation
AI mind or machine power point presentationAI mind or machine power point presentation
AI mind or machine power point presentation
 
State of the Smart Building Startup Landscape 2024!
State of the Smart Building Startup Landscape 2024!State of the Smart Building Startup Landscape 2024!
State of the Smart Building Startup Landscape 2024!
 
ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...
ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...
ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...
 
Linux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdf
Linux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdfLinux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdf
Linux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdf
 
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...
 
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
 
Overview of Hyperledger Foundation
Overview of Hyperledger FoundationOverview of Hyperledger Foundation
Overview of Hyperledger Foundation
 

Infrared simulation and processing on Nvidia platforms

  • 1. Infrared simulation and processing On Nvidia Platforms
  • 2. Neuronics • Developing projects in • Computer vision & AI • 3D generation and simulation • Nvidia Edge and server platforms and SDKs • Defense / Military • UAV / Autonomous vehicles • Autonomous vassals • robots • Medical COMPUTER VISION & AI NVIDIA PLATFORMS 3D GENERATION & SIMULATION
  • 3. Agenda •Solve IR camera processing challenges
  • 5. Camera connection •Camera Link -> Capture card •GMSL – Fiber ( for distanced camera) •USB •GIGE •HDMI •RAW Transfer
  • 6. Noise •DSNU - Dark Signal Non-Uniformity ~1% bias •PRNU - Photo-Response Non-Uniformity ~1% gain variability •Thermal noise •Sensor •Gain •Temp
  • 7. Data problems •Capture & Label Data – expensive / not possible •Augmentation - Sensitive •Just changing brightness like in RGB images wont work! •Simulation •3D World simulation (Real, Blender, Omniverse, Unity) •Thermal assignment (simple) • Glitter simulation / Ray tracing •Atmospheric simulation (Frequency, distance, angle)
  • 8. Simulation – Basic 3D World simulation Temperature Map Sensor Simulation Distance Map Glitter Map Shared config
  • 9. Sensor Simulation – Advance Temperature to electrons Lens Shading PSF ) Point spread function( Motion Blur PRNU / DSNU NUC Simulation Shot Noise Electrons to sensor out Temperature map Distance Map Glitter Map Per frame exposure Distance Map Wave Gliter to sensor out The simulation is not standalone since exposure is dynamic and changed by the processing pipe
  • 10. Frame Processor Inverse PSFs Fix NUC Exposure estimation To sensor simulation
  • 11. Target Acquisition Double Filter (Hadar) Morph operations Target to track association Kalman filtering Main target selection + reporting
  • 12. IR Recording and transfer •We DON’T compress IR sensor output and pixel targets •Store in raw format 8/10/12 bit!!! •Sometimes we have problem in streaming it •Lossless JPEG, Lossless AV1 •Process at the edge - > Jetsons!!! •Stream Raw over fiber using NVIDIA Rivermax
  • 13. Nvidia RiverMax •Nvidia architecture for stream 10s-100s of GB direct to GPU •Used to stream RAW 8/10/12bit video 4K •Broadcasting quality
  • 14. NVIDIA Video Analytics flow Sensors Actions Capture and Decode Pre-processing & Batching AI Inference Tracking Composition Business Rules and Analytics … … H.265
  • 15. Processing Pipe requirements • Process stream/s of video (and audio/other data synchronously) • Block based for each unit/function • Easy connection between camera / socket / encoder / decoder / file / parser • Ability to reuse and enhance existing modules (“Inheritance” + open source) • Fix all the threads / buffers on its own • Runs on Linux
  • 16. Edge Processing Architecture •DeepStream Nvidia Standard processing architecture – Not used •Python CPP Processing •Basic filtering using OpenCV/CUDA , RAPIDs • Double Filter – DoG • Inverse PSF • Neural Networks • Segmentation • Binary Operations •Tracking – MTT with Kalman/EKF/UKF