Camera Calibrationand Software Image EnhancementGuy Martin BEng MAScMarie Josée Perreault BAASerge Gagné BCom
Digital Imaging and Intelligent SystemsDigital imaging is an enabling technology, the key building block for a broad range of high end, highly automated technologies.The use of digital cameras as measuring tools in Intelligent Systems (IS) requires the camera to be calibrated.We corrected systematic errors found in every camera calibration model and technique known.Using our technique as a software lens correction filter,we demonstrated to National Defence Canada* 8:1 higher measurement accuracy (image or 3D telemetry) 4:1 faster computation time 30% added lossless video compression … * (June 2nd 2009)
Camera CalibrationMarket WiseBesides Aerospace and Military applications, the technologycan be found applied in a broad range of Intelligent Systems Major Markets Market Annual billions $US Growth Robotics 7.37 7-15% Service Robots 0.25 40-70% Artificial Vision 13.44 12.25% Tele-medicine 95.0 10% Robot Surgeon 1.0 100% Transportation 40.04 17% Security 14.00 20% Biometrics 1.20 20% Facial Recognition 0.14 100%Digital Imaging is forecasted to become in the present century thenext big technological trend setter, as big as the advent of thesilicon chip in the 20th century.
Camera CalibrationMarket Wise Our innovation reaches 60% of the optics and photonics market - 75 billions $ annual sales worldwide - 2/3 shared between 5,000 US companiesMilitary and Space Opportunities - Satellite imaging: digital compression, infrared fusion, sub pixel image analysis… - Enhanced night vision: infrared and color image fusion - Surveillance and object tracking: sub pixel contouring - Aerial cartography using UAV (Unmanned Airborne Vehicle): 3D from motion and 3D mapping using a camera video feed - Video feedback and robotics guidance: docking to ISS and Canadarm- Dextre visual feedback, rover exploration… …
Camera Calibration Is knowing how an image projects through the lens to print on the CCD camera surface We measure the behaviour related to - Camera posture with respect to the scene - Image deformation introduced by the lens - Image sensor and lens mechanical assembly tolerances Once calibrated, through software we can remove image artefacts: Errors introduced in the image by the imperfect nature of lenses. We can then use the corrected image as a feedback sensor for automated tasks in Intelligent Systems.
Camera Calibration We correct the biggest error found in digital imaging The distortion introduced by the lensLens distortion is known to cause •errors in object measurement and tracking using cameras •blurring of object edges •imaging software sensitivity to lighting conditions •lower digital image and video compression ratiosUsing off the shelf cameras and lenses, we demonstrated to NationalDefense Canada (DND) •an accuracy increase of 8:1 in 3D measurements using digital images •a 4:1 speed increase in image geometric distortion correction •a model extension for using zooming lenses in telemetry, a world first …We corrected a systematical error found in every camera model andcalibration technique published.We can retrieve the shape of a camera pixel with an accuracy better than10-10 mm (roughly the size of an hydrogen molecule), removing a bias inthe image center as big as 2 pixels.Every lens parameter is ‘polluted’ by the image center bias.Our results surpass MIT, NASA, JPL, Stanford, INRIA, INO, Creaform, …
Camera Calibration Lens distortion introduces the biggest error found in digital imaging Geometric distortion curves straight lines and shears the squareness of rectangles with increasing magnitude, going from the image center towards the outside border. Chromatic distortion splits white light as in a prism blurring the rectangle edges in blue on the short side, and in red on the far side with respect to the image center. Chromatic distortion introduces a systematic edge measurement error with varying object color or light spectrum changes. Both distortions are seen by image analysis algorithms as false image point motion while it actually is an imperfection created by the optics. We like to use short focal length lenses to gain image and surface definition, to see more details. Up until our work, we were faced with a trade off between image definition and measurement accuracy because of lens distortion. We are now able to compensate through software.
Camera Calibration How ? By taking the picture of a 3D known structure, and modeling how the image prints on the camera electronic surface.
Camera Calibration We corrected everyone’s error in locating the image center The Image center is the center of symmetry needed to model and correct lens distortion, both geometric and chromatic… Finding the true image center position corrects a systematic bias in all other lens parameters • the lens focal distance • the lens skew compensation • the lens geometric distortion • the lens chromatic distortion • the camera position w.r. to the outside world and all 3D measurements taken from camera images in telemetry…
Camera Calibration3D from a stereo camera pair All advanced imagery techniques require a pinhole ‘errorless’ camera image, where geometric and chromatic distortion are removed, and without bias in the image center. It enables sub pixel edge extraction and increases 3D triangulation accuracy.
Camera Calibration3D from a video sequence 3D from motion techniques behaves essentially like a camera stereo pair. Two pictures of a still scenery taken from two different vantage points will yield a 3D model. Still, lens geometric and chromatic distortion have to be removed from the images. This is the technique used by imaging satellites, UAV planes, rovers, panoramic cameras, … to achieve a 3D map of a given area. In satellite imaging, the earth’s atmosphere behaves like a lens and adds distortion to the image. It can be compensated for only if the camera distortion is accurately known.
Camera CalibrationEarth Observation and RovingA rover relies on its camera to map its Sub Pixel edge detection is biasedsurroundings. In aerial photography, we by lens distortion.can retrieve edge locations at a fraction of Geometric and chromatica pixel. In order to make the distortion correction require anmeasurement significant, lens chromatic accurate knowledge of the imageand geometric distortion have to be center.removed from the image. We removed a 2 pixel image center bias in image calibration using COTS cameras and lenses
Camera CalibrationImage fusion and Night Vision Seeing through fog, smoke, snow, camouflage The US Army is spending billions to gain a tactical advantage in night operations. They are seeking the next generation of wide angle imaging to replace the current NVG light amplification devices. Our proposed modification of SWIR (Short Wave Infra Red) cameras to DND Canada provides that kind of performance, and will cut down their supplying cost by 125 to 150$ millions with a smaller/lighter and more accurate night vision camera. Our initial involvement in this venture was to show DND Canada how camera calibration would solve the image fusion problem between SWIR and color images needed for Enhanced Night Vision Goggles (ENVG) We then proceeded to increase the camera sensitivity The same technology goes for ground or airborne surveillance cameras, soldier helmet, and satellite imaging.
Camera CalibrationLanding an Airplane Civil Aviation produces more CO2 emissions than the entire African continent. At 300,000 $ a piece, with a 2 year payback in fuel economy alone, a cryogenic infrared setup helps to locate the runway in low visibility conditions. The next generation of supersonic airplanes will rely on cameras so no cockpit or hull windows are needed, making the plane lighter. Our proposal to DND is an augmented reality helmet using the human eye as a pointing device to replace the mouse click, a device that doesn’t require cryogenic cooling.Synthetic vision withVision Amplificationshould appear in civilianairline transportationaround year 2018
Camera CalibrationMedical Imaging Historically, imaging technology developed for military uses has seen its way to medical imaging. The use of robotic surgery shortens the patient’s recovery period in hospital by more than half. Less invasive, it relies on cameras to provide access to the operating area. The US Army is developing a robot surgeon for NASA’s space program. Removing lens distortion makes the use of cameras more intuitive and comfortable for the surgeon, increasing his skill. Used with a robot, lens distortion removal makes measurements 8 times more accurate.
Camera CalibrationTelecommunications We demonstrated an added 30% lossless compression over existing codec. Removing lens distortion allowed us to use geometric redundancy to increase the compression ratio of images and video sequences. Video content is therefore faster to transmit. Internet saturates because of the increased use of digital video. The hardware cost for North America is estimated at 55 $billions. In satellite applications, our accuracy gain impacts two ways: • Smaller image files because of added compression • Less images because larger areas can be scanned In wireless transmission, added compression means more devices serviced using the same network. Since 75% of all Internet data content is video, a given telco operator could be more profitable than the owner of the network it is roaming on.
Camera CalibrationOther market targets include…•Pharmaceutical packaging•Robotics parcel handling•Highway traffic surveillance•Airport runway traffic management•Border surveillance control…As part of the Optics and Photonics market segment, computervision and digital imaging will impact on most of humanity’seconomic activities, part of a technological break trough assignificant as the advent of the microchip and the electron inthe 20th century (according to ISO’s market study).The use of zooming lenses cameras for telemetry was thoughtto be the only major technological barrier for the explosion ofthis specific market niche. We showed quite a few more…