This paper presents OpenLMD, a novel open-source solution for on-line multimodal monitoring of Laser Metal Deposition (LMD). The solution is also applicable to a wider range of laser-based applications that require on-line control (e.g. laser welding). OpenLMD is a middleware that enables the orchestration and virtualization of a LMD robot cell, using several open-source frameworks (e.g. ROS, OpenCV, PCL). The solution also allows reconfiguration by easy integration of multiple sensors and processing equipment. As a result, OpenLMD delivers significant advantages over existing monitoring and control approaches, such as improved scalability, and multimodal monitoring and data sharing capabilities.

1. OpenLMD, Multimodal Monitoring
and Control of LMD processing
Jorge Rodríguez-Araújo
AIMEN Technology Center, Porriño, Spain
Photonics WEST 2017, 1-2-2017

2. openlmd.github.io | jorge.rodriguez@aimen.es 2
Index
Index
1. Laser Metal Deposition
2. Motivation and innovative character
3. Open Laser Metal Deposition
4. ROS-based robot cell integration
5. Multimodal monitoring and virtualization
6. Image registration and data acquisition
7. Closed-loop laser power control
8. 3D geometrical monitoring
9. Adaptive LMD path planning
10.Conclusions and future work

3. openlmd.github.io | jorge.rodriguez@aimen.es 3
Laser Metal Deposition (LMD)
 Promising additive manufacturing technique
 Parts are built up layer by layer directly from a 3D CAD model
 The material is directly deposited on the previous surface
 For repair and direct fabrication of pieces
 Near-net-shape (close to the final shape)
 Manufacturing of large metallic parts
LMD issues
 Complex setup and adjustment of parameters
 Time consuming robot programming for repairing tasks
 Thermal heating accumulation and dimensional distortion
 Repeatability, rising defects, and metalurllical properties
Traditional off-line process (with constant parameters) becomes unsucessful for large metallic parts
LMD, Laser Metal Deposition

4. openlmd.github.io | jorge.rodriguez@aimen.es 4
Motivation and innovative character
Motion
Controller
Off-line
Path Programming
6-Axis
Robot
Laser
Powder
Feeder
Power
Controller
Flow
Controller
Motivation
 Lots of industrial robotized laser cells
 Mostly manually operated
 Lack of integration for monitoring and control
Innovation
 Retrofit current laser cladding facilities for LMD
 Empower robotized laser cells for effective AM
 Apply state of the art robotic software solutions
Goals
 Build a modular architecture for LMD full automation
 Reduce heating accumulation for large parts
 Adapt robot programming to the real part
 Increase geometry accuracy and repeatability
Conventional Robotized
Laser Cladding Cell

5. openlmd.github.io | jorge.rodriguez@aimen.es 5
Open Laser Metal Depositon
Concept and approach
 Open-source solution for on-line multimodal monitoring and control of LMD
 Modular set of software components. Built on ROS (Robot Operating System)
 Robotics, machine vision, embedded control, machine learning, big data
 Full compatible with current robotized laser cladding cells
 Focus on interoperability and standardization
ROS-based architecture
 Multiprocessing architecture based on message publishing
 Multi-node and multi-machine
 Modular (e.g. robot, laser, camera)
 Synchronized data acquisition (common timestamp)
 High bandwidth data management (i.e. images)
 Visualization tools and components (e.g. rviz)
 Advanced robotics environment

6. openlmd.github.io | jorge.rodriguez@aimen.es 6
ROS-based robot cell integration
Powder
Feeder
Fiber Laser
6-Axis Industrial Robot
PC
Controller
Cladding
Head
ROS-Driver
(ABB Rapid)
Geometrical
Cell Description
(URDF)
STATE
PUBLISHER
Laser Source
(slave)
Powder Feeder
(slave)
COMMAND
SERVER
Power
Speed
Powder flow
Motion path
States
Commands
ROBOT
Process
parameters
AIMEN’s LMD robotized laser cell
ROS-based integration of modular laser cells
 The PC commands the robot integrating interfaces and modules with ROS
 The robot controls all the cell elements
ROS components for robot integration
 Geometrical description (URDF)
 ROS driver

7. openlmd.github.io | jorge.rodriguez@aimen.es 7
Multimodal monitoring and virtualization
Multimodal Cladding Head
3D System
Tachyon
MWIR
NIR
MWIR+NIR
Multispectral
Imaging
LMD Cell Virtualization
Multimodal monitoring approach
 Coaxial SWIR/MWIR images (thermal monitoring): NIT microcore (1000fps) [1-3um]
 Coaxial NIR images (surface monitoring): CMOS camera (100fps) [830-880nm]
 Off-axis 3D system: on-line 3D point cloud scanning (50fps)

8. openlmd.github.io | jorge.rodriguez@aimen.es 8
Coaxial sensors registration
 Image registration: process of transforming different sets of data into one coordinate system
Data acquisition
 High throughput (28MB/s)
 NIR + MWIR + 3D point cloud + robot
 Data management and analysis
 Bag files and Pandas DataFrames
NIT NIR
(0, 0)
vel
x
y
Image registration and data acquisition
Calibration → Projection

9. openlmd.github.io | jorge.rodriguez@aimen.es 9
Closed-loop laser power control
 High speed SWIR/MWIR thermal meltpool monitoring (1000fps)
 NIT Europe Tachyon 1024 microcore camera
 Meltpool geometrical monitoring (elliptical approximation)
 Increased geometry repeatability and reduced dilution and heat accumulation
Closed-loop laser power control
Meltpool, ellipse
approach
Width
(mm)
Time
(s)

10. openlmd.github.io | jorge.rodriguez@aimen.es 10
3D
triangulation
working
table
nozzle
3D geometrical monitoring
Industrial Robotic Laser Cell
ROBOT
ROS-DRIVER
CAMERA
IDS-DRIVER
State Publisher Peak Finder
Robot Pose
Tool-Camera
Laser
TriangulationCalibration
3D ProfileCamera Pose
3D Point Cloud
Working Cell Coordinate
On-line 3D geometrical reconstruction
On-line 3D point cloud registration
 Real-time point cloud registration
 Actual metric measurement (mm)
 Direct acquisition in robot coordinates

11. openlmd.github.io | jorge.rodriguez@aimen.es 11
Adaptive LMD path planning
Adapts the path to the real geometry
 3D vision guided full automated laser cladding repair of complex metallic parts
 Automatic generation of robot trajectories
1. Part scanning and filtering
2. Surface selection and path generation
3. Repair job generation and supervision 3D Filtering
Initialization
(setup)
Scan layer
Depth mapTarget
Depth map
Disparity
Data
Layer path
planning
Layer Path Planning
(geometrical control)
Laser Cell
supervisor
Robotized Cell
0 Finished
Repair
Job
3D geometrical control
Coated surface

12. openlmd.github.io | jorge.rodriguez@aimen.es 12
Automatic coating of surfaces
3D scanning and robot path planning
Enabled by the 3D point cloud directly provided by the
3D geometrical monitoring solution
1. Workarea scanning (direct part information in
1. cell coordinates) [mm]
2. 3D point cloud projection (2D Zmap image)
3. Surface selection directly in the 2D image
4. Segmentation of the Zmap image
5. Contours calculation from the segmented surface
6. Contours and Zmap feed the path planner
7. A new path is automatically calculated from that
1. information
A second scanning after the process enables an
adaptive path planning strategy in a full automatic way
Surface selection
3D scanned surfaces

13. openlmd.github.io | jorge.rodriguez@aimen.es 13
Conclusions
 Integration and data acquisition
 Spatial reference system and temporally synchronized
 Monitoring and control
 Real-time closed-loop laser power control
 Adaptive path planning
Work in-progress
 Embedded image monitoring and control
 Big data and deep learning approaches
Robot
 Pose
 Process speed
3D geometry
 Point cloud
(<0.5mm)
SWIR/MWIR-NIR
 2D melt pool
geometry
 Thermal
distribution and
texture
Conclusions and future work
Reconfigurable
Modular and
reconfigurable
Interoperability
Large parts
Low-cost
solution
Scalability

14. AIMEN – Central y Laboratorios
c/ Relva 27 A
36410 – O PORRIÑO (Pontevedra)
Telf.+34 986 344 000 – Fax. +34 986 337 302
Thank you for your
attention
Jorge Rodríguez-Araújo | Research Engineer
Ph +34 986 344 000 | jorge.rodriguez@aimen.es
www.aimen.es | aimen@aimen.es
This work has receive funding from the European Union’s Horizon 2020
research and innovation programme under grant agreement Nº 637081.
The dissemination of results herein reflects only the author’s view and the
Commission is not responsible for any use that may be made of the
information it contains.
http://openlmd.github.io