Multi-modal attention method in welding defect recognition

A multi-modal welding defect identification method was proposed. A convolutional neural network with three branches was constructed to process welding pool pictures, arc sound, welding current and arc voltage, respectively. A channel attention module and a spatial attention module were added to image branch network to focus on important areas of weld pool image. In order to verify stability and reliability of the model in this paper, experiments were carried out on a self-constructed dataset containing 10 welding defects. The experimental results demonstrated that performance of introducing attention mechanism into front layers was better than that of back layers. Furthermore, attention mechanism contributed to promoting F value, which provided a reference for real-time welding classification and identification and was helpful for welding quality assessment.Highlights: (1) A multimodal convolutional neural network was proposed to automatically extract salient features of welding pool image, welding current, arc voltage and arc sound.(2) An attention mechanism was fused into image network branch to help model capture significant defect regions, and it was verified that effect of introducing attention in shallow convolution layer was better than that in deep layer.