Abstract: Bridges are an important part of railway infrastructure and need regular inspection and maintenance. Using UAV technology to inspect railway infrastructure is an active research issue. However, due to the large size of UAV images, flight distance and height changes, the object scale changes dramatically. At the same time, the elements of interest in railway bridges, such as bolts and corrosion, are small and dense objects, and the sample data set is seriously unbalanced, posing great challenges to the accurate detection of defects. In this paper, an Adaptive Cropping Shallow Attention Network (ACSANet) is proposed, which includes an adaptive cropping strategy for large UAV images and a shallow attention network for small object detection in limited samples. To enhance the accuracy and generalization of the model, the shallow attention network model integrates a CA (Coordinate Attention) attention mechanism module and an α-IOU (Alpha-IOU) loss function, and then carries out defect detection on the bolts, steel surfaces and railings of railway bridges. The test results show that the ACSANet model outperforms the YOLOv5s model using adaptive cropping strategy in terms of the total mAP and missing bolts mAP by 5% and 30%, respectively. Also, compared with the YOLOv5s model that adopts the common cropping strategy, the total mAP and missing bolt mAP are improved by 10% and 60% respectively. Compared with the YOLOv5s model without any cropping strategy, the total mAP and missing bolt mAP are improved by 40% and 67%, respectively.