Abstract: End-to-end object detection methods have attracted extensive interest recently since they alleviate the need for complicated human-designed components and simplify the detection pipeline. However, these methods suffer from slower training convergence and inferior detection performance compared to conventional detectors, as their feature fusion and selection processes are constrained by insufficient positive supervision. To address this issue, we introduce a novel query-selection encoder (QSE) designed for end-to-end object detectors to improve training convergence speed and detection accuracy. The QSE is composed of multiple encoder layers stacked on top of the backbone. A lightweight head network is added after each encoder layer to continuously optimize features in a cascading manner, providing more positive supervision for efficient training. Additionally, a hierarchical featureaware attention (HFA) mechanism is incorporated in each encoder layer, including in-level feature attention and cross-level feature attention, to enhance the interaction between features from different levels. HFA can effectively suppress similar feature representations and highlight discriminative ones, thereby accelerating the feature selection process. Our method is highly versatile in accommodating both CNN-based and transformer-based detectors. Extensive experiments were conducted on the popular benchmark datasets MS COCO, CrowdHuman and PASCAL VOC to demonstrate the effectiveness of our method. The results showed that CNN-based and transformer-based detectors using QSE can achieve better end-to-end performance in a shorter training setting.