Abstract: The increasing adoption of smart devices and cloud services, coupled with limitations in local computing and storage resources, prompts extensive users to transmit private data to cloud servers for processing. However, the transmission of sensitive data in plaintext form raises concerns regarding user's privacy and security. To address these issues, this study proposes an efficient privacy-preserving secure neural network inference scheme based on homomorphic encryption and secure multi-party computation, which ensures the privacy of both the user and the cloud server while enabling fast and accurate ciphertext inference. First, we divided the inference process into three stages, including the merging stage for adjusting the network structure, the preprocessing stage for performing homomorphic computations, and the online stage for floating-point operations on the additive secret sharing of private data. Second, we proposed an approach of merging network parameters, thereby reducing the cost of multiplication levels and decreasing both ciphertext-plaintext multiplication and addition operations. Finally, we proposed a fast convolution algorithm to enhance computational efficiency. Compared with other literature, our scheme reduces linear operation time in the online stage by at least 11%, significantly reducing inference time and communication overhead.