Abstract: A simultaneously transmitting and reflecting reconfigurable intelligent surface (STAR-RIS) assisted multi-user near-field wideband communication system is investigated, in which a robust deep reinforcement learning (DRL) based algorithm is proposed to enhance the users’ achievable rate by jointly optimizing the active beamforming at the base station (BS) and passive beamforming at the STAR-RIS. To mitigate the beam split issue, the delay-phase hybrid precoding structure is introduced to facilitate wideband beamforming. Considering the coupled nature of the STAR-RIS phase-shift model, the passive beamforming design is formulated as a problem of hybrid continuous and discrete phase-shift control, and the proposed algorithm controls the high-dimensional continuous action through hybrid action mapping. Additionally, to address the issue of biased estimation encountered by existing DRL algorithms, a softmax operator is introduced into the algorithm to mitigate this bias. Simulation results illustrate that the proposed algorithm outperforms existing algorithms and overcomes the issues of overestimation and underestimation.

基于深度强化学习的智能全向超表面

[1]Abeywickrama S, Zhang R, Wu QQ, et al., 2020. Intelligent reflecting surface: practical phase shift model and beamforming optimization. IEEE Trans Commun, 68(9):5849-5863.

[2]Dai LL, Tan JB, Chen Z, et al., 2022. Delay-phase precoding for wideband THz massive MIMO. IEEE Trans Wirel Commun, 21(9):7271-7286.

[3]ElMossallamy MA, Zhang HL, Song LY, et al., 2020. Reconfigurable intelligent surfaces for wireless communications: principles, challenges, and opportunities. IEEE Trans Cogn Commun Netw, 6(3):990-1002.

[4]Fujimoto S, van Hoof H, Meger D, 2018. Addressing function approximation error in actor-critic methods. https://arxiv.org/abs/1802.09477

[5]Gao XY, Dai LL, Zhou SD, et al., 2019. Wideband beamspace channel estimation for millimeter-wave MIMO systems relying on lens antenna arrays. IEEE Trans Signal Process, 67(18):4809-4824.

[6]Guo KF, Liu R, Alazab M, et al., 2023. STAR-RIS-empowered cognitive non-terrestrial vehicle network with NOMA. IEEE Trans Intell Veh, 8(6):3735-3749.

[7]Han C, Akyildiz IF, 2016. Distance-aware bandwidth-adaptive resource allocation for wireless systems in the terahertz band. IEEE Trans Terahertz Sci Technol, 6(4):541-553.

[8]Han C, Yan LF, Yuan JH, 2021. Hybrid beamforming for terahertz wireless communications: challenges, architectures, and open problems. IEEE Wirel Commun, 28(4):198-204.

[9]He XL, Xu HB, Wang J, et al., 2024. Joint active and passive beamforming in RIS-assisted covert symbiotic radio based on deep unfolding. IEEE Trans Veh Technol, 73(9):14021-14026.

[10]Headland D, Monnai Y, Abbott D, et al., 2018. Tutorial: terahertz beamforming, from concepts to realizations. APL Photon, 3(5):051101.

[11]Hua M, Wu QQ, Chen W, et al., 2024a. Intelligent reflecting surface assisted localization: performance analysis and algorithm design. IEEE Wirel Commun Lett, 13(1):84-88.

[12]Hua M, Wu QQ, Chen W, et al., 2024b. Secure intelligent reflecting surface-aided integrated sensing and communication. IEEE Trans Wirel Commun, 23(1):575-591.

[13]Huang CW, Alexandropoulos GC, Zappone A, et al., 2019. Deep learning for UL/DL channel calibration in generic massive MIMO systems. Proc IEEE Int Conf on Communications, p.1-6.

[14]Huang CW, Mo RH, Yuen C, 2020. Reconfigurable intelligent surface assisted multiuser MISO systems exploiting deep reinforcement learning. IEEE J Select Areas Commun, 38(8):1839-1850.

[15]Jiang CX, Zhang HJ, Ren Y, et al., 2017. Machine learning paradigms for next-generation wireless networks. IEEE Wirel Commun, 24(2):98-105.

[16]Kraus JD, Marhefka RJ, 2002. Antennas for All Applications (3^rd Ed.). McGraw-Hill Science/Engineering/Math, New York, USA.

[17]Li HC, Liu YW, Mu XD, et al., 2023. Near-field beamforming for STAR-RIS networks. https://arxiv.org/abs/2306.14587

[18]Li HY, Li M, Liu Q, et al., 2020. Dynamic hybrid beamforming with low-resolution PSs for wideband mmWave MIMO-OFDM systems. IEEE J Sel Areas Commun, 38(9):2168-2181.

[19]Li XW, Xie Z, Chu Z, et al., 2022. Exploiting benefits of IRS in wireless powered NOMA networks. IEEE Trans Green Commun Netw, 6(1):175-186.

[20]Li XW, Zhang JY, Han CZ, et al., 2024. Reliability and security of CR-STAR-RIS-NOMA-assisted IoT networks. IEEE Int Things J, 11(17):27969-27980.

[21]Liu R, Guo KF, Li XW, et al., 2024. RIS-empowered satellite-aerial-terrestrial networks with PD-NOMA. IEEE Commun Surv Tutor, 26(4):2258-2289.

[22]Mismar FB, Evans BL, Alkhateeb A, 2020. Deep reinforcement learning for 5G networks: joint beamforming, power control, and interference coordination. IEEE Trans Commun, 68(3):1581-1592.

[23]Mnih V, Kavukcuoglu K, Silver D, et al., 2015. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533.

[24]Mu XD, Liu YW, Guo L, et al., 2020. Exploiting intelligent reflecting surfaces in NOMA networks: joint beamforming optimization. IEEE Trans Wirel Commun, 19(10):6884-6898.

[25]Mu XD, Liu YW, Guo L, et al., 2022. Simultaneously transmitting and reflecting (STAR) RIS aided wireless communications. IEEE Trans Wirel Commun, 21(5):3083-3098.

[26]Ni WL, Liu YW, Eldar YC, et al., 2021. STAR-RIS enabled heterogeneous networks: ubiquitous NOMA communication and pervasive federated learning. https://arxiv.org/abs/2106.08592v1

[27]Pan L, Cai Q, Huang L, 2020. Softmax deep double deterministic policy gradients. Proc 34^th Int Conf on Neural Information Processing Systems, p.11767-11777.

[28]Samir M, Elhattab M, Assi C, et al., 2021. Optimizing age of information through aerial reconfigurable intelligent surfaces: a deep reinforcement learning approach. IEEE Trans Veh Technol, 70(4):3978-3983.

[29]Shafin R, Chen H, Nam YH, et al., 2020. Self-tuning sectorization: deep reinforcement learning meets broadcast beam optimization. IEEE Trans Wirel Commun, 19(6):4038-4053.

[30]Silver D, Lever G, Heess N, et al., 2014. Deterministic policy gradient algorithms. Proc 31^st Int Conf on Machine Learning, p.I-387-I-395.

[31]Wang J, Xiao J, Zou YX, et al., 2024. Wideband beamforming for RIS assisted near-field communications. IEEE Trans Wirel Commun, 23(11):16836-16851.

[32]Wang ZL, Mu XD, Xu JQ, et al., 2023. Simultaneously transmitting and reflecting surface (STARS) for terahertz communications. IEEE J Sel Top Signal Process, 17(4):861-877.

[33]Wu CY, Liu YW, Mu XD, et al., 2021. Coverage characterization of STAR-RIS networks: NOMA and OMA. IEEE Commun Lett, 25(9):3036-3040.

[34]Xiao J, Wang J, Wang ZL, et al., 2024a. Multi-scale attention based channel estimation for RIS-aided massive MIMO systems. IEEE Trans Wirel Commun, 23(6):5969-5984.

[35]Xiao J, Wang J, Wang ZL, et al., 2024b. Multi-task learning for near/far field channel estimation in STAR-RIS networks. IEEE Trans Commun, 72(10):6344-6359.

[36]Xu C, Ishikawa N, Rajashekar R, et al., 2019. Sixty years of coherent versus non-coherent tradeoffs and the road from 5G to wireless futures. IEEE Access, 7:178246-178299.

[37]Yu XH, Shen JC, Zhang J, et al., 2016. Alternating minimization algorithms for hybrid precoding in millimeter wave MIMO systems. IEEE J Sel Top Signal Process, 10(3):485-500.

[38]Zhang E, Huang C, 2014. On achieving optimal rate of digital precoder by RF-baseband codesign for MIMO systems. Proc IEEE 80^th Vehicular Technology Conf, p.1-5.

[39]Zhou Y, Zhou FH, Wu YP, et al., 2020. Subcarrier assignment schemes based on Q-learning in wideband cognitive radio networks. IEEE Trans Veh Technol, 69(1):1168-1172.

[40]Zhu BO, Chen K, Jia N, et al., 2014. Dynamic control of electromagnetic wave propagation with the equivalent principle inspired tunable metasurface. Sci Rep, 4(1):4971.

[41]Zhu FH, Wang BH, Yang ZH, et al., 2023. Robust millimeter beamforming via self-supervised hybrid deep learning. Proc 31^st European Signal Processing Conf, p.915-919.

[42]Zhu FH, Wang XQ, Huang CW, et al., 2024. Beamforming inferring by conditional WGAN-GP for holographic antenna arrays. IEEE Wirel Commun Lett, 13(7):2023-2027.