可重構智能表面(RIS)本質上缺少數位轉類比與功率放大器功，因此裝置只有受限的訊號處理與有限輻射涵蓋範圍。RIS反射器的設計通常需要依賴通道狀態訊息(CSI)。在傳統的遠場傳輸中，RIS設計可以通過僅考慮輻射方向來簡化。然而，在RIS輔助的毫米波傳輸中會產生近場效應，因此設計過程需要在近場模型中同時考慮訊號的傳播方向和發射器與接收器之間的距離。因此完整的CSI對於配置近場傳播的反射器至關重要，但獲取完整的CSI是不切實際的，尤其是在毫米波RIS輔助大規模MIMO系統中。在本文中，我們提出一種基於深度確定性策略梯度(DDPG)的強化學習(RL)方法，用來聯合優化RIS輔助多輸入單輸出(MISO)系統的波束成形和RIS反射器配置。所提出的方法不需要事先的CSI，而是利用與導頻訊號相對應的接收訊號功率來確定反射器配置。模擬結果表明，我們的方法優於基於遠場近似的現有方法，強調了近場效應在RIS設計中的影響。此外，我們的設計消除了對CSI估計的需要，增強了其在實際系統應用的適用性。

Reconfigurable Intelligent Surface (RIS) devices inherently lack A/D and D/A converters and power amplifiers, resulting in limited signal processing capability and effective radiation range. The design of RIS reflectors typically requires channel state information (CSI). Conventional RIS designs for far-field transmission can simplify the process by considering only the direction of radiation. However, in RIS-assisted mm-Wave transmissions, the near-field effect is generated, necessitating consideration of both the direction of propagation and the distance between the transmitter and receiver in the near-field model. Consequently, complete CSI is essential for accurately configuring reflectors for near-field propagation, but acquiring complete CSI is impractical, especially in mmWave RIS-assisted massive MIMO systems. In this thesis, we propose jointly optimizing beamforming and RIS reflectors for the RIS-assisted multiple-input and single-output (MISO) system using Deep Deterministic Policy Gradient (DDPG) reinforcement learning (RL). The proposed approach does not require prior CSI and directly determines the reflectors by leveraging received signal power corresponding to pilot signals. Simulation results demonstrate that our method outperforms existing approaches based on far-field approximations, emphasizing the influence of the near-field effect in RIS design. Moreover, our design eliminates the need for CSI estimation, enhancing its suitability for practical system applications.

論文審定書…………………………………………………………..………………..i
誌謝 ………………………………………………………..……..………………...ii
中文摘要 ………………….……………………………………..………………...iii
英文摘要 …………………………………………………..…………...……….....iv
目錄 ………………………………………………………..…………...………......v
圖次 ………………………………………………………….……………...……..vi
表次 ……………………………………..…………………………………..….vii
第1章 導論 …………………………………………………………………..1
第2章 系統模型………………………………………………………………....5
第2.1 節 可重製智能表面輔助MISO系統…………………………………5
第2.2 節 近場與遠場通道模型………………………………………………6
第2.3 節 遠場近似模型………………………………………………………9
第2.4節 優化問題描述………………………………………………………10
第3章 基站波束成形與RIS反射器設計………………………………………....12
第3.1 節 強化學習 ………………………………………………………12
第3.2節 Q-learning……………………………………………………14
第3.3節 深度Q學習……………………………………………………15
第3.4節 策略梯度…………………………………………………………17
第3.5節 演員評論家………………………………………………………18
第3.6節 深度確定性策略梯度 ………………………………………….19
第4章 系統模擬……………………………………………………………25
第5章 結論…………………………………………………………………32
參考文獻…………………………………………………………………….33

[1] L. U. Khan, I. Yaqoob, M. Imran, Z. Han, and C. S. Hong, “6G wireless systems: A vision, architectural elements, and future directions,” IEEE Access, vol. 8, pp. 147029–147044, 2020.
[2] C. Pan, et al., “Reconfigurable intelligent surfaces for 6G systems: Principles, applications, and research directions,” IEEE Commun. Mag., vol. 59, no. 6, pp. 14-20, June 2021.
[3] Y. Liu, Z. Wang, J. Xu, C. Ouyang, X. Mu, and R. Schober, “Near-field communications: A tutorial review,” IEEE Open Journal of the Communications Society, vol. 4, pp. 1999-2049, 2023.
[4] N. J. Myers and R. W. Heath, “Infocus: A spatial coding technique to mitigate misfocus in near-field LoS beamforming,” IEEE Trans. Commun., vol. 21, no. 4, pp. 2193-2209, April 2022.
[5] S. Lv, Y. Liu, X. Xu, A. Nallanathan, and A. Lee Swindlehurst, “RIS-aided near-field MIMO communications: Codebook and beam training design,” IEEE Trans. Wireless Commun., vol. 23, no. 4, pp. 2301-2316, April 2024.
[6] A. Nordio, L. Dossi, A. Tarable, and G. Virone, “Near-field IRS configuration techniques for wideband signals and THz communications,” in Proc. ICC Workshops, 2023, pp. 1198-1203.
[7] H. Li, Y. Liu, X. Mu, T. Chen, Z. Pan, and Y. C. Eldar, “Near-field beamforming for STAR-RIS networks,” arXiv:2306.14587v1, Jun. 2023.
[8] W. Hao, X. You, F. Zhou, Z. Chu, G. Sun, and P. Xiao, “The far-/near field beam squint and solutions for THz intelligent reflecting surface communications,” IEEE Trans. Veh. Technol., vol. 72, no. 8, pp. 10107- 10118, Aug. 2023.
[9] Y. Cheng, C. Huang, W. Peng, M. Debbah, L. Hanzo, and C. Yuen, “Achievable rate optimization of the RIS-aided near-field wideband uplink,” IEEE Trans. Wireless Commun., vol. 22, no. 8, pp. 5150-5164, Aug. 2023.
[10] D. Shen, L. Dai, X. Su, and S. Suo, “Multi-beam design for near-field extremely large-scale RIS-aided wireless communications,” IEEE Trans. Green Commun. Netw., vol. 7, no. 3, pp. 1542-1553, Sept. 2023.
[11] K. Singh, H. Albinsaid, S. K. Singh, C. Pan, and S. Biswas, “DRL-Based Beamforming Design in RIS-Aided Multi-user Wireless Networks,” Proc. 2023 IEEE Int. Conf. Adv. Netw. Telecommun. Syst. (ANTS), Jaipur, India, 2023, pp. 224-229.
[12] A. Faisal, I. Al-Nahhal, O. A. Dobre and T. M. N. Ngatched, “Deep Reinforcement Learning for RIS-Assisted FD Systems: Single or Distributed RIS?,” IEEE Commun. Lett., vol. 26, no. 7, pp. 1563-1567, July 2022.
[13] Q. Liu, Y. Zhu, M. Li, R. Liu, Y. Liu and Z. Lu, “DRL-Based Secrecy Rate Optimization for RIS-Assisted Secure ISAC Systems,” IEEE Trans. Veh. Technol., vol. 72, no. 12, pp. 16871-16875, Dec. 2023.
[14] Z. Xi and J. Ji, “Sum Rate Maximization for Active RIS MISO Systems Based on DRL,” IEEE Access, vol. 13, pp. 4315-4325, 2025.
[15] H. Mei, K. Yang, Q. Liu and K. Wang, “3D-Trajectory and Phase-Shift Design for RIS-Assisted UAV Systems Using Deep Reinforcement Learning,” IEEE Trans. Veh. Technol., vol. 71, no. 3, pp. 3020-3029, March 2022.
[16] R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, MIT Press 2nd ed., 2018.
[17] K. Feng, Q. Wang, X. Li, and C.-K, Wen, “Deep reinforcement learning based intelligent reflecting surface optimization for MISO communication systems,” IEEE Commun. Lett., vol. 9, no. 5, pp. 745-749, May 2020.
[18] Robert G. Gallager, Principles of Digital Communication, Cambridge, U.K.: Cambridge University Press, 2008.
[19] X. Zhang, H. Zhang, and Y. C. Eldar, “Near-field sparse channel representation and estimation in 6G wireless communications,” arXiv:2212.13527v1, Dec. 2022.
[20] P. Chen, X. Li, M. Matthaiou, and S. Jin, “DRL-Based RIS Phase Shift Design for OFDM Communication Systems,” IEEE Wireless Commun. Lett., vol. 12, no. 4, pp. 733-737, Apr. 2023.