Journal of Zhejiang University

Journal of Zhejiang University SCIENCE A

Accepted manuscript available online (unedited version)

An optimization scheduling strategy for electric-heat-hydrogen integrated energy systems based on memory-enhanced deep reinforcement learning

Author(s): Zhongli BAI
Affiliation(s): Tianjin Key Laboratory of New Energy Power Conversion, Transmission, and Intelligent Control, Tianjin University of Technology, Tianjin 300384, China; more
Corresponding email(s):
Key Words: Integrated energy system (IES), Electric-heat-hydrogen, Long short-term memory (LSTM), Soft actor–critic (SAC), Reinforcement learning, Scheduling strategy

<<< Previous Paper \|Next Paper >>>

Zhongli BAI. An optimization scheduling strategy for electric-heat-hydrogen integrated energy systems based on memory-enhanced deep reinforcement learning[J]. Journal of Zhejiang University Science A,in press.Frontiers of Information Technology & Electronic Engineering,in press.https://doi.org/10.1631/jzus.A2500598

@article{title="An optimization scheduling strategy for electric-heat-hydrogen integrated energy systems based on memory-enhanced deep reinforcement learning",
author="Zhongli BAI",
journal="Journal of Zhejiang University Science A",
year="in press",
publisher="Zhejiang University Press & Springer",
doi="https://doi.org/10.1631/jzus.A2500598"
}

%0 Journal Article
%T An optimization scheduling strategy for electric-heat-hydrogen integrated energy systems based on memory-enhanced deep reinforcement learning
%A Zhongli BAI
%J Journal of Zhejiang University SCIENCE A
%P 809-824
%@ 1673-565X
%D in press
%I Zhejiang University Press & Springer
doi="https://doi.org/10.1631/jzus.A2500598"

TY - JOUR
T1 - An optimization scheduling strategy for electric-heat-hydrogen integrated energy systems based on memory-enhanced deep reinforcement learning
A1 - Zhongli BAI
J0 - Journal of Zhejiang University Science A
SP - 809
EP - 824
%@ 1673-565X
Y1 - in press
PB - Zhejiang University Press & Springer
ER -
doi="https://doi.org/10.1631/jzus.A2500598"

Abstract
Chinese Summary
Academic Network
Reviewer Comment

Abstract: High wind-solar penetration drives integrated energy systems (IESs) to act as cross-vector buffers that absorb surplus electricity via hybrid storage, or convert it into heat and hydrogen for cross-medium energy peak shaving. However, conventional mixed-integer linear programming (MILP) solvers and static incentive schemes often struggle with the high-dimensional, strongly coupled, non-convex, and time-varying nature of electric-heat-hydrogen scheduling. Thus, we propose an end-to-end optimization framework for a renewable electric-heat-hydrogen IES, and evaluate it through offline dispatch computations and cost settlement on a 24-h simulated case study. The proposed framework incorporates a coupled power–state of charge (SOC) dual penalty mechanism to ensure consistent storage operation over time. Additionally, it includes a bidirectional incentive-based demand response (B-IDR) mapping that is differentiable and can capture asymmetric feedback in response to price fluctuations. Furthermore, a long short-term memory (LSTM)-augmented maximum-entropy soft actor–critic (SAC) scheduler is utilized for stable and efficient control in this continuous and high-dimensional setting. Comparisons with other methods under identical settings show that the proposed method achieves lower operating costs and carbon emissions, as well as improved training stability.

基于记忆增强深度强化学习的电-热-氢集成能源系统优化调度策略

作者：白忠立^1,2，高强^1,2,3，张宏志^1,2，刘俊杰^1,2，吉月辉^1,2，宋雨^1,2，程徐^1,4
机构：¹天津理工大学，天津市新能源电力变换传输与智能控制重点实验室，中国天津，300384；²天津理工大学，电气工程与自动化学院，中国天津，300384；³天津理工大学，海运学院，中国天津，300384；⁴天津理工大学，计算机科学与工程学院，中国天津，300384
目的：本研究专注于高可再生能源渗透的能源园区，利用双向激励需求响应（B-IDR）机制实现需求侧响应，通过电-电量状态（SOC）软惩罚降低储能使用频率，延长储能设备使用寿命。
创新点：1.提出了一种电-SOC软惩罚机制，以保持储能设备的灵活性和延长使用寿命。2.提出了基于B-IDR机制，通过反馈价格波动捕捉不对称需求响应。3.结合长短期记忆网络（LSTM）的时序感知能力，提高了强化学习调度的稳定性和效率。
方法：1.提出一种基于LSTM-SAC调度器的优化框架，用于解决高渗透可再生能源下的电-热-氢综合能源系统（IES）调度问题。该方法结合多能储存与双向激励需求响应（B-IDR）机制，通过LSTM增强的柔性动作-评价（SAC）调度器捕捉风电、光伏输出及负荷变化等短期动态特征，优化能源管理策略。2.在低碳经济调度模型中，综合考虑能源采购、碳排放、设备运行损耗等多个成本因素，并通过多场景模拟验证该方法在降低系统总成本、碳排放和可再生能源削减成本方面的有效性。
结论：1.通过结合B-IDR和储能系统的调度优化，本研究提出的方法显著降低了储能系统的循环负荷，提高了系统的经济性和碳减排效果。2.随着多能耦合技术的应用，尤其是在氢气储能路径的引入下，系统的低碳调度效果得到了显著改善。本研究的成果为高渗透可再生能源的低碳优化调度提供了新思路，并为未来能源市场的优化调度策略提供了实践依据。

关键词组：综合能源系统；电-热-氢；LSTM；SAC；强化学习；调度策略

Darkslateblue:Affiliate; Royal Blue:Author; Turquoise:Article

Reference

[1]Abbasimehr H, Shabani M, Yousefi M, 2020. An optimized model using LSTM network for demand forecasting. Computers & Industrial Engineering, 143:106435.

[2]Abri S, Abri R, 2025. Deep learning methods for LSTM-based personalized search: a comparative analysis. International Journal of Machine Learning and Cybernetics, 16(4):2747-2759.

[3]Alexopoulos A, 2021. The fractional Kullback–Leibler divergence. Journal of Physics A: Mathematical and Theoretical, 54(7):075001.

[4]Ampimah BC, Sun M, Han D, et al., 2018. Optimizing sheddable and shiftable residential electricity consumption by incentivized peak and off-peak credit function approach. Applied Energy, 210:1299-1309.

[5]Chen HP, Wu H, Li HL, et al., 2025. Bi-level optimal scheduling of integrated energy systems considering incentive-based demand response and green certificate-carbon trading mechanisms. Energy Reports, 13:330-344.

[6]Duan JL, Guan Y, Li SE, et al., 2022. Distributional soft actor-critic: off-policy reinforcement learning for addressing value estimation errors. IEEE Transactions on Neural Networks and Learning Systems, 33(11):6584-6598.

[7]Gao MF, Han ZH, Zhao B, et al., 2023. Optimal planning method of multi-energy storage systems based on the power response analysis in the integrated energy system. Journal of Energy Storage, 73:109015.

[8]Gea-Bermúdez J, Bramstoft R, Koivisto M, et al., 2023. Going offshore or not: where to generate hydrogen in future integrated energy systems? Energy Policy, 174:113382.

[9]Haarnoja T, Zhou A, Hartikainen K, et al., 2018. Soft actor-critic algorithms and applications. arXiv: 1812.05905.

[10]Hamzeh Aghdam F, Mohammadi-Ivatloo B, Abapour M, et al., 2025. Enhancing the risk-oriented participation of wind power plants in day-ahead, balancing, and hydrogen markets with shared multi-energy storage systems. Journal of Energy Storage, 107:114911.

[11]Han S, Sung Y, 2021. A max-min entropy framework for reinforcement learning. Proceedings of the 35th International Conference on Neural Information Processing Systems, Article 1970.

[12]Hannan MA, Wali SB, Ker PJ, et al., 2021. Battery energy-storage system: a review of technologies, optimization objectives, constraints, approaches, and outstanding issues. Journal of Energy Storage, 42:103023.

[13]Hu H, Zhao XN, Shang GZ, et al., 2026. Low-carbon optimization scheduling of hybrid energy storage in integrated energy system considering bidirectional interaction between green certificate and carbon trading. Energies, 19(1):70.

[14]Jeon S, Bae S, 2025. Integrated optimization for sizing, placement, and energy management of hybrid energy storage systems in renewable power systems. Journal of Energy Storage, 106:114793.

[15]Jiang MH, Xu ZJ, Zhu HY, et al., 2024. Integrated demand response modeling and optimization technologies supporting energy internet. Renewable and Sustainable Energy Reviews, 203:114757.

[16]Koronen C, Åhman M, Nilsson LJ, 2020. Data centres in future European energy systems—energy efficiency, integration and policy. Energy Efficiency, 13(1):129-144.

[17]Kurucan M, Özbaltan M, Yetgin Z, et al., 2024. Applications of artificial neural network based battery management systems: a literature review. Renewable and Sustainable Energy Reviews, 192:114262.

[18]Lee JO, Kim YS, 2022. Novel battery degradation cost formulation for optimal scheduling of battery energy storage systems. International Journal of Electrical Power & Energy Systems, 137:107795.

[19]Li FD, Shi ZH, Zhu ZH, et al., 2025. Energy management strategy for direct current microgrids with consideration of photovoltaic power tracking optimization. Energies, 18(2):252.

[20]Li P, Wang ZX, Wang N, et al., 2021. Stochastic robust optimal operation of community integrated energy system based on integrated demand response. International Journal of Electrical Power & Energy Systems, 128:106735.

[21]Li Q, Xiao XK, Pu YC, et al., 2023. Hierarchical optimal scheduling method for regional integrated energy systems considering electricity-hydrogen shared energy. Applied Energy, 349:121670.

[22]Li Y, Wang B, Yang Z, et al., 2022. Optimal scheduling of integrated demand response-enabled community-integrated energy systems in uncertain environments. IEEE Transactions on Industry Applications, 58(2):2640-2651.

[23]Liang HJ, Pirouzi S, 2024. Energy management system based on economic Flexi-reliable operation for the smart distribution network including integrated energy system of hydrogen storage and renewable sources. Energy, 293:130745.

[24]Liang T, Zhang XC, Tan JX, et al., 2024. Deep reinforcement learning-based optimal scheduling of integrated energy systems for electricity, heat, and hydrogen storage. Electric Power Systems Research, 233:110480.

[25]Lin JH, Gu YJ, Wang ZJ, et al., 2024. Operational characteristics of an integrated island energy system based on multi-energy complementarity. Renewable Energy, 230:120890.

[26]Liu JQ, Yu SS, Hu HJ, et al., 2023. Demand-side regulation provision of virtual power plants consisting of interconnected microgrids through double-stage double-layer optimization. IEEE Transactions on Smart Grid, 14(3):1946-1957.

[27]Luo YL, Gao Y, Fan DL, 2023. Real-time demand response strategy base on price and incentive considering multi-energy in smart grid: a bi-level optimization method. International Journal of Electrical Power & Energy Systems, 153:109354.

[28]Mochi P, Espegren KA, Korpas M, 2025. Short communication: local electricity-hydrogen market. International Journal of Hydrogen Energy, 116:17-22.

[29]Narayanan M, 2021. Annual evaluation of a model predictive controller in an integrated thermal-electrical renewable energy system using clustering technique. Journal of Energy Resources Technology, 143(5):051302.

[30]Nikoobakht A, Mokarram MJ, Moghaddam EM, 2023. Enhancing integrated energy systems’ resilience against windstorms through a decentralized cooperation model. Electric Power Systems Research, 225:109801.

[31]Ning LF, Liu F, Wang Z, et al., 2024. Day-ahead strategic bidding of renewable energy considering output uncertainty based on deep reinforcement learning. 6th Asia Energy and Electrical Engineering Symposium.

[32]Sadeghian O, Shotorbani AM, Ghassemzadeh S, et al., 2025. Energy management of hybrid fuel cell and renewable energy based systems—a review. International Journal of Hydrogen Energy, 107:135-163.

[33]Saglam B, Mutlu FB, Cicek DC, et al., 2023. Actor prioritized experience replay. Journal of Artificial Intelligence Research, 78:639-672.

[34]Shi MS, Vasquez JC, Guerrero JM, et al., 2023. Smart communities—design of integrated energy packages considering incentive integrated demand response and optimization of coupled electricity-gas-cooling-heat and hydrogen systems. International Journal of Hydrogen Energy, 48(80):31063-31077.

[35]Wang XW, Yang CL, Zhao JH, et al., 2025. Energy management of solar spectral beam-splitting integrated energy systems using soft actor-critic method. Applied Thermal Engineering, 269:125966.

[36]Wang YL, Dong HR, Ma KW, et al., 2024. Multi frequency stability optimization of integrated energy systems considering virtual energy storage characteristics of heating networks. Applied Thermal Engineering, 257:124254.

[37]Wang YL, Han L, Deng XL, et al., 2025. Optimization scheduling of household integrated energy systems for improving thermal comfort with low cost. Energy and Buildings, 329:115229.

[38]Washizu A, Ju YY, Yoshida A, et al., 2024. Modeling the distributed energy resource aggregator services in a macroeconomic framework: the application to Japan. Energy, 312:133561.

[39]Wu LZ, Zhang WJ, Chen W, et al., 2025. A multi-time scale optimal scheduling strategy for integrated energy systems considering the power randomness of wind and photovoltaic. Electrical Engineering, 107(7):9109-9123.

[40]Xu YH, Wei YR, Jiang KY, et al., 2023. Action decoupled SAC reinforcement learning with discrete-continuous hybrid action spaces. Neurocomputing, 537:141-151.

[41]Yan N, Zhao ZJ, Li XJ, et al., 2024. Multi-time scales low-carbon economic dispatch of integrated energy system considering hydrogen and electricity complementary energy storage. Journal of Energy Storage, 104:114514.

[42]Yan Z, Zhang YM, Yu JS, 2024. Allocative approach to multiple energy storage capacity for integrated energy systems based on security region in buildings. Journal of Energy Storage, 84:110951.

[43]Zheng KF, Wang XJ, Wu B, et al., 2020. Feature subset selection combining maximal information entropy and maximal information coefficient. Applied Intelligence, 50(2):487-501.

[44]Zuo LJ, Xi YF, Zhang JS, 2025. Leveraging electrochemical CO₂ reduction for optimizing comprehensive benefits of multi-energy systems: a collaborative optimization approach driven by energy-carbon integrated pricing. Energy, 322:135413.

Full Text: <1650>