Abstract: Real-time energy management of a hybrid excavator is addressed using reinforcement learning (RL). Due to the computational complexity and need for a priori knowledge of the load cycles, a traditional optimal control method, like dynamic programming (DP), is not feasible for real-time control. Real-time controllers derived from traditional optimal control methods compute the solutions either in a cycle-dependent manner or far away from the optimal. An RL-based energy management controller is proposed to solve this problem. The simulation and experimental results demonstrate that the RL controller has a better performance than the widely used thermostat and equivalent consumption minimization strategy (ECMS) controllers. It also shows that the RL controller is cycle-independent. Pontryagin’s minimum principle (PMP) is used to obtain the analytical solution of the energy management problem, and this can help to reduce the iteration time in the design process.

基于强化学习的混合动力挖掘机实时能量管理控制器设计

[1]Borhan, H.A., Vahidi, A., Phillips, A.M., et al., 2009. Predictive energy management of a power-split hybrid electric vehicle. American Control Conference, p.3970-3976.

[2]Brahma, A., Guezennec, Y., Rizzoni, G., 2000. Optimal energy management in series hybrid electric vehicles. American Control Conference, p.60-64.

[3]Busoniu, L., Babuska, R., Schutter, D., et al., 2010. Reinforcement Learning and Dynamic Programming Using Function Approximators. CRC Press, Boca Raton, USA, p.101-113.

[4]Chan, C.C., 2007. The state of the art of electric, hybrid, and fuel cell vehicles. Proceedings of the IEEE, 95(4):704-718.

[5]Ehsani, M., Gao, Y., Miller, J.M., 2007. Hybrid electric vehicles: architecture and motor drives. Proceedings of the IEEE, 95(4):719-728.

[6]Geering, H.P., 2007. Optimal Control with Engineering Applications. Springer, New York, USA, p.3-5.

[7]Gosavi, A., 2003. Simulation-based Optimization. Springer, New York, USA, p.197-211.

[8]Jalil, N., Kheir, N.A., Salman, M., 1997. A rule-based energy management strategy for a series hybrid vehicle. American Control Conference, p.689-693.

[9]Johri, R., Filipi, Z., 2009. Low-cost pathway to ultra efficient city car: series hydraulic hybrid system with optimized supervisory control. SAE International Journal of Engines, 2(2):505-520.

[10]Kim, H., Choi, J., Yi, K., 2012. Development of supervisory control strategy for optimized fuel consumption of the compound hybrid excavator. Proceedings of the Institution of Mechanical Engineers, Part D: Journal of Automobile Engineering, 226(12):1652-1666.

[11]Kim, H., Yoo, S., Cho, S., et al., 2016. Hybrid control algorithm for fuel consumption of a compound hybrid excavator. Automation in Construction, 68:1-10.

[12]Kim, N., Rousseau, A., 2012. Sufficient conditions of optimal control based on Pontryagin’s minimum principle for use in hybrid electric vehicles. Proceedings of the Institution of Mechanical Engineers, Part D: Journal of Automobile Engineering, 226(9):1160-1170.

[13]Kim, N., Cha, S., Peng, H., 2011. Optimal control of hybrid electric vehicles based on Pontryagin’s minimum principle. IEEE Transactions on Control Systems Technology, 19(5):1279-1287.

[14]Lin, C.C., Peng, H., Grizzle, J.W., et al., 2003. Power management strategy for a parallel hybrid electric truck. IEEE Transactions on Control Systems Technology, 11(6):839-849.

[15]Lin, T., Wang, Q., Hu, B., et al., 2010. Development of hybrid powered hydraulic construction machinery. Automation in Construction, 19(1):11-19.

[16]Lin, X., Pan, S.X., Wang, D.Y., 2008. Dynamic simulation and optimal control strategy for a parallel hybrid hydraulic excavator. Journal of Zhejiang University-SCIENCE A, 9(5):624-632.

[17]Musardo, C., Rizzoni, G., Guezennec, Y., et al., 2005. A-ECMS: an adaptive algorithm for hybrid electric vehicle energy management. European Journal of Control, 11(4-5):509-524.

[18]Salman, M., Schouten, N.J., Kheir, N.A., 2000. Control strategies for parallel hybrid vehicles. American Control Conference, p.524-528.

[19]Salmasi, F.R., 2007. Control strategies for hybrid electric vehicles: evolution, classification, comparison, and future trends. IEEE Transactions on Vehicular Technology, 56(5):2393-2404.

[20]Schouten, N.J., Salman, M.A., Kheir, N.A., 2002. Fuzzy logic control for parallel hybrid vehicles. IEEE Transactions on Control Systems Technology, 10(3):460-468.

[21]Sciarretta, A., Guzzella, L., 2007. Control of hybrid electric vehicles. IEEE Control Systems, 27(2):60-70.

[22]Sciarretta, A., Back, M., Guzzella, L., 2004. Optimal control of parallel hybrid electric vehicles. IEEE Transactions on Control Systems Technology, 12(3):352-363.

[23]Serrao, L., Onori, S., Rizzoni, G., 2009. ECMS as a realization of Pontryagin’s minimum principle for HEV control. American Control Conference, p.3964-3969.

[24]Serrao, L., Onori, S., Rizzoni, G., 2011. A comparative analysis of energy management strategies for hybrid electric vehicles. Journal of Dynamic Systems, Measurement, and Control, 133(3):031012.

[25]Watkins, C., Dayan, P., 1992. Q-learning. Machine Learning, 8(3-4):279-292.

[26]Xiao, Q., Wang, Q., Zhang, Y., 2008. Control strategies of power system in hybrid hydraulic excavator. Automation in Construction, 17(4):361-367.

[27]Zhang, S., Xiong, R., 2015. Adaptive energy management of a plug-in hybrid electric vehicle based on driving pattern recognition and dynamic programming. Applied Energy, 155:68-78.

[28]Zhu, Q., Wang, Q., Chen, Q., 2016. Component sizing for compound hybrid excavators. Proceedings of the Institution of Mechanical Engineers, Part D: Journal of Automobile Engineering, 230(7):969-982.