|
|
Frontiers of Information Technology & Electronic Engineering
ISSN 2095-9184 (print), ISSN 2095-9230 (online)
2020 Vol.21 No.7 P.1034-1046
MDLB: a metadata dynamic load balancing mechanism based on reinforcement learning
Abstract: With the growing amount of information and data, object-oriented storage systems have been widely used in many applications, including the Google File System, Amazon S3, Hadoop Distributed File System, and Ceph, in which load balancing of metadata plays an important role in improving the input/output performance of the entire system. Unbalanced load on the metadata server leads to a serious bottleneck problem for system performance. However, most existing metadata load balancing strategies, which are based on subtree segmentation or hashing, lack good dynamics and adaptability. In this study, we propose a metadata dynamic load balancing (MDLB) mechanism based on reinforcement learning (RL). We learn that the Q_learning algorithm and our RL-based strategy consist of three modules, i.e., the policy selection network, load balancing network, and parameter update network. Experimental results show that the proposed MDLB algorithm can adjust the load dynamically according to the performance of the metadata servers, and that it has good adaptability in the case of sudden change of data volume.
This article has been corrected, see doi:10.1631/FITEE.19e0121
Key words: Object-oriented storage system, Metadata, Dynamic load balancing, Reinforcement learning, Q_learning
1国家数字交换系统工程技术研究中心，中国郑州市，450002
2复旦大学计算机科学技术学院，中国上海市，200433
3复旦大学大数据研究院，中国上海市，200433
摘要：随着信息和数据量增长，面向对象的存储系统已被广泛应用到很多领域，包括Google文件系统、AmazonS3、Hadoop分布式文件系统和Ceph。其中元数据负载均衡在提高整个系统输入/输出性能方面起着重要作用，元数据负载不平衡会导致服务器出现严重的系统性能瓶颈问题。然而现有元数据负载平衡策略缺乏良好动态性和适用性，如基于子树分割或者哈希的负载策略。提出一种基于强化学习的动态负载平衡机制（MDLB）。采用Q_learning算法，所提基于强化学习机制由3个模块组成，即策略选择网络、负载均衡网络和参数更新网络。实验结果表明MDLB算法可根据元数据服务器的性能动态调节负载，在数据量骤变情况下仍具有很好适应性。
关键词组：
References:
Open peer comments: Debate/Discuss/Question/Opinion
<1>
DOI:
10.1631/FITEE.1900121
CLC number:
TP338.8
Download Full Text:
Downloaded:
7787
Download summary:
<Click Here>Downloaded:
1525Clicked:
5730
Cited:
0
On-line Access:
2020-07-10
Received:
2019-03-01
Revision Accepted:
2019-11-14
Crosschecked:
2020-06-02