Full Text:   <423>

Summary:  <203>

CLC number: TN333

On-line Access: 2023-07-24

Received: 2022-10-19

Revision Accepted: 2023-07-24

Crosschecked: 2023-06-13

Cited: 0

Clicked: 455

Citations:  Bibtex RefMan EndNote GB/T7714


Yunchuan GUAN




-   Go to

Article info.
Open peer comments

Frontiers of Information Technology & Electronic Engineering  2023 Vol.24 No.7 P.964-979


A disk failure prediction model for multiple issues

Author(s):  Yunchuan GUAN, Yu LIU, Ke ZHOU, Qiang LI, Tuanjie WANG, Hui LI

Affiliation(s):  Wuhan National Laboratory for Optoelectronics, Huazhong University of Science and Technology, Wuhan 430074, China; more

Corresponding email(s):   hustgyc@hust.edu.cn, liu_yu@hust.edu.cn

Key Words:  Storage system reliability, Disk failure prediction, Self-monitoring analysis and reporting technology (SMART), Machine learning

Yunchuan GUAN, Yu LIU, Ke ZHOU, Qiang LI, Tuanjie WANG, Hui LI. A disk failure prediction model for multiple issues[J]. Frontiers of Information Technology & Electronic Engineering, 2023, 24(7): 964-979.

@article{title="A disk failure prediction model for multiple issues",
author="Yunchuan GUAN, Yu LIU, Ke ZHOU, Qiang LI, Tuanjie WANG, Hui LI",
journal="Frontiers of Information Technology & Electronic Engineering",
publisher="Zhejiang University Press & Springer",

%0 Journal Article
%T A disk failure prediction model for multiple issues
%A Yunchuan GUAN
%A Qiang LI
%A Tuanjie WANG
%A Hui LI
%J Frontiers of Information Technology & Electronic Engineering
%V 24
%N 7
%P 964-979
%@ 2095-9184
%D 2023
%I Zhejiang University Press & Springer
%DOI 10.1631/FITEE.2200488

T1 - A disk failure prediction model for multiple issues
A1 - Yunchuan GUAN
A1 - Yu LIU
A1 - Ke ZHOU
A1 - Qiang LI
A1 - Tuanjie WANG
A1 - Hui LI
J0 - Frontiers of Information Technology & Electronic Engineering
VL - 24
IS - 7
SP - 964
EP - 979
%@ 2095-9184
Y1 - 2023
PB - Zhejiang University Press & Springer
ER -
DOI - 10.1631/FITEE.2200488

disk failure prediction methods have been useful in handing a single issue, e.g., heterogeneous disks, model aging, and minority samples. However, because these issues often exist simultaneously, prediction models that can handle only one will result in prediction bias in reality. Existing disk failure prediction methods simply fuse various models, lacking discussion of training data preparation and learning patterns when facing multiple issues, although the solutions to different issues often conflict with each other. As a result, we first explore the training data preparation for multiple issues via a data partitioning pattern, i.e., our proposed multi-property data partitioning (MDP). Then, we consider learning with the partitioned data for multiple issues as learning multiple tasks, and introduce the model-agnostic meta-learning (MAML) framework to achieve the learning. Based on these improvements, we propose a novel disk failure prediction model named MDP-MAML. MDP addresses the challenges of uneven partitioning and difficulty in partitioning by time, and MAML addresses the challenge of learning with multiple domains and minor samples for multiple issues. In addition, MDP-MAML can assimilate emerging issues for learning and prediction. On the datasets reported by two real-world data centers, compared to state-of-the-art methods, MDP-MAML can improve the area under the curve (AUC) and false detection rate (FDR) from 0.85 to 0.89 and from 0.85 to 0.91, respectively, while reducing the false alarm rate (FAR) from 4.88% to 2.85%.




Darkslateblue:Affiliate; Royal Blue:Author; Turquoise:Article


[1]Backblaze, 2018. The Backblaze Hard Drive Data and Stats. https://www.backblaze.com/b2/hard-drive-test-data.html [Accessed on Oct. 18, 2022].

[2]Botezatu MM, Giurgiu I, Bogojeska J, et al., 2016. Predicting disk replacement towards reliable data centers. Proc 22nd ACM SIGKDD Int Conf on Knowledge Discovery and Data Mining, p.39-48.

[3]Buffelli D, Vandin F, 2022. Graph representation learning for multi-task settings: a meta-learning approach. Int Joint Conf on Neural Networks, p.1-8.

[4]Finn C, Abbeel P, Levine S, 2017. Model-agnostic meta-learning for fast adaptation of deep networks. Proc 34th Int Conf on Machine Learning, p.1126-1135.

[5]Frikha A, Krompaß D, Köpken HG, et al., 2021. Few-shot one-class classification via meta-learning. Proc AAAI Conf on Artificial Intelligence, p.7448-7456.

[6]Han SJ, Lee PPC, Xu F, et al., 2021. An in-depth study of correlated failures in production SSD-based data centers. Proc 19th USENIX Conf on File and Storage Technologies, p.417-429.

[7]He HB, Garcia EA, 2009. Learning from imbalanced data. IEEE Trans Knowl Data Eng, 21(9):1263-1284.

[8]Hospedales T, Antoniou A, Micaelli P, et al., 2022. Meta-learning in neural networks: a survey. IEEE Trans Patt Anal Mach Intell, 44(9):5149-5169.

[9]Jiang TM, Zeng JF, Zhou K, et al., 2019. Lifelong disk failure prediction via GAN-based anomaly detection. IEEE 37th Int Conf on Computer Design, p.199-207.

[10]Lake BM, Salakhutdinov R, Gross J, et al., 2011. One shot learning of simple visual concepts. Proc 33rd Annual Meeting of the Cognitive Science Society, p.2568-2573.

[11]Ling CX, Huang J, Zhang H, et al., 2003. AUC: a statistically consistent and more discriminating measure than accuracy. Proc 18th Int Joint Conf on Artificial Intelligence, p.519-524.

[12]Lu SD, Luo B, Patel T, et al., 2020. Making disk failure predictions smarter! Proc 18th USENIX Conf on File and Storage Technologies, p.151-167.

[13]Luo C, Zhao P, Qiao B, et al., 2021. NTAM: neighborhood-temporal attention model for disk failure prediction in cloud platforms. Proc Web Conf, p.1181-1191.

[14]Mao YR, Wang ZK, Liu WW, et al., 2022. MetaWeighting: learning to weight tasks in multi-task learning. Findings of the Association for Computational Linguistics: ACL 2022, p.3436-3448.

[15]Nichol A, Achiam J, Schulman J, 2018. On first-order meta-learning algorithms. https://arxiv.org/abs/1803.02999

[16]Pereira FLF, dos Santos Lima FD, de Moura Leite LG, et al., 2017. Transfer learning for Bayesian networks with application on hard disk drives failure prediction. Brazilian Conf on Intelligent Systems, p.228-233.

[17]Pinheiro E, Weber WD, Barroso LA, 2007. Failure trends in a large disk drive population. Proc 5th USENIX Conf on File and Storage Technologies, p.17-28.

[18]Rincón CA, Pâris JF, Vilalta R, et al., 2017. Disk failure prediction in heterogeneous environments. Int Symp on Performance Evaluation of Computer and Telecommunication Systems, p.1-7.

[19]Shen J, Ren YJ, Wan J, et al., 2021. Hard disk drive failure prediction for mobile edge computing based on an LSTM recurrent neural network. Mob Inform Syst, 2021:1-12.

[20]Sun XY, Chakrabarty K, Huang RR, et al., 2019. System-level hardware failure prediction using deep learning. Proc 56th ACM/IEEE Design Automation Conf, p.1-6.

[21]Vaswani A, Shazeer N, Parmar N, et al., 2017. Attention is all you need. Proc 31st Int Conf on Neural Information Processing Systems, p.6000-6010.

[22]Wang JD, Lan CL, Liu C, et al., 2021. Generalizing to unseen domains: a survey on domain generalization. Proc 30th Int Joint Conf on Artificial Intelligence, p.4627-4635.

[23]Xiao J, Xiong Z, Wu S, et al., 2018. Disk failure prediction in data centers via online learning. Proc 47th Int Conf on Parallel Processing, p.1-10.

[24]Xie YW, Feng D, Wang F, et al., 2018. OME: an optimized modeling engine for disk failure prediction in heterogeneous datacenter. IEEE 36th Int Conf on Computer Design, p.561-564.

[25]Zhang J, Huang P, Zhou K, et al., 2020a. HDDse: enabling high-dimensional disk state embedding for generic failure detection system of heterogeneous disks in large data centers. Proc USENIX Annual Technical Conf, p.111-126.

[26]Zhang J, Zhou K, Huang P, et al., 2020b. Minority disk failure prediction based on transfer learning in large data centers of heterogeneous disk systems. IEEE Trans Parall Distrib Syst, 31(9):2155-2169.

[27]Zhang YQ, Hao WW, Niu B, et al., 2023. Multi-view feature-based SSD failure prediction: what, when, and why. Proc 21st USENIX Conf on File and Storage Technologies, p.409-424.

[28]Zhou Y, Wang F, Feng D, 2022. A disk failure prediction method based on active semi-supervised learning. ACM Trans Stor, 18(4):1-33.

[29]Zhu BP, Wang G, Liu XG, et al., 2013. Proactive drive failure prediction for large scale storage systems. IEEE 29th Symp on Mass Storage Systems and Technologies, p.1-5.

Open peer comments: Debate/Discuss/Question/Opinion


Please provide your name, email address and a comment

Journal of Zhejiang University-SCIENCE, 38 Zheda Road, Hangzhou 310027, China
Tel: +86-571-87952783; E-mail: cjzhang@zju.edu.cn
Copyright © 2000 - 2024 Journal of Zhejiang University-SCIENCE