CLC number: TP183
On-line Access: 2023-08-29
Received: 2022-06-21
Revision Accepted: 2023-08-29
Crosschecked: 2022-10-27
Cited: 0
Clicked: 836
Citations: Bibtex RefMan EndNote GB/T7714
Fengda ZHANG, Kun KUANG, Long CHEN, Zhaoyang YOU, Tao SHEN, Jun XIAO, Yin ZHANG, Chao WU, Fei WU, Yueting ZHUANG, Xiaolin LI. Federated unsupervised representation learning[J]. Frontiers of Information Technology & Electronic Engineering,in press.https://doi.org/10.1631/FITEE.2200268 @article{title="Federated unsupervised representation learning", %0 Journal Article TY - JOUR
联邦无监督表示学习张寅1,吴超2,吴飞1,庄越挺1,李晓林3,4,5 1浙江大学计算机科学与技术学院,中国杭州市,310027 2浙江大学公共管理学院,中国杭州市,310027 3同盾科技,中国杭州市,310000 4中国科学院基础医学与肿瘤研究所,中国杭州市,310018 5杭州灵思智康科技有限公司,中国杭州市,310018 摘要:为利用分布式边缘设备上大量未标记数据,我们在联邦学习中提出一个称为联邦无监督表示学习(FURL)的新问题,以在没有监督的情况下学习通用表示模型,同时保护数据隐私。FURL提出了两个新挑战:(1)客户端之间的数据分布转移(非独立同分布)会使本地模型专注于不同的类别,从而导致表示空间的不一致;(2)如果FURL中客户端之间没有统一的信息,客户端之间的表示就会错位。为了应对这些挑战,我们提出带字典和对齐的联合对比平均(FedCA)算法。FedCA由两个关键模块组成:字典模块,用于聚合来自每个客户端的样本表示并与所有客户端共享,以实现表示空间的一致性;对齐模块,用于将每个客户端的表示与基于公共数据训练的基础模型对齐。我们采用对比方法进行局部模型训练,通过在3个数据集上独立同分布和非独立同分布设定下的大量实验,我们证明FedCA以显著的优势优于所有基线方法。 关键词组: Darkslateblue:Affiliate; Royal Blue:Author; Turquoise:Article
Reference[1]Baevski A, Zhou H, Mohamed A, et al., 2020. wav2vec 2.0: a framework for self-supervised learning of speech representations. Proc 34th Conf on Neural Information Processing Systems. [2]Bonawitz K, Ivanov V, Kreuter B, et al., 2017. Practical secure aggregation for privacy-preserving machine learning. Proc ACM SIGSAC Conf on Computer and Communications Security, p.1175-1191. [3]Chen T, Kornblith S, Norouzi M, et al., 2020. A simple framework for contrastive learning of visual representations. Proc 37th Int Conf on Machine Learning, Article 149. [4]Chen XL, Fan HQ, Girshick R, et al., 2020. Improved baselines with momentum contrastive learning. https://arxiv.org/abs/2003.04297 [5]Coates A, Ng AY, Lee H, 2011. An analysis of single-layer networks in unsupervised feature learning. Proc 14th Int Conf on Artificial Intelligence and Statistics, p.215-223. [6]Deng J, Dong W, Socher R, et al., 2009. ImageNet: a large-scale hierarchical image database. Proc IEEE Conf on Computer Vision and Pattern Recognition, p.248-255. [7]Dosovitskiy A, Springenberg JT, Riedmiller M, et al., 2014. Discriminative unsupervised feature learning with convolutional neural networks. Proc 27th Int Conf on Neural Information Processing Systems, p.766-774. [8]Duan XY, Tang SL, Zhang SY, et al., 2018. Temporality-enhanced knowledge memory network for factoid question answering. Front Inform Technol Electron Eng, 19(1):104-115. [9]Gidaris S, Singh P, Komodakis N, 2018. Unsupervised representation learning by predicting image rotations. Proc 6th Int Conf on Learning Representations. [10]Hadsell R, Chopra S, LeCun Y, 2006. Dimensionality reduction by learning an invariant mapping. Proc IEEE Computer Society Conf on Computer Vision and Pattern Recognition, p.1735-1742. [11]Hassani K, Ahmadi AHK, 2020. Contrastive multi-view representation learning on graphs. Proc 37th Int Conf on Machine Learning, p.4116-4126. [12]He CY, Yang ZY, Mushtaq E, et al., 2021. SSFL: tackling label deficiency in federated learning via personalized self-supervision. https://arxiv.org/abs/2110.02470 [13]He KM, Zhang XY, Ren SQ, et al., 2016. Deep residual learning for image recognition. Proc IEEE Conf on Computer Vision and Pattern Recognition, p.770-778. [14]He KM, Fan HQ, Wu YX, et al., 2020. Momentum contrast for unsupervised visual representation learning. Proc IEEE/CVF Conf on Computer Vision and Pattern Recognition, p.9729-9738. [15]Hinton GE, Salakhutdinov RR, 2006. Reducing the dimensionality of data with neural networks. Science, 313(5786):504-507. [16]Jeong E, Oh S, Kim H, et al., 2018. Communication-efficient on-device machine learning: federated distillation and augmentation under non-IID private data. https://arxiv.org/abs/1811.11479v1 [17]Ji SX, Saravirta T, Pan SR, et al., 2021. Emerging trends in federated learning: from model fusion to federated X learning. https://arxiv.org/abs/2102.12920 [18]Jin YL, Wei XG, Liu Y, et al., 2020. Towards utilizing unlabeled data in federated learning: a survey and prospective. https://arxiv.org/abs/2002.11545 [19]Kairouz P, McMahan HB, Avent B, et al., 2021. Advances and open problems in federated learning. https://arxiv.org/abs/1912.04977 [20]Kempe D, McSherry F, 2008. A decentralized algorithm for spectral analysis. J Comput Syst Sci, 74(1):70-83. [21]Kingma DP, Welling M, 2014. Auto-encoding variational Bayes. Proc 2nd Int Conf on Learning Representations. [22]Konečný J, McMahan HB, Yu FX, et al., 2017. Federated learning: strategies for improving communication efficiency. https://arxiv.org/abs/1610.05492 [23]Krizhevsky A, 2009. Learning Multiple Layers of Features from Tiny Images. Technical Report TR-2009, University of Toronto, Toronto, Canada. [24]Krizhevsky A, Sutskever I, Hinton GE, 2012. ImageNet classification with deep convolutional neural networks. Proc 25th Int Conf on Neural Information Processing Systems, p.1097-1105. [25]Kuang K, Li L, Geng Z, et al., 2020. Causal inference. Engineering, 6(3):253-263. [26]Lei N, An DS, Guo Y, et al., 2020. A geometric understanding of deep learning. Engineering, 6(3):361-374. [27]Li QB, He BS, Song D, 2021. Model-contrastive federated learning. Proc IEEE/CVF Conf on Computer Vision and Pattern Recognition, p.10713-10722. [28]Li T, Sahu AK, Zaheer M, et al., 2020. Federated optimization in heterogeneous networks. Proc 3rd MLSys Conf. [29]Liang JL, Zhang MH, Zeng XY, et al., 2014. Distributed dictionary learning for sparse representation in sensor networks. IEEE Trans Image Process, 23(6):2528-2541. [30]Logeswaran L, Lee H, 2018. An efficient framework for learning sentence representations. Proc 6th Int Conf on Learning Representations. [31]Lyu YG, 2020. Artificial intelligence: enabling technology to empower society. Engineering, 6(3):205-206. [32]McMahan B, Moore E, Ramage D, et al., 2017. Communication-efficient learning of deep networks from decentralized data. Proc 20th Int Conf on Artificial Intelligence and Statistics, p.1273-1282. [33]Mikolov T, Sutskever I, Chen K, et al., 2013. Distributed representations of words and phrases and their compositionality. Proc 26th Int Conf on Neural Information Processing Systems, p.3111-3119. [34]Pan YH, 2020. Multiple knowledge representation of artificial intelligence. Engineering, 6(3):216-217. [35]Paszke A, Gross S, Massa F, et al., 2019. PyTorch: an imperative style, high-performance deep learning library. Proc 33rd Conf on Neural Information Processing Systems, p.8026-8037. [36]Pathak D, Agrawal P, Efros AA, et al., 2017. Curiosity-driven exploration by self-supervised prediction. Proc IEEE Conf on Computer Vision and Pattern Recognition Workshops, p.16-17. [37]Qiu JZ, Chen QB, Dong YX, et al., 2020. GCC: graph contrastive coding for graph neural network pre-training. Proc 26th ACM SIGKDD Int Conf on Knowledge Discovery & Data Mining, p.1150-1160. [38]Radford A, Metz L, Chintala S, 2016. Unsupervised representation learning with deep convolutional generative adversarial networks. Proc 4th Int Conf on Learning Representations. [39]Raja H, Bajwa WU, 2016. Cloud K-SVD: a collaborative dictionary learning algorithm for big, distributed data. IEEE Trans Signal Process, 64(1):173-188. [40]Sattler F, Wiedemann S, Müller KR, et al., 2020. Robust and communication-efficient federated learning from non-i.i.d. data. IEEE Trans Neur Netw Learn Syst, 31(9):3400-3413. [41]Sattler F, Korjakow T, Rischke R, et al., 2021. FEDAUX: leveraging unlabeled auxiliary data in federated learning. IEEE Trans Neur Netw Learn Syst, early access. [42]Sermanet P, Lynch C, Chebotar Y, et al., 2018. Time-contrastive networks: self-supervised learning from video. Proc IEEE Int Conf on Robotics and Automation, p.1134-1141. [43]Shakeri Z, Raja H, Bajwa WU, 2014. Dictionary learning based nonlinear classifier training from distributed data. Proc IEEE Global Conf on Signal and Information Processing, p.759-763. [44]Shi HZ, Zhang YC, Shen ZJ, et al., 2022. Federated self-supervised contrastive learning via ensemble similarity distillation. https://arxiv.org/abs/2109.14611v1 [45]Sohn K, 2016. Improved deep metric learning with multi-class N-pair loss objective. Proc 30th Int Conf on Neural Information Processing Systems, p.1857-1865. [46]Tian YL, Krishnan D, Isola P, 2020. Contrastive multiview coding. Proc 16th European Conf on Computer Vision, p.776-794. [47]van Berlo B, Saeed A, Ozcelebi T, 2020. Towards federated unsupervised representation learning. Proc 3rd ACM Int Workshop on Edge Systems, Analytics and Networking, p.31-36. [48]van den Oord A, Li YZ, Vinyals O, 2019. Representation learning with contrastive predictive coding. https://arxiv.org/abs/1807.03748 [49]Vinyals O, Blundell C, Lillicrap T, et al., 2016. Matching networks for one shot learning. Proc 30th Int Conf on Neural Information Processing Systems, p.3637-3645. [50]Wang HY, Yurochkin M, Sun YK, et al., 2020. Federated learning with matched averaging. Proc 8th Int Conf on Learning Representations. [51]Wang TZ, Isola P, 2020. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. Proc 37th Int Conf on Machine Learning, p.9929-9939. [52]Wu SX, Wai HT, Li L, et al., 2018. A review of distributed algorithms for principal component analysis. Proc IEEE, 106(8):1321-1340. [53]Wu YW, Zeng DW, Wang ZP, et al., 2021. Federated contrastive learning for volumetric medical image segmentation. Proc 24th Int Conf on Medical Image Computing and Computer-Assisted Intervention, p.367-377. [54]Wu ZR, Xiong YJ, Yu SX, et al., 2018. Unsupervised feature learning via non-parametric instance discrimination. Proc IEEE/CVF Conf on Computer Vision and Pattern Recognition, p.3733-3742. [55]Yang Q, Liu Y, Chen TJ, et al., 2019. Federated machine learning: concept and applications. ACM Trans Intell Syst Technol, 10(2):12. [56]Yang ZL, Dai ZH, Yang YM, et al., 2019. XLNet: generalized autoregressive pretraining for language understanding. Proc 33rd Int Conf on Neural Information Processing Systems, Article 517. [57]Zhao Y, Li M, Lai LZ, et al., 2022. Federated learning with non-IID data. https://arxiv.org/abs/1806.00582 [58]Zhou LK, Tang SL, Xiao J, et al., 2017. Disambiguating named entities with deep supervised learning via crowd labels. Front Inform Technol Electron Eng, 18(1):97-106. [59]Zhu YX, Gao T, Fan LF, et al., 2020. Dark, beyond deep: a paradigm shift to cognitive AI with humanlike common sense. Engineering, 6(3):310-345. [60]Zhuang WM, Gan X, Wen YG, et al., 2021a. Collaborative unsupervised visual representation learning from decentralized data. Proc IEEE/CVF Int Conf on Computer Vision, p.4892-4901. [61]Zhuang WM, Wen YG, Zhang S, 2021b. Joint optimization in edge-cloud continuum for federated unsupervised person re-identification. Proc 29th ACM Int Conf on Multimedia, p.433-441. [62]Zhuang WM, Wen YG, Zhang S, 2022. Divergence-aware federated self-supervised learning. Proc 10th Int Conf on Learning Representations. [63]Zhuang YT, Wu F, Chen C, et al., 2017. Challenges and opportunities: from big data to knowledge in AI 2.0. Front Inform Technol Electron Eng, 18(1):3-14. [64]Zhuang YT, Cai M, Li XL, et al., 2020. The next breakthroughs of artificial intelligence: the interdisciplinary nature of AI. Engineering, 6(3):245-247. Journal of Zhejiang University-SCIENCE, 38 Zheda Road, Hangzhou
310027, China
Tel: +86-571-87952783; E-mail: cjzhang@zju.edu.cn Copyright © 2000 - 2024 Journal of Zhejiang University-SCIENCE |
Open peer comments: Debate/Discuss/Question/Opinion
<1>