Similar articles

Abstract: To leverage the enormous amount of unlabeled data on distributed edge devices, we formulate a new problem in federated learning called federated unsupervised representation learning (FURL) to learn a common representation model without supervision while preserving data privacy. FURL poses two new challenges: (1) data distribution shift (non-independent and identically distributed, non-IID) among clients would make local models focus on different categories, leading to the inconsistency of representation spaces; (2) without unified information among the clients in FURL, the representations across clients would be misaligned. To address these challenges, we propose the federated contrastive averaging with dictionary and alignment (FedCA) algorithm. FedCA is composed of two key modules: a dictionary module to aggregate the representations of samples from each client which can be shared with all clients for consistency of representation space and an alignment module to align the representation of each client on a base model trained on public data. We adopt the contrastive approach for local model training. Through extensive experiments with three evaluation protocols in IID and non-IID settings, we demonstrate that FedCA outperforms all baselines with significant margins.

联邦无监督表示学习

[1]Baevski A, Zhou H, Mohamed A, et al., 2020. wav2vec 2.0: a framework for self-supervised learning of speech representations. Proc 34^th Conf on Neural Information Processing Systems.

[2]Bonawitz K, Ivanov V, Kreuter B, et al., 2017. Practical secure aggregation for privacy-preserving machine learning. Proc ACM SIGSAC Conf on Computer and Communications Security, p.1175-1191.

[3]Chen T, Kornblith S, Norouzi M, et al., 2020. A simple framework for contrastive learning of visual representations. Proc 37^th Int Conf on Machine Learning, Article 149.

[4]Chen XL, Fan HQ, Girshick R, et al., 2020. Improved baselines with momentum contrastive learning. https://arxiv.org/abs/2003.04297

[5]Coates A, Ng AY, Lee H, 2011. An analysis of single-layer networks in unsupervised feature learning. Proc 14^th Int Conf on Artificial Intelligence and Statistics, p.215-223.

[6]Deng J, Dong W, Socher R, et al., 2009. ImageNet: a large-scale hierarchical image database. Proc IEEE Conf on Computer Vision and Pattern Recognition, p.248-255.

[7]Dosovitskiy A, Springenberg JT, Riedmiller M, et al., 2014. Discriminative unsupervised feature learning with convolutional neural networks. Proc 27^th Int Conf on Neural Information Processing Systems, p.766-774.

[8]Duan XY, Tang SL, Zhang SY, et al., 2018. Temporality-enhanced knowledge memory network for factoid question answering. Front Inform Technol Electron Eng, 19(1):104-115.

[9]Gidaris S, Singh P, Komodakis N, 2018. Unsupervised representation learning by predicting image rotations. Proc 6^th Int Conf on Learning Representations.

[10]Hadsell R, Chopra S, LeCun Y, 2006. Dimensionality reduction by learning an invariant mapping. Proc IEEE Computer Society Conf on Computer Vision and Pattern Recognition, p.1735-1742.

[11]Hassani K, Ahmadi AHK, 2020. Contrastive multi-view representation learning on graphs. Proc 37^th Int Conf on Machine Learning, p.4116-4126.

[12]He CY, Yang ZY, Mushtaq E, et al., 2021. SSFL: tackling label deficiency in federated learning via personalized self-supervision. https://arxiv.org/abs/2110.02470

[13]He KM, Zhang XY, Ren SQ, et al., 2016. Deep residual learning for image recognition. Proc IEEE Conf on Computer Vision and Pattern Recognition, p.770-778.

[14]He KM, Fan HQ, Wu YX, et al., 2020. Momentum contrast for unsupervised visual representation learning. Proc IEEE/CVF Conf on Computer Vision and Pattern Recognition, p.9729-9738.

[15]Hinton GE, Salakhutdinov RR, 2006. Reducing the dimensionality of data with neural networks. Science, 313(5786):504-507.

[16]Jeong E, Oh S, Kim H, et al., 2018. Communication-efficient on-device machine learning: federated distillation and augmentation under non-IID private data. https://arxiv.org/abs/1811.11479v1

[17]Ji SX, Saravirta T, Pan SR, et al., 2021. Emerging trends in federated learning: from model fusion to federated X learning. https://arxiv.org/abs/2102.12920

[18]Jin YL, Wei XG, Liu Y, et al., 2020. Towards utilizing unlabeled data in federated learning: a survey and prospective. https://arxiv.org/abs/2002.11545

[19]Kairouz P, McMahan HB, Avent B, et al., 2021. Advances and open problems in federated learning. https://arxiv.org/abs/1912.04977

[20]Kempe D, McSherry F, 2008. A decentralized algorithm for spectral analysis. J Comput Syst Sci, 74(1):70-83.

[21]Kingma DP, Welling M, 2014. Auto-encoding variational Bayes. Proc 2^nd Int Conf on Learning Representations.

[22]Konečný J, McMahan HB, Yu FX, et al., 2017. Federated learning: strategies for improving communication efficiency. https://arxiv.org/abs/1610.05492

[23]Krizhevsky A, 2009. Learning Multiple Layers of Features from Tiny Images. Technical Report TR-2009, University of Toronto, Toronto, Canada.

[24]Krizhevsky A, Sutskever I, Hinton GE, 2012. ImageNet classification with deep convolutional neural networks. Proc 25^th Int Conf on Neural Information Processing Systems, p.1097-1105.

[25]Kuang K, Li L, Geng Z, et al., 2020. Causal inference. Engineering, 6(3):253-263.

[26]Lei N, An DS, Guo Y, et al., 2020. A geometric understanding of deep learning. Engineering, 6(3):361-374.

[27]Li QB, He BS, Song D, 2021. Model-contrastive federated learning. Proc IEEE/CVF Conf on Computer Vision and Pattern Recognition, p.10713-10722.

[28]Li T, Sahu AK, Zaheer M, et al., 2020. Federated optimization in heterogeneous networks. Proc 3^rd MLSys Conf.

[29]Liang JL, Zhang MH, Zeng XY, et al., 2014. Distributed dictionary learning for sparse representation in sensor networks. IEEE Trans Image Process, 23(6):2528-2541.

[30]Logeswaran L, Lee H, 2018. An efficient framework for learning sentence representations. Proc 6^th Int Conf on Learning Representations.

[31]Lyu YG, 2020. Artificial intelligence: enabling technology to empower society. Engineering, 6(3):205-206.

[32]McMahan B, Moore E, Ramage D, et al., 2017. Communication-efficient learning of deep networks from decentralized data. Proc 20^th Int Conf on Artificial Intelligence and Statistics, p.1273-1282.

[33]Mikolov T, Sutskever I, Chen K, et al., 2013. Distributed representations of words and phrases and their compositionality. Proc 26^th Int Conf on Neural Information Processing Systems, p.3111-3119.

[34]Pan YH, 2020. Multiple knowledge representation of artificial intelligence. Engineering, 6(3):216-217.

[35]Paszke A, Gross S, Massa F, et al., 2019. PyTorch: an imperative style, high-performance deep learning library. Proc 33^rd Conf on Neural Information Processing Systems, p.8026-8037.

[36]Pathak D, Agrawal P, Efros AA, et al., 2017. Curiosity-driven exploration by self-supervised prediction. Proc IEEE Conf on Computer Vision and Pattern Recognition Workshops, p.16-17.

[37]Qiu JZ, Chen QB, Dong YX, et al., 2020. GCC: graph contrastive coding for graph neural network pre-training. Proc 26^th ACM SIGKDD Int Conf on Knowledge Discovery & Data Mining, p.1150-1160.

[38]Radford A, Metz L, Chintala S, 2016. Unsupervised representation learning with deep convolutional generative adversarial networks. Proc 4^th Int Conf on Learning Representations.

[39]Raja H, Bajwa WU, 2016. Cloud K-SVD: a collaborative dictionary learning algorithm for big, distributed data. IEEE Trans Signal Process, 64(1):173-188.

[40]Sattler F, Wiedemann S, Müller KR, et al., 2020. Robust and communication-efficient federated learning from non-i.i.d. data. IEEE Trans Neur Netw Learn Syst, 31(9):3400-3413.

[41]Sattler F, Korjakow T, Rischke R, et al., 2021. FEDAUX: leveraging unlabeled auxiliary data in federated learning. IEEE Trans Neur Netw Learn Syst, early access.

[42]Sermanet P, Lynch C, Chebotar Y, et al., 2018. Time-contrastive networks: self-supervised learning from video. Proc IEEE Int Conf on Robotics and Automation, p.1134-1141.

[43]Shakeri Z, Raja H, Bajwa WU, 2014. Dictionary learning based nonlinear classifier training from distributed data. Proc IEEE Global Conf on Signal and Information Processing, p.759-763.

[44]Shi HZ, Zhang YC, Shen ZJ, et al., 2022. Federated self-supervised contrastive learning via ensemble similarity distillation. https://arxiv.org/abs/2109.14611v1

[45]Sohn K, 2016. Improved deep metric learning with multi-class N-pair loss objective. Proc 30^th Int Conf on Neural Information Processing Systems, p.1857-1865.

[46]Tian YL, Krishnan D, Isola P, 2020. Contrastive multiview coding. Proc 16^th European Conf on Computer Vision, p.776-794.

[47]van Berlo B, Saeed A, Ozcelebi T, 2020. Towards federated unsupervised representation learning. Proc 3^rd ACM Int Workshop on Edge Systems, Analytics and Networking, p.31-36.

[48]van den Oord A, Li YZ, Vinyals O, 2019. Representation learning with contrastive predictive coding. https://arxiv.org/abs/1807.03748

[49]Vinyals O, Blundell C, Lillicrap T, et al., 2016. Matching networks for one shot learning. Proc 30^th Int Conf on Neural Information Processing Systems, p.3637-3645.

[50]Wang HY, Yurochkin M, Sun YK, et al., 2020. Federated learning with matched averaging. Proc 8^th Int Conf on Learning Representations.

[51]Wang TZ, Isola P, 2020. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. Proc 37^th Int Conf on Machine Learning, p.9929-9939.

[52]Wu SX, Wai HT, Li L, et al., 2018. A review of distributed algorithms for principal component analysis. Proc IEEE, 106(8):1321-1340.

[53]Wu YW, Zeng DW, Wang ZP, et al., 2021. Federated contrastive learning for volumetric medical image segmentation. Proc 24^th Int Conf on Medical Image Computing and Computer-Assisted Intervention, p.367-377.

[54]Wu ZR, Xiong YJ, Yu SX, et al., 2018. Unsupervised feature learning via non-parametric instance discrimination. Proc IEEE/CVF Conf on Computer Vision and Pattern Recognition, p.3733-3742.

[55]Yang Q, Liu Y, Chen TJ, et al., 2019. Federated machine learning: concept and applications. ACM Trans Intell Syst Technol, 10(2):12.

[56]Yang ZL, Dai ZH, Yang YM, et al., 2019. XLNet: generalized autoregressive pretraining for language understanding. Proc 33^rd Int Conf on Neural Information Processing Systems, Article 517.

[57]Zhao Y, Li M, Lai LZ, et al., 2022. Federated learning with non-IID data. https://arxiv.org/abs/1806.00582

[58]Zhou LK, Tang SL, Xiao J, et al., 2017. Disambiguating named entities with deep supervised learning via crowd labels. Front Inform Technol Electron Eng, 18(1):97-106.

[59]Zhu YX, Gao T, Fan LF, et al., 2020. Dark, beyond deep: a paradigm shift to cognitive AI with humanlike common sense. Engineering, 6(3):310-345.

[60]Zhuang WM, Gan X, Wen YG, et al., 2021a. Collaborative unsupervised visual representation learning from decentralized data. Proc IEEE/CVF Int Conf on Computer Vision, p.4892-4901.

[61]Zhuang WM, Wen YG, Zhang S, 2021b. Joint optimization in edge-cloud continuum for federated unsupervised person re-identification. Proc 29^th ACM Int Conf on Multimedia, p.433-441.

[62]Zhuang WM, Wen YG, Zhang S, 2022. Divergence-aware federated self-supervised learning. Proc 10^th Int Conf on Learning Representations.

[63]Zhuang YT, Wu F, Chen C, et al., 2017. Challenges and opportunities: from big data to knowledge in AI 2.0. Front Inform Technol Electron Eng, 18(1):3-14.

[64]Zhuang YT, Cai M, Li XL, et al., 2020. The next breakthroughs of artificial intelligence: the interdisciplinary nature of AI. Engineering, 6(3):245-247.