Similar articles

Abstract: Most existing light field (LF) super-resolution (SR) methods either fail to fully use angular information or have an unbalanced performance distribution because they use parts of views. To address these issues, we propose a novel integration network based on macro-pixel representation for the LF SR task, named MPIN. Restoring the entire LF image simultaneously, we couple the spatial and angular information by rearranging the four-dimensional LF image into a two-dimensional macro-pixel image. Then, two special convolutions are deployed to extract spatial and angular information, separately. To fully exploit spatial-angular correlations, the integration resblock is designed to merge the two kinds of information for mutual guidance, allowing our method to be angular-coherent. Under the macro-pixel representation, an angular shuffle layer is tailored to improve the spatial resolution of the macro-pixel image, which can effectively avoid aliasing. Extensive experiments on both synthetic and real-world LF datasets demonstrate that our method can achieve better performance than the state-of-the-art methods qualitatively and quantitatively. Moreover, the proposed method has an advantage in preserving the inherent epipolar structures of LF images with a balanced distribution of performance.

MPIN：基于宏像素聚合的光场图像超分辨率网络

[1]Alain M, Smolic A, 2018. Light field super-resolution via LFBM5D sparse coding. Proc 25^th IEEE Int Conf on Image Processing, p.2501-2505.

[2]Bishop TE, Favaro P, 2012. The light field camera: extended depth of field, aliasing, and superresolution. IEEE Trans Patt Anal Mach Intell, 34(5):972-986.

[3]Bishop TE, Zanetti S, Favaro P, 2009. Light field super-resolution. Proc IEEE Int Conf on Computational Photography, p.1-9.

[4]Dong C, Loy CC, He KM, et al., 2014. Learning a deep convolutional network for image super-resolution. Proc 13^th European Conf on Computer Vision, p.184-199.

[5]Fan HZ, Liu D, Xiong ZW, et al., 2017. Two-stage convolutional neural network for light field super-resolution. Proc IEEE Int Conf on Image Processing, p.1167-1171.

[6]Farrugia RA, Galea C, Guillemot C, 2017. Super resolution of light field images using linear subspace projection of patch-volumes. IEEE J Sel Top Signal Process, 11(7):1058-1071.

[7]Glorot X, Bengio Y, 2010. Understanding the difficulty of training deep feedforward neural networks. Proc 13^th Int Conf on Artificial Intelligence and Statistics, p.249-256.

[8]Gul MSK, Gunturk BK, 2018. Spatial and angular resolution enhancement of light fields using convolutional neural networks. IEEE Trans Image Process, 27(5):2146-2159.

[9]Honauer K, Johannsen O, Kondermann D, et al., 2016. A dataset and evaluation methodology for depth estimation on 4D light fields. Proc 13^th Asian Conf on Computer Vision, p.19-34.

[10]Jin J, Hou JH, Chen J, et al., 2020. Light field spatial super-resolution via deep combinatorial geometry embedding and structural consistency regularization. Proc IEEE/CVF Conf on Computer Vision and Pattern Recognition, p.2257-2266.

[11]Kalantari NK, Wang TC, Ramamoorthi R, 2016. Learning-based view synthesis for light field cameras. ACM Trans Graph, 35(6):193.

[12]Levoy M, Hanrahan P, 1996. Light field rendering. Proc 23^rd Annual Conf on Computer Graphics and Interactive Techniques, p.31-42.

[13]Li M, Diao CY, Xu DQ, et al., 2020. A non-Lambertian photometric stereo under perspective projection. Front Inform Technol Electron Eng, 21(8):1191-1205.

[14]Liang CK, Ramamoorthi R, 2015. A light transport framework for lenslet light field cameras. ACM Trans Graph, 34(2):16.

[15]Lim B, Son S, Kim H, et al., 2017. Enhanced deep residual networks for single image super-resolution. Proc IEEE Conf on Computer Vision and Pattern Recognition Workshops, p.1132-1140.

[16]Lim J, Ok H, Park B, et al., 2009. Improving the spatail resolution based on 4D light field data. Proc 16^th IEEE Int Conf on Image Processing, p.1173-1176.

[17]Mitra K, Veeraraghavan A, 2012. Light field denoising, light field superresolution and stereo camera based refocussing using a GMM light field patch prior. Proc IEEE Computer Society Conf on Computer Vision and Pattern Recognition Workshops, p.22-28.

[18]Nava FP, Luke JP, 2009. Simultaneous estimation of super-resolved depth and all-in-focus images from a plenoptic camera. Proc 3DTV Conf: the True Vision—Capture, Transmission and Display of 3D Video, p.1-4.

[19]Ng R, Levoy M, Brédif M, et al., 2005. Light Field Photography with a Hand-Held Plenoptic Camera. Technical Report No. CTSR 2005-02, Stanford University, USA.

[20]Peng JY, Xiong ZW, Liu D, et al., 2018. Unsupervised depth estimation from light field using a convolutional neural network. Proc Int Conf on 3D Vision, p.295-303.

[21]Pérez F, Pérez A, Rodríguez M, et al., 2012. Fourier slice super-resolution in plenoptic cameras. Proc IEEE Int Conf on Computational Photography, p.1-11.

[22]Pérez F, Pérez A, Rodríguez M, et al., 2015. Super-resolved Fourier-slice refocusing in plenoptic cameras. J Math Imag Vis, 52(2):200-217.

[23]Rerabek M, Ebrahimi T, 2016. New light field image dataset. Proc 8^th Int Conf on Quality of Multimedia Experience, p.1-2.

[24]Rossi M, Frossard P, 2017. Graph-based light field super-resolution. Proc IEEE 19^th Int Workshop on Multimedia Signal Processing, p.1-6.

[25]Rossi M, Frossard P, 2018. Geometry-consistent light field super-resolution via graph-based regularization. IEEE Trans Image Process, 27(9):4207-4218.

[26]Shi WZ, Caballero J, Huszár F, et al., 2016. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. Proc IEEE Conf on Computer Vision and Pattern Recognition, p.1874-1883.

[27]Wang TC, Zhu JY, Hiroaki E, et al., 2016. A 4D light-field dataset and CNN architectures for material recognition. Proc 14^th European Conf on Computer Vision, p.121-138.

[28]Wang YL, Hou GQ, Sun ZN, et al., 2016. A simple and robust super resolution method for light field images. Proc IEEE Int Conf on Image Processing, p.1459-1463.

[29]Wang YL, Liu F, Zhang KB, et al., 2018. LFNet: a novel bidirectional recurrent convolutional neural network for light-field image super-resolution. IEEE Trans Image Process, 27(9):4274-4286.

[30]Wang YQ, Wang LG, Yang JG, et al., 2020. Spatial-angular interaction for light field image super-resolution. Proc 16^th European Conf on Computer Vision, p.290-308.

[31]Wanner S, Goldluecke B, 2014. Variational light field analysis for disparity estimation and super-resolution. IEEE Trans Patt Anal Mach Intell, 36(3):606-619.

[32]Wanner S, Meister S, Goldluecke B, 2013. Datasets and benchmarks for densely sampled 4D light fields. Proc 18^th Int Workshop on Vision, Modeling, and Visualization, p.225-226.

[33]Yeung HWF, Hou JH, Chen XM, et al., 2019. Light field spatial super-resolution using deep efficient spatial-angular separable convolution. IEEE Trans Image Process, 28(5):2319-2330.

[34]Yi P, Wang ZY, Jiang K, et al., 2019. Progressive fusion video super-resolution network via exploiting non-local spatio-temporal correlations. Proc IEEE/CVF Int Conf on Computer Vision, p.3106-3115.

[35]Yoon Y, Jeon HG, Yoo D, et al., 2015. Learning a deep convolutional network for light-field image super-resolution. Proc IEEE Int Conf on Computer Vision Workshop, p.57-65.

[36]Yuan Y, Cao ZQ, Su LJ, 2018. Light-field image superresolution using a combined deep CNN based on EPI. IEEE Signal Process Lett, 25(9):1359-1363.

[37]Yücer K, Sorkine-Hornung A, Wang O, et al., 2016. Efficient 3D object segmentation from densely sampled light fields with applications to 3D reconstruction. ACM Trans Graph, 35(3):22.

[38]Zhang S, Lin YF, Sheng H, 2019. Residual networks for light field image super-resolution. Proc IEEE/CVF Conf on Computer Vision and Pattern Recognition, p.11038-11047.

[39]Zhu H, Wang Q, Yu JY, 2017. Light field imaging: models, calibrations, reconstructions, and applications. Front Inform Technol Electron Eng, 18(9):1236-1249.