CLC number: TP312
On-line Access: 2021-10-08
Received: 2020-10-20
Revision Accepted: 2021-03-26
Crosschecked: 2021-08-31
Cited: 0
Clicked: 6003
Citations: Bibtex RefMan EndNote GB/T7714
Xinya Wang, Jiayi Ma, Wenjing Gao, Junjun Jiang. MPIN: a macro-pixel integration network for light field super-resolution[J]. Frontiers of Information Technology & Electronic Engineering, 2021, 22(10): 1299-1310.
@article{title="MPIN: a macro-pixel integration network for light field super-resolution",
author="Xinya Wang, Jiayi Ma, Wenjing Gao, Junjun Jiang",
journal="Frontiers of Information Technology & Electronic Engineering",
volume="22",
number="10",
pages="1299-1310",
year="2021",
publisher="Zhejiang University Press & Springer",
doi="10.1631/FITEE.2000566"
}
%0 Journal Article
%T MPIN: a macro-pixel integration network for light field super-resolution
%A Xinya Wang
%A Jiayi Ma
%A Wenjing Gao
%A Junjun Jiang
%J Frontiers of Information Technology & Electronic Engineering
%V 22
%N 10
%P 1299-1310
%@ 2095-9184
%D 2021
%I Zhejiang University Press & Springer
%DOI 10.1631/FITEE.2000566
TY - JOUR
T1 - MPIN: a macro-pixel integration network for light field super-resolution
A1 - Xinya Wang
A1 - Jiayi Ma
A1 - Wenjing Gao
A1 - Junjun Jiang
J0 - Frontiers of Information Technology & Electronic Engineering
VL - 22
IS - 10
SP - 1299
EP - 1310
%@ 2095-9184
Y1 - 2021
PB - Zhejiang University Press & Springer
ER -
DOI - 10.1631/FITEE.2000566
Abstract: Most existing light field (LF) super-resolution (SR) methods either fail to fully use angular information or have an unbalanced performance distribution because they use parts of views. To address these issues, we propose a novel integration network based on macro-pixel representation for the LF SR task, named MPIN. Restoring the entire LF image simultaneously, we couple the spatial and angular information by rearranging the four-dimensional LF image into a two-dimensional macro-pixel image. Then, two special convolutions are deployed to extract spatial and angular information, separately. To fully exploit spatial-angular correlations, the integration resblock is designed to merge the two kinds of information for mutual guidance, allowing our method to be angular-coherent. Under the macro-pixel representation, an angular shuffle layer is tailored to improve the spatial resolution of the macro-pixel image, which can effectively avoid aliasing. Extensive experiments on both synthetic and real-world LF datasets demonstrate that our method can achieve better performance than the state-of-the-art methods qualitatively and quantitatively. Moreover, the proposed method has an advantage in preserving the inherent epipolar structures of LF images with a balanced distribution of performance.
[1]Alain M, Smolic A, 2018. Light field super-resolution via LFBM5D sparse coding. Proc 25th IEEE Int Conf on Image Processing, p.2501-2505.
[2]Bishop TE, Favaro P, 2012. The light field camera: extended depth of field, aliasing, and superresolution. IEEE Trans Patt Anal Mach Intell, 34(5):972-986.
[3]Bishop TE, Zanetti S, Favaro P, 2009. Light field super-resolution. Proc IEEE Int Conf on Computational Photography, p.1-9.
[4]Dong C, Loy CC, He KM, et al., 2014. Learning a deep convolutional network for image super-resolution. Proc 13th European Conf on Computer Vision, p.184-199.
[5]Fan HZ, Liu D, Xiong ZW, et al., 2017. Two-stage convolutional neural network for light field super-resolution. Proc IEEE Int Conf on Image Processing, p.1167-1171.
[6]Farrugia RA, Galea C, Guillemot C, 2017. Super resolution of light field images using linear subspace projection of patch-volumes. IEEE J Sel Top Signal Process, 11(7):1058-1071.
[7]Glorot X, Bengio Y, 2010. Understanding the difficulty of training deep feedforward neural networks. Proc 13th Int Conf on Artificial Intelligence and Statistics, p.249-256.
[8]Gul MSK, Gunturk BK, 2018. Spatial and angular resolution enhancement of light fields using convolutional neural networks. IEEE Trans Image Process, 27(5):2146-2159.
[9]Honauer K, Johannsen O, Kondermann D, et al., 2016. A dataset and evaluation methodology for depth estimation on 4D light fields. Proc 13th Asian Conf on Computer Vision, p.19-34.
[10]Jin J, Hou JH, Chen J, et al., 2020. Light field spatial super-resolution via deep combinatorial geometry embedding and structural consistency regularization. Proc IEEE/CVF Conf on Computer Vision and Pattern Recognition, p.2257-2266.
[11]Kalantari NK, Wang TC, Ramamoorthi R, 2016. Learning-based view synthesis for light field cameras. ACM Trans Graph, 35(6):193.
[12]Levoy M, Hanrahan P, 1996. Light field rendering. Proc 23rd Annual Conf on Computer Graphics and Interactive Techniques, p.31-42.
[13]Li M, Diao CY, Xu DQ, et al., 2020. A non-Lambertian photometric stereo under perspective projection. Front Inform Technol Electron Eng, 21(8):1191-1205.
[14]Liang CK, Ramamoorthi R, 2015. A light transport framework for lenslet light field cameras. ACM Trans Graph, 34(2):16.
[15]Lim B, Son S, Kim H, et al., 2017. Enhanced deep residual networks for single image super-resolution. Proc IEEE Conf on Computer Vision and Pattern Recognition Workshops, p.1132-1140.
[16]Lim J, Ok H, Park B, et al., 2009. Improving the spatail resolution based on 4D light field data. Proc 16th IEEE Int Conf on Image Processing, p.1173-1176.
[17]Mitra K, Veeraraghavan A, 2012. Light field denoising, light field superresolution and stereo camera based refocussing using a GMM light field patch prior. Proc IEEE Computer Society Conf on Computer Vision and Pattern Recognition Workshops, p.22-28.
[18]Nava FP, Luke JP, 2009. Simultaneous estimation of super-resolved depth and all-in-focus images from a plenoptic camera. Proc 3DTV Conf: the True Vision—Capture, Transmission and Display of 3D Video, p.1-4.
[19]Ng R, Levoy M, Brédif M, et al., 2005. Light Field Photography with a Hand-Held Plenoptic Camera. Technical Report No. CTSR 2005-02, Stanford University, USA.
[20]Peng JY, Xiong ZW, Liu D, et al., 2018. Unsupervised depth estimation from light field using a convolutional neural network. Proc Int Conf on 3D Vision, p.295-303.
[21]Pérez F, Pérez A, Rodríguez M, et al., 2012. Fourier slice super-resolution in plenoptic cameras. Proc IEEE Int Conf on Computational Photography, p.1-11.
[22]Pérez F, Pérez A, Rodríguez M, et al., 2015. Super-resolved Fourier-slice refocusing in plenoptic cameras. J Math Imag Vis, 52(2):200-217.
[23]Rerabek M, Ebrahimi T, 2016. New light field image dataset. Proc 8th Int Conf on Quality of Multimedia Experience, p.1-2.
[24]Rossi M, Frossard P, 2017. Graph-based light field super-resolution. Proc IEEE 19th Int Workshop on Multimedia Signal Processing, p.1-6.
[25]Rossi M, Frossard P, 2018. Geometry-consistent light field super-resolution via graph-based regularization. IEEE Trans Image Process, 27(9):4207-4218.
[26]Shi WZ, Caballero J, Huszár F, et al., 2016. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. Proc IEEE Conf on Computer Vision and Pattern Recognition, p.1874-1883.
[27]Wang TC, Zhu JY, Hiroaki E, et al., 2016. A 4D light-field dataset and CNN architectures for material recognition. Proc 14th European Conf on Computer Vision, p.121-138.
[28]Wang YL, Hou GQ, Sun ZN, et al., 2016. A simple and robust super resolution method for light field images. Proc IEEE Int Conf on Image Processing, p.1459-1463.
[29]Wang YL, Liu F, Zhang KB, et al., 2018. LFNet: a novel bidirectional recurrent convolutional neural network for light-field image super-resolution. IEEE Trans Image Process, 27(9):4274-4286.
[30]Wang YQ, Wang LG, Yang JG, et al., 2020. Spatial-angular interaction for light field image super-resolution. Proc 16th European Conf on Computer Vision, p.290-308.
[31]Wanner S, Goldluecke B, 2014. Variational light field analysis for disparity estimation and super-resolution. IEEE Trans Patt Anal Mach Intell, 36(3):606-619.
[32]Wanner S, Meister S, Goldluecke B, 2013. Datasets and benchmarks for densely sampled 4D light fields. Proc 18th Int Workshop on Vision, Modeling, and Visualization, p.225-226.
[33]Yeung HWF, Hou JH, Chen XM, et al., 2019. Light field spatial super-resolution using deep efficient spatial-angular separable convolution. IEEE Trans Image Process, 28(5):2319-2330.
[34]Yi P, Wang ZY, Jiang K, et al., 2019. Progressive fusion video super-resolution network via exploiting non-local spatio-temporal correlations. Proc IEEE/CVF Int Conf on Computer Vision, p.3106-3115.
[35]Yoon Y, Jeon HG, Yoo D, et al., 2015. Learning a deep convolutional network for light-field image super-resolution. Proc IEEE Int Conf on Computer Vision Workshop, p.57-65.
[36]Yuan Y, Cao ZQ, Su LJ, 2018. Light-field image superresolution using a combined deep CNN based on EPI. IEEE Signal Process Lett, 25(9):1359-1363.
[37]Yücer K, Sorkine-Hornung A, Wang O, et al., 2016. Efficient 3D object segmentation from densely sampled light fields with applications to 3D reconstruction. ACM Trans Graph, 35(3):22.
[38]Zhang S, Lin YF, Sheng H, 2019. Residual networks for light field image super-resolution. Proc IEEE/CVF Conf on Computer Vision and Pattern Recognition, p.11038-11047.
[39]Zhu H, Wang Q, Yu JY, 2017. Light field imaging: models, calibrations, reconstructions, and applications. Front Inform Technol Electron Eng, 18(9):1236-1249.
Open peer comments: Debate/Discuss/Question/Opinion
<1>