Similar articles

Abstract: Light field (LF) imaging has attracted attention because of its ability to solve computer vision problems. In this paper we briefly review the research progress in computer vision in recent years. For most factors that affect computer vision development, the richness and accuracy of visual information acquisition are decisive. LF imaging technology has made great contributions to computer vision because it uses cameras or microlens arrays to record the position and direction information of light rays, acquiring complete three-dimensional (3D) scene information. LF imaging technology improves the accuracy of depth estimation, image segmentation, blending, fusion, and 3D reconstruction. LF has also been innovatively applied to iris and face recognition, identification of materials and fake pedestrians, acquisition of epipolar plane images, shape recovery, and LF microscopy. Here, we further summarize the existing problems and the development trends of LF imaging in computer vision, including the establishment and evaluation of the LF dataset, applications under high dynamic range (HDR) conditions, LF image enhancement, virtual reality, 3D display, and 3D movies, military optical camouflage technology, image recognition at micro-scale, image processing method based on HDR, and the optimal relationship between spatial resolution and four-dimensional (4D) LF information acquisition. LF imaging has achieved great success in various studies. Over the past 25 years, more than 180 publications have reported the capability of LF imaging in solving computer vision problems. We summarize these reports to make it easier for researchers to search the detailed methods for specific solutions.

用于计算机视觉任务的光场成像技术综述

[1]Adelson EH, Bergen JR, 1991. The plenoptic function and the elements of early vision. In: Landy MS, Movshon JA (Eds.), Computational Models of Visual Processing. MIT Press, Cambridge, USA, p.3-20.

[2]Afshari H, Akin A, Popovic V, et al., 2012. Real-time FPGA implementation of linear blending vision reconstruction algorithm using a spherical light field camera. Proc IEEE Workshop on Signal Processing Systems, p.49-54.

[3]Alperovich A, Goldluecke B, 2017. A variational model for intrinsic light field decomposition. Proc 13^th Asian Conf on Computer Vision, p.66-82.

[4]Balogh T, Kovács PT, 2010. Real-time 3D light field transmission. Proc SPIE 7724, Real-Time Image and Video Processing, Article 772406.

[5]Berent J, Dragotti PL, 2007. Segmentation of epipolar-plane image volumes with occlusion and disocclusion competition. Proc IEEE Workshop on Multimedia Signal Processing, p.182-185.

[6]Broxton M, Grosenick L, Yang S, et al., 2013. Wave optics theory and 3-D deconvolution for the light field microscope. Opt Expr, 21(21):25418-25439.

[7]Campbell NDF, Vogiatzis G, Hernández C, et al., 2010. Automatic 3D object segmentation in multiple views using volumetric graph-cuts. Image Vis Comput, 28(1):14-25.

[8]Campbell NDF, Vogiatzis G, Hernandez C, et al., 2011. Automatic object segmentation from calibrated images. Proc Conf for Visual Media Production, p.126-137.

[9]Chen XY, Dai F, Ma YK, et al., 2015. Automatic foreground segmentation using light field images. Proc Visual Communications and Image Processing, p.1-4.

[10]Cheng Z, Xiong ZW, Chen C, et al., 2019. Light field super-resolution: a benchmark. Proc IEEE/CVF Conf on Computer Vision and Pattern Recognition Workshops, p.1804-1813.

[11]Cohen N, Yang S, Andalman A, et al., 2014. Enhancing the performance of the light field microscope using wavefront coding. Opt Expr, 22(20):24817-24839.

[12]Criminisi A, Kang SB, Swaminathan R, et al., 2005. Extracting layers and analyzing their specular properties using epipolar-plane-image analysis. Comput Vis Image Underst, 97(1):51-85.

[13]Cui YL, Yu M, Jiang ZD, et al., 2021. Blind light field image quality assessment by analyzing angular-spatial characteristics. Dig Signal Process, 117:103138.

[14]Fang YM, Wei KK, Hou JH, et al., 2018. Light filed image quality assessment by local and global features of epipolar plane image. Proc IEEE 4^th Int Conf on Multimedia Big Data, p.1-6.

[15]Fiss J, Curless B, Szeliski R, 2014. Refocusing plenoptic images using depth-adaptive splatting. Proc IEEE Int Conf on Computational Photography, p.1-9.

[16]Gao Q, Han L, Shen J, et al., 2017. Focused-region segmentation for light field images based on PCNN. Proc Int Smart Cities Conf, p.1-6.

[17]Georgiev TG, Lumsdaine A, 2010. Focused plenoptic camera and rendering. J Electron Imag, 19(2):021106.

[18]Gershun A, 1939. The light field. J Math Phys, 18(1-4):51-151.

[19]Ghasemi A, Vetterli M, 2014. Detecting planar surface using a light-field camera with application to distinguishing real scenes from printed photos. Proc IEEE Int Conf on Acoustics, Speech and Signal Processing, p.4588-4592.

[20]Gryaditskaya Y, Masia B, Didyk P, et al., 2016. Gloss editing in light fields. Proc Conf on Vision, Modeling and Visualization, p.127-135.

[21]Guo BC, Wen JT, Han YX, 2020. Deep material recognition in light-fields via disentanglement of spatial and angular information. Proc 16^th European Conf on Computer Vision, p.664-679.

[22]Guo XQ, Lin HT, Yu Z, et al., 2015. Barcode imaging using a light field camera. Proc European Conf on Computer Vision, p.519-532.

[23]Hog M, Sabater N, Guillemot C, 2016. Light field segmentation using a ray-based graph structure. Proc 14^th European Conf on Computer Vision, p.35-50.

[24]Hsieh PY, Chou PY, Lin HA, et al., 2018. Long working range light field microscope with fast scanning multifocal liquid crystal microlens array. Opt Expr, 26(8):10981-10996.

[25]Huang ZJ, Yu M, Xu HY, et al., 2018. New quality assessment method for dense light fields. Proc SPIE 10817, Optoelectronic Imaging and Multimedia Technology V, Article 1081717.

[26]Jia C, Shi F, Zhao YF, et al., 2018. Identification of pedestrians from confused planar objects using light field imaging. IEEE Access, 6:39375-39384.

[27]Johannsen O, Sulc A, Goldluecke B, 2015. Variational separation of light field layers. Proc 20^th Int Symp on Vision, Modeling, and Visualization, p.135-142.

[28]Kalantari NK, Wang TC, Ramamoorthi R, 2016. Learning-based view synthesis for light field cameras. ACM Trans Graph, 35(6):193.

[29]Kim C, Zimmer H, Pritch Y, et al., 2013. Scene reconstruction from high spatio-angular resolution light fields. ACM Trans Graph, 32(4):73.

[30]Lee JY, Park RH, 2017. Separation of foreground and background from light field using gradient information. Appl Opt, 56(4):1069-1078.

[31]Levoy M, Hanrahan P, 1996. Light field rendering. Proc 23^rd Annual Conf on Computer Graphics and Interactive Techniques, p.31-42.

[32]Levoy M, Ng R, Adams A, et al., 2006. Light field microscopy. Proc ACM SIGGRAPH, p.924-934.

[33]Li NY, Ye JW, Ji Y, et al., 2014. Saliency detection on light field. Proc IEEE Conf on Computer Vision and Pattern Recognition, p.2806-2813.

[34]Li NY, Sun BL, Yu JY, 2015. A weighted sparse coding framework for saliency detection. Proc IEEE Conf on Computer Vision and Pattern Recognition, p.5216-5223.

[35]Li ZQ, Xu ZX, Ramamoorthi R, et al., 2017. Robust energy minimization for BRDF-invariant shape from light fields. Proc IEEE Conf on Computer Vision and Pattern Recognition, p.578-586.

[36]Liang CK, Lin TH, Wong BY, et al., 2008. Programmable aperture photography: multiplexed light field acquisition. ACM Trans Graph, 27(3):1-10.

[37]Lippmann G, 1908. Epreuves reversibles, photographies integrales. Comput R Acad Sci, 444:446-451.

[38]Lumsdaine A, Georgiev T, 2009. The focused plenoptic camera. Proc IEEE Int Conf on Computational Photography, p.1-8.

[39]Lv XQ, Wang X, Wang Q, et al., 2021. 4D light field segmentation from light field super-pixel hypergraph representation. IEEE Trans Vis Comput Graph, 27(9):3597-3610.

[40]Marquez M, Rueda-Chacon H, Arguello H, 2020. Compressive spectral light field image reconstruction via online tensor representation. IEEE Trans Image Process, 29:3558-3568.

[41]Mehajabin N, Pourazad M, Nasiopoulos P, 2020. SSIM assisted pseudo-sequence-based prediction structure for light field video compression. Proc IEEE Int Conf on Consumer Electronics, p.1-2.

[42]Meng CL, An P, Huang XP, et al., 2019. Objective quality assessment for light field based on refocus characteristic. Proc 10^th Int Conf on Image and Graphics, p.193-204.

[43]Mihara H, Funatomi T, Tanaka K, et al., 2016. 4D light field segmentation with spatial and angular consistencies. Proc IEEE Int Conf on Computational Photography, p.‍1-8.

[44]Murgia F, Giusto D, Perra C, et al., 2015. 3D reconstruction from plenoptic image. Proc 23^rd Telecommunications Forum Telfor, p.448-451.

[45]Ng R, Levoy M, Brédif M, et al., 2005. Light field photography with a hand-held plenoptic camera. Stanford Tech Report CTSR 2005-02.

[46]Nian ZC, Jung C, 2019. CNN-based multi-focus image fusion with light field data. Proc IEEE Int Conf on Image Processing, p.1044-1048.

[47]Paudyal P, Olsson R, Sjöström M, et al., 2016. SMART: a light field image quality dataset. Proc 7^th Int Conf on Multimedia Systems, Article 49.

[48]Paudyal P, Battisti F, Sjöström M, et al., 2017. Towards the perceptual quality evaluation of compressed light field images. IEEE Trans Broadcast, 63(3):507-522.

[49]Paudyal P, Battisti F, Carli M, 2019. Reduced reference quality assessment of light field images. IEEE Trans Broadcast, 65(1):152-165.

[50]Piao YR, Li X, Zhang M, et al., 2019a. Saliency detection via depth-induced cellular automata on light field. IEEE Trans Image Process, 29:1879-1889.

[51]Piao YR, Rong ZK, Zhang M, et al., 2019b. Deep light-field-driven saliency detection from a single view. Proc 28^th Int Joint Conf on Artificial Intelligence, p.904-911.

[52]Piao YR, Jiang YY, Zhang M, et al., 2021. PANet: patch-aware network for light field salient object detection. IEEE Trans Cybern, early access.

[53]Raghavendra R, Busch C, 2014. Presentation attack detection on visible spectrum iris recognition by exploring inherent characteristics of light field camera. Proc IEEE Int Joint Conf on Biometrics, p.1-8.

[54]Raghavendra R, Raja KB, Yang B, et al., 2013a. Combining iris and periocular recognition using light field camera. Proc 2^nd IAPR Asian Conf on Pattern Recognition, p.155-159.

[55]Raghavendra R, Raja KB, Yang B, et al., 2013b. A novel image fusion scheme for robust multiple face recognition with light-field camera. Proc 16^th Int Conf on Information Fusion, p.722-729.

[56]Raghavendra R, Raja KB, Busch C, 2016. Exploring the usefulness of light field cameras for biometrics: an empirical study on face and iris recognition. IEEE Trans Inform Forens Secur, 11(5):922-936.

[57]Rerabek M, Ebrahimi T, 2016. New light field image dataset. Proc 8^th Int Conf on Quality of Multimedia Experience.

[58]Sabater N, Boisson G, Vandame B, et al., 2017. Dataset and pipeline for multi-view light-field video. Proc IEEE Conf on Computer Vision and Pattern Recognition Workshops, p.1743-1753.

[59]Sepas-Moghaddam A, Pereira F, Correia PL, 2018. Light field-based face presentation attack detection: reviewing, benchmarking and one step further. IEEE Trans Inform Forens Secur, 13(7):1696-1709.

[60]Shan L, An P, Meng CL, et al., 2019. A no-reference image quality assessment metric by multiple characteristics of light field images. IEEE Access, 7:127217-127229.

[61]Sheng H, Deng SY, Zhang S, et al., 2016. Segmentation of light field image with the structure tensor. Proc IEEE Int Conf on Image Processing, p.1469-1473.

[62]Shi LK, Zhao SY, Zhou W, et al., 2018. Perceptual evaluation of light field image. Proc 25^th IEEE Int Conf on Image Processing, p.41-45.

[63]Shi LK, Zhao SY, Chen ZB, 2019. Belif: blind quality evaluator of light field image with tensor structure variation index. Proc IEEE Int Conf on Image Processing, p.3781-3785.

[64]Shi LK, Zhou W, Chen ZB, et al., 2020. No-reference light field image quality assessment based on spatial-angular measurement. IEEE Trans Circ Syst Video Technol, 30(11):4114-4128.

[65]Smith BM, Zhang L, Jin HL, et al., 2009. Light field video stabilization. Proc IEEE 12^th Int Conf on Computer Vision, p.341-348.

[66]Sulc A, Alperovich A, Marniok N, et al., 2016. Reflection separation in light fields based on sparse coding and specular flow. Proc Conf on Vision, Modeling and Visualization, p.137-144.

[67]Sun J, Hossain M, Xu CL, et al., 2017. A novel calibration method of focused light field camera for 3-D reconstruction of flame temperature. Opt Commun, 390:7-15.

[68]Tambe S, Veeraraghavan A, Agrawal A, 2013. Towards motion aware light field video for dynamic scenes. Proc IEEE Int Conf on Computer Vision, p.1009-1016.

[69]Tao MW, Srinivasan PP, Malik J, et al., 2015a. Depth from shading, defocus, and correspondence using light-field angular coherence. Proc IEEE Conf on Computer Vision and Pattern Recognition, p.1940-1948.

[70]Tao MW, Su JC, Wang TC, et al., 2015b. Depth estimation and specular removal for glossy surfaces using point and line consistency with light-field cameras. IEEE Trans Patt Anal Mach Intell, 38(6):1155-1169.

[71]Tian Y, Zeng HQ, Hou JH, et al., 2021. A light field image quality assessment model based on symmetry and depth features. IEEE Trans Circ Syst Video Technol, 31(5):2046-2050.

[72]Vizcaíno JP, Saltarin F, Belyaev Y, et al., 2021. Learning to reconstruct confocal microscopy stacks from single light field images. IEEE Trans Comput Imag, 7:775-788.

[73]Wang AZ, Wang MH, Li XY, et al., 2017. A two-stage Bayesian integration framework for salient object detection on light field. Neur Process Lett, 46(3):1083-‍1094.

[74]Wang HQ, Xu CX, Wang XZ, et al., 2016. Light field imaging based accurate image specular highlight removal. PLoS ONE, 11(6):e0156173.

[75]Wang TC, Efros AA, Ramamoorthi R, 2015. Occlusion-aware depth estimation using light-field cameras. Proc IEEE Int Conf on Computer Vision, p.3487-3495.

[76]Wang TC, Zhu JY, Hiroaki E, et al., 2016a. A 4D light-field dataset and CNN architectures for material recognition. Proc 14^th European Conf on Computer Vision, p.121-138.

[77]Wang TC, Chandraker M, Efros AA, et al., 2016b. SVBRDF-invariant shape and reflectance estimation from light-field cameras. Proc IEEE Conf on Computer Vision and Pattern Recognition, p.5451-5459.

[78]Wang TC, Zhu JY, Kalantari NK, et al., 2017. Light field video capture using a learning-based hybrid imaging system. ACM Trans Graph, 36(4):133.

[79]Wang TT, Piao YR, Li XC, et al., 2019. Deep learning for light field saliency detection. Proc IEEE/CVF Int Conf on Computer Vision, p.8837-8847.

[80]Wang YQ, Yang JG, Xiao C, et al., 2018. An efficient method for the fusion of light field refocused images. Proc SPIE 9^th Int Conf on Graphic and Image Processing, Article 1061536.

[81]Wanner S, Meister S, Goldluecke B, 2013a. Datasets and benchmarks for densely sampled 4D light fields. Proc 18^th Int Workshop on Vision, Modeling, and Visualization, p.225-226.

[82]Wanner S, Straehle C, Goldluecke B, 2013b. Globally consistent multi-label assignment on the ray space of 4D light fields. Proc IEEE Conf on Computer Vision and Pattern Recognition, p.1011-1018.

[83]Wilburn B, Smulski M, Lee HHK, et al., 2002. Light field video camera. Proc SPIE 6474, Media Processors, p.29-36.

[84]Wilburn B, Joshi N, Vaish V, et al., 2005. High performance imaging using large camera arrays. ACM Trans Graph, 24(3):765-776.

[85]Wu GC, Masia B, Jarabo A, et al., 2017. Light field image processing: an overview. IEEE J Sel Top Signal Process, 11(7):926-954.

[86]Xu YC, Nagahara H, Shimada A, et al., 2015. TransCut: transparent object segmentation from a light-field image. Proc IEEE Int Conf on Computer Vision, p.3442-3450.

[87]Xu YC, Nagahara H, Shimada A, et al., 2019. TransCut2: transparent object segmentation from a light-field image. IEEE Trans Comput Imag, 5(3):465-477.

[88]Yang JC, 2000. A Light Field Camera for Image Based Rendering. MS Thesis, Massachusetts Institute of Technology, Cambridge, USA.

[89]Yücer K, Sorkine-Hornung A, Wang O, et al., 2016. Efficient 3D object segmentation from densely sampled light fields with applications to 3D reconstruction. ACM Trans Graph, 35(3):22.

[90]Zhang C, Chen T, 2004. A self-reconfigurable camera array. Proc ACM SIGGRAPH Sketches, p.151.

[91]Zhang C, Hou GQ, Sun ZA, et al., 2013. Light field photography for iris image acquisition. Proc 8^th Chinese Conf on Biometric Recognition, p.345-352.

[92]Zhang J, Wang M, Gao J, et al., 2015. Saliency detection with a deeper investigation of light field. Proc 24^th Int Joint Conf on Artificial Intelligence, p.2212-2218.

[93]Zhang J, Wang M, Lin L, et al., 2017. Saliency detection on light field: a multi-cue approach. ACM Trans Multim Comput Commun Appl, 13(3):32.

[94]Zhang J, Liu YM, Zhang SP, et al., 2020. Light field saliency detection with deep convolutional networks. IEEE Trans Image Process, 29:4421-4434.

[95]Zhang M, Geng Z, Pei RJ, et al., 2017. Three-dimensional light field microscope based on a lenslet array. Opt Commun, 403:133-142.

[96]Zhang M, Li JJ, Wei J, et al., 2019. Memory-oriented decoder for light field salient object detection. Proc Advances in Neural Information Processing Systems32, p.2898-2909.

[97]Zhang XD, Wang Y, Zhang J, et al., 2015. Light field saliency vs. 2D saliency: a comparative study. Neurocomputing, 166:389-396.

[98]Zhou MY, Ding YQ, J Yiet al., 2020. Shape and reflectance reconstruction using concentric multi-spectral light field. IEEE Trans Patt Anal Mach Intell, 42(7):‍1594-1605.

[99]Zhou W, Shi LK, Chen ZB, et al., 2020. Tensor oriented no-reference light field image quality assessment. IEEE Trans Image Process, 29:4070-4084.

[100]Zhu H, Zhang Q, Wang Q, 2017. 4D light field superpixel and segmentation. Proc IEEE Conf on Computer Vision and Pattern Recognition, p.6709-6717.