Tutorial on

Deep Learning for Light Fields

IEEE Int'l Conf. on Visual Communications and Image Processing (VCIP)

Dec 13 – 16, 2022

Speaker: Dr. Junhui Hou

Department of Computer Science, City University of Hong Kong, Hong Kong SAR

Email: jh.hou@cityu.edu.hk

Brief Description

The light field describes the radiance of the light rays permeating the 3D free space as a function of their positions and directions. The light field image can be interpreted as a series of 2D images observed from different viewpoints, which implicitly encodes the depth information of the 3D scene. A high-quality 4D light field image records rich information of the scene in both appearance and geometry, and thus, enables worldwide applications in the fields of computer graphics and computer vision, such as novel view rendering, post-capture refocusing, scene reconstruction, and virtual/augmented reality. However, the high dimensionality of light field data also raises great challenges for the acquisition and processing compared with conventional 2D images. Therefore, recent researchers have taken advantage of the advanced deep learning techniques to explore the intrinsic characteristics of light field data.


This tutorial will introduce the basic knowledge of the light field and then focus on deep-learning-based light field reconstruction and processing algorithms. We will start with theoretical descriptions of the light field function and its basic applications, including light field rendering and post-capture refocusing, followed by the introduction of typical light field acquisition approaches, including multi-sensor, multi-exposure, and multiplexing capture. Next, we will comprehensively overview computational approaches to reconstruct the high-quality 4D light field image from low-cost inputs that are sparsely sampled in spatial or angular domains. Finally, we will introduce important techniques of light field processing, including depth estimation and compression, which are necessary intermediate steps for subsequent light field-based applications.

Outline

Background

Plenoptic function

4D light field organization and capture

4D light field representations

Light field applications

Light Field Depth Estimation

Non-learning-based methods

Learning-based methods

Supervised-based

Unsupervised-based

Summary

Light Field Reconstruction

Dense light field reconstruction from sparse light fields

Light field spatial super-resolution

Coded aperture-based light field reconstruction

Summary

Light Field Compression

Video coding-based methods

Optimization-based methods

View reconstruction-based methods

Summary

Conclusion

About the Speaker


Junhui Hou (Senior Member) is an Assistant Professor with the Department of Computer Science, City University of Hong Kong, since Jan. 2017. He received the B.Eng. degree in information engineering (Talented Students Program) from the South China University of Technology, Guangzhou, China, in 2009, the M.Eng. degree in signal and information processing from Northwestern Polytechnical University, Xian, China, in 2012, and the Ph.D. degree in electrical and electronic engineering from the School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore, in 2016. His research interests fall into the general areas of multimedia signal processing, such as image/video/3D geometry data representation, processing and analysis, semi/un-supervised data modeling, and data compression.


He received the Chinese Government Award for Outstanding Students Study Abroad from China Scholarship Council in 2015 and the Early Career Award (3/381) from the Hong Kong Research Grants Council in 2018. He is an elected member of MSA-TC and VSPC-TC, IEEE CAS, and MMSP-TC, IEEE SPS. He is currently an Associate Editor for IEEE Transactions on Image Processing, IEEE Transactions on Circuits and Systems for Video Technology, Signal Processing: Image Communication, and The Visual Computer. He also served as the Guest Editor for the IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing and as an Area Chair of ACM MM’19/20/21/22, IEEE ICME’20, VCIP’20/21, and WACV’21.



Acknowledgments

This tutorial is mainly composed of the outcomes of the research projects supported by the Hong Kong Research Grants Council under grants 11218121 and 21211518, and the Basic Research General Program of Shenzhen Municipality under grant JCYJ20190808183003968. This tutorial has been made possible with the help of my students, Dr. Jing Jin, Mr. Mantang Guo, and Mr. Xianqiang Lyu.



Expected Audiences

Graduate students, engineers, and researchers working on image processing, computational photography, computer vision, deep learning, immersive communication, etc.



Relevant Recent Publications

1. Jing Jin and Junhui Hou, Occlusion-aware Unsupervised Learning of Depth from 4-D Light Fields, IEEE Transactions on Image Processing (IEEE T-IP), vol. 31, pp. 2216 - 2228, 2022.

2. Mantang Guo, Jing Jin, Hui Liu, and Junhui Hou, Learning Dynamic Interpolation for Extremely Sparse Light Fields with Wide Baselines, IEEE/CVF International Conference on Computer Vision (ICCV), 2021, pp. 2450-2459.

3. Xianqiang Lyu, Zhiyu Zhu, Mantang Guo, Jing Jin, Junhui Hou, and Huanqiang Zeng, Learning Spatial-angular Fusion for Compressive Light Field Imaging in a Cycle-consistent Framework, ACM International Conference on Multimedia (ACM MM), 2021, pp. 4613-4621 (Oral presentation)

4. Yuchen Zhang, Wenrui Dai, Yong Li, Chenglin Li, Junhui Hou, Junni Zou, and Hongkai Xiong, Light Field Compression with Graph Learning and Dictionary-Guided Sparse Coding, IEEE Transactions on Multimedia (IEEE T-MM), 2022, DOI: 10.1109/TMM.2022.3154928

5. Mantang Guo, Junhui Hou, Jing Jin, Jie Chen, and Lap-Pui Chau, Deep Spatial-angular Regularization for Light Field Imaging, Denoising, and Super-resolution, IEEE Transactions on Pattern Analysis and Machine Intelligence (IEEE T-PAMI), 2021, DOI: 10.1109/TPAMI.2021.3087485.

6. Jing Jin, Junhui Hou, Jie Chen, Kwong Sam, and Jingyi Yu, Light Field Super-resolution via Attention-Guided Fusion of Hybrid Lenses, ACM International Conference on Multimedia (ACM MM), 2020, pp. 193-201. (Oral presentation)

7. Mantang Guo, Junhui Hou, Jing Jin, Jie Chen, and Lap-Pui Chau, Deep Spatial-Angular Regularization for Compressive Light Field Reconstruction over Coded Apertures, in Proc. European Conference on Computer Vision (ECCV), 2020, pp. 278-294. (Oral presentation, top 2%)

8. Jing Jin, Junhui Hou, Jie Chen, and Sam Kwong, Light Field Spatial Super-resolution via Deep Combinatorial Geometry Embedding and Structural Consistency Regularization, in Proc. IEEE/CVF International Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 2260-2269.

9. Jing Jin, Junhui Hou, Hui Yuan, and Sam Kwong, Learning Light Field Angular Superresolution via a Geometry-aware Network, in Proc. AAAI Conference on Artificial Intelligence (AAAI), 2020, pp. 11141-11148. (Oral presentation)

10. Jing Jin, Junhui Hou, Jie Chen, Huanqiang Zeng, Sam Kwong, and Jingyi Yu, Deep Coarse-to-fine Dense Light Field Reconstruction with Flexible Sampling and Geometry-aware Fusion, IEEE Transactions on Pattern Analysis and Machine Intelligence (IEEE T-PAMI), vol. 44, no. 4, pp. 1819 - 1836, 2022.

11. Chunle Guo, Jing Jin, Junhui Hou, and Jie Chen, Light Field Depth Estimation via an Occlusion-Aware Network, in Proc. IEEE International Conference on Multimedia & Expo (IEEE ICME), 2020, pp. 1-6. (Oral presentation)

12. Jie Chen, Lap-Pui Chau, and Junhui Hou, Surface Consistent Light Field Extrapolation over Stratified Disparity and Spatial Granularities, IEEE International Conference on Multimedia & Expo (IEEE ICME), 2020, pp. 1-6. (Oral presentation)

13. Yu Tian, Huanqiang Zeng, Junhui Hou, Jing Chen, and Kai-Kuang Ma, Light Field Image Quality Assessment via the Light Field Coherence, IEEE Transactions on Image Processing (IEEE T-IP), vol. 29, pp. 7945-7956, 2020.

14. Yu Tian, Huanqiang Zeng, Junhui Hou, et al., A Light Field Image Quality Assessment Model Based on Symmetry and Depth Features, IEEE Transactions on Circuits and Systems for Video Technology (IEEE T-CSVT), vol. 31, no. 5, pp. 2046-2050, 2021.

15. Henry Yeung*, Junhui Hou*, Xiaoming Chen, Jie Chen, Zhibo Chen, and Yuk Ying Chung, Light Field Spatial Super-Resolution Using Deep Efficient Spatial-Angular Separable Convolution, IEEE Transactions on Image Processing (IEEE T-IP), vol. 28, no. 5, pp. 2319-2330, 2019. (*joint 1st author)

16. Junhui Hou, Jie Chen, and Lap-Pui Chau, Light Field Image Compression based on Bi-Level View Compensation with Rate Distortion Optimization, IEEE Transactions on Circuits and Systems for Video Technology (IEEE T-CSVT), vol. 29, no. 2, pp. 517-530, 2019.

17. Henry Yeung*, Junhui Hou*, Jie Chen, Yuk Ying Chung, and Xiaoming Chen, Fast Light Field Reconstruction with Deep Coarse-to-Fine Modeling of Spatial-Angular Clues, in Proc. European Conference on Computer Vision (ECCV), 2018, pp. 138-154. (*joint 1st author)

18. Jie Chen, Junhui Hou, and Lap-Pui Chau, Light Field Compression with Disparity Guided Sparse Coding based on Structural Key Views, IEEE Transactions on Image Processing (IEEE T-IP), vol. 27, no. 1, pp. 414-324, 2018.

19. Jie Chen, Junhui Hou, Yun Ni, and Lap-Pui Chau, Accurate Light Field Depth Estimation with Superpixel Regularization over Partially Occluded Regions, IEEE Transactions on Image Processing (IEEE T-IP), vol. 27, no. 10, pp. 4889-4900, 2018.

20. Jie Chen, Junhui Hou, and Lap-Pui Chau, Light Field Denoising via Anisotropic Parallax Analysis in a CNN Framework, IEEE Signal Processing Letters (IEEE SPL), vol. 25, no. 9, pp. 1403-1407, 2018.

21. Jing Jin, Junhui Hou, Zhiyu Zhu, Jie Chen, and Sam Kwong, Deep Selective Combinatorial Embedding and Consistency Regularization for Light Field Super-resolution, https://arxiv.org/abs/2009.12537, 2021

22. Jing Jin, Hui Liu, Junhui Hou, and Hongkai Xiong, Light Field Reconstruction via Deep Adaptive Fusion of Hybrid Lenses, https://arxiv.org/abs/2102.07085, 2021

23. Mantang Guo, Jing Jin, Hui Liu, Junhui Hou, Huanqiang Zeng, and Jiwen Lu, Content-aware Warping for View Synthesis, https://arxiv.org/abs/2201.09023, 2022