Shadow Detection and Removal

Shadows often appear in images. A shadow appearing on a surface is caused by light that is supposed to hit on the surface but blocked by an occlude. Shadows, in particular the strong ones, may affect the performances of downstream vision tasks, such as object detection. They may also affect the visual quality of some images. Hence, shadow detection and removal tasks are popular low-level vision tasks, and a lot of research has been conducted for these two tasks.

We have conducted a number of projects on shadow detection as well as shadow removal.

Recasting Regional Lighting for Shadow Removal [paper] [suppl] [code] [dataset]

Yuhao Liu, Zhanghan Ke, Ke Xu, Fang Liu, Zhenwei Wang, and Rynson Lau

Proc. AAAI, Feb. 2024

Comparison of shadow removal results. Existing methods (b-e) may fail to completely remove the shadow in the homogenous region and to recover the details in the textured region. Our method explicitly estimates the reflectance (f) and illumination (g) of the shadow image, based on which we recast the lighting and correct the texture in the shadow region, resulting in a more visually pleasing prediction (h).

Abstract. Removing shadows requires an understanding of both lighting conditions and object textures in a scene. Existing methods typically learn pixel-level color mappings between shadow and non-shadow images, in which the joint modeling of lighting and object textures is implicit and inadequate. We observe that in a shadow region, the degradation degree of object textures depends on the local illumination, while simply enhancing the local illumination cannot fully recover the attenuated textures. Based on this observation, we propose to condition the restoration of attenuated textures on the corrected local lighting in the shadow region. Specifically, we first design a shadow-aware decomposition network to estimate the illumination and reflectance layers of shadow regions explicitly. We then propose a novel bilateral correction network to recast the lighting of shadow regions in the illumination layer via a novel local lighting correction module, and to restore the textures conditioned on the corrected illumination layer via a novel illumination-guided texture restoration module. We further annotate pixel-wise shadow masks for the public SRD dataset, which originally contains only image pairs. Experiments on three benchmarks show that our method outperforms existing state-of-the-art shadow removal methods.

Structure-Informed Shadow Removal Networks [paper] [code]

Yuhao Liu, Qing Guo, Lan Fu, Zhanghan Ke, Ke Xu, Wei Feng, Ivor Tsang, and Rynson Lau

IEEE Trans. on Image Processing, 32:5823-5836, October 2023

(a) State-of-the-art shadow removal methods (e.g., AEF [8]) typically learn a direct shadow-to-shadow-free mapping and may often produce shadow remnants with color artifacts. (b) We propose to incorporate image-structure information into the shadow removal process. We visualize the features of approaches (a) and (b) in (c) and (d), respectively, which show that features of (d) are structured according to region homogeneity. (e) Results of original AEF and its structure-enhanced counterpart, where red arrows indicate the region with shadow remnants exist, and RMSE metric are shown for reference.

Abstract. Existing deep learning-based shadow removal methods still produce images with shadow remnants. These shadow remnants typically exist in homogeneous regions with low-intensity values, making them untraceable in the existing image-to-image mapping paradigm. We observe that shadows mainly degrade images at the image-structure level (in which humans perceive object shapes and continuous colors). Hence, in this paper, we propose to remove shadows at the image structure level. Based on this idea, we propose a novel structure-informed shadow removal network (StructNet) to leverage the image-structure information to address the shadow remnant problem. Specifically, StructNet first reconstructs the structure information of the input image without shadows and then uses the restored shadow-free structure prior to guiding the image-level shadow removal. StructNet contains two main novel modules: (1) a mask-guided shadow-free extraction (MSFE) module to extract image structural features in a non-shadow-to-shadow directional manner, and (2) a multi-scale feature & residual aggregation (MFRA) module to leverage the shadow-free structure information to regularize feature consistency. In addition, we also propose to extend StructNet to exploit multi-level structure information (MStructNet), to further boost the shadow removal performance with minimum computational overheads. Extensive experiments on three shadow removal benchmarks demonstrate that our method outperforms existing shadow removal methods, and our StructNet can be integrated with existing methods to improve them further.

Adaptive Illumination Mapping for Shadow Detection in Raw Images [paper] [suppl] [code] [dataset]

Jiayu Sun^*, Ke Xu^*, Youwei Pang, Lihe Zhang, Huchuan Lu, Gerhard Hancke, and Rynson Lau (^* joint first authors)

Proc. IEEE ICCV, Oct 2023

When a non-shadow region has low contrast to a shadow region but high contrast to another non-shadow region (e.g., the black boundary of the object), existing methods may fail to detect the shadow region correctly (b-d). We propose to detect shadows in the raw images. By learning to project raw images into sRGB images of different intensity ranges adaptively, our method can detect the shadow region correctly (g).

Abstract. Shadow detection methods rely on multi-scale contrast, especially global contrast, information to locate shadows correctly. However, we observe that the camera image signal processor (ISP) tends to preserve more local contrast information by sacrificing global contrast information during the raw-to-sRGB conversion process. This often causes existing methods to fail in scenes with high global contrast but low local contrast in shadow regions. In this paper, we propose a novel method to detect shadows from raw images. Our key idea is that instead of performing a many-to-one mapping like the ISP process, we can learn a many-to-many mapping from the high dynamic range raw images to the sRGB images of different illumination, which is able to preserve multi-scale contrast for accurate shadow detection. To this end, we first construct a new shadow dataset with ∼ 7000 raw images and shadow masks. We then propose a novel network, which includes a novel adaptive illumination mapping (AIM) module to project the input raw images into sRGB images of different intensity ranges and a shadow detection module to leverage the preserved multi-scale contrast information to detect shadows. To learn the shadow-aware adaptive illumination mapping process, we propose a novel feedback mechanism to guide the AIM during training. Experiments show that our method outperforms state-of-the-art shadow detectors.

Mitigating Intensity Bias in Shadow Detection via Feature Decomposition and Reweighting [paper] [suppl] [video] [code]

Lei Zhu, Ke Xu, Zhanghan Ke, and Rynson Lau

Proc. IEEE ICCV, Oct. 2021

Intensity bias in shadow detection. Rows 1 and 3 show two original images, while rows 2 and 4 show the two images with 20% increase in intensity. Existing methods [16, 50, 5] heavily rely on the intensity cue, and suffer from two problems. On the one hand, they mis-recognize a relatively brighter region inside the shadow as non-shadow (e.g., yellow lanes in row 1), and dark non-shadow region as shadow (e.g., the traffic cone in row 3). On the other hand, their predictions change significantly due to the brightness change (rows 2 and 4). Our method mitigates this intensity bias and produces more consistent and accurate results.

Abstract. Although CNNs have achieved remarkable progress on the shadow detection task, they tend to make mistakes in dark non-shadow regions and relatively bright shadow regions. They are also susceptible to brightness change. These two phenomenons reveal that deep shadow detectors heavily depend on the intensity cue, which we refer to as intensity bias. In this paper, we propose a novel feature decomposition and reweighting scheme to mitigate this intensity bias, in which multi-level integrated features are decomposed into intensity-variant and intensity-invariant components through self-supervision. By reweighting these two types of features, our method can reallocate the attention to the corresponding latent semantics and achieves balanced exploitation of them. Extensive experiments on three popular datasets show that the proposed method outperforms state-of-the-art shadow detectors.

Distraction-Aware Shadow Detection [paper] [suppl] [code] [trained model] [distraction dataset]

Quanlong Zheng, Xiaotian Qiao, Ying Cao, and Rynson Lau

Proc. IEEE CVPR, pp. 5162-5171, June 2019

Shadow detection with distraction. Existing methods [42, 21, 12] wrongly detect a non-shadow region in the input image that appears like shadow (pointed to by a red arrow in (a) top row) as shadow ((b-d) top row), and also wrongly consider a shadow region that appears like a non-shadow pattern (pointed to by a green arrow in (a) bottom row) as non-shadow ((b-d) bottom row). Our distraction-aware model can detect shadow regions favorably in both cases. Best view in color.

Abstract. Shadow detection is an important and challenging task for scene understanding. Despite promising results from recent deep learning based methods. Existing works still struggle with ambiguous cases where the visual appearances of shadow and non-shadow regions are similar (referred to as distraction in our context). In this paper, we propose a Distraction-aware Shadow Detection Network (DSDNet) by explicitly learning and integrating the semantics of visual distraction regions in an end-to-end framework. At the core of our framework is a novel standalone, differentiable Distraction-aware Shadow (DS) module, which allows us to learn distraction-aware, discriminative features for robust shadow detection, by explicitly predicting false positives and false negatives. We conduct extensive experiments on three public shadow detection datasets, SBU, UCF and ISTD, to evaluate our method. Experimental results demonstrate that our model can boost shadow detection performance, by effectively suppressing the detection of false positives and false negatives, achieving state-of-the-art results.

DeshadowNet: A Multi-context Embedding Deep Network for Shadow Removal (Spotlight) [paper] [code] [train set] [test set]

Liangqiong Qu, Jiandong Tian, Shengfeng He, Yandong Tang, and Rynson Lau

Proc. IEEE CVPR, pp. 2308-2316, July 2017

Comparison with existing shadow removal methods. Existing methods fail to correctly remove the shadow cast on different semantic regions (i.e., horizontal ground and vertical trunk).

Abstract. Shadow removal is a challenging task as it requires the detection/annotation of shadows as well as semantic understanding of the scene. In this paper, we propose an automatic and end-to-end deep neural network (DeshadowNet) to tackle these problems in a unified manner. DeshadowNet is designed with a multi-context architecture, where the output shadow matte is predicted by embedding information from three different perspectives. The first global network extracts shadow features from a global view. Two levels of features are derived from the global network and transferred to two parallel networks. While one extracts the appearance of the input image, the other one involves semantic understanding for final prediction. These two complementary networks generate multi-context features to obtain the shadow matte with fine local details. To evaluate the performance of the proposed method, we construct the first large scale benchmark with 3088 image pairs. Extensive experiments on two publicly available benchmarks and our large-scale benchmark show that the proposed method performs favorably against several state-of-the-art methods.

Last updated in December 2023