S3: A Spectral and Spatial Measure of Local Perceived Sharpness in Natural Images
Cuong T. Vu, Thien D. Phan, and Damon M. Chandler
This webpage serves as the online supplement of the paper "S3 A Spectral and Spatial Measure of Local Perceived Sharpness in Natural Images", IEEE Transaction on Image Processing, 21 (3), September 2011.
S3 is a block-based algorithm which estimates the perceived sharpness of local image regions, and which does not require the presence of edges. Our measure is based on two factors: (1) a spectral measure based on the slope of the local magnitude spectrum, (2) a spatial measure based on local maximum total variation.
The S3 estimator can yield a local sharpness map in which greater values correspond to greater perceived sharpness within an image and across different images. To the best of our knowledge, there is no other existing blur/sharpness algorithm which was designed to generate a map. This type of map can be very useful for applications such as main subject detection and spatially adaptive processing. This S3 map can also be collapsed into a single scalar value which denotes overall perceived sharpness for a full-sized image.
The following supplementary results and analysis of the S3 algorithm are included on this page:
- Subjective rankings of sharpness
- Local sharpness maps
- No-reference quality assessment of JPEG2000 compressed images
- Analysis of parameters τ1, τ2, and η
- Details of the contrast computation for Figure 2 in the paper
- Monotonic prediction of blur parameter
- Download S3 code and sharpness maps database.
|Subjective rankings of sharpness|
We first demonstrate, via representative results, that S3 is able to accurately perform across-image and within-image sharpness prediction. Figure 1 depicts S3 maps and indices for a variety of images containing commonplace subject matter.
The images in Figure 1 have been ordered according to increasing overall perceived sharpness determined by asking eight naive subjects to rank-order the images in terms of sharpness. The interface of the experiment is shown in Figure 2.
In this experiment, the subjects were asked to rank the six images in terms of overall sharpness. The screen used was a LaCie 324 24-inch LCD monitor (1920x1200 at 60 Hz). The display yielded minimum and maximum luminances of 0.80 and 259 cd/m2, respectively, with a luminance gamma of γ = 2.2. Stimuli were viewed binocularly through natural pupils in a darkened room at a distance of approximately 60 cm. The six images yielded 15 pairs, and each subject took approximately five minutes to finish comparing all the pairs. The order of images shown in the Figure 1 was obtained by averaging the ranks from all subjects.
In terms of across-image prediction, comparing the S3 indices to the rank ordering, S3 is generally able to perform across-image sharpness prediction accurately. In terms of within-image sharpness prediction, all of the S3 maps quite accurately capture the sharp regions in each image.
|Local sharpness maps|
The interface of the experiment to generate ground-truth sharpness map (as described in Section IV-D in the manuscript) is shown the Figure 3. Two versions of the same image were displayed against a mid-gray background. The left image was divided into blocks of size 16x16 (shown by the grid) and the same image (without the grid) was displayed on the right-hand side for reference. Subjects rated the sharpness value for each block by using an integer scale from 1 to 3, where 1 denoted that the block was very sharp, 3 denoted that the block was not sharp, and 2 was anything in-between. The testing interface contained options to assign a sharpness level to multiple blocks and undo an assignment, which facilitated the testing.
Performances: In addition to Figure 12 in the manuscript, Figure 4 shows maps of all algorithms in comparison for all six images. We also include in this figure maps from the BLIINDS-II algorithm, which has been run on each 32x32 block with 24 pixels of overlap. In general, the S3 algorithm creates the best estimation of the ground truth maps.
Table I below shows the performance of all seven algorithms using three criteria: CC, SROCC and KLD (Kullback-Leibler Divergence). Again, this table shows that S3 holds the best performance in general. Note that this is not quite a fair comparison since the other algorithms were not designed to generate a sharpness map. There may certainly be more optimal techniques of applying these estimators locally rather than via the block-based fashion employed here.
|No-reference quality assessment of JPEG2000-compressed images|
We discuss in this section the performance of S3 on no-reference image quality assessment for JPEG2000-compressed images. One might argue that a sharpness/blurriness estimator can also be used for no-reference quality assessment of JPEG2000 images since JPEG2000 compression can induce blurring. To analyze the performance of S3 on this task, we used the JPEG2000 image subsets from LIVE, CSIQ, and TID2008, which contain 169, 150, and 96 JPEG2000 distorted images, respectively. For comparison, the same sets of images were analyzed by using the following sharpness algorithms: (1) JNB; (2) CPBD; (3) MMZ; (4) ST; (5) MDWE; and (6) BLIINDS-II. The results of this analysis are shown in Table II.
In comparison to JNB, ST, MMZ, and MDWE, except for images from the LIVE database, the performance of S3 is quite competitive. However, the CPBD and BLIINDS-II algorithm clearly outperform others. We believe this limitation of S3 is due to the fact that JPEG2000 compression does not generate blurring uniformly across an image. A highly compressed JPEG2000 image might still contain relatively sharp regions which yield a high S3 index because of our computation of the index in Equation (11) in the manuscript. On the other hand, the artifacts from JPEG2000 compression might ruin edges in the image, thus other sharpness/blur measures which rely on the appearance of edges in the image might give more veridical results for those images. Figure 5 demonstrates this argument. Even though most regions of the image have been blurred by JPEG2000 compression, some regions still appear quite sharp (such as the buildings in the middle of the image). Notice that the corresponding S3 map both correctly detects the blurred regions and is able to capture the sharp regions. The S3 index of this image, computed as the average of the maximum 1% pixel values in the S3 map is therefore fairly high.
|Analysis of parameters τ1, τ2, and η|
In Eq. (6) in the manuscript, the two parameters τ1, τ2 control the shape of the sigmoid transducer function (see Fig. 5 in the manuscript). Here we analyze the performance of S3 with different selections of τ1 and τ2. The corresponding sigmoid shapes are shown below in Figure 6.
Table III shows the performance of S3 on blurred images of TID2008, LIVE, and CSIQ databases with different selections of the pair (τ1, τ2) [see Equation (6) in the paper]. The best performance for each criterion on each database is highlighted. The selection of [τ1, τ2] = [-3, 2] was not chosen based on this analysis; rather it was determined empirically by fitting a sigmoid to subjective ratings of sharpness (using the authors as subjects) for the noise images shown in Figure 5 in the paper. Indeed, this pair gives the best CC only on the TID2008 and the best SROCC only on the LIVE database.
With the chosen constants τ1 and τ2, [-3 2], different values of the parameter η (see Equation (10) in the paper) were also tested to see the influence of this parameter. Table IV shows the performance of S3 when η = 1/4, 1/3, 1/2, 2/3, and 3/4. When η = 1/2, S3 gives the best CCs on TID2008 and LIVE, and SROCC on LIVE. Note that η = 1/2 was not chosen based on this analysis; rather, because S1 and S2 were designed to compensate for each other’s failures, the selection of η = 1/2 is a reasonable choice.
|Details of the contrast computation for Figure 2 in the paper|
While the slope of the magnitude spectrum can be a potential measure of sharpness, it does not take into account contrast, which is known to affect perceived sharpness. Two images can have the same spectral slope but appear to be of different sharpness due to a difference in contrast. Figure 7 demonstrates this assertion. Image (b) was generated from the image (a) by taking 40% of the value of every pixel in (a). This image (b) is then added a constant to make image (c) which has the same mean luminance as image (a). The magnitude spectrums of image (b) and (c) (ignoring the DC component) are therefore the same, and proportional to that of image (a). Thus the three images have the same slope factor. However, the image in (a) appears much sharper than (b) and (c) due to the low contrast of the latter two.
In this section, we computationally show the validity of this demonstration using Weber contrast, Michelson luminance contrast, and RMS luminance contrast.
Weber contrast is defined as: C = (Lmax – Lmin)/ Lmin
Michelson contrast (Peak-to-Peak contrast) is defined as: C = (Lmax – Lmin)/ (Lmax + Lmin)
RMS contrast is defined as: C = σL/μL in which the conversion to luminance L from 8-bit pixel value image I is given by: L = (a + bI)γ
Lmax, Lmin, σL, and μL are the maximum, minimum, standard deviation, and the mean of L, respectively.
Assuming the Adobe RGB color space, a = 0.7656; b = 0.0364 and γ = 2.2, the luminance contrast of each image in the demonstration above is given as:
These results support the validity of the demonstration. However, notice from these results that image (c), which is currently used in Figure 2 in the manuscript, better demonstrates the limitation of the slope of magnitude spectrum than image (b).
|Monotonic prediction of blur parameter|
Figure 8 repeats Figure 10 in the manuscript and adds the curve from BLIINDS-II. As can be seen from this Figure, BLIINDS-II shows several failure cases in this test. However, note that this behavior is expected given that BLIINDS-II was designed for quality assessment rather than sharpness estimation.
As discussed in the last paragraph of Section IV-C, notice from the curves of S3 that for σ ≥ 1.6, the S3 indices are close to zero. This fact agrees with a visual examination of these images; visually, the overall sharpnesses of images blurred with σ ≥ 1.6 are all very low and significantly lower than images blurred with σ ≤ 1.2. Examples of these images are shown in Figure 9 (click on the thumbnails for full-sized versions).
Figure 10 depicts graphs of the subjective ratings of DMOS (on the vertical axis; MOS for TID) plotted against S3 transformed outputs (on the horizontal axis). These graphs demonstrate that S3 performs quite well on very blurred images (high DMOS values for CSIQ and LIVE database, and low MOS values for TID database).
|Download S3 code and sharpness maps database.|