Online Supplement

S3: A Spectral and Spatial Measure of Local Perceived Sharpness in Natural Images

Cuong T. Vu, Thien D. Phan, and Damon M. Chandler

This webpage serves as the online supplement of the paper "S3 A Spectral and Spatial Measure of Local Perceived Sharpness in Natural Images", IEEE Transaction on Image Processing, 21 (3), September 2011.

S3 is a block-based algorithm which estimates the perceived sharpness of local image regions, and which does not require the presence of edges. Our measure is based on two factors: (1) a spectral measure based on the slope of the local magnitude spectrum, (2) a spatial measure based on local maximum total variation.

The S3 estimator can yield a local sharpness map in which greater values correspond to greater perceived sharpness within an image and across different images. To the best of our knowledge, there is no other existing blur/sharpness algorithm which was designed to generate a map. This type of map can be very useful for applications such as main subject detection and spatially adaptive processing. This S3 map can also be collapsed into a single scalar value which denotes overall perceived sharpness for a full-sized image.


The following supplementary results and analysis of the S3 algorithm are included on this page:

- Subjective rankings of sharpness
- Local sharpness maps
- No-reference quality assessment of JPEG2000 compressed images
- Analysis of parameters τ1, τ2, and η
- Details of the contrast computation for Figure 2 in the paper
- Monotonic prediction of blur parameter
- Download S3 code and sharpness maps database.


Subjective rankings of sharpness

We first demonstrate, via representative results, that S3 is able to accurately perform across-image and within-image sharpness prediction. Figure 1 depicts S3 maps and indices for a variety of images containing commonplace subject matter.

(1) Image ball, S3 = 0.384
(2) Image airplane, S3 = 0.509
(3) Image petal, S3 = 0.610
(4) Image zebra, S3 = 0.662
(5) Image pelicans, S3 = 0.760
(6) Image branches, S3 = 0.849
Fig. 1. S3 maps and indices for a variety of images containing commonplace subject matter. The ordering of these images was determined based on subjective rankings of sharpness: (1) denotes the image ranked to be least sharp; (6) denotes the image ranked to be the most sharp.

The images in Figure 1 have been ordered according to increasing overall perceived sharpness determined by asking eight naive subjects to rank-order the images in terms of sharpness. The interface of the experiment is shown in Figure 2.

Fig. 2. The interface of the rank-ordering experiment.

In this experiment, the subjects were asked to rank the six images in terms of overall sharpness. The screen used was a LaCie 324 24-inch LCD monitor (1920x1200 at 60 Hz). The display yielded minimum and maximum luminances of 0.80 and 259 cd/m2, respectively, with a luminance gamma of γ = 2.2. Stimuli were viewed binocularly through natural pupils in a darkened room at a distance of approximately 60 cm. The six images yielded 15 pairs, and each subject took approximately five minutes to finish comparing all the pairs. The order of images shown in the Figure 1 was obtained by averaging the ranks from all subjects.

In terms of across-image prediction, comparing the S3 indices to the rank ordering, S3 is generally able to perform across-image sharpness prediction accurately. In terms of within-image sharpness prediction, all of the S3 maps quite accurately capture the sharp regions in each image.

  1. In the image ball, ball is blurred, which is correctly shown as dark in the map. The map is also able to point out that, in the background, the right vertical line is sharper than the left line.

  2. The sharpest regions in the image airplane are the part of the mountain at the bottom of the image and the plane’s body. This fact is correctly captured in the corresponding S3 map where these regions appear brightest. On the other hand, the snow region, which is apparently blurry, correctly receives the smallest sharpness value.

  3. In the image petal, the flower’s stamens are the sharpest regions, which is accurately noted in its corresponding S3 map. The second sharpest region in this image occurs along the borders of the flower’s petals, particularly, the lower petals. This fact is also captured in the map: The borders of the upper petals receive a low sharpness value, the lower petals receive a higher sharpness value, and the stamens receive the greatest sharpness value.

  4. The perceived sharpness of the image zebra is due to the high contrast stripes of the two zebras. This fact is accurately shown in the sharpness map. Although most of the background is blurred, which correctly appears dark in the map, the map is also able to capture the sharp bushes between the two zebras.

  5. In the image pelicans, the sharpest region includes the pelicans and bushes in the middle of the image. The regions in the background, which include the building and the sky, are very blurred, and the water is increasingly blurred toward the bottom of the image. Accordingly, the S3 map indicates the greatest sharpness toward the center of the image (corresponding to the pelicans and bushes) and decreasing sharpness away from the center.

  6. The image branches includes two regions: the very sharp branches and very smooth sky. Again, as can be seen from the S3 map, the branches receive a very high sharpness value while the sky receives a very low sharpness value.

Go to top

Local sharpness maps
 
Fig. 3. The interface of the experiment to generate ground-truth sharpness map.

The interface of the experiment to generate ground-truth sharpness map (as described in Section IV-D in the manuscript) is shown the Figure 3. Two versions of the same image were displayed against a mid-gray background. The left image was divided into blocks of size 16x16 (shown by the grid) and the same image (without the grid) was displayed on the right-hand side for reference. Subjects rated the sharpness value for each block by using an integer scale from 1 to 3, where 1 denoted that the block was very sharp, 3 denoted that the block was not sharp, and 2 was anything in-between. The testing interface contained options to assign a sharpness level to multiple blocks and undo an assignment, which facilitated the testing.

Image dragon
Input image Ground truth sharpness map
Inter-subject std map CPBD sharpness map
ST sharpness map MMZ sharpness map
MDWE sharpness map S3 sharpness map
BLINDS-II sharpness map JNB sharpness map

 

       
Image flower
Input image Ground truth sharpness map
Inter-subject std map CPBD sharpness map
ST sharpness map MMZ sharpness map
MDWE sharpness map S3 sharpness map
BLINDS-II sharpness map JNB sharpness map

 

Image monkey

Input image Ground truth sharpness map
Inter-subject std map CPBD sharpness map
ST sharpness map MMZ sharpness map
MDWE sharpness map S3 sharpness map
BLINDS-II sharpness map JNB sharpness map

 

       

Image orchid

Input image Ground truth sharpness map
Inter-subject std map CPBD sharpness map
ST sharpness map MMZ sharpness map
MDWE sharpness map S3 sharpness map
BLINDS-II sharpness map JNB sharpness map

 

Image peak
Input image Ground truth sharpness map
Inter-subject std map CPBD sharpness map
ST sharpness map MMZ sharpness map
MDWE sharpness map S3 sharpness map
BLINDS-II sharpness map JNB sharpness map
       
Image squirrel
Input image Ground truth sharpness map
Inter-subject std map CPBD sharpness map
ST sharpness map MMZ sharpness map
MDWE sharpness map S3 sharpness map
BLINDS-II sharpness map JNB sharpness map
Fig. 4. Six original images, their corresponding average subjective sharpness maps, the inter-subject standard deviation maps, and the maps generated by different algorithms. (Black blocks in the MDWE maps are blocks on which MDWE failed to run).

Performances: In addition to Figure 12 in the manuscript, Figure 4 shows maps of all algorithms in comparison for all six images. We also include in this figure maps from the BLIINDS-II algorithm, which has been run on each 32x32 block with 24 pixels of overlap. In general, the S3 algorithm creates the best estimation of the ground truth maps.

Table I below shows the performance of all seven algorithms using three criteria: CC, SROCC and KLD (Kullback-Leibler Divergence). Again, this table shows that S3 holds the best performance in general. Note that this is not quite a fair comparison since the other algorithms were not designed to generate a sharpness map. There may certainly be more optimal techniques of applying these estimators locally rather than via the block-based fashion employed here.

  JNB CPBD ST MMZ MDWE BLIINDS-II S3
CC Dragon 0.6294 0.6628 0.8339 0.8622 0.6111 0.3352 0.9462
Flower 0.6653 0.8578 0.6963 0.8728 0.7607 0.7284 0.9353
Monkey 0.4283 0.4953 0.5167 0.9398 0.2755 0.2691 0.9200
Orchid 0.5004 0.8312 0.8137 0.8678 0.0971 0.2605 0.9432
Peak 0.1393 0.2733 0.8745 0.8807 0.5747 0.2900 0.9256
Squirrel 0.8492 0.9353 0.9051 0.9415 0.8523 0.8475 0.9583
Average 0.5353 0.6760 0.7734 0.8941 0.5286 0.4551 0.9381
SROCC Dragon 0.5886 0.6460 0.8657 0.8924 0.6057 0.3085 0.9295
Flower 0.7064 0.7559 0.6433 0.5467 0.6314 0.5665 0.7392
Monkey 0.3135 0.5057 0.5285 0.8848 0.0494 0.2537 0.9301
Orchid 0.4386 0.7964 0.7594 0.8412 0.4094 0.2706 0.9137
Peak 0.2025 0.0524 0.8601 0.8808 0.4896 0.2947 0.9006
Squirrel 0.835 0.8290 0.7044 0.8007 0.6841 0.7674 0.7987
Average 0.5141 0.5976 0.7269 0.8078 0.4783 0.4102 0.8686
KLD Dragon 2.5945 2.3732 1.8898 1.7963 2.7116 2.3439 1.5495
Flower 3.5302 2.0785 3.7803 3.3975 2.0298 2.5222 2.9532
Monkey 3.1292 2.6790 2.7549 1.1584 1.82663 1.4379 0.6097
Orchid 2.5450 1.6620 2.7284 1.8444 1.5889 2.6231 0.9858
Peak 1.7638 3.0033 2.4110 2.0405 2.4843 1.8834 0.7638
Squirrel 2.6573 2.3824 3.0013 2.2475 2.5408 3.2319 3.2155
Average 2.7033 2.3631 2.7610 2.0808 2.1970 2.3404 1.6796
Table I: Overall performance of S3 and other algorithms on the entire set of six images in our experiment. The two best performances are bolded.

Go to top

No-reference quality assessment of JPEG2000-compressed images

We discuss in this section the performance of S3 on no-reference image quality assessment for JPEG2000-compressed images. One might argue that a sharpness/blurriness estimator can also be used for no-reference quality assessment of JPEG2000 images since JPEG2000 compression can induce blurring. To analyze the performance of S3 on this task, we used the JPEG2000 image subsets from LIVE, CSIQ, and TID2008, which contain 169, 150, and 96 JPEG2000 distorted images, respectively. For comparison, the same sets of images were analyzed by using the following sharpness algorithms: (1) JNB; (2) CPBD; (3) MMZ; (4) ST; (5) MDWE; and (6) BLIINDS-II. The results of this analysis are shown in Table II.

  JNB CPBD ST MMZ MDWE BLIINDS-II S3
CC TID 0.831 0.926 0.800 0.840 0.875 0.921 0.902
LIVE 0.661 0.864 0.663 0.704 0.721 trained on 0.641
CSIQ 0.789 0.88 0.788 0.797 0.826 0.915 0.800
SROCC TID 0.826 0.929 0.823 0.889 0.878 0.901 0.880
LIVE 0.650 0.86 0.562 0.712 0.697 trained on 0.607
CSIQ 0.746 0.853 0.710 0.775 0.785 0.895 0.661
OR LIVE 81.70% 66.90% 82.20% 86.40% 78.10% trained on 87.00%
CSIQ 60.70% 50.70% 59.30% 76.00% 54.00% 42.70% 56.00%
OD LIVE 1822.9 1005.8 1921.4 2962.2 1603.5 trained on 2528.9
CSIQ 11.889 8.367 12.515 29.621 10.772 5.5 11.572
Table II: Overall performance of S3 and other algorithms on JPEG2000 distorted images from TID2008, LIVE and CSIQ database. The two best performances are bolded.

In comparison to JNB, ST, MMZ, and MDWE, except for images from the LIVE database, the performance of S3 is quite competitive. However, the CPBD and BLIINDS-II algorithm clearly outperform others. We believe this limitation of S3 is due to the fact that JPEG2000 compression does not generate blurring uniformly across an image. A highly compressed JPEG2000 image might still contain relatively sharp regions which yield a high S3 index because of our computation of the index in Equation (11) in the manuscript. On the other hand, the artifacts from JPEG2000 compression might ruin edges in the image, thus other sharpness/blur measures which rely on the appearance of edges in the image might give more veridical results for those images. Figure 5 demonstrates this argument. Even though most regions of the image have been blurred by JPEG2000 compression, some regions still appear quite sharp (such as the buildings in the middle of the image). Notice that the corresponding S3 map both correctly detects the blurred regions and is able to capture the sharp regions. The S3 index of this image, computed as the average of the maximum 1% pixel values in the S3 map is therefore fairly high.

S3 index is 0.7607
Fig. 5. A highly compressed JPEG2000 image might still contain relatively sharp regions (e.g., the buildings in the middle of the image). Notice that the corresponding S3 map both correctly detects the blurred regions and is able to capture the sharp regions.

Go to top

Analysis of parameters τ1, τ2, and η

In Eq. (6) in the manuscript, the two parameters τ1, τ2 control the shape of the sigmoid transducer function (see Fig. 5 in the manuscript). Here we analyze the performance of S3 with different selections of τ1 and τ2. The corresponding sigmoid shapes are shown below in Figure 6.

τ1 = -5; τ2 = 1 τ1 = -5; τ2 = 2 τ1 = -5; τ2 = 3
τ1 = -4; τ2 = 2 τ1 = -4; τ2 = 3 τ1 = -4; τ2 = 4
τ1 = -3.5; τ2 = 1.5 τ1 = -3.5; τ2 = 2.5 τ1 = -2.5; τ2 = 2.5
τ1 = -3; τ2 = 2 τ1 = -3; τ2 = 3 τ1 = -2; τ2 = 3
Fig. 6. The sigmoid transducer function with different selection of τ1 and τ2.

Table III shows the performance of S3 on blurred images of TID2008, LIVE, and CSIQ databases with different selections of the pair (τ1, τ2) [see Equation (6) in the paper]. The best performance for each criterion on each database is highlighted. The selection of [τ1, τ2] = [-3, 2] was not chosen based on this analysis; rather it was determined empirically by fitting a sigmoid to subjective ratings of sharpness (using the authors as subjects) for the noise images shown in Figure 5 in the paper. Indeed, this pair gives the best CC only on the TID2008 and the best SROCC only on the LIVE database.

[τ1, τ2] = [-5, 1] [-5, 2] [-5, 3] [-4, 2] [-4, 3] [-4, 4] [-3.5, 2.5] [-3.5, 1.5] [-3, 2] [-2.5, 2.5] [-3, 3] [-2, 3]
CC TID2008 0.61741 0.86033 0.86375 0.86513 0.86389 0.85200 0.86762 0.86784 0.86904 0.86698 0.86261 0.85974
LIVE 0.52675 0.91693 0.94691 0.93243 0.94702 0.92605 0.94411 0.93673 0.94324 0.94627 0.94528 0.94113
CSIQ 0.87556 0.88126 0.93508 0.89803 0.93985 0.95117 0.92250 0.90307 0.91913 0.93884 0.94521 0.95049
  Average 0.67324 0.88617 0.91525 0.89853 0.91692 0.90974 0.91141 0.90255 0.91047 0.91736 0.91770 0.91712
SROCC TID2008 0.84488 0.84373 0.83620 0.84518 0.83587 0.84733 0.83926 0.84259 0.83924 0.83485 0.83633 0.83943
LIVE 0.93183 0.93292 0.94325 0.94087 0.94128 0.90981 0.94308 0.94269 0.94380 0.93946 0.93624 0.93019
CSIQ 0.88173 0.88425 0.91925 0.89525 0.92119 0.92369 0.91072 0.89983 0.91066 0.92070 0.92346 0.92537
  Average 0.88615 0.88697 0.89957 0.89377 0.89945 0.89361 0.89769 0.89504 0.89790 0.89834 0.89868 0.89833
Table III: Performances of S3 algorithm with different τ1 and τ2, applied on TID2008, LIVE and CSIQ database. Best performances are highlighted.

With the chosen constants τ1 and τ2, [-3 2], different values of the parameter η (see Equation (10) in the paper) were also tested to see the influence of this parameter. Table IV shows the performance of S3 when η = 1/4, 1/3, 1/2, 2/3, and 3/4. When η = 1/2, S3 gives the best CCs on TID2008 and LIVE, and SROCC on LIVE. Note that η = 1/2 was not chosen based on this analysis; rather, because S1 and S2 were designed to compensate for each other’s failures, the selection of η = 1/2 is a reasonable choice.

η =

1/4

1/3

1/2

2/3

3/4

CC

TID2008

0.861640

0.865400

0.869040

0.859680

0.846570

LIVE

0.937009

0.940600

0.943239

0.914639

0.884017

CSIQ

0.948330

0.943320

0.919130

0.882080

0.866220

Average

0.915660

0.916440

0.910470

0.885466

0.865602

SROCC

TID2008

0.838870

0.834920

0.839240

0.842460

0.824530

LIVE

0.927921

0.934357

0.943804

0.928574

0.903870

CSIQ

0.926610

0.924540

0.910660

0.879160

0.867440

 

Average

0.897800

0.897939

0.897901

0.883398

0.865280

Table IV: Performances of S3 algorithm with chosen [τ1, τ2] = [-3, 2], and different η. Best performances are highlighted.

Go to top

Details of the contrast computation for Figure 2 in the paper
 
(a)
(b)
(c)
Fig. 7. The three images (a), (b), (c) all have the same spectral slope. However, image (b) and (c) appear much less sharp than image (a) due to the difference in contrast.

While the slope of the magnitude spectrum can be a potential measure of sharpness, it does not take into account contrast, which is known to affect perceived sharpness. Two images can have the same spectral slope but appear to be of different sharpness due to a difference in contrast. Figure 7 demonstrates this assertion. Image (b) was generated from the image (a) by taking 40% of the value of every pixel in (a). This image (b) is then added a constant to make image (c) which has the same mean luminance as image (a). The magnitude spectrums of image (b) and (c) (ignoring the DC component) are therefore the same, and proportional to that of image (a). Thus the three images have the same slope factor. However, the image in (a) appears much sharper than (b) and (c) due to the low contrast of the latter two.

In this section, we computationally show the validity of this demonstration using Weber contrast, Michelson luminance contrast, and RMS luminance contrast.

Weber contrast is defined as: C = (Lmax – Lmin)/ Lmin

Michelson contrast (Peak-to-Peak contrast) is defined as: C = (Lmax – Lmin)/ (Lmax + Lmin)

RMS contrast is defined as: C = σLL in which the conversion to luminance L from 8-bit pixel value image I is given by: L = (a + bI)γ

Lmax, Lmin, σL, and μL are the maximum, minimum, standard deviation, and the mean of L, respectively.

Assuming the Adobe RGB color space, a = 0.7656; b = 0.0364 and γ = 2.2, the luminance contrast of each image in the demonstration above is given as:
Weber contrast: (a) 287.2, (b) 47.71, (c) 5.877
Michelson contrast: (a) 0.993, (b) 0.959, (c) 0.746
RMS contrast: (a) 1.453, (b) 1.125, (c) 0.534

These results support the validity of the demonstration. However, notice from these results that image (c), which is currently used in Figure 2 in the manuscript, better demonstrates the limitation of the slope of magnitude spectrum than image (b).

Go to top

Monotonic prediction of blur parameter
 
Fig. 8. Relationship between the standard deviation of the Gaussian blurring filter’s impulse response and the sharpness/blurriness index of different algorithms.
 
 

Figure 8 repeats Figure 10 in the manuscript and adds the curve from BLIINDS-II. As can be seen from this Figure, BLIINDS-II shows several failure cases in this test. However, note that this behavior is expected given that BLIINDS-II was designed for quality assessment rather than sharpness estimation.


σ = 0.4

σ = 0.8

σ = 1.2

σ = 1.6

σ = 2

σ = 2.4

σ = 2.8

σ = 0.4

σ = 0.8

σ = 1.2

σ = 1.6

σ = 2

σ = 2.4

σ = 2.8
Fig. 9. Examples of images that have been blurred by a Gaussian filter of size 15 x 15 pixels and standard deviation of σ = {0.4, 0.8, 1.2, 1.6, 2.0, 2.4, 2.8}.

As discussed in the last paragraph of Section IV-C, notice from the curves of S3 that for σ ≥ 1.6, the S3 indices are close to zero.  This fact agrees with a visual examination of these images; visually, the overall sharpnesses of images blurred with σ ≥ 1.6 are all very low and significantly lower than images blurred with σ ≤ 1.2. Examples of these images are shown in Figure 9 (click on the thumbnails for full-sized versions).

Figure 10 depicts graphs of the subjective ratings of DMOS (on the vertical axis; MOS for TID) plotted against S3 transformed outputs (on the horizontal axis). These graphs demonstrate that S3 performs quite well on very blurred images (high DMOS values for CSIQ and LIVE database, and low MOS values for TID database).

Fig. 10. Subjective ratings of perceived distortion plotted agains S3 transformed outputs for CSIQ, LIVE and TID2008 database.
The subjective score is DMOS for CSIQ and LIVE and MOS for TID

Go to top

Download S3 code and sharpness maps database.

Sharpness maps database
S3 algorithm Matlab code

Go to top