4.1 Metrics based on 2DIQA

The SIQA-SET was chosen based on their reported performance, in the same way we collected 29 2DIQA in order to provide a baseline of 2D metrics (2DIQA-SET).

In the 2DIQA-SET we can find Statistical Image Quality Assessments (St-IQA), Full-Reference Image Quality Assessments (FR-IQA), and No-Reference Image Quality Assessments (NR-IQA). The fist twelve 2DIQA are part of the MetrixMux toolbox ^[40], while the rest of the metrics were collected from their respective authors.

On the one hand, for describing St-IQA, let I(i, j) and I^i,j be two images to be compared, being I(i, j) the original reference or source image, which has to be considered with perfect and unquestionable quality and I^i,j a distorted version of I(i, j), whose quality in comparison to I(i, j) is being evaluated.

In the other hand, both FR-IQA and NR-IQA algorithms just are listed in order to save space and we do not describe them here, so the reader is referred to the cited papers.

where Imax2 is the maximum possible error or the peak of the MSE value between I(i, j) and I^i,j. That is, for an 8 bit per pixel (bpp) intensity image, Imax2=28-12=65025^[43]. For 24bpp RBG images the PSNR is also defined by Eq. (2), where RGB MSE is the average of independent MSE values estimated in each 8bpp chromatic component, i.e. MSE_Red, MSE_Green, and MSE_Blue.

Once the 2DIQA-SET is defined, let us describe all the stereoscopic algorithms based on it.

— d₁, d₂, and d₃ are global disparity distortion measures described by Equation 9:

where D _dg is computed using the correlation coefficient between the original disparity maps and the corresponding disparity maps processed after image degradation. M define the the averaged left and right image distortion measures.

— A_v and V_a are defined by Equation 10. For A_v algorithm, a 2DIQA is separately applied on the left and right views then for producing a single measure of stereoscopic assessment, the calculated quality results are averaged. While V_a uses some values in order to separately weight a predicted score on the left and right views , these weights are equal to 0.43 and 0.57 respectively, which are similar from the weights used in the central approach (0.5). The unequal weights in V_a can be helpful 3D image databases such as LIVE 3D, since stereoscopic images are taken sequentially with a 2D camera, thus there are often important differences between the left and the right views, due to movement between the left and right shots.

— PSNR_edge is an algorithm, which makes use of a specific kind of quality metric, where the reference or original image is barely employed, since for stereoscopic depth map transmission this algorithm only uses extracted information from edges. Different depth levels are represented by edges and contours of the depth map and quality evaluations can be used this information. This metric uses a Sobel filter both in original and distorted images in order to obtain four binary edge marks. Left and right edge marks are applied to the original and distorted stereo-pairs. Then, these filtered source or original and presumably distorted stereo-pairs are tested using any image quality assessment of the 2DIQA-SET. Finally, the individual results obtained both in original and distorted images are averaged.

Then, PSNR_edge refers to full-reference 2DIQA rating for the depth map and f(2DIQA_bem) refers to the 2DIQA quality rating for the side information (i.e. edge information/binary edge mask).

— MSE_dp measures the Mean Squared Error or any 2DIQA between the disparity difference between reference and distorted stereoscopic image. It is defined as:

where DP_x, and DP_y are disparity map from reference and distorted image, respectively.

— YouDMOS_p, and OQ. The fist approach is obtained performing a nonlinear regression of a result of a certain metric (IQ) of the 2DIQA-SET, using the following function:

While the second approach, is a global combination, which computes two quality assessments of the distorted image. First, a result of a certain metric (IQ) and then, distorted disparity (DQ). This overall quality (OQ) is taken as the quality of the stereoscopic image using the following function:

where a = 3.465, b = 0.002, c = -0.0002, d = -1.083, and e = 2.2.

4.2 Stereoscopic Metrics

Similarly to the previous section, Stereoscopic Metrics are described by highlighting only the main features.

— AkMOS_p is a no-reference perceptual quality assessment based on features local segmentation of artifacts and disparity, i.e. this metric extracts information from edge and non-edge areas in addition to evaluate blockiness based relative disparity estimation. AkMOS_p is computed by the following equation:

where α and β are the model parameters, while D_Z, B and Z are the overall disparity feature, blockiness and zero crossing of each stereo-pair, respectively.

— Ddl₁ is obtained by evaluating Equation 17. The local SSIM measure map M_map is evaluated by measuring and fusing it with the distortion of the local disparity assessment using point-wise product. The disparity distortion is evaluated for each pixel p using the disparity map (DM) for both views (left and right) as follows (left view):

Thus, Ddl1 is the average value of the N pixels of Ddl_left and Ddl_right maps and by averaging both results as follows:

— Q_s analyzes the location of the artifacts by means of masking images, which are obtained by evaluating the difference between original and synthesized views. Then, a threshold Th is applied in order to identify critical areas. Th is defined as follows:

where I is the original image, and I' is the synthesized view. Then, this metric applies SSIM measure only on these critical areas and the final score is the mean SSIM scores averaged by the number of pixels.

— C_m is a cyclopean image synthesized and disparity-compensated from stereo views and it is calculated by:

where I_L and I_R are the left and right images respectively, and d is a disparity index that corresponds pixels from I_L to those in I_R. The weights W_L and W_R are calculated from the normalized Gabor filter magnitude responses.

— SBLC or Stereo Band Limited Contrast employs RANSAC algorithm to extract regions with high spatial frequency both in the left and right images, then, matched points are found. Thus, Surrounding pixels of these points are calculated and pixels outside them are discarded. SBLC is calculated as follows:

where C(x) is the the corresponding matched regions and then the average of matched regions founded both in the left and right images, L is the overall relative mean luminance, and p is the number of matched points in a certain region. SBLC estimates C(x)/L both in Original (Orig) and distorted (Comp) stereo-pairs.

— ODDM₄ is based on the ocular dominance theory and degree of parallax, the latter is calculated as follows:

where L and R represent left and right views respectively, and M indicates the central region of the image.

Ocular dominance theory is introduced by predicting separately the values of left and right image quality, i.e. Q_J(L) and Q_J(R) are the JPEG Quality Score ^[46] of the left and right images. Hence, ODDM₄ is defined by:

— MSE _ms is based on binocular human visual system, since it considers the cyclopean view and perceptibility of depth. This metric takes into account the masking effects of the Contrast Sensitive Function (CSF) and depth variability. Then, in order to differentiate the stereoscopic image structure, a downsampling into multi-scale images is applied to the left channel using a low pass filter. The size image is H × W, while number of pixels in the i-th downsampled image is (H × W)/(2²ⁱ). The final multi-scaled image of left channel is obtained as:

where Xileft and Yileft are the i-th downsampled left image both in reference and distorted image, MSE_hvsm is MSE version of the metric PSNR-HVS-M proposed by ^[33]; k_i is a constant parameter.

— PQM_3D is based on Perceptual Quality Metric (PQM), which is developed to measure slight changes in noises inducted into an normal image. PQM is obtained subtracting the PDM(f) from 1 and values less than 0 are equated to zero as the range of the metric is between 0 and 1, representing the worst and best qualities, respectively. The PQM is calculated as follows:

and

where PDM(t) is the distortion on all T block levels and finally weighted by the weighting factor W(t) to obtain the frame level Perceptual Distortion PDM(f).

Thus, PQM_3D is the average of individual PQM qualities of color and depth images, rendered into left and right views.

— Qmao is summarized in the following steps:

— Q_shao extracts distortion-specific features using only the distorted stereoscopic image. In this way, Q_shao is a two-phase feature fusion procedure, namely Training phase and Test phase. First, this metric employs four distortion categories (Gaussian blur, White noise, JPEG compression and JPEG2000 compression) in order to predict which kind of noise distorted the original stereo-pair. Then, a Support Vector Regression (SVR) is used to predict the relationship between stereoscopic features and subjective scores.

— HDPSNR includes a value of stereo vision into the classical PSNR formula and it is expressed by:

where S is a vector summation or Minkowski summation, defined by:

here e_n is the difference of left and right images after they are decomposed in N contourlet subbands and weighted by the following contrast sensitivity function, being f is the spatial frequency:

— 3VQM is a combination of three distortion measures:

Thus, 3VQM is defined as follows:

where SO, TO, and TI are normalized to range 0 to 1 and a, b, and c are determined by training. K is a constant for scaling 3VQM ranges. (SO∩TO) avoids to take into account a the outlier distortion more than once. Finally, Equation 29 is apply to left and right images obtaining two 3VQM matrices, which are averaged.

— IQA, and SSA assess stereo images from the perspective of average image quality and stereo sense, respectively.

IQA is defined as the arithmetic mean of the Left and Right image gauged by PSNR as follows:

SSA contains the absolute disparity image, namely it is the different information from the stereo-pair and is defined as:

here MSE_M is as follows:

where R_O and L_O refers to original stereo-pairs, while R_P and L_P are the processed or recovered ones. M is the number of nonzero-pixels in the original absolute disparity image (R_O - L_O).

— DQ_map1, DQ_map2, and DQ_map3. AI1 these three approaches are local combinations, which compute a quality map of the disparity image resulting an approximate distribution of the degradation on the distorted disparity image. Equation 33 computes the quality map on the disparity image in three different ways.

where D and D- denote the original disparity image and the distorted disparity image, respectively.

e_i is an overall visual quality measure between original stereo-pair (org) and the distorted stereo-pair (dst) and is calculated by:

where r_kθ is the output through Human Visual System (HSV) formulated as a subset s_kθ at scale k and phase θ after wavelet decomposition and r_z the perceptual response to depth of HVS. A_k and B_z are weight coefficients that are determined experimentally.

4.3 Discussion

Once SIQA-SET is described, we found some similarities among SIQA that we want to highlight.

In Equation 9, ^[3] defined M , which is employed in d₁ and d₂, in the same way that ^[7] did for A_v (Equation 10) in addition to ^[48] for IQA (Equation 30). Other authors such as ^[49] for YouDMOS_p (Equation 13) and OQ (Equation 14) or ^[3] for V_a (Equation 9) just modify these averaging algorithms either weighting individual qualities of the stereo-pair or performing a nonlinear regression.

In this sense, ^[21] proposed MSE_dp (Equation 12) in the same way that ^[3] did it for d₃ (Equation 9).

Other SIQA algorithms such as PSNR_edge^[16], Q_s^[5], SBLC^[14], or Q_mao^[24] use tools of image processing particularly within edge detection algorithms, since they employ Sobel operators, RANSAC algorithm or location of artifacts.

SSA^[48] and HDPSNR^[38] modify the well-known algorithm PSNR using respectively either an absolute squared difference of stereo-pair or a contrast sensitivity function after a con-tourlet transformation.

In the case of DQ_map3, ^[49] eliminated the parameter M_{map_left} of the Equation 16 of Ddl₁^[3] in order to improve its performance in local distortions such as JPEG or JPEG2000.

Q_shao^[35] is the only algorithms that employs a phase of Training, which is time-consuming task and had to be repeated every time the image database is changed or another kind of distortion is introduced.

While AkMOS_p^[1], e_i^[53], C_m^[9], and PQM_3D^[22] weight right and left views using some features of HSV such as Normalize Gabor Filters or responses of HSV in the Wavelet Domain. In the particular case of AkMOS_p, this metric performs a particular nonlinear regression including local segmentation of artifacts and disparity.

Finally, ODDM₄^[15], MSE_ms^[21], and 3VQM^[39] employ a perceptual 2DIQA, but not as a simple average, these metrics add another characteristics or weights such as degree of parallax, depth variability or temporal inconsistencies.

5.1 Metrics based on 2DIQA

In this subsection, we expose the results not only of SIQA based on 2DIQA but also of certain authors that propose more than one SIQA. We use ← to refer that we applied a certain 2DIQA algorithm into a SIQA.

5.2 Stereoscopic Metrics

In this subsection, we sketch only the results of metrics that do are not based on a normal metric or they just are based in one feature of a certain normal image quality assessment.

Table 9 just contains the results of all stereoscopic metrics exposed in subsection 4.2. Where Ddl₁ linear correlates in 86.29%, Figure 12(a), being the assessment that estimates the best Linear Correlation. Furthermore, MSE_ms is the best ranked metric obtaining the best results both in SROCC and KROCC with 89.52% and 70.22%, respectively.

Also, DQ_map2 is the most precise stereoscopic assessment for all whole of considered noises and image compression noises such as JPEG2000 and JPEG distortions.

Taking in to account only noises produced by a image compression coder, DQ_map3 is best metric in LCC and SROCC. Figure 12(b) depicts dispersion of the 365 results of DQ_map3.

5.3 ALL SIQA-SET

Regarding the overall experimental results, Table 10 depicts the performance of all SIQA of the SIQA-SET exposed in section 4.

Thus, Figure 11(a) shows the metric YouDMOS_p ← UQI, this assessment linear correlates in 93.71% and obtains the best Linear Correlation.

Furthermore, the best ranked metric is A_v ← MAD, because it is best correlated both in SROCC and KROCC with 93.94% and 77.72%, respectively.

Also, we can say that A_v ← MAD is the most precise algorithm for all set of distortions considered and JPEG2000 and JPEG noises. Regarding only these image compression distortions, A_v ← MAD is the best ranking metric, i.e. to average right and left MAD qualities, if we change the weighted parameters of the stereo-pair, namely V_a ← MAD, we get the best ranking metric for JPEG distortion.

In this paper we have presented several metrics (280) for gauging the quality of a stereo-pair intended for researchers interested in stereoscopic coding, visual discomfort or stereoscopic displaying. These researchers could want to found the best metric taking into account a certain respond in a certain distortion, for example.

The selected metric could be the best overall measure, which does not mean it would obtain the best results in all individual distortions. For those researchers who are interested in knowing the behavior of the top-ten overall SIQA-SET, we propose Tables 11 to 14, which were made considering the following aspects:

From Table 11 A_v ← UQI (Fig. 8(a)) and YouDMOS_p ← UQI (Fig. 11(a)) obtained the best results in linear correlation coefficient not only in overall performance (93.71%) but also in all individual distortions, except in White Noise. For White Noise d₁ ← FSIM (Fig. 7(a)) is the best metric with 93.07%.

Table 12 shows the performance across SIQA-SET in estimating subjective 3D/stereoscopic image quality using SROCC. Where A_v ← MAD (Fig. 8(b)) obtained the best results in overall performance (93.94%), JPEG2000 (92.47%), White Noise (94.97%), and Gaussian Blur (95.37%). While YouDMOS_p ← UQI (Fig. 11(a)) got the best results in JPEG (73.83%), and Fast Fading distortion (83.28%).

From Table 13 A_v - MAD (Fig. 8(b)) obtained the best results in linear correlation coefficient not only in overall performance (77.72%) but also in all individual distortions, except in Fast Fading distortion. For Fast Fading distortion YouDMOS_p ← UQI (Fig. 11(a)) is the best metric with 64.47%.

Table 14 shows the performance across SIQA-SET in prediction of the perceived stereoscopic image quality using Root Mean Squared Error. Where Av ←MAD (Fig. 8(b)) obtained the best results not only in overall performance (0.0732), but also in JPEG2000 (0.0630), JPEG (0.0529), White Noise (0.0805), Gaussian Blur (0.0919), and Fast Fading distortion (0.0861).