1.1 Background

To characterize a particular scene or object through locally salient regions, a general three stage process takes place. First, features need to be detected, using interest point or interest region detectors ³³. Second, a unique and informative numerical vector is constructed taking the local image region as input; these vectors are commonly referred to as local image descriptors ²². Finally, the detected image regions and their corresponding local descriptors are used as input to construct a problem specific representation or model. Afterwards, the stored models can be used to detect previously seen features by matching stored visual models with features detected online by the system.

In the process described above, the quality and synergy of a particular detector/descriptor combination is paramount, however the variety of available methods makes it difficult to determine which combination of methods are best suited for common tasks. CV literature is filled with many different proposals, such as the Harris detector ⁹, Shi-Tomasidetector ²⁷ or the FAST detector ²⁶. In the case of local descriptors, popular algorithms include SIFT ²¹ and SURF ¹, techniques that are now widely used and available in popular programming libraries, such as OpenCV or Matlab toolboxes.

1.2 Experimental Approach

The current paper compares these popular techniques with two more recent approaches towards the feature detection and description problem. In particular, we consider the Harris and Shi-Tomasidetectors and the SURF local region descriptor as standard machine baseline methods. Moreover, we explore other recent approaches to address these tasks.

First, we consider proposals that derive feature extraction algorithms using an automated design process based on meta-heuristic and global search algorithms. In particular, proposals that relied on genetic programming (GP), a form of evolutionary computation ¹⁹, to automatically synthesize feature detectors ^{31, 24, 25} and descriptors ^{32, 30}. In the case of feature detectors, ^{31, 24, 25} used widely accepted performance measures to pose an optimization problem and solve it using GP to produce novel image operators. Similarly, ³⁰ used GP to optimize the Holder image descriptor ³², providing a fast and simplified feature description algorithm. It is important to note that automatically designing computer vision algorithms through GP is not a commonly used approach within the CV community. Moreover, those works present a limited and controlled experimental validation of the new methods, so it is still not clear if the GP-generated methods can compete with standard techniques in real-world scenarios.

Second, this work proposes a new approach towards feature description, based on popular techniques from statistical image processing and optics literature ⁸. In particular, we develop a new local description method based on an improved version of the synthetic discriminant filter (SDF) ⁷, a type of composite correlation filter that is widely used in distortion tolerant object recognition and target tracking applications ^{14, 12}, but not often used in computer vision applications based on local features.

To compare these different paradigms we have chosen a difficult real-world problem, widely addressed in vision based robot navigation, referred to as visual simultaneous localization and mapping, or visual SLAM.

Klippenstein and Zhang ^{18, 17} also present a comparison of feature extraction methods applied to visual SLAM, using several well-known computer vision methods. The authors used two comparative metrics to evaluate a SLAM system based on the Extended Kalman Filter. First they determine if the estimated trajectory is consistent with the ground truth trajectory, based on the normalized estimation error squared (NEES). However, their results showed that the SLAM system was almost never consistent, so they relaxed this criteria in their comparative work. Therefore, as a second evaluation measure they used the accumulated uncertainty as their comparative measure. Their experimental work showed no significant difference between the considered methods. On the other hand, the recently proposed SLAMBench evaluation platform ²³ identifies the average trajectory error (ATE) error as the most important comparative measure between SLAM systems; i.e., the error between the estimated robot trajectory and the ground truth trajectory, This measure is useful since most visual SLAM systems will not produce consistent trajectories, making uncertainty a misleading measure ^18,17. Moreover, it is easier to determine the ground truth position and trajectory of the robot than the position of the scene landmarks. However, a more direct comparison of local feature extractors is to consider the accuracy they provide when detecting previously stored landmarks; the main task they are intended to solve. It is also important to consider the computational efficiency of each method, since real-time performance is necessary.

In this work we use the well known Mono-SLAM system of Davison et al. ⁴, a visual SLAM system that builds a sparse map using local features as landmarks. In all our experimental test, three detectors are considered: two common computer vision approaches (Harris and Shi-Tomasi) and an automatically generated detector (MOP). Also, four local descriptors are tested: Normalized Cross Correlation (NCC, the method used in Mono-SLAM), a common computer vision method (SURF), a GP optimized descriptor (Holder descriptor) and the newly proposed Sdf descriptor. Every detector/descriptor combination is tested on the Mono-SLAM system, using two experimental environments: (1) two paths in a controlled environment where the real trajectory of the robot is known; (2) a freely captured video sequence provided with the Mono-SLAM system.

To evaluate each feature detector and descriptor algorithm, three performance measures are used. Firstly, the average error between the estimated robot trajectory and the ground truth trajectory (ATE), in some experiments a precise ground truth is obtained by using a high-precision industrial robotic arm to perform the trajectory in a hand-eye configuration of the visual SLAM system.

Secondly, an evaluation based on the ability of the system to detect and match previously seen landmarks within the scene, this measure is particularly useful since it only considers the feature matching process.

Thirdly, the computational efficiency of each detector and descriptor algorithm, which is reported in CPU time. Results suggest that the non-standard approaches towards local feature extraction can help to improve the accuracy of the matching process, achieve high quality estimations of the trajectory of the camera produced by the Mono-SLAM system, while also substantially reducing the total computational costs.

1.3 Contributions and Organization

Let us summarize the main contributions of the present research work. First, we propose a new local feature descriptor based on the SDF, a technique from optical and signal processing that is not widely known or studied within the computer vision community. Second, we evaluate and compare a GP-based point detector (MOP), a GP-optimized descriptor (Holder descriptor) and the proposed SDF-based descriptor, with common techniques from CV literature on a difficult real-world problem. Finally, our experimental work suggests that these unconventional approaches to the local feature extraction problem compare favorably with widely used CV techniques.

The paper is organized as follows. Section 2 provides a short review of the more popular methods towards local feature extraction in computer vision systems. Section 3 provides alternative approaches for feature extraction, namely: automatic design methods based on the GP ^{24, 25, 31, 32, 30} and a method based on composite correlation filtering ⁷. Section 4 provides a brief overview of the SLAM problem and the Mono-SLAM system. Section 5 presents the experimental work and main results. Finally, Section 6 presents our concluding remarks.

2.1 Interest Point Detectors

Interest points are simple point features within an image; that is, they are image pixels that are salient or unique when compared with neighboring pixels. The algorithms used to detect interest points analyze the intensity patterns within local image regions and only make weak assumptions regarding the underlying structure. Interest points are quantitatively and qualitatively different from other points, and they usually represent only a small fraction of the total number of image pixels.

A measure of how salient or interesting each pixel is can be obtained using a mapping of the form K(x): ℝ⁺ → ℝ called an interest point operator. Applying the mapping K to an image I produces what can be called an interest image I*. Afterwards, most detectors follow the same basic process: non-maxima suppression that eliminates pixels that are not local maxima, and a thresholding step that obtains the final set of points. Therefore, a pixel x is tagged as an interest point if the following conditions hold,

where W is a square neighborhood of size n x n around x, and h is an empirically defined threshold. The first condition in Equation 1 accounts for non-maximum suppression and the second is the thresholding step (see Figure 1).

Fig. 1 Example of the interest point detection process, where O_MOP (I) is the MOP operator and the N(·) represents the non-maxima suppression and thresholding function 

The problem of detecting interest points has been well-studied and a large variety of proposals exist in current literature. For instance, the most widely used methods employ image operators that are based on the local second-moment matrix A(x, σ_I, σ_D), defined as

where σ_D and σ_I are the differentiation and integration scales respectively, G_σ is a Gaussian smoothing function, and L_u = L_u(x, σ_D) is the Gaussian derivative in direction u of image I at point x. For instance, the interest point operator used by the Harris detector ¹⁰ is

where k is a scale parameter, Det defines the determinant and Tr defines the trace. The Shi-Tomasioperator ²⁸ is given by

where λ_1; λ₂ are the two eigenvalues of A. The subindex of K_H and K_S-T are used as shorthand to refer to the namesake of each operator.

2.2 Local Image Descriptors

The problem of local image description has been extensively studied and many algorithms have been proposed. One of the most popular local descriptors is the Scale Invariant Feature Transform (SIFT) ²¹, which is still widely used and available in many computer vision libraries. Similar to other state-of-the-art descriptors, SIFT uses a distribution-based approach, characterizing image information using histograms that attempt to capture the main properties of local shape or appearance. For instance, the simplest approach would be to use histograms of pixel intensities. However, SIFT builds an histogram of the gradient distributions within the detected local region, a 3D histogram of gradient orientations at different locations (discretized by a uniform grid), weighted by the gradient magnitudes. Moreover, random noise is suppressed using bilinear interpolation and a Gaussian function is used to increase the relative importance of pixels near the center of the local region.

One important drawback of the SIFT descriptor is its computational complexity, making it difficult to use in low performance computer systems or in scenarios where real-time response is critical. This is the case of a low cost embedded computing system for a mobile robot that needs to solve a visual SLAM problem.

The Speeded-Up Robust Features (SURF) descriptor was proposed as a more efficient method ¹, and it has achieved strong results in many scenarios ^{3, 11}. This descriptor is also distribution-based, building a histogram of Haar-wavelet responses within the interest point neighborhood. Basically, building the SURF descriptor consists of the following steps. First, the dominant gradient orientation of the local image region is determined. Afterwards, a square region is aligned with the dominant orientation. The region is then divided into a 4 by 4 grid, and the Haar-wavelets responses are then estimated based on a uniform sampling, using the sum of the response in each direction (vertical and horizontal) and the sum of the absolute values. Note that a total of four attributes for each subregion are obtained. This gives a descriptor of 64 dimensions, half the size of the SIFT descriptor. Moreover, integral images are used to increase efficiency, since they allow for a fast implementation of box convolution filters.

3.1 Automatic Design of Local Feature Extractors with Genetic Programming

It is instructive to consider that most published feature detectors and descriptors are normally accompanied with experimental evidence from a particular domain that illustrates the superiority of the method under some conditions when compared with other techniques. However, such results can rarely provide assurance that a particular method will be well suited for a new or unique scenario. Therefore, researchers have performed extensive experimental evaluations and comparisons of these methods, using domain independent criteria that capture the underlying characteristics that such methods are expected to have ^{22, 32, 30}.

Based on those works, recent contributions have followed the opposite approach, using these experimental criteria as the basis for objective functions, and then posed a search and optimization problem to automatically synthesize high-performance feature detectors or descriptors ^{24, 25, 31, 32, 30}. The general goal is to exploit meta-heuristic and hyper-heuristic search methods to help researchers during the design process of specialized operators. However, we do not suggest that such an approach is in some way superior to a traditional design process. Instead, we agree with ²⁴, hyper-heuristic searchers such as GP can produce novel designs that can assist in the development of high-performance and possibly unconventional solutions to difficult problems. In particular, this paper studies the interest point detector generated with a multi-objective GP ^{24, 31, 25}, and the GP-optimized Holder descriptor from ^{32, 30}, both of which are described next. But first a brief introduction to GP is provided, which is the core algorithm used to derive both operators.

3.2 GP-Optimized Regularity-based Descriptor

The Holder descriptor was proposed in ³², and is based on capturing the regularity (or irregularity) of each element of a 2D signal, given by the pointwise Holder exponent. After computing the regularity of each image element, we are left with a regularity matrix H, that characterizes how the signal varies over the image plane. Then, for an image region extracted with an interest point detector, the Holder descriptor is constructed by sampling the regularity matrix in a polar grid around the central point, and ordering the sampling based on the dominant orientation; this process is illustrated in Figure 2.

Fig. 2 Example of the basic process used to construct the Holder image descriptor as proposed in ³²  

However, as noted in ³², the Holder descriptor is not useful in real-time scenarios given the high computational cost of estimating the Holder exponent for each image point. Therefore, in ³⁰ GP is used to synthesize an operator O_Holder to estimate image regularity more efficiently, given by

The proposed evolutionary process used a similar function and terminal set as the one proposed in ^{24, 25}, and was also implemented in GPLAB ²⁹. Two different fitness functions were used to drive the search process. First, the goal was to evolve operators that could compute the ground-truth regularity of synthetically generated images, where the underlying regularity can be prescribed and is given as the ground truth. While the evolved operators achieved a high accuracy on the synthetic images, they were overfitted and did not generalize to real-world scenes.

Therefore, the second approach was to reproduce the estimated regularity of a known but computationally expensive method. In ³⁰ we showed that the operator generated by GP could estimate image regularity at a fraction of the computational cost of traditional methods, without sacrificing performance on the feature description problem evaluated over a set of standard benchmarks. However, such as in ^{24, 25}, the evolved operators have not been evaluated in real-world scenarios, where the computed local features are used as input to a higher level process. In what follows, the SLAM problem is described emphasizing how local image features are used in this common robotics task.

3.3 Synthetic Discriminant Functions Filter for Local Feature Description

Composite correlation filters are widely used for distortion tolerant object recognition, and in tracking applications for computer vision ^{16, 2}. These filters represent the impulse response of a linear system designed in such a manner that the coordinates of the system's maximum output are estimates of the target's location within an observed scene. In the last decades plenty of proposals have been suggested for the design of composite filters for distortion tolerant pattern recognition, by optimization of several performance criteria ¹⁶.

In the present work, we are interested in the design of a filter capable of matching a local image feature when it is embedded in a disjoint background and the observed scene is corrupted with additive noise. Additionally, the filter must be able to recognize geometrically distorted versions of the target, such as rotated and scaled versions.

Let I = {t_i(x,y); i = 1, ⋯, N} be a set of availabe image templates representing different views of the image feature t(x, y) to match. We assume that the observed scence patch f(x, y) contains an arbitrary view of the target t_i (x, y) embedded into a disjoint background b(x, y) at unknown coordinates (τ_x, τ_y), and the image is corrupted with zero-mean additive white noise n(x, y), as follows:

where w¯(x,y) is the region of support of the target (binary function) defined as zero within the area occupied by the target, and unity elsewhere. It is known that the best filter for detecting a single view of the target t(x, y) in Equation 6, in terms of the signal to noise ratio (SNR) and the minimum variance of measurments of location errors (LE), is the matched filter (MF) whose frequency response is given by ¹⁵:

In Equation 7, T(u, υ) and W¯(u,υ) are the Fourier transforms of t(x,y) and w¯(x,y), respectively, μ_b is the mean value of b(x, y), and P_b(u, υ) and P_n(u, υ) are the spectral densities functions of b₀(x, y) = b(x, y) - μ_b and n(x, y), respectively.

To successfully employ the MF for the feature description problem used by a SLAM system, the following issues must be addressed. First, the support function w¯(x,y) of the target is explicitly unknown, since new unseen image features are matched online during system operation. Second, the statistical properties of the background and the additive noise process, required to synthesize the MF, are unknown; so, they need to be estimated online.

Note that the target can be located at any coordinates within the observed scene patch. Thus the support function of the whole patch can be taken as w¯(x,y). In such a case, the MF for detecting a target with an explicitly unknown support function, is given by

Now, suppose that the background within the scene patch has a separable exponential covariance function; then, P_b(u, υ) can be computed as ¹³

where σ^b2 is the variance of b₀(x, y), and p_x and p_y are correlation coefficients of the background in x and y directions. These parameters can be known beforehand, otherwhise, they can be easily estimated from input signals ⁵.

Furthermore, assume that the noise-free image p(x,y)=ti(x-τx,y-τy)+b(x,y)w¯(x-τx,y-τy) within the scene patch, and the zero-mean additive noise n(x, y) are independent. So, the covariance funtion of the observed scene patch is Cf(x, y) = C_p(x, y) + C_n(x, y), where C_p(x, y) is the covariance function of p(x, y) and Cn(x, y)=σ^b2δx, y is the covariance function of white noise. The noise variance can be estimated as σn2=Cp(0,0)-Cf(0,0); however, C_p(0,0) is unknown. Note that C_n(x, y) = 0, ∀(x, y) ≠ 0. So, the values of {Cf(x, y);(x, y) ≠ 0} can be used to estimate C_p(0,0). This can be done using simple linear extrapolation, as follows:

Now, let h_i(x, y) be the impulse response of a MF constructed to match the ith available view of the target t_i(x, y) in I. Let H = {h_i(x, y); i = 1, · · · , N} be the set of all MF impulse responses constructed for all training images t_i(x, y). We want to synthesize a filter capable to recognize all target views in I, by combining the optimal filter templates contained in H, and by using only a single correlation operation. The required filter p(x, y) is designed according to the SDF filter model, whose impulse response is given by ¹⁶

where {α_i; i = 1,⋯, N} are weighting coefficients that are chosen to satisfy the following conditions: 〈p(x, y), t_i(x, y)〉 = u_i; where "<, >" denotes inner-product, and u are prespecified output correlation values at the origin, produced by the filter p(x, y) in response to the training patterns t_i(x, y).

Let us denote a matrix R with N columns and d rows (d is the number of pixels), where its ith column contains the elements of the ith member of H in lexicographical order. Let a = [α_i; i = 1,⋯,N]^T be a vector of weighting coefficients. Thus, Equation 11 can be rewritten as

Furthermore, let u = [u_i = 1; i = 1..., N]^T be a vector of correlation constraints imposed to the filter's output in response to the training patterns t_i(x, y), and let Q be a d x N matrix whose ith column is given by the vector version of the ith view of the target in I. Note that the filter's constraints can be expressed as

where superscript "+" denotes conjugate transpose. By substituting (12) into (13), we obtain u = Q⁺ Ra. Thus, if matrix Q⁺ R is nonsingular the solution for a, is

Finally, by substitution of (14) into (12), the solution for the SDF filter is given by

For the purpose of local feature description, we propose the following process based on the SDF filter design, depicted in Fig. 3. First, given an interest point x, we extract a scene patch of 11 x 11 pixels around x; this represents our reference target pattern t(x, y). Next we create two synthetically rotated versions of t(x, y), one rotated 15 degrees clockwise and another rotated 15 counterclockwise, and construct a three element set I. Next, set H is created by syhthetising the MF impulse responses of all patterns in I. Finally, an SDF filter template is synthesized with Equation 15.

Fig. 3 Procedure to obtain an image descriptor based on the SDF filter. Left: target object; Center: training templates used to construct the matrices R and Q; Right: the SDF image template. The images are normalized for easier understanding 

5.1 Experimental Setup

The Mono-SLAM system is conigured to use 10 visible features at each iteration with a patch size of 11 x 11 pixels. The parameters of the methods are given in Table 2.

The parameters in Table 2 were experimentally tuned to obtain the lowest ATE for each detector/descriptor combination. To determine a match for the SURF and Holder descriptors the Euclidean distance between two descriptors is computed, and a match is scored if the distance is below a threshold t_m. Similarly for the SDF and NCC descriptors, a threshold for the output correlation peak t_c is used to determine a positive match. The values of the threshold vary for each descriptor to obtain the best possible performance; the threshold values are given in Table 3.

5.2 Results

As stated above, the experimental results are presented based on performing 30 independent runs of the Mono-SLAM system with each detector/descriptor combination for each experiment. For experiments A and B, where the ground truth is known, a graphical representation is provided by plotting the average estimated trajectories relative to the ground truth. We group the results based on the interest point detector used and we show the trajectories from four different perspectives: a 3D view, and a view from each plane from the robot's frame of reference (X-Z, Z-Y, X-Z). Additionally, the median of the ATE and of the total correct matches are provided for each detector/descriptor combination. For the number of correct matches, we first take the median number of matches over all the iterations of the system, and then take the median over all 30 runs. In the best case scenario, the trajectory error should be close to zero, while the number of total matches should tend to the total number of visible features (10 in our setup). Statistical tests are performed using a 1XN setup (N methods are compared with a single control method), where the base Mono-SLAM detector/descriptor combination (Shi-Tomasi/NCC) is used as the control method against which all other methods are compared. The non-parametric Friedman test is used to perform the statistical tests, and the resulting p-values are corrected by the Bonferroni-Dunn method. The null hypothesis is rejected at the α = 0.05 confidence level.

Finally, to test the efficiency of each method the following procedure is performed. We use a single highly textured image of 640x480 pixels. Then we detect a total of 100 interest points and extract 100 corresponding feature vectors. The total CPU time required to detect and describe these interest points is also reported.

5.2.1 Experiment A

Figures 8 to 10 show the average estimated trajectories of each detector/descriptor combination for Experiment A. For the Harris detector (Figure 8), the SDF descriptor provides the best trajectory estimation. For the MOP detector (Figure 9), SDF also provides a good estimation, as well as the Hölder descriptor, while the SURF descriptor is noticeably worse. Finally, for the Shi-Tomasi detector, the best estimation is achieved by the Hölder descriptor, while NCC and SURF are noticeably worse.

Fig. 8 Illustrations of the average estimated trajectories for the Harris detector for experiment A; where (a) 3D view, (b) X-Y plane view, (c) Y-Z plane view, (d) X-Z plane view 

Fig. 9 Illustrations of the average estimated trajectories for the MOP detector for experiment A; where (a) 3D view, (b) X-Y plane view, (c) Y-Z plane view, (d) X-Z plane view 

Fig. 10 Illustrations of the average estimated trajectories for the Shi-Tomasi detector for experiment A; where (a) 3D view, (b) X-Y plane view, (c) Y-Z plane view, (d) X-Z plane view 

Table 4 summarizes the numerical comparisons for Experiment A and Table 5 provides the p-values of the statistical tests. For this experiment, the ATE of the system appears to be quite robust with respect to the detection/description process, except for two outliers that did exhibit a statistical difference with respect to the control method, these are: ST-SDF and MOP-NCC. While the MOP-Hölder combination shows the best ATE, the only combination that is statistically different with respect to the control method is ST-SDF, while MOP-NCC is the only combination that is statistically worse. However, if we focus on the quality of the detection, description and matching of visual landmarks the results are somewhat different. The median and maximum values are very similar for all combinations, but several methods do outperform the control method, these are: ST-SURF, ST-Hölder, ST-SDF, MOP-SURF, MOP-Hölder, MOP-SDF, Harris-NCC and Harris-Hölder. In summary, for Experiment A the ST-SDF combination is able to improve system performance based on both the ATE and the total correct matches. Similarly, MOP-Hölder also exhibited a noticeable improvement relative to the base Mono-SLAM algorithms.

Table 4 Numerical comparisons for Experiment A, where bold indicates best results 

Experiment A	Match		ATE [meters]
	Max	Median	Max	Min	Median
ST-NCC	10	7	0.0551	0.0371	0.0471
ST-SURF	10	8	0.0543	0.0353	0.0446
ST-Hölder	10	7	0.0554	0.0343	0.0431
ST-SDF	10	6	0.0549	0.0363	0.0466
MOP-NCC	9	6	0.0566	0.0391	0.0502
MOP-SURF	8	5	0.0559	0.0370	0.0469
MOP-Hölder	10	7	0.0523	0.0335	0.0422
MOP-SDF	10	8	0.0544	0.0365	0.0450
Harris-NCC	10	8	0.0569	0.0354	0.0445
Harris-SURF	10	7	0.0555	0.0358	0.0442
Harris-Hölder	10	8	0.0546	0.0353	0.0441
Harris-SDF	10	7	0.0554	0.0343	0.0431

Table 5 Results of the 1xN statistical tests, showing the p-values of the Friedman test after Bonferroni-Dunn corrections for Experiment A, with ST-NCC as the control method; bold indicates that the null hypothesis is rejected at the a = 0.05 confidence level 

Experiment A ST-NCC	ST-SURF	ST-HOL	ST-SDF	MOP- NCC	MOP-SURF	MOP-HOL	MOP-SDF	HA- NCC	HA-SURF	HA-HOL	HA-SDF
ATE	1.5854	3.0065	0.0000	0.0001	1 1.0000	0.1165	3.0065	5.1173	5.1173	0.3131	3.0065
Match	0.0071	0.0005	0.0464	1 1.0000	0.0046	0.0439	0.0194	0.0469	9.2563	0.0046	2.2091

5.3 Experiment B

In this experiment, while the trajectory is simpler than the one used in Experiment A, it introduces scale changes in the observed scene which were not present in Experiment A. This is an important issue, since the interest point detectors (Harris, ST and MOP) are not scale invariant, so the detection process is expected to degrade relative to the performance reported for Experiment A. Therefore, the heavy lifting in the detection process must be done by the feature descriptors extracted from the scene. Moreover, while scale invariant detectors do exist, they are computationally expensive and would degrade system efficiency substantially ³³.

Table 6 summarizes the numerical comparisons for Experiment B and Table 7 provides the p-values of the statistical tests. For this experiment, the ATE of the system is more dependent on the detection/description process. We observe much larger performance differences, the median ATE of the best method (MOP-Hölder) is about 70% smaller than the ATE of the worst (ST-NCC). In this case, all combinations are statistically different with respect to the control method, based on the p-values that allow us to reject that null hypothesis. Therefore, to determine the best overall method a multigroup statistical test is performed using an NxN (all pairwise comparisons of N methods) approach with all pairwise comparisons done the Friedman test, and p-values corrected using the Holm procedure. Results are reported in Table 8, where bold indicates that the null hypothesis is rejected at the α = 0.05 confidence level. This table clearly shows that the best ATE performance is achieved by the MOP-Hölder, MOP-NCC and MOP-SURF combinations, suggesting that the MOP detector provides the best detection under scale changes. Figure 11 shows the estimated trajectories obtained with MOP-Hölder, MOP-NCC, MOP-SURF and the control method ST-NCC.

Fig. 11 Illustrations of the estimated trajectories of the best detector/descriptor combinations for Experiment B: (a) 3D view, (b) X-Y plane view, (c) Y-Z plane view, (d) X-Z plane view 

In the case of feature matching, results are quite similar. Most detector/descriptor combinations (except Harris-NCC) outperform the control method. In particular, once again the MOP-Hölder and MOP-SURF combinations achieve the best performance, along with Harris-Hölder. This suggest that the Hölder descriptor is robust to scale changes even without the use of a scale invariant detector. For this scenario both of the methods that were automatically designed by a GP search (MOP and Hölder) exhibit the best performance.

5.4 Experiment C

Table 10 summarizes the numerical comparisons for Experiment C and Table 9 provides the p-values produced by the statistical tests. For this experiment, since the ground truth trajectory is not known, we only compare the detector/descriptor combinations based on matching performance. The median values of the total correct matches suggest that most combinations outperform the control method; in particular MOP-SURF and MOP-Hölder exhibit the best performance with perfect median values. Moreover, the statistical tests show that the null hypothesis is rejected in all cases except for the ST-NCC combination. The Shi-Tomasi detector exhibits the worst performance, while the MOP detector achieves the best overall performance. To visualize these results, Figure 12 compares the matching performance of the control method and the best detector/descriptor combinations: MOP-SURF and MOP-Hölder. The plots show the median value of correct matches over all runs of the system at each iteration, plus/minus the irst and third quartiles. Notice that the maximum number of possible correct matches at each iteration is 10, which is achieved at many iterations by the MOP-SURF and MOP-Hölder combinations, but not by ST-NCC. These plots conirm that matching performance in the Mono-SLAM system can be greatly improved by the appropriate detector/descriptor combination.

Fig. 12 Plots of the total Matches over all the iterations of the Mono-SLAM system using Experiment C. The plots show the median value over all runs at each iteration, plus/minus the first and third quartiles 

5.5 Comparison of CPU Time

Finally, we evaluated each method based on the total CPU time required to either detect a 100 interest points or build their corresponding descriptors; the results are given in Table 11. First, it is clear that all detectors require about the same computational time, with Harris the slightly slower method and MOP the fastest. On the other hand, the differences between the descriptors are larger. While SURF and SDF are relatively similar, though SDF is faster by 3 ms, the Hölder descriptor is one order of magnitude faster than all other methods, requiring about half the time of the popular SURF descriptor. Given the strong performance achieved by Hölder on the tests reported above, this descriptor is recommended for use in future applications.