6.1 I3GO+ (First Place)

We start describing the solution that won the RICATIM challenge. The proposal consists of generating a new low dimensional space by approximating the k centers problem using the Farthest First Traversal algorithm (FF-traversal), along with a kernel function. The new representation feeds a kNN classifier. By using this approach, the dimension is reduced, and the final performance is improved. Below we describe each of the steps involved in this solution, a graphical description of this method is provided in Figure 7.

Fig. 7 Flow diagram for FF-traversal scheme 

6.1.1 The k-Centers Problem

Finding a solution for k-centers is equivalent to the generate є-nets problem (^[40]). From this point of view, k-center is an optimization problem, where the objective is to find a set of centers C={c_i,c₂, … ,c_k} of cardinality k = |C|, and where all elements x_i ∈ X are at most distance є from its nearest center in C. The precise value of є is optimized for a particular value of k, and conversely, k can also be optimized for some є.

6.1.2 Farthest First Traversal

The Farthest First Traversal (FF-traversal) algorithm was proposed simultaneously in ^[15] and ^[20]. The algorithm finds the best know approximation (in polynomial time) for the k-centers problem, i.e., at most two times the optimal solution. Finding any best approximation is NP-Hard (see ^[15,20,11]).

The algorithm of T.F. Gonzalez ^[15] calculates a set C, where all c_j ∈ C are farthest among them. So, for each x ∈ X, the distance to the nearest centers is kept in C, the distance for each x_i is updated each time that a new c_j is added to C. In order to ease the algorithm definition let use define d_min(x) as the distance between x and its nearest center in C, then d_min(x) = min{d(x, c) | c ∈ C}. Algorithm 1 specifies the FF-traversal:

Algorithm 1: The farthest first traversal algorithm 

Given some starting element for C, the farthest object in X \ C to all objects in C is located in each iteration, and then it is added to C. This order resembles a farthest-first walk or traversal over X, and this fact gives the name to the algorithm. Let r be the distance used to choose the new center; it follows that at each FF-traversal iteration, a Delaunay set is built; the following properties are preserved:

So, the FF-traversal algorithm finds k elements that minimize the maximum distance between the elements x ∈ X to some c among the available centers C. Figure 8a shows the Delaunay set generated from 5 normal distributions with four items each one. Each ball radius is the optimal r found at iteration 5. Moreover, Figure 8b illustrates the order of how FF-traversal selected the five centers in C.

Fig. 8 FF-traversal phases 

6.1.3 Kernelized k-Centers

The proposed solution is based on the hypothesis that chosen centers are good enough to represent the complete database variation; hence selected centers are used to create a lower dimensional space. The new feature space is created by mapping all the objects in X to the new space X’, and X’ is used as training dataset for a k-nn classifier. The new features for each x_i are generated using a function Φ(x, C) over each x_i and the elements in C; i.e. x i'=Φkxi,C=ϕxi,c1,ϕxi,c2,…,ϕxi,ck, where ϕ(x_i, c_j) can be any kernel function (similarity or distance d). For instance to use a linear kernel ϕ(x_i,c_j)=d(x_i, c_j) is used, and for a RBF (Gaussian) kernel the function given by equation 1 can be used instead:

For the presented experiments when using RBFs, the parameter σ is set to the є found at the last iteration of the FF-traversal algorithm. Algorithm 2 describes the feature generation algorithm:

Algorithm 2: Farthest First Traversal for feature condensation 

This approach is named FF-traversal for Feature Condensation (FFTFC) and the generated feature space as k-Centers Features. The resulting process is composed of two main stages:

6.1.4 Experimental Setup

The authors tried several related approaches. Firstly, this proposal used the FFTFC over the official image vectors and the plain word2vec features. To control the significance of each kind of feature, the authors introduced a weighted metric d_W, defined in Equation 2:

where W ∈ [0,1]. By using Equation 2 the weight W is used to adjust the contribution of each feature portion (i.e., image and text). The kernel function may use the set of centers C, or the set of centroids Ĉ. Each centroid ĉ∈Ĉ is computed using c, where each component of ĉ is computed as the average of that component over all items having c as its nearest center c∈C.

Also, a set of additional features for text labels were produced with μTC library (see ^[41]). Feature vectors obtained with μTC are sparse vectors of dimension m = 3951, these features were concatenated to the image data to get 8047-dimensional vectors. A new set of k-Centers features were created using this new representation. A standard PCA dimensionality reduction and the nonlinear Kernel PCA (KPCA) were compared with FFTFC. All results in this approach were computed using an RBF kernel, for both KPCA and FFTFC.

The selection of the classifier and its parameters were performed using a cross-validation split of 70% for the training set and 30% for the validation set. The resulting features were used to fit a kNN (k=3) classifier. Table 3 shows the obtained results for the four evaluated metrics at the RICATIM Challenge. As can be seen, over the 70-30 split, the k-centers and KPCA outperforms PCA and exhibit similar scores. However, k-Centers features have lower dimensionality. For the sake of completeness, the scores using the original feature space are also shown at Table 3. The best result of each score is marked using boldface.

6.1.5 Performance at Development Phase

During the development phase, an ensemble of three of the previous classifiers was tested. Results obtained are shown at Table 1, the top result for I3GO+ team are the results of an ensemble with a majority class for the three k-Centers features show at Table 3, while second and third rows at Table 1 are the resulting scores for EvoDAG based classifiers (see ^[16]) over k-Centers features. Table 4 shows the top three results for the proposed scheme. It is relevant to mention that the first one was the best score at the development phase.

6.2 An Approach based on Contrast Patterns (Second Place)

This section describes the proposal that obtained a second place in the RICATIM challenge. Additionally, this section shows the experimental results attained for each pattern-based approach using the original RICATIM database as well as the experimental results achieved but using a new RICATIM database representation, which was introduced by the I3GO+ team (see section 6.1).

6.2.1 Classification Based on Contrast Patterns

Over the past years, several classifiers have been proposed in the literature but nowadays obtaining a high accuracy is not the only desired characteristic for a classifier; experts in the application domain should understand the results associated to each class of the problem ^[44,25].

Contrast pattern-based classifiers are an essential family of both understandable and accurate classifiers. The main reasons are that contrast pattern-based classifiers provide patterns in a language that is easy for a human to understand, and they have demonstrated to be more accurate predictions than other popular classification models, such as decision trees, naive Bayes, nearest neighbor, bagging, boosting, and the support vector machine (SVM) ^[44,25].

A classifier based on contrast patterns uses a collection of contrast patterns to create a classifier that predicts a query object class ^[44].

A pattern is represented by a conjunction of relational statements, each with the form: [f # v_j], where v_j is a value in the domain of feature f_i and # is a relational operator from the set {=, ≠, ≤, >} ^[28,25]. For example, [Height ∈ [1.9, 2.1]] ˄ [Weight ≤ 120] ˄ [Agility = "Excellent"]] I a pattern describing a collection of basketball players. Let p be a pattern and T be (often) a (training) dataset; then, the support of p (with respect to T) is a fraction resulting from dividing the number of objects in T described (covered) by p by the total number of objects in T. Now, a contrast pattern (CP) for a class c is a pattern whereby the support of CP for c is significantly higher than any support of CP for every class other than c^[6,5,25].

Contrast pattern-based classifiers are used in various real-world applications, such as gene expression profiles ^[7], structural alerts for computational toxicology ^[33], gene transfer and microarray concordance analysis ^[27], characterization for leukemia subtypes ^[24], classification of spatial and image data ^[23], and heart disease prediction ^[21], in which they have reported effective classification results.

For building a contrast pattern-based classifier, there are three phases: mining, filtering, and classification strategy^[22,25].

Pattern Mining: This phase is dedicated to finding a set of candidate patterns by an exploratory analysis using a search-space, which is defined by a set of inductive constraints provided by the user. There are several algorithms for mining patterns, those that extract patterns from the tree (e.g., from decision trees) and those directly generating the patterns (e.g., rule miners)

Pattern Filtering: This phase focus on selecting a subset of patterns coming from a large collection of patterns produced in the preceding phase. For selecting a subset of patterns, a quality measure for patterns is commonly used ^[12].

Classification Strategy: This phase is responsible for searching the best strategy for combining the information provided by a subset of patterns and so builds an accurate model based on patterns. Usually, combining the support provided by each pattern into the subset is a widely used classification strategy ^[5,25].

6.2.2 Mining and Filtering Patterns

For RICATIM challenge, the team that won the second place have used an approach based on decision trees for mining contrast patterns. Using contrast pattern mining based on decision trees has two main advantages. First, the local discretization performed by decision tree miners with numeric features avoids doing a priori global discretization, which might cause information loss. Second, with decision trees, there is a significant reduction of the search space of potential patterns, since, even in longer paths of decision trees, a small proportion of candidate attributes are obtained. Moreover, the authors of ^[14] argued that creating the collection of extracted patterns from all the generated trees, reducing it through a filtering procedure, and obtaining an accurate model, is a simple procedure.

Two strategies for inducing decision trees were used with the aim of mining diversity contrast patterns. Diversity is an essential property for generating a collection of decision trees since it is possible that a collection of nearly identical trees cannot outperform any of their components.

In ^[13], an experimental comparison of different diversity generation procedures, was performed. From this work, two of the best strategies were selected: Bagging and Random Forest.

Bagging creates diversity by generating each tree with a bootstrap replicate of the training set. Since small changes in the training sample lead to significant changes in the model of a decision tree, Bagging is an excellent way to obtain a diverse collection. On the other hand, Random Forest creates different trees by selecting a Random Subset of attributes at each node. The best feature of the selected subset is then used to build the node. The success of Random Forest can also be explained because injecting randomness at the level of nodes tends to produce higher accuracy models.

Once the collection of trees is generated, all paths from the root nodes to all leaf nodes in each tree is considered as a pattern. This list of patterns could have duplicated patterns, which are eliminated. In addition, particular patterns are also removed. A pattern p is considered as particular if there is another pattern q in the collection such that the item set in p is a subset of the items in q. Thus, the final list of patterns has not duplicates or particular patterns.

6.2.3 The Pattern-based Classifier PBC4cip

As the pattern-based classifier, PBC4cip ^[25] was selected. This classifier reports competitive results in both problems with and without class imbalance. In the training phase, PBC4cip weights the sum of supports in each class, for all contrast patterns covering a query object, taking into account the class imbalance level. This strategy is different from traditional classifiers, which only sum the supports. The weighted expression is:

where |c| represents the number of objects belonging to the class c, |T| is the number of objects in the training dataset, P is the set of all the patterns for the class c, and support(p, c) is the support of the pattern p into the class c. This expression punishes the high sum of supports computed for the majority class.

In the classification phase, PBC4cip computes the sum of supports in each class for all patterns matching with the query object o. This sum is also multiplied by the weight w_c of its corresponding class c. Thus, the query object is classified in the class where it reaches the highest value, according to Equation (4).

6.2.4 Experimental Results

Table 5 shows the experimental results using two feature representations of the RICATIM database (see section 4). These results lead us to three main observations. First, the reduced representation allows the classifier to achieve a higher accuracy no matter the contrast patterns miner and whether pruning is used or not. Second, using pruning slightly outperforms the accuracy of every contrast patterns miner. Also, third, using Random Forest with the reduced representations consistently improves the accuracy of the Bagging miner.

Finally, it is recommended to use PBC4cip with patterns mined with Random Forest (pruning the trees) from the reduced representations of features. It is essential to mention that, in the recommended version, the Bhattacharyya coefficient ^[1] was used to evaluate candidate splits in the decision trees generation.

6.3 Phoenix Team (Third Place)

This section describes the methodology that obtained a third place in RICATIM challenge. The proposed approach was derived from two observations over the dataset. First, it was noted that several images were redundant in the training set, which could affect the performance of any machine learning algorithm. Second, it was noticed that the initial problem could be partitioned into smaller problems, where each problem consists of classifying the new pair (image, word) only by using the pairs with the same word between them. For these reasons, first, the proposed method filters the training set to enhance the performance of the classifier.

Second, the dataset has the inconvenience of high dimensionality. In this regard, the proposed method transforms the feature space into a low-dimensional space trying to group instances with similar content. Finally, for the classification phase k-Nearest Neighbor, was used trying to exploit the similarity between images.

In general terms, the algorithm is divided into these three stages (Figure 9). The first stage consists of an instance selection process to discard irrelevant and noisy data, only focusing on those images that share the same word representation. The second stage attacks the high dimensionality problem by performing a feature selection/generation process to get a new reduced feature space. It is important to note that the proposed approach takes advantage of one of the most known feature reduction techniques: Principal Component Analysis (PCA) and Linear Discriminant Analysis (LDA). The last stage consists of classifying the new sample with kNN; in this phase, the input of the kNN was both the representation of the new sample and condensed training set. More details of the stages mentioned above are described as follows:

Fig. 9 Diagram of the methodology proposed by the Phoenix team 

During the validation phase of the challenge, this method was evaluated with the following k values: 1, 3, 5, and 7. At the end of the validation phase, the best value for k was 3, which obtained a second place in the scoreboard at the validation phase.

6.4 Discussion

The academic challenge has produced positive results, achieving results on accuracy more significant than 0.8 and surpassing the proposed baselines. In a successful way, the methods proposed by the top-ranked teams, although diverse, they proved to be competitive at the exploitation of the proposed dataset. After analyzing these methods, we found that ensembles and feature preprocessing are necessary and helpful for handling noisy instances and for creating diversity. The most crucial property of classifier ensembles is the diversity; it is based on the rationale that a set of nearly identical classifiers cannot outperform any of their components.

In this regard, both, first and second places have proposed methods that include ensembles where they approach the problem differently. I3GO+ used ensembles of kNN and EvoDAG over an augmented data representation trying to extract the most information possible.

On the other hand, the second place used ensembles of decision trees by using a random approach, and after, this solution extracts several patterns from each decision tree. Furthermore, it is interesting that when both approaches are used in conjunction, then better results can be reached (i.e., see Table 5). In this case, first performing an augmentation in data using the I3GO+ approach, and second by using a practical approach to classify them as the approach introduced by the second place to discover patterns.

However, it is essential to highlight that the discovered patterns are complicated to understand by experts in the application domain because the representation of both datasets is based on word2vec representations. Nevertheless, although both mentioned proposals are effective, they have a limitation regarding the high computation power, being this characteristic alleviated it for the third place that proposed a method purely based on efficient machine learning techniques.

Regarding the benefits of this problem formulation, we foresee a promising approach for combining and taking advantage of multimodal data. It is possible to match representations from diverse sources, extending the capabilities of the main approaches for AIA. For these reasons, we plan to extend our dataset by considering new scenarios for generating synthetic instances from a word embedding perspective rather than an image perspective (presented here).