1 Introduction

In recent years, class imbalance problems have emerged as one of the challenges in data mining community [²⁸]. This kind of data appear in many real-world classification problems like fault diagnosis [²⁹], anomaly detection [²⁴], medical diagnosis [¹⁹], circuit breaker maintenance diagnosis [²¹], among others. In binary classification, this problem occurs when the number of instances of one class is much lower than the instances of the other class. The overrepresented class is called the majority or negative class, and the other class the minority or positive class. Traditional classifiers generally tend to classify almost all instances as negative (i.e., the majority class) [²²].

Many techniques for dealing with class imbalance have emerged: those that modify the data distribution by preprocessing techniques (data level solutions) [², ⁶, ¹⁸, ²³], those at the level of the learning algorithm which adapt a base classifier to deal with class imbalance (algorithm level solutions) [⁴, ¹⁶, ²²], those that apply different costs to misclassification of positive and negative samples (cost-sensitive solutions) [⁸, ¹⁸, ²⁵, ²⁷], and ensemble based solutions that combine the previous solutions by means of an ensemble [¹⁰].

In this paper, we present an ensemble solution to classify imbalanced using the Imbalanced Fuzzy-Rough Ordered Weighted Average Nearest Neighbor (IFROWANN) algorithm [²²]. This classifier combine fuzzy rough set theory and ordered weighted average aggregation to taking into account the imbalance of the classes. Each strategy of weight and fuzzy-rough indiscernibility relation produce different classification results. It is difficult to know the best configuration to obtain the best classification result in different datasets.

The ensemble generates the same classifier (IFROWANN), but with different weight strategy and different fuzzy relation. Final classification is given by one of these strategies. To evaluate the quality of our model, we have carried out an extensive experimental analysis on a collection of 66 imbalanced datasets with different imbalance ratios (IR) (between 1.82 and 129.44), originating from the SCI²S site (sci2s.ugr.es). In the experiments, we have compared our algorithm with the base IFROWANN proposal to show that it is similar to it without to test the weight and fuzzy relation strategies. Also, we have compared the ensemble with a set of 16 state-of-the-art methods designed for imbalanced classification, obtaining that it is better positioned than 14 and without significant differences with the rest. To assess the classification performance, we have use the Area Under the Curve (AUC) metric [²²], and the significance of the results has been supported by the Friedman tests and post hoc Holm procedure.

The remainder of this paper is organized as follows. In Section 2, we introduce the imbalanced classification problem, including an overview of the state-of-the-art methods for solving it. In Section 3, we recall the standard IFROWANN algorithm. In Section 4, we introduce the WIFROWANN algorithm and present the proposed strategies to instance classification. In Section 5, we discuss the setup of the experimental study. In Section 6, we present and discuss the results. In Section 7, we draw some conclusions and future work about the study.

2 Classifications in Imbalanced Datasets

In this section, we first introduce the problem of imbalanced datasets in classification. Then, several techniques to address the class imbalance problem are presented. In binary classification, it is considered a set of data samples 𝑈, characterized by their values for the set 𝐴 = {𝑎₁,..., 𝑎_𝑚} of attributes. Moreover,𝑈 = 𝑃 ∪ 𝑁 where 𝑃 represents the positive class, and 𝑁 represents the negative class. We denote 𝑝 = 𝑃 ∨, 𝑛 = 𝑁 ∨, and 𝑡 = 𝑈 ∨ 𝑝 + 𝑛. The imbalance rate is then defined as 𝑛/𝑝. The imbalanced classification problem can be tackled using four main types of solutions:

1) Sampling (solutions at the data level) [¹⁸]: This kind of solution consists of balancing the class distribution by means of a preprocessing strategy. Techniques at data level are undersampling, oversampling and hybrid methods. Some examples of this technique are Synthetic Minority Oversampling Technique (SMOTE) algorithm [⁶], SMOTE-ENN [²] and SMOTE-RSB* [²³].

2) Design of specific algorithms (solutions at the algorithmic level) [⁴, ¹⁶]: Traditional classifier is adapted to deal directly with the imbalance between the classes. This is the case of Imbalanced Fuzzy Rough Ordered Weighted Average Nearest Neighbor (IFROWANN) [²²].

3) Cost-sensitive solutions [⁸, ¹⁸]: These kind of methods incorporate solutions at data level, at algorithmic level, or at both levels together. They try to minimize higher cost errors where the cost of misclassifying a positive instance should be higher than the cost of misclassifying a negative one. The main examples are Cost-sensitive C4.5 decision tree (CS-C4.5) [²⁵] and Cost-sensitive support vector machine (CS-SVM) [²⁷].

4) Ensemble solutions [¹¹]: Usually combine an ensemble learning algorithm and one of the techniques above, specifically, data level and cost-sensitive. For example the EUSBOOST algorithm [¹⁰], which uses evolutionary undersampling.

Next, we will discuss the evaluation of machine-learning algorithms in imbalanced domains.

3 Imbalanced Fuzzy-Rough Ordered Weighted Average Nearest Neighbor (IFROWANN)

In this section, we introduce the IFROWANN classification algorithm proposed in [²²]. This algorithm is a variation of the Fuzzy-Rough Nearest Neighbor (FRNN) algorithm [¹⁷]. In order to predict the class of a new instance 𝑥, the IFROWANN algorithm calculate the memberships degrees of 𝑥 to the fuzzy-rough lower and upper approximation of each class and assigns the instance to the class with higher degree.

More precisely, let 𝑈 be the universe, I an implicator, T a t-norm defined by I(𝑎, 𝑏) = 𝑚𝑎𝑥(1 − 𝑎, 𝑏) and T(𝑎, 𝑏) = 𝑚𝑖𝑛(𝑎, 𝑏),for 𝑎, 𝑏 in [0,1], and 𝑅 a fuzzy relation that represents approximate indiscernibility between instances, WPl and WNl OWA weight vectors.

An implicator I is a [0,1]² → [0,1] mapping that is decreasing in its first argument and increasing in its second argument, and that satisfies I (0,0) = I(0,1) = I(1,1) = 1, and I(1,0) = 0. The membership degrees to the positive class 𝜇_𝑃(𝑥) and negative class 𝜇_𝑁(𝑥) are defined by:

where P¯WPl(x) is the lower approximation to the positive class 𝑃 under the fuzzy relation 𝑅 with the OWA weight vector WPl, and N¯WNl(x) is the lower approximation to the negative class 𝑁 under 𝑅 and the OWA weight vector WNl for negative class. 𝑥 is classified to the positive class if 𝜇_𝑃(𝑥) ≥ 𝜇_𝑁(𝑥); otherwise, it is classified to the negative class. The lower approximation for the negative and positive class are calculated by the equation 3 and 4:

where OWA is the operator to take into account the imbalance. Given a sequence 𝐴 of 𝑡 real values 𝐴 = ⟨𝑎₁, …, 𝑎_𝑡⟩, and a weight vector 𝑊 = ⟨𝑤₁, …, 𝑤_𝑡⟩ such that 𝑤_𝑖 ∈ [0,1] and ∑i=1twi=1 the OWA aggregation of 𝐴 by 𝑊 is given by: OWAw(A)=∑i=1twibi where 𝑏_𝑖 = 𝑎_𝑗if 𝑎_𝑗 is the ith largest value in 𝐴.

IFROWANN algorithm has two fundamental factors: indiscernibility fuzzy function and weight vector. There is no weight and fuzzy relation strategies to obtain the best results for any 𝐼𝑅. In order to obtain the best result of classification we must to try with several combinations of weight vectors and fuzzy relations. For this reason, this paper proposes an ensemble with the IFROWANN algorithm, which combine different weight strategies and fuzzy relations.

4 Extending the IFROWANN Algorithm

The Wrapper Imbalanced Fuzzy-Rough Ordered Weighted Average Nearest Neighbor (WIFROWANN) is an ensemble that uses the IFROWANN algorithm in base, with different configurations of OWA weight vector and fuzzy relations. The description of the algorithm is divided in two main parts. First, we discuss the configurations to run the classifier, and second in Section 4.1, the different strategies to output the ensemble classification of the new instance.

To predict the class of a new instance 𝑥, the ensemble creates a set of classifiers 𝐶 = {𝐶₁, …, 𝐶_𝐿}, each one with a weight strategy and an indiscernibility fuzzy relation strategy too. Given a vector with different weight strategies 𝑊 = [𝑊₁, …, 𝑊_𝑛] and a vector with different configurations of indiscernibility fuzzy-rough relations 𝑅 = [𝑅₁, …, 𝑅_𝑚]; the ensemble build 𝑊 × 𝑅 classifiers. Each classifier 𝐶_𝑖computes the membership degree to the positive and negative class and the final result is given by the strategy of fusion or selection chose by us. Next section we explain these strategies. Weights strategy define which weight vector the classifier 𝐶_𝑖 will use to calculate the belong degree to the lower approximation to the positive and negative class in equations 3 and 4 respectively. The different weight vectors proposed in [²²] are:

where 𝑝 = |𝑃| and 𝑛 = |𝑁|, are the number of instances of the Positive 𝑃 and Negative 𝑁 classes.

A variation of the WPl1 vector is: Given 0 ≤ 𝛾 ≤ 1:

where 𝑟 = [𝑝 + 𝛾(𝑛 − 𝑝)].

In particular, the first 𝑝 positions in WPl can be put to 0, taking into account that this correspond to the highest values of I (𝑅(𝑥, 𝑦), 𝑃(𝑦)) = 𝑚𝑎𝑥(1 − 𝑅(𝑥, 𝑦), 0) = 1 − 𝑅(𝑥, 𝑦), and thus to the p positive samples in the training data. The remaining 𝑛 positions in the weight vector WPl correspond to the instances in 𝑁. In a completely analogous way, the weight vectors WNl, can we put 0 to the first 𝑛 positions too.

We consider the following three alternatives for defining the fuzzy relation 𝑅 Average t-norm, Łukasiewicz t-norm and Minimum t-norm 𝑅_𝐴𝑣, 𝑅_𝑇𝐿𝑜𝑅𝑀𝑖𝑛:

where Łukasiewicz t-norm is defined by, for 𝑢₁, 𝑢₂,..., 𝑢_𝑚 in [0,1]:

and 𝑅_𝑎is the similarity function between 𝑥 and 𝑦 instances with the attribute 𝑎.

For a quantitative attribute and nominal attribute, we use the equations respectively:

We explain below in detail the strategies followed to build the ensemble.

5 Experimental Setups

In this section, we describe the experimental framework used to validate our proposal, including the benchmark datasets, the state-of-the-art methods, and the statistical tests used in order to carry out the performance comparison.

6 Experimental Results

In this section, we present the results of our experimental analysis.

In Section 6.1, we compare our proposal with IFROWANN baseline methods and its best configurations. Next, in Section 6.2, we compare the algorithms with the state-of-the-art methods for imbalanced classification.

6 Conclusions

In this paper, we have presented the WIFROWANN method, a new ensemble level solution for two-class imbalanced classification problems that is based on the IFROWANN algorithm. In particular, the W-All, W-Weights, W-SF, W-W6W4W5 and W-F2 variants of WIFROWANN method, considering six weighting strategies and no weighting strategy, combined with three different indiscernibility fuzzy relations. Our experimental results and statistical analysis have shown that:

W-F2 obtains better AUC mean respect to IFROWANN and the best position in the Friedman ranking for low imbalance datasets. Holm’s procedure shows that this variant present significant difference with AV-W6 for low imbalance and no significant difference with the same IFROWANN configuration for high imbalance.

WIFROWANN outperforms 14 state-of-the-art representative algorithms that cover preprocessing level, cost-sensitive, and ensemble solutions specifically designed for imbalanced learning and similar behavior with IFROWANN, SMOTE+RSB and EUSBOOT methods.

For future work, we will consider extend WIFROWANN method for multiclass and multi-labels classification problems.