2.1 The LASSO logistic regression model

The least absolute shrinkage and selection operator, better known by its acronym LASSO, was designed for linear regression models, but its concept of shrinkage has been generalized to classification tasks through a combined framework with logistic regression that also performs variable selection. Formally speaking, consider a labeled dataset {(x ₁ , y ₁), .., (x _n , y _n )}, where xiT = (x _i1 , ..., x _ip ) is the vector of the ith individual’s features (also called predictors), and y _i is its corresponding label. We will assume that the objective (or response) variable Y is categorical and can take only two possible values (or classes), for instance, y _i = 1 if the ith individual is poor and y _i = 0 otherwise.⁵

The coefficients for the LASSO logistic regression are then obtained by solving the following penalized log-likelihood problem:

where the non-negative tuning parameter λ is chosen via crossvalidation. The fitted probabilities P^ (Y = 1|X = x _i ) are then computed with the estimated coefficients and the logistic transformation as in a logit model, and a simple classifier can be set according to the maximum probability criterion, that is, by assigning an observation x _i to the kth class if P^ (Y = k|X = x _i ) ≥ P^ (Y = k´|X = x _i ) for k ≠ k´ ∈ {0, 1}.

Note that in expression (1), λ controls the penalty in the optimization problem, and when it is equal to zero, the result gives the usual logit fit. This model has also the advantage of shrinking some coefficient estimates to zero, yielding a parsimonious model that is easier to interpret. Unlike traditional LASSO, its predicted probabilities do not fall outside the unit interval, which makes it a useful method for classification.

2.2 Random forest

The RF method is a well-known tree-based approach that aggregates a collection of classification trees to provide an accurate prediction for a new observation.⁶ Each tree T (x; θ) in the forest segments the predictor space X ₁ , ..., X _P into J mutually disjoint regions (or nodes) R ₁ , ..., R _J and classifies new observations according to the most frequent class k _j into the region in which they belong.⁷ Importantly, every region R _j depends on the minimum number of observations in each terminal node n _min or leaf), which is usually determined via cross-validation. Figure 2 depicts a classification tree with five terminal nodes in a three-dimensional feature space, and a binary response. Note that two advantages of classification trees are their graphic interpretability, which makes them easy to explain and understand compared to other black-box algorithms, and their good performance with highly non-linear decision boundaries. However, the method is very sensitive, since a small change in the training data can produce significant changes in the final tree. In other words, classification trees suffer from high variance.

Note: In this case the tree has three predictors, X1, X2, X3, two possible classes, k ∈ {1, 0}, and a partition of five regions {R1, ..., R5}.

Figure 2 Example of a classification tree T (x; θ) diagram 

Fortunately, this problem can be addressed by including a large number of trees in a single model. The RF classifier takes this idea and develops a powerful prediction algorithm with lower variance, in which similar training data yields almost identical results. Intuitively speaking, the RF iterative algorithm decorrelates its classification trees in two main steps. First, it draws different bootstrap samples from the training data to grow each tree, and second, it considers only a subset of variables to optimally split the final nodes.

We thus obtain a collection of decorrelated classification trees Tx;θbb=1B , best known as a RF, which takes the majority vote of its trees as the class prediction for a new observation x.⁸ There is a trade-off between RF’s variance and its interpretability: a decrease in variance is gained at the expense of an intuitive graphical representation of classification trees. The method has three main tuning parameters: the number of trees in the forest B, the number of variables considered at each split m, and the minimum node size on every leaf n _min. In practice, we use a value of B sufficiently large such that the error settles down (B ≥ 500 is generally sufficient), and the trees are grown deep (generally with n _min = 1; ^{James et al., 2013}). The tuning process is therefore centered around the parameter m, which controls the similarity among trees and prevents them from being highly correlated, resulting in a more reliable classifier.

The RF building process allows for simultaneous estimation of the error rate without the need to perform an extra validation process. Approximately one-third of the observations are left out in every bootstrap iteration; for every observation x _i it is thus possible to predict a response y^ _i using the majority vote of trees which did not include that observation in their training data. Then, the estimated misclassification rate, known as the out-of-bag error rate, would be given by ER=1n∑i=1nIyi≠y^i . In addition, the RF method quantifies importance of variable to the predictive power of the model using two different measures: the mean decrease in accuracy (MDA), and the mean decrease in Gini index (MDGI). The former results from averaging, for all B trees, the total decrease in accuracy due to a random permutation of the values of a given feature on the omitted observations in the corresponding trees, while MDGI is the average, for all B trees, of the total reduction in the Gini index over all splits of a given predictor.

2.3 Support vector machines

Another supervised learning method is support vector machines (SVM), whose main idea relies on establishing non-linear decision boundaries in the original feature space X ₁ , ..., X _P through linear boundaries in a transformed and augmented space.⁹ Like RF, SVM is considered a “black-box” algorithm because of its complex internal process. The SVM classifier exploits the idea of the natural separation produced by a hyperplane H (β, β ₀) = {x ∈ ℝ ^p : x ^T β + β ₀ = 0} , where new observations are classified according to the side on which they fall.

More specifically, a kernel function K : ℝ ^p × ℝ ^p → ℝ₊ is chosen to characterize the transformation h = (h ₁ , ..., h _m ) : ℝ p → ℝ ^m that will expand the original p-feature space into a higher-dimensional one (m ≥ p), where K and h are related by K (x, x´) = h(x) ^T h(x), ∀x, x´ ∈ ℝ ^p .¹⁰ After obtaining the basis functions h _j , j = 1, ..., m, SVM finds the optimal hyperplane using the transformed observations as training data: {(h(x ₁), y ₁), .., (h(x _n ), y _n )}. An important characteristic of this method is that it depends only on those observations that are closest to the hyperplane and lie exactly on its margin; these are known as support vectors. Figure 3 shows an example of linearly separable training data in a transformed bidimensional feature space, and its corresponding optimal hyperplane and support vectors.

Note: Linearly separable training data, in a transformed bidimensional space. Source: ^{Lantz (2015)}.

Figure 3 Optimal hyperplane and support vectors 

Note that, depending on the kernel used, there might be different support vector machines even with the same data. Table 1 shows some kernels popular in the ML literature. As expected, the radial basis and dth-degree polynomial kernels result in more flexible decision boundaries than the linear kernel. In addition, SVM has a cost parameter C, which controls the smoothness of the hyperplane boundary, and together with the kernel’s parameters is selected through crossvalidation using a tuning grid.

Nonetheless, SVM suffers from a major drawback: it can be very slow to train, particularly if the input dataset has a large number of features or observations (^{Lantz, 2015}). ^{Steinwart and Thomann (2017)} suggest an alternative approach, in which the feature space is split into spatial cells where local SVMs with the same kernel are trained, thus considerably decreasing runtime with large samples. Furthermore, cross-validation is performed on every cell, resulting in the same number of optimal tuning parameters as cells. Then, to classify a new observation x, the SVM of the cell to which x belongs is used. According to the authors, the solution of the cell approach is similar to that of SVM.

3.1 Selected variables

Comparable features of the MCS-ENIGH and the ENOE are divided into four sets of variables. The first is the sociodemographic set, which consists of state of residence, binary variables for rural/urban and gender, and continuous variables for age and household size. The second is the set of economic characteristics, which includes a categorical variable for individual economic status, continuous variables for household hours of work per week and real labor income per month, and an indicator variable that equals one if household per capita real labor income is below the minimum welfare line (mwl), and zero otherwise.¹³ The third set consists of educational variables, formed by an indicator variable for not in school, a categorical variable for educational level, and an indicator variable for educational lag. Finally, there is a set of binary variables for access to public health care (Instituto Mexicano del Seguro Social, IMSS; and Instituto de Seguridad y Servicios Sociales de los Trabajadores del Estado, ISSSTE).

Table 3 summarizes these predictors and shows the input variables from MCS-ENIGH and ENOE that are used to create them. The ENOE input variables are taken directly from INEGI releases, while the MCS-ENIGH input variables, with the exception of household size and hours worked per week, are constructed following the official CONEVAL methodology for the multidimensional poverty measurement.¹⁴ Although the number of predictors is relatively small, it is worth noting that they not only describe the dimension of household income, but also individual educational trajectory (especially educational lag) and household access to health care. The collected predictors consider three out of nine of the dimensions established in the General Law of Social Development for multidimensional poverty to expand the one-dimensional approach used to identify labor poverty. The objective variable for poverty is constructed in the same way as in ^{CONEVAL (2014)} and included in the training data. This variable identifies poor and non-poor individuals according to the multidimensional poverty measurement methodology, and is the dependent variable in the prediction data since it is not possible to obtain it deterministically from the ENOE.

3.2 Training and prediction data

Some ML algorithms display high variance: their output may vary widely with small changes in the training data. They may produce unexpected estimates if the distribution of characteristics differs substantially from one survey to the other. It is thus important to identify similarities and differences in the proportions and dynamics of the series in both training and prediction data.

Descriptive statistics for both data sources are shown in Appendix A.¹⁵ Tables A.1 and A.3 to A.8 show descriptive statistics for the household-level variables from training and prediction data. Mean household size and average hours worked per week have remained relatively stable over time, but only household size is comparable between surveys; households in the prediction data work an average of eleven hours less per week than those in the training data. Mean labor income is also not comparable between datasets (and thus the proportion of households with per capita labor income below the minimum welfare line is also not comparable). Figure 4 shows the contrast between average real labor income of Mexican households in training and prediction data over time; the income in the training data is always higher. The confidence intervals suggest the existence of statistically significant differences between the biennial means, although they exhibit similarly increasing behavior since 2010. Table A.15 summarizes the t-test results for this variable, where the null hypothesis of mean equal income between surveys within their corresponding biennial window (from 2008 to 2009, from 2010 to 2011, etc.) is rejected at all significance levels, meaning that the predictions with ML algorithms will be affected to some extent by the income patterns of the training data. Additionally, the methodological changes inherited by the 2016 MEC bring about a noteworthy increase in labor income dispersion over previous years, mainly due to the attempt by INEGI to improve its household income data by means of stricter capture and verification field criteria; these changes affect the income distribution in the 2016 training data. This pattern is not displayed in the 2018 MEC.

Note: 95% confidence intervals are depicted with dotted lines. Source: Author’s calculations using deflated income data from MCS-ENIGH and ENOE, from 2008 to 2019, without expansion factors.

Figure 4 Average labor income of Mexican households 

Similarly, tables A.2 and A.9 to A.14 show individual-level descriptive statistics from the training and prediction data. It can be seen that most variables display steady behavior that does not differ substantially over time or across datasets. However, the fraction of individuals in rural localities in the training data is, on average, 12 pp greater, which in turn means a smaller number of people affiliated with IMSS and a higher level of educational lag. There is a significant decrease in the training data on the number of individuals who do not have any access to health care services, from 34.8% in 2008 to 14% in 2018; in the prediction data this variable hovers around 25%. Finally, the objective variable of multidimensional poverty is balanced over time: neither poor nor non-poor individuals exceed 60% of the total number of observations in the training datasets, and the number remains very close to 44%. These numbers are equivalent to the official poverty estimates using the corresponding expansion factors.

4.1 Poverty estimates of the LASSO logistic model

As described in section 2.1, the LASSO logistic model can handle several regressors in a logistic regression framework and select only a subset of them by shrinking the coefficients of some variables to zero. In this analysis, quadratic terms for continuous variables and interactions among categorical variables (except for the state of residence variable) are added to the baseline specification in (2), and the algorithm decides which variables to keep. The penalty term λ is determined using a tuning grid and 10-fold cross-validation.¹⁸

Table 5 presents a summary of the estimation results, where the algorithm drops an average of one-tenth of the variables each year. This amount is partially determined by the small value of the penalty parameter λ: the larger it is, the small the number of non-zero coefficients in the model. Although some coefficients are close to those in table 4, there is an average 1.8 pp gain in accuracy over the logit out of-sample performance. Even if the interaction and quadratic terms are not included, the LASSO logistic model outperforms the logit by 0.7 pp, as shown in table B.2. In other words, adding more variables to the baseline specification in this algorithm to cover possible nonlinearities yields a small improvement in the misclassification rate.

However, the LASSO logistic ex-post and ex-ante estimates provide closer approximations to the multidimensional poverty rate than labor poverty in the sample analyzed, as shown in figure 6, panel (a). Both poverty measures show an increasing trend from 2008 to 2010, where ex-post estimates increase by 2.3 pp during the world financial crisis. LASSO’s multidimensional poverty estimates then hovers around 48%, while labor poverty continues to increase until 2015.

Note: Solid and dashed gray lines in panel (a) represent ex-post and ex-ante LASSO logistic estimates, respectively, while gray areas indicate where ex-ante estimates are seen before their updated ex-post version. Ex-ante and ex-post poverty rates in panel (b) are the averages of their respective quarterly estimates that fall within each publication year of multidimensional poverty. LASSO logistic estimates are computed using the maximum probability criterion. Source: Author’s calculations using the MCS-ENIGH and ENOE from the first quarter of 2008 to the last quarter of 2019, and the glmnet package (version 4.1-1) in R.

Figure 6 Multidimensional, labor, and LASSO logistic poverty rates 

For the rest of the time window, the two rates show similar decreases, with the LASSO logistic estimates less volatile than those for labor poverty. The ex-ante and ex-post estimates are further apart during 2018, revealing some degree of sensitivity to the methodological changes in the 2016 and 2018 training data. Finally, according to the quarterly averages in figure 6, panel (b), ex-ante and ex-post measures overestimate the biennial multidimensional rate by an average of 1.2 pp and 2 pp, respectively.¹⁹

4.2 Poverty estimates of random forest

Similar to the baseline analysis, the random forest (RF) method uses all of the variables in table 3 to train the ML algorithm, and it uses the optimal cutoffs in table 4. As described in section 2.2, the RF method has three main tuning parameters: the number of trees in the forest (B), the number of possible predictors at each split (m), and the minimum size on every terminal node (n _min). For this application, the parameters B and n _min are fixed at 500 and 1, respectively. RF calibration is thus focused on parameter m, for which a grid of possible values is built and tested using out-of-sample performance in order to choose the best model.²⁰ Optimal values of the tuning parameter m and out-of-bag error rates of their corresponding fitted models are shown in table 6. The optimal parameters are very close to the total number of predictors, and there is a significant improvement in RF of out-of-sample performance, exceeding the LASSO logistic and logit models by an average of 7.7 pp and 9.4 pp, respectively. This result suggests that decision boundaries are highly non-linear, and that the RF structure helps to assimilate them better than a traditional regression approach.

The RF quarterly poverty rate is depicted in figure 7, panel (a), which shows ex-post levels that are close to the official poverty rate. The increasing trend during the world financial crisis shows a jump of 2.8 pp, a magnitude similar to that of the benchmark model. From 2010 to 2015, it hovers around 46%, decreasing toward the end of the period. The RF ex-ante quarterly predictions are close to their corresponding ex-post updates, with an average absolute deviation of 0.5 pp. The quarterly averages in panel (b) show an interesting result: the RF ex-post estimates generally improve their ex-ante poverty predictions, in contrast to the LASSO logistic estimates. Even in 2016 and 2018, RF adjusts better to the methodological changes in the training data than the logit model.

Note: Solid and dashed gray lines in panel (a) represent ex-post and ex-ante RF estimates, respectively, while gray areas are spans where ex-ante estimates are seen before their updated ex-post version. Ex-ante and ex-post poverty rates in panel (b) are the averages of their respective quarterly estimates that fall within each publication year of multidimensional poverty. RF estimates are computed using the optimal cutoffs for the benchmark model, in table 4. Source: Author’s calculations using the MCS-ENIGH and ENOE from the first quarter of 2008 to the last quarter of 2019, and the randomForest package (version 4.6-14) in R.

Figure 7 Multidimensional, labor, and RF poverty rates 

In spite of being a black-box algorithm, RF offers a picture of what is happening inside through the mean decrease in accuracy (MDA) and the mean decrease in Gini index (MDGI) as measures of variable importance. These measures are shown in relative terms in figure 8. According to the MDGI metric, per capita real labor income below the mwl (pob) and labor income (lab inc) are the most important variables over the six years. For MDA, the four most important variables are household size (hsize), state of residence (state), the indicator variable of rural localities (rururb), and IMSS affiliation (imss). Gender, educational level (ed lev), not in school (no school), no health insurance (no hserv), and economic status (eap) are the variables with least importance over the six years. Surprisingly, access to social security has greater importance for the predictive power of the algorithm than educational level or school attendance, highlighting the critical role of formal employment in predicting poverty.

Note: Both mean decrease in accuracy (MDA) and mean decrease in Gini index (MDGI) are expressed relative to their corresponding largest measure and sorted by MDA, for each year of training data. Source: Author’s calculations using the MCS-ENIGH for 2008 to 2018, and the randomForest package (version 4.6-14) in R.

Figure 8 Variable importance for the random forest method 

4.3 Poverty estimates of support vector machines

Like the RF method, SVM analysis uses all the features of table 3. As described in section 2.3, the training times of local SVMs are considerably less expensive with large samples than training a single SVM, so this exercise, with approximately 230 000 training observations per year, uses the alternative SVM cell approach of ^{Steinwart and Thomann (2017)}. The radial basis transformation is employed as the default kernel function, leading to two tuning parameters, C and λ, for which a tuning grid is built and tested, using 10-fold cross-validation to pick the best model for every cell.²¹

Table 7 shows summary results of the SVM cell calibration. There are more cells in 2016 and 2018 than in other years, mainly due to the greater number of observations and, to some extent, to the difference in the distribution of training data. The data in 2010 yields only an increase in average cell size, without affecting the number of cells. For every year most cells pick a small optimal value for λ, which is related to classifiers with high variance given that their kernels would be large, and it would thus have a greater impact on the decision function h(x)^T β^ + β^0 ; higher values of λ are related to more non-linear fits and more bias classifiers. The mode of the cost parameter C remains constant over time, with a value of 100 000. According to ^{Hastie et al. (2009)}, a large value of C encourages a large value of ||β||, which brings about a narrower margin and fewer support vectors, so the model tends to overfit the data. Roughly two-fifths of the observations are support vectors on every cell, where the results for 2016 suggest that the greater the dispersion in the training data, the more support vectors are needed. Finally, the mean error rate of the SVM cells is taken as the estimate for the model’s out-of-sample performance, which is similar to the LASSO logistic performance, slightly better than the benchmark model, but not as good as RF.

Figure 9 shows the poverty rates using the optimal models of SVM cells. At first glance, the ex-post curve of SVM in panel (a) overestimates the multidimensional poverty rate during the first seven years of analysis, showing a decreasing trend from 2010, but even so, it is closer to the official rate than the labor poverty rate. The alternative SVM approach estimates the greatest increase of all four algorithms in the world financial crisis, 3.8 pp, which is still far from the labor poverty increase of 6.4 pp. The SVM ex-ante poverty predictions are the most distant of all the algorithms from their corresponding ex-post updates, with an average absolute distance of 1.36 pp. Quarterly averages of SVM estimates in panel (b) reaffirm that although the ex-post estimates generally improve the ex-ante predictions, they are behind the benchmark model and RF.

Note: Solid and dashed gray lines in panel (a) represent ex-post and ex-ante SVM estimates, respectively, while gray areas indicate the spans of ex-ante estimates before their updated ex-post version. Ex-ante and ex-post poverty rates in panel (b) are the average of their respective quarterly estimates that fall within each publication year of multidimensional poverty. Source: Author’s calculations using the MCS-ENIGH and ENOE from the first quarter of 2008 to the last quarter of 2019, and the liquidSVM package (version 1.2.1) in R.

Figure 9 Multidimensional, labor, and SVM poverty rates 

4.4 Poverty dynamics in the COVID-19 pandemic

As the COVID-19 pandemic evolves, the alleviation of poverty has been threatened by disruptions in living standards and the economy.²² This crisis is an excellent example of a situation where timely information about multidimensional poverty is fundamental to targeting policy. In order to estimate the dynamics of the pandemic, ex-ante poverty rates are provided for 2020 using the fitted models with the 2018 training data. Unfortunately, INEGI canceled the ENOE during the second quarter of 2020, but implemented a hybrid version, ENOEN, in the third quarter, 72% of whose sample is compiled from face-toface interviews and 28% from telephone interviews.

Figure 10 shows LASSO logistic, RF, and SVM quarterly ex-ante estimates during 2020 and their corresponding ex-post poverty rates along with the labor poverty series. The pandemic seems to affect Mexican poverty after the first quarter of 2020, in line with the first national lockdown campaign in late March. Although the impact cannot be fully estimated since there is no information for the second quarter, the increase from 2020 QI to 2020 QIII is likely a lower bound of the true effect, considering that other macroeconomic variables, such as the unemployment rate and Mexican GDP, were at their worst levels in 2020 QII.²³

Note: Because of the COVID-19 pandemic, the ENOE was canceled in the second quarter of 2020, and replaced with a hybrid version (ENOEN) in subsequent quarters. Source: Author’s calculations using the MCS-ENIGH, ENOE, and ENOEN from the first quarter of 2008 to the last quarter of 2020.

Figure 10 Ex-post poverty series of machine learning and 2020 ex-ante estimates 

Ex-ante numbers are presented in table 8. Labor poverty shows the highest increase from 2020 QI to 2020 QIII, approximately 8.9 pp, while the other increases range from 3.8 pp to 6.4 pp. These increases are an average of 2.5 pp larger than the corresponding impacts of the world financial crisis, though in 2020 the increasing trend is of shorter duration. The preferred projections for the 2020 multidimensional poverty rate are the average of the third and fourth quarters of 2020, with which RF predicts the lowest poverty rate (40.92%) and the LASSO logistic model the highest (44.32%). Either way, it is highly likely that the multidimensional poverty rate for 2020 will be greater than in 2018. This would also depend on further methodological changes in CONEVAL’s poverty identification strategy.

4.5 Model assessments

All quarterly ex-post poverty rates seen in figure 10 are closer to the official biennial levels than the ITLP labor poverty rate, and they display similar behaviors over the period analyzed. It seems, however, that none of the ML algorithms greatly exceeds the baseline logit model in approximating the multidimensional poverty rate, so what is the real advantage of having more complex classification models? To answer this question, it would be helpful to keep in mind an observation made by Olivier Dupriez, Lead Statistician at the Development Data Group, at the 2018 Machine Learning and the Future of Poverty Prediction conference hosted by the World Bank in Washington, D.C. Dupriez presented an empirical comparison of several ML classification algorithms using data from Indonesia and Malawi, and highlighted the importance of assessing ML algorithms with multiple performance metrics.²⁴ Despite making good predictions of the official poverty rate, he noted, there is no guarantee that the model correctly classifies poor individuals: “The people identified as poor might be in the right numbers but they are not the right people”.

In this sense, the four models are fitted with default settings for their tuning parameters and assessed via 10-fold CV using the accuracy metric (basically 1-Error rate), and six complementary metrics: recall, specificity, precision, negative predictive value (NPV), F1 score, and kappa (κ). None of these metrics is better than the others; they simply evaluate different things and provide different, but equally important information.²⁵ Local rankings are then made for every metric and year to establish the reliability and robustness of the algorithms, and finally they are all summarized by their mean rank.

Table 9 shows these results, where RF shows the highest scores over the period analyzed, followed by SVM, the LASSO logistic model, and logit, reinforcing the findings already presented. For 2016 in particular, RF leads the rest, but the logit and LASSO overtake SVM in five metrics, possibly because of the dispersion in the 2016 training data. Interestingly, if a given model is assessed with the same metric over time, 2016 and 2018 turn out to be the years with the lowest scores, while all of them show their best performance in 2008 and 2010. The effect, however, of the 2016 methodological changes on the performance of the algorithm is more perceptible in SVM, where for instance the recall score falls from 86.7% in 2014 to 72.6% in 2016. That is, the proportion of the truly poor who are correctly classified by SVM decreases by approximately 14 pp from 2014 to 2016.

The SVM, the LASSO logistic regression, and the logit model show similar performance in the multidimensional poverty classification task in Mexico, while RF outperforms all of them. Although the logit performs well in targeting the official biennial poverty rate, the real advantage of using RF lies in its ability to correctly identify poverty patterns that are imperceptible with other algorithms, as well as its flexibility and robustness to different training data distributions and methodological changes in the poverty detection mechanisms. A state-level analysis rather than a national one, as well as consideration of the correlations between predictors, could be additional ways to explore the strengths of RF.