texto en 
Inglés (pdf)
Artículo en XML
Referencias del artículo
Traducción automática
Enviar artículo por email
Citado por SciELO 
Similares en
SciELO 
Permalink
Highlights:
Thirteen indicators of the National Forest Program (2012-2018) were evaluated qualitatively.
Level of compliance for the following criteria was evaluated: clarity, relevance, monitoring and adequacy.
Compliance with the criteria of the indicators was assessed appropriately using fuzzy logic.
The indicator "Rate of change of timber forest production" had the best evaluation.
Introduction
In the area of public policy, there is a variety of indicators used for the evaluation of different stages and outcomes of interest (De la Cuesta, Pardo, & Paredes, 2015; Guayanlema, Fernández, & Arias, 2017; Guillen, Badi, Garza, & Acuña, 2015). Users and proponents notoriously recognize the need for quality indicators. In Mexico, the literature reports very few studies that address compliance levels that the criteria must satisfy. One of these studies was conducted by the National Council for the Evaluation of Social Development Policy (CONEVAL, 2014), and another was performed by the World Bank (2016).
The CONEVAL's evaluation proposal consists of two stages: integral assessment and specific assessment of results indicators. This process involves the participation of public servants, those responsible for the programs to which the indicators correspond, and representatives of the federal public sector. The first stage consists of determining whether indicators meet the minimum design criteria: clarity, relevance, monitoring, and adequacy. Two methodological support sheets are used: the results indicator matrix sheet and the indicator evaluation sheet. All questions used in the sections of these sheets are binary (yes/no). The second stage (specific assessment) is intended to determine the compliance with the indicators' minimum consistency criteria. Questions included are also binary, and the evaluator is asked to add the reason for his/her answer. Furthermore, the CONEVAL performs a statistical validation by applying a Rasch metric model used in a wide range of situations to measure subjective valuation. Finally, based on the results of these two stages and observations derived from statistical validation, an approval report is issued, if applicable, for the set of indicators of the corresponding program.
On the other hand, the instrument proposed by a World Bank Group, “Tool for evaluating the quality of indicators,” like the CONEVAL evaluation process, involves a broad group of invitees with sufficient experience in the sector to which the programs being evaluated with the indicators in question correspond. The process includes three parts: (a) evaluation of the quality of indicators, (b) evaluation of the quality of indicator targets, and (c) evaluation of the quality of information sources. Most of the data collection forms have four response options: 0 when the question's assumption is not satisfied, 3 when there is an intermediate level of satisfaction, 5 when the level of satisfaction is null, and 99 when it is not applicable. Finally, averages are calculated and converted to percentages with which the quality of the indicators analyzed is rated.
Evaluations and analyses of indicators have been conducted in several areas of knowledge. For example, Kühnen and Hahn (2017) reviewed trends, consistencies, inconsistencies, and gaps in investigating social life cycle assessment indicators. Whitehead (2017) analyzed and prioritized a list of sustainability indicators. Vignesh et al. (2017) evaluated a set of pollution indicators. The list of this type of indicator assessment includes Evans, Strezov, and Evans (2009), and Chenoweth (2008).
Evaluation processes that frequently use quantitative and qualitative information commonly correspond to a complex framework with a high degree of uncertainty. which has traditionally been approached from probabilistic models in classical decision theory; however, uncertainty involves aspects with a non-probabilistic character, because they are related to imprecision and vagueness of the meaning perceived by those performing the evaluation (Martínez, Rodríguez, & Herrera, 2015; Torres & Tranchita, 2014). When the information used in the evaluation process is qualitative, the use of linguistic variables represents a good option to model it (Herrera & Martínez, 2001a). In this respect, “Human beings possess two remarkable capabilities; the first is the ability to communicate, reason, and make rational decisions in an environment of imprecision, uncertainty, and incomplete information. The second is the ability to execute a wide variety of physical and mental tasks without any measurement and no calculations” (Mendel et al., 2010). Also, there are examples of successful applications in different fields where assessments have been performed using fuzzy logic (Gothwal & Raj, 2019; Montignac et al., 2015; Pirlot, Teghem, Ulungu, Bulens, & Goffin, 2015; Wang, Yang, & Cheng, 2019).
The present research objective was to apply a multigranular approach based on the 2-tuple fuzzy linguistic model to the evaluation of 13 indicators of the National Forestry Program established in the system of social policy indicators derived from the National Development Plan 2012-2018 of Mexico.
Materials and methods
The method proposed in this study for the evaluation of indicators differs from those already mentioned. It uses tools developed in the framework of fuzzy logic, specifically, the 2-tuple fuzzy linguistic representation model proposed by Herrera and Martinez (2000) and the extended linguistic hierarchy approach (ELH) developed by Espinilla, Liu, and Martínez (2011).
The evaluation of indicators is framed in a decision-making process. In this sense, Martínez et al. (2015) consider that such a process comprises at least the following five phases: intelligence, modeling, information gathering, analysis, and selection. Figure 1 describes the proposed process for evaluating forest policy indicators based fundamentally on the phases mentioned above with some modifications.
Figure 1 Forest policy indicator evaluation process used in this study. Source: Modified from Martínez et al. (2015).
Forestry indicators evaluation process
Defining indicators
The definition must be clear and include at least the objective to which the indicator is grouped within the public policy to which it corresponds, calculation method, clearly specified units, and source of information. The indicators evaluated in this study are shown in Figure 2.
Model of evaluation
The development considers the 2-tuple fuzzy linguistic fuzzy representation model and the JLE approach designed to solve problems with multigranular linguistic information.
The 2-tuple fuzzy linguistic representation model proposed by Herrera and Martinez (2000) is symbolic and is defined under computing with words approach where the linguistic results are obtained starting from linguistic premises. While other linguistic approaches carry out approximation processes to obtain the results, which implies a loss of information and precision in the results, the 2-tuple fuzzy linguistic model provides a continuous fuzzy representation for linguistic values, overcoming precision limitations of previous models (Pei, Ruan, Liu, & Xu, 2009). The model represents the linguistic information using a pair of values known as 2-tuple (s, α), where s is a linguistic term, and α is a numerical value representing the symbolic translation (Herrera & Martínez, 2001a).
Definition 1 (Herrera & Martínez, 2000): Let β be the result of an aggregation of the indices of a set of labels evaluated on a set of linguistic terms S; that is, the result of a symbolic aggregation operation
The model also defines a set of functions to make transformations between 2-tuple values and numeric values.
Definition 2 (Herrera & Martínez, 2000): Let
The literature indicates several advantages in the computing with words approach in favor of the 2-tuple fuzzy linguistic representation method (Martínez et al., 2015; Rodríguez & Martínez, 2013). One of the most important is that the linguistic domain can be treated as continuous, whereas it is treated as discrete in classical models. The 2-tuple-based computational linguistic model easily performs computing with words processes without loss of information; the results of the computing with words processes are constantly exposed in the initial domain expression; and the aggregation of multi-granular linguistic information is possible in an easy way.
On the other hand, the JLE approach arises in the context of evaluation processes. The participating experts have different levels of knowledge of the variable of interest, which justifies using different sets of linguistic terms, i.e., a multi-granular approach. In addition to the JLE approach, multi-granular approach methods based on the 2-tuple fuzzy linguistic model include the fusion approach to multi-granular linguistic information management (Herrera, Herrera-Viedma, & Martínez, 2000) and linguistic hierarchies (LH) (Herrera & Martínez, 2001b).
The JLE approach solves the limitation shown in the granularity of the basic set of linguistic terms proposed by the fusion approach in the unification phase, which would be larger than the other term sets; it also solves the disadvantages related to precision and domain of expression for the computed results.
Following the methodology proposed by Espinilla et al. (2011), the JLE approach is based on LH, which are understood as the union of all levels, i.e. t: JL = U
t
l(t,n(t)), where each level t of LH corresponds to the set of linguistic terms with an uncertainty granularity of n(t) denoted as:
It defines the set of previous nodal points of level t such as
It replaces the two basic rules of the LH approach by forcing to keep the previous nodal points from one level t to the next, t + 1. According to Espinilla et al. (2011), the extended hierarchical rules are: (a) to build a JLE with a finite number of levels l(t,n(t)) with t = 1,…, m that defines the multigranular F MS structure required by the experts to express their knowledge (it is necessary to maintain the previous nodal points between them); and b) to have a JLE, where a new level l(t*,n(t*)) with t* = m + 1 must be added to keep all nodal points of all previous levels l(t,n(t)), t = 1,…, m within this new level.
To build a JLE, the m linguistic scales are given to the experts to express their information. Then the term set l(t*,n(t*)) with t* = m + 1 will be aggregated according to the following theorem (Espinilla et al., 2011): Let
It proposes an optimised structure minimising the granularity of t*, which can keep all the above nodal points, using the least common multiple (LCM) as follows:
Within the unification phase, the JLE approach uses the transformation functions defined in the LH approach,
Because the transformation is unified by means of 2-tuple linguistic values, the computational phase is executed using the 2-tuple linguistic representation model. The results obtained are expressed by means of 2-tuple linguistic values in a unified
Framework
In this step, the structure of the problem, preferences, and uncertainty is established. The components are the criteria
The criteria assessed are those used by CONEVAL (2014):
Clarity (c1). It refers to whether there are doubts about what is intended to be measured, whether the indicator has any ambiguous terms or technical aspects that could be interpreted differently.
Relevance (c2). It should be verified that the most important elements of the indicator are directly related to some fundamental aspect of the objective (relevant factors).
Monitoring (c3). The clarity of the means of verification and the calculation method are analyzed to determine whether the indicator can be subject to independent verification.
Adequacy (c4). It refers to whether the indicator provides a sufficient basis for making a judgment about the program’s performance and whether the information that the indicator provides is relevant and appropriate to describe the program’s achievements over a period.
Experts: Regarding the multi-granular nature of the assessment, the participation of six experts grouped in two sets is considered: forest administrative sector,
Granularity structure. Given the participation of two groups of experts with different levels of knowledge, a linguistic hierarchy composed of two linguistic term sets with granularity of five and seven terms is used. The group of experts from the forestry administrative sector expresses its assessment of the linguistic variable compliance in S
5
, and the group of experts from the academic sector in S
7
; i.e.,
Semantics. The linguistic terms were defined by means of a triangular membership function µ
A
(u), which is represented by a 3-tuple (a, b, c) where b indicates the point at which the membership value is equal to 1, and a and c indicate the left and right boundaries, respectively, of the domain of the membership function, where,
Thus, we have finite, ordered sets of linguistic terms with an odd cardinality (g + 1), where each term is equally informative. The mean linguistic term is the intermediate term and represents an approximate rating of 0.5 (Figure 3).
Figure 3 The semantics of linguistic term sets of linguistic hierarchies with granularity of five (S5) and seven (S7) terms.
Once all the ratings are obtained, and making use of the two extended rules of the JLE approach, the new set (level) l(t*, n(t*)) with t* = m + 1 is added to the linguistic hierarchy, that is: n(t*,n(t*)) = n(t
3
,n(t
3
)), where the granularity of (n(t
3
)) is defined by applying the theorem 1
Figure 4 Linguistic term set aggregated to the extended linguistic hierarchy l(t * , n(t * )) = S 13.
Assuming that, within each group, the experts have the same importance in the decision process, the aggregation operator used is the 2-tuple arithmetic mean defined as
The assessments were collected using designed formats (questionnaires of five [Appendix 1] and seven linguistic terms [Appendix 2]) and each respondent e i was provided with all relevant information: 1) objective, 2) name of the indicator, 3) description of the indicator, 4) method of calculation, 4) sources of information and 5) definition of the criteria.
All calculations were performed using the transformation equations defined in the resolution method, using the FLINTSTONES software (Estrella, Espinilla, Herrera, & Martínez, 2014). A decision support system developed within the Intelligent Systems Based on Fuzzy Decision Analysis (SINBAD) research group at the University of Jaén, Spain. The acronym FLINTSTONES stands for Fuzzy LINguisTic deciSion TOols eNhacemEnt Suite. The program was designed to solve decision-making problems under uncertainty, following a computing with words approach, which allows obtaining, starting from linguistic premises, results also represented linguistically, which facilitates interpretation of the results. Therefore, through FLINTSTONES, it is possible to model decision-making problems of any nature, specifying experts involved, alternatives or possible solutions to the problem, and evaluating those alternatives. Furthermore, the tool allows the definition of domains of expression of different nature that experts can use to elicit their opinions; therefore, they can be modelled by numerical or linguistic values. Once the problem has been defined, FLINTSTONES includes decision models widely used in decision-making literature that allow to obtain the solution. In this study, the 2-tuple linguistic fuzzy model was selected under a complex multigranularity uncertainty framework. FLINTSTONES is available for download at https://sinbad2.ujaen.es/flintstones/eshttp://sinbad2.ujaen.es/es
Results
The ratings expressed by the group of experts from the official administrative sector are presented in Table 1. Table 2 shows the same information, but the ratings correspond to the group of experts from the academic sector and are expressed in the set of seven linguistic terms (S 7 ).
Table 1 Ratings expressed by the experts from the official administrative sector, provided in the linguistic term set S 5 , for assessing the criteria of indicators of the National Forest Program 2012-2018.
| Indicator | Criteria | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Expert 1 | Expert 2 | Expert 3 | ||||||||||
| C1 | C2 | C3 | C4 | C1 | C2 | C3 | C4 | C1 | C2 | C3 | C4 | |
| i1 | (s4 5) | (s3 5) | (s3 5) | (s3 5) | (s3 5) | (s4 5) | (s4 5) | (s3 5) | (s4 5) | (s4 5) | (s4 5) | (s3 5) |
| i2 | (s0 5) | (s3 5) | (s3 5) | (s3 5) | (s2 5) | (s4 5) | (s4 5) | (s3 5) | (s2 5) | (s3 5) | (s4 5) | (s3 5) |
| i3 | (s0 5) | (s2 5) | (s3 5) | (s3 5) | (s2 5) | (s3 5) | (s3 5) | (s3 5) | (s4 5) | (s4 5) | (s4 5) | (s3 5) |
| i4 | (s3 5) | (s2 5) | (s3 5) | (s3 5) | (s2 5) | (s3 5) | (s3 5) | (s3 5) | (s2 5) | (s3 5) | (s4 5) | (s3 5) |
| i5 | (s3 5) | (s2 5) | (s3 5) | (s3 5) | (s3 5) | (s4 5) | (s4 5) | (s3 5) | (s1 5) | (s2 5) | (s2 5) | (s3 5) |
| i6 | (s3 5) | (s2 5) | (s3 5) | (s3 5) | (s3 5) | (s3 5) | (s3 5) | (s3 5) | (s4 5) | (s4 5) | (s3 5) | (s3 5) |
| i7 | (s3 5) | (s2 5) | (s3 5) | (s3 5) | (s3 5) | (s4 5) | (s4 5) | (s3 5) | (s3 5) | (s3 5) | (s4 5) | (s3 5) |
| i8 | (s3 5) | (s2 5) | (s3 5) | (s3 5) | (s2 5) | (s2 5) | (s2 5) | (s3 5) | (s2 5) | (s1 5) | (s3 5) | (s3 5) |
| i9 | (s3 5) | (s2 5) | (s3 5) | (s3 5) | (s2 5) | (s3 5) | (s3 5) | (s3 5) | (s3 5) | (s4 5) | (s3 5) | (s3 5) |
| i10 | (s2 5) | (s1 5) | (s2 5) | (s2 5) | (s2 5) | (s2 5) | (s2 5) | (s2 5) | (s2 5) | (s3 5) | (s2 5) | (s3 5) |
| i11 | (s2 5) | (s1 5) | (s2 5) | (s2 5) | (s2 5) | (s2 5) | (s2 5) | (s2 5) | (s2 5) | (s1 5) | (s2 5) | (s3 5) |
| i12 | (s3 5) | (s2 5) | (s3 5) | (s3 5) | (s3 5) | (s3 5) | (s4 5) | (s4 5) | (s1 5) | (s1 5) | (s2 5) | (s3 5) |
| i13 | (s3 5) | (s2 5) | (s3 5) | (s3 5) | (s1 5) | (s2 5) | (s2 5) | (s1 5) | (s3 5) | (s1 5) | (s3 5) | (s3 5) |
The indicators correspond to those shown in Figure 2.
Table 2 Ratings expressed by experts from the academic sector, provided in the linguistic term set S 7 , evaluate the criteria of indicators of the National Forest Program 2012-2018.
| Indicator | Criteria | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Expert 1 | Expert 2 | Expert 3 | ||||||||||
| C1 | C2 | C3 | C4 | C1 | C2 | C3 | C4 | C1 | C2 | C3 | C4 | |
| i1 | (s3 7) | (s4 7) | (s3 7) | (s4 7) | (s6 7) | (s5 7) | (s5 7) | (s4 7) | (s6 7) | (s6 7) | (s6 7) | (s6 7) |
| i2 | (s3 7) | (s4 7) | (s3 7) | (s2 7) | (s6 7) | (s6 7) | (s5 7) | (s4 7) | (s5 7) | (s6 7) | (s4 7) | (s6 7) |
| i3 | (s2 7) | (s4 7) | (s2 7) | (s4 7) | (s6 7) | (s5 7) | (s6 7) | (s5 7) | (s4 7) | (s6 7) | (s5 7) | (s6 7) |
| i4 | (s3 7) | (s4 7) | (s2 7) | (s4 7) | (s3 7) | (s6 7) | (s6 7) | (s4 7) | (s5 7) | (s5 7) | (s6 7) | (s6 7) |
| i5 | (s4 7) | (s4 7) | (s2 7) | (s4 7) | (s6 7) | (s6 7) | (s6 7) | (s3 7) | (s5 7) | (s5 7) | (s6 7) | (s6 7) |
| i6 | (s2 7) | (s3 7) | (s3 7) | (s3 7) | (s4 7) | (s4 7) | (s5 7) | (s3 7) | (s5 7) | (s5 7) | (s6 7) | (s5 7) |
| i7 | (s4 7) | (s3 7) | (s2 7) | (s2 7) | (s6 7) | (s6 7) | (s4 7) | (s3 7) | (s4 7) | (s6 7) | (s3 7) | (s5 7) |
| i8 | (s2 7) | (s2 7) | (s3 7) | (s2 7) | (s6 7) | (s5 7) | (s2 7) | (s2 7) | (s5 7) | (s6 7) | (s3 7) | (s5 7) |
| i9 | (s2 7) | (s4 7) | (s2 7) | (s4 7) | (s6 7) | (s6 7) | (s6 7) | (s6 7) | (s5 7) | (s5 7) | (s4 7) | (s4 7) |
| i10 | (s2 7) | (s3 7) | (s1 7) | (s3 7) | (s5 7) | (s5 7) | (s5 7) | (s5 7) | (s3 7) | (s4 7) | (s4 7) | (s4 7) |
| i11 | (s2 7) | (s3 7) | (s2 7) | (s3 7) | (s4 7) | (s5 7) | (s5 7) | (s5 7) | (s3 7) | (s4 7) | (s3 7) | (s4 7) |
| i12 | (s2 7) | (s2 7) | (s2 7) | (s3 7) | (s6 7) | (s6 7) | (s5 7) | (s2 7) | (s4 7) | (s5 7) | (s3 7) | (s4 7) |
| i13 | (s4 7) | (s4 7) | (s4 7) | (s4 7) | (s6 7) | (s6 7) | (s6 7) | (s6 7) | (s5 7) | (s4 7) | (s6 7) | (s4 7) |
The indicators correspond to those shown in Figure 2.
Following the JLE approach, the values expressed by the experts in the two linguistic term sets (S 5 and S 7) were unified (transferred) to the linguistic term set added to the linguistic hierarchy (S 13). Table 3 shows the transformation of the results for the academic experts. The same procedure was used for the ratings of the experts from the forest administrative sector.
Table 3 Values expressed by experts from the forestry academic sector, transferred to the aggregated linguistic term set S 13 and represented by 2-tuple values for the assessment of the criteria of indicators of the National Forest Program 2012-2018.
| Indicator | Criteria | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Expert 4 | Expert 5 | Expert 6 | ||||||||||
| C1 | C2 | C3 | C4 | C1 | C2 | C3 | C4 | C1 | C2 | C3 | C4 | |
| i1 | (s6 13,0) | (s8 13,0) | (s6 13,0) | (s8 13,0) | (s12 13,0) | (s12 13,0) | (s10 13,0) | (s8 13,0) | (s12 13,0) | (s12 13,0) | (s12 13,0) | (s12 13,0) |
| i2 | (s6 13,0) | (s8 13,0) | (s6 13,0) | (s4 13,0) | (s12 13,0) | (s12 13,0) | (s10 13,0) | (s8 13,0) | (s10 13,0) | (s12 13,0) | (s8 13,0) | (s12 13,0) |
| i3 | (s4 13,0) | (s8 13,0) | (s4 13,0) | (s8 13,0) | (s12 13,0) | (s10 13,0) | (s12 13,0) | (s10 13,0) | (s8 13,0) | (s12 13,0) | (s10 13,0) | (s12 13,0) |
| i4 | (s6 13,0) | (s8 13,0) | (s4 13,0) | (s8 13,0) | (s6 13,0) | (s12 13,0) | (s12 13,0) | (s8 13,0) | (s10 13,0) | (s10 13,0) | (s12 13,0) | (s12 13,0) |
| i5 | (s8 13,0) | (s8 13,0) | (s4 13,0) | (s8 13,0) | (s12 13,0) | (s12 13,0) | (s12 13,0) | (s6 13,0) | (s10 13,0) | (s10 13,0) | (s10 13,0) | (s12 13,0) |
| i6 | (s4 13,0) | (s8 13,0) | (s6 13,0) | (s6 13,0) | (s8 13,0) | (s8 13,0) | (s10 13,0) | (s6 13,0) | (s10 13,0) | (s10 13,0) | (s10 13,0) | (s10 13,0) |
| i7 | (s8 13,0) | (s6 13,0) | (s4 13,0) | (s4 13,0) | (s12 13,0) | (s12 13,0) | (s8 13,0) | (s6 13,0) | (s8 13,0) | (s12 13,0) | (s6 13,0) | (s10 13,0) |
| i8 | (s4 13,0) | (s4 13,0) | (s6 13,0) | (s4 13,0) | (s12 13,0) | (s10 13,0) | (s4 13,0) | (s4 13,0) | (s10 13,0) | (s12 13,0) | (s6 13,0) | (s10 13,0) |
| i9 | (s4 13,0) | (s8 13,0) | (s4 13,0) | (s8 13,0) | (s12 13,0) | (s12 13,0) | (s12 13,0) | (s12 13,0) | (s10 13,0) | (s10 13,0) | (s8 13,0) | (s8 13,0) |
| i10 | (s4 13,0) | (s6 13,0) | (s2 13,0) | (s6 13,0) | (s10 13,0) | (s10 13,0) | (s10 13,0) | (s10 13,0) | (s6 13,0) | (s8 13,0) | (s8 13,0) | (s8 13,0) |
| i11 | (s4 13,0) | (s6 13,0) | (s4 13,0) | (s6 13,0) | (s8 13,0) | (s10 13,0) | (s10 13,0) | (s10 13,0) | (s6 13,0) | (s8 13,0) | (s6 13,0) | (s8 13,0) |
| i12 | (s4 13,0) | (s4 13,0) | (s4 13,0) | (s6 13,0) | (s12 13,0) | (s12 13,0) | (s10 13,0) | (s4 13,0) | (s8 13,0) | (s10 13,0) | (s6 13,0) | (s8 13,0) |
| i13 | (s8 13,0) | (s6 13,0) | (s8 13,0) | (s8 13,0) | (s12 13,0) | (s12 13,0) | (s12 13,0) | (s12 13,0) | (s10 13,0) | (s8 13,0) | (s12 13,0) | (s8 13,0) |
The indicators correspond to those shown in Figure 2. Criteria: c1 = clarity, c2 = relevance, c3 = monitoring and c4 = adequacy).
As the collective ratings were calculated using the 2-tuple fuzzy linguistic model, they were expressed in the aggregated linguistic term set S 13, including the ranking order of indicators according to the experts' ratings (Table 4).
Table 4 Collective assessments expressed in the linguistic term set S 13 for evaluating the criteria of indicators of the National Forest Program 2012-2018.
| Indicator | Criteria | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Expert 1 | Expert 2 | Expert 3 | ||||||||||
| C1 | C2 | C3 | C4 | C1 | C2 | C3 | C4 | C1 | C2 | C3 | C4 | |
| i1 | (s4 5) | (s3 5) | (s3 5) | (s3 5) | (s3 5) | (s4 5) | (s4 5) | (s3 5) | (s4 5) | (s4 5) | (s4 5) | (s3 5) |
| i2 | (s0 5) | (s3 5) | (s3 5) | (s3 5) | (s2 5) | (s4 5) | (s4 5) | (s3 5) | (s2 5) | (s3 5) | (s4 5) | (s3 5) |
| i3 | (s0 5) | (s2 5) | (s3 5) | (s3 5) | (s2 5) | (s3 5) | (s3 5) | (s3 5) | (s4 5) | (s4 5) | (s4 5) | (s3 5) |
| i4 | (s3 5) | (s2 5) | (s3 5) | (s3 5) | (s2 5) | (s3 5) | (s3 5) | (s3 5) | (s2 5) | (s3 5) | (s4 5) | (s3 5) |
| i5 | (s3 5) | (s2 5) | (s3 5) | (s3 5) | (s3 5) | (s4 5) | (s4 5) | (s3 5) | (s1 5) | (s2 5) | (s2 5) | (s3 5) |
| i6 | (s3 5) | (s2 5) | (s3 5) | (s3 5) | (s3 5) | (s3 5) | (s3 5) | (s3 5) | (s4 5) | (s4 5) | (s3 5) | (s3 5) |
| i7 | (s3 5) | (s2 5) | (s3 5) | (s3 5) | (s3 5) | (s4 5) | (s4 5) | (s3 5) | (s3 5) | (s3 5) | (s4 5) | (s3 5) |
| i8 | (s3 5) | (s2 5) | (s3 5) | (s3 5) | (s2 5) | (s2 5) | (s2 5) | (s3 5) | (s2 5) | (s1 5) | (s3 5) | (s3 5) |
| i9 | (s3 5) | (s2 5) | (s3 5) | (s3 5) | (s2 5) | (s3 5) | (s3 5) | (s3 5) | (s3 5) | (s4 5) | (s3 5) | (s3 5) |
| i10 | (s2 5) | (s1 5) | (s2 5) | (s2 5) | (s2 5) | (s2 5) | (s2 5) | (s2 5) | (s2 5) | (s3 5) | (s2 5) | (s3 5) |
| i11 | (s2 5) | (s1 5) | (s2 5) | (s2 5) | (s2 5) | (s2 5) | (s2 5) | (s2 5) | (s2 5) | (s1 5) | (s2 5) | (s3 5) |
| i12 | (s3 5) | (s2 5) | (s3 5) | (s3 5) | (s3 5) | (s3 5) | (s4 5) | (s4 5) | (s1 5) | (s1 5) | (s2 5) | (s3 5) |
| i13 | (s3 5) | (s2 5) | (s3 5) | (s3 5) | (s1 5) | (s2 5) | (s2 5) | (s1 5) | (s3 5) | (s1 5) | (s3 5) | (s3 5) |
Finally, the collective values were retranslated into one of the two linguistic term sets used (Table 5); in this case, the linguistic term set used was S 7. As can be seen, this process of retranslation did not change the ranking order of indicators and it was necessary because usually the set t * added to the linguistic hierarchy corresponds to a high degree of granularity, which makes it difficult to interpret. The expression of the values is more interpretable because linguistic terms are closer to the usual rating scales and experts get results represented in any of the linguistic term sets used.
Table 5 Collective assessments expressed in the linguistic term set S 7, for the evaluation of the criteria of indicators of the National Forest Program 2012-2018.
| Indicator | Name | Classification | Collective assessment |
|---|---|---|---|
| i1 | Rate of change of timber wood production | 1 | (very high, 0.042) |
| i9 | Percentage of ejidos and communities changing to a higher typology of producers by increasing organizational capacities. | 2 | (high, 0.438) |
| i3 | Rate of change of certified area under good forest management practices | 3 | (high, 0.417) |
| i5 | Percentage of coverage of reinstated or restored | 4 | (high, 0.396) |
| i4 | Rate of change in area conserved via payments for environmental services | 5 | (high, 0.375) |
| i2 | Percentage of value of production derived from sustainable harvesting of natural resources | 5 | (high, 0.375) |
| i7 | Rate of change in the average annual area of adult and regrowth trees affected by forest fires | 6 | (high, 0.313) |
| i6 | Annual net deforestation rate of forests and rainforests | 7 | (high, 0.270) |
| i13 | Rate of change of financial resources granted to the forestry sector by the development bank | 8 | (high, 0.167) |
| i12 | Avoided CO2 emissions by reduction of deforestation and forest degradation | 9 | (high, -0.104) |
| i8 | Proportion of timber forest products belonging to the legal market | 10 | (high, -0.333) |
| i10 | Index of social participation in forestry sector | 11 | (moderate, 0.333) |
| i11 | Index of the National REDD+ Strategy in operation | 12 | (moderate, 0.223) |
Even though the 2-tuple fuzzy linguistic model is widely supported by results obtained by researchers who have addressed and solved several theoretical and real problems (Carmona, González, Gacto, & del Jesus, 2012; Doukas, Tsiousi, Marinakis, & Psarras, 2014; Herrera-Viedma, López-Herrera, Luque, & Porcel, 2007; Montes, Sánchez, Villar, & Herrera, 2015), there are so far no known applications in the evaluation of public policy indicators that would allow us to contrast the results obtained here. However, there is confidence in the quality of tools used and the proper character of the structure of information gathered in the process of evaluating indicators; therefore, it can be said that, according to the linguistic term and value of the symbolic translation α estimated for each of the indicators, indicator i 1 “Rate of change of timber forest production” is the one that had the best evaluations from the experts with “very high” level of compliance in the criteria considered. After this, indicators i 1 to i 9, including indicators i 12 and i 13, had a “high” level of compliance with differences in the value of the symbolic translation which, in this case, determined the differences in the ranking order (Table 5). Only i 10 and i 11 had the collective rating corresponding to the linguistic term “moderate”.
Conclusiones
The structure defined in evaluating social policy indicators corresponds appropriately to that used with the 2-tuple fuzzy linguistic model, which is relevant because it allows an adequate evaluation of the indicators considered. Furthermore, the multi-granular approach considers assessments beyond the traditional binary values (yes, no). Through the participation of experts with different levels of knowledge, it enriches the experts’ assessments expressed with greater precision. CONEVAL and the World Bank represent valuable contributions to assessing indicators and evaluating aspects beyond the four criteria considered in this study; however, such aspects are also susceptible to being evaluated through the approach used here. In the future, it is intended to extend the study to the assessment of all qualitative aspects considered in the attributes of the indicators.
