text new page (beta)
English (pdf)
Article in xml format
Article references
Automatic translation
Send this article by e-mail
Cited by SciELO 
Similars in
SciELO 
Permalink
Introduction
When talking about pro-environmental behavior (CPA), Páramo (2017) points out that this concept refers to those human activities whose intention is the protection of natural resources or, at least, the reduction of environmental deterioration, which is intertwined with various values interacting from a person to a society in different settings.
Environmental degradation is mainly caused by anthropogenic activities (Toledo, 2013), and the lack of environmental education jeopardizes natural resources for future generations. Therefore, it is essential to re-educate people in terms of achieving sustainable development (Saldaña-Almazán et al., 2020). Since 1970s, research that has been done regarding sustainability, and the environment has been substantial; however, much more needs to be done. For instance, higher education institutions are committed to the development of human resources and the improvement of the physical space where they operate, but environmental education as a transversal axis has not appeared on their programs regardless of the area of study. Therefore, it is necessary to implement informal environmental education programs to provide information that will eventually result in improvements in sustainable practices (Ashraf & Alanezi, 2020).
An evaluation instrument is a technique that will allow a numerical assignment to quantify the expressions of a construct that is only indirectly measurable. Research instruments are operational tools that allow data collection; therefore, all instruments must be the product of an articulation of the theory, methodology, and techniques for data collection and analysis (Soriano, 2014). According to Cea-D'Ancona (1999), a validation instrument as a standardized procedure that is used to collect information from a group of subjects, whose main objective is to gather information on behavior, values, and attitudes regarding a research topic, might be designed using a Likert-type scale, in that this type of scale can measure the intensity of an attitude (Kerlinger & Lee, 2002). Instruments made with this type of scale have shown a very acceptable performance in research experiences since the score per item allows to indicate a greater or lesser presence of variables.
The content validity of an instrument is understood as “the extent to which the elements that comprise the scale accurately represent or measure the information that is being assessed” (Gil-Gómez & Pascual-Ezama, 2012). Different ways have been suggested to assess if an instrument has high content validity or not. The most common method is based on working with judges who are experts on the subject that is going to be studied. This method has been used to structure an instrument consisting of a series of items that are going to be examined by a group of people with professional experience in the subject and who can make valuable judgments (Silva et al., 2002). Since the goal of this method is to obtain a reliable consensus among expert opinions, its main advantages are the feasibility for specialists to carry out the assessment and the generation of a controlled interaction between the participants and the judges. In the same way, Cabero (2014) recommends this validation method when it is desirable to maintain the heterogeneity of the experts to ensure the validity of the results and to facilitate the participation of judges who are physically dispersed. The method follows a procedure that, according to Aponte et al. (2012), can be summarized in eight stages: 1) to identify the problem, 2) to select the group of experts, 3) to present the problem by utilizing a survey, 4) to synthesize the responses into a series of statements, 5) to send the items to the group of specialists for a first evaluation, 6) to analyze the experts’ responses or suggestions, 7) to identify convergences or divergences to achieve a consensus regarding the evaluations obtained, and 8) to prepare the final instrument.
The main properties of measurement are reliability and validity. On the one hand, reliability refers to the fact that the object of study, when measured repeatedly with the same instrument, will always give the same results. According to Aponte et al. (2012), validity in a measurement instrument will be the degree of ownership of the inferences and interpretations resulting from the results of a test that includes its social and ethical consequences. Thus, reliability is empirical, which focuses on the probability of the instrument yielding the same results over multiple trials. On the other hand, validity refers to the extent to which the instrument measures meaningfully and adequately the subject that is to be measured. The validation of an instrument requires continuous inspections; hence, it only indicates certain degrees of validity for specific uses and certain populations (Jiménez & Montero, 2013). In addition to this, Gronlund (1976) indicates that validity matters when it is required to use the subject’s performance on the instrument to infer the possession of certain traits.
The objective of this research was to design and validate an instrument that evaluates the pro-environmental behavior of university students from the Autonomous University of Guerrero (UAGro) and later, with the information obtained, to create an informal environmental education program using information and communication technologies (ICT) as supporting tools in the teaching-learning process, while adapting to the needs of the students (Saldaña-Almazán & Saldaña-Almazán, 2019).
Materials and Methods
A quantitative methodological approach was established to carry out this process. The process of validity and reliability of the instrument was based on research reported by Soriano (2014). It consists of the elaboration of an instrument, its design and adaptation, and finally the analysis of its psychometric properties. This approach was applied at each stage to determine the validity of the content, to improve reliability, and to assess and improve a scientific paper. This analysis is used to validate the content, construction, and reliability of the instrument to evaluate the pro-environmental behavior.
Design and peer review
For the elaboration of the instrument, the delimitation of the indicators was established in the form of a survey with five response options according to a Likert scale whose options were: Never = 1, Almost never = 2, Sometimes = 3, Almost always = 4, and Always = 5 (García et al., 2017). Other aspects considered were academic training, environmental themes, sociocultural issues, the university community, management, extension, research, associations, and socioeconomic impact.
Items’ score
A first draft of the instrument was designed in which the information was delimited, and the number of items to be included in the instrument were formulated, ordered, and established. Then, the validation process involved carrying out a pilot test and evaluating the metric properties of the scale. For this, specialists with experience in the areas of research, teaching, and environmental sciences were selected.
Pilot test
To evaluate the pilot test (25 items), the experts considered the following aspects:
Types of questions: Is the statement adequate and comprehensible?; Does it have the appropriate length?
Categorization of responses
Are there any psychological resistances or rejections towards some questions?
Does it present a logical internal order?
Is the application time acceptable according to the participants?
The elaboration of the instrument was carried out by performing:
Evaluation of scale metrics
The measurement scale was designed to ensure reliability and validity. It not only allows to have a score but also to compare the score of the participants (or that obtained from the same participant at different times). The reliability of the instrument is the indication that it can offer true and constant results during its repeated use under similar measurement conditions. It was assessed through consistency, temporal stability, and inter-observer concordance as established and listed below (Manterola et al., 2018).
1. Consistency refers to the level at which the different items on a scale are related to each other. This indicates the degree of agreement between them and, therefore, will determine that they can accumulate and give an overall score. Consistency will be tested using Cronbach's alpha coefficient statistical method, its values range from 0 to 1. A good internal consistency will be considered to exist when the alpha value is greater than 0.7 (Frías-Navarro, 2021).
2. Temporal stability is the consistency obtained between the test results when the same sample is evaluated by the same evaluator in two different situations (test-retest reliability). Reliability (normally calculated with the intraclass correlation coefficient, for continuous variables and distant temporal evaluations) indicates that the measurement result has temporal stability. A correlation of 70% would indicate acceptable reliability.
3. Inter-observer concordance refers to the analysis of the level of agreement obtained when the same sample is evaluated by two different evaluators under the same conditions, and at different times, obtaining the same reliability results. The agreement between observers can be analyzed by the percentage of agreement and Kappa index.
The sequence for the instrument’s validation, writing, and design process can be observed in Figure 1.
Results
Phase I
The research aims and its rationality were considered to proceed with the psychometric validation for the construction of the instrument. Likewise, all the data related to the research was collected for the preparation of the items. For the selection of the experts, a survey was applied (Table 1) in addition to the fact that they must meet at least one of the following criteria: 1) having doctoral training, 2) being a specialist on subjects related to environmental sciences or education with a master's degree, and 3) the experts must come from different contexts.
Table 1 Survey designed for the selection of experts.
| Dimension | Question |
| Knowledge | 1. Are you familiar with the environmental conditions in your area? |
| 2. Do you think that it is important to consider the environment in your daily life? | |
| 3. Are you interested in the environment? | |
| 4. What are the benefits of taking care of the environment? | |
| 5. Do you contribute with your actions to the caring of the environment? | |
| 6. Have you participated in sustainability activities from the university? | |
| 7. Do you think these activities are important in university life? | |
| 8. Are you clear about the objectives of sustainability? | |
| University | 9. Are there official sustainability programs at the university? |
| 10. Is there a specific office or area at the university for this topic? | |
| 11. Are there cross departmental processes regarding sustainability issues? Are there partnerships with other universities, industries or businesses to address and discuss environmental issues? | |
| 12. Is there a specific plan that is implemented at the university, which includes water and electricity conservation, sanitation, wastewater management, garbage recycling? | |
| 13. Do the facilities of the institution include aspects of environmental sustainability? | |
| 14. Are there environmental or sustainability policies at the institution? | |
| 15. How do you consider the subject of university sustainability? | |
| Education | 16. Are there sustainability themes in the educational programs? |
| 17. Is sustainability emphasized in classes? | |
| 18. Are teachers interested in these topics? | |
| 19. Do teachers make proposals for these topics? | |
| 20. Do the study plans include the theme throughout the educational program? | |
| 21. Are professors prepared to teach topics related to sustainability? | |
| 22. Are topics discussed in a transversal way? | |
| 23. Are there lines of research related to the sustainability of resources? | |
| 24. Does the university have an impact on the social and productive sector on sustainability issues? |
Source: Authors’ own elaboration.
25 experts were contacted, and 15 were selected. 14 experts corresponded to various UAGro academic units and one from an external institution. Among the experts, 73.33% correspond to specialists with a post-doctoral degree, which fulfill the first desired criteria for the selection, and 26.67% have a master’s degree, which corresponds to the second condition.
Also, 93.33% are experts working in different areas of UAGro, and 6.66% working at another university. All of them (100%) are full-time teachers and dedicated to research and teaching activities.
Phase II
At the end of the first phase (drafting phase), the instrument was submitted to expert judgment. This allowed them to have a clear sense of the objectives, as well as the theoretical positioning of the research, facilitating evaluation based on the theoretical construct and the validation of format. The experts are specialists in the area, with academic and research experience related to the topic of environmental education, which allowed them to assess the content and form of each of the items included in the instrument. Figure 2 shows the validation format for the experts.
The observations made by the experts were submitted to consensus, by means of Cohem kappa coefficient. After the evaluation, the instrument was distributed to a focus group to establish whether the items were understandable and to receive the group’s interpretations. The participating experts validated each of the items, this allowed improving the instrument in terms of writing, content, selection, form, and style. Figure 3 shows the first example of the pilot survey delivered to the specialists.
Phase III
After reviewing the results of Phase II, the judges carried out their analysis, and a second draft of the items was prepared. It was later applied as a pilot test, where the sample was selected randomly. The instrument was applied to the participants under the same conditions, and the data and statistical analysis of the results were processed. The selected population has similar characteristics to those of the population to be researched: 36 women and 21 men; they were high school, undergraduate, and graduate students.
The application of the instrument had as objectives (1) to determine the degree of clarity of the items and its instructions, resolution time, and satisfaction with the instrument, and (2) to determine the variance of each item (Table 2) and so to analyze the reliability of the instrument using Cronbach's alpha coefficient.
Table 2 Analysis of the variance of the pilot test.
| Items | Initial Eigen values | Sums of charge extraction squared | ||||
| Total | % of variance | % accumulated | Total | % de variance | % accumulated | |
| 1 | 3.857 | 15.430 | 15.430 | 3.857 | 15.430 | 15.430 |
| 2 | 2.022 | 8.088 | 23.518 | 2.022 | 8.088 | 23.518 |
| 3 | 1.460 | 5.839 | 29.356 | 1.460 | 5.839 | 29.356 |
| 4 | 1.190 | 4.758 | 34.115 | 1.190 | 4.758 | 34.115 |
| 5 | 1.155 | 4.621 | 38.736 | 1.155 | 4.621 | 38.736 |
| 6 | 1.089 | 4.355 | 43.091 | 1.089 | 4.355 | 43.091 |
| 7 | 1.021 | 4.084 | 47.175 | 1.021 | 4.084 | 47.175 |
| 8 | 0.994 | 3.977 | 51.152 | |||
| 9 | 0.955 | 3.822 | 54.974 | |||
| 10 | 0.931 | 3.725 | 58.699 | |||
| 11 | 0.880 | 3.519 | 62.218 | |||
| 12 | 0.843 | 3.371 | 65.589 | |||
| 13 | 0.826 | 3.302 | 68.891 | |||
| 14 | 0.805 | 3.221 | 72.112 | |||
| 15 | 0.769 | 3.075 | 75.187 | |||
| 16 | 0.751 | 3.004 | 78.191 | |||
| 17 | 0.721 | 2.884 | 81.075 | |||
| 18 | 0.683 | 2.734 | 83.808 | |||
| 19 | 0.667 | 2.669 | 86.477 | |||
| 20 | 0.632 | 2.527 | 89.004 | |||
| 21 | 0.624 | 2.495 | 91.499 | |||
| 22 | 0.579 | 2.315 | 93.813 | |||
| 23 | 0.533 | 2.133 | 95.947 | |||
| 24 | 0.517 | 2.069 | 98.016 | |||
| 25 | 0.496 | 1.984 | 100.000 | |||
Source: Authors’ own elaboration.
Based on the obtained results, the reliability was determined. That is, the first percentage was 15%, and from the second onwards the percentages were lower than 10%. According to Soriano (2014), the first must be greater than 20% and all the following must be lower than 10%. Therefore, the instrument could be considered one-dimensional since only the first result obtained does not comply with the following sedimentation graph (Figure 4). One-dimensionality is observed since only the first component exceeds 3 points, and the following 24 components are below 2 points, which means that the instrument is suitable for educational measurement.
The definition of the reliability coefficient refers to the degree to which an instrument constructed by several items presents a high correlation and measures the stability of a sample. Values fluctuate between 0 and 1; Cronbach's Alpha analysis allowed determining the internal stability of the items. Soriano (2014) states that “the minimum acceptable value for Cronbach's alpha coefficient is 0.70; below this value, the internal consistency of the scale is low”. The result obtained in the instrument of the magnitude of the reliability coefficient was 0.772, which is considered acceptable according to Palella & Marins (2003).
Phase IV
The final review of the experts and the application of the instrument as a pilot test allowed for improvements to be made, some of them focused on the drafting of environmental and sustainability concepts and the group’s interpretation of them. Furthermore, the responses of the instrument were reduced to four to avoid ambiguity with the intermediate option called “Sometimes”. The final instrument to be used for the research is shown in (Figure 5).
The final instrument was applied to 1013 students from 29 UAGro academic units at the three educational levels in UAGro. The characteristics of the population to which the instrument was applied are shown in Table 3.
Table 3 Characterization of surveyed students (Final instrument).
| Students | ||
| Gender | Female | 589 |
| Male | 403 | |
| Prefer not to answer | 21 | |
| Level of study | High School | 198 |
| Undergraduate | 751 | |
| Postgraduate | 61 | |
| No answer | 3 | |
| Age | < 18 years old | 387 |
| 18-25 years old | 531 | |
| > 25 years old | 56 | |
| No Answer | 39 | |
Source: Authors’ own elaboration.
The reliability coefficient of the final instrument was determined by a Cronbach's alpha analysis value of 0.778, coinciding as the initial instrument.
Discussion
The main purpose of the research was to design a reliable and viable instrument to evaluate pro-environmental behavior and to provide the basis for a subsequent implementation of good sustainable practices as mentioned by some authors (Ashrat & Alanezi, 2020). These analyses allowed the integration of the knowledge with respect to environmental issues presented in the selected population, as well as its influence on academic, social, and ethical training in students regarding their behavior associated with the environment, as indicated by Soriano (2014). Based on published research, the methodology was addressed. The validation was carried out in four phases (Soriano, 2014), which are appropriate for the design and validation of an instrument that is suitable for the purposes of academic evaluation. While carrying out the first phase, the objectives of the desired results were very clear; it was necessary to carefully develop the items (Ruiz, 2014). At the end of the drafting phase, the instrument was submitted to expert judgment (Landeta, 2002). The selection of experts is one of the fundamental stages since the reliability of the results will depend on the adequate selection of reviewers. They allowed to identify strengths and deficiencies in the instrument that is being evaluated, which will lead to decisions in respect to which items need to be modified or eliminated. The specialists selected for instrument validation were 15 out of 25. The number of specialists falls within the ranged proposed by the literature, which is 7 to 30 individuals (Landeta, 2002), although sometimes it is not possible to access a large number of experts or to be conditioned to a predetermined range (Cabero, 2014). The design of instruments and their corresponding items, either for evaluation or academic tests, must go through the entire previous process to ensure that the information obtained is valid and allows effective decision-making (Soriano, 2014). During the fourth phase, in which the final instrument can be determined, the response options were modified. In the first instrument, five answers were provided, and in the final instrument they were reduced to four in order to eliminate bias of the respondents in the middle option. In fact, it has been recorded as difficult readability and is related to an increased choice of intermediate alternatives; that is, given the difficulty to understand the item, respondents more frequently choose the intermediate options on the scale (Velez & Ashworth, 2007). As part of the validation, statistical tests were carried out, such as the variance test, to corroborate if the items are correlated and to determine whether the instrument is one-dimensional. As mentioned by Soriano in 2014, the importance of applying a survey that goes through a whole validation process is to have the certainty of obtaining reliable results. The first example of the instrument was applied to 57 students of the University of Guerrero (UAGro) as a pilot test, then pertinent changes were made, and the final instrument was applied to 1013 students of the UAGro. With these data, the degree of pro-environmental behavior of UAGro students in Acapulco can be measured; moreover, some of the environmental activities in which students do not participate were identified. This could be used to promote environmental conservation according to students’ needs. In addition, it was concluded that it can be applied at another public and private educational institutions.
Conclusions
The instrument in the first instance was applied to 57 UAGro students as a pilot test, the relevant changes were reported, and the final instrument was applied to 1013 UAGro students. With these data, the degree of pro-environmental behavior of students of the UAGro can be measured. In addition to this, it can be applied in other public and private educational institutions.
The application of instruments to obtain information in some type of research is increasingly common; however, it is necessary to achieve an acceptable degree of validation and reliability before their implementation. Based on this, research results can be more accurate and reliable.
After four validation phases, and obtaining a Cronbach alpha value of 0.778, the instrument was approved and administered to upper secondary, higher and postgraduate level students; this will allow to know the pro-environmental behavoir and detect what dimension can be improved.
Conflicts of Interest
The authors declare no conflict of interest.
