nueva página del texto (beta)
Inglés (pdf)
Artículo en XML
Referencias del artículo
Traducción automática
Enviar artículo por email
Citado por SciELO 
Similares en
SciELO 
Permalink
1 Introduction
The health contingency generated by the COVID-19 pandemic has disrupted all sectors of society in the world, the education sector therefore, is no exception, at all levels of this sector they have seen the emerging need to implement remote education programs, that allow continuity to the activities of the educational field. In Mexico, and in particular for higher education institutions, it has been necessary to include different models for conducting the teaching-learning process in order to adapt to the current context. However, some of the problems in this sense have not only been related to the infrastructure needed to carry out the activities, but teachers have tried to do the same thing they did in the classroom in person. Therefore, evaluation has become one of the most complex points because the application of written examinations is not possible. Online exams are difficult to monitor and become a complex issue as teachers consider that students have a wide possibility of copying in an uncontrollable context. This is where the intelligent system for customizing evaluation activities implemented in virtual learning environments becomes relevant again.
Regarding the evaluation, the Dictionary of the Royal Spanish Academy mentions that, to evaluate means: To point out the value of something; estimate, appraise, calculate the value of something; estimate the knowledge, skills and performance of students [2]. Assessment is a process that integrates a sequence of planned activities, with the purpose of feedback to improve student learning.
However, when it comes to the evaluation of e-learning, it requires a sensible and valid performance evaluation strategy that can detect changes in knowledge constructions. Peñalosa [12] proposes three stages in the assessment of learning: a) The initial assessment, at the beginning of the course, to know the level of knowledge and skills of students; b) The formative evaluation, during the teaching action, in continuous and established periods, and c) The summative evaluation, which is carried out at the end of the course. The importance of continuous assessment of learning lies in the feedback that the learner receives [12].
In this context, attention is focused on the problem of evaluation as the culmination of reprobation, so the object of study of this research, proposes an alternative to improve student approval rates in courses for large groups of students, using Virtual Learning Environments (VLE).
This research paper presents the design and implementation of an ontological model that recommends the type of personalized formative and mediating assessment that promotes meaningful learning in the student, to value the diversity in the styles of thought and of the opportunity to show the knowledge acquired in various ways, not properly through the application of written exams and other practices that were carried out in the classroom in person, rather by considering the approach of e-learning evaluation, which includes initial assessment, formative evaluation and summative evaluation.
2 Related Work
Technology in the field of education covers the development/creation of resources, processes, and information tools, among others, which have a direct impact on the central teaching-learning process at all levels. In this sense, it is necessary to consider that technology has been a fundamental piece in the process of transition in which society finds itself, and that technological advances in the computational area have changed and supported teaching-learning in various aspects within the education sector.
The context in which this research was developed was considered a Computational dimension integrated by the application of Artificial Intelligence through Adaptive Systems and Systems based on knowledge (ontologies), integrating ICT-mediated teaching and learning, e-learning, b-learning (learning in semi-classroom modality), c-learning (cooperative learning), m-learning (mobile learning) and u-learning (ubiquitous learning).
And on the other hand, the Neuropedagogical dimension integrated by evaluation and assessment, as well as by learning styles and cognitive styles. The two dimensions mentioned above intersect with personalization as a mechanism for addressing diversity in education.
The state of the art of research includes 6 lines: 1) Evaluation; 2) E-Assessment (b-learning, c-learning, m-learning, u-learning); 3) Ontologies; 4) Adaptive systems; 5) Cognitive styles and learning styles; and 6) Customization of learning activities and evaluation.
Ontologies can be considered in the computational dimension, some works of it are described below:
In [3], a mobile learning system is presented that uses a network of ontologies, which allow the representation of the contextual dimensions of a teaching-learning environment (for example, location, time, user profiles and knowledge areas). The study seeks to provide students and teachers with personalized and relevant academic information in their current study context.
Belazoui, et al. [1] propose an intelligent tutoring system that can access the educational content available on the web automatically and offer them to students as additional sources of information, highlighting the use of an improved traditional tutoring system architecture that makes use of ontologies and description logic to allow them to access various data sources on the web.
There are works that are various stages for the construction of adaptive educational systems, such is the case of [9] where the design of the Multiagent Pedagogical Support System (MaPSS) and the different scenarios of its use are presented, the system is based on a coupling of ontology and multi-agent systems for a synergy of their forces and the important contribution they can make to improve the learning-teaching process. On the other hand, in [8] shows the use of an implemented MaPSS prototype, the experiments, results and discussion. The evaluation part is also investigated in [7] and in this work a solution for the diagnostic evaluation based on the competences approach is described, especially in integration pedagogy.
In [11] a novel approach is presented that uses ontologies and knowledge models to support accessibility analysis and evaluation of learning objects. The model extends existing ontologies such as WCAG 2.0, Schema DRD, and SKOS to represent how accessibility metadata relates to the most common types of disability.
In relation to the evaluation process in adaptive systems, there is also the work of [10] who present an e-evaluation model, which can accept objects developed under open standards.
In [20] an IMS-compliant ontological learning model is developed for adaptive e-learning environments. And in [4] a review of ontology-based recommender systems in e-learning is done, the full survey shown gives an overview of the research in progress
3. Background
From the Neuropedagogical point of view, cognitive is taken up again, based on the development of psychological theories about personality and intelligence, with a systemic approach characterized by a didactic centered on processes.
When the evaluation of learning is presented in virtual learning environments, which support distance education, under a modality of b-learning (learning in semi-classroom modality), c-learning (cooperative learning), or u-learning (ubiquitous learning), systemic and process vision is critical to improving student learning.
The systemic approach also includes teachers, students, and educational resources (which of course integrate teaching strategies and techniques), as well as their interrelations in a special way with personalized assessment.
The present article aims to fully integrate several previous works that describe parts of the planned system. In previous works, an architecture of a strategic learning meta-model [17] has been designed that is divided into three layers, one of them is the intelligent layer where an ontological model was designed [17] to present the relationship between courses, students, their profiles and learning activities. Likewise, the infrastructure layer has been designed where Cloud Computing is used as the infrastructure of a learning management system (LMS) [16] and also in [15] the design of the reactive layer that contains the architecture of the regulation model.
4. General Structure of the System
As can be seen in Fig. 1, the system consists of three modules: intelligent, reactive and infrastructure module. Each of these modules is described in detail below and they are in relation to the strategic learning meta-model described in [17]. The intelligent module considers a set of ontological models that constitute the intelligent part of the architecture.
The reactive module, whose objective is to maintain a mechanism of communication and registration of activities that allow the regulation of learning until the student is self-regulated.
The infrastructure module, which allows the student to interact with the teacher, facilitators and students from various devices and media.
4.1 Intelligent Module - Ontological Model Design and Implementation
Based on the context presented, In the intelligent module, the ontological model was designed and implemented, which integrates a set of ontologies that allow us to intelligently suggest a series of recommendations for the design of the learning path of a specific student based on their learning profile of a given course.
The design and implementation of the ontological system allow to model the domain at the conceptual level from the acquisition of knowledge, and later to be transformed into a formal model in an ontological language like OWL. The methodology used for its design and implementation was Graphical Ontology Design Methodology (GODeM), composed of 7 essential steps [18]: 1) Establish the knowledge domain and the ontology scope, 2) Identification of the ontology requirements, 3) Integration of existing ontologies, 4) Design of the ontological model, 5) Implementation of the ontological model, 6) Evaluation of ontology, 7) Documentation of the ontology.
Figure 2 shows the conceptual hierarchy of classes of the ontological model. It is student-centered, based on your profile and the cognitive skills you want to develop during the course, it also determines the type of learning activities that are convenient to assess the student in a personalized way. Once the classes and subclasses were hierarchized, forming the taxonomy of ontology, 14 main classes were obtained.
Also, you can observe how the learning activities are separated from the course and relate to the tools that can be used to carry out the learning activity, at the same time it is associated with the type of cognitive ability that can help.
The ontological model is constituted by a set of four ontologies: 1) Students, 2) Course, 3) Learning Activities, and 4) Profiles. During the construction process of each of them the main features were first detected, later the design was carried out through the notation OntoDesign Graphics [19] and later the implementation in Protégé [13].
The Students ontology is fundamental in the ontological model, since the information recorded in it will be decisive to make adjustments in the learning path for the student. This ontology considers the information of each course taken by the student recording the period in which he or she enrolls, the teacher assigned to the group and the grades obtained in each evaluation activity. In addition, it incorporates information regarding the student’s learning profile, since together with the course information, the ontological model through inference rules will determine the personalized activities for the student. Next, the implementation with the different classes of the Students ontology are presented in Figure 3.
The Course ontology considers that the course is part of a curriculum. The course is composed of general objectives, diagnostic evaluation (which determines the previous knowledge and the learning profile), summative evaluation (which determines the final grade of the course), evaluation criteria, the assessment scale specifying the values used to determine the final rating. In addition, it is considered that the course is constituted by a set of teaching-learning units, each unit is associated with a topic or sub-topic, has objectives, evaluation activities and multimedia educational resources or learning objects. Following, the classes created in the Course ontology are presented (Figures 4).
The LearningActivities ontology, integrates the different types of evaluation: a) diagnostic, formative, mediating and summative; b) the evaluation activities that correspond to each type of evaluation and that integrate: 1) the evaluation criteria (if the evaluation is carried out individually or in a group); 2) the context of the evaluation (determines whether only the final product, the process performed or both are evaluated); 3) the type of evaluation that can be analytical (the activity is evaluated globally) or holistic (the evaluation is divided into sub-areas of the activity); 4) the scope of the assessment that determines whether all team members are rated with the same rating or whether criteria that define the differentiated rating for each team member are used; and 5) the learning profile corresponding to the evaluation activity.
Figure 5 shows: the implementation with the different classes and Individuals of the LearningActivities ontology.
The Profiles ontology, is used to classify students according to their learning profile, considering the two moments of the evaluation immersion model in the cognitive process. The dominant styles of perception and thought are included; however, it should not be forgotten that there can be multiple dominances. Following, the classes created in the Profiles ontology are presented (Figures 6).
By means of inferences the system determines the set of activities that are recommended contemplating the cognitive skills that the student must develop in the course. It also integrates activities according to the dominant cognitive style of the student with the aim of combining activities that the student will be able to develop efficiently what generates the motivation to continue advancing.
4.1.1. Intelligent Analysis for the Selection of Learning Activities
The intelligent analysis process is integrated by two subprocesses. In the first, the population of ontology is performed with the data of the course and the cognitive profile of the student. The second process of analysis consults the knowledge model with the SPARQL language for the selection of learning activities. The diagram of the analysis process is shown in Fig. 7
The recommendation section issues twenty-four types of recommendations based on the analysis of the cognitive skills that the student must develop, as well as on the cognitive profile of the student. Now each combination is associated with a set of learning activities that can be selected from the ontological model.
Figure 8 shows an example in pseudocode with the conditions that are applied to obtain the recommended learning activities according to the cognitive type "creative" that it is wanted promote in the course, in addition to those learning activities according to the learning profile of student.
Fig. 8 An example in pseudocode of a query to get a list of learning activities according to the cognitive type “creative”
From the recommended list, the teacher may select the activities that he or she deems most appropriate to integrate into the Personalized Virtual Learning Environment (PVLE).
The automation of the recommendations in the PVLE is in process, now the loading of the activities selected by the ontological system is done manually.
Since the objective of the ontological model is to recommend appropriate learning activities to optimize the student’s learning, it is possible to carry out this purpose once the ontologies were populated.
4.2 Reactive Module - Regulation Model
The reactive module contains a regulation model where learning activities are integrated based on the recommendations made by the ontological system. The reactive module consists of a mechanism for recording activities and accesses made in the system, as well as a set of notifications that report the progress of students to both teachers and students. The information of this layer is fundamental since it feeds the regulation model and allows to make decisions in an appropriate way to adjust the recommended learning activities, the learning path of the student.
The elements that constitute the regulation model are: teacher, students, motivational facilitator, facilitator of processes and, finally, recommended content and learning activities. The regulation model is responsible for monitoring students, motivating them and identifying recurring errors, through cooperative work among those involved.
The teacher guides and supervises teaching and learning processes. The feedback with the students is carried out through online sessions in real time once a week, in it the doubts of the students are attended, problem solving and final evaluation to validate students' learning and to be able to assign a grade. The role of the teacher is very oriented to the brain quadrant A (Analytical), its focus is to provide information, problem-solving exercises, among others.
When a course begin it starts an induction session. Students must complete a series of tests to determine their learning profile. They register with the PVLE and are ready to start working on the course learning activities. It is mandatory for each student to attend classroom teacher assessments to validate their learning. The student’s role is oriented to quadrant D (Creative); it is focused on problem solving, decision making and proposing solutions, among others.
The motivational facilitator has the task of seeking a balance in the cooperative environment, carries out an accompaniment in the training process. Its role is oriented to the cerebral quadrant C (Interpersonal) and is focused on maintaining continuous interaction.
While the process facilitator is responsible for identifying recurring mistakes of students when conducting learning activities. Notify the teacher of errors to take corrective action. The role of this facilitator is oriented to quadrant B of the brain (processes), among its functions are to monitor the effectiveness of the recommendations of implemented learning activities, verify the delivery times and the performance of students.
The diversity of learning styles is managed through a set of educational resources including animation, audio, and interactivity. For this, conceptual maps, diagrams, synoptic tables, videos, audios, reading guides, chats, forums, social networks, interactive resources, blackboards, games, among others, are used. The contents are developed as reusable learning objects.
More details of this module and its relationship with the reactive layer are found in [15].
4.3 Infrastructure Module - Personalized Virtual Learning Environment
The infrastructure module is responsible for enabling the interconnection between the intelligent module and the reactive module. Its objective is to provide interfaces that allow users to connect to the system from any type of device (computer, laptop, tablet, iPad, cell phone, among others).
The architecture of the computational solution integrates four components: Learning Management System (LMS), Personalized Virtual Learning Environment (PVLE), Management Information System (MIS) that handles the information of the Ontological Model (OM). The components are implemented as a Cloud Computing solution, based on the work of [6] and [5]. The proposal presented in this study also integrates the ontological model for the personalization of learning activities that replaces the MIS module proposed by Gusev [5].
The infrastructure module is associated with the PVLE that enables interaction between the actors of the system. It is integrated by the SAKAI platform and the integration of the various types of evaluation. The course of Structured programming is divided into 8 units, each has individual and collaborative activities, self-study educational resources and personalized learning activities according to the student’s profile. These activities are configured based on the recommendation received from the system.
The Personalized Virtual Learning Environment is a suitable environment for learning, as it integrates multiple tools that are traditionally used on the internet such as the use of social networks, as well as multimedia educational resources. Various multimedia educational resources were developed for this purpose. Students are free to use any resource they wish. Even if the topic is not clear to you from one resource, you can access resources from another style and complement your learning while developing your other channels of perception and your skills as a critical receiver. In addition, the use and creation of mental maps is incorporated as a tool to integrate knowledge. The self-evaluations that are integrated in the tool allow the student to verify if they have understood the fundamental concepts of each unit.
The PVLE is constituted by an application server, three frameworks: one for the centralization of services (Spring), another for the mapping to database (Hibernate), and the third for the implementation of the controller and Vista (Struts).
More details of this module and its relationship with the infrastructure layer are found in [16].
5. Experiments and Results
During the development of the research, tree stages of experiments were carried out that allowed building and modifying parts of the system.
Were developed 11 experiments with a duration of twelve weeks each one: at autumn 2011 (11-O), at winter 2012 (12-I), at spring 2012 (12-P), at autumn 2012 (12-O), at winter 2013 (13-I), at spring 2013 (13-P), at autumn 2013 (13-O), at winter 2014 (14-I), at spring 2014 (14-P), at winter 2015 (15-I) and at spring 2015 (15-P).
To carry out the experiments an experimental, descriptive-explanatory methodology was used, the following parameters were considered:
– Reference population: Engineering students who enroll in the course in each trimester (11-O - 15-P).
– Characteristics of the sample: Students who are in the first year of any of the 10 degrees.
– Type of schools and location: Universidad Autónoma Metropolitana-Azcapotzalco. México.
– Student course: Structured Programming.
– Characteristics of the students: The thinking style of the students was determined.
– Sample size: approximately 120 students out of a population of approximately 600 students per trimester.
5.1. Stage 1
The first experiments aimed to determine the impact of six variables on the retention rate and student approval. These experiments were performed in the 11-O and 12-I trimester. Customization of learning activities was not considered. The results of these experiments were the point of comparison with those of the subsequent stages in which an intervention is made in the teaching and learning process.
The first stage of experiments made it possible to identify the right moments to make an intervention in the teaching and learning process. Some problems to be addressed were also identified, including the need for a pedagogical model and an instructional design to formalize, standardize and control the experiments of the next stages.
The experiments in this stage integrate a proof of concept where the course is carried out in a normal way, that is, without personalizing the learning activities.
5.2. Stage 2
In the second stage of experiments, the Cognitive Instructional System mediated by ICT was applied and learning activities were customized. These experiments were performed in the 12-P, 12-O, 13-I, 13-P, and 13-O trimesters.
The second stage of experiments made it possible to identify the elements that make up the ontological model to obtain recommendations and personalize learning activities. The GODeM methodology was applied [18], and the Ontological Model was designed to support the personalization of activities, being implemented in Protégé. Finally, the technology architecture of the Personalized Virtual Learning Environment was designed and implemented as a Cloud Computing solution [16].
In the second stage, experiments are carried out in which the personalization of learning activities is applied.
5.3. Stage 3
During the third stage of experiments a model was applied for the selection of learning activities according to their degree of difficulty. After the conclusion of an experiment, the results in the grades obtained by the students are evaluated and the learning activities whose degree of difficulty is in an intermediate range (0.2 and 0.8) are selected.
The results are evaluated to determine the impact they have on dropout and approval rates. These experiments were conducted in the 14-I, 14-P, 15-I, 15-P trimesters.
The third stage of experiments identified that the degree of difficulty of learning activities has a direct impact on the student’s school performance. For this reason, a model was defined that allows assessing the degree of difficulty of learning activities, identifying learning activities that are very easy or very difficult to replace with others and applying them in the next course. The model allows the acceptable range of the degree of difficulty from 0 to 1 to be established.
In stage 3, a variable not contemplated was identified: the level of complexity of the learning activity, then, several experiments are carried out in which the level of complexity of each activity is assessed and it is decided to establish a threshold to eliminate the learning activities very easy and very complicated [14].
The results of the application of these three stages can be seen in Table 1 and Figure 9, which illustrates the trend of the approval rate with the R2. The graph is divided into the 3 stages considered in the research. Stage 1 presents the results of the concept test where no change is applied in the delivery of the Structured Programming course. In Stage 2, learning activities are customized according to the learning style. Finally in Stage 3, the model is applied to maintain learning activities with an intermediate difficulty level. It is noted that the pass rate improves by maintaining the level of difficulty of learning activities at an intermediate level.
Table 1 Experiments data (trimesters 11-O to 15-P)
| # | Trimester | Approval |
| 1 | 11-O | 51.4619883 |
| 2 | 12-I | 47.3372781 |
| 3 | 12-P | 23.715415 |
| 4 | 12-O | 49.2307692 |
| 5 | 13-I | 31.3609467 |
| 6 | 13-P | 20.2020202 |
| 7 | 13-O | 18.1818182 |
| 8 | 14-I | 24.137931 |
| 9 | 14-P | 36.8421053 |
| 10 | 15-I | 52.6315789 |
| 11 | 15-P | 68 |
6. Conclusions and Future Work
The situation of school failure is a complex problem that must be analyzed from a systemic point of view, for the approach of alternative solutions. One of the high-impact variables is evaluation since it is the culmination of failure and therefore one of the factors that impact school failure. For this reason, a system was developed with the central objective of improving pass rates while facilitating and promoting learning within the evaluation process.
This work makes contributions from two dimensions, neuropedagogy and computational. This research has come a long way, from the 11-O and 12-I trimester, that is to say stage 1 where the concept tests were carried out, which made it possible to identify the ideal moments for making an intervention in the teaching-learning process and going through stage 2 where important contributions were made from the computational dimension, among them an ontological model as support for the personalization of activities, the design of a technological architecture of the personalized virtual learning environment that was implemented as a cloud computing solution, all these elements are part of the strategic learning meta-model and are key parts of the intelligent system modules for the customization of evaluation activities.
Stage 3 identified that the degree of difficulty of learning activities has a direct impact on the student’s school performance. These findings have implications that allow to base the instructional design, considering the diversity and helping in the cognitive student process to produce a strategic learning.
In the future, it is intended to automate the process of including the recommendations drawn from the ontological model in the PVLE. Also, this system could be implemented in other units of the university.
