SciELO - Scientific Electronic Library Online

 
vol.19 número2Competencia matemática de los alumnos en el contexto de una modelización: aceite y aguaPortefólio reflexivo de matemática enquanto instrumento de autorregulação das aprendizagens de alunos do ensino secundário índice de autoresíndice de materiabúsqueda de artículos
Home Pagelista alfabética de revistas  

Servicios Personalizados

Revista

Articulo

Indicadores

Links relacionados

  • No hay artículos similaresSimilares en SciELO

Compartir


Revista latinoamericana de investigación en matemática educativa

versión On-line ISSN 2007-6819versión impresa ISSN 1665-2436

Relime vol.19 no.2 México jul. 2016

http://dx.doi.org/10.12802/relime.13.1922 

Artículos

On-line assessment of the process involved in maths problem - solving in fifth and sixth grade students: self - regulation and achievement

Evaluación on-line del proceso de resolución de problemas matemáticos en estudiantes de quinto y sexto curso: auto - regulación y logro

Trinidad García* 

Lucy Betts** 

Paloma González-Castro*** 

Julio A. González-Pienda**** 

Celestino Rodríguez***** 

*Universidad de Oviedo, España. garciatrinidad@uniovi.es

**Nottingham Trent University, UK. lucy.betts@ntu.ac.uk

***Universidad de Oviedo, España. mgcastro@uniovi.es

****Universidad de Oviedo, España. julioag@uniovi.es

*****Universidad de Oviedo, España. rodriguezcelestino@gmail.com

Abstract:

The aim of this study was to test a method to assess the processes involved in mathematical problem solving, based on the Triple Task methodology and Self - Regulated Learning principles. This protocol was administered to 510 fifth and sixth grade students from Northern Spain, who carried out two mathematical tasks of varying difficulty. The results derived from the total sample indicated the presence of ineffective planning strategies and a lack of revision mechanisms. However, comparisons between groups with different achievement in the tasks revealed the sub - processes involved in planning (especially the use of representation strategies) as important determining factors in students' success rates, with these exerting a greater effect as task - difficulty increased.

Key words: Self - Regulated Learning; mathematics; problem - solving; process

Resumen:

El objetivo de este estudio ha sido poner a prueba un método de evaluación del proceso implicado en la resolución de problemas matemáticos, basado en la metodología de la Triple Tarea y en los principios del Aprendizaje Auto-regulado. Este protocolo se administró a 510 estudiantes de quinto y sexto curso procedentes del Norte de España, los cuales realizaron dos tareas matemáticas de diferente dificultad. Los resultados indicaron la presencia de unas estrategias de planificación ineficaces, así como la ausencia de mecanismos de revisión. Sin embargo, el análisis de las diferencias entre los grupos con diferente rendimiento en las tareas reveló los sub-procesos implicados en la planificación, y especialmente el empleo de estrategias de representación de la información, como determinantes importantes en el éxito de los estudiantes, ejerciendo un efecto mayor conforme la dificultad de la tarea aumentó.

Palabras clave: Aprendizaje Auto - regulado; matemáticas; proceso; resolución de problemas

Resumo:

O objetivo deste estudo foi testar um método de avaliação do proceso envolvido na resolução de problemas matemáticos, com base na metodologia da Tarefa Tripla e nos princípios da aprendizagem auto - regulada. Este protocolo foi administrado a 510 alunos de quinto e sexto ano do norte da Espanha, que fizeram duas tarefas de matemáticas com diferentes graus de dificuldade. Os resultados indicaram a presença de algumas estratégias de planeamento ineficazes e a falta de mecanismos de avaliação. No entanto, a análise das diferenças entre os grupos com um desempenho diferente na tarefa, revelou que os sub - processos envolvidos no planeamento, especialmente o uso de estratégias de representação da informação, como determinantes importantes para o sucesso dos alunos, exercendo um efeito maiorquando a dificuldade da tarefa aumento.

Palavras chave: Aprendizagem auto - regulação; matemática; processo; resolução de problemas

Résumé:

L'objectif de cette étude était de tester une méthode d'évaluation du processus impliqué dans la resolution de problèmes mathématiques, basée sur la méthodologie de la Triple Tâche et les príncipes de l'Apprentissage Autorégulé. Ce protocole a été administré à 510 élèves en sixième et cinquième année, du Nord de l'Espagne, quiont effectué deux tâches mathématiques de différente difficulté. Les résultats indiquent la présense de quelques stratégies de planification inefficaces et le manque de mécanismes de contrôle. Cependant, l'analyse des différences entre les groupes avec l'exécution des tâches différentes a révélé les sous - processus impliqués dans la planification, et en particulier l'utilisation de la représentation des informations stratégiques, comme des déterminants importants dans la réussite des élèves, en exerçant un effet plus grand que la difficulté de la tâche a augmenté.

Mots clés: L'auto - apprentissage; le traitement; les mathématiques; la résolution de problèmes

1. Background

It is widely known that the development of problem - solving skills is an important issue in formal education. In fact, authors such as Lazakidou and Retalis (2010) linked it to Life - long Learning in the sense that "engagement in problem - solving activities helps students to acquire useful attitudes such as thinking, flexibility, creativity, and productivity, which are very important to real life " (p. 3). In the current educational context, mathematical problem solving takes a greater relevance; this activity, which starts in the first years of Elementary Education, may be the basis to develop basic problem - solving skills.

Mathematical problem solving is a complex cognitive activity that involves multiple processes. A student's achievement in these tasks relies on the integrated application of cognitive, metacognitive, and motivational components (Cleary & Chen, 2009; Kajamies, Vauras, & Kinnunen, 2010; Montague, Enders, & Dietz, 2011; Voyer, 2011). In fact, while it has been supported for many years that a well - organized and flexibly accessible knowledge about mathematical facts, symbols, algorithms, concepts and rules may be the core of problem - solving skills, current literature has established that strategy use based on Self - regulated Learning (SRL) is linked to mathematical capacities which are an important determining factor in solving mathematical problems (Gálvez, Cosmelli, Cubillos et al., 2011; Geary, 2004; Jarero, Aparicio y, Sosa, 2013; Lazakidou & Retalis, 2010; Montague, 2008; Pennequin, Sorel, Nanty, & Fontaine, 2010; Pereis, Dignath, & Schmitz, 2009; Schmitz & Perel, 2011; Verschaffel, Greer, & De Corte, 2000).

SRL refers to those proactively initiated thoughts, feelings and behaviors which are planned and cyclically adapted based on self - generated or performance feedback in order to attain personal goals (Zimmerman, 2000). SRL involves those aspects which facilitate the control and regulation of students' cognitive systems and learning processes, and comprises three sequential phases: forethought (i.e., processes that precede efforts to learn or perform), performance control (i.e., processes occurring during learning efforts), and self - reflection (i.e., processes occurring after learning or performance such as evaluate the effectiveness of one's learning methods or results; Zimmerman, 2000; in Cleary & Chen, 2009). These processes, which are based on an interplay of personal and task characteristics and the strategies available in a learning situation, help students to monitor and check their thoughts and are particularly useful when solving novel or challenging problems (Lazakidou & Retalis, 2010; Pennequin et al., 2010; Rosenzweig, Krawec, & Montague, 2011; Throndsen, 2011; Zimmerman & Schunk, 2008). However, studies have shown that students tend to demonstrate poor metacognitive skills when they are engaged in mathematical problem - solving situations. Many students forge ahead without considering alternative decisions, jumping immediately into calculations, giving impulsive reponses, and using trial and error as strategies. Students often get stuck in irrelevant details of the task or fail to verify solution paths and evaluate answers, focusing on superficial measures of progress (Cleary & Chen, 2009; Kramarski & Gutman, 2006; Montague, 2008; Montague et al., 2011; Pennequin et al, 2010; Pereis et al., 2009; Veenman, 2005). In this sense, not only do students with learning disabilities show problems in these issues. In fact, typical students may also not realize the importance of regulating their thoughts and behavior, or they may simply not know how to self - regulate properly. Thus, many students often need support to regulate their learning processes (Azevedo & Cromley, 2004; Butler & Cartier 2005; Ifenthaler, 2012; Kramaski & Gutman, 2006).

In this context, one of the biggest challenges for researchers is to design measures that provide access to internal cognitive structures and functions involved in both, self - regulation and problem solving. Most studies have been based on applying standardized measures such as questionnaires or structured interviews; this kind of measures involves asking students about how they solve a problem or the extent to which they use different strategies. Additionally, students are also asked about different situations in which they would use (or not use) a specific strategy or what approximation would be the best in each case. Students' responses to the metacognitive questions are scored depending on the quality of the response and a total score is calculated. These kind of assessments, although widely applied (Greene & Azevedo, 2010; Shea & Bidjerano, 2010; Zimmerman & Schunk, 2008), may yield inconsistent or inaccurate information due to inaccurate memory or perhaps even response biases such as social desirability, and do not provide information about how learners transfer their knowledge to regulate their problem - solving activities. As many authors have pointed out, these tools are useful to assess a student's declarative and situational metacognitive knowledge, but do not provide any information about the mental processes underlying task performance and the student's achievement levels which together represent important issues in learning and instruction (Clearly & Chan, 2006; Veenman, 2011). In response to this, Veenman (2011) established the distinction between off-line and on-line methods in the assessment of strategy use and learning. Off-line methods refer to questionnaires and interviews which are administered either before or after task performance, while on-line methods concern measurements taken concurrent to task performance, such as Think - aloud or Triple Task procedures. These measures may be especially useful to provide data about student reasoning abilities during problem - solving activities.

In Think - aloud protocols (Rosenzweig et al., 2011) learners verbalize their thoughts and cognitive activities while engaged in task execution. The usefulness of these procedures lie in the fact that they provide access to students' short-term memory abilities, which reflects cognitive processing during task completion. This approach allows the researcher to obtain accurate information about metacognitive skills and strategies while students are asked to simply verbalize what they are thinking at each moment. After transcription and coding, undirected verbalizations are recognized as valid expressions of the students' cognitive and metacognitive processes. Studies in this area are limited, although some interesting data support the usefulness of the protocols in assessing metacognitive processes during mathematical problem-solving tasks (Lazakidou & Retalis, 2010; Ostad & Sorenson, 2007; Throndsen, 2011).

On the other hand, Triple Task procedures have been widely applied in the scientific study of reading comprehension and written composition, and more recently in note - taking activities (Fidalgo, Torrance, Robledo, y García, 2009; García, Rodríguez, Pacheco, & Diez, 2009; Piolat, Barbier, & Roussey, 2008; Piolat, Kellog, & Farioli, 2001; Olive et al., 2002; Piolat & Olive, 2000; Piolat, Olive, & Kellogg, 2005; Torrance & Galbraith, 2006).

The Triple Task method is rooted in the so-called "Doble Task " protocols. Although the present study focuses on the use of Triple Task procedures, a description of both methods is provided below:

The Double Task procedure provides information about the cognitive effort engaged in higher - order cognitive tasks, such as comprehension or text production. Participants are asked to perform concurrently a primary task and a secondary probe task. For example, while composing a text (the primary task), participants must react as fast as possible to tones (the secondary task by pressing a mouse button or by saying 'stop' to a microphone linked to a vocal key) that are periodically distributed in a random interval (generally between 15 and 45s). Reaction time (RT) in this dual - task situation is compared with a control condition when the probe is responded to as a single task. The degree of interference in RT (IRT) caused by the primary task provides a measure of the amount of cognitive effort designated to composition or another cognitive task. However, although widely used, this measure does not provide information about the process that underlies the cognitive task under consideration.

It is the Triple Task procedure which provides additional information about the temporal organization and the cognitive processes underlying a cognitive activity by adding a third task, consisting of asking students for an immediate directed introspection after each tone's detection. Students have to categorize their thoughts at the moment the tone is presented. Thus, in Triple Task studies students perform: a) the primary task under investigation, b) the secondary probe task, and c) a third task in which participants are asked to label the process that was interrupted by the probe. For this last task, participants are trained in direct retrospection in order to better identify and report the cognitive processes present at each moment. As a measure of the process, the Triple Task method can be differentiated from Think - aloud protocols in the following aspects: a) the Triple Task uses directed introspection as opposed to the undirected introspection involved in Think - aloud, b) a system of categories can be given in order to facilitate categorization and facilitate exploration in to sub-processes, c) response transcription and coding are unnecessary, and d) inter-rater agreement does not have to be calculated. To date however, this method has neither been applied to the analysis of the processes underlying mathematical problem - solving, nor from the point of view of investigating the metacognitive skills involved in these tasks. Additionally, as happens with Think - aloud protocols, most Triple Task studies have been conducted with small sample sizes, mainly due to their own methodology feautures, which make them less economic to implement.

In this context, we introduce our own process assessment tool, the Triple Task Procedure in Mathematics - TTPM (García y González - Pienda, 2012). It is based on the Triple Task technique and suitable for application in larger samples through a MOODLE platform. One of the novelties of this method is the introduction of a new system of categories to analyze student processes during problem solving. It was designed according to the PLEJE model of SRL ("Planificación - Ejecución - Evaluación" in Spanish: Rosário, Mourão, Núñez, González - Pienda, & Solano, 2008; Zimmerman, 2000), which establishes three main phases: Planning, Execution, and Revision; and Bransford and Stein's (1993) IDEAL problem - solving model, which stablishes five stages to successfully solve a problem: Identifying potential problems, Defining and representing the problem, Exploring possible strategies, Acting on those strategies, and Looking back and evaluating the effects of those activities. A more detailed description of the assessment process and this tool, which was administered to 510 students from fifth and sixth grade belonging to 12 private and state schools in Asturias (Northern Spain), is provided in the method section. The main aim of this study was to analyze the usefulness of TTPM as a measure of the process during the performance of two mathematical problems of varying difficulty. Specifically, this study sought to answer the following questions: a) do students follow Self - Regulated Learning stages while trying to solve mathematical problems? b) are there differences in the process followed by students with different achievement in the tasks? and c) do these differences remain constant when the difficulty of the tasks varies?

2. Method

2.1. Participants

The study involved 510 students from fifth and sixth grade students from 12 private and state primary schools in northern Spain (Mean age: 11 years, SD:0.71). Students belonged to 32 different classrooms. Gender distribution was balanced. Specifically, 255(50%) were female. Of the total sample, 210 students (41.2%) attended fifth and 300 (58.8%) sixth grade. Gender and level (fifth or sixth grade) were taken into account as potential mediating variables in subsequent analyses.

Sample selection was made through convenience or accessibility procedures. Students volunteered for the study and presented informed consent from their parents.

2.2. Measures

Mathematical problems: Students completed two relatively complex mathematical word problems taken from the book "Problem - solving and comprehension" (Whimbey & Lochhead, 1999) published in Spanish. This book provides a systematic review of the characteristics that define a good problem solver, while proposing strategies and activities to develop these skills, mainly based on mathematical reasoning. Tasks differ in the amount of information to be processed, the number of associations presented and the number of parameters to calculate.

Process measures: Evidence of student metacognitive process was obtained by means of the TTPM (García y González - Pienda, 2012). It consists in the following: after a RT probe task, and while completing the mathematical problems, students are presented with an electronic tone at a randomized interval of 40-45 seconds. At this point students are asked to categorize their activities or thoughts. In order to help students, categorize these processes, students are provided with a category system (see Table I). They have to indicate by mouse - click which of nine the categories (reading, drawing or summarizing, recalling similar problems, thinking about a solution, mental calculation, writing, reviewing, correcting mistakes, or "other") best describes the processes in which they are involved when the tone is presented. The category system is based on Bransford and Stein's (1993) IDEAL problem - solving model and the PLEJE model of SRL (Rosário et al., 2008; Zimmerman, 2000), previously described in this study. The category called "other" was incorporated to gather all those thoughts or actions unrelated to the mathematical task (e.g., day - dreaming). Nevertheless, previous research indicates that such unrelated processes are generally very scarce (Olive et al., 2002). The eight proposed processes are also organized into three, higher - level categories corresponding to SRL phases: Planning, Execution and Revision. Table I shows the category system used for students to categorize their processes during the task, based on the correspondence between the stages and phases involved in both SRL and IDEAL models.

Table I Category system. Based on the Self - Regulation Model (Rosário et al., 2008; Zimmerman, 2000) and the IDEAL Model (Bransford & Stein, 1993

2.3. Procedure

The study was conducted in accordance with The Helsinki Declaration of the World Medical Association (Williams, 2008), which reflects the ethical principles for research involving humans. Participants were tested colectivelly within a 45 - minute time - frame. A maximum of 20 participants were tested at a time. The first phase consisted of training students to familiarize them with the system of categories and the assessment procedure. The hypothetical case of a boy who was the same age as them (Álex) and who tried to solve a mathematical problem was used for this purpose. After training, students performed a category recognition test consisting of 12 multiple - choice items with four alternatives. Students were asked to indicate the category that best expressed each proposed activity (e.g., for the statement "Alex realised he made a mistake, so he is erasing", alternatives were: Álex is thinking about a solution, writing, reviewing or correcting mistakes).

Once the system of categories was undertood, students were informed that they would occasionally hear a tone coming from the computer at varing intervals, and they were instructed to quickly react to every tone by clicking on the computer mouse with their dominant hand. This tone was presented in a randomized interval of 10-15 seconds. Then they were informed that this same sound signal would appear while they tried to solve the tasks. They were asked to choose the category that best represented what they were doing in each moment. The tone was presened in a randomized interval of 40-45 seconds. Concurrent to the tone, a pop-up appeared on the computer screen. It showed a box with the category system. Students were able to select a category with a mouse-click. Mathematical problems were provided on paper. Students could use that paper to write whatever they needed, with the condition that they had to write their answer on the paper when they finished the task. As data were collected from individual students simultanteoulsy, head - phones were provided in order that other students were not disturbed. Data collection was implemented through Moodle platform (https://moodle.org). A special module was created on the platform, that hosted the evaluation procedure and stored data. In order to acoomplish that, a multidisciplinary team, including psychologists, teachers, and a computer engineer (responsible for all technical issues) collaborated during the study. Students accessed this platform through an individual username and password in order to guarantee their anonymity. Once data were obtained and stored, they were automatically transferred to an Excel file for subsequent processing.

Finally, students who showed difficulties understanding the category system (less than 90% correct responses in categorization test) were excluded from the analyses. Process variables were based on frequency counts. Frequency of each sub-process or category was established by dividing the frequency of election of that category by the total number of elections done across categories. These frequency counts were then transformed into percentages by multiplying the quotient by 100. Finally, students' achievement in the mathematical tasks was established in terms of success (1) or failure (0) according to the answers given on paper. Given the main aims of this study, centered in the process, RT's as a meaure of effort were not taken into account for data analyses.

2.4. Data analyses

Ex - post - facto descriptive and comparative design was used in this study. After describing the profile of process followed by the total group in each task, Multivariate Analysis of Covariance (MANCOVA) analyses with process variables (8 categories and 3 higher - level categories) as dependent variables, and gender and level as covariates, were conducted in order to investigate differences between groups with different achievement. Tasks were analysed separately in order to determine the importance of task difficulty as a factor explaining these differences. Effects were significant at p < .05 unless otherwise stated. Data were analyzed with SPSS 18. The additional category named "Other" was not taken into account for further analyses.

3. Results

3.1. Student problem - solving process

In order to demonstrate if a student's problem-solving process responds to SRL phases, Figure 1 shows the profile of the process followed by students in Tasks 1 and 2. This profile indicates similar processes for both tasks, characterized by a higher frequency of selected categories in planning and execution phases. The means (Table II) indicated a percentage of 49.46% in Task 1 and 45.53% in Task 2 of the students' reports referring to planning categories, while those related to execution were 40.43% and 45.95%, respectively. With respect to the revision phase, mean percentages of categories reported within this phase were 10.15% in Task 1 and 7.57% in Task 2 across categories. It must be noted in this case the high means found in the planning phase (even above those found in the execution phase). This is because the planning phase consists of four sub-processes or categories, while the other phases comprise two.

Figura 1 Process profile of the total sample. Tasks 1 and 2.1 = reading; 2 = Drawing or summarizing; 3 = recalling similar problems; 4 = thinking about solutions; 5 = mental calculation; 6 = writing; 7 = reviewing; 8 = Correcting mistakes; Plan = planning phase; Exec = execution phase; Rev = revision phase 

In respect to the categories or sub-processes within each phase, in the planning phase "thinking about solutions" and "reading" were the most frequently reported categories, with mean percentages of 19.20% in Task 1 and 17.02% in Task 2 for the first category, and 15.98% and 13.57% respectively for the second category. "Recalling similar problems" was the least reported category or sub-process, although higher in the second task (2.97% in Task 1 and 6.67% in Task 2). "Drawing or summarizing" represented 11.29% of the total categories reported in Task 1 and 12.36% in Task 2, being an important sub-process within planning. With respect to execution phase, "mental calculation" was the category most frequently reported in this phase, with mean percentages of 28.37% in Task 1 and 31.83% in Task 2. "Writing" also represented an important percentage of the categories used, with a 12.09% frequency reported in Task 1, and 14.14% in Task 2. In relation to the revision phase, "reviewing" and "correcting mistakes" were two of the least frequently reported categories by students in both tasks (6.36% in Task 1 and 4.29% in Task 2 for "reviewing"; 3.79% and 3.29 respectively for "correcting mistakes").

Finally, standard deviations in Table II showed a high variability among the processes shown by students in both tasks, mainly in the categories less reported ("recalling similar problems", "reviewing" and "correcting mistakes").

Table II Means and Standard Deviations in process variables. Total sample (N=510). Tasks 1 and 2. 

3.2. Process differences between groups with different achievement (success / failure)

Tables III and IV present the mean scores and standard deviations in process variables for groups with different achievement (Group 1 = success; Group 2 = failure) in Tasks 1 and 2, respectively. Group - compositions revealed the presence of a high proportion of students solving the mathematical problems unsuccessfully. Specifically, 356 students (69.80%) failed to solve the first task while 327 students (60.11%) gave an incorrect answer on the second task. As one of the aims of this study was to know whether there were differences between tasks with varying difficulty, separate analyses for each task were conducted.

For the first task, means in Table III indicated a trend to plan more, and to execute and review less, in the Group 1 (success). MANCOVA analyses showed the presence of statistically significant differences between the groups (λ = .957; F (11,496) = 2.012; p =.026; ηp 2 = .043). However, these differences were only found in "drawing or summarizing" (F (1,506) = 4.548; p = .033; ηp 2 = .009). As happened with planning, this sub-process was more frequently reported by students who successfully solved the task. Gender and grade level, which were included as covariates, did not generate any differences in the process (p = .461 and p = .123, respectively).

Table III Means and Standard Deviations in process variables for each group. Task 1 

With respect to the second task, the means in Table IV showed a different pattern of results, with students who successfully solved the problem reporting using less sub-processes related to planning, and more related to the execution and revision phases. MANCOVA analyses indicated the presence of statistically significant differences between groups (λ = .951; F (11,496) = 2.303; p = .009; ηp 2 = .049). These differences were found in those categories or sub-processes within the planning phase: "reading" (F (11,506) = 5.499; p = .019; ηp 2 = .011), "drawing or summarizing" (F(11,506) = 4.604; p = .032; ηp 2 = .009), "recalling similar problems" (F (11,506) = 5.209; p = .023; ηp 2 = .010) and "thinking about solutions" (F (11,506) = 5.387; p = .021; ηp 2 = .011). Differences in the planning phase were close to statistical significance (F (11,506) = 3.825; p = .051; ηp 2 = .006). Means revealed that students who successfully solved the task (Group 1) reported reading, recalling similar problems, and thinking about solutions less frequently than the other group. As with the first task, successful students also reported using representation strategies (i.e., drawing or summarizing) more frequently than their peers. Gender and level did not generate differences in the process (p = .590 and p = .230, respectively).

Table IV Means and Standard Deviations in process variables for each group. Task 2 

4. Discussion and conclusions

It has been argued that it is essential to identify economic, fast, reliable, and valid techniques to elicit and analyse cognitive processes involved in different cognitive tasks (Ifenthaler, 2008; Ifenthaler, Masduki, & Seel, 2011). In this context, the main aim of this study was to analyse the usefulness of a new assessment tool based on the Triple Task procedure and designed to assess the metacognitive and self - regulated processes shown by the 510 fifth and sixth grade students during the performance of two mathematical problems of varying difficulty. Specifically, this study tried to answer these questions:

Do students follow Self - Regulated Learning stages while they try to solve mathematical problems? Global results indicated the presence of important differences among processes followed by students, as indicated by the high standard deviations. This variability was even higher in the categories or sub-processes less frequently reported by students in general: recalling similar problems, reviewing and correcting mistakes. The pattern of results was similar in both tasks, and was characterized by a lack of evaluation strategies. However, sub-processes related to Planning and Execution phases were the most frequently reported, respectively. These result may seem to contradict some previous research pointing out that student commonly tend to be impulsive in problem - solving situations, spending little time making a plan to develop (Cleary & Chen, 2009; Kramarski & Gutman, 2006; Montague, 2008; Montague et al., 2011; Pennequin et al, 2010; Pereis et al., 2009).

However, despite the fact that these results could indicate that once students properly planned an activity, making mistakes became less likely and reviewing activities less necessary, students' achievements in both tasks revealed a high proportion of failure. This indicated that planning strategies were not effective enough. In fact, an analysis of the sub-processes involved in the planning phase revealed that thinking about solutions and reading were the most frequently reported sub-processes within planning, while the use of potentially more useful strategies such as information representation and organization were less frequent among students. This result could indicate a trend to use passive solving strategies or even comprehension difficulties.

Are there differences in the processes followed by students with different achievements in the tasks? Analyses revealed the existence of statistically significant differences between groups. These differences were located in the subprocess of drawing or summarizing in Problem 1, while significant differences were found in all the sub-process within the planning phase in Problem 2. The existence of more significant differences in this second problem could be related to its characteristics (i.e., more complex relationships to stablish, parameters to calculate, etc.). Thus, this task may result more difficult, suggesting a more important role of planning strategies when problem difficulty and demands increase. Regarding the pattern of differences, students who successfully solved the tasks tended to draw and summarize more, but they recalled similar problems, read and thought about solutions proportionally less than those who failed to solve the tasks. These results are consistent with the statement above about the effectiveness of planning strategies, and suggest the use of more active processes by students with high achievement. In this sense, the strategy that showed to be effective in both tasks was drawing or summarizing (i.e. using information representation and organization strategies). Making this kind of representations would correspond to the "translation" process, characterized as an essential part of solving mathematical problems by many authors (Abdullah, Zakaria, & Halim, 2012; Csíkos, Szitányi, & Keleme, 2012; Díez-Palomar, Menéndez, & Civil, 2011; Pantziara, Gagatsis, & Elia, 2009; Stylianou, 2011). An example of graphical representation, made by a participant in this study, is shown in Figure 2.

Figura 2 Example of students' answers in the second task 

The use of graphic representations supposses an important part of planning strategies. As Montague and others (2011, p. 263) argued, "problem representation involves translating and transforming linguistic and numerical information into verbal, graphic, symbolic, and quantitative representations that show the relationships among the problem parts prior to generating appropriate mathematical equations or algorithms for problem solution" . In fact, Abdullah and others (2012) showed in their study that when the teaching approaches encourage students to apply thinking strategies through using visual representation, students are able to gain a better conceptual understanding and eventually improve their mathematics achievement. They attributed this effect to a more active involvement of students in their learning process. However, representation is not only a matter of copying what one sees. As the example above shows, it involves a process of personal re-organization, inventing or adapting conventions of a representational system for the purpose at hand (Stylianou, 2011).

Are there differences between tasks with varying difficulty? Although differences in the main phases of planning, execution and review were not statistically significant, analyses revealed the existence of significant differences in the sub-process involved in planning, separately. In this way, differences in the metacognitive process showed by the groups with different achievement were stronger in Task 2, which implied more relationships to stablish and maybe more complex. This would initially suggest a more important role of planning strategies when task difficulty and demands increase.

In summary, the results found in this study would seem to contradict those studies that indicated that students do not plan, jumping immediately to calculations without considering alternatives to solve problems or giving impulsive responses (Cleary & Chen, 2009; Kajamies et al., 2010; Kramaski & Gutman, 2006). In fact, students in this study reported having planned. However, an in-depth analysis of the sub-process involved in planning revealed the presence of ineffective planning strategies implemented by the overall group, which was translated into a greater percentage of failure in both problems. In this sense, another important finding of this study is that students had serious difficulties to evaluate their progress. They also showed a tendency to use familiar procedures, such as performing calculations, as a method to solve the problems. Thus, students in this study did not self - regulate or implement effective metacognitions during problem solving. However, differences between groups with different achievement levels suggested that, despite the fact that students did not show a properly self - regulated process, a proportion of them were able to use more effective and active strategies while solving the problems, such as organizing or representing the information. This is in fact an important determinining factor in successful mathematical problem solving. Finally, the use of effective planning strategies seemed to be most important when the task difficulty and demands are greater.

These findings add evidence to the utility of process measures, specifically those based on directed and concurrent self - reporting, in the assessment of cognitive activities (in this case mathematical problem - solving tasks). The usefulness of the assessment tools presented in this studylies, however, in the systems of categories designed, which makes it possible to divide mathematical problem - solving tasks into sub-processes or isolated activities, and provides in-depth insights about a student's strengths or weaknesses, thereby helping to understand their success or failure from the point of view of SRL. This is not only important for instruction, but specially for student learning. In this sense, as Ifenthaler (2012) pointed out, the key link between knowledge about (and the regulation of) one's own problem - solving activities may lie in reflective thinking. Making students aware of their own learning processes may help them generate information about the efficiency of their problem-solving strategies and successfully implement that knowledge in the ongoing problem-solving process, thereby being able to control and regulate their cognition, effort and behavior.

It is necessary to acknowledge, however, the following limitations in the present study: first, as self - regulation is a cyclical process (Zimmerman, 2000), analyzing the recursion of the process followed by students may bring useful information about its temporal course and the metacognitive mechanisms involved. In this sense, identifying different patterns of performance may also help to reduce the high variability whithing - subjets found in this study; second, motivational and affective components, such as self - efficacy beliefs, task interest and perceived instrumentality, have been demonstrated to play an important role in mathematical performance and problem solving situations (Ahmed, Minnaert, Kuyper, & Van der Werf, 2012; Cueli, García y González - Castro, 2013; Dettmers et al. 2011; Hoffman, 2010). The study conducted by Ahmed et al. (2012) revealed that a good students' performance in addition and subtraction tasks was related not only to the student's use of advanced mathematics strategies and SRL competence, but also to the ability attribution for success, effort attribution for failure, and high perceived self - efficacy when using specific strategies; third, it would be interesting to extend the number of mathematical problems, also adding different levels of difficulty, which would provide more accurate information about the influence of task difficulty in problem - solving processes; additionally, in order to provide support to the usefulness of the tool applied in this study as a measure of self - regulatory processes involved in mathematical problem solving, analyzing its relationship with other standardized instruments such as questionnaires and interviews would be useful. Complementary use of both measures would provide a better understanding of self - regulatory mechanisms exhibited by the students, thus helping to design more adapted instructional strategies; finally, and given the characteristics of the sample (students belonged to different schools in a specific area in Northern Spain), extending the research to other areas would be advisable in order to better stablish the scope of the findings. This is, however, the first approach to the study of the usefulness of the Triple Task technique in the assessment of the process involved in solving mathematical problems. Therefore, the results from the present study actually open the way for future research. In this sense, it would be interesting, for example, to conduct multi - level modeling that would take in to account school level effects.

Acknowledgments

Grants awarded to the authors from a project of the Principality of Asturias (FC-15 - GRUPIN14-053), and the Severo Ochoa Program - FICYT (BP: 11-067) supported this work. We thank Steve Loew for his kind assistance on the preparation of this manuscript.

References

Abdullah, N., Zakaria, E. & Halim, L. (2012). The Effect of a Thinking Strategy Approach through Visual Representation on Achievement and Conceptual Understanding in Solving Mathematical Word Problems. Asian Social Science , 8 (16), 16-340. doi:10.5539/ ass.v8n16p30 [ Links ]

Ahmed, W., Minnaert, A., Kuyper, H. & Van der Werf, G. (2012). Reciprocal relationships between math self - concept and math anxiety. Learning and individual differences , 22 (3), 385-389. doi: 10.1016/ j.lindif.2011.12.004 [ Links ]

Azevedo R. & Cromley, J.G. (2004). Does training of self - regulated learning facilitate student's learning with hypermedia? Journal of Educational Psychology , 96 (3), 523-535. doi: 10.1037/ 0022-0663.96.3.523 [ Links ]

Bransford, J. D. & Stein, B. S. (1993). The ideal problem solver: A guide for improving thinking, learning and creativity 2 New York, United States: W. H. Freeman. [ Links ]

Butler D. L. & Cartier S. C. (2005). Multiple complementary methods for understanding self - regulated learning as situated in context Accepted for presentation at the April 2005 Annual Meetings of the American Educational Research Association, Montreal, QC. [ Links ]

Cleary, T. J. & Chen, P. P. (2009). Self - regulation, motivation, and math achievement in middle school: Variations across grade level and math context. Journal of School Psychology , 47 (5), 291-314. doi: 10.1016/ j.jsp.2009.04.002 [ Links ]

Csíkos, C., Szitányi, J. & Keleme, R. (2012). The effects of using drawings in developing young children's mathematical word problem solving: A design experiment with third - grade Hungarian students. Educational Studies in Mathematics , 81 (1), 47-65. doi: 10.1007/ s10649-011-9360-z [ Links ]

Cueli, M., García, T. & González - Castro, P. (2013). Autorregulación y rendimiento académico en Matemáticas. ENT#091;Self - regulatio and academic achievement in MathematicsENT#093;. Aula Abierta , 41 (1), 39-48. [ Links ]

Dettmers, S., Trautwein, U., Lüdtke, O., Goetz, T., Frenzel, A. & Pekrun, R. (2011). Students' emotions during homework in mathematics: Testing a theoretical model of antecedents and achievement outcomes. Contemporary Educational Psychology , 36 (1), 25-35. doi: 10.1016/ j.lindif.2011.04.006 [ Links ]

Díez - Palomar, J., Menéndez, J. M. & Civil, M. (2011). Learning mathematics with adult learners: drawing from parents' perspective. Revista Latinoamericana de Investigación en Matemática Educativa , 14 (1), 71-94. [ Links ]

Fidalgo, R., Torrance, M., Robledo, P. y García, J. N. (2009). Dos enfoques metacognitivos de intervención: auto - conocimiento del producto textual frente a auto - regulación del proceso de escritura. Revista de Psicología INFAD, International Journal of Developmental and Educational Psychology , 1 (1), 303-312. [ Links ]

Gálvez, G., Cosmelli, D., Cubillos, L., Leger, A. M., Mena, A., Tanter, E.,... y Soto - Andrade, J. (2011). Estrategias cognitivas para el cálculo mental. Revista Latinoamericana de Investigación en Matemática Educativa , 14 (1), 9-40. [ Links ]

García, T. y González - Pienda, J. A. (2012). Evaluación del proceso de Aprendizaje Autorregulado en el área de las Matemáticas mediante pizarras digitales. En J. Dulac y C. Alconada (Coord.), III Congreso Pizarra Digital. Publicación de comunicaciones (pp.105-117). Madrid, España: Pluma y Arroba. [ Links ]

García, J. N., Rodríguez, C., Pacheco, D. & Diez, C. (2009). Influence of cognitive effort, sustained attention, working memory and ADHD symptoms in the process and product of written composition. An experimental study. Estudios de Psicología , 30 (1), 31-50. doi: 10.1174/ 021093909787536326 [ Links ]

Geary, D. C. (2004). Mathematics and learning disabilities. Journal of Learning Disabilities , 37 (1), 4-15. doi: 10.1177/ 00222194040370010201 [ Links ]

Greene, J. A. & Azevedo, R. (2010). The measurement of Learners' Self - Regulated Cognitive and Metacognitive Processes While Using Computer - Based Learning Environments. Educational Psychologist , 45 (4), 203-209. doi: 10.1080/ 00461520.2010.515935 [ Links ]

Hoffman, B. (2010). "I think I can, but I'm afraid to try": The role of self - efficacy beliefs and mathematics anxiety in mathematics problem solving efficiency. Learning and individual differences , 20 (3), 276-283. doi: 10.1016/ j.lindif.2010.02.001 [ Links ]

Ifenthaler, D. (2008). Practical solutions for the diagnosis of progressing mental models. In D. Ifenthaler, P. Pirnay - Dummer & J. M. Spector (Eds.), Understanding models for learning and instruction. Essays in honor of Norbert M. Seel (pp. 43-61). New York, United States: Springer. [ Links ]

Ifenthaler, D., Masduki, I. & Seel, N. M. (2011). The mystery of cognitive structure and how we can detect it. Tracking the development of cognitive structures over time. Instructional Science , 39 (1), 41-61. doi: 10.1007/ s11251-009-9097-6 [ Links ]

Ifenthaler, D. (2012). Determining the effectiveness of prompts for self - regulated learning in problem - solving scenarios. Educational Technology & Society , 15 (1), 38-52. [ Links ]

Jarero, M., Aparicio, E. & Sosa, L. (2013). Pruebas escritas como estrategia de evaluación de aprendizajes matemáticos. Un estudio de caso a nivel superior. Revista Latinoamericana de Investigación en Matemática Educativa , 16 (2), 213-243. doi: 10.12802/ relime.13.1623 [ Links ]

Kajamies, A., Vauras, M. & Kinnunen, R. (2010). Instructing low - achievers in mathematical word problem solving. Scandinavian Journal of Educational Research , 54 (4), 335-355. doi: 10.1080/ 00313831.2010.493341 [ Links ]

Kramarski, B. & Gutman, M. (2006). How can self - regulated learning be supported in mathematical E-learning environments? Journal of Computer Assisted Learning , 22 (1), 24-33. doi: 10.1111/ j.1365-2729.2006.00157.x [ Links ]

Lazakidou, G. & Retalis, S. (2010). Using computer supported collaborative learning strategies for helping students acquire self - regulated problem - solving skills in mathematics. Computers & Education 54 (1), 3-13. doi: 10.1016/ j.compedu.2009.02.020 [ Links ]

Montague, M. (2008). Self - Regulation and Mathematics Instruction. Learning Disabilities Research & Practice , 22 (1), 75-83. doi: 10.1111/ j.1540-5826.2007.00232.x [ Links ]

Montague, M., Enders, G. & Dietz, S. (2011). Effects of cognitive strategy instruction on math problem solving of middle school students with learning disabilities. Learning Disability Quarterly , 34 (4), 262-272. doi: 10.1177/ 0731948712463368 [ Links ]

Olive, T., Kellogg, R. T. & Piolat, A. (2002). The Triple Task technique for studying the processus of writing: why and How? In G. Rijlaarsdam (Series Ed.), Studies in Writing & T. Olive, & C. M. Levy (Vol. Eds.), Contemporary tools and techniques for studying writing (pp. 31-59). Dordrecht, Netherlands: Kluwer Academic Publishers. [ Links ]

Ostad, A. & Sorenson, P. M. (2007). Private speech and strategy-use patterns: Bidirectional comparisons of students with and without mathematical disabilities in a developmental perspective. Journal of Learning Disabilities , 40 (1), 2-14. [ Links ]

Pantziara, M., Gagatsis, A. & Elia, I. (2009). Using diagrams as tools for the solution of non - routine mathematical problems. Educational Studies in Mathematics , 72 (1), 39-60. doi: 10.1007/ s10649-009-9181-5 [ Links ]

Pennequin, V., Sorel, O., Nanty, I. & Fontaine, R. (2010). Metacognition and low achievement in mathematics: The effect of training in the use of metacognitive skills to solve mathematical word problems. Thinking and Reasoning , 16 (3), 198-220. doi: 10.1080/ 13546783.2010.509052 [ Links ]

Pereis, F., Dignath, C. & Schmitz, B. (2009). Is it possible to improve mathematical achievement by means of self - regulation strategies? Evaluation of an intervention in regular math clases. European Journal of Psychology of Education , 24 (1), 17-29. doi: 10.1007/ BF03173472 [ Links ]

Piolat, A., Kellogg, R. T. & Farioli, F. (2001). The Triple Task technique for studying writing processes: on which task is attention focused? Current Psychology Letters. Brain, Behavior and Cognition , 4 , 67-83. [ Links ]

Piolat, A. & Olive, T. (2000). Comment étudier le coût et le déroulement de la rédaction de textes? La méthode de la triple - tâche: Un bilan méthodologique. L'Anneé Psychologique , 100 (3), 465-502. [ Links ]

Piolat, A., Olive, T. & Kellogg, R. T. (2005). Cognitive effort during note taking. Applied Cognititive Psychology , 19 (3), 291-312. doi: 10.1002/ acp.1086 [ Links ]

Piolat, A., Barbier, M. L. & Roussey, J. Y. (2008). Fluency and cognitive effort during first and second - language note taking and writing by undergraduate students. European Psychologist , 13 (2), 114-125. doi: 10.1027/ 1016-9040.13.2.114 [ Links ]

Rosário, P., Mourão, R., Núñez, J. C., González - Pienda, J. A. & Solano, P. (2008). Storytelling as a promoter of Self - Regulated Learning (SRL) throughoutschooling. In A. Valle, J. C. Núñez, R. G. Cabanach, J. A. González - Pienda, & S. Rodríguez (Eds.), Handbook of instructional resources and their applications in the classroom (pp. 107-122). New York, United States: Nova Science. [ Links ]

Rosenzweig. C., Krawec, J. & Montague, M. (2011). Metacognitive strategy use of eighth - grade students with and without learning disabilities during mathematical problem solving: A Think - Aloud analysis. Journal of Learning Disabilities , 44 (6), 508-520. doi: 10.1177/ 0022219410378445 [ Links ]

Schmitz, B. & Perel, P. (2011). Self - monitoring of self - regulation during math homework behaviour using standardized diaries. Metacognition and Learning , 6 , 255-273. doi: 10.1007/ s11409-011-9076-6 [ Links ]

Shea, P. & Bidjerano, T. (2010). Learning presence: Towards a theory of self - efficacy, self - regulation, and the development of a communities of inquiry in online and blended learning environments. Computer & Education , 55 (4), 1721-1731. doi:10.1016/ j.compedu. 2010.07.017 [ Links ]

Stylianou, D. A. (2011). An examination of middle school students' representation practices in mathematical problem solving through the lens of expert work: towards an organizing scheme. Educational Stududies in Mathematics , 76 (3), 265-280. doi: 10.1007/ s10649-010-9273-2 [ Links ]

Throndsen, I. (2011). Self - regulated learning of basic arithmetic skills: A longitudinal study. British Journal of Educational Psychology , 81 , 558-578. doi: 10.1348/ 2044-8279.002008 [ Links ]

Torrance, M. & Galbraith, D. (2006). The rocessing demands of writing. In MacArthur, C., Graham, S. & Fitzgerald, J. (Eds.), Handbook of Writing Research (pp. 67-80). New York, United States: The Guilford Press. [ Links ]

Veenman M. V. J. (2005). The assessment of metacogni-tive skills: what can be learned from multi-method designs? In B. Moschner & C. Artelt (Eds.), Lernstrategien und Metakognition: Implikationen fur Forschung und Praxis (pp. 75-97). Berlin, Germany: Waxmann. [ Links ]

Veenman, M. V. J. (2011). Learning to self-monitor and self - regulate. In R. Mayer & P. Alexander (Eds.), Handbook of research on learning and instruction (pp. 197-218). New York, United States: Routledge. [ Links ]

Verschaffel, L., Greer, B. & De Corte, E. (2000). Making sense of word problems Lisse, Netherlands: Swets & Zeitlinger. [ Links ]

Voyer, D. (2011). Performance in mathematical problem solving as a function of comprehencion and aritmetic skills. Journal of Science and Mathematics Education , 9 (5), 1073-1092. doi: 10.1007/ s10763-010-9239-y [ Links ]

Whimbey, A. & Lochhead, J. (1999). Problem - solving and comprehension Hillsdale, NJ: Erlbaum. [ Links ]

Williams, J. R. (2008). Revising the Declaration of Helsinki. World Medical Journal , 54 , 120-125. Obtained from http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2681053/Links ]

Zimmerman, B. J. (2000). Attaining self - regulation. A social cognitive perspective. In M. Boekaerts, P. R. Pintrich, & M. Zeidner (Eds.), Handbook of self - regulation (pp. 13-39). San Diego, United States: Academic Press. [ Links ]

Zimmerman, B. J. & Schunk, D. H. (2008). Motivation: An essential dimension of self - regulated learning. In D. H. Schunk & B. J. Zimmerman (Eds.), Motivation and self - regulated learning. Theory, research, and applications (pp. 1-30). New York, United States: Routledge . [ Links ]

Received: October 10, 2013; Accepted: March 03, 2015

Creative Commons License This is an open-access article distributed under the terms of the Creative Commons Attribution License