THE CONSTRUCTION OF CATEGORIES: OBSERVING EVALUATION PRACTICES

To investigate the workings of the Mexican evaluation system, I first observed the grading of the two main evaluation tools, the portfolio and the standardized test. This process involves teacher-evaluators-teachers who have already been evaluated with good results and have been certified to evaluate their colleagues-as well as tools such as an evaluation grid and statistical discrimination techniques operated by the personnel of Ceneval. This is a confidential process, and I was only granted access to observe it after signing a confidentiality agreement with the institution, which prevents me from disclosing the specific content of the evaluation grid and other evaluation methods. However, beyond the content of these tools, my primary interest was in understanding how the various actors involved used them and how their evaluation practices contributed to the construction of the categories and groups that emerge from the Teacher’s Professional Service. What should one observe in a process where hundreds of teacher-evaluators largely work on their computers in silence? How can one identify social practices and representations in an almost automatized process? How to observe this kind of process without disturbing it, and thus potentially affecting the work chances of teachers? To address these challenges, I adopted two strategies.

The first strategy involved focusing on moments of “trial,” where the rules, criteria, and even the legitimacy of the evaluation process became explicit and open to discussion. One such instance of “trial” is the grading of the standardized test. Unlike the grading of portfolios, which includes hundreds of evaluators, this process is more intimate, lasting two days and involving only three teacher-evaluators, a coordinator, and a Ceneval statistician who intervenes intermittently. This setup facilitates the observation of evaluation practices. During a full day, the evaluators discuss the relative difficulty of each of the 120 questions on the test. To do this, they must imagine four “hypothetical candidates,” one for each performance category, and assess the probability that each would answer each question correctly. Surprisingly, this technique does not lead to many discussions, and it seems as though the evaluators shared these avatars.¹ The moment of trial occurs on the second day, at the end of the grading process, when the evaluators and the statistician determine the cut-points between each performance category. These cut-points have to be derived from the difficulty of the exam, as assessed by the evaluators the previous day, rather than from a theoretical or a priori definition of how many teachers are “insufficient” or how many correct answers are necessary to be “excellent.”

The final cut-points emerge from the collective work of evaluators and the statistician. While the statistician wants to produce a “normalized” distribution of results (one that places few teachers in the highest and lowest categories and most in the middle), evaluators seek to avoid a situation where many of their colleagues end up in the lowest category (“insufficient”), which involves potential sanctions. These positions are not contradictory. In the grading process I observed, the initial cut-points resulted in 14% of teachers classified as insufficient, 42% as sufficient, 34% as good, and 10% as excellent. This distribution was deemed too “spread out” by the statistician, and the evaluators agreed that 14% was an excessive number of “insufficients.” They thus revised the difficulty estimates they had made the previous day, which led to a second distribution curve: 8% insufficient, 46% sufficient, 35% good, and 12% excellent. This new result “saved” an important portion of the previously insufficient teachers but was even more “spread out”. The Ceneval statistician thus encouraged the evaluators to revise their expectations and increase the number of insufficient teachers, which should lower the ranks of the “sufficient” category-the highest category since the beginning. Following this advice, the evaluators adjusted their estimates again, arriving at a final, almost perfect normal distribution: 12% insufficient, 39% sufficient, 37% sufficient, and 12% excellent. The “normalization” goal of the dispositif and its embedding in different statistical and professional objectives thus became apparent. This process reflects the dual nature of discrimination both as statistical description (the normal distribution) and as social prescription (sanctioning or “saving” those deemed insufficient). However, the observation of evaluation practices alone did not allow me to fully grasp the social and professional meaning of the evaluation categories and the statistical manipulations involved in their construction. As a result, I adopted a second strategy during my observations: engaging in informal situations and conversations. During lunch, coffee breaks, or even after-work drinks, I could ask evaluators to reflect on the work they were doing. These moments also provided space for anecdotes from previous years. For instance, some evaluators recalled how some of their colleagues had been excluded from the procedure after grading all portfolios as “excellent”-a practice identified and sanctioned by the internal auditing process. They also shared their strategies to avoid audits, such as grading the different components of portfolios as “sufficient” and “good” (audits are triggered when two different evaluators grade the same components of the same portfolio as “insufficient” and “excellent”). When having a drink with the evaluators involved in the grading of the standardized test described above, I could ask them if the normalization of results was a common practice. They acknowledged this practice while expressing their discomfort, as many of the test questions are discarded to construct the distribution (those questions answered correctly or incorrectly by more than 90% of teachers, and thus deemed ineffective for discriminating their performance). After one of the evaluators raised this issue, another answered, “Yes, yet you validated the procedure when you signed at the end and took your check.” These informal discussions are thus an occasion to contextualize the observed practices, their meaning, the strategies of the actors involved, and the role they perceive themselves as playing in the evaluation process. Nevertheless, the social meaning of the categories remained elusive.

GRASPING THE MEANING OF CATEGORIES THROUGH INTERVIEWS

While the ethnographic observation of evaluation practices seems indispensable when studying policy instruments such as the Teacher’s Professional Service, I found that understanding the processes of categorization required in-depth conversations with teacher- evaluators. These interviews provided an opportunity to reconstruct and compare their representations, their grammars of evaluation, and the ideas of merit and justice on which their work was based. To avoid common issues with semi-structured interviews-such as the forgetting of the precise practices that actors realized in the past or the a posteriori illusion of coherence in practices and representations-I could use my observations to resituate my interviewees (some of whom I had observed) in specific grading situations and ask them to walk me through their practices and thought-processes.

Returning to the construction of “hypothetical candidates” proved particularly useful in understanding the discriminations involved in the evaluators’ practices. Evaluators would explain that they use various techniques to imagine these avatars, ranging from the personalization of the “hypothetical candidates,” where relatives or friends are made the representatives of a particular category, to modes of abstraction and generalization where specific professional sub-groups are invoked as “insufficient.” The insufficient category was typically the easiest for them to imagine and talk about, as was the “excellent” category, where they always placed themselves. As one evaluator put it, “You’ll say I have a great ego, but I imagined myself in level 4 [excellent].” This is not surprising, as reflecting on both the top and the bottom of the new evaluation system allowed them to construct its symbolic coherence and discriminate and distribute performances in it.

As with other forms of contemporary discrimination, evaluators did not rely on categorical differentiation between their colleagues but rather on “infracategorical discriminations” (Monk, 2023). This practice was facilitated by the fact that, aside from the written texts and the few photographs included in the portfolio, evaluators did not have access to any personal information (such as the names) of the teachers being evaluated. Moreover, the interview setting made them remember only portfolios that stood out, either for their quality or lack thereof-contrary to the observation of the continuous flow of grading, where most portfolios are “sufficient” or “good.” This led them to reconstruct the evaluated person and ascribe her identity and professional situation. For instance, instead of explicitly referring to the racial or indigenous identity of an “insufficient” teacher, evaluators would say that the text was “as if written by a child” (as in the excerpt at the beginning of this paper) or that it was a teacher “from la sierra” (the mountains). Another common form of discrimination involving personal characteristics was the association between old age and “insufficiency.” One evaluator recalls thinking about “that old teacher that knows a lot, that reads a lot, why doesn’t she make an effort? […] But the fear, the tension, are stronger than her…” In other cases, reflecting on who qualifies as an “insufficient” teacher allowed evaluators to activate intra- professional cleavages. For example, one evaluator thought about the “insufficient” teacher as “teacher Óscar”, an actual teacher from her school: “He was in an administrative position and when he came back, he was lost.” I encountered similar remarks in other interviews, where evaluators referred to the “lazy teacher from the school” or “those that have been teaching for more than ten years and don’t even have their degree.” These cues associated insufficiency with union militancy, echoing the scandals of the (illegal) selling or the (legal) inheriting of teacher positions or the union’s political patronage, including placing phantom employees in educational administrations. Insufficiency was also a way of discriminating against teachers with alternative educational backgrounds, such as those who graduated from universities rather than from the normal schools of teacher training, that many consider to “lack vocation,” to teach “while they look for another job,” and to have “insufficient interest in didactics and pedagogy.” These snapshots bear witness to how interviews allowed me to access the categorization processes evaluators follow to assess and classify their colleagues. In this process, pre-existing social and professional identities, categories, and cleavages are activated. I could not have grasped this mechanism through observation alone, as reflecting and making explicit the thought-processes underlying the categorization of individuals requires an intimate and time-consuming setting that only the interview situation-conducted in the evaluators’ homes or quiet coffeeshops-can provide. Interviews thus reinforce and complement the materials produced via observation, allowing to better objectify discriminations without overgeneralizing from anecdotal evidence. These materials were especially crucial given that I was granted access to the grading process for only a week (40 hours) and could not be certain about the generalizability of my observations.

FROM CATEGORIES TO GROUPS: COMBINING QUALITATIVE AND QUANTITATIVE EVIDENCE

The problem of generalization is a recurrent challenge in qualitative studies. One potential solution is the combination of different sources (such as observations and interviews) to triangulate information and minimize biases. In this third and final section, I extend this combination of sources to include quantitative materials, specifically descriptive statistics. These are particularly important when studying the discriminations produced by policy instruments such as Mexico’s Teacher’s Professional Service. As is often the case, its results are public, which allowed me to test whether the discriminations identified during the categorization process translate into group differentiation and hierarchization within the profession. I did not excessively extrapolate from the reading of descriptive statistics, and I adopted a reflexive approach to these materials. Indeed, these statistics do not always rely on the categorical and infracategorical differences that would allow me to test the qualitative evidence gathered via observations and interviews. While some variables, such as age and gender, are well reflected, others, such as degree or unionization, are not (for personal confidentiality reasons), which led me to rely on complementary statistical sources. This cross-fertilization of qualitative and quantitative materials allowed me to infer correlations with confidence but not causality-a point developed below.

To test the correlation between age and the categorization as insufficient, I relied on the public presentation of the first performance evaluation results in the Mexican Senate (^{SEP, 2016}). This analysis showed that among the 19,679 teachers categorized as “insufficient” in 2015, the oldest segment of the profession was overrepresented. On average, 15% of male teachers had insufficient results, a rate that doubled for the 55-60 age group (31%) and that went up to 35% for those over 60 years old. While women generally achieved better results (11% categorized as insufficient), the proportions remained similar. Among women aged 55- 60, 22% were categorized as insufficient, rising to 28% for those over 60. In light of my previous qualitative analysis, I could understand the processes that led to this overrepresentation of old teachers as insufficient. Not only did they probably experience problems with an online evaluation that reflected on their portfolios, but they were also often considered less “effortful” and more “fearful” by evaluators.

By crossing evaluation results with measures of poverty (^{CONEVAL, 2020}), I could indirectly interrogate the correlation between race, geographical location, and “insufficiency.” Or, in other words, verify if teachers from “la sierra” were discriminated against by the evaluation standards. Indeed, teachers categorized as insufficient primarily come from the poorest regions of the country. Five out of the six states whose rate of insufficient teachers at least doubles the national average (12.43%) also present higher poverty rates than the national average (41.9%). These states are Zacatecas (29% insufficient teachers), Sonora (31%), Oaxaca (31%), Guerrero (31%), Michoacan (35%), and Chiapas (44%).² Except for Sonora, these regions have high poverty rates, with up to 66% of the population in Guerrero and 76% in Chiapas living in poverty. These states also share an important characteristic: they are all rural, low-density population areas, which probably hinders their teachers’ opportunities for professional socialization and preparation of the evaluations. Their situation contrasts with that of the states achieving the best evaluation results (half or less the national average of insufficient teachers). Despite their many differences, all these states present poverty rates below the national average, ranging from 40.8% (Yucatan) to 18.1% (Baja California Sur). Although less straightforward, these results allowed me to objectify the intuition stemming from my interviews and observations: many insufficient teachers come from poor, rural, and sometimes indigenous areas.

Intra-professional cleavages were harder to test. As with the race and ethnicity of the teachers and their students, variables such as unionization or academic background were not included in the individual (and anonymous) results made public by the administration. I thus used two sources to interrogate the new forms of intra-professional competition: results in the new competitive examinations to access the teaching profession (^{Cordero Arroyo & Jimenez Moreno, 2018}) and enrollment data in normal schools for teacher training (normales) (^{INEE, 2017}). If students from normales achieved better results in three of the four competitive exams held between 2014 and 2018, the proportion reversed in 2016- 2017, when the percentage of university graduates eligible for a teaching position reached 68%, compared to 50% for students from normales. Furthermore, enrollment in these schools has plummeted since the introduction of the Teacher’s Professional Service. Private normales have lost 45% of their students, becoming almost irrelevant. The much more historically important public normales have also lost around 21% of their students. These points to a recomposing of professional values and valorization criteria, with university graduates considered as having “less vocation” by normalistas, and probably less tied to the teachers’ union, becoming increasingly important in the ranks of the profession.

This quantitative evidence allows me to reinforce the objectivation of the modes of discrimination produced and reproduced by the Teacher’s Professional Service. To be sure, I do not identify causal mechanisms but rather point at strong correlations to confirm the generalization of the qualitative evidence that represented the heart of my doctoral research. If multivariate analysis could be conducted on this data, my goal was not to identify the exact effect of each variable but rather to show the multiple meanings and the categorization processes that lay behind the “insufficient” result. Echoing recent results in the sociology of instruments (^{Krippner & Hirschman, 2022}), I thus show that it is not a single group or category that is considered insufficient, but rather that different individuals, reduced to a series of attributes, can be categorized as such for various reasons (but with the same implications). This, in turn, hinders the potential identification of these individuals with the category and, thus, their potential political mobilization.

DISCUSSION

In this paper, I have proposed a methodological framework to study the production of discrimination by policy instruments. The Teacher’s Professional Service raises specific questions regarding the conceptualization of categorization, hierarchization, and discrimination processes, as well as about data availability and limitations. I adopted a strategy combining observations, in-depth interviews, and descriptive statistics to study this instrument. These materials complement and extend each other. Focusing on “moments of trial” during the observation of practices of evaluation allowed me to identify the rationale of the instrument and the role that various actors and tools play during the grading process. These elements enabled me, during a second fieldwork, to refer to specific moments and practices in my interviews and to objectify the representations and discriminations (both categorical and infracategorical) used by evaluators while assessing their colleagues. Finally, I scaled up the analysis through descriptive statistics to test if these group discriminations could be observed at the aggregate level. In doing so, I could escape the anxiety often felt by qualitative scholars when attempting to generalize their findings, and I was able to confirm both the existence of evaluation-induced discriminations and the social processes leading to their production.

Of course, this proposal has limitations and should be further developed and adapted to address the specificities of other policy instruments. One important limitation is the absence of causal identification in my quantitative analysis, which could be explored in further studies. Another limitation is that I did not include the “phenomenological dimension” (^{Fourcade & Healy, 2017}) of evaluation in my inquiry into the production of discrimination by the Teacher’s Professional Service. What does it mean to be insufficient or excellent? How does it affect teacher subjectivities and daily work? Only more systematic ethnographic work in schools could address these questions. Beyond the materials presented in this paper, like most studies of instruments, I also analyzed official documentation in my dissertation (e.g., technical documents, guidelines for evaluators, and examples of portfolios and evaluation grids), which may deserve a methodological reflection regarding its status as archival material and its role in the production of discriminations.