Introduction

Globally, the selection of physicians for medical residency courses involves many issues: applicants outnumber positions, applicants are applying to increasing numbers of programs, and the costs of the selection process are a burden to society.¹

In Mexico, general practitioners have very few opportunities for medical practice. This is because there is a strong preference for people with specializations and a majority of job positions are filled by people who have completed a medical residency. Thus, the great majority of medical school graduates hope to be trained as specialists.^2,3,4

Medical specialization refers to the training that physicians receive after obtaining a medical degree. A specialization expands and deepens their knowledge and skills in a specific medical field. This training helps them develop sound clinical judgment and performance skills that can be used to solve complex medical problems with high professional competence.^4,5

Other countries have variations in admission policies to residency training: United States and Canada use a matching program which purpose is to allow applicants and specialty training programs to know each other and make their selections. In Australia, the United Kingdom, Germany and the Netherlands admission is through an open selection. There are further issues, for example, in Australia, candidates must show some clinical experience prior to application.⁵

In Mexico, to take a medical specialty training course, one must have a medical degree and pass the Examen Nacional de Aspirantes a Residencias Medicas (ENARM), a national examination for medical residency admission. Subsequently, candidates are admitted to the specialty training course based on their score and chosen specialty.⁶ This test is conducted by the Interinstitutional Commission for Human Resources Formation in Health (Comisión Interinstitucional para la Formación de Recursos Humanos para la Salud, CIFRHS). The number of selected candidates corresponds to the number of positions offered for each discipline in the hospitals that provide training; i.e., it is a norm-referenced test. Annually, a little more than 35 000 candidates take the test for about 7 000 resident positions. Thus, a large number of candidates are excluded; for example, in 2016, only 22% of applicants were selected for residency training.⁷ The majority of candidates, therefore, remain as general practitioners. Furthermore, some reports estimate that around 14% of all physicians are unemployed.⁸

The ENARM is taken for any direct entry specialty (not requiring previous specialty training) applicant.⁹ It is structured in a different version for every day of application with a format of multiple choice questions.¹⁰ ENARM explores examinees’ knowledge of general medical practice in different areas. It is a high-stakes summative test with determinant results. It defines, in a way, the professional future of thousands of physicians. In this regard, although multiple choice questions have limitations, it has been demonstrated that well-structured multiple-choice questions can explore not only the recall of facts, but higher levels in the cognitive taxonomy of Bloom (e.g. application or synthesis).^11,12

An analysis of this test helps identify some aspects of academic performance, as indicated by the number of correct answers, and assesses the amount of knowledge that the candidates possess in order to get into a medical residency course. It also identifies the strengths and weaknesses of the candidates.⁵

Academic performance does not depend exclusively on personal effort but also on variables such as academic background, learning strategies for the test, as well as socioeconomic, psychological, and vocational factors.^13,14,15 Type of medical school has been investigated in other contexts: at least one research paper has showed a greater possibility of matching into a family medicine residency if the candidate was from a public institution.¹⁶

In this context, research papers identifying academic variables that influence performance in ENARM are scarce. This kind of research allows feedback to candidates and gathers relevant information for design and implementation of education related interventions that strengthen educational plans and enhance quality. In Mexico, quality of medical care is usually associated with accreditation status considering that medical education accreditation processes can encourage institutional self-review and improvement, and in consequence ensure that medical students receive high-quality education experiences based on established standards.¹⁷ An accredited medical school is one which complies with the requirements of the Mexican Council for the Accreditation of Medical Education (Consejo Mexicano para la Acreditación de la Educación Médica, COMAEM); that is getting a minimum of 80% on the assessment instrument. Another probable determinant of quality would be the geographic region where the medical school is located or to where the examinee belongs; this, related to the available resources: economic and academical. A recently published paper¹⁸ demonstrated differences in performance and selection rates related to geographical and socioeconomical factors; the authors based their research on historical data (annual public report of the ENARM for 17 years from 2001 to 2017) and concluded a better ENARM performance in the northwestern region and for private schools. In our opinion, the method of socioeconomic segmentation according to levels and their geographic allocation process, although sophisticated and made through specialized software, is not as easy to understand for the general medical community. A useful geographic segmentation process is that proposed by the National Association of Universities and Higher Education Institutions (Asociación Nacional de Universidades e Instituciones de Educación Superior, ANUIES), a non-government organization that comprises 191 of the most renowned public and private institutions in our country. These institutions are spread across 32 Mexican states and together represent almost 60% of higher education students and perform 90% of scientific research.¹⁹ In agreement with the aforementioned organization, regions of Mexico are Northwest, Northeast, Center-West, Metropolitan area (Mexico City and surroundings), Center-South and South-Southeast; they could constitute a geographic predictor of performance. Table I shows the Mexican states grouped by these criteria.

Thus, we aimed to determine any difference in ENARM performance and selection rate between candidates from accredited medical schools and non-accredited medical schools and between public (government sponsored) and private schools. In the same way our study tried to demonstrate these differences between candidates from different regions of the country.

Materials and methods

Between July 2019 and March 2020, a retrospective analysis was executed.

Results

Our study analyzed the data of 153 654 examinees of the ENARM test from 2014 to 2018. Of these, 51.2% were women and 49.7% were men.

Figure 1 shows a frequencies polygon with results obtained in ENARM. Data allows us to see a performance mean for the whole group of 62.3 (standard deviation 8.9) ranking from 1.11 to 91.11; median (50th percentile) was located at 62.4 and mode at 60.9 indicating an asymmetrical and unimodal distribution. The great majority of candidates performed in the range between 60 and 70, and 50.8% of the whole sample performed above the mean; the 75th percentile was located above 68.9. For a proper interpretation of the results, we must underscore that candidates know the exam content beforehand and usually spend a considerable amount of time in preparation for this high stakes test.⁹

Figure 1: Performance in ENARM: frequencies polygon. Mexico City, Mexico, 2020 

Geographic region and ENARM performance

In table III, under “Geographic Region,” we can see how different geographic regions showed different and statistically significant performance means. Nevertheless, some cross comparisons, e.g., Metropolitan compared to Center-South and Center-West, or North-west compared to North-east, demonstrated very similar means. The geographic region with the lowest performance was South-Southeast.

Selection rate showed some differences: North-East and North-West reached 31%, contrasting with South-Southeast obtaining 20.7%; p< 0.001 after Chi-squared test (figure 2).

ENARM: Examen Nacional de Aspirantes a Residencias Médicas

Selection rate for Mexico City and Metropolitan Area was 25.73

Figure 2: Geographic regions and ENARM selection rate, Mexico, 2020  

Table III: ENARM performance for different geographic regions in the period 2014-2018. Number of candidates per region and performance means are shown. Mexico City, Mexico 2020 

Geographic region	n	ENARM
		x̄ ± SD
Metropolitan*	37 885	62.6 ± 8.3
Center-South	22 701	61.2 ± 8.8
Center-West*	32 656	62.8 ± 9.3
North-West‡	14 164	63.3 ± 8.9
North-East‡	22 248	63.8 ± 9.0
South-Southwest	22 854	58.9 ± 8.2
Total	152 508	62.3 ± 8.9

Analysis of variance p< 0.001

^*,‡Compared groups with no difference after post-Hoc Bonferroni test

Data lost: 1 146

ENARM: Examen Nacional de Aspirantes a Residencias Médicas

Discussion

Our study aimed to identify the medical school characteristics associated with performance in a selection test (ENARM) among general physicians who want to be trained in a medical specialty.

Our results show that heterogeneous ENARM performance can be related to certain variables: accreditation status of the medical school, type of medical school (public or private), and geographic region.

We consider these findings as a form of validation of the ENARM and provide evidence of the test in relation to other variables.^22,23 This last assumption must be emphasized since other researchers have pointed out the lack of validity evidence regarding the ENARM.²⁴ Our study provides validity evidence considering terms of sample size and statistical analysis that proves our hypotheses. We expected statistically significant p-values because of the population size; nevertheless although effect sizes can be considered small, important information can be obtained in relation to the studied variables. For example, in a real scenario, differences in selection rates around 4% may signify hundreds of candidates in or out of the residency system.

Upon examining accreditation status in relation to ENARM performance, our findings are similar to those encountered by Vázquez-Martínez and colleagues²⁵ in the ENARM test of 2016. They found a greater selection rate (without statistical significance) for accredited medical schools; however, they did not examine performance (average number of correct answers). In our analysis, examinees belonging to accredited medical schools performed better on ENARM and had a higher selection rate compared to non-accredited medical schools (p<0.001); nevertheless, we must acknowledge that these differences (62.5 vs. 61.4 in performance) are marginal and thus must be interpreted properly. Difference in selection rate was somewhat greater (26.2 vs. 22.9%); this specific aspect can be of interest for stakeholders.

In relation to type of medical school (public vs. private), even though differences were not great from a numerical standpoint, examinees belonging to public schools performed better, which augments their probability of being selected for medical residency (p<0.001). Although differences were statistically significant, effect sizes were small. We could say that public and private medical schools are at least equivalent in terms of performance and selection rate in ENARM with a tendency of public schools to get better results.

This finding can have several meanings: a) supportive or corrective measures related to assessment and accreditation processes can improve education quality in medical schools; b) the inherent struggle for students to transcend their possible financial constraints by becoming better prepared professionals as a way to accomplish social mobility is enhanced;²⁶ c) although the number of private medical schools is almost double than that of public ones, because of the probably more lax requirements for the establishment of private institutions and the marked growth in their absolute numbers (e.g. there are over 3 000 medical schools in the world, twice as many as there were 20 years ago),²⁷ structural and academical limitations may predominate in this group. A similar phenomenon can be seen in other Latin-American countries.^28,29,30 Our results contrast with those of Hernández-Gálvez;¹⁸ a possible explanation for this is that by having access to raw data we could determine in a more precise way the distinct characteristics of applicants, their performance and selection rates; i.e. means and proportions.

Geographical regions also impact ENARM performance; in our study, based in raw performance data, Northeast and Northwest regions show a better test performance, Center-West, Metropolitan, and Center-South areas are located midway, while South-Southeast areas are placed at the bottom. This reflects the unequal rate of socioeconomic development among regions of the country as well as a heterogeneous distribution of monetary and educational resources (e. g. more resources for Metropolitan areas); the number of applicants by region can give a hint, and an analysis of this situation may enable a thoughtful consideration of inequalities for students.

In our interpretation, general physicians who take the ENARM test are exposed to varied and heterogeneous medical school educational programs, as is the case in other parts of the world;³¹ this may be one of the main determinants for varied ENARM performance. This heterogeneity has been demonstrated, and it is rather obvious; some researchers have aimed at identifying medical schools as better or worse, or assigning them places in a competition.³² The quid of the matter is to know why this difference happens.

In a constructive way, ENARM has recently been more and more criticized in relation to issues of validity and equity;^23,33 for example, internal structure of the test has not been assessed publicly and the use of established psychometric theories, such as classical test theory or item response theory for creation and item analysis in this test is unknown;³³ we believe that our study can shed a light in the issue of validity.

Finally, when using ENARM to make inferences about medical school performances, we must not forget that this is a norm-referenced test; the number of “winners” depends directly on the number of available spots. Moreover, as we do ignore difficulty and discrimination indices at the moment, classifying candidates as underprepared based only on selection rates or means of performance is at least risky. Conversely, although ENARM is not intended to certify medical knowledge or academic quality of medical schools, in our opinion its inherent value lies in being the only test taken from graduates of every medical school in the country. A cautionary note has to be pointed out since the evidence obtained in this aspect is indirect: in terms of validity in education what we see is evidence in relation to other variables and evidence regarding consequences of testing.²² As figure 1 shows, the performance means of candidates (e.g. 75th percentile above 68.9) indicate that as a whole they are prepared to continue with training as medical residents and somehow point to the quality level of medical education in our country as being adequate; once more, this estimation is indirect.

Previous studies have indicated that in order to align scarce healthcare resources we have to make efforts to know more about “how medical students are trained, who are the students who graduate, and the economic, political, and social factors that foster or hinder the establishment and operation of medical schools”.^34,35 Thus, analysis of high-stakes tests like ENARM are valuable and strongly related to these topics.