Study design and data acquisition

This study was cross-sectional and used historical data that did not require approval by an Institutional Review Board. We based our analyses in the annual public report of the ENARM during 8 years from 2012 to 2019. The Interinstitutional Commission issued the reports for Human Resources Training for Health (CIFRHS, Comisión Interinstitucional para la Formación de Recursos Humanos para la Salud) a department of the Undersecretariat of Innovation and Quality of the Mexican Ministry of Health¹³. The reports contain quantitative information of the academic performance at each medical specialty from graduate students who took the ENARM; the reports are freely available as PDF files at the CIFRHS website¹⁴.

Logistics of ENARM and assessed variables

Five test forms are created each year, each comprising 450 multiple-choice single-best answer items; no item is used in more than one test form. All test forms contain the same number of items per area of knowledge (specialty/subspecialty), with an approximate item distribution of 37.5% internal medicine, 25% pediatrics, 22% gynecology-obstetrics, and 15% surgery. Applicants for each specialty are ranked from highest to the lowest according to their total ENARM score. Ranked applicants receive a "pass" certificate until the quota is met according to that specialty's available positions⁸.

For each year (2012-2019), we recorded the minimum and maximum scores (calculated by dividing the absolute number of correct answers by the total number of items) clustered by nationality (Mexican or IMG) and chosen specialty (five direct-entry specialties) that coincidentally appear in the annual CIFRHS report.

Statistical analysis and data visualization techniques

Our analysis was performed at two steps, first we compare the minimum and maximum scores among surgical specialties, and second we compared the minimum and maximum scores between Mexican and IMG.

In the first part of our analysis, we compare the minimum (MinSco) and maximum (MaxSco) scores of the five direct-entry surgical specialties evaluated by the ENARM (general surgery, gynecology and obstetrics, ophthalmology, otorhinolaryngology, and traumatology and orthopedics); the Kolmogorov-Smirnoff and Shapiro-Wilk tests showed a non-significant p-value for each specialty, which indicated a normal distribution of data in both variables (MinSco and MaxSco). Then, we performed a one-way ANOVA to reveal the differences in the scores achieved by each specialty; variables were tested for homogeneity of variance and post hoc tests used the least significant difference method. To test the assumption that MinSco and MaxSco increase every year, we assessed if there was a significant linear trend for the scores to increase across the specialties. For this assessment, we use the polynomial option (in the ANOVA menu of SPSS), in its contrast box with chose the Degree:Linear (defaul) option. Detailed descriptions of the ANOVA test in clinical settings have been previously published by our group¹⁵,¹⁶. Descriptive statistics were used for each variable, 95% confidence intervals (C.I.)¹⁷. The effect size assessment (proportion of the variance in the dependent variable that can be explained by the independent variable) of each result was obtained using the Partial Eta Squared (η²)¹⁸, where 0.01-0.06 = small effect, 0.06-0.14 = moderate effect, and > 0.14 = large effect. To visualize the results, we use graph lines showing the evolution of MinSco and MaxSco every year for each specialty, we also drew bar graphs with the global means indicating those specialties whose mean were above or below a global mean for all specialties.

For the second part of our analysis, we looked for significant differences between Mexican and IMG in their scores by analyzing independently each specialty, the comparison of means was done using the independent T-test. To analyze the trend of the MinSco and MaxSco every year for each specialty, the Pearson's correlation coefficient helped us to reveal direction trends: positive for increasing scores (↑) with every year (2012-2019) or negative for decreasing scores (↓). We completed the analysis using the forecasting method to calculate a 5-years trend in the MinSco and MaxSco of each specialty and detected if there was a crossing point between Mexican and IMG for each medical specialty. Similar to medical specialties, we used our previously calculated global means for the MinSco and MaxSco to group the Mexican and IMG of the specialties that lied above or below the mean for specialty.

Score comparisons and trend analyses were performed using the IBM^® SPSS^® Statistics software (version 25.0.0.1 IBM Corporation; Armonk, NY, USA). Data visualization of score trends and forecasting analysis used ©Tableau software (version 2019.1.3, Seattle, Washington, USA). Statistical significance considered p < 0.05 (two-tailed).

Scores included in the analysis

For each score (MinSco and MaxSco) we evaluated 80 measures, 16 for each specialty (eight scores for Mexicans and eight for IMG for the years 2012-2019), with a total 160 measures included in the analyses.

Grouping of specialties above or below a global mean

We calculated a MinSco global mean of 72.572. Specialties above this mean were ophthalmology, otorhinolaryngology, and general surgery. Specialties below the mean corresponded to traumatology and orthopedics and gynecology and obstetrics.

The global mean for the MaxSco was 81.559; only two specialties were above this mark: ophthalmology and general surgery. The other three specialties below the global mean were traumatology and orthopedics, gynecology and obstetrics, and otorhinolaryngology. Figure 1A and B showed the scores above or below the global mean for surgical specialties.

Figure 1 A-B: scores above or below the global mean for surgical specialties. C-D: mean comparison of surgical specialties showing the trend by year. E-F: global trend of the MinSco and MaxSco during 8 years (2012 to 2019). 

Comparison of minimum and maximum scores achieved by surgical specialties

The one-way ANOVA depicted a significant difference among the minimum scores achieved by the five surgical specialties; F (4, 78) = 24.586, p ≤ 0.001; the η² = 0.570 indicated a great effect size. Post hoc tests showed that significant differences were observed between each one of the surgical specialties (Bonferroni adjusted p = 0.01). Only two pairs of specialty-comparisons were non-significant: gynecology and obstetrics versus traumatology and orthopedics (p = 0.102), and ophthalmology versus otorhinolaryngology (p = 0.566). There was a significant linear trend for the increasing scores with every year F (1, 7) = 18.558, p ≤ 0.001; the η² = 0.164 indicated a great effect size.

We found an opposite result in the comparison of the MaxSco between surgical specialties, as the ANOVA test was not significant F (4, 78) = 0.708, p = 0.590 which indicated that there was not difference in the MaxSco between surgical specialties; the η² = 0.04 indicated a small effect size. The test for a linear trend of the MaxSco with every year did not show significance F (1, 7) = 1.610, p = 0.209; with a small effect size, η² = 0.020.

Table 1 depicts the means, standard deviation, standard error, and 95% CI for the MinSco and MaxSco scores in each specialty. Figure 1C and D shows mean comparison of surgical specialties showing the trend by year. Figure 1E and F depicts the global trend of the MinSco and MaxSco during 8 years (2012-2019).

Comparison of minimum and maximum scores between Mexicans and IMG in each surgical specialty General Surgery

For this specialty, we did not find a significant difference in the MinSco, but significance was found in the MaxSco between Mexicans and IMG. A similar finding was revealed for gynecology and obstetrics, ophthalmology, otorhinolaryngology, and traumatology and orthopedics. Table 2 depicts the means, SD, standard error of mean between Mexicans and IMG for each surgical specialties; p-values were calculated with the independent t-test.

Positive and negative trends in the minimum and maximum scores between Mexicans and IMG in each surgical specialty

For the minimum score in general surgery, both groups showed a positive and significant correlation, Mexican R = 0.803, p = 0.016, and IMG R = 0.785, p = 0.021. Gynecology and obstetrics, both groups showed a positive but no significant correlation; Mexicans R = 0.632, p = 0.093. IMG, and R = 0.562, p = 0.147. Ophthalmology, Mexicans showed a positive and non-significant correlation, R = 0.596, p = 0.119; on the other hand, IMG showed a positive and significant correlation R = 0.767, p = 0.044. Otorhinolaryngology for both specialties showed a positive but non-significant correlation between MinSco and years; Mexicans R = 0.658, p = 0.076. IMG, and R = 0.529, p = 0.178. In the last category, traumatology and orthopedics, both groups showed a positive and significant correlation; Mexicans R = 0.851, p = 0.007. IMG, and R = 0.828, p = 0.011.

For the maximum score, in general surgery, Mexicans showed a positive non-significant correlation, R = 0.362, p = 0.378. IMG showed a positive and significant correlation R = 0.866, p = 0.005. In the gynecology and obstetrics, both groups showed a negative and no significant correlation; Mexicans R = -0.300, p = 0.470. IMG, and R = -0.414, p = 0.308. Ophthalmology, both groups showed a positive and no significant correlation; Mexicans R = 0.000074, p = 1.00; and IMG R = 0.327, p = 0.474. Otorhinolaryngology for both specialties showed a positive but non-significant correlation between MaxSco and years; Mexicans R = 0.171, p = 0.686. IMG, R = 0.459, p = 0.253. For the last category, traumatology and orthopedics, both groups showed a positive but non-significant correlation; Mexicans R = 0.681, p = 0.063. IMG, and R = 0.474, p = 0.235.

Table 3 shows the trends of minimum and maximum scores grouped as Mexican and IMG and the statistical significance. Figure 2 shows the graphical representation of the observed means for both SMinS and SMaxS scores.

Figure 2 Selected specialties showing the increasing and decreasing trends in the SMinS and SMaxS. 

Comparison of 5-year forecasting trends between minimum and maximum scores of Mexicans and IMG

We identified convergent and divergent forecasting trends between the minimum and maximum scores depending if the lines will eventually touch each other during or after the 5-year forecasted period (2020-2024 years).

Four specialties showed a convergent pattern for Mexicans between the MinSco and MaxSco: general surgery, gynecology and obstetrics, ophthalmology, and otorhinolaryngology. Only traumatology and orthopedics showed a very mild divergent trend. In IMG, three specialties depicted a convergent trend: gynecology and obstetrics, ophthalmology, and traumatology and orthopedics; and the other two, otorhinolaryngology and surgery, showed a divergent trend. Figure 3 shows the forecasting trends between minimum and maximum scores of Mexicans and IMG.

Figure 3 Forecasting trends between minimum and maximum scores of Mexicans and International Medical Graduates. 

Ranking of specialties between Mexicans and IMG

In addition, we ranked the specialties based in the mean of the SMinS between Mexican and IMG for each specialty.

Adjacent rows with connecting arrows show the displacement in the ranking from the rank each specialty reached for Mexican to the position they had for IMG. It was evident that the ranking of medical specialties was similar between both groups, with the exception of ophthalmology which move from 5^th place for Mexicans to the 1^st position for the IMG. Figure 4 shows the ranking displacement in Mexican specialties when we compared with the scores of IMG.

Figure 4 Ranking displacement in Mexican specialties when compared with the scores of International Medical Graduates. 

Grouping of specialties above or below a global mean

The use of an overall mean to compare above or below this mark is helpful to reflect the performance of five-different groups of test-takers that revealed us which specialties had the students with the best scores. The ENARM global mean for the minimum score (from 2012 to 2019) was 72.572 a score above the previous observation made in a study by de la Garza-Aguilar⁶; this number is also above the mean for the past 7 years for the test known as MIR (Medical Intern Resident) in Spain with 57.29 reported by the Ministry of Health²¹,²². Our findings showed that the surgical specialties whose applicants achieve scores above this mean were ophthalmology, otorhinolaryngology, and general surgery. Our findings coincide with the study of Rinard et al.,¹⁹ where otolaryngology was one of the best-ranked specialties among surgical specialties in Texas, USA. The specialties below the mean corresponded to traumatology and orthopedics and gynecology and obstetrics, this observation of low scores at the ENARM contrast with results of the matching program in USA, where this specialties achieved higher accepted than general surgery¹⁹.

Comparison of minimum and maximum scores achieved by surgical specialties

Along the 8 years assessed, it was evident that the ranking of the five surgical specialties was preserved for the MinSco (Fig. 1C), in descendent order otorhinolaryngology, ophthalmology, General surgery, traumatology and orthopedics, and gynecology and obstetrics. On the contrary, for the MaxSco, an entanglement of scores was evident along the 8 years, representing the change of ranking for the surgical specialties at different years (Fig. 1D). For this visualization of data, we also did not find no publications where the performance among medical specialties were compare⁶.

Comparison of minimum and maximum scores between Mexicans and IMG in each surgical specialty

Our findings reveled that Mexicans and IMG got similar passing grades, which might indicate an equivalent level of education in their medical schools; this finding differs from a previous report from USA that observed in 8 years for the orthopedic surgery residency applicants that national got better scores than IMG³. The absence of significant differences in the minimum scores in all specialties comparing Mexican and IMG can also be interpreted as a high competitiveness across all specialties. However, the MaxSco clearly revealed the superiority of Mexicans above IMG for all specialties; which reflect a better level of preparation for this exam. This score revealed a large gap in knowledge between Mexicans and IMG test-takers¹.

Positive and negative trends in the minimum and maximum scores between Mexicans and IMG in each surgical specialty

The limited information about trends for applicants matching into USA specialties has been previously addressed; most of the foreign articles describe the performance of specific specialties, without a comparison between their nationals and IMG²³. The use of the minimum score in our study revealed a fierce competition among medical specialties as four of them showed a positive and significant trend, with the exception of gynecology and obstetrics. This trend is similar to a USA report for the surgical specialties (surgery, plastic surgery, orthopedic surgery, otolaryngology, and obstetrics and gynecology) since each specialty has a different mean score for individuals that are accepted in the match¹⁹. We learned from our findings that there is still missing information and we do not know which scores at specialties are ruled by the applicants every year and which others by the level of difficulty of the exam; an additional analysis will be necessary to understand how the number of residency positions influences the scores at each medical specialties.

Comparison of 5-year forecasting trends between minimum and maximum scores of Mexicans and IMG

The predictive images help us to understand that for Mexicans the gap between MinSco and MaxSco will decrease for general surgery, gynecology and obstetrics, and ophthalmology, however, for IMG gynecology and obstetrics, traumatology and orthopedics, and ophthalmology. It means there are only two out of five surgical specialties (gynecology and obstetrics, and ophthalmology) between Mexicans and IMG share the same learning trend.

Ranking of specialties between Mexicans and IMG

From this analysis we learned that both, Mexicans and IMG depict the same ranking in the order of selected specialties; although Mexicans observed mildly lower MinSco (Fig. 4). For the MaxSco, the 1^st specialty with the highest scores is general surgery, this fact represent a challenge for future applicants, as they would have to get the best scores to be selected for a residency position. In general, Mexicans achieved the highest scores. It was interesting to observe that traumatology and orthopedics was the 2^nd place for Mexicans but the 5^th for IMG.

Limitations of the study

Several limitations need to be acknowledged for this study. With the ENARM, the Mexican Secretariat of Health get to select the best candidates each year with reasonable confidence, but a number much higher than the accepted is left without entering a medical specialty; we did not analyze those numbers as this topic was out of the scope of our study. Furthermore, we did not comment the context regarding the offer and demand of Mexican physicians per number of inhabitants; in 2015, Mexico had 2.2 physicians per 1000 population, including professionals in the private sector, these numbers represent a significant disparity in the distribution of human health resources in the country. The same year, the USA reported a ratio of 3.1 physicians per 1000 inhabitants²⁴. Although the number and needs of the medical specialists are not found fully identified, the number of existing doctors and possible training at the current rate will be insufficient for the needs of the country⁵. We did not get deep in the analysis of which medical schools correspond the test-takers with the highest scores, as this information was not available in the annual CIFRHS reports. Our assessment did not perform subgroup performance differences considering age, gender, the race of test takers, English as a second language because all these items were not publicly available. The same limitations had been addressed in previous reports from USMLE; residency program directors look in the ENARM results for the best candidates for their programs, considering all aspects of a student's application and an interview; however, we did not took into account intangible factors such as away rotations, personal interactions, membership, and research experience, although all of them might influence the chance of matching¹⁹, they were not assessed in the context of this paper. Other topics no included in this study were the need needed to examine whether there is an ideal applicant-to-position ratio that would allow surgical residency coordinators to remain selective in their choices or whether increasing the number of surgical residency positions would dilute the quality of successful candidates.