Implementation and evaluation of a National External Quality Control Program for Cervical Cytology in Mexico
Ana Flisser, Biól, DSc,(1) Felipe García-Malo, MD,(2) María de los Angeles Canepa, MD,(2) Susana Doncel, Comp Tech,(2) Raquel Espinoza, MD,(3) Ramiro Moreno, MD,(3) Isidro Avila, MD,(3) Gregorio Pérez-Palacios, MD, PhD,(4) Roberto Tapia-Conyer, MD, MPH,(5) Juan Ramón de la Fuente, MD, PhD.(6)
Flisser A, García-Malo F, Canepa MA, Doncel S, Espinoza R, Moreno R, Avila I, Pérez-Palacios G, Tapia-Conyer R, De la Fuente JR.
Implementation and Evaluation of a National External Quality Control Program for Cervical Cytology in Mexico ]]>
Abstract
Objective. To evaluate cytology laboratories and the performance of cytotechnologists for establishing efficient external quality control for Mexico's National Program for the Prevention and Control of Cervical Cancer. Material and Methods. During January and February 1998, an on-site evaluation of all cytology laboratories of the Ministry of Health found that only 70% of the microscopes were in adequate working conditions, reagents were out of date, and working conditions were sub-optimal. A program for external quality control based on proficiency testing was established for cytotechnologists. Fifty slide sets with 20 Papanicolaou slides and 10 photographic slides were prepared. The sets were given to the cytotechnologists for evaluation and again one year later by courier. Results. Twenty-one percent of microscopes were repaired and 9% replaced; reagents were distributed and laboratory facilities improved. Only 16% of cytotechnologists passed the initial proficiency test. Cytotechnologists received a refresher training course: one year later 67% of them passed the proficiency test. To ascertain that each slide was correctly diagnosed, 41 sets were rescreened by expert cytopathologists or cytologists and their diagnoses compared to the original ones. Thirty-seven sets had 86% to 96% concordance. Conclusions. This new system for external quality control of cervical cytology allowed the opportune and reliable evaluation of the performance of cytotechnologists. The English version of this paper is available too at: http://www.insp.mx/salud/index.html
Key words: cervix neosplasms; Papanicolaou; proficiency testin; quality control; México
Flisser A, García-Malo F, Canepa MA, Doncel S, Espinoza R, Moreno R, Avila I, Pérez-Palacios G, Tapia-Conyer R, De la Fuente JR.
Instrumentación y evaluación de un programa nacional de control de calidad externo para citología cervical en México.
Salud Publica Mex 2002;44:431-436.
El texto completo en inglés de este artículo también está disponible en: http://www.insp.mx/salud/index.html
Cervical cancer is the most frequent type of malig- nant disease among women in Latin America and the Caribbean. In most of these countries mortality rates from cervical cancer have remained unchanged over the past 30 years.1 It has been clearly established that cervical cancer incidence and mortality can be reduced through appropriate treatment of pre-cancerous lesions2 with early diagnosis using exfoliative cytology, introduced by Papanicolaou in 1941.3 The effectiveness of screening programs has been shown in Iceland, where cervical cancer mortality has been practically eradicated.4 In Canada, mortality was reduced from 11.5 to 4.8 per 100,000 women.5 According to different studies in Latin America, between 20% and 60% of all deaths attributed to cervical cancer can be avoided through screening programs using the Papanicolaou (Pap) test.6 In Mexico, the use of the Pap test started in 1949 in the Women's Hospital (Hospital de la Mujer) of the Ministry of Health7 and is presently performed in all health institutions.
The Instituto de Diagnóstico y Referencia Epidemiológicos (Institute of Epidemiological Diagnosis and Reference, InDRE) belongs to the Ministry of Health. InDRE has, among many other activities, the responsibility of quality control of cervical cytology since 1985.8 Quality control was based on rescreening Pap smears obtained from the Mexican population without social security that had been diagnosed in public cytology laboratories belonging to the Ministry of Health in Mexico. All smears with atypical images and those associated with any degree of dysplasia or carcinoma, and up to 10% of negative smears, underwent rescreening. Results of the comparison of smears diagnosed in the 32 states of Mexico and rescreened in InDRE during 1994-1995, alerted us of the low diagnostic performance. An overall concordance of 70% was found, ranging from 93% in negative smears and 63% in moderate dysplasia, to 13% in carcinomas.9 These discrepant results, together with the increasing number of Pap smears received in InDRE since 1995, from the Program for Prevention and Control of Cervical Cancer, compelled to streamlining the quality control system for diagnosis of cervical cytology. In this paper we present the conception, establishment, evaluation and follow up of the new program.
Material and Methods
A group of experts from Mexico's Ministry of Health was assembled, composed by cytopathologists (MD with 2 years in pathology and one in cytology) and cytologists (MD with 2 years in cytology), leaders in laboratory diagnosis of cervical cytology. The main objective of the group was to define a new external quality control program, based on proficiency testing, following the recommendation of Rodrigo Prado (Pan American Health Organization consultant, Chile). Fifty slide sets were prepared in InDRE. Each set included 20 Pap smears with unequivocal diagnoses (as to their own criteria), provided by some of the experts of the group. Slide sets included smears with the following diagnoses: negative, inflammatory or infectious (40%), low dysplasia (10%), moderate dysplasia (10%), severe dysplasia (10%), in situ cancer (10%), invasive squamous cancer (10%), adenocarcinoma (5%), and inadequate smears (5%). The official classification in Mexico,10 which is that of the World Health Organization,10 was used. The slide sets also included 10 photographic slides with images of the same diagnoses as the smears and one picture of either Trichomonas, Chlamydia or Entamoeba. No photographs of inadequate smears were included. Each set also contained a sheet with clinical data of the slides and a sheet where the cytotechnologist who took the test had to write all 30 diagnoses. A portable viewfiner was provided for those laboratories where no slide projector was available.
]]> Initially the group of experts developed a procedures manual that included the general characteristics of cytology laboratories, type, number, and activities of the personnel, equipment, reagents, and staining methods used. This manual was distributed to all cytology laboratories. A comprehensive evaluation of all cytology laboratories belonging to the Ministry of Health was undertaken by 18 microscopists from InDRE who routinely perform microscopic diagnosis, such as thick smears for malaria, acid-fast bacilli for tuberculosis, stool parasitoscopic studies, and cervical cytology. The group was trained for two weeks on the characteristics of the cervical cancer national program, specially the cytology laboratory procedures, and on how to clean and oil microscopes and identify those in non-working conditions.During January and February 1998, 111 laboratories were visited, 319 microscopes were checked, and slide sets were given to 216 cytotechnologists. Cytotechnologists read the slides in the laboratory with their usual microscope during that workday. Due to the low performance of cytotechnologists, InDRE organized five three-day courses in March and April 1998, with conferences given by members of the expert group, and workshops where slides and Pap smears were shared and discussed. All cytotechnologists attended the five courses. To evaluate their progress in a short time, a pre-test of 10 slides and a pos-test of five Pap smears were included. The diagnosis of each slide in the slide sets and in the tests was considered as correct or incorrect and results were scored as excellent (80 to 100% of correct diagnoses), good (70-79%), fair (60-69%), and failures (less than 60%). Official diplomas were provided for those having excellent and good results. Cytotechnologists who had fair results received official probation letters conditioning their work until the following evaluation. Cytotechnologists with scores <60% received official letters indicating that they were not authorized to perform cytological diagnosis until completion of a three-week training course. Finally, a national workshop divided in two two-day sessions was organized for all cytotechnologists, independently of their grading, in April 1999.
During the second semester of 1998 and 1999, the national external quality control was performed again. This time InDRE sent by courier to state public health laboratories (SPHL) one slide set for each cytotechnologist in the state. SPHL distributed the sets to all laboratories and one week later returned to InDRE the sets and results sheets. Official diplomas, letters of conditional work or disapproval to perform cytological diagnosis, were again prepared.
Finally, to determine the reliability of the slide sets, since they are constituted by smears obtained from routine Pap studies, 41 sets were randomly distributed among the group of experts. The experts work in the Ministry of Health, the National Institute of Cancer, the General Hospital of Mexico, the General Hospital "Dr. Manuel Gea Gonzalez", the Ministry of National Defense, the Mexican Institute of Social Security, the Institute for Social Security for State Workers in Mexico City and in Sonora, the Ministry of Health of Mexico City, and the Ministry of Health of Zacatecas State. Each expert rescreened 3-5 slide sets during December 1999 and January 2000 and diagnoses were compared to the original ones. To assess the concordance of the slide set readings, each of the 20 slides were given five points if both diagnoses were identical, four points if the difference was one diagnosis above or below, two points if the difference was two steps (for example between low and severe dysplasia, between in situ cancer and adenocarcinoma, etc) and zero points if the difference was higher. These results were compared to the grades that the cytotechnologists received in the on-site evaluation and in the courier-sent evaluation.
Results
The results obtained from the national evaluation of all cytology laboratories revealed problems with microscopes, inadequate supply of reagents, and inadequate laboratory facilities (Table I). Between 85% and 18% of the cytotechnologists gave wrong diagnoses (Table I). For example 61% of the cytotechnologists were not able not identify an inadequate sample and 45-46% did not identify carcinomas. Of the 319 microscopes, 70% were in adequate working conditions, 21% were brought to InDRE for repair, and 9% were replaced. There were inadequate amounts of dyes and reagents; for example, there was a higher amount of staining reagents than needed (130%), but 78% were outdated. Around one fourth of the laboratories consisted of a washstand for staining and a table for the microscope, hence having a working area of 0 m2, while another fourth had adequate areas (Table I). Microscopes, dyes, reagents, and staining items were bought and distributed to all laboratories according to the needs detected. State Departments of Health have been invited to improve cytology laboratories and to centralize them in order to have adequate volumes of slides, so as to maintain the ability to perform correct diagnoses of Pap smears. Microscopes for double reading were provided to the central laboratory or to the SPHL of each state for internal quality control; in this way several cytotechnologists working in the same place with a cytologist or a cytopathologist can review together atypical, positive, and non-concordant slides, and a given percentage of negative ones to confirm diagnosis.
Table 2 shows the performance of cytotechnologists before and after the external quality control intervention. The training program improved the performance of cytotechnologists from 59% of failures before to only 12% afterwards. During 1998 and 1999 slide sets were sent by courier to all laboratories; the same cytotechnologists and the new ones were evaluated. Results were reviewed and graded in InDRE and are also shown in table 2. The proportion of excellent and good results was 56%, 15% had fair results and 29% failed. When considering individual changes between both national evaluations, 14% passed both evaluations and 54% of cytotechnologists failed in the on-site one but passed the courier-sent evaluation; 13% remained as failures and a similar proportion moved from failure to conditional work. Only 3 and 2% had a lower performance; that is, they were downgraded from excellent or good results to conditional work or to failure, respectively. Regarding the technologists that were incorporated to cytology laboratories along the time of this study, 84% had excellent or good results in the courier-sent evaluation.
Since the slides used for quality control were obtained during clinical work, although diagnosed by experts, we wanted to ascertain their performance. Between 86 and 96 points (out of 100) of concordance were obtained by expert rescreening, with one exception that showed 73 points of concordance. The percentage of cytotechnologists that had excellent or good results with the same sets during the on-site evaluation was between 0 and 50%, increasing to 47-67% with the courier-sent evaluation of 1998-1999. The non-concordant slides were replaced with new slides provided by the same group of experts.
Discussion
A previous study performed in Mexico found a poor diagnostic reproducibility with an overall rate of 35% false negative results in the Pap test.12 The study performed in InDRE showed great discrepancies between the results obtained in the Institute and those from the cytology laboratories across the country.9 Other studies have shown that the validity and reproducibility of cytology diagnosis vary greatly within and between institutions13 and between Latin American countries.14 The authors argue that the main problem is the overwhelming lack of methods to evaluate cytotechnologists and cytopathology laboratories, and that quality control of diagnostic precision is central to the consistency of results in cytology laboratories.13,14 Therefore, a new external quality control program for cervical cytology was created based on proficiency testing.
]]> To have a complete panorama of laboratory issues, an on-site evaluation was performed for all laboratories belonging to the Ministry of Health from across the country, which allowed a short time evaluation of the performance of all cytotechnologists and the conditions of the laboratories. Cytology laboratories received the microscopes and reagents needed; this effort included purchasing and distribution of adequate supplies, replacing microscopes, and providing those for co-observation. Moreover, each of 319 microscopes were checked, cleaned, and oiled or replaced; this was an important support for the daily work of cytotechnologists. No correlation was found between the screening efficiency and the working conditions of the microscopes, reagents, and laboratories, probably because many cytotechnologists had poor performance and many microscopes were in good working conditions. Training of cytotechnologists in short courses and workshops clearly increased their diagnostic performance. Before training, 59% of cytotechnologists failed and only 29% afterwards. It should be noted that more or less similar proportions were obtained with the slide sets during the on-site evaluation and the 10 slides used in the pre-course test, or with the post-course evaluation and the courier sent-sets; these results support the notion that the sets provided reproducible results and therefore were useful for quality control. Clearly the intervention was responsible for the change in the proportion of cytotechnologists who had excellent and good results, since 54% were upgraded from failing the first evaluation to good and excellent results in the second one, as compared to only 14% who passed both evaluations. Also, most new cytotechnologists had good diagnostic ability.Until 1997 external quality control of cervical cytology was performed in the Ministry of Health of Mexico by rescreening of Pap smears. The high discrepancy of results9 and the increasing number of slides received in InDRE prompted a change of system. Proficiency testing is now recognized as an alternative way for evaluating the diagnostic quality of slide reading.15, 16 For this purpose 50 sets were prepared with slides provided by experts in cervical cytology and used in two national quality control evaluations. Sets were prepared with slides having similar proportions of different types of diagnoses to those found after rescreening 10,098 slides from the Departments of Health across the country in 1994-1995.9 This system allowed detecting the diagnostic ability of cytotechnologists. Finally, when the concordance value of diagnoses given by experts was associated with the percentage of cytotechnologists that had excellent or good results with the same slide sets in the on-site evaluation (white circles) or in the courier-sent evaluation (black squares), it was found that when cytotechnologists had a poorer performance (white circles), sets with higher concordance between experts had higher percentages of excellent or good results. This trend was not statistically significant but suggests that training facilitated reading of more difficult slides (Figure 1).
This new system for external quality control of cervical cytology allowed timely and reliable evaluation of the performance of cytotechnologists working in the National Program for the Prevention and Control of Cervical Cancer. After providing training courses, microscopes, and reagents, the quality control program demonstrated improvement in the diagnosis performed by the cytotechnologists. International standards recommend external quality control not to be performed in an individual basis but on laboratories as evaluation units.14,16,17 The improvement in diagnostic capabilities of cytotechnologists in Mexico now allows for this change. Also, external quality control programs should allow keeping the slide sets in the laboratories, until results are graded at the national laboratory and returned, so that each laboratory can go over the slides to review wrong diagnoses. Only after that should sets be sent back to the national laboratory.16 For this purpose more Pap smears of excellent quality are being collected in InDRE, to assemble more test slide sets.
Acknowledgements
The authors wish to thank all Mexican cytotechnologists who participated in the on-site evaluation and in the new external quality control program, as well as the group of experts for their participation and collaboration, PAHO and Rodrigo Prado (PAHO, Chile) for support, and Dolores Correa (Instituto Nacional de Pediatria), Maribeth Gagnon, and John Ridderhoff (CDC) for critical review of the manuscript.
]]> References
1. Robles S. Introduction to the special issue: Timely detection of cervical cancer. Bull PAHO 1996:30:285-289. [ Links ]
2. Laara E, Day E, Hakama M. Trends in mortality from cervical cancer in the Nordic countries: Association with organized screening programs. Lancet 1987;1:1279-1281. [ Links ]
3. Papanicolaou GN, Traut HF. The diagnostic value of vaginal smears in carcinoma of the uterus. Am J Obstet Gynecol 1941;42:193. [ Links ]
4. Johannesson G, Geirsson G, Day NE. The effect of mass screening in Iceland. 1965-1974, on the incidence and mortality of cervical carcinoma. Intl J Cancer 1978;21:418. [ Links ]
5. Boyes DA, Nichols TM, Millner AM, Worth AJ. Recent results from the British Columbia screening program for cervical caner. Am J Obstet Gynecol 1977;128:692. [ Links ]
6. Restrepo HE, González JL, Roberts E, Litvak J. Epidemiología y control del cáncer del cuello uterino en América Latina y El Caribe. Bol Oficina Sanit Panam 1987;102:578. [ Links ]
7. García G. La detección de cáncer del cuello uterino en 75 855 mujeres. Gac Med Mex 1970;100:154. [ Links ]
8. Weissbrod-de Bonder D, Magis-Rodríguez C. Control de calidad del diagnóstico citológico de displasias en México. Higiene 1994;II: 207-212. [ Links ]
9. Weissbrod D, Torres M, Rodríguez A, Ureña I, Estrada J, Reyes ML et al. Comparación del examen de citología cervical efectuado por el método Papnet y por microscopía convencional. Bol Oficina Sanit Panam 1996;121:528-535. [ Links ]
10. Secretaría de Salud. Norma Oficial Mexicana. NOM-014-SSA2-1994, para la Prevención, Detección, Diagnóstico, Tratamiento, Control y Vigilancia Epidemiológica del Cáncer Cérvico Uterino. México, D.F.: Diario Oficial, Marzo 6, 1998. [ Links ]
11. Organización Panamericana de la Salud. Manual de normas y procedimientos para el control del cáncer de cuello uterino. PALTEX 1990;6;37-43. [ Links ]
12. Lazcano-Ponce EC, Buiatti E, Nájera-Aguilar P, Alonso-de Ruiz P, Hernández-Avila M. Evaluation model of the Mexican national program for early cervical cancer detection and proposals for a new approach. Cancer Causes Control 1998;9:241-251. [ Links ]
13. Lazcano-Ponce EC, Alonso-de Ruiz P, López-Carrillo L, Nájera-Aguilar P, Avila-Cisneros R, Escandón-Romero C et al. Validity and reproducibility of cytologic diagnosis in a sample of cervical cancer screening centres in Mexico. Acta Cytol 1997;41:277-284. [ Links ]
14. Alonso-de Ruiz P, Lazcano-Ponce EC. Quality control in cytopathology laboratories in six Latin American countries. In: Compendium on quality assurance, proficiency testing and workload limitations in clinical cytology. 1995:115-121. [ Links ]
15. United States Department of Health and Human Services. Health care financing administration. Medicare, Medicaid and CLIA Programs. Regulations implementing the clinical laboratory improvement amendments of 1988 (CLIA). Federal Register 1992;57(49):7002-7186. [ Links ]
16. Organización Panamericana de la Salud. Módulo de Citología. Procedimientos. Washington, D.C., 2000:1-20. [ Links ]
17. Alonso-de Ruiz P, Lazcano-Ponce EC, Hernández-Avila M. Cáncer cervicouterino. Diagnóstico, prevención y control. México, D.F.: Editorial Médica Panamericana, 2000: 203-209. [ Links ]
]]> (1) Instituto de Diagnóstico y Referencia Epidemiológicos, Secretaría de Salud (SSA), México, D.F., México. Actualmente en el Departamento de Microbiología y Parasitología, Facultad de Medicina. Universidad Nacional Autónoma de México (UNAM). México, D.F., México.
(2) Instituto de Diagnóstico y Referencia Epidemiológicos, Secretaría de Salud. México, D.F., México.
(3) Programa Nacional de Prevención y Control de Cáncer Cérvico Uterino, SSA, México, D.F., México.
(4) Programa de Salud de la Mujer, SSA, México, D.F. Actualmente en el Hospital General de México, SSA. México, D.F., México.
(5) Subsecretaría de Prevención y Control de Enfermedades, SSA. México, D.F., México.
(6) Secretaría de Salud, México, D.F., México. Actualmente en la Rectoría de la UNAM, México, D.F., México.
Received on: January 29, 2001 • Accepted on: April 17, 2002
Address reprint requests: Ana Flisser. Departamento de Microbiología y Parasitología, Facultad de Medicina, UNAM, Ciudad Universitaria, San Angel, 04510 México, D.F., México.
e-mail: flisser@servidor.unam.mx