nueva página del texto (beta)
Inglés (pdf)
Artículo en XML
Referencias del artículo
Traducción automática
Enviar artículo por email
Citado por SciELO 
Similares en
SciELO 
Permalink
Introduction
Information and Communication Technologies (ICT) are an important resource for improving the performance of health systems. The applications of ICT in health are closely related to the concept of electronic health (e-Health), which is defined as “the cost-effective and secure use of information and communications technologies in support of health and health-related fields, including health-care services, health surveillance, health literature, and health education, knowledge and research” [1]. A similar concept, under the term digital health, has also been recently introduced [2,3].
The application of ICT in the areas of health surveillance, monitoring, prevention, promotion and care, at various levels or fields of application, has evolved into the concepts of e-Health and digital health [4].
As the principal components of e-health are the following: health information systems (HIS) supported by ICT (public health informatics, clinical informatics, and consumer health informatics); Electronic Medical Records (EMR); Electronic Health Records (EHR); Picture Archiving and Communication Systems (PACS); Clinical Decision Support Systems (CDSS); remote care systems (telehealth and telemedicine); and, mobile systems (mobile health or m-Health). It should be noted that the eHealth innovations have been continually growing [5][6][7][8][9].
Various studies and systematic reviews have found evidence on the effectiveness and efficiency of the components of eHealth [10-13], on the basis of which, global and regional organizations have issued diagnostics, policies, and recommendations for improving the implementation of ICT in the health systems of developing and developed countries [3, 14-21]. To this end, various frameworks and recommendations have been developed from different disciplinary perspectives [22-27].
In order to discuss eHealth in Mexico, it is essential to first describe the country’s Sistema Nacional de Salud (SNS or National Health System), which is characterized by its fragmentation into two large public and private sectors. The public sector is subdivided into health systems for people with formal employment and their direct family members (essentially a social security scheme) and health systems for people without formal employment. The social security system is made up the Instituto Mexicano del Seguro Social (IMSS or Mexican Social Security Institute), which comprised 39.2% of the sector in 2015; the Instituto de Seguridad y Servicios Sociales de los Trabajadores del Estado (ISSSTE or Social Security and Services Institute for State Workers), which comprised 7.7% of the sector; and the health services used by the armed forces (Secretaría de la Defensa Nacional, SEDENA or Ministry of Defense, and Secretaría de Marina, SEMAR or Ministry of Navy) and employees of Petróleos Mexicanos (Pemex), which are provided by their respective institutions and comprised 1.2%. Health services for people without formal employment are mainly provided by the Secretaría de Salud (SSA or Ministry of Health) through the Servicios Estatales de Salud (SESA or State Health Services), which comprised 49.9% of the sector, although it is currently undergoing a process of incorporation into the newly-formed Instituto de Salud para el Bienestar (INSABI or Institute of Health for Welfare). There are also some special programs within the public sector, such as the IMSS-Bienestar (IMSS-Welfare) and the Sistema Nacional para el Desarrollo Integral de la Familia (DIF Nacional or National System for Integral Family Development) programs, that aim to provide healthcare to vulnerable populations. Finally, the private sector comprises a multiplicity of service providers for people with the ability to pay for healthcare [28-30].
Recent federal administrations have pursued policies promoting the adoption and implementation of ICT, particularly in the public health sector. The last such policy was the Estrategia Digital Nacional (EDN or National Digital Strategy) [31]. However, individual and organizational factors have limited the adoption and implementation of ICT, such as the persistently fragmented structure of the public health system.
Research on the adoption and implementation of ICT in the public health system in Mexico is scarce, which impedes a broad understanding of the problems faced by those making the decisions in this policy area. However, some general overviews have been published [14,32-34], as well as some studies related to particular eHealth components, such as telemedicine [35] and mHealth [36,37], and initiatives undertaken by private institutions [38-40]. Regarding EMR systems, some studies have analyzed the planning, adoption, and its implementation in public health institutions [32,41,42], focusing on identifying the factors that enable or hinder the success of this implementation. Some of these factors are related to technology (i.e., problems with diagnostic catalogs), the user (i.e., the age of some of the physicians providing primary care), and the organization itself (i.e., a lack of training or quality in its provision).
While the afore mentioned studies have focused principally on exploring the advances in and barriers to the implementation of EMR systems, they have not applied models or frameworks taken from behavioral sciences or social psychology [43,44] as a guide. In this sense, some socio-technical frameworks have been developed to evaluate information systems in organizations, such as DeLone and McLean’s Model of Information Systems Success (MISS) [27]. This approach focuses on evaluating information system quality and implementation in an organizational context. This model focuses on evaluating information systems from the users’ perspectives, via three main domains: 1) variables for evaluating the quality of the system, the quality of the information provided, and the quality of the service provided to users via the implementation of the system; 2) both system use and user satisfaction; and, 3) the perceived benefits to the health service in which the system was implemented.
The MISS has been used around the world in the evaluation of EMR systems and other eHealth components generating evidence about the validity of this application in different contexts [45-53]. The objective of the present study was to explore the use of the MISS to evaluate an EMR implemented in the public health services of Aguascalientes, as a case study that could inform its application in other health institutions of Mexico.
Materials and methods
Study design and theoretical model
The case study design[54] applied in the present study used, as a theoretical guide, DeLone and McLean’s Model of Information Systems Success (MISS), particularly its most recent update [27].
Study setting
The present study was performed at the Instituto de Servicios de Salud del Estado de Aguascalientes (ISSEA), which is the main health service provider for the population in the state not covered by social security. In 2015, the total population of Aguascalientes was estimated at 1,292,901 inhabitants, of whom an estimated 42.9% do not have social security coverage [55]. The ISSEA comprises the following: 21 urban health centers (Centros de Salud Urbanos or CSUs); and, 62 rural health centers (Centros de Salud Rurales or CSRs) providing primary care across three sanitary jurisdictions. The ISSEA has six second-level hospitals (385 beds in total) and one third-level hospital (60 beds in total) [56]. A previous published study described with more detail the healthcare provision system of the ISSEA and the general e-Health ecosystem that have been implemented [57].
The EMR system has been progressively implemented in all the primary care clinics and emergency services functioning at the second and third level of care. The advances in EMR development at the ISSEA are shown in Table 1. The EMR consists of four main modules: 1) Patient registration and payment; 2) Medical consultation; 3) Pharmacy; and, 4) Statistics. The findings presented in Table 1 reveal the significant development of the EMR system [57], with the users of these EMR modules comprising the following, respectively: 1) Administrative and management personnel; 2) Physicians; 3) Pharmacy technicians; and, 4) Administrative personnel (statisticians).
Table 1 Development of the EMR in the study setting
| Patient registration and payment |
|---|
| • Registration of service users. • Identification of user affiliation to other health subsystems and special programs. • User identification via a unique identifier (Clave Única de Registro de Población - CURP, or Unique Population Registry Code). • Administration of fees for service provision • Schedule for medical appointments. • Human resources administration. |
| Medical consultation |
| • Integrated padlocks that automatically issue patient-dependent reminders for screening and health prevention activities. • Integrated clinical information requests to promote proper patient management or control (mainly chronic disease patients). • Integrated alerts or recommendations to improve the quality of care (prompts for referrals to specialists once need has been identified based on clinical parameters). • Links to medical diagnostic catalogs. • Provision of information about patient drug coverage (based on their membership of particular programs) • Provision of pharmacy stock information. • Incorporation of clinical practice guidelines and medication information. • Provision of information on recommended dosages in order to make dosage adjustments and promote the reasonable use of medication. |
| Pharmacy |
| • Provision of information on the essential medication chart (Cuadro Básico de Medicamentos). • Support for the management of stocks of medication and other health supplies. |
| Statistics |
| • Provision of the necessary forms for maintaining the monthly health service provision reports and compiling epidemiological statistics. |
Source: Prepared by the author.
Sampling of medical personnel
In particular, the medical consultation module of the EMR was evaluated from the perspective of medical personnel either working or providing services in ISSEA CSU or CSR primary care health centers. Physicians were selected as informants, as it is, they who use said module of the EMR. A convenience sample was obtained based on physicians working as medical directors or in charge of personnel in each health center of the state’s sanitary jurisdictions I and III in the period August 2018 to April 2019. In circumstances where the medical director was not available as an informant, medical personnel were included as informants in the sample, in their stead. The inclusion criterion was to have worked or to be currently working with the EMR system.
Data collection
An adaptation of the online questionnaire developed by Canada Health Infoway, which is based on the Benefits Evaluation Framework and the MISS, was applied in the present study as an instrument for evaluating the EMR [46,58,59]. Some items from the questionnaire were selected, translated into Spanish and placed on a Google Forms sheet, which was sent to the informants of the study. The variables included in the model were mainly measured using five-level Likert scales, only the use variable was measured through a proposed 10-level scale. A description of the variables and items included in the instrument is shown in Annex 1.
Data analysis
The data collected was analyzed using STATA 15 software, with Structural Equations Modeling (SEM) analysis used to test the theoretical model comprehensively. Adjustments were implemented in order to optimize the model according to suggestions provided by the software. The model was tested using the maximum likelihood method and the Satorra-Bentler method for standard error computation [60].
Ethical considerations
The research protocol was reviewed and approved by the Research Management of the Centro de Investigación e Innovación en Tecnologías de la Información y Comunicación (INFOTEC or Center for Research and Innovation in Information and Communication Technologies). Prior to responding to the survey, the informants were informed about the research objectives and those responsible for the study. All informants gave their consent to participate in the study. The identity of the informants was protected using alphanumeric codes.
Results and discussion
Sample characteristics
The sample characteristics are shown in Table 2. A total of 62 physicians answered the online questionnaire, working at a total of 46 health centers, namely 88.5 % of all the primary care health centers, comprising a total of 52 health centers (15 CSUs and 37 CSRs) in sanitary jurisdictions I and III. Moreover, these represented 38.3 % of the 162 physicians working at these health centers, as previously reported by the SSA [56]. This sample comprised mainly qualified general medical practitioners (66.1%), although the sample also included personnel who had yet to qualify and were carrying out their one-year period of social service (30.6%). The majority had worked with the EMR system for more than three years (46.8%), meaning that, therefore, they were qualified physicians.
Table 2 Descriptive statistics of the medical staff surveyed (n=62).
| Data | n (%) |
|---|---|
| Sanitary Jurisdiction | |
| I | 40 (64.5) |
| III | 22 (35.5) |
| Age | |
| > 20 | 1 (1.6) |
| 21 to 30 | 22 (35.5) |
| 31 to 40 | 10 (16.1) |
| 41 to 50 | 18 (29.0) |
| 51 to 60 | 8 (12.9) |
| > 61 | 3 (4.8) |
| Sex | |
| Female | 24 (38.7) |
| Male | 38 (61.3) |
| Education | |
| Non-qualified physician (social service) | 19 (30.6) |
| General medical practitioner | 41 (66.1) |
| Specialist physician | 1 (1.6) |
| Master’s degree in Public Health | 1 (1.6) |
| Health service type | |
| Rural (CSR) | 39 (62.9) |
| Urban (CSU) | 23 (37.1) |
| Time working with EMR | |
| > 1 month | 2 (3.23) |
| 1 to 3 months | 10 (16.1) |
| 4 to 6 months | 6 (16.1) |
| 7 to 12 months | 7 (11.3) |
| 1 to 2 years | 8 (12.9) |
| > 3 years | 29 (46.8) |
| Total | 62 (100) |
Notes: aStatistical significance levels: * p<0.05, **p<0.01, ***p<0.001
Source: Prepared by the author.
Measurement of the MISS variables
The results for the measurement of the MISS variables via their respective items are shown in Table 3. As can be seen, means above three were obtained for all items on the five-level Likert scale used in the present study. The lowest means were obtained for two items related to service quality (SQ1 and SQ2, with means of 3.27 and 3.15, respectively), while the highest means were obtained for items related to system quality (SQ5, with a mean of 4.08) and net perceived benefits (NB5 and NB6, with means of 4.05 and 4.11, respectively). It should be noted that the four MISS variables that were measured using Likert items showed high internal consistency values (Coefficient ( > 0.85).
Table 3 Measurement of the variables from the Model of Information Systems Success (MISS).
| Variable | Itema | Nob | Mean | Standard deviation | Coefficient αc | |
|---|---|---|---|---|---|---|
| System quality | SQ1 | L5 | The EMR is easy to use | 3.94 | 0.14 | 0.9503 |
| SQ2 | L5 | The effort required to complete an action or access information in the EMR is acceptable | 3.84 | 0.13 | ||
| SQ3 | L5 | The EMR response time is acceptable | 3.84 | 0.13 | ||
| SQ4 | L5 | The EMR is integrated with my workflow | 3.76 | 0.14 | ||
| SQ5 | L5 | The access to the EMR is acceptable | 4.08 | 0.12 | ||
| SQ6 | L5 | The characteristics of the EMR enable me to perform my job well | 3.63 | 0.14 | ||
| SQ7 | L5 | The EMR is consistent in its performance | 3.64 | 0.13 | ||
| SQ8 | L5 | In general, the quality of the EMR is excellent | 3.42 | 0.14 | ||
| Information quality | IQ1 | L5 | The information provided by the EMR is complete | 3.71 | 0.13 | 0.9410 |
| IQ2 | L5 | The information provided by the EMR is timely | 3.95 | 0.11 | ||
| IQ3 | L5 | The information provided by the EMR is appropriate | 3.87 | 0.12 | ||
| IQ4 | L5 | The information provided by the EMR is relevant | 3.97 | 0.11 | ||
| IQ5* | L5 | The information provided by the EMR is available when needed | 3.79 | 0.13 | ||
| IQ6 | L5 | The information provided by the EMR enables me to make patient care decisions or recommendations more quickly | 3.69 | 0.14 | ||
| Service quality | SQ1 | L5 | The current level of EMR training is acceptable | 3.27 | 0.15 | 0.8584 |
| SQ2 | L5 | The level of ongoing support provided for the use of the EMR is acceptable | 3.13 | 0.15 | ||
| Use | U1 | N10 | On a scale of 0 to 10, What amount do you use the EMR for patient care? | 8.7 | 0.27 | 1 |
| User satisfaction | US1 | L5 | Overall, how satisfied are you with the EMR? | 3.8 | 0.13 | 1 |
| Net benefits | NB1 | L5 | The EMR improves my productivity | 3.69 | 0.14 | 0.9128 |
| NB2 | L5 | The EMR improves the quality of care I provide | 3.56 | 0.16 | ||
| NB3 | L5 | The EMR makes my job easier | 3.71 | 0.15 | ||
| NB4 | L5 | The EMR improves our capacity for continuous patient care | 3.74 | 0.14 | ||
| NB5* | L5 | The EMR improves our ability to share patient information among healthcare staff | 4.05 | 0.12 | ||
| NB6* | L5 | The EMR improves the efficiency of requesting laboratory tests, and X-rays, as well as generating prescriptions | 4.11 | 0.11 | ||
| NB7 | L5 | The EMR improves the quality of my decision-making | 3.43 | 0.15 | ||
| NB8 | L5 | The EMR reduces fragmentation in the care provided to my patients and facilitates their mobility on the continuum of care | 3.53 | 0.14 | ||
| NB9 | L5 | The EMR reduces risks to patient safety | 3.89 | 0.13 |
Notes: aItem eliminated in order to improve the internal consistency of the respective variable (Coefficient α) and facilitate model adjustment
bNumber of categories used for the item response: L5 - Likert scale of 5 categories; and, N - numeric scale
cCoefficient α obtained after model adjustment
Source: Prepared by the author.
Estimations for the MISS
The structural model tested is shown in Figure 1, while the results of the calculation of direct and indirect effects are shown in Table 4. As can be seen, in terms of the dependent variable of use, only system quality was a significant variable (β = 0.389). With regard to the dependent variable of user satisfaction, the effects of system quality (β = 0.907) and service quality (β = 0.331) were significant, while, in terms of the dependent variable of net benefits, only user satisfaction was significant (β = 0.728). The values of the coefficient of determination (R2) were high for user satisfaction and net benefits (0.931 and 0.877, respectively), but low for use (0.413).
aThe rectangles represent the variables comprising the MISS, while the arrows show the direct effects of independent variables on the dependent variables of this model (the indirect and total effects are described in Table 4).
bStatistical significance of the path coefficients: * p<0.05; **p<0.01; and, ***p<0.001
Source: Prepared by the author.
Figure 1 Adjusted MISS.a,b
Table 4 Direct, indirect and total effects on the dependent variables obtained in the adjusted model.a
| A predicts B | Standarized Effects | |||
|---|---|---|---|---|
| A | B | Direct | Indirect | Total |
| System quality | Use | 0.389** | ----- | 0.389** |
| Information quality | Use | 0.228 | ----- | 0.228 |
| Service quality | Use | 0.105 | ----- | 0.105 |
| System quality | User satisfaction | 0.907*** | -0.168 | 0.739*** |
| Information quality | User satisfaction | 0.064 | -0.098 | -0.034 |
| Service quality | User satisfaction | 0.331* | -0.045 | 0.286** |
| Use | User satisfaction | -0.431 | ----- | -0.431 |
| System quality | Net benefits | ----- | 0.696*** | 0.696*** |
| Information quality | Net benefits | ----- | 0.068 | 0.068 |
| Service quality | Net benefits | ----- | 0.251** | 0.251** |
| Use | Net benefits | 0.406 | -0.314 | 0.092 |
| User satisfaction | Net benefits | 0.728* | ----- | 0.728* |
Notes: aStatistical significance levels: * p<0.05; **p<0.01; and, ***p<0.001
Source: Prepared by the author.
Model adjustments were made by means of the software used in the present study, with some items (IQ5, NB5, and NB6) excluded, thus enabling better reliability and model fit. Through the adjustments made, a model was obtained that meets some of the criteria suggested in the literature [60], such as the significance of the Chi2 statistical test (p <0.000), Chi2/df < 2 (result = 1.47) and the SRMR indicator < 0.1 (result = 0.068). Also, this model was near to meet other criteria such as RMSEA < 0.05 (result = 0.110) and CFI > 0.95 (result = 0.877) indicators.
Improvement opportunities for the EMR
Various improvement opportunities were identified from the physicians’ perspectives, principally regarding the medical consultation module (Table 5). Some of the most significant suggestions made by the physicians were: 1) The continuous updating of information, such as diagnostic catalogs, stock information, and information pertaining to the co-payment systems for drug catalogs; 2) The implementation of interoperable information systems between the second and third levels of care to improve the availability of information (i.e., results of laboratory tests performed in hospital and counter-referral information); 3) The automation of the monthly reporting of institutional statistics, including the data taken from private sector information systems (Fundación Carlos Slim [38,61]).
Table 5 Opportunities for improving the EMR from the physicians’ perspective
| Medical consultation |
|---|
| Continuous updating of the diagnostics catalogs and improved participation of medical staff in this endeavor. Continuous updating of the clinical practice guidelines. Implementation of interoperable information systems between the second and third levels of care, which will improve follow-up of patient referrals and counter-referrals, thus promoting integrated care and giving access to laboratory, radiology and imaging tests. Implementation of systems to identify users or patients with chronic diseases who travel continuously for work purposes (i.e., migrants), to avoid limiting their medication supply. Continuous monitoring of medication stock held in the pharmacy, in order that the computerized system corresponds to physical availability. Continuous monitoring to prevent errors in the lists of medication covered by special programs. |
| Statistics |
| Automation of the process for filing monthly reports, in order that the information for the monthly service provision reports is obtained automatically from the EMR. Also, for the filing of information systems that have been provided by private entities. |
Source: Prepared by the author
This study used a socio-technical and psychometric approach to evaluate an EMR system by applying the MISS, a comprehensive conceptual framework. The main contribution of the study could be found in the manner in which it applied this model in the primary care physicians and in the context of the study setting.
As an approach of reliability, the data gotten from the instrument adapted for and translated into Spanish obtained good results in terms of the internal consistency (Coefficient () in the different constructs. Besides, an structural equations model (SEM) analysis was used to study the relationship among the different variables of the MISS allowing to identify the relationship between the variables related to EMR quality (quality of the system / quality of the information) and implementation (quality of the service) and the closely related satisfaction / use variables, as well as the relationship with the perceived net benefits reported by medical personnel. However, it was not possible to prove some relationships, mainly those with the use variable, which could be due to the fact that it was measured with a single item, when there may be different modalities and dimensions of the use of the EMR by medical personnel, including the frequency, types, objectives, and quality of use.
In terms of the statements of an international consensus regarding psychometric research reporting [62], this study contributes on generating evidence about the validity of the data gotten from the adapted instrument derived from the MISS, as an approach to evaluating EMR systems in public health services in Mexico. However, it is considered that more studies should be performed in order to improve the adapted instrument and generate robust evidence taking into account the different contexts derived from the country’s fragmented health system. Also, the incorporation of the perspectives of physicians working at the second or third levels of care as well as other healthcare professionals could be of great relevance to particular healthcare systems, considering the different levels of EMR implementation in the public health system. In this regard, it should be noted that various published studies have contributed too on generating evidence in this field, using the MISS, and some modified versions, in other countries at different levels of healthcare, from different health workers’ perspectives, and focusing on the evaluation of different eHealth systems, such as HIS, EMR and EHR [45-50], and PACS [51-53].
Moreover, beyond the focus on developing and validating the MISS or an integrative instrument based on it, the building of indicators from the measurement of MISS variables could also be used as a way to guide and inform the implementation of EMR systems (or other eHealth components) in healthcare institutions. Expanding on this topic, these indicators could be tested and later used in the context of eHealth or digital health policy implementation in Mexico in different contexts. It should be noted that this has been the focus of studies conducted in other countries [63,64], some of which have focused their analysis on identifying opportunities to improve the application of these technologies [65,66].
The following were identified as strengths of the study: 1) The use of MISS, an integrated framework that incorporates various dimensions related to the success of information systems and goes beyond the technology adoption or acceptance models; 2) The use of a structural equation model (SEM) as an analytical methodology that enables an analysis of all the model variables and their relationships (such as their direct and indirect effects); and, 3) The identification of opportunities for improving the EMR system as well as its implementation from the perspective of medical staff, through which the authorities or management can be made aware informed (a formative evaluation approach, which focuses on improving implementation [67,68] and has been used in other eHealth studies [48]).
As weaknesses of the present study, the following were identified: 1) The case-study design was applied to only one state health service thus limiting the external validity of the study to other state health institutions (SESA, SSA), as these have implemented their own EMR systems; 2) a complete set of responses by the physicians to the questionnaire was not obtained due to the low response rate; and, 3) The need to build a more integrated measurement of the use variable. These weaknesses are identified as further research opportunities.
Conclusion
The present study contributes generating evidence on the use of the MISS and the adapted instrument to evaluating EMR systems in public health services of Mexico, although some improvements should be performed in this instrument. Besides, more evidence should be generated in this field in order to promote the continuous improvement of EMR systems, in the context of the e-Health (or Digital health) policy.
