SciELO - Scientific Electronic Library Online

 
vol.47 número3Adaptación cultural y propiedades psicométricas de la escalade actitudes hacia el cigarro convencional en estudiantes universitarios mexicanosProblemas emocionales y conductuales de preescolares mexicanos antes y durante la pandemia de COVID-19 índice de autoresíndice de materiabúsqueda de artículos
Home Pagelista alfabética de revistas  

Servicios Personalizados

Revista

Articulo

Indicadores

Links relacionados

  • No hay artículos similaresSimilares en SciELO

Compartir


Salud mental

versión impresa ISSN 0185-3325

Salud Ment vol.47 no.3 México may./jun. 2024  Epub 21-Feb-2025

https://doi.org/10.17711/sm.0185-3325.2024.015 

Original articles

Design, reliability, and validity of the acceptability of internet-based psychological interventions questionnaire in Mexican university students

Diseño, confiabilidad y validez del cuestionario de aceptabilid ad de intervenciones psicológicas en internet en estudiantes universitarios mexicanos

Raquel Mondragón Gómez 1  
http://orcid.org/0000-0002-1785-3291

Nora Angélica Martínez Vélez 2  
http://orcid.org/0000-0001-8907-2155

Marcela Tiburcio Sainz 2   * 
http://orcid.org/0000-0001-7548-7800

Morise Fernández Torres 2  
http://orcid.org/0000-0001-8573-5126

1Programa de Maestría y Doctorado, Universidad Nacional Autónoma de México, Ciudad de México, México.

2Departamento de Ciencias Sociales en Salud, Dirección de Investigaciones Epidemiológicas y Psicosociales, Instituto Nacional de Psiquiatría Ramón de la Fuente Muñiz, Ciudad de México, México.


Abstract

Introduction

Internet-based psychological interventions are an effective option for treating mental health problems. Identifying the acceptability of these services makes it possible to improve their design and user adherence. However, only a few psychometric instruments exist to evaluate this acceptability.

Objective

To design and evaluate the psychometric properties of an internet-based psychological interventions questionnaire, based on the theory of technology acceptance.

Method

The study was divided into three parts: 1) Design of instrument items, 2) analysis of psychometric properties and exploratory factor analysis, and 3) confirmatory factor analysis.

Results

The instrument proved to have adequate psychometric properties, with the following goodness-of-fit measurements: χ2/df = 168.92/74 = 2.28, CFI = .935, TLI = .920, RMSEA = .080, 95% CI [.64, .096]. The analysis of internal consistency found an α = .91 for the total scale, an α = .91 for the first factor, “Approval of use,” an α = .79 for the second factor, “Perceived usefulness,” and an α = .59 for the third factor, “Perceived risk.”

Discussion and conclusion

The evaluation of factors associated with greater acceptability is a potential tool for improving awareness of the use of online psychological interventions.

Keywords: Internet-based interventions; acceptability; internet; users of mental health services; eMental Health

Resumen

Introducción

Las intervenciones psicológicas a través de Internet son una opción eficaz para tratar distintos problemas de salud mental. Identificar la aceptabilidad de estos servicios por parte de los usuarios permite mejorar su diseño y la adherencia de los usuarios. Sin embargo, existen pocos instrumentos psicométricos para evaluar esta aceptabilidad.

Objetivo

Diseñar y evaluar las propiedades psicométricas de un instrumento de aceptabilidad para las intervenciones psicológicas en línea, basado en la teoría de la aceptación de la tecnología.

Método

El estudio se dividió en tres partes: 1) Elaboración de los ítems del instrumento, 2) análisis de las propiedades psicométricas y análisis factorial exploratorio, y 3) análisis factorial confirmatorio.

Resultados

Se muestra que el instrumento tiene propiedades psicométricas adecuadas, con las siguientes medidas de bondad de ajuste: χ2/df = 168.92/74 = 2.28, CFI = .935, TLI = .920, RMSEA = .080, IC 95% [.64, .096]. El análisis de consistencia interna encontró un α = .91 para la escala total, α = .91 para el primer factor, "Aprobación de uso", α = .79 para el segundo factor, "Utilidad percibida", y α = .59 para el tercer factor, "Riesgo percibido".

Discusión y conclusión

La evaluación de los factores asociados a una mayor aceptabilidad es una herramienta potencial para mejorar la concienciación sobre el uso de intervenciones psicológicas en línea.

Palabras clave: Psicoterapia en línea; aceptabilidad; internet; usuarios de servicios de salud mental; salud mental electrónica

INTRODUCTION

The use of online resources to provide and administer health care services is known as electronic health, or eHealth (World Health Organization, 2016). This technology seeks to provide the patient, health care service user, or client with greater responsibility, power, and information, helping them to take an active role in making decisions about their health (Scheibner et al., 2021). eHealth also contributes to greater efficiency and effectiveness and enhances the interaction between the patient and the primary and secondary health care provider (Abolade & Durosinmi, 2018). In mental health, various eHealth strategies are utilized in the design, evaluation, and practical implementation of internet interventions. The use of eHealth in mental health is known as eMental Health (Blankers, 2011).

A range of internet-based psychological interventions (IPIs) exist. These include telephone calls, videoconferencing, text messaging for cognitive behavioral interventions, text messages or applications that send reminders, automated messages with general and personalized information, interventions through social media, virtual reality, and gaming. They provide services that include psychoeducation or more complex interventions (Mohr et al., 2014).

Although most published studies on the use of the electronic strategies offered in mental health have focused on evaluating their efficiency and effectiveness, an additional area of interest is the identification of factors that enhance their adoption by patients, such as acceptability (Musiat et al., 2014; Simon et al., 2019; Sobowale et al., 2016).

The concept of acceptability is heterogeneous and can be confused with terms such as treatment satisfaction, engagement, usability, and feasibility (Ellis & Anderson, 2023; Ng et al., 2019), which can hamper its evaluation (Ellis & Anderson, 2023). For this research, acceptability refers to attitudes or beliefs about the use of this type of intervention (Molloy & Anderson, 2021; Schröder et al., 2015).

Evaluating the acceptability of IPIs is essential since it can influence adherence and intervention results (Santana & Fontenelle, 2011).

There are two types of procedures for developing assessment instruments to study acceptability as a construct: 1) acceptability assessment of different types of ICTs without a theoretical basis, and 2) theory-based acceptability assessment.

In regard to the first group, in 2010, Banna et al. (2010) explored perceptions about eHealth among the general population in Australia, finding that accessibility was the most important advantage, since it enables people to make decisions about their health. Dinesen et al. (2013) investigated patients with chronic obstructive pulmonary disease (COPD) to evaluate their attitudes toward telerehabilitation in the Danish TELEKAT project, finding four types of attitudes: indifference, learning, feelings of security, and motivation to engage in physical training. Lee et al. (2014) evaluated the attitudes and preferences of older adults in the United States undergoing warfarin treatment about the use of mHealth technology and health games for acquiring self-control skills. Their findings indicated that study participants believed that mHealth could be useful for managing medication, and that they needed help from their families in using technological devices. Simon et al. (2009) evaluated the opinions and preferences of patients in the United States concerning the electronic provision of health information. They found that patients were enthusiastic about the process, recognizing its capacity for improving the safety and quality of health care, although they also expressed concern about privacy and improper use of their data.

Kok et al. (2014) found that users in the Netherlands rated this type of intervention as acceptable in terms of difficulty, time spent on each module, and usefulness. In a study of the general population in England, Musiat et al. (2014) assessed the acceptability of different types of mental health care: 1) in-person therapy, 2) self-help books, 3) eMental Health (internet-based interventions), and 4) mMental Health (smartphone apps). They evaluated a variety of criteria, including whether they helped solve the problem, provided motivation to improve, were credible, accessible without waiting periods, and available at convenient times, at no cost, and in convenient locations. They also explored whether they could be used anonymously, whether they included individual support, provided feedback, and were adaptable to individual learning styles. They found that participants did not believe that computerized treatment or mental health apps met these criteria, except for accessibility, and that they were unlikely to use computerized treatments for mental health in the future.

Regarding the use of videoconferencing to provide psychotherapy, Morland et al. (2015) evaluated the acceptability of this resource among women with PTSD using the Telemedicine Satisfaction and Acceptability Scale (TSAS, Frueh et al., 2005). The results indicated that participants were satisfied with the intervention and would recommend it to family and friends. Olden et al. (2017) also administered the TSAS to patients with post-traumatic stress disorder receiving the intervention. They found high satisfaction with and acceptability of the clinical interaction in the videoconferencing intervention. These studies provided valuable information for understanding the acceptability of the electronic strategies offered in mental health but did not use the theory and models for the variables associated with information and communication technologies (ICT) in health care. These include the theory of reasoned action (Fishbein & Ajzen, 1980), the technology acceptance model (Davis, 1989), the extended technology acceptance model (Venkatesh & Davis, 2000), the innovation diffusion model (Rogers, 1963), and the unified theory of acceptance and use of technology (UTAUT; Venkatesh et al., 2003), as described in Table 1.

Table 1  Variables Associated with ITC Use in Health Care  

Perceived
usefulness
Degree to which a person believes that use
of a particular system would improve their
health (Davis, 1989).
Perceived risk Degree to which a person believes that use
of a particular system can affect their per-
formance, finances, time, and privacy in the
health care they receive (Pavlou, 2003).
Compatibility Degree to which a person likes to use internet
services for various purposes (Pavlou,2003).
Expectation
of functioning
Degree to which a person believes that the use
of a particular system will be advantageous for
their health care (Venkatesh et al., 2003).
Expectation of effort Ease of use associated with a particular sys-
tem for mental health care (Venkatesh et al.,
2003).
Social factors Degree to which a person perceives that
other people important to them believe they
should use a system (Venkatesh et al., 2003).
Facilitating
conditions
Degree to which a person believes that orga-
nizational structure and technical infrastructure
can provide support for the use of a system.

Some studies have used this theory. Jung and Loria (2010) investigated the acceptance of eHealth services among seniors in Sweden using the technology acceptance model (TAM), finding that 1) usefulness, perceived ease of use, compatibility of services with user needs, and trust in the service provider are the major determinants of intention to use the service, and 2) most of those interviewed expressed positive attitudes toward the use of eHealth services, finding them useful, convenient, and easy to use. In an evaluation of the acceptability of online therapy for depression in Indonesia, Arjadi et al. (2018) also used the TAM, find that people were open to using this type of intervention, and that personal acceptance of online services, understood as the degree of individual predisposition or an attitude reflecting the tendency to experiment with mobile health care technologies, regardless of the experience reported by others, is the strongest factor predicting use.

Zhang et al. (2015) used Rogers’ innovation diffusion model in a study of factors influencing the acceptance and use of electronic resources for health care among primary care patients in Australia. Their main findings included an association between a low adoption rate and the inability to use the service to make an online appointment, the preference of most patients to use the telephone, the incompatibility of the new service with patients’ preference for oral communication with a receptionist, and patient characteristics such as unfamiliarity with the internet and lack of experience with online health care services. Lamela et al. (2020) used the UTAUT to evaluate the acceptability of IPIs for depression among the Portuguese population, finding that the expectation of efficiency, social influence, and the stigmatization of depression were significantly associated with the acceptance of this type of intervention.

Many studies on the acceptability of IPIs have been conducted internationally. Among the Mexican population, exploratory and descriptive analyses have been conducted. Although they constitute an initial approach to the topic (Lara et al., 2022), they are not based on a specific theoretical framework that would provide a more in-depth understanding of acceptability or its relationship with other variables predicting the adoption of IPIs in the general population.

The literature review identified studies focused on assessing the acceptability of psychological service provision through technological tools that included items without a theoretical foundation to support their psychometric properties. It was not possible to locate psychometric instruments developed in Mexico to assess IPI acceptability, the study of which could benefit the implementation of IPIs for people with limited access to in-person services.

The purpose of the present study was to evaluate the acceptability of IPIs, with a focus on the construction and assessment of the psychometric properties of theory-based instruments. It was conducted in three stages. The first involved the design of items for the instrument based on the concepts of the TAM. The second focused on evaluating psychometric properties and exploring the dimensions of the instrument based on an exploratory factor analysis. The third used a confirmatory factor analysis to determine the factor structure of the instrument.

METHOD

Phase 1. Construction of items for the instrument

Design of the study

An exploratory-descriptive study was conducted to identify the key words and phrases for an instrument to assess the acceptability of IPI use in Mexico.

Participants

A convenience sample was formed of undergraduate students aged 18 years and over requiring psychological care.

Measurements

Six open, self-report questions were designed and administered based on 1) beliefs, 2) attitudes, and 3) subjective norms concerning the use of psychological interventions through information technologies. These included the following: “List the advantages of having access to a psychological service based on information technologies,” “List the negative adjectives associated with having access to a psychological service based on information technologies,” and “Describe who would disapprove if you used a psychological service based on information technologies.”

Procedure

Prospective participants were contacted at various faculties at a public university, informed of the objectives of the study, and told that participation was voluntary. The process took an average of ten minutes, and data collection was face to face.

Data analysis

A content analysis was conducted of participants’ responses (Neuendorf, 2019).

Phase 2. Analysis of Psychometric Properties and Exploratory Factor Analysis (EFA)

Study Design

A non-experimental, descriptive, exploratory analysis was conducted to evaluate the internal consistency and validity of an instrument to determine the acceptability of use of internet-based psychological interventions in Mexico.

Participants

A convenience sample was formed of university students contacted at various faculties in a public university.

Procedure

Prospective participants were contacted in person and told of the objectives of the project. Those who were interested filled in the informed consent form and subsequently completed the questionnaire. The process took an average of twenty minutes, and data collection was in-person and conducted on the university premises.

Measurements

  • Sociodemographic data questionnaire: four items about sex, age, academic major, and year students were enrolled in at university.

  • Questionnaire on the acceptance of internet-based psychological interventions: forty-two items with responses on a five-point Likert scale, where 1 is “completely disagree” and 5 is “completely agree.” The following definition of psychological intervention on the Internet was given at the beginning of the questionnaire: Psychological treatment that is not provided in a traditional face-to-face setting and instead uses information and communication technologies, specifically the Internet, with tools such as chat, video calls (Zoom, Skype) and self-help programs (interactive activities integrated in a program) with and without the intervention of a health professional.

Statistical analysis

Frequency, asymmetry, discrimination and item directionality were analyzed using Cronbach’s alpha, and exploratory factor analysis (EFA), Bartlett’s test of sphericity and the Kaiser-Meyer Olkin (KMO) test of sampling adequacy. Factor analysis was performed with the extraction of principal components and varimax rotation; items with factor loads of less than 0.4 or more than 0.4 in two or more factors were eliminated.

Phase 3. Confirmatory Factor Analysis

Design of the study

A non-experimental, descriptive, exploratory study was conducted to confirm the factor structure of the instrument.

Participants

A convenience sample was drawn from mental health service users contacted through universities in Mexico City.

Measurements

  • Sociodemographic questionnaire: twelve items on age, sex, educational attainment, profession, and residence.

  • Internet use questionnaire: sixteen questions used in other studies, exploring internet use, place of connection, ease of access, and activities on the internet. These included “Do you know how to use the internet?”, “Do you use the internet regularly?”, “Where do you connect to the internet?”, and “How often do you do the following activities on the internet?” (Tiburcio et al., 2018).

  • Questionnaire on the acceptability of internet-based psychological interventions: fourteen items with responses on a five-point Likert scale where 1 is “completely disagree” and 5 is “completely agree.” The following definition of psychological intervention on the Internet was given at the beginning of the questionnaire: Psychological treatment not conducted in a traditional face-to-face setting but through the use of information and communication technologies, specifically the Internet, using tools such as chats, video calls (Zoom, Skype) and self-help programs (interactive activities incorporated into a program) with and without the intervention of a health professional.

Procedure

Prospective participants were contacted in person and informed of the objectives of the project. Those who were interested answered the informed consent form and subsequently completed the questionnaire. The process took an average of twenty minutes. The data collection was face-to-face and conducted on university premises.

Statistical analysis

The confirmatory factor analysis (CFA) used the maximum likelihood estimation method. The goodness of fit indicators were: 1) χ² with degrees of freedom ≤ 5 for acceptable fit, ≤ 3 for perfect fit; root mean square error of approximation (RMSEA) ≤ .10 for weak fit, ≤ .08 for good fit, ≤ .05 for perfect fit; comparative fit index (CFI) ≥ .90 for acceptable fit, ≥ .95 for good fit, ≥ .97 for perfect fit; and Tucker Lewis index (TLI/NNFI) ≥ .90 for acceptable fit, ≥ .95 for good fit (Hu & Bentler, 1999). Internal consistency overall and for each factor were analyzed using Cronbach’s alpha. Data analysis was performed with Stata.

Ethical considerations

The study protocol was approved by the Committee on Research Ethics of the Ramón de la Fuente Muñiz National Institute of Psychiatry (CEI/C/015/2015).

RESULTS

Phase one

The sample included fifty participants: twenty-five female and twenty-five male students majoring in range of subjects at a public university. Their answers were grouped into four categories: 1) usefulness, 2) obstacles to use, 3) risks, and 4) approval of use (Table 2). Based on this analysis, forty-two items were drafted with responses on a five-point Likert scale, in an initial version of a questionnaire on the acceptability of online psychological interventions.

Table 2  Content analysis for the creation of the instrument to evaluate university students’ attitudes to the use of online interventions  

Categories
Usefulness Obstacles to use Perceived risk Approval of use
Saves money
Access
Convenience
Speed
Time-saving
Adapts to a person’s schedule
Easy
Communication when necessary
Innovative
Privacy
Inexpensive
Interesting
Effective
Practicality
Frequent internet use
Lack of electronic resources
Technical problems
Lack of ability to use new technologies
Impersonal service
Lack of interaction
Misuse of information
Lack of security
Virtual
Anxiety
Not useful
Unreliable
Doubts about professionalism
Dishonesty
Lack of commitment
Lack of available information
Incomplete
Poor communication
Lack of guarantees of confidentiality
of information provided
Self
Friends
Family
Physicians
Psychologist
Partner
Parents
Young people
Traditionalists

Phase two

The sample comprised 223 participants, 50.9% of whom were women ages18-26. The largest proportion were 20 years old (21.9%), followed by those ages 21 (21%) and 19 (17.4%). Participants were pursuing undergraduate degrees in law (15.6%), chemistry (12.1%), engineering (11%), education (6.7%), medicine (5.8%), biopharmaceutical chemistry (5.8%), architecture (5.4%), and other fields (37.4%).

Table 3 shows that the different response options were used in the forty-two items in the instrument. The asymmetry analysis found that nine items showed a typical bias and that thirty-three had a normal distribution. Eight items (10, 13, 18, 24, 30, 36, 37, 38) were eliminated based on the discriminant analysis because no significant difference was found in the group means. The directionality analysis found a clear trend in thirty-four items. Three items (11, 21, and 35) were eliminated based on the initial analysis of internal consistency.

Table 3  Evaluation of psychometric properties of items  

Item Use of
response options
Bias t/discriminant Direction Decision
1. I am interested in using online interventions. -.006 Eliminated in EFA
2. I believe the functions of online psychological interventions are useful for my family. .131 Factor 1
3. I believe my friends would agree with my using an online psychological service. .054 Eliminated in EFA
4. I believe that accessing an online psychological service from anywhere is an advan-
tage.
-.288 Factor 2
5. I believe online interventions are private. .250 Eliminated in EFA (indicator)
6. If someone in my family had a personal problem, I would probably suggest they used
online therapy.
.281 Factor 1
7. I believe communication in online psychological interventions is poor. -.013 Factor 3
8. I believe my partner would agree with my using an online psychological service. .128 Eliminated in EFA
9. I think it is easy to use online psychological interventions. -.357 Factor 2
10. Technical problems could interrupt internet-based psychological services. .792 Eliminated in t-test
11. I am concerned that online psychological interventions are unsafe for my family. .384 Eliminated in total correlation
of corrected elements
12. My partner would use online psychological services. .341 Eliminated in EFA
13. I use the internet frequently. -2.366 Eliminated in t-test
14. I believe that using the internet provides useful information for improving mental
health.
-.217 Eliminated in t-test
15. My parents would use internet-based psychological services. .825 Eliminated in EFA (indicator)
16. I think the use of online interventions is innovative. -.612 Factor 2
17. I believe it would be advisable to use an online psychological intervention. -.187 Eliminated in factor analysis
18. I would feel anxious using an online psychological service. -.343 Eliminated in t-test
19. My family members would use internet-based psychological services. .492 Eliminated in EFA (indicator)
20. I believe the confidentiality of the information provided in online interventions is
guaranteed.
.184 Eliminated in EFA (indicator)
21. If my friends knew I used online psychological interventions, they would disapprove. -.505 Eliminated in item correlation –
total less than 0.2
22. I believe online psychological interventions provide quality service. .139 Factor 1
23. I think internet-based psychological services are incomplete. .160 Factor 3
24. I believe there is limited commitment among people who use online interventions. .120 Eliminated in t-test
25. My friends would use internet-based psychological services. .281 Factor 1
26. I believe the functions of online psychological interventions are useful for my friends. .082 Factor 1
27. I believe my parents would approve of my using an online psychological intervention. -.136 Eliminated in EFA (indicator)
28. I would feel unsafe using an online psychological service. -.064 Factor 3
29. If a friend had a personal problem, I would probably recommend online therapy. .050 Factor 1
30. My friends often use the internet. -2.367 Eliminated in t-test
31. I believe my family would agree with my using an online psychological service. -.003 Eliminated in factor analysis
32. Using online psychological services is easy for anyone. -.011 Eliminated in EFA
33. I will certainly use internet services for my mental well-being. -.073 Factor 1
34. I would trust these types of online psychological support with my personal problems. .345 Factor 1
35. I have doubts about the professionalism of therapists who provide internet-based
services.
-.079 Eliminated in item correlation –
total less than 0.2
36. Online psychological interventions are accessible to most of the population. .401 Eliminated in t-test
37. I am concerned that online psychological interventions are unsafe for my friends. .105 Eliminated in t-test
38. My family uses the internet frequently. -.671 Eliminated in t-test
39. I think that in online psychological interventions there is a lack of honesty in the
relationship between therapists and clients.
-.006 Eliminated in EFA
40. I believe the use of internet-based psychological interventions will allow me to adjust
my schedule to access the service.
-.671 Eliminated in EFA
41. If my family knew I used online psychological interventions, they would disapprove. -.504 Eliminated in EFA
42. I believe the functions of online psychological interventions are useful for me. -.037 Eliminated in EFA

The test of sampling adequacy found thirty-one items with values within the parameters, KMO = .90 and test of sphericity 3365.15 (df = 465, p =.001). Exploratory factor analysis found that seven components explained 51.6% of the variance, with eleven items with factor loads of less than .40 or showing loads in more than one factor being eliminated (1, 3, 8, 12, 14, 17, 32, 39, 40, 41, 42). Six items (5, 15, 19, 20, 27, 31) were eliminated that were grouped in a way that did not constitute a factor (Table 2). The reliability test of the fourteen items on the final scale showed an alpha of .89. The alpha for each of the factors was .89 for factor 1, .62 for factor 2, and .53 for factor 3 (Table 4).

Table 4  Exploratory factor analysis  

Factor 1
Approval
of use
Factor 2
Perceived
usefulness
Factor 3
Perceived
risk
2. I believe the functions of online psychological interventions are useful for my family (p. 8) .441
6. If someone in my family had a personal problem, I would probably recommend they used
online therapy (p. 4)
.448
22. I believe online psychological interventions provide quality service (p. 12) .562
25. My friends would use internet-based psychological services (p. 14) .453
26. I believe the functions of online psychological interventions are useful for my friends (p. 15) .596
29. If a friend had a personal problem, I would probably recommend online therapy (p. 17) .654
33. I will certainly use internet services for my mental well-being (p. 19) .581
34. I would trust these types of psychological support with my personal problems (p. 2) .745
4. I believe accessing an online psychological service from anywhere is an advantage (p. 20) .557
9. I think it is easy to use online psychological interventions (p. 6) .501
16. I think the use of online interventions is innovative (p. 9) .594
7. I believe the communication in online psychological interventions is poor (p. 5) .464
23. I think internet-based psychological services are incomplete (p. 13) .624
28. I would feel unsafe using an online psychological service (p. 16) .446
Alpha .890 .620 .530
Total Alpha .890

Phase three

The sample comprised 201 participants, of whom 51.2% were male, 73.1% lived in Mexico City, and 78.1% were single. The largest proportion were students (48.8%), 56.7% of whom were undergraduate and 21.1% high school students. The results of the CFA showed the following measures of goodness of fit: χ² / df = 168.92/74 = 2.28, CFI = .935, TLI = .920, and RMSEA = .080 (95% CI [.064, .096]). The analysis of internal consistency showed an alpha of .91 overall, .91 for factor 1, “approval of use”; .79 for factor 2, “perceived usefulness”; and .59 for factor 3, “perceived risk” (Figure 1).

Figure 1 Confirmatory factor structure. 

DISCUSSION AND CONCLUSION

This study constructed and evaluated the psychometric properties of an instrument for exploring the acceptability of internet-based psychotherapy to users of mental health services. The results showed that the instrument has a good index of reliability (.86) and an internal structure of three factors: approval of use, perceived usefulness, and perceived risk. These are factors in the acceptance of technology model designed by Davis (1989), which has been extensively applied to predict the use of new technologies in different areas, including psychological care. In addition, it contributes to the importance mentioned in the introduction of having theory-based psychometric instruments to support their factor structure.

This study is one of the first in Mexico to provide a means of exploring the acceptability of IPIs. Acceptability assessment would make it possible to identify those who would adhere better to their treatment. It could also serve as an approach for providing psychoeducation to patients who perceive IPIs as risky so that they can benefit from them.

Developing an instrument with these characteristics is relevant as IPI use increased sharply as a result of the confinement measures due to the COVID-19 pandemic. This radical shift in psychological care provision from in-person to online was observed both in Mexico and internationally (de la Rosa-Gómez & Waldherr, 2023). This change has been maintained in several spheres of private consultation where tools such as Zoom, Skype, and WhatsApp are regularly used as a result of the multiple benefits offered by online psychotherapy. This includes increased privacy and decreased stigma towards psychological care, schedule flexibility, low costs, caution regarding therapist-patient proximity, better access for patients in remote areas to psychological care centers (Rojas-Jara et al., 2022).

Additionally, in Mexico, the implementation of online interventions is expected to expand in comparison with traditional interventions, since there are various online programs designed to treat mental health problems, such as the online self-help program for alcohol use (Schaub et al., 2021), the self-help program for drug use (Tiburcio et al., 2018), the self-help program for depression, the online self-help program to address the emotional health of adolescents in the pandemic (de la Rosa-Gómez et al., 2020), intervention for adults who lost a loved one to COVID-19 (Dominguez-Rodriguez et al., 2023); and psychological assistance via chats from a mental health platform in regard to COVID-19 (Arenas-Landgrave et al., 2022).

Moreover, Internet use is steadily increasing in Mexico. In 2016, the Mexican Internet Association reported that there were approximately 65 million internet users, increasing to 79 million by 2018, an annual growth rate of 17.5% vs. 12% respectively. Connection time also increased during this period, from seven hours and fourteen minutes a day to eight hours and twelve minutes. In both years, the principal online activities included social networks (79% vs. 89%), sending and receiving emails (70% vs. 84%), sending and receiving messages (68% vs. 83%), and searching the internet for information (64% vs. 82%; AMIPCI, 2016 ; 2018).

We recommend additional research to enable the identification of psychosocial and clinical variables that will predict greater acceptability, so that this instrument can serve as a screening tool. Further research should also complement the development of instruments for the evaluation of other ICTs, such as the use of WhatsApp as a means of communication between therapists and patients via messages or the use of mobile applications to monitor patients’ moods.

It is important to note that this study has limitations. Firstly, participants were not randomly selected. Some of them were already familiar with traditional psychological care and given that this study was conducted prior to the pandemic, some were unfamiliar with internet-based psychotherapy. Strengths of the study include the fact that it was based on the conditions of the Mexican population, and that its various phases were sufficient to obtain information on its psychometric properties.

It is important to note that the sample used to design the instrument primarily comprised college students, which affects the generalization of the results. In this respect, it would be useful to test the factor structure with other populations that could benefit from IPIs, such as the elderly.

This study found evidence for the validity and reliability of th questionnaire on the acceptability of online psychotherapy, one of the first Spanish-language psychometric instruments for evaluating this construct. Use of this instrument could yield multiple benefits, such as improving adherence to this type of intervention and increasing awareness among potential mental health service users of the possibilities of online psychotherapy.

Acknowledgements

The authors would like to thank the mental health care institutions where the study was conducted for the facilities provided, particularly the mental health care users who agreed to participate in the study.

REFERENCES

Abolade, T. O., & Durosinmi, A. E. (2018). The Benefits and Challenges of E-Health Applications in Developing Nations: A Review. In Proceedings of the 14th ISTEAMS International Multidisciplinary Conference. Ilorin, Nigeria: AlHikmah University. [ Links ]

Arenas-Landgrave, P., de la Rosa-Gómez, A., Carreón-Martínez, A. E., Esquivel-González, D., Martínez-Luna, S. C., Hernández-Aguirre, O., Olivares-Avila, S. M., Plata-Ochoa, A.Y., González-Santiago, E., & Domínguez-Rodríguez, A. (2022). Atención psicológica vía chat desde una plataforma de salud mental ante la COVID-19. Revista de Investigación en Psicología, 25(2), 185-202. doi: 10.15381/rinvp.v25i2.22916 [ Links ]

Arjadi, R., Nauta, M. H., & Bockting, C. L. (2018). Acceptability of internet-based interventions for depression in Indonesia. Internet Interventions, 13, 8-15. doi: 10.1016/j.invent.2018.04.004 [ Links ]

Asociación Mexicana de Internet [AMIPCI]. (2016). 12 – Estudio sobre los Hábitos de los usuarios de internet en México 2016. Retrieved from https://www.asociaciondeinternet.mx/estudios/habitos-de-internetLinks ]

Asociación Mexicana de Internet [AMIPCI]. (2018). 14 – Estudio sobre los Hábitos de los usuarios de internet en México 2018. Retrieved from https://www.asociaciondeinternet.mx/estudios/habitos-de-internetLinks ]

Banna, S., Hasan, H., & Meloche, J. (2010). A subjective evaluation of attitudes towards E-health. The 2010 International Conference on Innovation and Management. Taiwan: EBRC. [ Links ]

Blankers, M. (2011). E-Mental Health Interventions for Harmful Alcohol Use: Research Methods and Outcomes [Doctoral Dissertation]. Amsterdam: University of Amsterdam. [ Links ]

Davis, F. (1989). Perceived Usefulness, Perceived Ease of use and User Acepptance of Information Technology. MIS Quarterly, 319-340. doi: 10.2307/249008 [ Links ]

de la Rosa-Gómez, A., & Waldherr, K. (2023). Highlights in digital mental health 2021/22. Frontiers in Digital Health, 4, 1093375. doi: 10.3389/fdgth.2022.1093375 [ Links ]

de la Rosa-Gómez, A., Moreyra, L., & De la Rosa-Montealvo, N. (2020). Intervenciones eficaces vía internet para la salud emocional en adolescentes: una propuesta ante la pandemia por COVID-19. Hamut´ay, 7(2), 18-33. doi: 10.21503/hamu.v7i2.2128 [ Links ]

Dinesen, B., Huniche, L., & Toft, E. (2013) Attitudes of COPD patients towards tele-rehabilitation: across-sector case study. International Journal of Environmental Research and Public Health, 10(11), 6184-98. doi: 10.3390/ijerph10116184 [ Links ]

Dominguez-Rodriguez, A., Sanz-Gomez, S., González Ramírez, L. P., Herdoiza-Arroyo, P. E., Trevino Garcia, L. E., de la Rosa-Gómez, A., González-Cantero, J. O., Macias-Aguinaga, V., & Miaja, M. (2023). The Efficacy and Usability of an Unguided Web-Based Grief Intervention for Adults Who Lost a Loved One During the COVID-19 Pandemic: Randomized Controlled Trial. Journal of Medical Internet Research, 25, e43839. doi: 10.2196/43839 [ Links ]

Ellis, D. M., & Anderson, P. L. (2023). Validation of the Attitudes Towards Psychological Online Interventions Questionnaire Among Black Americans: Cross-cultural Confirmatory Factor Analysis. JMIR Mental Health, 10, e43929. doi: 10.2196/43929 [ Links ]

Fishbein M., & Ajzen, I. (1980). Understanding attitudes and predicting social behavior. Londres: Prentice Hall International. [ Links ]

Frueh, B. C., Henderson, S., & Myrick, H. (2005). Telehealth service delivery for persons with alcoholism. Journal of Telemedicine and Telecare, 11(7), 372-375. doi: 10.1258/135763305774472060 [ Links ]

Hu, L. T., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1-55. [ Links ]

Jung, M., & Loria, K. (2010). Acceptance of Swedish e-health services. Journal of Multidisciplinary Healthcare, 3, 55-63. Retrieved from https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3024889/Links ]

Kok, G., Bockting, C., Burger, H., Smit, F., & Riper, H (2014). Mobile Cognitive Therapy: Adherence and acceptability of an online intervention in remitted recurrently depressed patients. Internet Interventions, 1(2), 65-73. doi: 10.1016/j.invent.2014.05.002 [ Links ]

Lamela, D., Cabral, J., Coelho, S., & Jongenelen, I. (2020). Personal stigma, determinants of intention to use technology, and acceptance of internet-based psychological interventions for depression. International Journal of Medical Informatics, 136, 104076. doi: 10.1016/j.ijmedinf.2020.104076 [ Links ]

Lara, M. A., Patiño, P., Tiburcio, M., & Navarrete, L. (2022). Satisfaction and acceptability ratings of a web-based self-help intervention for depression: retrospective cross-sectional study from a resource-limited country. JMIR Formative Research, 6(4), e29566. doi: 10.2196/29566 [ Links ]

Lee, J., Nguyen, A., Berg, A., Amin, A., Bachman, M., Guo, Y., & Evangelista, L. (2014). Attitudes and Preferences on the Use of Mobile Health Technology and Health Games for Self-Managment: Interviews with older adults on anticoagulation therapy. JMIR Mhealth and Uhealth, 2(3), e32. doi: 10.2196/mhealth.3196 [ Links ]

Mohr, D., Schueller, S., Montague, E., Burns, M. N., & Rashidi, P. (2014) The Behavioral Intervention Technology Model: An Integrated Conceptual and Technological Framework for eHealth and mHealth Interventions. Journal of Medical Internet Research, 16(6), 1-15. doi: 10.2196/jmir.3077 [ Links ]

Molloy, A., & Anderson, P. L. (2021). Increasing Acceptability and Outcome Expectancy for Internet-Based Cognitive Behavioral Therapy During the COVID-19 Pandemic. Telemedicine and e-Health, 28(6), 888-895. doi: 10.1089/tmj.2021.0393 [ Links ]

Morland, L. A., Mackintosh, M. A., Rosen, C. S., Willis, E., Resick, P., Chard, K., & Frueh, B. C. (2015). Telemedicine versus in‐person delivery of cognitive processing therapy for women with posttraumatic stress disorder: A randomized noninferiority trial. Depression and Anxiety, 32(11), 811-820. doi: 10.1002/da.22397 [ Links ]

Musiat, P., Goldstone, P., & Tarrier, N. (2014). Understanding the acceptability of e-mental health:attitudes and expectations towards computerised self-help treatments for mental health problems. BMC Psychiatry, 14(1), 1-8. doi: 10.1186/1471-244X-14-109 [ Links ]

Neuendorf, K. A. (2019). Content analysis and thematic analysis. In P. Brough (Ed.). Advanced, research methods for applied psychology: Design, analysis and reporting (pp. 211-223). New York: Routledge. [ Links ]

Ng, M. M., Firth, J., Minen, M., & Torous, J. (2019). User Engagement in Mental Health Apps: A Review of Measurement, Reporting, and Validity. Psychiatric Services, 70(7), 538-544. doi: 10.1176/appi.ps.201800519 [ Links ]

Olden, M., Wyka, K., Cukor, J., Peskin, M., Altemus, M., Lee, F. S., Finkelstein-Fox, L., Rabinowitz, T., & Difede, J. (2017). Pilot study of a telehealth-delivered medication-augmented exposure therapy protocol for PTSD. The Journal of Nervous and Mental Disease, 205(2), 154-160. doi: 10.1097/NMD.0000000000000563 [ Links ]

Pavlou, P. (2003). Consumer Acceptance of Electronic Commerce: Integrating Trust and Risk with the Technology Acceptance Model. International Journal of Electronic Commerce, 7(3), 101-134. doi: 10.1080/10864415.2003.11044275 [ Links ]

Rogers, C. R. (1963). Toward a science of the person. Journal of Humanistic Psychology, 3(2), 72-92. [ Links ]

Rojas-Jara, C., Polanco-Carrasco, R., Caycho-Rodríguez, T., Muñoz-Vega, C., Muñoz-Marabolí, M., Luna-Gómez, T., & Muñoz-Torres, T. (2022). Telepsicología para psicoterapeutas: lecciones aprendidas en tiempos del Covid-19. Revista Interamericana de Psicología/Interamerican Journal of Psychology, 56(2), e1733. doi: 10.30849/ripijp.v56i2.1733 [ Links ]

Santana, L., & Fontenelle, L.F.(2011). A review of studies concerning treatment adherence of patients with anxiety disorders. Patient Preference and Adherence, 5, 427-439. doi: 10.2147/PPA.S23439 [ Links ]

Schaub, M. P., Tiburcio, M., Martínez-Vélez, N., Ambekar, A., Bhad, R., Wenger, A., Baumgartner, C., Padruchny, D., Osipchik, S., Poznyak, V., Rekve, D., Landi Moraes F., Monezi Andrade A. L., Souza-Formigoni, M. L. O., & WHO E-Health Project On Alcohol And Health Investigators Group. (2021). The effectiveness of a web-based self-help program to reduce alcohol use among adults with drinking patterns considered harmful, hazardous, or suggestive of dependence in four low-and middle-income countries: randomized controlled trial. Journal of Medical Internet Research, 23(8), e21686. doi: 10.2196/21686 [ Links ]

Scheibner, J., Sleigh, J., Ienca, M., & Vayena, E. (2021). Benefits, challenges, and contributors to success for national eHealth systems implementation: a scoping review. Journal of the American Medical Informatics Association, 28(9), 2039-2049. doi: 10.1093/jamia/ocab096 [ Links ]

Schröder, J., Sautier, L., Kriston, L., Berger, T., Meyer, B., Späth, C., Köther, U., Nestoriuc, Y., Klein, J. P., & Moritz, S. (2015). Development of a questionnaire measuring Attitudes towards Psychological Online Interventions-the APOI. Journal of Affective Disorders, 187, 136-141. doi: 10.1016/j.jad.2015.08.044 [ Links ]

Simon, N., McGillivray, L., Roberts, N. P., Barawi, K., Lewis, C. E., & Bisson, J. I. (2019). Acceptability of internet-based cognitive behavioural therapy (i-CBT) for post-traumatic stress disorder (PTSD): a systematic review. European Journal of Psychotraumatology, 10(1), 1646092. doi: 10.1080/20008198.2019.1646092 [ Links ]

Simon, S., Evans, S., Benjamin, A., Delano, D. & Bates, D. (2009). Patients´Attitudes Toward Electronic Health Information Exchange: Qualitive Study. Journal of Medical Internet Research, 11(3), 132-139. doi:10.2196/jmir.1164 [ Links ]

Sobowale, K., Nguyen, M., Weiss, B., Van, T. H., & Trung, L. T. (2016). Acceptability of internet interventions for youth mental health in Vietnam. Global Mental Health, 3, e22. doi: 10.1017/gmh.2016.18 [ Links ]

Tiburcio, M., Lara, M., Martínez, N., Fernández, M., & Aguilar, A. (2018) Web-Based Intervention to Reduce Substance Abuse and Depression: A Three Arm Randomized Trial in Mexico. Substance Use & Misuse, 53(13) 2220-2231. doi: 10.1080/10826084.2018.1467452 [ Links ]

Venkatesh, V., & Davis, F. (2000). A Theorical Extension of the Technology Acceptance Model: Four Longitudinal Field Studies. Management Science, 46(2), 186-204. doi: 10.1287/mnsc.46.2.186.11926 [ Links ]

Venkatesh, V., Morris, M., Davis, G., Davis, F. (2003). User acceptance of information technology: toward a unified view. MIS Quarterly, 27(3), 425-478. doi: 10.2307/30036540 [ Links ]

World Health Organization. (2016). Global diffusion of eHealth: making universal health coverage achievable. Report of the third global survey on eHealth. Geneva: World Health Organization. [ Links ]

Zhang, X., Yu, P., Yan, J., & Spil, I. (2015). Using diffusion of innovation theory to understand the factors impacting patient acceptance and use of consumer e-health innovations: a case study in a primary care clinic. BMC Health Services Research, 15, 1-15. doi: 10.1186/s12913-015-0726-2 [ Links ]

Funding None.

Citation: Mondragón Gómez, R., Martínez Vélez, N. A., Tiburcio Sainz, M., & Fernández Torres, M. (2024). Design, reliability, and validity of the acceptability of internet-based psychological interventions questionnaire in Mexican university students. Salud Mental, 47(3), 107-116. DOI: 10.17711/SM.0185-3325.2024.015

Received: February 22, 2023; Accepted: May 22, 2023

Correspondence: Marcela Tiburcio Sainz Calz. México-Xochimilco 101, Col. San Lorenzo Huipulco, Tlalpan, 14370 Ciudad de México, Mexico. Phone: +52 (55) 4160-5162 E-mail: tibsam@imp.edu.mx; mtiburcio3@gmail.com

Conflicts of interest The authors declare they have no conflicts of interest.

Creative Commons License This is an open-access article distributed under the terms of the Creative Commons Attribution License