martes, 4 de mayo de 2021

 Teacher performance evaluation model in COVID19 times

 

1st Olger Gutiérrez Aguilar

Faculty of philosophy and humanities

Universidad Nacional de San Agustín
Arequipa, Perú
https://orcid.org/0000-0002-6657-7529


2nd Aleixandre Brian Duche Pérez

Faculty of Sciences and Social Technologies and Humanities

Universidad Católica de Santa María

Arequipa, Perú

https://orcid.org/0000-0001-9905-1489


3rd Ananí Gutiérrez Aguilar
Faculty of philosophy and humanities
Universidad Nacional de San Agustín

Arequipa, Perú
https://orcid.org/0000-0002-2916-2957
line 5: email address or ORCID

 Abstract— This study purpose was to validate a model for evaluating teacher performance at university level, in the modality of non-attendance education, based on the management of teaching and learning processes with the LMS - Moodle and its integration with the MS Teams, according to the adaptation provided by the National Superintendence of Higher University Education (SUNEDU) in Peru, due to the social distancing declared by the government in Covid19 times. The evaluation model involves first validating the 15-item questionnaire, with 5 factors each one, with 3 items using the 5-value Likert scale, applying the Item Response Theory (IRT) model for graduated response polytomics items. The latent structure of the items was analyzed, applying firstly the exploratory factorial analysis using SPSS, secondly, for the confirmatory analysis the PLS-SEM methodology was used, which is a second generation multivariate method called Structural Equation Modeling with Partial Least Squares. The model was validated with a sample of 358 undergraduate students in 5 subjects of the Professional School of Psychology. It is concluded that, the statistical tests for exploratory and confirmatory analyses, have revealed acceptable results in their assessment, therefore, it is stated that the instrument is reliable and consistent, therefore, its relationship with the variable (NPS), validates the model in its integrity.

Keywords— evaluation model, teaching performance, digital competences, satisfaction index, Covid19, PLS-SEM.

                                                                      I.        Introduction

The decisions taken by the different national governments with respect to minimizing the spread of Covid19 , in the face of a health emergency that several countries have been experiencing, and on the other hand, the need to continue with the educational service provided by universities, has led to processes of adaptation and change of curricula from face-to-face to non-presential modalities, in Peru the Ministry of Education and the National Superintendence of University Higher Education (SUNEDU), have fostered in the universities, processes of implementation of resources with digital technology, using virtual learning environments, developing in turn, teaching skills in the proper management of the Learning Management System (LMS), thus ensuring the continuity of the provision of training service according to the criteria of accessibility, adaptability, quality and other essential conditions for learning.

On the other hand, there is consensus that the quality of the educational service is achieved through the teachers  performance, on the one hand, by adequately managing the learning process of their students, especially those who experience processes of adaptation from secondary education to university [1], and on the other hand the development and implementation of appropriate and relevant strategies for the integration of Information and Communication Technologies (ICT) into educational processes, leading to the development and evaluation of e-learning models for higher education [2].

Evaluating for decision making about teaching performance is a need and a latent concern of the current universities.

Thus, the performance of the university teacher, would be framed in the following dimensions, the Planning of the teacher, which allows adequate management of resources and the time needed before the start of teaching activities, proper management of educational materials and content, and how these are being adapted to learning situations by the Covid19 [3-5], the Didactic Methodology, especially suitable for models of e-learning educational management, the Relationship with the Students, in such a way that it favour favourable environments for learning and the Evaluation and Academic Monitoring of the student, related to aspects of tutoring and accompaniment of the student from synchronous and asynchronous communication networks  [6, 7].

In this particular context of non-face-to-face educational experiences, the use of Web 2.0 concepts for virtual exchanges is beginning to take hold, especially Synchronous Computer-Mediated Communication (SCMC), which allows teachers and students to use video conference applications, [8, 9] to interact face-to-face, tools like Skype [10], Zoom [11], YouTube [12], MS Teams, are of unusual value, added to which is the proper management of the LMS - Moodle [13] which is based on a constructivist approach to education [14-17].

The learning experiences developed through management in virtual learning environments are achieving significant effects, as preceded by other research [18-20], and in particular those related to health, which integrate the Zoom for the recording of learning situations, distinguishing three methods. The first is self-study which involves motivated study by each resident; the second is a pedagogical method which requires independent interpretation by the study resident and the third is a subsequent review with an experienced faculty mentor [21].

To measure or control the satisfaction of the services and especially, the educational ones, is a permanent concern for the educational institutions, especially for the universities, the Net Promoter Score (NPS), is being used by diverse organizations for this purpose, we have the properly business, health and education, it is so that the organizations have incorporated this tool inside their organizational culture and a practice of continuous improvement to establish quality experiences. The NPS is a contribution made by F. F. Reichheld [22, 23]. The NPS is the application of a rating scale, whose values range from 0 - 10, with a key question: "How likely are you to recommend the product or service to a family member or friend" the same F. F. Reichheld, & Markey, R. [24], for the book Spanish version he would title it «La Pregunta Decisiva 2.0», thus asking the user of the service to rate it, where 0 is «Very unlikely» and 10 is «I would definitely recommend it». According to the results, clients are classified into promoters, passive and detractors:

                                                                    II.       Methodology

The sample for the validation of the model was made up by 358 students from the professional school of psychology of the Universidad Católica de Santa María de Arequipa - Peru. Table 1 presents the data. The students were randomly selected from the 5 odd semesters, being 51% male and 49% female, whose ages are between 17 and 23 years old. The total average was 19.86 (SD=2,015). The instrument was implemented in June 2020 in a condition of social isolation and quarantine, as decreed by Peru government with the aim of minimizing Covid19 contagion.

TABLE I.           Sample studied

Subjects of the odd semester 2020

Total

%

Methodology of university work

73

20.39%

Child Developmental Psychology

61

17.04%

Behavioral Neuroscience

56

15.64%

Motivation and emotion

57

15.92%

Oral and written communication

55

15.36%

Introduction to Psychology

56

15.64%

Total

358

100.00%

 

For the construction of the questionnaire, five observable variables were established, these are Planning; Educational materials and contents; Methodology, Relationship with students and Evaluation and academic follow-up, with 3 items, as shown in table 2:

TABLE II.          Questionnaire structure for the teacher performance assessment

Components

Items

Planning (PLA)

3

Educational Materials and Content (MAT)

3

Methodology (MET)

3

Relationship with students (REL)

3

Evaluation and academic monitoring (EVA)

3

Total

15

 

A scale from (1 to 5) is used to fill in the questionnaire, with the value given based on the following criteria: 1 Never; (2) Hardly ever; (3) Sometimes; (4) Almost always and (5) Always. With fully agreed criteria and through an expert evaluation with background research of this nature.

                                                        III.   Results and discussion

As part of the instrument analysis, the reliability statistics of the instrument were processed with the SPSS software, resulting in a Cronbach's alpha of 0.958, which means that it is an excellent measuring instrument. The analysis started with a preliminary analysis of the 15 items by means of factorial analysis that aims to condense the information into a series of original variables to smaller series [25]. To establish the coupling level of the components, the KMO Bartlett's Test of Sphericity were used, which is based on the observed correlation coefficients [26, 27]. The process was carried out using the Principal Component Analysis (ACP) technique, which allows the joint treatment of variables formed from a combination of them, in order to analyze the correlation matrix. The result of the KMO test is 0.958 which means an excellent sample adequacy, as shown in table 3.

TABLE III.        KMO test and Barlett sphericity test

Kaiser-Meyer-Olkin Measurement of Sampling Adequacy

 

0.958

Bartlett's Test of Sphericity

Aprox. Chi-square

4069.925

 

gl

105

 

Sig.

0.000

 

After checking the sample adequacy, the communalities were evaluated, which are the sum of the factor weights squared by each of the rows, these indicate the proportion of the variance explained by the common factors for each variable. From the analysis it is observed that the exogenous latent variables, present communalities that oscillate between 0.471 and 0.708, which means that in the worst case the observed items would explain the model in 47.1% and in the best case they would manage to explain the model in 70.8%. The data are presented in the table 4.

TABLE IV.        Communalities, first test with the SPSS

Items

Initial

Extraction

In the development of the subject, a planning of theoretical and practical activities is evident (PLA_01).

1.000

0.663

The selection of contents for the learning, are the adequate, updated and pertinent. (PLA_02).

1.000

0.596

The teacher demonstrates mastery on the subject matter (PLA_03).

1.000

0.594

There is a variety of educational resources and activities in the implementation of the subjects programmed during the semester (MAT_01).

1.000

0.643

The resources and activities used by the teacher contribute to the achievement of the competencies (learning objectives) (MAT_02).

1.000

0.687

The teacher clearly explains the relationship between educational resources and learning activities (MAT_03).

1.000

0.708

The computer skills at the beginning of the course are clearly established by the teacher (MET_01).

1.000

0.674

The competences of the course, express results that are measurable or achievable (MET_02).

1.000

0.655

The relationship of the educational activities and resources are clearly established in the virtual classroom (MET_03).

1.000

0.641

The teacher's communication plan to interact with the students in the course is clearly established (Teams, Chat, Whatsapp, E-mail and others) (REL_01).

1.000

0.505

The communication you establish with your teacher is satisfactory (REL_02).

1.000

0.554

The dialogue and the participation of the students are encouraged (REL_03).

1.000

0.600

The teacher, besides the exams, considers other possibilities of evaluation of the learning (oral, practical work, portfolios, projects, etc.) (EVA_01).

1.000

0.471

The evaluations used are sequenced (of process), varied, and adequate for the level of the subject (EVA_02).

1.000

0.644

The evaluation of the teacher is objective (EVA_03).

1.000

0.576

Extraction method: principal component analysis

 

The total variance explained must be greater than 0.50 to be considered acceptable, therefore, percentages of 0.70 or more are desirable [28-30], being 1.00 the perfect reliability, in this sense, the results obtained from the 5 components, would explain the model by 79.198%.

The analysis was carried out for each of the factors, in terms of the factor: Planning, the Kaiser-Meyer-Olkin test - KMO, determined a value of 0.721, which is equivalent to an acceptable sample suitability. Table 5 explains the results of the component matrix.

TABLE V.          Planning component matrix  (PLA)

Component

1

In the development of the subject, a planning of theoretical and practical activities is evident.

0.887

The selection of contents for learning, are adequate, updated and relevant.

0.869

The teacher demonstrates mastery of the subject matter.

0.851

Main Component Extraction Method

 

The second factor, educational materials and content, showed a KMO measure of sampling adequacy of 0.725, which is equivalent to acceptable.

TABLE VI.        Component matrix of educational materials and content (MAT)

Component

1

The teacher clearly explains the relationship between educational resources and learning activities.

0.903

The resources and activities used by the teacher contribute to the achievement of competencies (learning objectives).

0.893

There is a variety of educational resources and activities in the implementation of the subjects programmed during the semester.

0.856

Main Component Extraction Method

 

The third analysis factor: Relationship to students, KMO test results, revealed a value of 0.727, which is equivalent to an acceptable sample fit. The results of the component matrix are presented in Table 7.

TABLE VII.       Relationship with students component matrix (REL)

Component

1

The communication you establish with your teacher is satisfactory.

0.885

Dialogue and student participation are encouraged.

0.872

The teacher's communication plan for interacting with students on the course is clearly established (Teams, Chat, Whatsapp, Email and others).

0.863

Main Component Extraction Method

 

The fourth factor, methodology, showed a KMO measure of 0.731, which is equivalent to an acceptable sample adequacy. The results of the component matrix are presented in Table 8.

TABLE VIII.      Methodology component matrix (MET)

Component

1

The competencies of the subject, express results that are measurable or achievable

0.876

The list of educational activities and resources are clearly established in the virtual classroom.

0.875

The computer skills at the beginning of the course are clearly established by the teacher

0.873

Main Component Extraction Method

 

The fifth factor: Evaluation and academic monitoring, which gives a KMO measure of sampling adequacy of 0.714 which is equivalent to an acceptable adequacy; the component matrix, is detailed in table 9.

TABLE IX.        Evaluation and academic monitoring component matrix (EVA)

Component

1

The teacher, in addition to the exams, considers other possibilities of evaluation of the learning (oral, practical work, portfolios, projects, etc.).

0.850

The teacher's evaluation is objective.

0.849

The evaluations used are sequential (process), varied, and appropriate to the level of the subject

0.847

Main Component Extraction Method

 

The preliminary analysis, called exploratory, allowed us to obtain the necessary inputs to proceed with the confirmatory analysis, for which we used SmartPLS (v.3.3.2), which is a tool for the modeling of structural equations, based on the variance that uses the least-squares path modeling method, devised by Christian M. Ringle, Sven Wende and Jan-Michael Becker[31].

PLS-SEM (Partial Least Square Structural Equation Modeling), is a second generation multivariate method called Partial Least Square Structural Equation Modeling [32], which is composed by two elements, the first one proposes a measurement model that tries to analyze the factorial loads of the variables through their items, in such a way that the relationship with their corresponding variables can be established. With this method, the reliability and validity of the dimensions of the theoretically supported model are evaluated. On the other hand, we have the structural model, which analyzes the causal relationships between independent and dependent latent variables, this last variable is the Satisfaction Index - Net Promoter Score NPS.

Figure 1, presents the internal consistency of the model expressed through Cronbach's alpha to establish the degrees of reliability of the indicators, the results range from 0.805 to 0.860, with acceptable values for this from 0.700 to more [33].

 

Fig. 1. Cronbach's Alpha of the SmartPLS model

 

As regards the analysis of the reliability and validity of the model, explained from R2 (Pearson's coefficient) for the dependent variable: satisfaction index - SPL, according to the data obtained R2 has a value of 0.561, which is equivalent to saying that 56.1% of the variance is explained by the model, as can be seen in figure 2.

 

Fig. 2. SmartPLS model R-square

 

The matrix of external loads, explains the measurement model and is composed of the indicators and their trajectories that connect them to their factors, the result represents the contribution of the indicators to the definition of the dependent variable. The values of the external loads could have a variation between 0 and 1, so the closer to the unit they are, the stronger they are to explain the validity of the model.

In table 10, the matrix of external loads is presented with their respective values, these are considered acceptable, and therefore none of the items were discarded.

TABLE X.          External loads matrix – smart PLS

 

(EVA)

(MAT)

(MET)

(PLA)

REL)

NPS

EVA_01

0.834

 

 

 

 

 

EVA_02

0.855

 

 

 

 

 

EVA_03

0.856

 

 

 

 

 

MAT_01

 

0.831

 

 

 

 

MAT_02

 

0.908

 

 

 

 

MAT_03

 

0.91

 

 

 

 

MET_01

 

 

0.868

 

 

 

MET_02

 

 

0.889

 

 

 

MET_03

 

 

0.866

 

 

 

PLA_01

 

 

 

0.898

 

 

PLA_02

 

 

 

0.875

 

 

PLA_03

 

 

 

0.833

 

 

REL_01

 

 

 

 

0.858

 

REL_02

 

 

 

 

0.893

 

REL_03

 

 

 

 

0.868

 

NPS

 

 

 

 

 

1

 

The internal consistency of the items of the latent variables of the instrument (Planning; Educational materials and content; Methodology, Relationship with students and Evaluation and academic follow-up), the Composite Reliability is within the values ranging from 0.885 to 0.914, which is equivalent to satisfactory. Table 11.

TABLE XI.        Construct reliability and validity

 

Cronbach’s alpha

rho_A

Composed Reliability                 

Mean extracted variance (AVE)

Evaluation and academic monitoring

0.805

0.81

0.885

0.719

Educational materials and content

0.86

0.883

0.914

0.781

Methodology

0.846

0.852

0.907

0.765

Planning

0.838

0.851

0.902

0.755

Relationship with students

0.844

0.849

0.906

0.762

Satisfaction Index (NPS)

1

1

1

1

 

Taking into account the correlation coefficients in all its dimensions, the reliability and validity of the construct, correlated with the satisfaction index variable, the 5 factors proposed in the model and that are manifested in the construct are statistically significant. The values of the average variance extracted - Average Variance Extracted - (AVE), obtained with SmartPLS, range from 0.719 to 0.781, table 11. These results exceed the minimum recommended value of 0.50 [34]. The assessment of this analysis allows us to conclude that convergent validity is acceptable in all components of the model. As for composite reliability, for a social science approach, different authors have suggested its application [35, 36], Therefore, if the values are greater than 0.6 then high levels of internal consistency reliability are demonstrated in each of the reflective latent variables of the proposed model. However, the discussion today around the application of PLS, it is suggested instead of using Cronbach's alpha and composite reliability for the analysis, it is recommended to use the coefficient (rho_A) to verify the reliability of the values obtained in the construction and design of the PLS [37], the results obtained in the (rho_A) should have values of 0.7 or higher to demonstrate composite reliability, the results obtained are between 0.81 and 0.883

There are 2 accepted ways to verify the discriminant validity of the model, the first one corresponds to the one proposed by Fornell and Larcker [38], which suggests that the square root of AVE in each variable, its resulting values should be greater than the results of the correlation between the latent variables, the result indicates that the discriminant validity is well established, as shown in table 12.

TABLE XII.       Fornell-Larcker criteria analysis for verification and discriminant validity

EVA

NPS

MAT

MET

PLA

REL

EVA

0.848

NPS

0.648

1

MAT

0.776

0.688

0.884

MET

0.795

0.708

0.825

0.874

PLA

0.727

0.616

0.86

0.788

0.869

REL

0.707

0.658

0.71

0.757

0.674

0.873

 

The second way to verify the discriminant validity of the model is proposed by Henseler, Ringle and Sarstedt [39] based on the heterotrait-monotrait correlation (HTMT), whose essence is the multitrait-multimethod matrix. In an ideal model, the heterotrait correlations should be smaller than the monotrait correlations, therefore, the HTMT ratio should be below 1.0. Table 13.

TABLE XIII.      Heterotrait-monotrait (HTMT) correlations

EVA

NPS

MAT

MET

PLA

REL

EVA

 

 

 

 

 

 

NPS

0.719

 

 

 

 

 

MAT

0.931

0.731

 

 

 

 

MET

0.958

0.766

0.963

 

 

 

PLA

0.881

0.668

1.024

0.936

 

 

REL

0.851

0.714

0.833

0.895

0.801

 

 

Similarly, as part of the proposed model validation process, it was considered necessary to obtain the variance inflation factor (VIF), which will make it possible to avoid the collinearity problem, if the values exceed 5.0 they will indicate collinearity within the indicator [34]. The results of the collinearity test reveal that the rates range from 1,731 to 2,503 and therefore satisfy the criteria, Table 14.

 

TABLE XIV.     Results of the indicator collinearity test

 

VIF

EVA_01

1.749

EVA_02

1.731

EVA_03

1.743

MAT_01

1.907

MAT_02

2.374

MAT_03

2.503

MET_01

2.019

MET_02

2.055

MET_03

2.038

PLA_01

2.163

PLA_02

1.991

PLA_03

1.831

REL_01

1.93

REL_02

2.145

REL_03

2.015

 

                                                                    IV.   Conclusions

The methodology of PLS-SEM, has allowed to evaluate the proposed model, using for this purpose an initial exploratory analysis and later a confirmatory analysis using the SmartPLS, as part of the model, is the validation of the instrument for the evaluation of the teaching performance, key aspect in the quality of the university educational services, which consists on 5 factors, as observed latent variables, like they are Planning (PLA); Educational Materials and Content (MAT); Methodology (MET); Relationship with students (REL); Evaluation and academic follow-up (EVA), as a dependent variable, the Net Promoter Score (NPS).

Statistical tests for exploratory and confirmatory analyses have revealed acceptable results in their assessment, therefore, it is stated that the instrument is reliable and consistent, therefore, its relationship with the variable (NPS), validates the model in its totality.

Finally, it is concluded that, the statistical analysis allowed by the PLS-SEM is a technique of great interest for social research, based on an alternative approach of structural equation modeling with partial least squares.

                                                                    V.    Future works

As future work, the instrument will be applied to university students studying in the virtual modality; the model will also serve to implement a quality system in terms of university teacher performance.

References

[1]    A. B. Duche-Perez, F. M. Paredes-Quispe, and O. A. Gutierrez-Aguilar, "The Transition from high school to university: Identifying internal and external factors for a successful transition in peruvian students of Architecture and Engineering," in EDUNINE 2019 - 3rd IEEE World Engineering Education Conference: Modern Educational Paradigms for Computer and Engineering Career, Proceedings, 2019, doi: 10.1109/EDUNINE.2019.8875751. [Online]. Available: https://www.scopus.com/inward/record.uri?eid=2-s2.0-85074787943&doi=10.1109%2fEDUNINE.2019.8875751&partnerID=40&md5=4ddb2e09ff96cab86778bcd102f446f0

[2]    A. B. Duche Perez, F. M. Paredes Quispe, O. A. Gutierrez Aguilar, and D. Arias Chavez, "Development and evaluation of an e-learning model of teaching-learning in multidisciplinary education subjects in technological higher education," 2019, pp. 359-366, doi: 10.1109/LACLO49268.2019.00067. [Online]. Available: https://www.scopus.com/inward/record.uri?eid=2-s2.0-85081124307&doi=10.1109%2fLACLO49268.2019.00067&partnerID=40&md5=d8e062d5301f250565cbc6431d4a1fc5

[3]    L. C. Warth, N. O. Noiseux, S. T. Duncan, S. B. Daines, and C. R. Mahoney, "How Has COVID-19 Affected Our Orthopedic Implant Industry Partners? Implications for the Surgeon-Industry Relationship in 2020 and Beyond," The Journal of Arthroplasty, vol. 35, no. 7, Supplement, pp. S56-S59.e10, 2020/07/01/ 2020, doi: https://doi.org/10.1016/j.arth.2020.04.063.

[4]    C. Carolan, C. L. Davies, P. Crookes, S. McGhee, and M. Roxburgh, "COVID 19: Disruptive impacts and transformative opportunities in undergraduate nurse education," Nurse Education in Practice, vol. 46, p. 102807, 2020/07/01/ 2020, doi: https://doi.org/10.1016/j.nepr.2020.102807.

[5]    E. E. Lewis, L. J. Taylor, J. L. Hermsen, D. P. McCarthy, and A. G. Fiedler, "Cardiothoracic Education in the Time of COVID-19: How I Teach It," The Annals of Thoracic Surgery, 2020/04/10/ 2020, doi: https://doi.org/10.1016/j.athoracsur.2020.04.002.

[6]    J. Brouwer, C. Downey, and C. Bokhove, "The development of communication networks of pre-service teachers on a school-led and university-led programme of initial teacher education in England," International Journal of Educational Research, vol. 100, p. 101542, 2020/01/01/ 2020, doi: https://doi.org/10.1016/j.ijer.2020.101542.

[7]    S. Rubinelli, K. Myers, M. Rosenbaum, and D. Davis, "Implications of the current COVID-19 pandemic for communication in healthcare," Patient Education and Counseling, vol. 103, no. 6, pp. 1067-1069, 2020/06/01/ 2020, doi: https://doi.org/10.1016/j.pec.2020.04.021.

[8]    C. A. Lenkaitis, "Teacher candidate reflection: Benefits of using a synchronous computer-mediated communication-based virtual exchange," Teaching and Teacher Education, vol. 92, p. 103041, 2020/06/01/ 2020, doi: https://doi.org/10.1016/j.tate.2020.103041.

[9]    D. Healey, "Language learning and technology," The Routledge handbook of language learning and technology, pp. 9-23, 2016.

[10]  N. M. Terhune, "Language learning going global: linking teachers and learners via commercial Skype-based CMC," Computer assisted language learning, vol. 29, no. 6, pp. 1071-1089, 2016.

[11]  C. A. Bohinski and N. Mulé, "Telecollaboration: Participation and negotiation of meaning in synchronous and asynchronous activities," MEXTESOL Journal, vol. 40, no. 3, pp. 1-16, 2016.

[12]  M. Farag, D. Bolton, and N. Lawrentschuk, "Use of YouTube as a Resource for Surgical Education—Clarity or Confusion," European Urology Focus, vol. 6, no. 3, pp. 445-449, 2020/05/15/ 2020, doi: https://doi.org/10.1016/j.euf.2019.09.017.

[13]  J. Cabero Almenara, M. L. Arancibia Muñoz, and A. D. Prete, "Dominio técnico y didáctico del LMS Moodle en Educación Superior: más allá de su uso funcional," 2019.

[14]  L. Vygotsky and J. P. T. Abadía, Pensamiento y lenguaje. Grupo Planeta, 2013.

[15]  J. Piaget, The child's construction of reality. London, 1955.

[16]  J. Bruner, "Aprendizaje por descubrimiento," NYE U: Iberia, 2011.

[17]  N. Ausubel and J. Novak, "Hanesian.(1983)," Psicología Educativa: Un punto de vista cognoscitivo, vol. 2, pp. 161-181, 1986.

[18]  L. Ruan, "Effects of the Combination of Synchronous Web-Based Teaching With Visually Creative Teaching on Art Students’ Creativity," Eurasia Journal of Mathematics, Science and Technology Education, vol. 14, pp. 3245-3251, 05/25 2018, doi: 10.29333/ejmste/90627.

[19]  P. J. Lewis, T. M. Catanzano, L. P. Davis, and S. G. Jordan, "Web-based Conferencing: What Radiology Educators Need to Know," Academic Radiology, vol. 27, no. 3, pp. 447-454, 2020/03/01/ 2020, doi: https://doi.org/10.1016/j.acra.2019.05.017.

[20]  J. Chiodini, "Online learning in the time of COVID-19," Travel Medicine and Infectious Disease, vol. 34, p. 101669, 2020/03/01/ 2020, doi: https://doi.org/10.1016/j.tmaid.2020.101669.

[21]  C. H. Li et al., "Virtual Read-Out: Radiology Education for the 21st Century During the COVID-19 Pandemic," Academic Radiology, vol. 27, no. 6, pp. 872-881, 2020/06/01/ 2020, doi: https://doi.org/10.1016/j.acra.2020.04.028.

[22]  F. F. Reichheld, "The one number you need to grow," Harvard business review, vol. 81, no. 12, pp. 46-55, 2003. [Online]. Available: http://marketinglowcost.typepad.com/files/the-one-number-you-need-to-grow-1.pdf.

[23]  F. F. Reichheld, The Ultimate Question: Driving Good Profits and True Growth (Harvard Business Press). Boston, MA., 2006.

[24]  F. F. Reichheld, & Markey, R., La Pregunta Decisiva 2.0. Madrid, España: LID Editorial, 2012.

[25]  J. F. Hair, R. E. Anderson, R. L. Tatham, and W. C. Black, Análisis multivariante. Prentice Hall Madrid, 1999.

[26]  M. S. Bartlett, "Tests of significance in factor analysis," British journal of psychology, 1950.

[27]  H. F. Kaiser, "A second generation little jiffy," Psychometrika, vol. 35, no. 4, pp. 401-415, 1970.

[28]  M. A. Cea, Análisis multivariabl: teoría y práctica en la investigación social. Editorial síntesis, 2004.

[29]  R. Furry and V. R. Bacharach, "Psychometrics: An introduction (pp. 180)," ed: Thousand Oaks, CA: SAGE Publications, 2014.

[30]  J. Nunnally and I. Bernstein, "Psychometric theory, 3rd edn., internat. stud. ed.,[Nachdr.]," ed: McGraw-Hill Series in Psychology. Tata McGraw-Hill Ed, New Delhi, 2010.

[31]  C. Ringle, S. Wende, and J.-M. Becker, "SmartPLS 3. SmartPLS GmbH, Boenningstedt," Journal of Service Science and Management, vol. 10, no. 3, 2015.

[32]  K. K.-K. Wong, "Partial least squares structural equation modeling (PLS-SEM) techniques using SmartPLS," Marketing Bulletin, vol. 24, no. 1, pp. 1-32, 2013.

[33]  D. George and M. Mallery, "Using SPSS for Windows step by step: a simple guide and reference," 2003.

[34]  J. F. Hair Jr, G. T. M. Hult, C. Ringle, and M. Sarstedt, A primer on partial least squares structural equation modeling (PLS-SEM). Sage publications, 2016.

[35]  R. P. Bagozzi and Y. Yi, "On the evaluation of structural equation models," Journal of the academy of marketing science, vol. 16, no. 1, pp. 74-94, 1988.

[36]  J. F. Hair, M. Sarstedt, T. M. Pieper, and C. M. Ringle, "The use of partial least squares structural equation modeling in strategic management research: a review of past practices and recommendations for future applications," Long range planning, vol. 45, no. 5-6, pp. 320-340, 2012.

[37]  T. K. Dijkstra and J. Henseler, "Consistent partial least squares path modeling," MIS quarterly, vol. 39, no. 2, 2015.

[38]  C. Fornell and D. F. Larcker, "Evaluating structural equation models with unobservable variables and measurement error," Journal of marketing research, vol. 18, no. 1, pp. 39-50, 1981.

[39]  J. Henseler, C. M. Ringle, and M. Sarstedt, "A new criterion for assessing discriminant validity in variance-based structural equation modeling," Journal of the academy of marketing science, vol. 43, no. 1, pp. 115-135, 2015.