Evaluation of System Usability Scale as A Marker of Non-Human Computer Interface’s Usability: A Sanitizer Container-Based Study

The use and the production of sanitizers have increased in the post-pandemic situation to prevent the further spread of COVID-19. Usability assessment of sanitizer containers is essentially required to evaluate the effectiveness, efficiency, and satisfactory use of the sanitizer containers. This study aimed to evaluate the system usability scale (SUS) as a marker of perceived usability of non-human computer interfaces, such as - sanitizer containers. The perceived usability of three types of sanitizer containers was evaluated using SUS. The authors have measured the reliability, convergent, and discriminant validity to evaluate SUS as a marker of usability of sanitizer containers. The result showed that SUS lacks convergent validity though it has a higher reliability coefficient. Thus, this is not the best measure of usability for non-human computer interfaces like-sanitizer containers. The SUS was applied on the flip cap, finger-pressure pump, and spray-type sanitizer containers. The finger-pressure pump container has exhibited a higher SUS score than others. The results of this study have given an idea about the usability of three different kinds of sanitizer containers. This research provides an overview of the application of SUS on non-Human computer interfaces like sanitizer containers. This study provides limitations like lack of convergent validity and ideas to overcome common method bias.

Keywords: Usability Testing and Evaluation, Sanitizer Usability, Validity, Reliability, Gender

The deadly novel SARS-CoV-2 (COVID-19) originated in Wuhan, China, which resulted in a massive outbreak around the globe [1]. The World Health Organization (WHO) declared COVID-19 as a pandemic on 11th March 2020 [2]. COVID-19 spreads through different human activities like- talking, coughing, sneezing, etc [3]. Apart from the vaccination, some precautionary measures likewearing the mask(s), maintaining social (physical) distancing, and washing hands including repeated hand sterilizing with alcohol- based sanitizer are required to stop COVID-19 from further being spread. Alcohol-based sanitizer denatures the natural structure of the protein envelope of SARS-CoV-2 [4]. As a result, the sales of hand sanitizer have hiked by 600% [5]. The hand sanitizer production capacity of India has grown 1,000 times during the Covid-19 pandemic [6]. Usability assessment of sanitizer containers is necessary for three main reasons. Firstly, usability marks out the effectiveness of a product. Secondly, usability describes the efficiency of a product [7]. Third, usability can lead to users’ satisfaction. Lack of usability is the source of users’ dissatisfaction [8]. Moreover, usability plays an important role in influencing the choices of users while purchasing a product [9].

Participants of this study were divided into two groups based on their gender and their type of work. Therefore, the objectives of this study were to compare the perceived usability score of different sanitizer containers and to find out the differences in the perceived usability of sanitizer containers between the males & females as well as work from home participants & commuting worker participants. Thus, the hypotheses of this study are-

H1: System Usability Scores of different sanitizer containers are different.

H2: There is different perceived usability of sanitizer containers between males and females.

H3: There is different perceived usability of sanitizer containers between work from home participants & commuting worker participants.

Brooke developed the System Usability Scale (SUS) [10]. Initially, it was unidimensional [11]. Two latent factors were identified in 2009. The first factor comprised of eight items (items 1,2,3,5,6,7,8 and 9) concerning usability and the second factor comprised of two items (items 4 and 10) related to learnability [12, 13]. But most satisfactory model revealed that the two-factor structure was based on positive (odd-numbered items) and negative (even-numbered items) items [14]. This system usability scale can be used to assess the usability of various products, software, hardware, and websites [15]. These products include- everyday products (microwave, landline, automated teller machine, etc.), customer equipment, face coverings, and safety signs [16-19]. Nonetheless, there is no evidence of using SUS as a measure of perceived usability of different sanitizer containers. Therefore, this study aimed to evaluate the system usability scale as a marker of sanitizer containers’ usability. Thus, the research question of this study is-

Q: Can SUS be a reliable and valid marker of the perceived usability of non-human computer interfaces such as sanitizer containers?

The distinctiveness of this study is that exploratory and confirmatory factor analysis of system usability scale (SUS) items along with validity and reliability tests were used to interpret the result instead of accounting only the descriptive statistics. Different statistical analyses (discussed later) were performed to compare the usability of different sanitizer containers.

This is a descriptive type of cross-sectional study. The authors have categorized perceived usability as a dependent variable that was measured by using SUS. Three independent variables of this study are - gender, type of work, and type of sanitizer containers.

Participants were randomly selected from different cities in India. A total of 135 participants (N) were selected for the usability measurement, among them 72 (53.33%) were males and 63 (46.67%) were females (there was no third gender). Further, participants were divided into two groups based on their work type- commuting workers and work from home employees. The number of commuting workers and work-from-home employees were 57 and 78 respectively. The age range of all the participants is 18-65 years. The mean (±Standard Deviation) age of participants was 28.42±8.282 years. Inclusion & exclusion criteria were satisfied based on the information acquired from the participants.

Inclusion Criteria: Daily sanitizer users were included in this study. Participants in the age group of 18-65 years were included in this study. Participants who have used all three types of containers were included in this study.

Exclusion Criteria: Participants who do not use sanitizer and those participants who do not know how to use sanitizer containers were excluded from this study.

Ethical Approval: This research complied with the American Psychological Association Code of Ethics and was approved by the Institutional Human Ethical Committee at the University of Calcutta. Informed consent was obtained from each participant.

Sanitizer Container Selection: Three (3) sanitizer containers were selected through an online market with different usability. These sanitizer containers are designated as Type 1, Type 2 & Type 3. Type 1 is a flip cap container with gel-based alcohol sanitizer. Type 2 container consists of a finger pressure pump with gel-based alcohol sanitizer. Type 3 is a spray container with liquid-based alcohol sanitizer. Figure 1 shows three kinds of sanitizer containers and their usability.

Measurement of Usability: The system usability scale (SUS) was adapted to measure the usability of sanitizer containers [18]. It was a 5-point rating scale that includes - “I strongly disagree”, “I disagree”, “I don’t disagree nor agree”, “I agree”, and “I strongly agree”. SUS consists of 10 reliable (Cronbach’s alpha = 0.85) questions[20, 21].

The SUS questions are as follows [15, 22]:

Item 1. I think that I would like to use this system frequently.
Item 2. I found the system unnecessarily complex.
Item 3. I thought the system was easy to use.
Item 4. I think that I would need the support of a technical person to be able to use this system.
Item 5. I found the various functions in this system were well integrated.
Item 6. I thought there was too much inconsistency in this system.
Item 7. I would imagine that most people would learn to use this system very quickly.
Item 8. I found the system very cumbersome to use.
Item 9. I felt very confident using the system.
Item 10. I needed to learn a lot of things before I could get going with this system.

The odd number questions (1,3,5,7 & 9) were positive statements and the even number questions (2,4,6,8 & 10) were negative statements [12]. To compute the overall SUS score, the following method was applied- The item score based on positive statements or odd number questions was deducted by 1 (x-1) and the item score based on negative statements or even number questions was calculated by deducting the score from 5 (5-x). The summation of these item scores was then multiplied by 2.5 to provide a total SUS score. The range of the SUS score is between 0 (extremely poor usability) and 100 (excellent usability)[23].

Table 1 represents comparison of System Usability Scale with other usability assessment methods

For this study, the word “System” from the questionnaire was replaced with “Sanitizer container”.

Participants were randomly asked to use three different types of sanitizer containers one after another. Because the random presentation of the containers will counterbalance and minimize the carrying over effects of the participants. The participants used sanitizer containers using the following steps-

1. Participants held the container,
2. Then participants have used the head (flip cap or pressure pump or spray) of the sanitizer container to dispense the sanitizer,
3. Participants have poured sanitizer into their palms.
4. The SUS was applied after the use of each sanitizer container.
5. This study was documented using a google form. The contents of the form were a few demographic questions along with a SUS questionnaire.

Descriptive Statistics and Normality Test: After the collection of data, total SUS scores of three different types of sanitizer containers were expressed in terms of average (mean) with standard error (SE) and percentile values. Kolmogorov-Smirnov test was performed to check the normality of the variables [24, 25].

Comparison Test: The related-sample Friedman’s two-way ANOVA (two-tailed) by ranks was performed to compare the distributions of three SUS scores [8]. After obtaining significance, Bonferroni correction was performed as a post-hoc test. The Mann-Whitney U test (two-tailed and unpaired) was performed to compare the SUS scores distributions of three types of sanitizer containers between males and females as well as between commuting workers and work from home employees. The strength of association and effect size between the parameters were measured using Pearson’s correlation coefficient (r) [26]. The effect size (r) is small if the value is within 0.1 to 0.3, medium if the value is within 0.3 to 0.5 and large if r varies more than 0.5 irrespective of sign (negative or positive) [27].

Exploratory Factor Analysis (EFA): EFA was performed to evaluate the system usability scale as a marker of sanitizer containers’ usability. The exploratory factor analysis (EFA) of SUS was performed by using principal component analysis (PCA) with ProMax rotation [28, 29]. EFA identifies the number of latent factors which are involved in SUS and the correlation between each item of the SUS questionnaire [11].

Confirmatory Factor Analysis (CFA): CFA has done using maximum likelihood (ML) estimation to confirm the latent factors which are involved in SUS [30]. The fitness of the model was assessed by using Chi-square (χ2)/degrees of freedom(df), Comparative Fit Index (CFI, close to 0.95 or greater), Root Mean Square Error of Approximation (RMSEA, close to 0.06 or below) along with upper and lower limit as suggested by Brown [31].

Test for Validity: Convergent and discriminant validity of the confirmatory factor analysis model was measured by construct reliability (CR>0.70), average variance extracted (AVE>0.50) and maximum shared squared variance (MSV

Test for Reliability: The reliability assessment of the SUS was done by using the Omega hierarchical (HA) coefficient [34]. Omega is a reliability evaluation that does not pivot on the assumption of tau equivalence. There is no specific benchmark for omega to evaluate acceptable reliability but a minimum of 50 and values with closer proximity to 0.75 are recommended for acceptable and good reliability. The Cronbach’s alpha was measured to check the reliability of this questionnaire. A Cronbach’s alpha coefficient higher than 0.70 is to be considered reliable [35].

Test for Common Method Bias Detection: Common method bias (CMB) was detected using an unmeasured latent marker construct (ULMC) [30]. A common latent factor (CLF) is used to represent the CMB[36]. The standardized regression weights of the bi-factor model with and without CLF were subtracted to measure the effect of the CMB. The difference of less than 0.200 indicates that there is a less weighted effect of the common factor [37].

Previously mentioned statistical tests were performed by using Microsoft Excel 2016 and Statistical Package for the Social Sciences (SPSS) software (version 26) including AMOS (version 23) [28, 38].

Table 2 represents the mean (±SE) System Usability Scale (SUS) score of three types of sanitizer containers which are categorized as - general users, male users, female users, commuting workers, and working from home employees.

Individual SUS scores were arranged according to the adjective rating scale [39]. It has been found that the maximum number (80) SUS scores in the acceptable range were for type 2 sanitizer containers (Table 3). These 80 individuals for type 2 sanitizer containers comprised 59.3% of the total participants. The lowest number (11) of SUS scores were for type 3 sanitizer container. These 11 individuals for type 3 sanitizer container comprised only 8.1% of the total participants.

Percentile values of three types of sanitizer containers indicated that the number of distributions of individual SUS scores isgreater in type 2 sanitizer containers in the 25th, 50th (median), 75th, and 95th percentiles than other two types (Table 4). Only the 95th percentile distributions of type 1 and type 2 were the same. Whether these distributions were significantly different or not is discussed in later sections of the result section.

The type 2 sanitizer container had a higher SUS score than type 1 and 3. This result has indicated that the participants preferred type 2 sanitizer container over type 1 and 3 in terms of usability. To confirm the differences between the above-mentioned SUS scores, several tests were performed. Kolmogorov-Smirnov normality test was performed to check the type of distribution. It has specified that the set of observations was not normally distributed (p<0.05) [25]. Related-samples Friedman's Two-Way ANOVA test by ranks has indicated that the distributions of the SUS scores of type1, type 2, and type 3 are significantly different (p<0.05). Post-hoc paired wise comparison (with Bonferroni’s correction) of SUS scores of previously mentioned sanitizer containers also dissimilar with one another (p<0.05). Hence, the type 2 sanitizer container had the highest usability over the type 1 and type 3 sanitizer container respectively. The effect size of type 2 and type 3, type 1 and type 3 were very large (very strong) and large (strong) respectively. But the effect sizes of type 1 and type 2 were medium (moderate) [40]. The related-sample Friedman's two-way ANOVA by rank test has revealed significant differences (p<0.05) in perceived system usability scale scores among three different types of sanitizer containers (N= 135, Chi-square= 82.093, df= 2). Table 5 indicates the Wilcoxon signed-rank pair-wise comparisons of type 3 vs type 1 SUS score, type 3 vs type 2 SUS score, and type 1 vs type 2 SUS score along with significance level including Bonferroni adjustment correction and effect sizes.

The Mann-Whitney U test has revealed that the distribution of the SUS scores was the same between the genders across all three types of sanitizer containers (intra-type) which are type 1, type 2, and type 3 respectively. In other words, there were no significantly different (p>0.05) distributions of SUS scores observed between males and females in the same type of sanitizer container. The comparison was as follows:

1. Type 1 SUS score of males and Type 1 SUS score of females.
2. Type 2 SUS score of males and Type 2 SUS score of females.
3. Type 3 SUS score of males and Type 3 SUS score of females.

The Mann-Whitney U test has revealed that the distribution of the SUS scores was the same between the work types across all three types of sanitizer containers (intra-type) which are type 1, type 2, and type 3 respectively. In other words, there were no significantly different (p>0.05) distributions of SUS scores observed between commuting workers and work from home employees in the same type of sanitizer container. The comparison was as follows:

1. Type 1 SUS score of commuting workers and Type 1 SUS score of work from home employees
2. Type 2 SUS score of commuting workers and Type 2 SUS score of work from home employees
3. Type 3 SUS score of commuting workers and Type 3 SUS score of work from home employees

There were significantly different (p<0.05) distributions of SUS scores observed for inter-type of sanitizer containers (among type 1, type 2, and type 3) in the following categories:

1. Gender-based- a.Male
b. Female
2. Work type based-
a. Commuting workers
b. Work from home employees

Table 6 represents Related-Samples Friedman's Two-Way ANOVA by Ranks test summary of both categories. From table 6, the authors obtained that the distributions of the SUS scores of type1, type 2, and type 3 are significantly different (p<0.05) in both categories.

Pairwise comparison along with Bonferroni correction has been utilized as a post-hoc test. It has been found that the SUS scores of type 1 and type 2 sanitizer containers are not significantly different in both male and female users. In other words, the distribution of the SUS scores of type 1 and type 2 sanitizer containers under the male user sub-category were the same.

A similar result was also found in the gender-based category. The SUS scores of type 1 and type 2 sanitizer containers among work from home employees and commuting workers were insignificantly distributed. In other words, the distribution of the SUS scores of type 1 and type 2 sanitizer containers under the work from home employee sub-category were identical. Table 7 indicates the pairwise comparison of the SUS scores based on gender and work type categories respectively. The effect size of type 2 and type 3 in both gender-based and work-based categories was very large (very strong). The effect size of type 1 and type 3 in both gender-based and work-based categories was large (strong) respectively. But the effect size of type 1 and type 2 in both gender-based and work-based categories was medium (moderate) [40].

The factor analysis of SUS was done by using principal component analysis with pro-max rotation (κ, kappa=4). The initial inspection of the correlation matrix of SUS questionnaire items indicated considerable numbers of the coefficients were above ±0.30 (Table 8). Table 8 indicates that the correlation between an odd item with other odd items was positive and an odd item with other even items was negative, and vice versa

The Kaiser-Meyer-Olkin (KMO) measure of sampling adequacy index was 0.846 which was above the recommended value of 0.60 [41]. Bartlett's test of sphericity has reached statistical significance (χ2=1679.873, p<0.001) [42, 43]. Communalities were the proportion of variances of each variable that can be explained by the components or factors [44]. No item had a communality (h2) below 0.30. These results infer that the data obtained from the SUS questionnaire was suitable for principal component analysis [45, 46]. There were two factors with Eigenvalues over 1. Factor 1 and 2 have initial eigenvalues of 4.029 and 2.190, respectively. These factors cumulatively explained 62.19% of the total variances. Pattern matrix of exploratory factor analysis of ten SUS questionnaire items has revealed that there are two latent components/factors (Table 9). The first component consisted of item numbers 5, 9, 3, 7, and 1 and the second component consisted of item numbers 10, 4, 8, 6, and 2. Factor 1 or component 1 symbolized the userfriendliness of a system (sanitizer container) and Factor 2 or component 2 symbolized the perplexity (confusion and complexity) of the system (sanitizer container).

Confirmatory factor analysis (CFA) has confirmed the previously-mentioned issue. Further, two models were developed. The first model was developed using two latent factors- user-friendliness and perplexity as a bifactor model. The second model indicated that there may be a common method bias that was linked with the items. Table 10 indicates the model fit indices for both models. The model with a common method bias exhibited a better model fitness than the bifactor model.

Path diagrams of the bifactor model with and without common method bias (CMB) are provided in figure 2. Error terms (e) of item 4 and item 10 and item 5 and item 7 are interrelated because they had a 0.49 and 0.20 correlation value respectively. As suggested by Brooke, SUS items were dependent on- the product’s effectiveness, efficiency, and user satisfaction. Item 5 was scored for the system integration and item 7 was scored for quick learnability. A well-integrated system can cause better learnability of that system [10]. On the other hand, learnability has a relation to usability [47], and items 4 and 10 are scored for learnability. Thus, error terms of items 4 and 10 and 5 and 7 were interlinked to form the model.

The convergent and divergent validity of the common latent factor model was explained using table 11. Construct reliability (CR) was higher (>0.700) in both of the factors (User-friendliness and perplexity) but very low for the common bias. The average variance extracted (AVE) was almost the same as the cut-off value (0.50) for the factor user-friendliness, but slightly lower than the cut-off value for the perplexity factor. The maximum shared squared variance (MSV) of the User-friendliness and perplexity was 0.225 which was lower than the AVE. These factors indicated that the convergent and divergent validity were above the threshold level. The effect of common factor bias was low.

There was a confirmed discriminate validity among user-friendliness and perplexity because the HTMT correlation ratio of these two was 0.456 which is lower than 0.850.

Omega and Cronbach’s alpha coefficients of the SUS were reliable because both of these coefficients were well above the acceptable (>0.70) limit. Table 12 represents the Omega and Cronbach’s alpha coefficients.

The post-hoc ULMC method has detected the presence of common method bias (CMB). Standardized estimates of “userfriendliness” factor items have changed more than 0.2 in the presence of CMB (Table 13).

The chi-square (χ2) and degrees of freedom (df) values were significantly different(p<.05) between the bifactor model and the bifactor model with CMB.

In this study, the authors have found the presence of two latent factors which can affect perceived usability. These factors are perceived user-friendliness and perplexity (confusion and complexity) of the product. This bifactor model has comprised of positive and negative items of the SUS which represent user-friendliness and perplexity respectively. Further, confirmatory factor analysis has established the presence of two latent factors in SUS. Model fit indices revealed that it is moderately fit. Post-hoc unmeasured latent marker construct has confirmed the presence of common method bias in the “user-friendliness” latent factor items. This bias may be due to the common scale format, common scale anchor (5-point scale), and positive-negative wording [30]. The order of presentation of the products can be a potential source of method bias. The construct reliability (CR) of convergent validity has construed that the system usability scale (SUS) can be used as a marker of sanitizer containers’ usability. But the average variance explained (AVE) has failed to establish a convergent validity for the “perplexity” factor items. AVE is a strict measure of convergent validity because consideration of CR only will conclude that the convergent validity of a construct is adequate, even if more than 50% of the variance is due to the error [48]. Maximum shared squared variance (MSV) and heterotrait-monotrait (HTMT) ratio have successfully established discriminant validity. The contents of the SUS questionnaire were reliable enough to assess the usability of sanitizer containers because Omega hierarchical and Cronbach’s alpha coefficients were well above the threshold limit. Furthermore, the previously-stated model has indicated the relationship between user-friendliness, perplexity, and usability. Therefore, the answer to our research question is that the SUS cannot be a valid marker of the perceived usability of sanitizer containers though it has a decent reliability coefficient. Table 14 represents the comparison of the reliability coefficients (Cronbach's alpha) of different studies to validate the effectiveness of the proposed experimental condition [18].

The system usability scale has been applied to real-time applications such as- Web sites, Cell phones, Television, Interactive Voice Responses (IVR), Graphical User Interface (GUI) and the mean SUS scores of these products were 68.2, 65.9, 67.8, 72.4, 76.2 respectively. The perceived usability of 14 everyday products has been measured using the SUS. Previously mentioned study has shown that Excel, Global Positioning System (GPS), Digital Video Recorder (DVR), PowerPoint presentation (PPT), Word, Wii (game console), iPhone, Amazon, Automated teller machine (ATM), Gmail, Microwaves, Landline, Browser, Google search have the mean SUS scores of 56.5, 70.8, 74.0, 74.6, 76.2, 76.9, 78.5, 81.8, 82.3, 83.5, 86.9, 87.7, 88.1, 93.4 respectively. Hence, the afore-mentioned study validates the acceptability of SUS. It has been obtained from the previous study that the SUS score does not interpret whether the usability of a product is poor or good. This decision requires some comparison with other products. One form of the comparison is to statistically compare the SUS score of two or more groups. The comparison can be done among diverse products or user groups [18]. The percentile values and descriptive statistics of adjective ratings for individual System Usability Scale (SUS) scores have shown that the type 2 sanitizer container has better usability over type 1 and type 3. Because a higher SUS score means better usability [11]. According to the comparative chart of mean SUS score, type 2 sanitizer container has ‘good’ usability of 71.39 which is higher than type 1(63.50) and type 3(41.06) which are in the range of ‘ok’ and ‘poor’ respectively [11, 39]. Friedman’s ANOVA has revealed that the result is not only true for the general users but also true for the user categories based on gender and work type. But post-hoc analysis indicated that the females and commuting workers have found that the usability of type 1 and 2 sanitizer containers were similar. All of these outcomes indicate that the type 2 sanitizer container is much easier to use, less complex, and has better learnability. On the other hand, there was no significant difference in perceived usability of a particular sanitizer container between the genders and between the type of work. Additional reasons for the greater SUS score of type 2 sanitizer are- more accurate dispense of sanitizer solution and anyone can comfortably use it in public or at home. Usability influences the choice of a product [8]. The higher usability of type 2 can result in more preference for pressure pumps like sanitizer container (type 2).

It can be concluded that the system usability scale is not the best measure of usability for sanitizer containers though it has a good reliability coefficient. One of the reasons is that the SUS has a poor convergent validity. Another reason is that it can be affected by CMB. Additionally, this research revealed that the type 2 sanitizer container has better usability, learnability, user-friendliness, and lower perplexity irrespective of the users. These factors helped users to give a higher SUS score to the type 2 sanitizer container when compared to the other two. The result of this study suggested that both the genders (male and female) and workers (commuting and working from home) widely accepted the type 2 sanitizer container for their day-to-day lives.

● Initially, the SUS was developed for HCI. It was applied to sanitizer containers that were non-HCI in the current study. For that reason, it has faced convergent validity issues and it has made the SUS a poor marker of usability for non-HCI.
● The present study has used a post-hoc unmeasured latent marker construct (ULMC) method to detect Common method bias (CMB). ULMC is not the best method to detect and minimize the effect of CMB.
● Only three types of sanitizers have been used as non-HCI to evaluate SUS as a marker of non-human computer interface’s usability. The application of SUS on other products with non-HCI has needed to confirm the existing results.

Usability researchers should not rely on SUS only for non-human computer interfaces (non-HCI) like- sanitizer containers. They should use more than one usability scale or questionnaires precisely for non-human computer interface (non-HCI). If practitioners are unable to use more than one usability questionnaire, then they are recommended to use a-priori ideal markers along with the SUS questionnaire as an effective measure against common method bias [49]. Because it will statistically remove the effect of CMB from the SUS responses [50].

SUS with all positive items can be an effective measure of perceived usability. The previous study has shown that it has better reliability [51]. All positive SUS questionnaire was adapted to measure perceived usability for non-human computer interface (non-HCI) like face coverings[16].

Practitioners should separate the usability measurement of each product to reduce common method bias. This separation can be done through psychological barriers. For example, the psychological barrier can be created by using a story to make it appear that the usability of the first product is not related to the usability of the other product [30].

The randomized or counterbalance order of the presentation of the products along with a higher number of participants can minimize the effect of common method variance [52].

The authors would like to thank all subjects for their participation and would like to acknowledge Ms. Madhuri Datta, Mr. Romit Majumder, and Ms. Sutithi Dey, Department of Physiology, Research scholars, University of Calcutta, India for proofreading the article.