MIS Quarterly, 2014, Volume 38,
Issue 1, Page 1/28/2017.
A reliability generalization study (a meta-analysis of reliability coefficients) was conducted on three widely studied information systems constructs from the technology acceptance model (TAM): perceived ease of use, perceived usefulness, and behavioral intentions. This form of meta-analysis summarizes the reliability coefficients of the scores on a specified scale across studies and identifies the study characteristics that influence the reliability of these scores. Reliability is a critical issue in conducting empirical research as the reliability of the scores on well-established scales can vary with study characteristics, attenuating effect sizes. In conducting this study, an extensive literature search was conducted, with 380 articles reviewed and coded to perform reliability generalization. Study characteristics, including technology, sample, and measurement characteristics, for these articles were recorded along with effect size data for the relationships among these variables. After controlling for number of items, sample size, and sampling error, differences in reliability coefficients were found with several study characteristics for the three technology acceptance constructs. The reliability coefficients of PEOU and PU were lower in hedonic contexts than in utilitarian contexts, and were higher when the originally validated scales were used as compared to when other items were substituted. Only 27 percent of the studies that provided the measurement items used the original PEOU items, while 39 percent used the original PU items. Scales that were administered in English had higher reliability coefficients for PU and BI, with a marginal effect for PEOU. Reliability differences were also found for other study characteristics, including reliability type, subject experience, and gender composition. While average reliability coefficients were high, the results show that, on average, relationships among these constructs are attenuated by 12 percent with maximum attenuation in the range of 35 to 43 percent. Implications for technology acceptance research are discussed and suggestions for addressing variation in reliability coefficients across studies are provided.
Keywords: Reliability generalization; reliability; meta-analysis; technology acceptance model (TAM); ease of use; usefulness; behavioral intentions; effect size attenuation