- © 2009 American Dental Education Association
Computer-Assisted Learning in Orthodontic Education: A Systematic Review and Meta-Analysis
- Thikriat S. Al-Jewair, B.D.S., M.B.A.,
- Amir Azarpazhooh, D.D.S., M.Sc., FRCD(C),
- Sunjay Suri, B.D.S., M.D.S., M Orth RCS (Edinburgh), DNB, DIBO and
- Prakesh S. Shah, M.Sc., M.B.B.S., M.D., MRCP, FRCPC
- Direct correspondence and requests for reprints to Dr. Thikriat S. Al-Jewair, Department of Community Dentistry, Faculty of Dentistry, University of Toronto, 124 Edward Street, Toronto, Ontario, Canada M5G 1G6; 647-242-4490 phone; .
- Received December 1, 2008.
- Accepted February 16, 2009.
The purpose of this systematic review and meta-analysis was to compare the efficacy of computer-assisted learning (CAL) with traditional methods of learning in orthodontic education. Comprehensive electronic and manual searches of randomized controlled trials and prospective studies were conducted. Participants considered were undergraduate or postgraduate orthodontic students or orthodontic educators. The main outcome measure of CAL efficacy was knowledge gain. The time efficiency of the method was assessed based on the time spent learning the material, while its qualitative effect was tested by the attitudes of participants. Nine studies assessing CAL in teaching orthodontic diagnosis and treatment planning met the inclusion criteria. A statistically significantly higher knowledge gain favoring CAL was identified in studies that used pre- and post-intervention tests (weighted mean difference [WMD] 9.78 percent, 95 percent confidence intervals [CI] 2.89 percent, 16.67 percent; test of heterogeneity p=0.25). For studies that used only post-intervention tests, significantly greater efficacy was noted, but the effect size was smaller (WMD 3.79 percent, 95 percent CI 0.31 percent, 7.28 percent; test of heterogeneity p=0.003). Overall, student attitudes were positive towards CAL. No conclusions can be drawn about the time efficiency of CAL. Further studies are warranted to examine other important outcomes, including CAL efficacy in teaching other orthodontic topics, cost-effectiveness, and knowledge retention.
Computer-assisted learning (CAL) has been used in orthodontics since the early 1980s to support learning in diagnosis and treatment planning. It has been argued that CAL is more effective and faster than traditional methods for inculcating comprehension and for knowledge retention.1 Yet, evidence of its efficacy is unclear. Studies of CAL in orthodontics have reported mixed results, ranging from unfavorable effects to significant beneficial effects over traditional learning methods.1–3 Rosenberg et al.4 reviewed four studies of CAL in orthodontics teaching and reported conflicting results. Clark et al.2 identified no significant difference between the two methods of learning, whereas Irvine and Moore5 and Luffingham3 reported a significant advantage for CAL over conventional methods. In contrast, Hobson et al.1 reported a superior effect using conventional methods.
Rosenberg et al.4 reviewed studies of CAL published until July 2003. There have been several subsequent studies,6–9 suggesting that this is a growing area of research and thus warranting revisiting the question. Moreover, a comprehensive quantitative summary (meta-analysis) of the effect of CAL in orthodontics has not been reported. The purpose of this systematic review and meta-analysis was to compare and quantify the efficacy of CAL in comparison with traditional methods of learning in orthodontic education.
A comprehensive search of English-language articles to November 2008 was conducted using the following databases: Ovid MEDLINE, Ovid OLDMEDLINE, EMBASE, Database of Abstracts of Reviews of Effectiveness (DARE), Educational Resources Information Center (ERIC), International Pharmaceutical Abstracts, Cumulative Index to Nursing & Allied Health (CINAHL), HealthSTAR/Ovid HealthSTAR, Cochrane Controlled Trials Register, Cochrane Database of Systematic Reviews, Health and Psychosocial Instruments, Library and Information Science Abstracts (LISA), PubMed, and the World Wide Web using Google Scholar. Additionally, a manual search was performed of reference lists of relevant articles, theses, and dissertations. Table 1⇓ shows the search strategy carried out by two of the authors (TJ and AA), who independently assessed the studies by inspecting the titles, abstracts, and full articles for validity and quality assessment.
The selection criteria for the studies were set a priori using the PICO-C (population, intervention, comparator, outcome, and critical appraisal) system. Only randomized controlled trials (RCTs) and prospective studies were included. The participants considered were undergraduate dental students taking a clinical or preclinical orthodontic course, postgraduate orthodontic residents, or orthodontic educators. Intervention methods explored included any of the following: interactive multimedia course packages, web-based cases, computer-based hypertext tutorials, computer-based tutorials, and CDs. Control methods included traditionally taught lectures, seminars, tutorials, or live demonstrations.
All articles were reviewed, and data were extracted using a customized data abstraction sheet. Agreement between the reviewers was assessed by calculating the Kappa score, which showed a high level of agreement (0.99 and 0.96 on the title and abstract stages respectively). Any disagreement was resolved by discussion. Methodological quality assessment was carried out using the guidelines of the Canadian Task Force on Preventive Health Care10 and a checklist for appraising evidence in studies of efficacy in health care.11 The maximum possible score was 17, and studies were included if they scored ≥ 11.The criteria for high-quality studies were 1) an objective pre- and post-intervention assessment, 2) homogeneous groups of participants, 3) allocation concealment, 4) controlling for confounders, and 5) clear reporting of results.
The primary outcomes were as follows:
Quantity and/or quality of knowledge gained (including recall and application) by the students (objectively measured by student’s performance on pre- and post-intervention tests [multiple-choice, oral, or written tests] or measured at the end of the teaching [post-intervention only]).
Knowledge retention of at least three months after the test date, measured as above.
The secondary outcomes were as follows:
Diagnostic accuracy as compared with a gold standard.
Performance in clinical procedures.
Attitudes of students and educators towards CAL.
Cost and labor of structuring and delivering CAL courses (cost-effectiveness).
Burden of study on students.
Degree of student engagement in the subject.
Efficacy of CAL for specific areas of orthodontics.
Use of CAL as an adjunct educational method versus a replacement method.
For the purpose of meta-analysis, the sample size, mean percentage, and standard deviation percentage were obtained from each study. In case of missing information, the authors were contacted. In studies with pre- and post-intervention test scores, the mean difference and the standard deviation of the difference (SD diff) for each learning group were obtained from the report if provided or were calculated by using the following formula: SD(diff) = ✓SD(A)2 + SD(B)2 – [2 x R x SD(A) x SD(B)], where A and B are the learning group categories investigated (CAL and traditional respectively) and R is the correlation coefficient. A sensitivity analysis was performed using different R-values (R=0.4; R=0.5; R=0.6) to test the stability of the data results. An R of 0.5 was used in the analysis.
The data were analyzed using RevMan software (version 4.3, Cochrane Collaboration, Canada).12 The statistical effect size was calculated using weighted mean differences (WMD) and 95 percent confidence intervals [CI]. Individual studies had their weight calculated based on the inverse of variance. To use that, the maximum possible dissimilar pre- and post-intervention test scores in the studies were converted into percentages, along with the appropriate 95 percent CI. The effect size was considered significant if p<0.05. Due to expected heterogeneity in the studies, a random effects model was used for analyses. Statistical homogeneity of the studies was assessed using the Cochrane test (p<0.1 was considered significant) and by examining the observed variance in effect sizes and residual variance. The I2 statistic was calculated to quantify heterogeneity.13 In accordance with convention, no statistical corrections were utilized to adjust for multiple comparisons. Included studies assessed reliability of the test instruments and the questionnaires developed.
The initial search identified 459 relevant articles (Table 1⇑). After elimination of duplicates and evaluation of titles and abstracts, twelve articles were considered for full review. After quality assessment, three articles were excluded. In two of these,14,15 the efficacy of CAL was subjectively measured, and for the third article, the acceptability of CAL was measured cross-sectionally.16
A total of nine articles met the inclusion criteria for this systematic review, seven of which were RCTs1–3,5–7,17 and the other two were prospective controlled trials.8,9 The studies assessed the efficacy of CAL in a number of orthodontic educational areas including diagnosis, treatment planning, use of digital records, cephalometrics, and mixed-dentition analysis. Characteristics of the included studies are presented in Table 2⇓.
Participants in all the included studies were undergraduate dental students in various years of education. The students in the CAL group received the learning material in several ways: an interactive multimedia course package,6 web-based cases,7 computer-based hypertext tutorials,2,17 computer-based tutorials,1,3,5,8 and CDs.9
In one study,8 CAL was used in conjunction with a seminar. Students in the traditional learning group received one or more of the following: conventional tutorials,3,17 seminars,1,8 traditional records,7 and lectures.2,5,6,9 The studies assessed one or more of four outcomes: knowledge gain, time efficiency, attitudes toward CAL, and cost and labor of structuring and delivering CAL courses.
Quantity and/or Quality of Knowledge Gain
Data were extracted from nine studies that compared the efficacy of CAL in terms of knowledge gain. Data from one study1 that assessed knowledge gain were not utilized for meta-analysis because the scores of post-intervention standard deviation were missing and the principal investigator did not have the information when contacted.
Three studies3,5,8 found statistically significant efficacy of CAL over traditional methods; p-values ranged from 0.01 to 0.03. Five studies2,6,7,9,17 showed no statistically significant differences. Only one study1 found a negative effect of CAL compared to traditional methods (p=0.032).
Three trials2,6,9 that assessed the quantity of knowledge gained using pre- and post-intervention tests were statistically combined (Figure 1⇓), and a statistically significant gain in knowledge favoring CAL was observed (WMD 9.78 percent, 95 percent CI 2.89 percent,16.67 percent; test of heterogeneity p=0.25; I2=27.6 percent).
Five studies3,5,7,8,17 examined the same outcome by administering post-intervention tests only. In one study,3 an RCT cross-over design was implemented, and participants were randomized to receive tutorial instructions on both Class II Div 1 and Class III malocclusions using two different methods (CAL or traditional). Although no information about the washout period was provided, the differences in the tutorial topics between the learning groups allowed the inclusion of the results of each group separately in the meta-analysis. In another study,8 students were taught by CAL or lecture about the application of the dental health component (DHC) and aesthetic component (AC) of the Index of Orthodontic Treatment Need (IOTN). Subjects were then tested on the following: AC, DHC grade, DHC qualifier, and DHC grade and qualifier. For this meta-analysis, the AC and DHC grade and qualifier scores were included as separate entities since the correlation between the two is low.18 The DHC grade or qualifier was not used because they represent the same measure components of the combined DHC grade and qualifier scores. Statistically significant knowledge gain favoring CAL was detected (WMD 3.79 percent, 95 percent CI 0.31 percent, 7.28 percent; test of heterogeneity p=0.003; I2=69.2 percent) (Figure 2⇓).
Overall, a significantly greater knowledge gain using CAL was evident in studies that controlled for pre-intervention knowledge and studies that did not.
Time spent in learning material on the computer was assessed in four studies.2,5,7,9 One study7 measured the time spent for both methods of learning and reported no significant difference between mean times (p>0.05). Time spent was objectively reported in all studies except two,7,9 where it was self-reported by students. The mean time spent on the computer ranged from forty-one to 360 minutes. In three trials,2,5,7 the correlation between time spent using the computer and improvement in test scores was explored. No correlation was reported in the two studies,5,7 and a low correlation was discovered in the third.2 Meta-analysis was not performed for this outcome because a comparison between the two methods of learning was conducted in only one study.
Attitudes of Students and Educators Towards CAL
Six studies1,2,5,7,9,17 administered questionnaires to students to assess their attitudes towards the CAL method provided. In one study,2 73 percent of those who received CAL found the method enjoyable, but only 46 percent found it easy to use. The same positive finding was evident in four other studies,5,7,9,17 but in one study,1 the students favored the traditional method over CAL. None of the studies assessed the attitudes toward CAL among educators. Statistical comparisons of attitudes toward both learning methods were not made in any of the studies, so meta-analysis was not performed.
Cost and Labor of Structuring and Delivering CAL Courses
The direct design and set-up cost of the CAL website was reported to be $1,850 in one study.7 However, the author in that study spent 40 percent of his time annually planning and administering the website programming process. Another study5 stated that 300 hours were needed to create a one-hour lecture on mixed-dentition analysis on a CAL program. None of the studies compared the cost of CAL to the cost of structuring and developing traditional courses or evaluated its cost-effectiveness.
The rest of the secondary outcome measures (e.g., diagnostic accuracy, performance in clinical procedures) were not investigated in any of the studies, and therefore, no conclusions can be drawn.
The aim of this systematic review and meta-analysis was to determine the objective efficacy of CAL in orthodontic education. Nine studies of adequate quality were included.
Pooled results of this meta-analysis indicate a 10 percent greater knowledge gain after using CAL, a finding in similar direction to that reported in a previous meta-analysis19 comparing individualized instruction (CAL) to conventional methods of instruction. In comparison with Rosenberg’s9 review, which reported mixed results from four included RCTs on the use of CAL in orthodontics, our review included nine primary studies of adequate quality and captured a significantly better outcome of CAL methods over conventional methods.
A small but statistically significant knowledge gain favoring CAL (3.79 percent) was also demonstrated in five studies3,5,7,8,17 that used only post-intervention tests. Nonetheless, the heterogeneity of these studies was statistically significant. This can be attributed to unequal pre-intervention knowledge, different ways of measuring CAL efficacy, and different ages and educational backgrounds of participants. Despite the limitations in methodology of these studies, the findings suggest a modest improvement in knowledge after using CAL.
The modest increases of 3.79 percent and 10 percent may be educationally meaningful to student learning. However, different confounders (e.g., level of motivation, prior knowledge) that can influence evaluation in education may not have allowed the detection of greater beneficial effects of CAL.
It is not clear whether CAL improved the quantity and/or the quality of knowledge gained, since none of the studies presented their test instruments. Although knowledge retention was one of the outcomes considered for this review, no relevant data were available regarding it.
Interestingly, all trials that found significant advantages of CAL used post-intervention tests alone and were published in the 1980s, when the use of computers in orthodontic education was quite novel and there was great eagerness to explore their application;20 this may explain the significant findings. Subsequent studies2,6–9,17 did not find significant effects, and one even found a negative effect.1 The potential complexity of some CAL packages may have negatively affected students’ learning outcomes.1 The unexplored confounding effect of the burden imposed by other courses undertaken simultaneously in dental school is another aspect that needs further investigation.
The reported time spent on CAL varied greatly between students. This is not surprising, as time spent on learning depends upon the amount of students’ prior knowledge of the subject and degree of motivation,4 learning style,21 and the perceived complexity of the learning material or CAL package.22
Most of the studies showed positive perceptions toward CAL among students. This finding can be attributed to the possibility of greater interactive learning offered by CAL compared with the often one-way communication in traditional didactic learning.23 It may also reflect change in student demographics over time, as the contemporary student generation is generally more computer savvy than the previous generation. In using CAL, students are able to personalize the learning environment and accommodate it to their schedules and locations.24 CAL is also potentially advantageous to educators, as it frees up more time for them to devote to other modes of knowledge transfer and reinforcement of understanding.15,25 However, traditional learning methods still have the advantage of providing an opportunity for human interaction26 with an educator who also may be viewed as a role model, thus enriching the learning experience.
While two studies reported extremely high costs for CAL,5,7 no conclusions can be drawn. The cost of structuring and developing a CAL package may be affected by the expertise of the CAL content developer and the nature, quantity, and quality of the subject material being developed. Additionally, recurrent cost analyses were not performed in any of the studies, even though the advantage of CAL is that it usually takes far less time to set up subsequent sessions after setting up the initial session.
The included studies examined CAL as an adjunct for traditional education and not as a replacement. This is acceptable, as previous studies have shown that students were not ready to use CAL as a replacement to textbooks or lectures.27,28
Our study has several limitations. The search strategy was limited to studies published in the English language. Also, many of the included studies did not measure the pre-intervention knowledge, subjecting the results to confounding and measurement bias. This may have weakened the pooled weighted effect of studies examining the post-intervention knowledge gain. Further, the different scoring systems and varied tutorial subjects affected the homogeneity of the studies included. Nonetheless, utilizing the strongest available evidence, this study was able to capture a positive effect of CAL on knowledge gain that supports its further integration in orthodontic education.
From the limited number of studies available on this growing area of education research in orthodontics, pooled evidence indicates that CAL, when applied for teaching orthodontic topics related to diagnosis and treatment planning, is at least as effective as conventional modes of orthodontic learning, and adds an additional small (4–10 percent) but significant gain in student knowledge acquisition. Further studies are warranted to examine other important outcomes, such as the efficacy of CAL on performance in clinical procedures, cost-effectiveness, knowledge retention, the burden it imposes on students, and its effect on educators in terms of attitudes and effort involved. Future studies should also investigate the efficacy of CAL in diverse orthodontic subject areas.
Dr. Al-Jewair is a candidate for the M.Sc. in Dental Public Health, Faculty of Dentistry, University of Toronto; Dr. Azarpazhooh is a candidate for the Ph.D./Specialty in Endodontics, Lecturer in Dental Public Health, and Researcher with the Community Dental Health Services Research Unit, Faculty of Dentistry, University of Toronto; Dr. Suri is Assistant Professor of Orthodontics, Faculty of Dentistry, University of Toronto and Staff Orthodontist, Department of Dentistry, The Hospital for Sick Children in Toronto; and Dr. Shah is a Neonatal Intensivist and Clinical Epidemiologist, Mount Sinai Hospital and Assistant Professor, Departments of Paediatrics and HPME, University of Toronto. Direct correspondence and requests for reprints to Dr. Thikriat S. Al-Jewair, Department of Community Dentistry, Faculty of Dentistry, University of Toronto, 124 Edward Street, Toronto, Ontario, Canada M5G 1G6; 647-242-4490 phone;.