This article requires a subscription to view the full text. If you have a subscription you may use the login form below to view the article. Access to this article can also be purchased.
- Sarah B. Pelton, MS, MLSCM SBBCM⇑
- Address for Correspondence: Sarah Pelton, MS, MLSCM SBBCM, Assistant Professor, Department of Clinical, Laboratory Sciences, Respiratory Care, and Health Administration, Ferris State University, 200 Ferris Drive, VFS 406, Big Rapids, MI 49307, 231-591-2207, SarahPelton{at}ferris.edu
Abstract
This retrospective study evaluated the ability to predict certification by the American Society for Clinical Pathology (ASCP) Board of Certification (BOC), using an overall score cutoff of 60% on a university comprehensive exam. The study also evaluated overall and content area scores (Blood Bank, Chemistry, Hematology, Immunology, Laboratory Operations, Microbiology, and Urinalysis and Other Body Fluids) for correlation between the university and BOC exams. Overall university exam scores ranged from 35-86% (percentage of correct answers) for students completing both exams from 2006-2015 (n = 152). BOC exam scores ranged from 287-755 (scaled from 0-999, with 400 required to pass). The overall correlation between scores was 0.65. Content area correlations ranged from 0.00 (Immunology) to 0.55 (Microbiology) for students completing both exams from 2012-2015 (n = 51). A receiver operating characteristic curve resulted in an overall university exam score cutoff of 55% showing the highest sensitivity and specificity for predicting success. Using the hypothesized 60% cutoff, one student showed a false positive result. All students scoring above 67% on the comprehensive exam passed the certification exam. In general, this study indicates that there are large variations when comparing results between comprehensive and certification exams.
ABBREVIATIONS: ASCP - American Society for Clinical Pathology; BOC - Board of Certification; BOR - Board of Registry; GPA - grade point average; MLS - medical laboratory science/scientist; NAACLS - National Accrediting Agency for Clinical Laboratory Sciences; ROC - receiver operating characteristic; SD - standard deviation
INTRODUCTION
The American Society for Clinical Pathology (ASCP) Board of Certification (BOC) offers national certification examinations in varying categories, including Medical Laboratory Science (MLS). These certification exams utilize computer adaptive technology, whereby questions are given at increasing difficulty levels until the examinee fails to answer correctly, at which time the difficulty level is lowered.1 Exam scoring is weighted based on assigned difficulty rankings of each question answered correctly. A minimum score of 400 out of a possible 999 is required to obtain certification. No public formulas are available for calculating scores or for equating the minimum passing score to a percentage of questions answered correctly.
The National Accrediting Agency for Clinical Laboratory Sciences (NAACLS) uses ASCP BOC pass rates as an outcome measure for accredited MLS programs, requiring a three-year rolling average pass rate of 75% for students taking the exam within one year of graduation.2 Due to this accreditation requirement, MLS programs must identify students at risk of failing the certification exam and variables that may be responsible. Establishing benchmarks is an essential step in implementing a comprehensive testing program.3
A retrospective study was conducted in 2016 to determine the extent to which a university comprehensive exam could predict successful BOC exam performance. The primary aim of the study was to determine whether a 60% cutoff was most appropriate for predicting students' likelihood of passing the BOC MLS exam, and to recommend a more appropriate passing score for the comprehensive exam if necessary. The university hypothesized that a minimum overall score of 60% on the comprehensive exam would correlate with passing the BOC exam, consistent with a minimum passing grade on traditional academic grading scales. The second aim of the study was to identify content areas of the university exam that showed high concordance with the BOC exam and those that did not.
Review of the literature showed only one study published in the field of laboratory medicine comparing results of a comprehensive exam to ASCP certification exam scores.4 In this study, the authors examined scores of students who took both exams between 1993 and 1999 (n = 44). Students who took the national exam from 1994-1999 used computer adaptive testing.5 The study showed that comprehensive exam scores >74.36% correlated (r = 1.000) with passing the ASCP Board of Registry (BOR, now BOC) exam. Lower comprehensive exam scores did not correlate (r = −0.091) with BOR outcomes.
Although there has been only one study specific to laboratory science, numerous studies have been published that showed correlation between medical residency in-training preparation exams and pass rates of corresponding written board qualifying exams.6–11 One study evaluated the American Board of Emergency Medicine in-training and qualifying examinations and found a significant correlation between in-training exams taken during the final year of residency and qualifying exam scores.7 For each point increase above the mean on the in-training exam, a 0.6 point increase was observed on the qualifying exam. Another study compared senior pathology residents' in-training exam scores to the American Board of Pathology certification exam and showed increased pass rates for each quartile group on the in-training exam.9
In addition to medical residency exams, studies in nursing and other allied health professions have also showed correlations between mock certification/comprehensive exams and national certification/licensure exams.12–14 One study in radiologic technology found a significant correlation (r = 0.693) between scores on a mock certification exam and national certification exam, with the mock exam administered approximately three months prior to the certification exam.12 Another study examined correlation of multiple variables with passing the physical therapy licensure exam, including mock exams administered after each semester in the program.14 All three mock exams showed significant correlation to passing the licensure exam. The strongest correlation (r = 0.569) was seen on the exam given after the first semester of the program.
Aside from individual scores, one study compared scores from Health Information Administration programs in the United States to determine if administration of a comprehensive exam correlated with improved overall program scores on the Registered Health Information Administrator exam.15 It showed that higher pass rates were not achieved in programs that administered comprehensive exams as part of their curriculum.
In addition to comprehensive exams, other factors have been studied to identify their effects on certification exam pass rates. These include time lapse between exam eligibility and exam completion, grade point average (GPA), age, and student motivation. A study published in 2015 examined pass rates on the ASCP BOC MLS certification exam and found that the failure rate increased with longer time lapses after eligibility.16 Exams in nursing, surgery, and emergency medicine showed similar findings.7,17,18
Grade point average (GPA) is another factor that has been identified to influence exam pass rates. Lanier and Lambert19 showed that higher GPA for science courses taken prior to program admission correlated with higher scores on the ASCP certification exam and a comprehensive exam. This study, however, did not examine the direct relationship between the two exams. In addition to science GPA, overall GPA and program-specific GPA demonstrated significant impact on variation in certification exam scores in radiologic technology, physical therapy assistant, and nursing programs.12,14,20
Another factor is age. According to Karni and Lofsness,21 an unpublished dissertation suggested that age is a confounding factor in predicting laboratory certification exam success, in that older students perform more poorly on certification exams. In contrast, many studies have shown that age is not a significant factor in predicting scores on national qualifying, certifying, or licensure examinations.6,20,22–24 Overall, results suggest that age is not a significant factor. These divergent findings regarding age as a factor may spur additional research. One final factor that may influence performance on a certification exam is student motivation.3 Two studies suggested that motivation is a contributing factor to certification exam success.4,21 Student motivation to study for a comprehensive exam may be lower if there are no consequences for achieving a low score, such as a negative impact on grade. Because MLS programs are not permitted by NAACLS to make the BOC exam a precondition for graduation, motivation to study for the certification exam is driven by certification requirements of prospective employers.
MATERIALS AND METHODS
This retrospective study included MLS students who completed both the campus comprehensive and BOR/BOC MLS exams between 2006 and 2015. The university comprehensive exam was administered in a course during the final semester on campus (prior to internship). The comprehensive exam score comprised 10% of the final course grade. Students took the comprehensive exam twice, on the first and last day of class. The higher of these two scores were analyzed. Overall exam scores were compared for all students. Data for individual questions on comprehensive exams, including question category and student score, were available only for years 2012-2015. Comparison of individual content area scores included students completing both exams during this time. The content areas used were consistent with the BOC exam: Blood Bank, Chemistry, Hematology, Immunology, Laboratory Operations, Microbiology, and Urinalysis and Other Body Fluids.
Comprehensive exam scores were collected from course archives. For years 2012-2015, questions were assigned to the content areas listed above and individual content area scores were manually calculated for each student. BOR/BOC scores were collected from the Program Performance Reports compiled annually by ASCP. Scores from the first attempt were used for this comparison. All scores were transcribed to a Microsoft Excel® spreadsheet. The university Institutional Review Board determined that formal review was not necessary for this study.
Regression data was calculated to determine the extent to which the two exams correlated. Using paired data, a correlation coefficient (r) was obtained for the total score and each content area. Bivariate regression was performed for both total and content area scores, with the independent variable being the university exam and the dependent variable being the BOR/BOC score. The squared multiple correlation (R2) was used to determine the extent to which the regression predicted BOR/BOC scores. A receiver operating characteristic (ROC) curve was created to determine the cutoff resulting in the highest sensitivity and specificity for prediction of passing the national certification exam, with passing both exams representing sensitivity and passing the university exam but failing the BOR/BOC exam representing false positives. Statistical analysis was performed using Microsoft Excel®.
RESULTS
A total of 157 students completed the university exam from 2006-2015. Five of these students did not take the BOC exam. Overall university exam scores ranged from 35-86% (n = 152), and overall BOC exam scores ranged from 287-755. Fifty-three students completed the university exam between 2012 and 2015. Two of these students did not take the BOC exam, leaving 51 students for content area analysis. Mean and standard deviation results for overall and section scores are shown in Table 1.
Mean and Standard Deviation (SD) for overall and section scores
As shown in Table 2, overall scores showed moderate correlation (r = 0.65). It is believed that test correlations above 0.60 indicate validity.25 In contrast, individual content areas showed poor correlation and ranged from 0.00 (Immunology) to 0.55 (Microbiology). The correlation coefficient for overall scores was calculated with the reduced sample size for years 2012-2015 to better compare with the section correlations. This increased correlation to 0.72.
Correlation (r), bivariate regression, and squared multiple correlation (R2) data for overall and section scores.
Bivariate regression is also shown in Table 2, along with the associated squared multiple correlations. Based on these values, most of the overall BOC scores could not be accurately predicted by the regression model, with only 42% showing correlation with the equation. For the content area with highest correlation (Microbiology), only 30% of the scores were predicted by the equation. Squared multiple correlation coefficients for Chemistry, Blood Bank, Immunology, Laboratory Operations, and Urinalysis and Other Body Fluids were less than half of those for Hematology and Microbiology, indicating that for these areas, the university exam score had little relation to the BOC exam outcome. These low correlation results may have been due to the large standard deviations in the section scores.
An ROC curve was prepared from overall exam scores (Figure 1). Based on this, a cutoff value around 55% on the university exam showed the highest sensitivity and specificity. At this cutoff, 4 students in the study (14% of all students failing the BOC exam) showed false positive results by passing the campus exam. The hypothesized cutoff of 60% showed lower sensitivity and a lower number of false positives. Using the 60% cutoff, only one student in the study showed a false positive result. This student scored 67% on the campus exam and waited approximately three years before attempting the BOC exam. Based on the known adverse effect of time lapse prior to attempting national certification exams7,16–18, one can infer that the 3-year wait was a confounding factor in this case.
Receiver operating characteristic curve of campus comprehensive exam scores in prediction of passing Board of Certification exam
DISCUSSION
The objective of this study was to evaluate the extent to which a comprehensive exam can predict performance on a national certification exam. In a previous study by Wiggers and Holton,4 it was found that comprehensive exam scores above 74.36% correlated perfectly (r = 1.000) with passing scores on the national certification exam. In this study, all students who scored above 67% on the comprehensive exam passed the certification exam. Excluding one student who waited approximately three years before attempting the BOC exam, all students scoring above 60% passed the certification exam.
The primary aim of the study was to determine if the hypothesized cutoff of 60% on the university exam is appropriate for predicting certification exam success. Although the ROC curve indicates highest sensitivity and specificity at a university score of 55%, the program will utilize a 60% cutoff. At this score, all students who pass the campus exam and attempt the BOC exam within one year of graduation should also pass the national exam. While using this cutoff will result in a larger number of false negatives, it will allow the program to detect 100% of students at risk of failing the BOC exam and direct them to additional resources.
The second aim of the study was to determine the extent of correlation in each content area. Despite moderate correlation between overall scores, correlation in individual content areas is weak. Microbiology and Hematology show moderate correlation (r = 0.55 and 0.53, respectively), while Immunology shows no correlation between the two exams (r = 0.00). This lack of correlation may be due to the variation and fragmentation of Immunology testing in clinical laboratories, leading to decreased emphasis of content during internship rotations.
There are multiple possible explanations for the weak correlation in the remaining areas. One possibility may be differences in question categorization between the campus and BOC exams. Although the ASCP BOC Examination Content Guidelines were referenced when manually categorizing the campus exam questions, some questions address multiple content areas, making categorization difficult.
A second reason for these results may be the timing and structure of the corresponding classes. The weakest content correlations are in Immunology and Urinalysis and Other Body Fluids. Both courses are taken during students' first semester in the MLS program, approximately 15 months prior to completing the comprehensive exam. In addition, these courses are a shorter duration and do not have an associated upper-level advanced course, unlike Blood Bank, Chemistry, Hematology, and Microbiology. Students may not adequately review this information prior to taking the campus exam.
A third possibility for the low correlations within the content areas is the time when students complete the comprehensive exam; that is prior to internship. Students receive further training, hands-on experience, and additional study time during internship. This learning experience may help improve scores on the certification exam when compared to scores obtained on the campus exam.
A final reason for the difference in results may be that the difficulty levels of questions on the campus exam as compared to the BOC exam differ. Because the university exam includes fixed questions while the BOC exam is computer adaptive, it is possible that the questions in each content area differ in difficulty level between the two exams.
There are several limitations to this study. First, confounding factors such as time delay, GPA, and motivation to study that have been shown to impact or influence certification exam outcomes7,12,14,16–20 are not evaluated. Information regarding these factors was not known for many students included in this study.
Second, the content covered in the comprehensive exam is not consistent throughout the years included in the study. Four different faculty members taught Chemistry and Blood Bank during the study period, leading to multiple changes in exam questions in those content areas. Questions in other content areas are also updated each year, even with long-standing faculty. In contrast, changes to certification exam questions first undergo a vetting process before being added, limiting abrupt changes to the exam in any one year.
A third limitation is the difference in exam formats. The university exam is neither criterion referenced nor computer adaptive. Instead, student score is based strictly upon the number of questions answered correctly. On the BOC exam, scoring also considers the difficulty level of the questions answered successfully. These two scoring processes are not likely to be equivalent.
Finally, the results are not applicable to other programs due to the limited sample size and demographics. The sample in this study represents only the local population that may be more homogeneous than other programs with respect to age, academic background, gender, and other baseline characteristics. In addition, there are differences in comprehensive exam questions between programs.
Based on study findings and identified limitations, there are four recommendations which may improve section correlations between the exams. The first is to develop a larger question pool for the comprehensive exam. This may better represent the computer adaptive format of the national exam, where each student receives different questions in each area. The second recommendation is to develop a more consistent method of distributing difficulty levels of university exam questions. This will help eliminate variance between the proportions of easy and difficult questions within each section. The third recommendation is to purchase mock certification exams from an outside source, such as ASCP, to replace the campus-generated exam. This may result in a better representation of BOC questions, offering improved detection of at-risk students. A final recommendation is to employ the university exam at the end of the internship, allowing better assessment of student knowledge at the completion of the program.
In conclusion, this study shows moderate correlation between university and BOR/BOC exam scores (r = 0.65). Highest correlation is seen in the content areas of Hematology and Microbiology, and lowest correlation in the areas of Immunology, Laboratory Operations, and Urinalysis and Other Body Fluids. A previous study indicated scores >74.36% best predict passing the certification exam4, while this study indicates scores >60% would predict passing with 100% specificity for those taking the certification exam within one year of graduation. This difference supports the notion that application of these results to other programs is limited. Generally, this study indicates large variations exist when comparing results between a campus comprehensive exam and a national certification exam. Future studies may consider limiting the study population to include only those students who attempt the certification exam within one year of graduation to better predict student success and evaluate program outcomes
- © Copyright 2018 American Society for Clinical Laboratory Science Inc. All rights reserved.
