A Comprehensive Assessment Plan for Medical Laboratory Science Programs

INTRODUCTION

The role of medical laboratory scientists in the clinical laboratory has changed dramatically over the years. Thus, substantial and ongoing modifications are introduced to the curriculum of educational programs in order to prepare graduates for technologic and regulatory changes in laboratory medicine. Additionally, assessment of program learning outcomes in higher education has become increasingly important, as the educational effectiveness of many academic programs has been debated and survival of academic programs has been questioned. More than ever, academic programs are held more accountable for the use of financial and human resources and the level of student achievement associated with well-developed educational outcomes.

In the United States, most medical laboratory science (MLS) and medical laboratory technician (MLT) academic programs are accredited by the National Accrediting Agency for Clinical Laboratory Sciences (NAACLS).¹ Like other accrediting agencies, NAACLS adopted an outcome-based approach in the accreditation process.² Since then, MLS and MLT programs have taken various approaches at integrating, implementing, and assessing outcome-based teaching and learning. This paper provides a sample comprehensive assessment model designed to provide guidance to MLS and MLT programs for more effective program assessment and accreditation processes while assisting programs in revising and updating their curriculum.

BACKGROUND

Colleges and universities are required to evaluate the effectiveness of instructional programs using a variety of methods. A well-developed assessment plan involves gathering and analyzing both quantitative and qualitative data that demonstrate student achievement. Accreditation standards, on the other hand, examine instructional delivery, adequacy of facilities and equipment, standardized tests, completion rates, job placement rates, results of licensing examinations, evaluations by employers, and follow-up surveys of alumni.

Simply, assessment is the process of gathering evidence of student learning, reviewing the evidence to determine if the desired transfer of knowledge occurred, and using this evidence to alter the direction of the curriculum. For the assessment to be effective, it should be non-anecdotal, systematic, and periodic. It should include all students or a representative sample, and use direct and indirect measures of learning. An effective assessment strategy also uses multiple measures and assessors to determine which aspects of the program are successful and which are not.

MISSION AND GOALS

Programs are required to have a written mission statement that states the program purpose and goals that guide the activities, policies, and allocation of resources.³ The mission statement represents the core values of the program and must be aligned with the mission and goals of the institution. It assists in developing the organizational structure and program learning outcomes. Goals describe broad learning outcomes and objectives that support the program mission.⁴ The program goals are reviewed annually in conjunction with the overall objectives and program assessment plan and are communicated to faculty, students, and staff, as well as to prospective students, student sponsors, and the public. A sample of a general mission statement of an MLS program may be found in Box 1.

CURRICULUM

The curriculum of the MLS and MLT programs should be consistent with the mission of the program and appropriate to meet student professional needs. The curriculum and course objectives should also meet the program goals. The instructional materials, exercises, and teaching methods should contribute to the students’ mastery of the content and skills identified in the course objectives. The program curriculum, therefore, should demonstrate a natural progression of content and skills development from basic to more complex laboratory diagnostics. To do this, the program should provide the students with materials that are relevant, comprehensive, and current. The sequence in which the courses are delivered in programs must be logical.

An MLS or MLT program curriculum provides students with knowledge of the fundamental nature of human diseases and the strategies and procedures to characterize and differentiate such disorders using laboratory investigation and analyses. In MLS and MLT programs, students typically complete supporting and foundation courses during the first phase of the program. In the latter phase, students complete a variety of MLS and MLT courses that include on-campus coursework and laboratory experiences either on campus or at affiliated clinical training sites. Programs maintain a balance of courses in didactic and student practicum experiences in four major core scientific disciplines: clinical chemistry, medical microbiology, hematology, and immunohematology or transfusion medicine. The curriculum covers all three phases of laboratory testing, pre-analytical (all aspects of specimen integrity from the time the physician places the order to the time the specimen is received in the laboratory), analytical (all processes associated with specimen testing once received in the clinical laboratory), and post analytical (all processes involved in result reporting and delivery).

Striving to prepare competent medical laboratory scientists and technicians, MLS and MLT programs establish academic policies and procedures to ensure that program graduates meet entry-level competencies in all four of the core disciplines and possess the knowledge and critical thinking to successfully pass a professional certification examination. All graduates are expected to possess the manual skills and dexterity to perform moderate to high complexity testing within a clinical laboratory. Upon completion of the academic requirements of the program, graduates are eligible to take the national laboratory science certification examination of the Board of Certification (BOC) of the American Society of Clinical Pathology (ASCP).⁵

EDUCATIONAL OBJECTIVES

The program should have objectives or student learning outcomes (SLOs) that clearly support the overarching mission and goals of the program.⁶ The objectives should be measurable and align with the quality assurance program of the institution. The objectives should also be consistent with the defined level of the degree as outlined by NAACLS accreditation standards and the BOC certification exam. Objectives for the didactic and clinical aspects of the program must address the cognitive, psychomotor, and affective domains.⁷

An objective (or SLO) needs to specify who is to perform (student), what action they are to take, the criterion associated with how the action will be performed, and some result that must come from their action. An educational objective for a program should meet the following criteria⁸:

• • Should be written in terms of what students can do at the end of a course/program

• • Should keep it short and simple (KISS)

• • Should be specific, measurable, attainable, realistic, and timely (SMART)

• • Should identify a target performance level for success (for example, 70%)

• • Should be a manageable and meaningful (M&M) assessment process

• • Should use Bloom’s Taxonomy and acceptable active verbs (create, analyze, demonstrate, etc.)

• • Should be written in the positive sense instead of the negative

• • Should reflect measurable standards or address the basic knowledge and skills that students are accountable for

• • Should combine higher-order thinking skills and supporting or enabling skills

An objective is the specific learning behavior that the student should demonstrate in the context of achieving the program or course goal. Because an objective is measurable, programs are able to determine if the desired student learning and skill development is achieved to meet program goals. A list of sample objectives that can be used in the MLT or MLS program is shown in Box 2.

It is acceptable to use more than one objective for each goal. Table 1 shows the relationship between the program goals and objectives.

CURRICULUM MAP

A curriculum map is a process to align courses with desired goals and program outcomes in an attempt to identify core skills and content, processes employed, and assessments used for each subject area and grade level.⁹ It is a planning tool for examining and organizing a curriculum that allows the academic programs to determine how content, skills, and assessments will unfold throughout the program.⁹ It also serves as a roadmap for an in-depth view of the distribution of learning opportunities, the level of student learning in the courses that are taught, and how they blend together in a chronological and coherent manner.^10,11

To develop a cohesive curriculum map, program faculty must first engage in dialogue about how the students learn, what pedagogies or educational experiences demonstrate value to the learning process, and how the courses and experiences are intentionally built to achieve the outcomes.¹¹ The curriculum map is organized as a matrix with one column for each objective and one row for each program course. Program administrators should begin with the program’s intended objectives and match them to the required, supporting, and elective courses and internship coursework. The courses are marked to indicate the desired learning outcomes. Introduction to the outcome is marked “I,” followed by development “D,” and ultimately “M,” to demonstrate mastery. Program administrators should review the curriculum so that a learning outcome (objective) is introduced, developed, and then mastered. The curriculum map should be revisited during the assessment cycle, making changes as disciplines and desired outcomes evolve. Table 2 shows an example curriculum map for a baccalaureate degree in MLS.

ASSESSMENT PLAN

Outcome assessment allows a program to systematically examine the alignment between student learning, instructional or institutional expectations, and instructional activities. An effective assessment plan is one that is closely linked to the curriculum and uses available information and resources to the greatest degree possible.⁸ Program assessment should focus on assessing student learning and experience to determine whether students have acquired the skills, knowledge, and competencies associated with the program goals and objectives. The plan should include ways to assess competencies that reflect a range of critical thinking skills, from basic knowledge to high-level application of knowledge and skills. The plan should also include the program expectations for the student’s level of achievement for the SLOs. Targets and benchmarks that will be used in evaluating the student performance for each of the assessed SLOs and/or performance indicator, ie, 80% of students are expected to achieve a score of 3 or more for this outcome/indicator, should be described. Students who know their learning expectations are more successful. Similarly, faculty who have a clear idea of what they expect their students to learn can align their instructional activities to these outcomes. Therefore, clearly articulated outcomes are essential to student learning and achieving program goals.

Once the mission, goals, and educational objectives are established and approved by the program faculty and administration, an effective assessment plan can be implemented as a systematic process. It can span a period of two to three academic years, assessing four to five learning outcomes each year. Table 3 shows an example of an assessment model.

ASSESSMENT PRACTICES

There are many ways to assess SLOs. An assessment matrix is a useful way of linking goals and objectives to assessment tools and course curricula. As a tool, an assessment matrix may identify direct and indirect methods of measure. Direct methods require students to display their knowledge and skills as they respond to the instrument. Objective tests, essays, presentations, and classroom assignments are examples of direct methods of assessment. On the other hand, indirect methods, such as surveys and interviews, ask students to reflect on their learning.⁵

Other opportunities to directly assess learning outcomes include capstone courses, internships, and portfolios. Capstone courses are designed to enable students to review, evaluate, integrate, and synthesize information and skills gained from other courses in the program. These courses are the best place to assess program goals. Internship experiences, when they occur at the end of the program, may serve as capstone experiences. An accumulation of student-produced work collected in a portfolio may be designed to assess a student’s attainment of program goals. Portfolios that contain early or unrevised work and revised work later or done later can assess the student’s growth and skill development.

Program objectives may be assessed through assignments embedded in required courses. Writing assignments such as summaries or reports and oral presentations may be used to assess a student’s mastery of course content. In these activities, their writing, reading, critical thinking, or speaking skills are evaluated. With planning, a single assignment or project can be designed to assess several program goals. Additional examples of assessment opportunities are case study analysis, journal article critiques, and oral presentations.

A tool that may be used in the direct method of assessment is a standardized scoring rubric. A rubric is a criterion-based scoring tool that specifies levels of achievement (eg, exemplary, satisfactory, and unsatisfactory) for each dimension of the outcome.¹² In essence, rubrics translate outcome statements into performance indicators as measures of the student’s level of achievement as well as weaknesses.¹¹

Rubrics may be used to evaluate portfolios and other forms of assigned work.¹² Like program outcomes, rubrics should be carefully developed to ensure that they represent and clearly define the standard of quality, criteria, and student levels of achievement. The rubric should be shared with the student.

An external advisory committee composed of individuals who are working professionals in the field is particularly useful in curriculum review. These individuals are generally practicing in the field and are familiar with new technology and other advances or changes impacting clinical laboratory science education and practice.⁷ Often, the advisory committee can give valuable insight by reviewing the goals and objectives to help plan future directions of a program.

Professional agencies such as NAACLS and ASCP provide an opportunity to tie the curriculum to a national standard serving as a valuable assessment technique. Individual student and overall program median scores from the ASCP BOC certification licensing exam or other comparable exam can serve as a valuable direct program assessment technique. Successful results demonstrate credibility of the curriculum through student performance when compared with similar educational programs.⁵

The Association of American Colleges and Universities (AACU) provides an extremely helpful assessment tool for educators. AACU launched a nationwide initiative that examines direct evidence of student learning. The initiative is called Valid Assessment of Learning in Undergraduate Education (VALUE) and is aimed to develop rubrics to assess SLOs. The VALUE rubrics include Inquiry and Analysis, Critical Thinking, Creative Thinking, Written Communication, Oral Communication, Quantitative Literacy, Information Literacy, Reading, Teamwork, Problem Solving, Civic Knowledge and Engagement – Local and Global, Intercultural Knowledge and Competence, Ethical Reasoning and Action, Global Learning, Foundations and Skills for Lifelong Learning, and Integrative Learning.^11,13

Surveys are a common indirect method that may be used to assess program effectiveness. The use of surveys is a way to gain a perception about student learning or performance of educational preceptors that may directly impact a program.⁹ Program faculty may develop the questions that ask for this information and send to current students and employers of program graduates. There are many types of surveys. The ones most often used are graduate surveys, employer surveys, and student surveys. Results sometimes raise issues that would not be apparent in other types of assessment and provide students as well as external stakeholders an opportunity to recommend or influence improvements in the program.

Conducting focus groups or interviews is another indirect method to assess program effectiveness and can be used to address a specific issue or perception of a student group or stakeholders, such as employers or alumni. This method, however, is often time-consuming, and analysis of the data can be cumbersome and difficult. Nevertheless, when combined with direct methods of assessment, it may provide valuable insight into student and/or preceptor attitudes and practices that directly affect student performance and outcomes assessment.

ASSESSMENT AND GRADING

In outcomes assessment, the terms scoring and grading have different meanings. Scoring refers to the process of marking an assessment instrument to collect data about how well the course and instructional methods are achieving the desired outcomes. Grading is the process of marking an assessment instrument for assigning a student a grade for the course. Traditional grading that offers one “score” to represent the sum total of a student’s performance across a host of outcomes does not provide the detailed and specific information necessary for linking student performance to program objectives and, ultimately, to improvement.⁵ Because grades do not represent student performance on individual or specific learning goals or outcomes, they provide little information on the overall success of the program in helping students attain specific and distinct learning objectives of interest.

ANALYSIS OF ASSESSMENT PLAN

Assessment offers a useful insight into program effectiveness and improvement when carefully analyzed and interpreted in the context in which it is collected. After the assessment cycle, data and findings for each educational objective should be thoroughly analyzed and described to include strengths and weaknesses.¹³ The faculty of the academic program should prepare a narrative describing the assessment process, including how the data are analyzed and the process in which assessment strategies are implemented. Finally, the assessment plan should be used to determine whether the goals of the program are being met and therefore the identification of the necessary actions needed to address any concerns or deficiencies. Upon completion, a plan to implement changes in the curriculum, instructional methods, or methods of assessment should be determined, thus closing the loop.

CORRECTIVE ACTION PLAN

Keeping the program’s resources in mind, the faculty of the program should consider the actions that would be effective to enhance student learning and then systematically follow up on the efficacy of the implemented changes. This is a lengthy process and can take several years. Accurate data collection and regular faculty engagement and discussions concerning program assessment are vital for effective and continued assessment practices. As with initial program curriculum mapping, it is important to utilize an effective method of documentation for the implementation of specific changes or corrective actions that have been identified to improve student outcomes.

One method for the documentation of plans for change or corrective action can be organized in a matrix. The matrix is based upon the program faculty’s review of the program strengths and weaknesses and the program’s interaction with other areas of the institution, whether it be a community college, hospital-based program, or university. As with all continued assessment practices, upon implementation and completion of the desired change, the program faculty must evaluate the efficacy of those actions to assess whether the changes achieved the intended results. Follow-up action does not necessarily mean drastic change, rather an ongoing attention to those actions. Sometimes a slight change in the assessment practice of the learning outcome or the teaching method can result in dramatic changes in the actual student success or simply an improved demonstration or measurement of that success. Table 4 represents a proposed action plan that explains the recommendation to take place and the rationale for it. The plan should also capture in enough detail the action to be taken, by whom, and when.

A well-established, quality assessment process also includes faculty development, policy adjustments, or curriculum changes. However, the assessment process can also include drawbacks if no actions are taken following compliance with external demands. It is not effective to simply have an assessment plan; it is essential that the process is ongoing, provides value, and is clearly embedded into the culture and practice of the program and institution to enhance student learning.

CONCLUSION

Entry-level medical laboratory scientists and technicians are expected to apply principles learned in the educational programs for use in a broad spectrum of laboratory activities in the work environment. Thus, educational programs must prepare graduates for the workplace that is rapidly changing because of new technology in the healthcare system. The quality of education must be readily measured using a comprehensive assessment plan that complies with the institutional mission and goals and follows accreditation standards and best practices.