Using Multiple Choice Tests in Maritime Training Assessments
Multiple choice tests are one of the oldest assessment techniques in existence. Yet they are also one of the most highly maligned. Why, then, do they continue to be used so pervasively in maritime training? Are they effective or aren’t they? This article is the first of a two-part series that looks at Multiple Choice Questions (MCQs) in maritime training.
Maritime Mentoring: International Maritime Mentoring Community - Find a Mentor, Be a Mentor
Multiple choice tests are one of the oldest assessment techniques in existence. Yet they are also one of the most highly maligned. Why, then, do they continue to be used so pervasively in maritime training? Are they effective or aren’t they? The answers to these questions are not as simple as one might hope. However, every maritime trainer needs to be aware of the strengths and weaknesses of multiple choice question (MCQs), and how to use them to greatest effect. This article is the first of a two-part series that looks at MCQs in maritime training. This first article examines some of the criticisms of MCQs, and then moves on to focus on their strengths. The second article in the series then provides some tips on how to create and use MCQs to take advantage of their strengths.
If you would like to receive a notification when the second part of this series is available (as well as for subsequent maritime training articles) and have not yet done so, please click here to sign up to receive notifications.
Multiple choice question (MCQ) exams have been used for 100 years or more to assess training effectiveness both within and outside the maritime industry. Even so, trainers and educational researchers love to hate them. There are many valid arguments frequently levied against MCQs.
One of the main arguments is that MCQs can’t test higher order thinking, but instead only test factual knowledge. This is not technically correct; MCQs can be constructed to test the ability to synthesize knowledge and apply reasoning, but it requires some skill to construct such questions. So much so, that the criticism is, in fact, true in practice. And even in the case that such questions are constructed well, they are unlikely to be as effective at the assessment of reasoning ability as are other forms of assessment including questions requiring written responses, oral responses, or demonstrations of skills.
Another criticism often levied against MCQs is that performance on an MCQ generally can only fit into one of two categories: correct or incorrect. That is, they produce a very “black and white” analysis. The problem is that this does not reflect the reality that knowledge and competencies are rarely either completely known or completely unknown. Instead, candidates are much more likely to have a partial grasp of the knowledge or competency being measured. As was the case for higher order thinking, MCQs, either singly or in groups, can be constructed to test partial knowledge, but doing so requires some skill and effort. And also as above, even if that effort is put in, they are unlikely to be as effective as other forms of assessment at revealing partial knowledge.
Another issue with MCQs is that, all too frequently, MCQs are written ambiguously, are poorly defined or confusing, and can sometimes be guessed correctly even if the correct answer is not known. These are all examples of a general class of problem with MCQs: simply said, they are hard to write well. I can personally attest to this fact. At British Columbia Ferry Services Inc (BC Ferries) there is a testbank of some 13,000 questions (and growing) which have largely been written by BC Ferries subject matter experts (SMEs) - the same people who deliver the training at BC Ferries. Despite the high level of expertise and experience of the SMEs, and despite the great deal of care and effort put into the creation of these questions, it is not unusual at all for questions to require one or more cycles of use and review before they are considered to be well written and unambiguous. As such, the question bank at BC Ferries is, in fact, quite an asset as it has been carefully cultivated, reviewed and refined (and continues to be so) through continuous use, evaluation and feedback. But it is not as easy as one might imagine.
This is certainly not an exhaustive list of criticism of MCQs. There are others. With all of these drawbacks, one might wonder why they remain so popular. While it is difficult to find compelling statistics on their use, it is clearly pervasive. For example, while I was a faculty member at the University of British Columbia, their use was quite common - especially in first-year courses with large enrollments.
So again, why are they so popular? One of the obvious reasons is that they are easy to grade, either by hand or through some automated tool. And as a former faculty member myself, I can tell you that I and most of my colleagues considered grading exams only slightly preferable to having one’s eyes gouged out with blunt sticks. This reason for their use (the ability to avoiding ever having to grade a paper) adds fuel to the fire of arguments against their use: not only do they have many shortcomings, but their main reason for use is arguably because trainers and faculty members are too lazy to grade answers to “better” assessment methods.
This may be true, but before jumping to this seemingly logical conclusion, it is important to consider some of the benefits of MCQs. They actually have many, and these benefits need to be considered carefully. MCQs bring to the table benefits that other assessment mechanisms cannot offer. And likewise, their shortcomings can be accommodated by other forms of assessment. Therefore, MCQs, as one component of a comprehensive assessment strategy, are a valuable tool. However, their value can only be realized if the assessment strategy is constructed to use each tool (including MCQs) for what it is best at. So - what are MCQs best at?
One advantage of MCQs is that they can actually be quite effective at testing well defined factual knowledge. This is important in the maritime industry because all skills and competencies are rooted in a foundation of knowledge. While this knowledge is not sufficient to ensure that a competency is held or a skill can be performed, it is necessary. That is, even if a trainee can demonstrate a skill under assessment conditions, if they do not possess the knowledge that informs that skill, then they cannot be expected to perform the skill safely under varying or novel conditions. Knowledge must be tested separately as part of any comprehensive assessment strategy, and MCQs can be very effective at testing certain forms of knowledge.
Another significant advantage of MCQs is that they completely remove subjectivity in grading. In my view this is one of their greatest strengths - one that cannot be said about the vast majority of other assessment techniques. In fact, variability in grading is a pervasive problem for any large organization. Regardless of how standardized the training may be, different trainers are going to have different ideas of what constitutes the ability to perform a skill or demonstrate a competency. This problem can be somewhat alleviated through the use of rubrics and other instructions to assessors, but even these can never completely remove the subjectivity inherent in other assessment techniques. This variability can be a weak link in the quest for standardization when trying to answer the question “do all of our employees have the necessary knowledge, competency and skills to operate safely”? The grading of MCQs is 100% objective - a valuable contribution to an assessment program.
One of the main reasons for the pervasive use of MCQs (as mentioned above) is that they are easy to grade - especially if they are done online where grading is automatic. This is an advantage which ostensibly accrues only to the trainer. However, it is, in fact, an even more important advantage for the trainee.
The reason for this is that frequent and regular assessment is a cornerstone of effective training. Assessment, as most readers will know, comes in two flavors - formative and summative. Summative assessment is used at the end of training to determine whether someone has learned the concepts and therefore is fit for duty. It is, in general, for the benefit of the training organization.
Formative assessment is different. It exists to benefit the trainee and inform the training process. It can be used throughout training to provide feedback to the trainee (and the trainer) as to how effectively the lessons are being learned. Formative assessment provides the feedback necessary to make course corrections. The more frequent the feedback through formative assessment, the earlier (and less costly) the training course correction are. The problem, of course, is that most forms of assessment are costly. They require a lot of expert time - time of the instructor - to grade. Therefore, although formative assessment is critically important, it is rarely provided in sufficient quantity.
This is where MCQs are especially useful. MCQs, especially those delivered on line, require no time to grade. This means that they can be applied as frequently as desired as long as there are sufficient questions available. In the absence of an instructor who has sufficient free time to grade frequent formative assessment, MCQs are an exceedingly valuable tool as they enable frequent, regular, and even on-demand formative assessment.
Another feature of MCQ tests is that technology can be used to randomize the exams such that no two exams will be exactly the same. For those not familiar with MCQ question randomization, most learning management systems (LMS) have the ability to dynamically create exams by randomly selecting questions from an exam database. In the case of our LMS at Marine Learning Systems, we employ a form of intelligent randomization where the difficulty and coverage of questions on every exam is normalized. This helps ensure that all candidates receive exams of the same coverage and relatively equal difficulty even though the question mix is different. In addition, questions can be tagged to indicate the vessel or equipment they pertain to. This way, a trainee undergoing vessel-specific training can indicate which vessel they are about to be deployed on. The LMS then ensures that both the general knowledge as well as the vessel-specific knowledge is tested. In addition to being able to support vessel-specific training, the ability to randomize exams has two other advantages over other kinds of assessment methodologies.
First, randomized examinations further enhance the opportunities for frequent and regular formative assessments. As discussed in the previous section, formative assessment is critical to the training process and opportunities for it are often limited. Randomized examinations allow trainees to continue to assess themselves, as often as they like, each time receiving a somewhat different examination.
Second, for summative assessments, randomized exams ensure that no two trainees receive the same exam. This helps prevent the opportunity for information about the exam to spread from one trainee to another. This is a very real problem which randomized MCQ tests do not suffer from.
Another wonderful advantage of technology-delivered MCQs is the ability for them to generate analytics which give training managers excellent visibility into the learning process. I have written about continuous improvement and analytics before.
Each time a trainee answers a MCQ on line, a record is made of that response. This data, when analysed and presented, can provide insights into how well trainees - both as individuals and collectively - are learning the materials. For example, which exams are done well and which are done poorly? How long does it take for candidates to grasp a particular concept? Which knowledge and concepts are not being learned well? What common misconceptions do our trainees have?
These and many more questions can be answered by MCQ analytics, yet they can be difficult at best (and impossible at worst) to obtain from most other types of assessment. If the goal is to identify gaps in knowledge early - before they lead to performance or safety issues, MCQ analytics can help tremendously.
So - are MCQs of value despite their obvious shortcomings? I say yes - absolutely. Should they be used as the sole technique to assess seafarers? Absolutely not. In fact, there is no single technique which could comprehensively and reliably do so. Each assessment tool offers advantages the others cannot. And while each tool has limitations, those limitations are accommodated by the strengths of the other tools. MCQs are no exception. Therefore, by combining assessment techniques in your overall assessment strategy, you can take advantage of the benefits of each.
The next article in this series is going to present some guidelines on how to create good multiple choice questions. While it is not difficult to do so, a little bit of advice goes a long way in ensuring the time you put into it yields good results.
If you would like to receive notification of the next article of this series (and subsequent maritime training articles) and have not yet done so, please click here to sign up to receive notifications.
Until then - thanks for reading!
# # #
About The Author:
Murray Goldberg is the founder and President of Marine Learning Systems (www.marinels.com), the creator of MarineLMS - the learning management system designed specifically for maritime industry training. Murray began research in eLearning in 1995 as a faculty member of Computer Science at the University of British Columbia. He went on to create WebCT, the world’s first commercially successful LMS for higher education; serving 14 million students in 80 countries. Murray has won over a dozen University, National and International awards for teaching excellence and his pioneering contributions to the field of educational technology. Now, in Marine Learning Systems, Murray is hoping to play a part in advancing the art and science of learning in the maritime industry.
Maritime Mentoring: International Maritime Mentoring Community - Find a Mentor, Be a Mentor