We’ve all heard the stories, and many of us have seen it first-hand: the completely ineffective assessment practice of giving the same multiple-choice exam to people over and over. The only time this is ever acceptable is when all of the people writing the exam are doing so at the same time and are supervised to ensure they are not sharing answers. However, giving the same exam to different people over a period of months or years ensures that exam grades are going to go up over time – sometimes quite markedly. Sadly, this is not because people are learning more. It is because the people who wrote the exam earlier are sharing the answers with those writing it later. This is no surprise, and it creates real issues.
The most obvious problem with the practice of exam reuse is that the assessment results become meaningless. They are not an accurate reflection of the trainee’s knowledge, and they cannot be used as a measure of training quality. And, of course, with exam answers in hand, trainees will bypass the learning materials and go straight to the exam – rendering even well-designed training useless.
Possibly the most damaging effect, however, is that the training program loses credibility and respect in the eyes of trainees. To a trainee, reuse of exams signals laziness and a lack professionalism in the training organization. They will respond in kind.
Thus, the lesson here can hardly be overstated: never give the same exam twice. If you are doing so, now is a great time to stop.
Does This Mean Constantly Creating New Exams?
No – though it may mean some changes. Most people who are reusing exams are doing so because they are delivering paper-based exams or using poorly built computer-based training. In the case of paper exams, it is difficult and expensive to create a new exam each time it is given. In the case of poorly constructed computer-based training, the program is “static”, so the same exam is delivered to every person who completes the training.
The alternative is to randomize your exams. With exam question randomization, no two trainees will get the same exam. Instead, they will each get a different mix of exam questions, and in a different order. This kind of exam randomization requires a learning management system (LMS). But fortunately, nearly all modern professional LMSs support exam question randomization. So if you have access to an LMS, there is a good chance that you already have the capability.
What Does This Mean for Assessment Consistency?
The idea of giving each trainee a different exam may raise the concern that we lose consistency in our assessment practices. After all, it is critical that we measure everyone by the same standard. However, making it easy to share answers by reusing exams completely distorts the playing field – so we already have a problem. Additionally, exam randomization does not have to be quite as “random” as it sounds. Instead, done correctly, controls can be applied to ensure that the same competencies are covered, to the same degree, with questions of roughly equal difficulty. How do we do this?
Check back here next month and we will see in Training Tips for Ships! Until then, safe sailing!
(As published in the September 2019 edition of Maritime Reporter & Enginerering News)
