Assessment in Maritime Job Training and Familiarization
This is the first in a series of articles discussing current and best-practice assessment methods in maritime job training and familiarization. This first article discusses the limits and purpose of assessment. Subsequent articles will look at assessment reliability and validity, professional judgement, the goals and topics of assessments, and the merits of specific assessment practices in the maritime industry.
This is the first in a series of articles discussing current and best-practice assessment methods in maritime job training and familiarization. Specifically, I am speaking about the testing vessel operators administer to officers and crew to determine whether they are sufficiently prepared to perform their duties on their assigned vessels safely and efficiently.
In planning this series of articles, I have reminded myself how much there is to say about assessment. It is a complex and rich topic and I expect to generate a number of articles on the subject. It is my hope that these articles will provide a basic understanding of assessment principles - an understanding that everyone involved in maritime training should have.
This short introductory article discusses the limits and purpose of assessment, creating a foundation for subsequent articles on assessment. Subsequent articles will look at assessment reliability and validity, professional judgement, the goals and topics of assessments, and the merits of specific assessment practices in the maritime industry. Please click “follow this blog” to receive notification of those upcoming articles.
It is often the case that we give a great deal more thought to training than we do to assessment. This is unfortunate because training cannot be successful (or at the very least cannot be shown to be successful) without an objective and comprehensive assessment process. Your training may be excellent at this moment, but without quality assessments you have no way of knowing this for sure, and you won’t have the tools necessary to keep it on track and continuously improve it. We need to realize that assessment is a critical and necessary part of training, not just something we do at the end in order to apply a credential. It is a primary safety and operations tool to:
- Determine whether a candidate is fit for duty
- Determine what gaps in knowledge and skills exist for a candidate
- Provide key performance indicators for your organization to be used as a basis for analysis and continuous improvement.
Assessment is an activity that very few organizations do well, and fewer still understand well.
There is good reason for that. It is not a cut and dried science. Assessment is often based on intuition rather than concrete fact. Assessment tries to peer into the future of an individual and answer the question “does he or she have the knowledge necessary to perform when called upon”? “Can he or she perform this skill”?
But how can we truthfully say? After all, so much of what we would like to assess is hidden from view inside the head of the candidate. And while the candidate may be able to demonstrate a skill under one set of conditions, what if those conditions change?
But as hard as it is to assess someone’s skills and knowledge, true assessment in the maritime industry needs to do more than that. It needs to asses their cognitive abilities as well. Can the candidate assimilate disparate information and synthesize it into a plan of action when presented with unexpected events? There is simply no way to know for sure. As such, it is something that the academic community has been wrestling with for ages, and many consider it to be as much art as it is science.
Having said this, there is still much we can do to improve the validity and reliability of the assessments we administer on board. A little bit of knowledge and planning can go a long way.
One of the first things we must realize when designing an assessment program is that, as I alluded to in the paragraphs above, full assessment is an impossibility. You cannot devise an assessment program which will completely assess a candidate’s knowledge or abilities. Instead, at best, assessment is a statistical process - much like an audit, that samples bits of knowledge here, or components of an ability there, and assigns a score which is an extrapolation of the sample taken. If the sample size is very small or the assessment techniques are flawed (or both, as is sometimes the case), then the margin of error is going to be very large rendering the assessment inaccurate much of the time. But even with a reasonable “sample size” and sound techniques, assessments can never be treated as absolute indicators. Some candidates will assess well and perform poorly, while others will assess poorly and perform well. This begs the question “If assessment is flawed, then why do we assess”?
Having been a university faculty member for 10 years (and one who really dislikes grading exams) I have often asked myself that question. But it turns out there are very good answers. I will list two of them.
First is the obvious answer. In the absence of any other indicators about a candidate, an imperfect assessment is usually better than no assessment at all. Some form of assessment is required to obtain an estimate of gaps in knowledge and abilities as well as the prospects for future performance. Because any one assessment is imperfect, we should not treat its results as an absolute indicator of knowledge or competence. Having said that, even an imperfect assessment provides data that, when combined with professional judgement, can be used to make decisions.
The second answer is, to me, the most important. If nothing else, assessment is incentive to learn. Every candidate knows that successful assessment performance is their key to employment. They are also keenly aware that not everything they need to know or do will be tested. But in the absence of knowing specifically what will be tested, they are faced with having to learn as much as they possibly can about all testable knowledge and skills. There is no greater incentive to deep and broad learning. This is an important fact to keep in mind because anything you do to purposely or inadvertently “teach to the test” or make candidates aware of specifically what their assessment will consist of or cover, takes away their incentive to learn as much as they can. The implications of this statement for assessment techniques will be discussed further in subsequent articles on the subject.
We need to always keep in mind that assessment is a largely imperfect exercise. Knowing this should cause us to place assessment results in perspective, to ensure our assessment techniques create an incentive to learn, and to take a keen interest in other indicators of our mariners’ abilities. Most importantly, we should treat assessment as a tool which informs conclusions, not a conclusion in itself.
Subsequent assessment articles will look at how we assess, what we assess, and specific techniques to make our assessments reliable and valid. Please click “Follow this blog”, below, to be informed of these articles as they are released.
# # #
About The Author:
Murray Goldberg is the founder and President of Marine Learning Systems (www.marinels.com), the creator of MarineLMS - the first learning management system specifically for maritime industry training. Murray began research in eLearning in 1995 as a faculty member of Computer Science at the University of British Columbia. He went on to create WebCT, the world’s first commercially successful LMS for higher education; serving 14 million students in 80 countries. Now, in Marine Learning Systems, Murray is hoping to play a part in advancing the art and science of learning in the maritime industry.