Implementing Continuous Improvement in Familiarization Training - Part 3
Continuous Improvement (or CI for short) is the process of continually analysing the performance of some aspect of operations, and then applying changes intended to improve that performance. This is the third and final installment in a series of articles intended to introduce you to CI, and to give some practical tips for using CI in your maritime organization to improve training effectiveness and efficiency.
Implementing Continuous Improvement in Familiarization Training - Part 3
Continuous Improvement (or CI for short) is the process of continually analysing the performance of some aspect of operations, and then applying changes intended to improve that performance. It is a critical component in maintaining the health of any operation and can be applied to almost any activity. This is the third and final installment in a series of articles intended to introduce you to CI, and to give some practical tips for using CI in your maritime organization to improve training effectiveness and efficiency. Implementing a modest CI process for your in-house training is neither expensive nor difficult, and even a small program can produce a tremendous ROI in safety, efficiency, trainer engagement and trainee satisfaction.
In the first article of this series, (which can be found here) I introduced the idea of applying the principle of CI to your organization’s maritime job and familiarization training. That article provided an overview of CI, talked about why it is important and very timely, and introduced the cornerstone of any CI process - measurement and KPIs or “Key Performance Indicators”. In the second article of this series (which can be found here), I discussed useful training-related KPIs which you can implement now - whether or not you employ a learning management system in the delivery and management of your job and familiarization training. To get the most out of this article, I do suggest you take a few minutes to go back to those posts before progressing with this one if you have not read them already.
In this article, I complete this series by talking about specific key performance indicators which are likely to be available to you if (or once) you employ a learning management system in the delivery and management of your job and familiarization training. These KPIs provide deep insight into learning activities and performance, and can be used as part of the foundation upon which you implement continuous improvement.
If the first two articles in this series are fresh in your mind, please feel free to skip directly to the section entitled LMS-Enabled KPIs, below. For those of you who would rather not go back and read the first two articles in this series, below is a quick refresher on continuous improvement and KPIs.
CI is the continuously repeating process of reflecting on some activity (training in our case) and identifying and eliminating suboptimal practices in a series of incremental, rather than drastic steps. In the maritime training environment the focus of CI should be on improving the effectiveness of training foremost, with a secondary focus on improving efficiency.
This topic is especially timely right now. Vessels, equipment and job routines in the maritime industry continue to become more and more complex and sophisticated. As a result, deeper knowledge and more specialized skills are required to operate safely - knowledge and skills which can only be taught by vessel operators or equipment manufacturers. However, familiarization training in most organizations has not changed much over the years and as a result there is an increasingly large gap between the sophistication of operational training and the knowledge needs of modern seafarers. A program of continuous improvement for operational training is necessary (and possibly sufficient) to close this gap.
Key Performance Indicators
The focus of the second and this third article is Key Performance Indicators. As I introduced in the first article of this series, Key Performance Indicators (KPIs) are the foundation of any CI process. KPIs are measurements used to evaluate effectiveness and efficiency. This is critical - unless you have a way of measuring whether changes made in the name of CI have created an improvement in outcomes (effectiveness and efficiency), you’ll have no way of telling whether you are moving closer to, or further away from, your goal. KPIs provide that information by measuring the success of every change you make.
As a reminder, KPIs should minimally satisfy the following requirements:
- They must be aligned with corporate goals - otherwise, what is the point of measuring them?
- They should react reasonably quickly to changes in training so that when we do make changes, our KPIs will tell us whether they were effective or not.
- They must track something that is actually “measurable”. This may seem obvious, but some important things can be difficult or impossible to measure.
- They must track something you have some control over. Otherwise, although they may be interesting to track, they are not helpful in the process of tuning training outcomes.
As with many data-driven systems (and KPIs are just that - data), technology is your friend here. If many or most aspects of your training activities are delivered and managed with a computer-based learning management system (or LMS), you can use that system to report on metrics about the learning process which cannot otherwise be practically obtained. In fact, metrics are one of the greatest strengths and benefits of using an LMS and that is one of the reasons that most large training organizations use an LMS. However, LMSs are only now starting to be used in maritime familiarization and job training. If the maritime industry follows the world-wide trend in other industries (and I see no reason why it won’t), LMSs will become more and more of a factor in maritime training. Therefore, it is time to begin thinking about how an LMS can improve the process of continuous improvement in your operatonal training.
In order to understand how an LMS can generate the KPIs we are interested in here, it is important to have a basic understanding of what an LMS is. Here is a short, edited definition from Wikipedia:
A learning management system (commonly abbreviated as "LMS") is a software application for the delivery, management, tracking, and reporting of training and assessments. A robust LMS should be able to do the following:
- Assemble and deliver standardized training content rapidly
- Consolidate training initiatives on a scalable web-based platform
- Personalize content and enable knowledge reuse
- Centralize and automate training administration
Note the emphasis in the definition on “management, tracking and reporting”. This is what we are concerned about here. However, from the trainee and trainer perspective, the LMS is really about delivery of learning materials and assessments. An example of its use in maritime job and familiarization training is that of the British Columbia Ferries Services Inc. They use an LMS called MarineLMS (full disclosure: MarineLMS is the product of Marine Learning Systems, the company I work for). BC Ferries uses MarineLMS to deliver their familiarization learning materials and examinations to trainees, all customized to the route and vessel (or terminal) the person is training for. As an aside, BC Ferries has created a very comprehensive process for familiarisation that I think all maritime organizations can learn from. If you are interested, you can read more about it here.
Now that you understand that LMSs are computer-based systems involved in the delivery of training materials and assessments, you probably are beginning to get a glimpse of their potential in generating trackable KPIs. Let’s take a look at some of these “LMS-enabled KPIs” and discuss how they can be used in a process of continuous improvement.
LMSs can be used to deliver exams, and while doing so, are able to record various kinds of metrics. The exam metrics I find to be the most compelling are per-question statistics. Questions which are asked in online exams are recorded by the LMS to generate metrics on how often each question is correctly answered. Some LMSs will even indicate the distribution of answer choices for each question in the system. This is outstanding information to have because it highlights hidden weaknesses in your training which are unlikely to otherwise be apparent. For example, it may be the case that in general, your trainees are passing exams with flying colors. However, despite their overall performance, they may routinely be incorrectly answering a small set of very important questions. The knowledge that these questions test may be critical to safe operations. If not for the metrics generated by your LMS, this would be a very difficult fault to uncover on an organization-wide basis, yet may represent a real risk to safety or operational efficiency. Your LMS will alert you to these issues.
So - once you have found a set of questions which are routinely answered incorrectly, how do you remedy the situation? First - you need to examine the question to determine whether the question is at fault by being incorrectly worded or ambiguous. If that is the case, then you simply need to repair the question. However, if the question looks acceptable, the problem is likely in the learning materials or training program. The next step in this case, if your LMS supports it, is to look at the average distribution of answers for that question. Is there one incorrect answer that most candidates are choosing instead of the correct one? Or are the incorrect answers uniformly distributed amongst the various possible choices? This metric can help you identify common misunderstandings and be very helpful in updating the learning materials to correct the issue.
In either case, this metric is very useful at bringing to the surface any hidden training issues, allowing you to correct them before they result in a loss.
Click here to view a sample of question performance metrics.
Click here to view a sample of per-question metrics including incorrect answer distribution from an LMS.
One of the most informative and useful metrics delivered by many LMSs is the learning page access log. This metric shows every learning page in the system and indicates, on average, how long learners are spending on that page. Often the metrics can be segmented to show pages pertaining to a particular job function.
This can be useful as a general metric to inform you on how much time your trainees are spending learning the web based content. Trends in this number can be watched to determine whether your training time expectations are valid, and to alert you to changes which could be caused by candidate differences or changes in the training materials.
More importantly, this metric can be used to identify pages whose access times represent outliers. For example, if most learning pages fall into a range of 30 seconds to one minute (in terms of how long trainees are spending on that page), but you have a small number of pages at 10 seconds, and another small group at three minutes, this could be alerting you to an issue which needs attention. If the page access duration is unusually short, it may be the case that the content there is either not compelling or not useful. Of course, it could also just be that there is very little content on that page. In either case, visiting that page and speaking with a couple trainees will usually make it immediately apparent whether there is a problem, and if there is, how it can be resolved.
More telling are pages whose average time on page is unusually high. Typically these pages contain content with is complex or confusing to learners. These pages often require special attention to either break up the content, or to provide more digestible explanations.
Whatever the results, these metrics provide a window of insight into how trainees use the learning materials which you would otherwise have no insight into at all. This is a great example of how technology can be our friend when analyzing training activities for the purpose of continuous improvement.
Click here to view a sample of learning content access metrics from an LMS.
Another metric that is easily generated by most LMSs is that of overall exam performance. This is a metric which you can maintain separately without an LMS, but an LMS does the work for you and usually presents tidy and flexible reporting options.
Typically, for each exam in the system, the LMS will tell you the average score that trainees have been getting on that exam. This simple metric allows you to “keep your fingers on the pulse” of assessment performance in your organization. It is a useful audit metric that can be tracked over time to watch for increases or decreases. Such variations must be examined carefully since they could be due to a number of factors including changes in the level of knowledge of incoming candidates, changes in the exams themselves, changes in trainer preparation and performance, changes in training curriculum, and so on. All of these are important and may require attention. Regardless of the cause, they provide insight which should be tracked. Numbers which are (or are trending to be) too high or too low in relation to organizational goals indicate a problem with training or assessment and must be dealt with.
Click here to view a sample of overall exam performance metrics from an LMS.
Most learning management systems include self-evaluation (or self-tests) as one of their features. Self-tests are typically automatically graded, but no grade is recorded and the trainee is not evaluated on the basis of any self-tests they take. Instead, they are used as a formative tool by the trainee to determine whether they posses the knowledge required to progress to the next learning module or exam.
Because self-tests are used for a completely different purpose than summative assessments (such as “final” exams to test competency), metrics about self-tests tell us something very different than the metrics discussed above. Summative assessment metrics tell us whether the trainee eventually learned the material. Self-test metrics, on the other hand, tell us how quickly the trainees learned the material.
The reason for this is due to the typical usage patterns of self-tests. Trainees typically first review the learning materials, and then when they believe they know it reasonably well, they move on to the self test to validate their belief. If all goes well, they move on to the next module (or the summative test if they are finished). If, instead, the self-test yields a low score, they return to the learning materials and then, later, re-attempt the self test. This latter pattern will create a lower average score on the self-tests.
Keep in mind that self-test scores are relative - they can only be examined by comparing them to either past experience, or to experience on different self-tests. There is no specifically desirable range for self test scores to fall into, though very high or very low scores are problematic for obvious reasons. However, self test scores which change significantly, or self test scores for a program which are much lower than the average of other programs are telling us something about that program or the test and should be investigated.
In general, a learning management system will provide far deeper insight into the efficiency and effectiveness of our training program than we could ever hope for without the use of an LMS. In some cases, these metrics can highlight significant shortcomings in training which can lead to safety or operational problems. This is one of the benefits of LMSs which have made LMSs very common outside the maritime industry, and which is now beginning to make them more common within the maritime industry.
Having said that, most vessel operators do not yet employ an LMS in training. Until such time as they do, there are still some excellent KPIs which can be tracked and analysed as indicated in the second article of this series. Start there, and eventually you can make your way here.
# # #
Let me write about your story so we can all learn from it! Is your organization leading the way in job and familiarization training? I’d like to write about it. Or, do you have an example of a poor familiarization experience or practice that we can all take lessons from? I would like to write about that too (without naming you or your company, in this case). Contact me by email at Murray@MarineLS.com. You have a familiarization tale to tell. You can benefit everyone by sharing it.
About The Author:
Murray Goldberg is the founder and President of Marine Learning Systems (www.marinels.com), the creator of MarineLMS - the first learning management system specifically for maritime industry training. Murray began research in eLearning in 1995 as a faculty member of Computer Science at the University of British Columbia. He went on to create WebCT, the world’s first commercially successful LMS for higher education; serving 14 million student in 80 countries. Now, in Marine Learning systems, Murray is hoping to play a part in advancing the art and science of learning in the maritime industry.