Learner success and job readiness is easier to assess with the rich data that can be collected in VR. Careful planning for assessment can help you generate meaningful results and feedback for your learners.
The xAPI profile includes an optional “result” property that you can leverage to extract results from the learning experience. xAPI results can be in the following forms:
- Success: a binary variable, also known as a boolean value, that tracks the “success” or “pass/failure” of an activity
- Note that success can be tracked on a more complex activity (made up of smaller, sub-activities) and cumulative “success” of a larger activity could depend on the success/failure of a combination of sub-activities.
- Completion: a boolean value criteria for completion can be defined by the learning designer and can depend on the results of a set of activities in the scenario
- Score: a numerical result which can be represented as a raw score (one number) or a scaled score (percentage)
Rubrics & Criteria Based Assessment
Defining what success looks like in a training scenario will vary from project to project, but there are some guiding principles and techniques that you can use to ensure that your assessment reflects the learning goals you set in the planning phase. In order to demonstrate how you might build a thoughtful and informative assessment, we will use a food safety example from a fictional food preparation company. This company wants to reduce contamination from 1% to below 0.2%. They currently use a Foodbot 3000 machine for food preparation.
The company’s overall learning goal is: “Learners will be able to disassemble, sanitize, and reassemble a Foodbot 3000.” This learning goal was set to address a business goal of decreasing the problematic contamination of food equipment. To read more about setting goals for your VR training program, view our blog post here.
In order to properly achieve the learning goal, employees need to demonstrate the following key performance indicators:
- Learner will disassemble the Foodbot for cleaning
- Learner must take safety precautions, specifically donning safety gloves and unplugging the machine before they begin cleaning
- Learner will loosen all 6 screws on the cover plate to reveal the inner chamber of the Foodbot
- After unscrewing the cover plate, it must be removed and placed in the sink
- When reassembling, screws on the cover plate must be in the locked position
- Learner will scrub all widgets in the inner chamber a minimum of 10 times using the Foodbot cleaning solvent
- There are 8 widgets, each needs at least 10 passes with the cloth
- Learner must check pH level of the cleaning solvent before using it
- Learner will test the surface of the Foodbot upon completion to check for contaminants
- Swab surface
- Place swab in reagent
- Read results
Example Rubric and Assessment Strategy
You can use a combination of “result” types to define what overall completion and overall success of this scenario means (note that this an example only – there are many ways to plan the assessment).
|Performance Indicator||Result Type||Score|
|Put on safety gloves||Success||2|
|Loosen screws and remove plate||Success||2 for screws|
2 for plates
|Lock screws on reassembly||Success||1 point per screw locked for up to 6 points|
|Check PH of solvent||Success||2|
|Wipe each widget 10 times||Success||0-3 times: 0 points|
4-7 times: 1 point
8-10 times: 2 points
8 widgets, maximum of 2 points each => 16 possible points
|Check for Contamination|
|Dip swab in reagent||Success||2|
|Total Possible Points||38|
Overall Success: Achieve a scaled score of over 90% or 35/38
Overall Completion : Success on ALL steps.
Breaking down assessment in this way checks for learner action as opposed to memorized knowledge. Traditionally, you might check for job readiness by having a quiz at the end of a video. Questions like “How many times should you wipe each widget?” can easily be answered. But checking if the learner actually wipes the required number of times to achieve sanitation is a true measure of success.
You will want to ensure that you have a solution in place that can handle this data and allow you to visualize it for each learner. Analyzing results will allow you to refine the training as you go so that the skills that need the most focus are highlighted.
Teach Mode vs Test Mode
The above noted assessment strategy will be a lot more meaningful if you create a version of the training that is unguided. At Motive, we refer to this as “test mode”. Generally, trainees experience a guided, “teach mode”, first where they can learn the procedure or task and then attempt test mode to see if they are able to complete the task with no help. Multiple repetitions of teach and test mode may be required before trainees can achieve completion and success in test mode.
Assessing the ways your learners perform will be key in determining the job readiness of your employees. Using your own key performance indicators alongside the data extracted from your employees VR training scenarios will help you better understand your employees training success.
Stay in the Know
Want to stay up-to-date with what is going on in the world of immersive training? Subscribe to the Motive Blog.