Dolly: Brandon ranted about ineffective evaluations while I was holed up under the 2 feet of snow here. The snow is still here, but my Internet is finally back. His comments about evaluation made me wonder what I consider effective evaluation to look like. Cumulative evaluation should be as authentic as possible. If you want someone to be an effective salesperson, you’d better identify what competencies are required and then have them replicate and practice as best they can in the classroom environment.
Brandon: Good point. I concur. However, more and more organizations are hit with the high-cost of travel and the inefficiency of removing people away from their jobs for instructor-led training. We need to provide effective strategies for authentic evaluation in an online format.
So I can think of two ways to recreate these situations in the digital world: virtual simulation and/or role-playing. Learners will have different profiles. Does someone who is technically minded need the same practice as someone who is naturally a people person? Both these individuals need to come out with the same skill sets at the end, but perhaps they need different practice. Formative self assessments combined with flexible course sequencing can allow individuals to focus on their areas of real need.
That just describes a sales training course I completed using Thiagi’s 4-Door Model. It allowed the learner to “self-adjust” the content based on their own self-leveling of knowledge. It does beg the question though: how do you “branch” an evaluation based on different learner profiles? Can you have an effective evaluation instrument via a dynamic system that presents a contextual series of assessment items based on the learner’s individual profiles?
I think that you can have a self-assessment that gives the learner guidance in their own strengths and weaknesses.
OK, so you’re talking about self-assessment. What about an actual skills certification? One that can affect a learner’s job status, salary, or in the case of compliance or regulatory situations, a learner’s knowledge that could have life or death consequences? Can a dynamic evaluation instrument provide the appropriate assessment of knowledge?
I don’t know what you mean by “dynamic evaluation instrument”.
“Dynamic pooling” is when the system displays content based on a learner’s input at the time of input.
What you are talking about now is less about the assessment/evaluation used and more the consequences related to that assessment or evaluation. The SATs are pretty weighty and they’ve been using a dynamic response for years. A student gets progressively harder questions until incorrect responses are entered. Then a cycle commences where easier and harder questions are given to the student until the logarithm determines what the student’s level of mastery is. I’d say that stakes are high, but the College Board feels comfortable using this dynamic response.
Right. So I’m saying we need a similar system in corporate learning where appropriate assessment techniques seem to be a missing factor. In eLearning, there is a cycle of ineffective self-check systems instead of situational problem-based assessments. For example, a colleague was discussing with me the usage of “misconception” problems in assessment. They can present more authentic situations such as posing the problem in a scenario and requiring the learner to identify the parts of the scenario that are wrong or inaccurate. Even using open-ended question types where the system evaluates based on a keyword or series of keywords can be quite effective. How are we truly able to measure whether knowledge or skills are transferred if we’re not willing to properly evaluate?
I like that because some people are just natural or trained to be good test takers. That kind of exercise actually accurately tests knowledge that they have, rather than their ability to suss out a poorly written stem or item. I know you are all about performance. Shouldn’t people’s training be based upon actual deficits in their performance and knowledge? And if they are high performers or can demonstrate mastery of course materials, shouldn’t they be rewarded by being allowed to choose their own training regimen, anyhow?
This post first appeared in January 2010 at The Total Learner Experience.
