6. Measuring Skills
Another challenge is measuring skills. I was once questioned by a colleague when I said my students were learning to think critically.
‘How do you know?’ he said.
My answer was: ‘I know it when I see it in their assessments.’
‘But how will your students know what you are looking for if you can’t describe it in advance?’
The Higher Education Quality Council of Ontario (HEQCO) published a report in 2018 that claimed to be ‘one of the first major attempts to measure employment-related skills in university and college students on a large scale’. The second study used a test designed to evaluate students’ ability to analyze evidence, understand implications and consequences, and develop valid arguments.
The HEQCO study concluded that final-year students had somewhat higher scores in literacy and numeracy than their first-year counterparts, although there was considerable variation among programs, but little difference between the test scores of incoming and graduating students in critical-thinking abilities, although critical thinking ability too showed considerable variation among programs.
There are a number of possible criticisms of this study. One of the challenges that the HEQCO study faced was finding valid and reliable ways to assess soft skills. The first study measured literacy, numeracy, and problem-solving abilities of adults using everyday scenarios. Why assess these skills outside the knowledge domains in which they were taught, given the importance of context? Were the measurements sensitive enough to really discriminate differences in skill development over time?
Nevertheless, it is worrying that HEQCO found that after four years of post-secondary study there was no noticeable difference in critical thinking skills. Is this because this is not being well taught, or because the tests used were not valid? Any attempt to identify learning outcomes involving skills requires consideration from the beginning of how these skills can validly be assessed. Instructors should not complain about HEQCO’s assessment methods if they cannot justify their own methods of identifying and assessing skills.