12. Step Nine: Evaluate and Innovate
12.2. What to Evaluate: Summative
In Step 1, I defined quality very narrowly:
Teaching methods that successfully help learners develop the knowledge and skills they will require in a digital age.
It will be clear from reading this book that I believe that to achieve these goals, it will be necessary to re-design most courses and programs. So it will be important to know whether these redesigned courses are more effective than the ‘old’ courses. One way of evaluating these new courses is to see how they compared with the older courses, for instance:
- Completion rates will be at least as good if not better for the new version of the course(s)
- Grades or measures of learning will be at least as good if not better for the new version
The first two criteria are relatively easily measured in quantitative terms. We should be aiming for completion rates of at least 85 per cent, which means of 100 students starting the course, 85 complete by passing the end of course assessment (unfortunately, many current courses fail to achieve this rate, but if we value good teaching, we should be trying to bring as many students as possible to the set standard).
The second criterion is to compare the grades. We would expect at least as many As and Bs in our new version as in the old classroom version, while maintaining the same (hopefully high) standards or higher.
However, to be valid the evaluation will also would need to define the knowledge and skills within a course that meet the needs of a digital age, then measuring how effective the teaching was in doing this. Thus a third criterion would be:
The new design(s) will lead to new and different learning outcomes that are more relevant to the needs of a digital age.
This third criterion is more difficult, because it suggests a change in the intended learning goals for courses or programs. This might include assessing students’ communication skills with new media, or their ability to find, evaluate, analyze and apply information appropriately within the subject domain (knowledge management), which have not previously been (adequately) assessed in the classroom version. This requires a qualitative judgement as to which learning goals are most important, and this may require endorsement or support from a departmental curriculum committee or even an external accreditation body.
With a new design, and new learning outcomes, it may be difficult to reach these standards immediately, but over two or three years it should be possible.