Evaluation model leads to credibility gap
The evaluation model used to grade Sweden’s engineering programmes risks creating a credibility gap. Aspects of quality that teachers themselves consider to be most important hardly receive any attention at all. That is the view of pedagogic researcher Ola Svärd, who has studied one of the cornerstones of the model – the degree project – in a new thesis.
“Unless teachers and assessors agree on the meaning of “good quality” in a degree project, there is a risk that issues of validity or legitimacy will arise,” Svärd says. “My own studies indicate that many teachers perceive the quality assessments as being unreliable.”
The inspections of Sweden’s higher education programmes that were carried out in recent years by the Swedish Higher Education Authority (UKÄ) have drawn criticism from several quarters. One area of contention is the suitability of placing such great weight on degree projects.
For KTH Royal Institute of Technology, the evaluation resulted in 8 programmes being assessed as being of insufficient quality. Svärd says one explanation could be that the teachers and the UKÄ view the purpose of the final degree project (thesis project) differently.
“My study shows that the perception of the concept of quality is by no means uniform. Teachers who work with degree projects on a daily basis have a quality agenda that is somewhat different from the steering documents that the UKÄ relies on in the assessment work,” Svärd says.
Assessments not relevant
Interviews he conducted during a Bachelor of Science in Engineering degree programme at KTH show that the teachers place great emphasis on preparing their students for professional careers. The focus is placed on making them employable. In this context, the degree project constitutes a “bridge between the academic world and working life”.
“The teachers are enthusiastic about wanting to give their students a good lift-up into working life. The final degree project is seen as an opportunity for the students to make a good impression on a company. And in the longer term they hope that the degree project can lead on to a job.”
The problem is that the evaluation grounds used by the UKÄ take such considerations into account only to a small extent. Consequently, many teachers feel that the educational programmes are evaluated in accordance with a scale that is not really relevant, he says.
“If an evaluation can have such serious consequences that an entire degree programme is considered unsatisfactory, then it should also have a strong legitimacy amongst those being reviewed,” he states.
Misses meaningful aspects
Svärd’s case study shows that the teachers fail to take sufficient account of certain goals that the UKÄ considers central to the evaluations. This involves, among other things, the necessity for the final degree projects to highlight aspects of sustainable development from a social, ecological and economic perspective.
In the interview survey, the teachers state that they know about the steering documents and are convinced that they comply with them in their work as supervisor and assessor.
“Despite this, they invest a lot of time in aspects that are not emphasised in the quality evaluation,” Svärd says. On the other hand, they have missed things that are accorded considerable significance, he says.
The teachers in the study also value the work process behind the final degree project in a different way than the UKÄ.
On many occasions, the degree project for a Bachelor of Science in Engineering degree aims to develop a product together with a company. As a rule, teachers consider the process to be more important than the concluding written report.
“A general problem is that the work process, which frequently implies challenging and stimulating collaboration between supervisor and student, is not visible in the report writing. And, therefore, it is not subject to the quality evaluation.”
Ola Svärd believes that the concerns he discusses in the thesis affect vocational education and training programmes at KTH more than general programmes. What is more, the degrees of Bachelor of Science in Engineering are probably more affected than the Master of Science in Engineering degrees, he says.
“Even though I didn’t investigate this in the thesis, I do believe that the more a degree programme is focused on a specific professional or vocational competence, the more it is disadvantaged when assessed according to the norms that have been applied so far.”