Teachers have experienced unreliability in quality assessments. (Photo: Håkan Lindgren)

Evaluation model leads to credibility gap

Published Apr 04, 2014

The evaluation model used to grade Sweden’s engineering programmes risks creating a credibility gap. Aspects of quality that teachers themselves consider to be most important hardly receive any attention at all. That is the view of pedagogic researcher Ola Svärd, who has studied one of the cornerstones of the model – the degree project – in a new thesis.

“Unless teachers and assessors agree on the meaning of “good quality” in a degree project, there is a risk that issues of validity or legitimacy will arise,” Svärd says. “My own studies indicate that many teachers perceive the quality assessments as being unreliable.”

The inspections of Sweden’s higher education programmes that were carried out in recent years by the Swedish Higher Education Authority (UKÄ) have drawn criticism from several quarters. One area of contention is the suitability of placing such great weight on degree projects.

For KTH Royal Institute of Technology, the evaluation resulted in 8 programmes being assessed as being of insufficient quality.  Svärd says one explanation could be that the teachers and the UKÄ view the purpose of the final degree project (thesis project) differently.

The concept of quality is by no means uniform

“My study shows that the perception of the concept of quality is by no means uniform. Teachers who work with degree projects on a daily basis have a quality agenda that is somewhat different from the steering documents that the UKÄ relies on in the assessment work,” Svärd says.

Assessments not relevant

Interviews he conducted during a Bachelor of Science in Engineering degree programme at KTH show that the teachers place great emphasis on preparing their students for professional careers. The focus is placed on making them employable. In this context, the degree project constitutes a “bridge between the academic world and working life”.

“The teachers are enthusiastic about wanting to give their students a good lift-up into working life. The final degree project is seen as an opportunity for the students to make a good impression on a company. And in the longer term they hope that the degree project can lead on to a job.”

The problem is that the evaluation grounds used by the UKÄ take such considerations into account only to a small extent. Consequently, many teachers feel that the educational programmes are evaluated in accordance with a scale that is not really relevant, he says.

“If an evaluation can have such serious consequences that an entire degree programme is considered unsatisfactory, then it should also have a strong legitimacy amongst those being reviewed,” he states.

Misses meaningful aspects

Svärd’s case study shows that the teachers fail to take sufficient account of certain goals that the UKÄ considers central to the evaluations. This involves, among other things, the necessity for the final degree projects to highlight aspects of sustainable development from a social, ecological and economic perspective.

In the interview survey, the teachers state that they know about the steering documents and are convinced that they comply with them in their work as supervisor and assessor.

“Despite this, they invest a lot of time in aspects that are not emphasised in the quality evaluation,” Svärd says. On the other hand, they have missed things that are accorded considerable significance, he says.

Ola Svärd, pedagogic researcher

The teachers in the study also value the work process behind the final degree project in a different way than the UKÄ.

On many occasions, the degree project for a Bachelor of Science in Engineering degree aims to develop a product together with a company. As a rule, teachers consider the process to be more important than the concluding written report.

“A general problem is that the work process, which frequently implies challenging and stimulating collaboration between supervisor and student, is not visible in the report writing. And, therefore, it is not subject to the quality evaluation.”

Ola Svärd believes that the concerns he discusses in the thesis affect vocational education and training programmes at KTH more than general programmes. What is more, the degrees of Bachelor of Science in Engineering are probably more affected than the Master of Science in Engineering degrees, he says.

“Even though I didn’t investigate this in the thesis, I do believe that the more a degree programme is focused on a specific professional or vocational competence, the more it is disadvantaged when assessed according to the norms that have been applied so far.”

Christer Gummeson

Final degree project - a quality indicator within higher ed?

Ola Svärd conducted a case study of the Bachelor of Science in Engineering degree at KTH. He has interviewed 12 teachers who have been examiners and supervisors and compared their view of quality in the degree projects with the steering documents and research literature within the area. Svärd presents his thesis The final degree project- a quality indicator within higher education? The example of the Bachelor of Science in Engineering degree programme  at Uppsala University on 10 April.

In previous studies, he has carried out quality evaluations of three Bachelor of Science in Engineering programmes at KTH.
Ola Svärd worked as a teacher at KTH during the years 1997–2011, in the subject areas of Maths and Physics, mostly within teacher training programmes and teachers’ further professional development.

Editors Note: At present, the Higher Education Authority is working on preparing a new version of the evaluation model that will be applied as of 2015.