Home Page Company Clients Partners Reports CallWeb SampleCalc EPlanCalc French
Doubt is not a pleasant condition, but certainty is an absurd one.
Evaluation supports the Government of Canada's aim to becoming a learning organization. It does this by helping senior executives, program managers and policy makers discover whether or not their initiatives work and are meeting objectives, whether or not there is a continued need for their initiatives, and how their initiatives can be better designed and delivered to meet objectives in a cost-effective manner. The quality of evaluation reports is fundamental if the evaluation function is to deliver upon these information needs.
In 2001, TBS created the Centre of Excellence for Evaluation (CEE) and established a new Evaluation Policy to strengthen the evaluation function and the quality of reporting. A key objective of this report is to address whether the quality of reports is acceptable and whether there has been an improvement in quality. An important aspect of this work is to promote quality evaluation reports. This review represents one piece of CEE's overall strategy to monitor and strengthen the quality of reporting. Other activities include: best practice research; an annual survey of the health of departmental and small agency evaluation units; individual meetings; ongoing review of evaluations, RMAFS, departmental evaluation plans; and, an annual report documenting evaluation findings and how they contribute to strengthening accountability and the government's Expenditure Review exercise.
A number of sources were used to develop the criteria used for this review including the "Guide for the Review of Evaluation Reports", prepared by the Centre of Excellence for Evaluation, TBS, January 2004 and excerpts from the OAG 1993 Report on Program Evaluation ("Criticisms re Evaluation Reports"). A reference group of department and agency evaluation units was also consulted.
The findings of this review indicate that most federal evaluation reports are acceptable in quality, though almost one-quarter of the evaluations (23 per cent) were rated as inadequate overall. No clear and consistent variations in quality were observed for federal organizations of different sizes and for departments versus agencies. A comparison of reports completed pre- versus post-April 2002 indicates, however, that quality has improved on a number of criteria in the more recent evaluations. For example, this includes: addressing cost-effectiveness issues; methodological rigour; identifying alternatives; presentation of evidence-based findings; and, formal recommendations. This increase in quality over time suggests that TBS's efforts to improve the quality of evaluation may be having a positive impact (i.e., allowing one year, until April 2002, for the Policy to be fully understood by departments/agencies and for the new Centre for Excellence in Evaluation to begin operating). Still, there is a pressing need for further improvement as indicated by the findings noted below.
Key strengths of the evaluations examined in this review include:
On the other hand, some weaknesses of evaluations/reports are:
Conclusions and Recommendations
On balance, most evaluations that were assessed in this review are of reasonable quality. The majority received an overall rating of adequate (45 per cent) or more than adequate (32 per cent). Still, a considerable proportion of the evaluations (23 per cent) were rated as inadequate and this finding warrants attention. To this end, the report recommends that the TBS Centre of Excellence for Evaluation:
To reach us:
General address : firstname.lastname@example.org
Benoît Gauthier : email@example.com, @BGauthierCEEQ
Tel. : +1 819 775-2620, Fax : (no more fax; sorry)
238 Fleming Road, Cantley, Québec J8V 3B4