Home Page Company Clients Partners Reports CallWeb SampleCalc EPlanCalc French

In a time of universal deceit, telling the truth is a revolutionary act.
(George Orwell)

JANUARY 4, 2005

Review of the quality of evaluations across Canadian departments and agencies

(Reproduction of the executive summary of a study conducted by the Centre of Excellence for Evaluation and available on its Web site)

Introduction

Evaluation supports the Government of Canada's aim to becoming a learning organization. It does this by helping senior executives, program managers and policy makers discover whether or not their initiatives work and are meeting objectives, whether or not there is a continued need for their initiatives, and how their initiatives can be better designed and delivered to meet objectives in a cost-effective manner. The quality of evaluation reports is fundamental if the evaluation function is to deliver upon these information needs.

Purpose

In 2001, TBS created the Centre of Excellence for Evaluation (CEE) and established a new Evaluation Policy to strengthen the evaluation function and the quality of reporting. A key objective of this report is to address whether the quality of reports is acceptable and whether there has been an improvement in quality. An important aspect of this work is to promote quality evaluation reports. This review represents one piece of CEE's overall strategy to monitor and strengthen the quality of reporting. Other activities include: best practice research; an annual survey of the health of departmental and small agency evaluation units; individual meetings; ongoing review of evaluations, RMAFS, departmental evaluation plans; and, an annual report documenting evaluation findings and how they contribute to strengthening accountability and the government's Expenditure Review exercise.

Methodology

A number of sources were used to develop the criteria used for this review including the "Guide for the Review of Evaluation Reports", prepared by the Centre of Excellence for Evaluation, TBS, January 2004 and excerpts from the OAG 1993 Report on Program Evaluation ("Criticisms re Evaluation Reports"). A reference group of department and agency evaluation units was also consulted.

Findings

The findings of this review indicate that most federal evaluation reports are acceptable in quality, though almost one-quarter of the evaluations (23 per cent) were rated as inadequate overall. No clear and consistent variations in quality were observed for federal organizations of different sizes and for departments versus agencies. A comparison of reports completed pre- versus post-April 2002 indicates, however, that quality has improved on a number of criteria in the more recent evaluations. For example, this includes: addressing cost-effectiveness issues; methodological rigour; identifying alternatives; presentation of evidence-based findings; and, formal recommendations. This increase in quality over time suggests that TBS's efforts to improve the quality of evaluation may be having a positive impact (i.e., allowing one year, until April 2002, for the Policy to be fully understood by departments/agencies and for the new Centre for Excellence in Evaluation to begin operating). Still, there is a pressing need for further improvement as indicated by the findings noted below.

Key strengths of the evaluations examined in this review include:

On the other hand, some weaknesses of evaluations/reports are:

Conclusions and Recommendations

On balance, most evaluations that were assessed in this review are of reasonable quality. The majority received an overall rating of adequate (45 per cent) or more than adequate (32 per cent). Still, a considerable proportion of the evaluations (23 per cent) were rated as inadequate and this finding warrants attention. To this end, the report recommends that the TBS Centre of Excellence for Evaluation:

  1. Encourage evaluation divisions in federal departments and agencies to strengthen their evaluation reports by addressing the major weaknesses identified in this review;
  2. Refine Treasury Board guidelines/criteria for the expected features of (1) evaluation methodologies and (2) evaluation reports and disseminate them;
  3. Continue to implement a rigorous approach to monitoring the quality of evaluations, and use this as a basis for the development of individual report cards on the quality and overall health of the evaluation function by department and small agency; and,
  4. Identify measures, including an incentive structure and standards, to ensure that departments and agencies submit completed evaluations and reviews in a responsible, reasonable manner. Departments and agencies adherence to such standards should be made a public record.

 Return to the home page
 List all news

To reach us:

General address : service@circum.com
Benoît Gauthier : gauthier@circum.com, @BGauthierCEEQ
Tel. : +1 819 775-2620, Fax : (no more fax; sorry)
238 Fleming Road, Cantley, Québec J8V 3B4