Home Page Company Clients Partners Reports CallWeb SampleCalc EPlanCalc French

Doubt is not a pleasant condition, but certainty is an absurd one.
(Voltaire)

Document T086

Evaluation of NSERC's Scholarships and Fellowships

Prepared by Circum Network Inc. in collaboration with Program Evaluation and Beyond Inc. and SSHRC’s and NSERC’s Evaluation Division For the Natural Sciences and Engineering Research Council of Canada

Prepared by Circum Network Inc. in collaboration with Program Evaluation and Beyond Inc. and SSHRC's and NSERC's Evaluation Division For the Natural Sciences and Engineering Research Council of Canada

This report presents the evaluation of NSERC's Scholarships and Fellowships, for the period 2003-04 to 2013-14. Specific funding opportunities under review are Postgraduate Scholarships (PGS), Industrial Postgraduate Scholarships (IPS), Industrial Innovation Scholarships (IIS), the Collaborative Research and Training Experience (CREATE), and to a lesser extent the Postdoctoral Fellowships (PDF), Undergraduate Student Research Awards in Universities (USRA-U), and International Exchanges (IE).

The funding opportunities support a significant number of students and postdoctoral fellows at various stages of their university studies, from the undergraduate level to post-doctoral studies. Actual expenditures under the sub-program amounted to $73.3 million in 2013-2014.

The evaluation was based on the following seven lines of evidence:

As described below, the findings from this evaluation indicate that the Scholarships and Fellowships sub-program is effective at supporting graduate students and post-doctoral fellows in the natural sciences and engineering (NSE). Each funding opportunity has its own niche consistent with its design, and together they constitute an effective toolbox for NSERC's intervention in support of NSE training.

Relevance

The funding opportunities were found to be relevant given government priorities and the needs of stakeholders. Federal government involvement in scholarships and fellowships is supported by the priority given to investing in people as part of a science, technology and innovation strategy, and is aligned with the objectives of NSERC. The funding opportunities are part of a larger system of financial support that is needed by graduate students and postdoctoral fellows. However, this evaluation finds that in the NSE, in general, all graduate students receive financial support. Finally, the professional training emphasized by some funding opportunities is highly valued.

Effectiveness

Although this evaluation finds evidence that the funding opportunities are contributing to longer-term outcomes, this contribution is not always as evident for more immediate outcomes, occurring during degree studies. This may be explained in part because of the complex funding environment in which NSE students find themselves. Because a typical NSERC recipient is in receipt of multiple other types of funding adding up to a much greater amount than their NSERC award or stipend, it is difficult to control for the effect of these other funding sources when assessing the contribution of the NSERC funding opportunity.

Across all highly qualified personnel (HQP) in the earlier survey cohort (both recipients and non-recipients), more than 90% had completed their degree; the average Master's student took two years eight months to finish, and the average doctoral student took five years. It is possible that funding opportunities supported the completion of studies or the time to completion, but not in a way that is discernible via a comparison between recipients and non-recipients; this may be because all NSE graduate students are financially supported through their studies, be that by the NSERC funding opportunities or through other awards, prizes, or stipends.

The training environments provided under the various funding opportunities showed specific characteristics that are representative of their design: CGS recipients had more interactions within Canada while PGS awardees had more interactions outside Canada; IPS/IIS had more interactions with the private sector; IIPS/IIS and CREATE students enjoyed more exchange and internship opportunities.

On average, nine out of 10 recipients and non-recipients from the earlier cohort were working full-time at the time of the survey, and reported annual employment-related income between $70,000 and $80,000. The primary employer for Master's students was the private sector, while for doctoral students it was universities.

Although it was too early to assess long-term outcomes for CREATE, the PGS and IPS/IIS funding opportunities had observable impacts on employment outcomes. PGS recipients at both Master's and doctoral levels, compared to non-recipients, had higher incomes and reported their training was more useful to their careers. At the doctoral level, PGS recipients were more likely to be working full-time than non-recipients, and if employed in academia, were more likely to be research faculty as opposed to non-recipients who were more likely to be in a postdoctoral position. IPS/IIS recipients were more likely to be working in the private sector at both levels, and at the Master's level had higher income than non-recipients.

Participants from all funding opportunities contributed to research productivity. Although there were few observable differences across funding opportunities with regard to the types and amounts of academic outputs, the use of these outputs differed across funding opportunities: a greater number of citations of articles for PGS-M and PGS-D; more software and databases as well as more direct cost savings for IPS/IIS-M recipients; more professional practice outputs and more improved policies and programs for IPS/IIS-D participants; more new practices for IPS/IIS and CREATE participants. These particularities were congruent with the design of the funding opportunities. PGS and IPS/IIS supervisors were satisfied with the awardee's research contributions to their research program or their impact on the supervisor's organization.

In order to address the question of whether direct or indirect funding of students provided better outcomes, a separate analysis was undertaken of the survey data, encompassing the full funding package of each student surveyed, assessing the extent to which types of Tri-Agency funding predicted better outcomes for these students. This analysis showed that direct Tri-Agency funding (scholarships or fellowships) had stronger positive associations with academic and employment outcomes than indirect Tri-Agency funding (stipends), and primarily at the doctoral rather than Master's level.

One of the goals of CREATE is to influence the NSE academic system in a sustainable way; CREATE initiatives are expected to be self-sustaining by the end of the grant. The evaluation finds that this sustainability may not be assured. Many aspects of CREATE require funding that is unlikely to exist after the end of CREATE support.

Efficiency and Delivery

In terms of program delivery, few substantial issues were found; this is not surprizing considering that most of the funding opportunities are long standing.

Funding opportunities were generally delivered in an efficient manner: about $4.60 to $5.50 in administrative costs were spent for every $100 in grant funds, across all funding opportunities. For PGS, PDF, CGS and IIS, the cost-efficiency ratios have tended to increase over time (ranging from 80¢ to $1.10 for every $100 of grants awarded); however, they are still within acceptable ranges.

Although the majority of CREATE initiatives appear able to meet the program guideline requiring 80% of funds be directed to trainee stipends, CREATE recipients would prefer that this threshold be lowered. Also, some CREATE recipients indicated that limiting the payment of stipends to students from outside the NSE to 30% of payments was constraining in research domains at the interface of health, social sciences, and humanities.

Under-spending by CREATE initiatives was an issue. It was found that, on average, CREATE initiatives spent 35% less than the balance available to be spent in that year. Under-spending was greatest in Year 1 of the initiative's implementation (at 48%). By Year 3, annual expenditures met or exceeded annual instalments. If instalment amounts were reduced in Years 1 and 2, most CREATE initiatives would be better able to align expenditures with instalment amounts on a yearly basis.

While recipients were mostly satisfied with the management of the funding opportunities, the clarity (and fairness to a lower degree) of the selection process, the promptness of the notification, and the length of some awards generated lower satisfaction levels.

Recommendations

Based on the findings of the evaluation and noting that the results suggest that these programs are relevant and well managed, the following recommendations are offered to improve the operations and outcomes of these funding opportunities.

Recommendation 1: The quality and quantity of information communicated on the PGS and IPS/IIS selection processes should be improved.

Recommendation 2: CREATE program management should consider design adjustments to adapt to the initiatives' ability to spend in Year 1.

Recommendation 3: CREATE program management should consider revisiting the limits placed on certain CREATE spending categories.

Recommendation 4: Program management should continue to monitor administrative costs relative to grant expenditures.

60 pages, 909K [PDF format]


To reach us:

General address : service@circum.com
Benoît Gauthier : gauthier@circum.com, @BGauthierCEEQ
Tel. : +1 819 775-2620, Fax : (no more fax; sorry)
238 Fleming Road, Cantley, Québec J8V 3B4