Reviewer training to assess KT in funding applications is long overdue

31 August 2017

In line with MSFHR’s commitment to meeting the highest scientific standards in peer review, we regularly review our processes to ensure we are meeting the needs of the health research community.

As part of this, while reflecting on MSFHR’s peer reviewer and applicant guidance on knowledge translation (KT), the KT team at the Foundation discovered a distinct lack of evidence on how to train funding program peer reviewers, and no evidence to guide peer reviewer training in KT.

As a result, MSFHR’s Director of KT Gayle Scarrow, Interim Director of KT Donna Angus*, and Interim President & CEO Bev Holmes, conducted a literature search and key informant interviews on assessing KT in funding applications. Published in Research Integrity and Peer Review, they propose that peer reviewer training in KT is critical to ensuring that the KT components of funding applications are as rigorously evaluated as the scientific components.

Reviewer training to assess KT in funding applications is long overdue

Most funding agencies, MSFHR included, now require a KT plan as a central component of funding applications. The intention is to ensure that the research we fund is relevant and applicable, and that the resulting evidence makes it into the hands of those who are best positioned to make use of it. However, to properly assess the KT components of funding applications, you need peer reviewers with KT expertise, or a dedicated KT expert to sit on review panels, but htis can be difficult in a field where hte pool of expert KT scientists and practitioners is small.

To date, funders mostly provide peer reviewers with resources or a KT orientation prior to a review meeting, or embed specific review criteria for KT in applications. However, this may not be sufficient to ensure that review panel members, including research users, have the knowledge and skills to adequately assess the quality of KT plans in applications.

A combination of panel composition, orientation and training can provide KT reviewers with the competencies they require

Our environmental scanning was unable to uncover any evidence that points to a single best practice for training reviewers in KT. As a result, we drew on extrapolated evidence as well as expert and experiential knowledge to develop a set of recommendations to support rigorous KT plan evaluation. These recommendations include a variable mix of panel composition, orientation and training tailored to the skills of reviewers and dependent on the type of funding program and research funded. The recommendations can be summarized as follows:

  • Funding program guidelines must clearly articulate the funder’s expectation of excellence in KT planning.
    Funders should build a set of application and review criteria for each kind of KT (end-of-grant and integrated throughout the research process), with reviewer guidelines that have enough detail to support an assessment of the adequacy of an application’s KT components based on the research type, stage and expected audience. These guidelines can also be used as a basis to train reviewers by providing the conceptual tools to assess the KT components of grant applications.
  • Embed a KT orientation into peer review meetings.
    Training is time-consuming and seasoned reviewers may be unwilling to participate. Embedding a KT orientation as part of a pre-meeting introduction can overcome this and allow training to be calibrated to the knowledge level of the reviewers.
  • Include KT experts on panels, particularly where grants have a substantive or more involved KT component.
    For more involved KT, multiple KT experts might be valuable, for example in areas such as commercialization or implementation science. Having KT experts on panels also provides a learning opportunity for hte panel through discussion. Similarly, including research users on review panels not only serves the purpose of ensuring research relevance but also serves to educate other panel members, particularly when discussing community-based research.

MSFHR’s commitment to rigorous evaluation of KT plans

Funders of health research have an obligation to ensure that the knowledge translation components of funding applications are as rigorously evaluated as the scientific components. At MSFHR, we regularly review the latest scientific research and both reviewer and applicant feedback to establish and guide best practice.

Currently, as well as detailed guidelines, we provide a KT orientation for all reviewers before each review panel meeting. However, we have found that the level of KT support required varies dramatically dependent on the field of research, with population health researchers often feeling more confident in their KT expertise. TO address this we are working with reviewers and national KT experts to tailor our training for and identify the most impactful mode of delivery.

By providing reviewers with the KT competencies they require, we can take an important step towards ensuring that all funded research is not only excellent science but also highly relevant and useful to our society.

Read the full article: Reviewer training to assess knowledge translation in funding applications is long overdue

*Then supporting MSFHR as a KT consultant