Peer review: Committing to continuous learning and improvement

Peer review has been a hot topic in the Canadian research community over the past year and MSFHR’s Valerie To, Director, Research Competitions, and Zena Sharman, Director, Strategy, have been monitoring the conversations to stay abreast of emerging trends and best practices.

Here, Val (who leads the team responsible for leading MSFHR’s program implementation and peer review) and Zena (who leads the team responsible for designing and evaluating MSFHR’s funding programs) discuss the latest developments in peer review and MSFHR’s commitment to meeting the highest scientific standards.


Peer review: Committing to continuous learning and improvement

 

Peer review is usually a closed-door process mostly of interest to researchers and research institutions. But last year, it made front page news. The media took notice when the health research community voiced concerns about a series of peer review and funding program reforms implemented at the Canadian Institutes of Health Research (CIHR). In response to these concerns, CIHR convened an International Peer Review Expert Panel in September 2016 to “examine the design and adjudication processes of CIHR's investigator-initiated programs in relation to the CIHR mandate, the changing health sciences landscape, international funding agency practices, and the available literature on peer review.” The Panel released their final report and recommendations in February 2017.

We read the report with keen interest because peer review is integral to our work at MSFHR. Between us, we have almost 20 years’ experience working at Canadian health research funding agencies, spanning program design and implementation, peer review, award administration, program management, and monitoring and evaluation. Zena is also a PhD-trained health researcher with experience as a peer reviewer. Together, we’ve got a deep understanding of the application review process and how it functions within the context of a health research funding agency.

 

Peer review relies on all stakeholders seeing the process as fair and trustworthy

We agree with the Panel’s assertion that peer review “relies on all stakeholders seeing the process as fair and trustworthy.” Trust is integral to both successful peer review and strong relationships between funding agencies and our stakeholders. At MSFHR, we build trust through a review process that operates on the basis of three core principles: confidentiality (keeping applicants’ information, reviewer reports and review panel discussions confidential), impartiality (managing conflict of interest), and parity (demonstrating fairness, meticulous scientific judgement, and awareness of context). In the interest of transparency, we publish our evaluation criteria and an overview of the evaluation process for each competition, as well as our reviewer tutorials and webinars.

As an organization we’re committed to continuous learning and improvement of our review processes. In addition to routine evaluation, in 2013 we commissioned a comprehensive external review of MSFHR’s peer review and funding approval process. It concluded that, “Staff are well-versed in their duties, processes are well-documented, applicants are given good advice and direction in submitting their proposals, and reviewers are provided with helpful guidance about their responsibilities, which they discharge conscientiously in order to provide high-quality reviews.” We’re proud of our review processes and are continually looking to expand our knowledge and skills. To this end, we recently commissioned an environmental scan of best practices in application review that included a literature review and interviews with key informants at health research funding agencies in Canada, the US and Europe.

Our teams are in the process of synthesizing the results of this scan into a set of draft recommendations for MSFHR and we plan to seek stakeholder input on these in the coming months. We’ve also been closely reviewing the Panel’s report to determine whether any of their findings might be applicable to our work. This is particularly timely given MSFHR’s new suite of programs, because application review isn’t a “one-size-fits-all” process. For example, we wouldn’t assess our Innovation to Commercialization Award using the same evaluation criteria or reviewers as we would for our Health Professional-Investigator Award.

Effective application review processes are integral to the successful allocation of health research funding. In defining what success looks like and figuring out how to get there, it’s important to consider both the technical and strategic aspects of peer review (how we do it, and why we do it that way). The Panel’s report offers us helpful food for thought here, as well as raising provocative – and likely controversial – questions about the merits of practices like face-to-face peer review and how funding agencies balance excellence and impact in their funding strategies and application review processes. We will definitely be re-reading this report in the coming months, and look forward to continuing the conversation about excellence in peer review.

More information on MSFHR’s review process is available here.

Add new comment

Plain text

  • No HTML tags allowed.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Lines and paragraphs break automatically.