Understanding the peer review process

16 July 2014

Peer review is a cornerstone of MSFHR’s Scholar awards and other funding programs.

This mechanism ensures the research we support meets the highest scientific and academic standards. It also maintains our accountability to the provincial government and to British Columbia’s health research community.

But what actually happens between the application deadline and the announcement of funding decisions? To demonstrate the excellent work of MSFHR’s dedicated Programs staff, we have prepared a brief overview of the peer-review process.

How are reviewers selected?

The recruitment of review panel members and external readers is a huge undertaking.

Immediately after the competition’s letter-of-intent deadline, MSFHR programs staff create a pivot table of keywords submitted with applications and analyze which terms appear most often. Those keywords that are consistently repeated must be represented by subject matter experts on the appropriate review panel.

Two types of reviewers must be recruited:

Review panel members

These reviewers are selected for their research excellence, breadth of knowledge, and maturity of judgment. They provide a critical assessment of the application, participate in rating applications, and vote during the peer-review panel meeting. Each application is reviewed in depth by two lead reviewers.

Key facts:

  • Review panels correspond to CIHR’s four research pillars (biomedical, clinical, health services, population health)
  • Recruitment of committee members can take up to 2-3 weeks
  • Review panels consist of 50% BC researchers and 50% from outside BC but within Canada
External readers

These reviewers are subject matter experts whose research corresponds exactly to the content of a particular award application. Each external reader is assigned only one application to review. They provide constructive feedback on the research approach, as well as the project’s originality and potential impact. Panel members refer to the external reader reports when evaluating applications.

Key facts:

  • Recruitment of external readers for a competition can take up to 4-5 weeks
  • It can be necessary to contact 20-25 researchers to secure one external reviewer

What happens at the review panel meeting?

  • Panel members in conflict with an application leave the room for the duration of the review panel’s discussion of that application.
  • The two lead reviewers for each application summarize its strengths and weaknesses with their initial scores displayed to the rest of the review panel.
  • The committee chair leads the panel discussion, inviting feedback from all members.
  • Based on the panel’s discussion, the lead reviewers are given the opportunity to revise their initial scores.
  • The panel’s scientific officer reads out notes summarizing the discussion that took place and the recommendations of panel members.
  • The lead reviewers agree on a consensus score. To ensure consistency, a common scale is used, ranging from 0 to 4.9. Panel members are encouraged to use the full range of scores.
  • All panel members, including the lead reviewers, confidentially score each application within +/- 0.5 points of the consensus score.
  • MSFHR’s Programs team produces a list of the applications in ranked order. A ranked list, with the identity of each applicant masked, is presented to the review panel for a final review.
  • Funding decisions are based on this ranked list, with the funding cut-off varying from one competition to another.

What are the principles governing peer review?

  • Confidentiality
    All information contained in applications submitted to MSFHR, reports made by reviewers, and review panel discussion is strictly confidential and subject to the BC Personal Information Protection Act.
  • Impartiality
    All reviewers must read and agree to abide by MSFHR’s guidelines on conflict of interest prior to viewing any application information.
  • Parity
    Reviewers are expected to be fair and reasonable, to exercise meticulous scientific judgment, and to understand and take into account the particular context of each application.