Introduction: Peer review

In recent years, research grants from various bodies—such as the National Institute of Health in America or the Australian Research Council—have become increasingly competitive.  The reason, as De Peuter and Conix (2022) argue, is because the number of researchers who submit grant applications has increased but the availability of funds has not been boosted to the same extent.

 

To decide which applicants they will fund, most of these bodies apply an approach called peer review.  Specifically

 

• a panel of relevant scientists, called reviewers, evaluate and score these proposals

• the scores of these reviewers are combined, and the proposals are ranked

• only the applicants that are ranked the highest receive funding

 

The majority of scientists generally believe that peer review, despite possible flaws, is the best approach to allocate research funding (Philipps, 2021).  Yet, scholars in this field have shown that peer review may culminate in a range of concerns and problems. 

 

To resolve this concerns, since around 2000 (e.g., Boyle, 1998; Brezis, 2007; Greenberg, 1998), scholars have advocated the possibility of random grant allocation, sometimes called lottery procedures, to decide which grant submissions should be funded.  For example, Greenberg (1998) recommended that funding bodies should allocate a specific percentage of their funds—perhaps 15 to 20%--to a lottery.  If applicants were chosen in these lotteries, they would receive the funding they sought, provided their projects were first deemed as respectable. Random grant allocation is assumed to overcome several limitations of peer review—the approach that funding bodies usually apply to allocate grants—such as the steep costs or bias towards research that is not especially novel. Specifically, random grant allocation is not only efficient but can increase the diversity of funded projects and can override some injustices around funding. 

 

Because of these benefits, several funding bodies, such as the New Zealand Health Research Council, the Swiss National Science Foundation, the Austrian Science Fund, and the German Volkswagen Foundation have applied this approach.  Other funding bodies, including the Council of Canadian Academies and Royal Danish Academy of Science and Letters, have piloted or plan to pilot this approach.  Nevertheless, despite some enthusiasm towards random grant allocation, many scholars criticize this approach (Bedessem, 2020; Reinhart & Schendzielorz, 2020)  

 

Limitations of the peer review approach: Expenses

Compared to other approaches, such as random allocation, the peer review approach is expensive to applicants, to reviewers, and to funding bodies.  To illustrate the expense to applicants, Herbert and colleagues (2013) estimated the time that researchers had dedicated to one funding round—the Australian National Health and Medical Research Council in 2012.  They estimated the 3727 research proposals would have consumed 550 hours of work, equivalent to AU$66 million in salaries.  This value represents 14% of the total fund.  Only 21% of the projects received funding.   

 

Admittedly, when researchers construct research proposals, they may receive some benefit, even if these projects are not funded.  For example, they need to consider their past research and future aspirations more diligently, sometimes clarifying their ideas, goals, and priorities.  Nevertheless, as Gross and Bergstrom (2019) revealed, after estimating the return on these grants, scientists would tend to be significantly more productive to society if they could dedicate more time to productive research rather than to grant applications.

 

Besides the applicants, the reviewers also dedicate significant time to this approach, also detracting time from more productive activities.  For example, as Graves et al. (2011) estimated, about 9% of the costs or expenses of one grant scheme or round of the Australian National Health and Medical Research Council can be ascribed to peer review.    

 

Peer review does not only incur these costs but also increases waste.  To illustrate, most funding bodies will distribute most the funds to a limited subset of renowned authors.  The researchers that receive the funding, therefore, often earn more research income than needed.  Hence, these researchers do not tend to use these funds as effectively (Mongeon et al. 2016).  Similarly, according to Wahls (2018), research projects in health that receive about $400k annually from the National Institute of Health are the most efficient.  Yet, a few projects receive significantly more, and many projects receive significantly less, than $400k annually.  

 

Limitations of the peer review approach: Limited precision of peer evaluations

Peer reviewers will tend to assign each application or proposal a score.  Yet, because peer reviewers depend on intuition and judgment, these scores are not especially precise.  If the reviewers evaluated the same applications twice, the scores would inevitably be different.

 

The problem, however, is that a difference in decimals, such as 3.09 out of 5 and 3.08 out of 5, is often enough to differentiate successful grant applications from unsuccessful grant applications.  Indeed, as Kaplan et al (2008) estimated, if the NHS wanted to limit this random error to .01 on a 5-point scale, over 38 000 reviewers would need to be engaged in evaluate the applications.  If funding bodies merely wanted to limit this random error to .1 on a 5-point scale, 384 reviewers would still need to be engaged.

 

Limitations of the peer review approach: Relevance of ratings

To evaluate grant applications, funders usually receive explicit instructions on how to evaluate the various criteria, such as quality, impact, and track record.  But whether the scores they assign to each application predict return on investment is contentious.  That is, because of several reasons, the criteria do not necessarily culminate in scores that reflect possible quality or return on investment.  To illustrate

 

• experienced reviewers often deviate from these instructions, because they overestimate their knowledge of peer reviews (Sattler et al. 2015)

• reviewers cannot predict which research projects are likely to be successful—but instead may derive these predictions from similar projects in the past; hence, biases in the past that favor specific projects will magnify over time (Mallapaty, 2018)

• typically, the reviewers of grant applications, unlike the reviewers of publications, are seldom leading specialists in the field

 

Because of these concerns, the scores tend to vary considerably across reviewers.  As Gallo et al. (2016) estimated, over 75% of the variance in these scores can be attributed to either random error, variations across reviewers, or the interaction between variation across reviewers and the specific project.  Nevertheless, reviewers often concur on which of the projects outstanding (Li & Agha 2015) but not as likely to concur on the quality of other projects

 

Limitations of the peer review approach: Bias in ratings

Not only are peer evaluations often inaccurate rather than precise and meaningful, the scores are sometimes biased towards particular demographics, even if these biases are unconscious and inadvertent.  To illustrate

 

• males often receive higher scores, even when gender is concealed, because they tend to use more abstract language (Else 2019; for a nuanced discussion, see Sato et al 2020)

• reviewers tend to evaluate the same projects more favorably if the track record of applicants is extensive (Guthrie et al. 2019)

• research that is especially innovative tends to be rated unfavorably partly because the results of innovative studies is harder to predict and partly because innovative projects may not confirm to the conventional standards that reviewers espouse (Brezis 2007; Gillies 2014; Li & Agha 2015)

• research that integrates multiple disciplines or FOR codes is more likely to be rated unfavorably, because the projects may not conform to the standards the reviewers have adopted (e.g., Bromham et al., 2016).

 

Limitations of the peer review approach: A potential source of research misconduct

Because peer review is so competitive, this approach might discourage research integrity.  To illustrate

 

• because reviewers are not compensated but often need to evaluate many applications, they often skim application and thus inflate their confidence in their scores (e.g., Herbert et al 2013)

• to be competitive, applicants may feel obliged to exaggerate the milestones they can deliver within a specific timeframe (De Peuter & Conix, 2022) or the implications of their research (Serrano Velarde, 2018)

 

Possible improvements to the peer review approach: Measures to diminish the impact of inconsistent scores

Because of the problems and concerns about the peer review approach, many scholars have recommended entirely novel alternatives—most commonly random grant allocation.  However, even smaller changes to the peer review approach could offset or diminish some of these problems.   

 

First, many of the criticisms that are levelled at the peer review approach revolve around the inaccuracy or bias of scores—inaccuracies or biases that often manifest as inconsistencies across reviewers.  Therefore, measures that diminish the impact of these inconsistencies might significantly improve the integrity and utility of the peer review approach.  To diminish these inconsistencies, several measures have been recommended:

 

• reviewers could receive more training (Sattler et al. 2015), although past research implies this approach may not be effective (Pier et al. 2018)

• the lowest score that is assigned to each applicant should be deleted, to diminish the established impact of low outliers (Daniels, 2015)

• perhaps variability in reviewer scores could be deemed as an indication the project could be innovative and the returns, as well as risks, could be elevated (Kaplan et al., 2008)

• the panel of reviewers could also include other relevant stakeholders, such as farmers (Bedessem, 2020)

 

Possible improvements to the peer review approach: Measures to diminish costs

Scholars have also proposed several measures to diminish the costs and expenses of peer review.  For example

 

• whether shorter applications save time would need to be investigated empirically

• an approach that comprises two phases—in which researchers submit a full application only after initial expressions of interest are accepted—has been shown to reduce expense by 30% in particular circumstances (Morgan et al. 2020)

 

Random grant allocation strategies: An example

To override the problems and limitations of peer evaluations, many scholars have advocated random allocation, at least in specific circumstances.  But few funding agencies have implemented this approach. 

 

One of the first exceptions was the Health Research Council of New Zealand (for a discussion, see Liu et al., 2020).  Since 2013, random allocation has been applied to one scheme, called the Explorer Grant—a scheme that revolves around transformative research in health.  To be eligible, the proposed ideas, methodologies, or technologies must be deemed as transformative, innovative, or unconventional.  These applications would not be eligible to receive other Health Research Council funding.  Successful applications receive NZD $150,000.

 

To apply, researchers submit brief applications, comprising six pages or fewer, and need to enter limited administrative information.  A panel of three members, selected because of their capacity to think innovatively and their expertise in either biomedical, clinical, or public health, assess each anonymous application on two criteria only:

 

• whether the research is potentially transformative rather than merely an incremental refinement to past research

• whether the research is viable—such as whether the research could be completed

 

If two or more of the panel members perceive the research as both potentially transformative and viable, the application is included in the lottery.  Then, to randomly allocate the applications, the funding agency deploy Microsoft Excel to randomly assign a number to each proposal.  The projects that are assigned smaller numbers receive funding, until the funding pool is exhausted.  All applicants are informed about this procedure.   

 

Random grant allocation strategies: The perspective of researchers

Presumably, random grant allocation may be applicable in some circumstances but not in other circumstances.  Barlösius and Philipps (2022) thus explored the circumstances in which researchers felt that random grant allocation may be useful.  In particular, this study examined which the perspective of researchers on which problems this random grant allocation could address.    To address this question, Barlösius and Philipps interviewed 32 researchers, ranging from doctoral candidates to professors, specializing in biology or physics.

 

In general, the researchers felt that random allocation may address a range of problems with peer review.  Specifically, random allocation might diminish a range of problems.  For example, when funding bodies depend on peer review  

 

• reviewers often reject the applications of gifted researchers who are not yet renowned

• reviewers sometimes reject the applications that might challenge their own research or favor researchers in their network

 

Random allocation can overcome these problems, because the biases of reviewers do not affect the allocation of funding.  However, according to the participants, random allocation cannot address some of the other limitations of peer review, such as the problem that

 

• members of funding bodies sometimes use the ideas of submitted research proposals

• the feedback that applicants receive is often inadequate

• researchers do not know which topics will be prioritized and thus cannot readily choose the issues to explore

 

More interestingly, according to participants, some of the matters that random allocation may address may not be problems at all but are legitimate features of science.  To illustrate, according to participants

 

• bold and unconventional research tends to be rejected, diminishing the tendency of researcher to propose daring research projects.

• reviewers tend to apply their intuition and judgment, rather than a methodical approach, underpinned by science, to rank applicants

 

Although random allocation may address these matters, the rejection of daring projects and the reliance on reviewer judgment are essential features of scientific progress.  For example, this rejection of daring projects, although potentially stifling progress, also confines funding to projects that are more likely to be effective.   Likewise, the reliance on judgment and intuition enables reviewers to consider priorities that are hard to characterize in algorithms or formulas.     

 

Because of these considerations, the participants felt the random allocation of grants may be applicable, but only in specific circumstances and if preceded by attempts to confirm the scientific quality of applications.  According to participants, random allocation may be suitable when

 

• funding bodies want to confine some of their funding to support daring, innovative projects and researchers

• multiple applications are deemed as equally worthy but only one application can be funded

• funding bodies want to override the inherent bias towards more established researchers

• funding bodies need to reach decisions more efficiently.

 

Random grant allocation strategies: The perspective of applicants

In addition to the attitudes of researchers in general, some research has explored the attitudes of applicants to specific instances of random allocation.  To illustrate, the Explorer Grants in New Zealand are allocated randomly, provided the research is deemed to be transformative and viable.  Liu et al. (2020) conducted a survey to evaluate the attitudes of applicants towards this scheme.  Interestingly, 78% of the applicants who received funding deemed this random allocation is acceptable, whereas only 44% of the applicants who did receive funding deemed this random allocation is acceptable.  However, in general, these individuals felt this random allocation is suitable only to grant schemes that revolve around transformational research.

 

Features of randomized grant allocation: A triage phase

One problem with randomized grant allocation is that research proposals that are patently inadequate might be funded.  Conversely, research proposals that are patently outstanding might be rejected.  To address this problem, many if not most proponents of randomized grant allocation recommend a preliminary review before the random allocation.  This preliminary review might comprise several phases.

 

First, an exclusive team of reviewers might evaluate the grant applications, but perhaps more expeditiously than typical peer reviews.  They might, for example, evaluate only a brief summary (e.g., Gross & Bergstrom, 2019).  Second, grant applications that are obviously unsuitable—perhaps because they do not fulfill several minimum criteria or because all reviewers criticized this proposal—would be rejected before the lottery (e.g., Gillies, 2014; Liu et al. 2020).  Third, grant applications that are conspicuously outstanding may be funded and bypass the lottery.  Finally, all the other grant applications—applications that are not obviously unsuitable or outstanding (Graves et al., 2011) or applications that did not reach consensus (Brezis 2007)—would be submitted to the lottery.  

 

Features of randomized grant allocation: Minimal criteria

Most scholars who advocate random allocation concede that some preliminary activity, such as triage phase, must precede the lottery.  Otherwise, anyone could submit an application and receive a grant.   The question, then, becomes how should funding agencies decide which applications to withhold from the lottery.  

 

According to Avin (2018), funding agencies should stipulate the minimum criteria that an application must fulfill.  To illustrate, funding agencies might stipulate that researchers cannot submit an application unless

 

• they have attained a doctoral qualification in institutions that are ranked above a particular threshold

• they have published at least one paper in one or several relevant journals, and so forth

 

To develop these minimum criteria, Avin (2018) recommends that

 

• researchers should be able to pursue multiple pathways to fulfill each criterion; otherwise, particular segments of the population might be unfairly disadvantaged

• these criteria should be discussed and refined regularly and collaboratively with all relevant stakeholders

 

Features of randomized grant allocation: Short proposals

If proposals merely need to fulfill minimum criteria to be included in a lottery, the proposals do not need to be lengthy.  According to Avin (2018), proposals should be detailed enough only to fulfil four purposes:

 

• The proposal should clarify the topic, to enable the funding body to distribute this submission to the right panel

• The authors should demonstrate knowledge of some relevant literature; even radical proposals should be informed by some past knowledge

• The proposal should include enough detail to enable reviewers to ascertain promptly whether the merit of this research is broadly likely to be high, moderate, low, or uncertain; this information could be used to identify applications that should circumvent the lottery or be rejected outright.

 

Features of randomized grant allocation: Graduated lottery

Most random allocation schemes assume that applications are either included or excluded from the lottery.  However, funding bodies could use a graduated lottery, foreshadowed by Boyle (1998) and suggested by De Peuter and Conix (2022).  To illustrate, according to this scheme

 

• the grant applications would be ranked, at least cursorily

• half the applications in the lowest quartile would be randomly selected

• the selected applications would be combined with the next lowest quartile—and half this set of applications would be randomly selected

• this procedure would be extended to all four quartiles

 

The benefit of this scheme is that applicants, initially evaluated favorably, would be more likely funded.  However, this scheme does not depend excessively on the evaluations of reviewers

 

Limitations of random allocation: Circumstances in which randomized allocation may not be appropriate

Most scholars recognize that randomized allocation of research grants may be more applicable in some circumstances than other circumstances. Avin (2018) discussed some of the circumstances in which randomized allocation may be unsuitable:

 

• Randomized allocation should not be applied to fund massive projects—like the Human Genome Project—that need to receive millions of dollars over decades.  No project that is only randomly selected should receive decades of funding

• Randomized allocation is not as likely to be effective when a funding agency wants to encourage research that solves a specific problem, such as an epidemic

  

References

• Avin, S. (2018). Policy considerations for random allocation of research funds. RT. A Journal on Research Policy and Evaluation, 6(1).

• Barlösius, E., & Philipps, A. (2022). Random grant allocation from the researchers’ perspective: Introducing the distinction into legitimate and illegitimate problems in Bourdieu’s field theory. Social Science Information, 61(1), 154-178.

• Bedessem, B. (2020). Should we fund research randomly? An epistemological criticism of the lottery model as an alternative to peer review for the funding of science. Research Evaluation, 29(2), 150-157.

• Bornmann, L., & Daniel, H. D. (2009). The luck of the referee draw: The effect of exchanging reviews. Learned Publishing, 22(2), 117-125.

• Boyle, C. (1998). Organizations selecting people: how the process could be made fairer by the appropriate use of lotteries. Journal of the Royal Statistical Society: Series D (The Statistician), 47(2), 291-321.

• Brezis, E. S. (2007). Focal randomisation: An optimal mechanism for the evaluation of R&D projects. Science and Public Policy, 34(10), 691-698.

• Bromham, L., Dinnage, R., & Hua, X. (2016). Interdisciplinary research has consistently lower funding success. Nature, 534(7609), 684-687.

• Chawla, D. S. (2021). Swiss funder draws lots to make grant decisions. Nature.

• Cole, S., Cole, J. R., & Simon, G. A. (1981). Chance and consensus in peer review. Science, 214(4523), 881-886.

• Daniels, R. J. (2015). A generation at risk: young investigators and the future of the biomedical workforce. Proceedings of the National Academy of Sciences, 112(2), 313-318.

• De Peuter, S., & Conix, S. (2022). The modified lottery: Formalizing the intrinsic randomness of research funding. Accountability in Research, 29(5), 324-345.

• Else, H. (2019). Male researchers''vague'language more likely to win grants. Nature.

• Fang, F. C., & Casadevall, A. (2016). Research funding: The case for a modified lottery. MBio, 7(2), e00422-16.

• Gallo, S. A., Sullivan, J. H., & Glisson, S. R. (2016). The influence of peer reviewer expertise on the evaluation of research funding applications. PloS one, 11(10).

• Gildenhuys, P. (2020). Lotteries make science fairer. Journal of Responsible Innovation, 7(sup2), S30-S43.

• Gillies, D. (2014). Selecting applications for funding: Why random choice is better than peer review. RT. A Journal on Research Policy and Evaluation, 2(1).

• Graves, N., Barnett, A. G., & Clarke, P. (2011). Funding grant proposals for scientific research: retrospective analysis of scores by members of grant review panel. BMJ, 343.

• Greenberg, D. S. (1998). Chance and grants. The Lancet, 351(9103), 686.

• Gross, K., & Bergstrom, C. T. (2019). Contest models highlight inherent inefficiencies of scientific funding competitions. PLoS Biology, 17(1)

• Guthrie, S., Rincon, D. R., McInroy, G., Ioppolo, B., & Gunashekar, S. (2019). Measuring bias, burden and conservatism in research funding processes. F1000Research, 8(851).

• Herbert, D. L., Barnett, A. G., Clarke, P., & Graves, N. (2013). On the time spent preparing grant proposals: an observational study of Australian researchers. BMJ open, 3(5), e002800.

• Höylä, T., Bartneck, C., & Tiihonen, T. (2016). The consequences of competition: Simulating the effects of research grant allocation strategies. Scientometrics, 108(1), 263-288.

• Kaplan, D., Lacetera, N., & Kaplan, C. (2008). Sample size and precision in NIH peer review. PLoS One, 3(7), e2761.

• Langfeldt, L. (2001). The decision-making constraints and processes of grant peer review, and their effects on the review outcome. Social Studies of Science, 31(6), 820-841.

• Lee, C. J., Grant, S., & Erosheva, E. A. (2020). Alternative grant models might perpetuate Black–White funding gaps. The Lancet, 396(10256), 955-956.

• Li, D., & Agha, L. (2015). Big names or big ideas: Do peer-review panels select the best science proposals? Science, 348(6233), 434-438.

• Liu, M., Choy, V., Clarke, P., Barnett, A., Blakely, T., & Pomeroy, L. (2020). The acceptability of using a lottery to allocate research funding: a survey of applicants. Research Integrity and Peer Review, 5(1), 1-7.

• Luukkonen, T. (2012). Conservatism and risk-taking in peer review: Emerging ERC practices. Research Evaluation, 21(1), 48-60.

• Magua, W., Zhu, X., Bhattacharya, A., Filut, A., Potvien, A., Leatherberry, R., ... & Kaatz, A. (2017). Are female applicants disadvantaged in National Institutes of Health peer review? Combining algorithmic text mining and qualitative methods to detect evaluative differences in R01 reviewers' critiques. Journal of women's health, 26(5), 560-570.

• Mongeon, P., Brodeur, C., Beaudry, C., & Larivière, V. (2016). Concentration of research funding leads to decreasing marginal returns. Research Evaluation, 25(4), 396-404.

• Morgan, B., Yu, L. M., Solomon, T., & Ziebland, S. (2020). Assessing health research grant applications: A retrospective comparative review of a one-stage versus a two-stage application assessment process. Plos one, 15(3), e0230118.

• Pier, E. L., Brauer, M., Filut, A., Kaatz, A., Raclaw, J., Nathan, M. J., ... & Carnes, M. (2018). Low agreement among reviewers evaluating the same NIH grant applications. Proceedings of the National Academy of Sciences, 115(12), 2952-2957.

• Philipps, A. (2021). Science rules! A qualitative study of scientists’ approaches to grant lottery. Research Evaluation, 30(1), 102-111.

• Reinhart, M., & Schendzielorz, C. (2020). The lottery in Babylon—On the role of chance in scientific success. Journal of Responsible Innovation, 7, S25-S29.

• Roumbanis, L. (2019). Peer review or lottery? A critical analysis of two different forms of decision-making mechanisms for allocation of research grants. Science, Technology, & Human Values, 44(6), 994-1019.

• Roumbanis, L. (2020). Two dogmas of peer-reviewism. Journal of Responsible Innovation, 7(sup2), S129-S133.

• Roumbanis, L. (2021). Disagreement and agonistic chance in peer review. Science, Technology, & Human Values.

• Sato, S., Gygax, P. M., Randall, J., & Schmid Mast, M. (2021). The leaky pipeline in research grant peer review and funding decisions: challenges and future directions. Higher Education, 82(1), 145-162.

• Sattler, D. N., McKnight, P. E., Naney, L., & Mathis, R. (2015). Grant peer review: improving inter-rater reliability with training. PloS one, 10(6), e0130450.

• Serrano Velarde, K. (2018). The way we ask for money… The emergence and institutionalization of grant writing practices in academia. Minerva, 56(1), 85-107.

• Wahls, W. P. (2018). Point of view: The NIH must reduce disparities in funding to maximize its return on investments from taxpayers. Elife, 7, e34965.