TY - JOUR
KW - peer review
KW - Rating and ranking
KW - Research funding
AU - Stephen A. Gallo
AU - Michael Pearce
AU - Carole J. Lee
AU - Elena A. Erosheva
AB - In many grant review settings, proposals are selected for funding on the basis of summary statistics of review ratings. Challenges of this approach (including the presence of ties and unclear ordering of funding preference for proposals) could be mitigated if rankings such as top-k preferences or paired comparisons, which are local evaluations that enforce ordering across proposals, were also collected and incorporated in the analysis of review ratings. However, analyzing ratings and rankings simultaneously has not been done until recently. This paper describes a practical method for integrating rankings and scores and demonstrates its usefulness for making funding decisions in real-world applications.
BT - Research Integrity and Peer Review
DA - 2023-07-24
DO - 10.1186/s41073-023-00131-7
LA - en
N2 - In many grant review settings, proposals are selected for funding on the basis of summary statistics of review ratings. Challenges of this approach (including the presence of ties and unclear ordering of funding preference for proposals) could be mitigated if rankings such as top-k preferences or paired comparisons, which are local evaluations that enforce ordering across proposals, were also collected and incorporated in the analysis of review ratings. However, analyzing ratings and rankings simultaneously has not been done until recently. This paper describes a practical method for integrating rankings and scores and demonstrates its usefulness for making funding decisions in real-world applications.
PY - 2023
EP - 10
ST - A new approach to grant review assessments
T2 - Research Integrity and Peer Review
TI - A new approach to grant review assessments: score, then rank
UR - https://doi.org/10.1186/s41073-023-00131-7
VL - 8
Y2 - 2024-02-06
SN - 2058-8615
ER -