Abstract
Using an online experiment, this study examines the efficacy of two idea evaluation processes: scoring vs. ranking in the context of innovation management. In the scoring process, the evaluators are asked to rate the quality of each idea by assigning it a score (e.g., from 0 to 10), while in the ranking process the evaluator simply orders all ideas according to their perceived qualities. Our results suggest that the scoring process strictly outperforms the ranking process in terms of the likelihood of selecting the highest-quality ideas. This result remains robust, irrespective of the possibility of allowing ties in the ranking process. However, when the number of ideas to be evaluated is reduced from eight to three, the efficacies of the two idea evaluation processes become similar. Additionally, we find that the efficacy of the ranking process is significantly improved when additional information is provided, yet the efficacy of the scoring process does not change with more information. Based on the observations from the experimental data, we propose and test an explanatory model in which the information becomes a cue that directs the participants’ efforts to evaluate the ideas (i.e., the time taken for evaluation).
| Original language | English |
|---|---|
| Pages (from-to) | 176-188 |
| Number of pages | 13 |
| Journal | Production and Operations Management |
| Volume | 28 |
| Issue number | 1 |
| DOIs | |
| Publication status | Published - Jan 1 2019 |
Fingerprint
Keywords
- data-driven research
- experimental study
- idea evaluation
- ranking
- scoring
ASJC Scopus subject areas
- Management Science and Operations Research
- Industrial and Manufacturing Engineering
- Management of Technology and Innovation
Cite this
Scoring vs. Ranking : An Experimental Study of Idea Evaluation Processes. / Cui, Zhijian; Kumar PM, Shijith; Gonçalves, Dilney.
In: Production and Operations Management, Vol. 28, No. 1, 01.01.2019, p. 176-188.Research output: Contribution to journal › Article
}
TY - JOUR
T1 - Scoring vs. Ranking
T2 - An Experimental Study of Idea Evaluation Processes
AU - Cui, Zhijian
AU - Kumar PM, Shijith
AU - Gonçalves, Dilney
PY - 2019/1/1
Y1 - 2019/1/1
N2 - Using an online experiment, this study examines the efficacy of two idea evaluation processes: scoring vs. ranking in the context of innovation management. In the scoring process, the evaluators are asked to rate the quality of each idea by assigning it a score (e.g., from 0 to 10), while in the ranking process the evaluator simply orders all ideas according to their perceived qualities. Our results suggest that the scoring process strictly outperforms the ranking process in terms of the likelihood of selecting the highest-quality ideas. This result remains robust, irrespective of the possibility of allowing ties in the ranking process. However, when the number of ideas to be evaluated is reduced from eight to three, the efficacies of the two idea evaluation processes become similar. Additionally, we find that the efficacy of the ranking process is significantly improved when additional information is provided, yet the efficacy of the scoring process does not change with more information. Based on the observations from the experimental data, we propose and test an explanatory model in which the information becomes a cue that directs the participants’ efforts to evaluate the ideas (i.e., the time taken for evaluation).
AB - Using an online experiment, this study examines the efficacy of two idea evaluation processes: scoring vs. ranking in the context of innovation management. In the scoring process, the evaluators are asked to rate the quality of each idea by assigning it a score (e.g., from 0 to 10), while in the ranking process the evaluator simply orders all ideas according to their perceived qualities. Our results suggest that the scoring process strictly outperforms the ranking process in terms of the likelihood of selecting the highest-quality ideas. This result remains robust, irrespective of the possibility of allowing ties in the ranking process. However, when the number of ideas to be evaluated is reduced from eight to three, the efficacies of the two idea evaluation processes become similar. Additionally, we find that the efficacy of the ranking process is significantly improved when additional information is provided, yet the efficacy of the scoring process does not change with more information. Based on the observations from the experimental data, we propose and test an explanatory model in which the information becomes a cue that directs the participants’ efforts to evaluate the ideas (i.e., the time taken for evaluation).
KW - data-driven research
KW - experimental study
KW - idea evaluation
KW - ranking
KW - scoring
UR - http://www.scopus.com/inward/record.url?scp=85050473453&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85050473453&partnerID=8YFLogxK
U2 - 10.1111/poms.12910
DO - 10.1111/poms.12910
M3 - Article
VL - 28
SP - 176
EP - 188
JO - Production and Operations Management
JF - Production and Operations Management
SN - 1059-1478
IS - 1
ER -