Scoring vs. Ranking

An Experimental Study of Idea Evaluation Processes

Zhijian Cui, Shijith Kumar PM, Dilney Gonçalves

Research output: Contribution to journalArticle

2 Citations (Scopus)

Abstract

Using an online experiment, this study examines the efficacy of two idea evaluation processes: scoring vs. ranking in the context of innovation management. In the scoring process, the evaluators are asked to rate the quality of each idea by assigning it a score (e.g., from 0 to 10), while in the ranking process the evaluator simply orders all ideas according to their perceived qualities. Our results suggest that the scoring process strictly outperforms the ranking process in terms of the likelihood of selecting the highest-quality ideas. This result remains robust, irrespective of the possibility of allowing ties in the ranking process. However, when the number of ideas to be evaluated is reduced from eight to three, the efficacies of the two idea evaluation processes become similar. Additionally, we find that the efficacy of the ranking process is significantly improved when additional information is provided, yet the efficacy of the scoring process does not change with more information. Based on the observations from the experimental data, we propose and test an explanatory model in which the information becomes a cue that directs the participants’ efforts to evaluate the ideas (i.e., the time taken for evaluation).

Original languageEnglish
Pages (from-to)176-188
Number of pages13
JournalProduction and Operations Management
Volume28
Issue number1
DOIs
Publication statusPublished - Jan 1 2019

Fingerprint

Innovation
Experiments
Experimental study
Ranking
Scoring
Evaluation
Efficacy

Keywords

  • data-driven research
  • experimental study
  • idea evaluation
  • ranking
  • scoring

ASJC Scopus subject areas

  • Management Science and Operations Research
  • Industrial and Manufacturing Engineering
  • Management of Technology and Innovation

Cite this

Scoring vs. Ranking : An Experimental Study of Idea Evaluation Processes. / Cui, Zhijian; Kumar PM, Shijith; Gonçalves, Dilney.

In: Production and Operations Management, Vol. 28, No. 1, 01.01.2019, p. 176-188.

Research output: Contribution to journalArticle

Cui, Zhijian ; Kumar PM, Shijith ; Gonçalves, Dilney. / Scoring vs. Ranking : An Experimental Study of Idea Evaluation Processes. In: Production and Operations Management. 2019 ; Vol. 28, No. 1. pp. 176-188.
@article{0a53e421972844c185532cedea47830d,
title = "Scoring vs. Ranking: An Experimental Study of Idea Evaluation Processes",
abstract = "Using an online experiment, this study examines the efficacy of two idea evaluation processes: scoring vs. ranking in the context of innovation management. In the scoring process, the evaluators are asked to rate the quality of each idea by assigning it a score (e.g., from 0 to 10), while in the ranking process the evaluator simply orders all ideas according to their perceived qualities. Our results suggest that the scoring process strictly outperforms the ranking process in terms of the likelihood of selecting the highest-quality ideas. This result remains robust, irrespective of the possibility of allowing ties in the ranking process. However, when the number of ideas to be evaluated is reduced from eight to three, the efficacies of the two idea evaluation processes become similar. Additionally, we find that the efficacy of the ranking process is significantly improved when additional information is provided, yet the efficacy of the scoring process does not change with more information. Based on the observations from the experimental data, we propose and test an explanatory model in which the information becomes a cue that directs the participants’ efforts to evaluate the ideas (i.e., the time taken for evaluation).",
keywords = "data-driven research, experimental study, idea evaluation, ranking, scoring",
author = "Zhijian Cui and {Kumar PM}, Shijith and Dilney Gon{\cc}alves",
year = "2019",
month = "1",
day = "1",
doi = "10.1111/poms.12910",
language = "English",
volume = "28",
pages = "176--188",
journal = "Production and Operations Management",
issn = "1059-1478",
publisher = "Wiley Blackwell",
number = "1",

}

TY - JOUR

T1 - Scoring vs. Ranking

T2 - An Experimental Study of Idea Evaluation Processes

AU - Cui, Zhijian

AU - Kumar PM, Shijith

AU - Gonçalves, Dilney

PY - 2019/1/1

Y1 - 2019/1/1

N2 - Using an online experiment, this study examines the efficacy of two idea evaluation processes: scoring vs. ranking in the context of innovation management. In the scoring process, the evaluators are asked to rate the quality of each idea by assigning it a score (e.g., from 0 to 10), while in the ranking process the evaluator simply orders all ideas according to their perceived qualities. Our results suggest that the scoring process strictly outperforms the ranking process in terms of the likelihood of selecting the highest-quality ideas. This result remains robust, irrespective of the possibility of allowing ties in the ranking process. However, when the number of ideas to be evaluated is reduced from eight to three, the efficacies of the two idea evaluation processes become similar. Additionally, we find that the efficacy of the ranking process is significantly improved when additional information is provided, yet the efficacy of the scoring process does not change with more information. Based on the observations from the experimental data, we propose and test an explanatory model in which the information becomes a cue that directs the participants’ efforts to evaluate the ideas (i.e., the time taken for evaluation).

AB - Using an online experiment, this study examines the efficacy of two idea evaluation processes: scoring vs. ranking in the context of innovation management. In the scoring process, the evaluators are asked to rate the quality of each idea by assigning it a score (e.g., from 0 to 10), while in the ranking process the evaluator simply orders all ideas according to their perceived qualities. Our results suggest that the scoring process strictly outperforms the ranking process in terms of the likelihood of selecting the highest-quality ideas. This result remains robust, irrespective of the possibility of allowing ties in the ranking process. However, when the number of ideas to be evaluated is reduced from eight to three, the efficacies of the two idea evaluation processes become similar. Additionally, we find that the efficacy of the ranking process is significantly improved when additional information is provided, yet the efficacy of the scoring process does not change with more information. Based on the observations from the experimental data, we propose and test an explanatory model in which the information becomes a cue that directs the participants’ efforts to evaluate the ideas (i.e., the time taken for evaluation).

KW - data-driven research

KW - experimental study

KW - idea evaluation

KW - ranking

KW - scoring

UR - http://www.scopus.com/inward/record.url?scp=85050473453&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85050473453&partnerID=8YFLogxK

U2 - 10.1111/poms.12910

DO - 10.1111/poms.12910

M3 - Article

VL - 28

SP - 176

EP - 188

JO - Production and Operations Management

JF - Production and Operations Management

SN - 1059-1478

IS - 1

ER -