Scoring vs. Ranking: An Experimental Study of Idea Evaluation Processes

Zhijian Cui, Shijith Kumar PM, Dilney Gonçalves

Research output: Contribution to journalArticlepeer-review

8 Citations (Scopus)

Abstract

Using an online experiment, this study examines the efficacy of two idea evaluation processes: scoring vs. ranking in the context of innovation management. In the scoring process, the evaluators are asked to rate the quality of each idea by assigning it a score (e.g., from 0 to 10), while in the ranking process the evaluator simply orders all ideas according to their perceived qualities. Our results suggest that the scoring process strictly outperforms the ranking process in terms of the likelihood of selecting the highest-quality ideas. This result remains robust, irrespective of the possibility of allowing ties in the ranking process. However, when the number of ideas to be evaluated is reduced from eight to three, the efficacies of the two idea evaluation processes become similar. Additionally, we find that the efficacy of the ranking process is significantly improved when additional information is provided, yet the efficacy of the scoring process does not change with more information. Based on the observations from the experimental data, we propose and test an explanatory model in which the information becomes a cue that directs the participants’ efforts to evaluate the ideas (i.e., the time taken for evaluation).

Original languageEnglish
Pages (from-to)176-188
Number of pages13
JournalProduction and Operations Management
Volume28
Issue number1
DOIs
Publication statusPublished - Jan 1 2019

Keywords

  • data-driven research
  • experimental study
  • idea evaluation
  • ranking
  • scoring

ASJC Scopus subject areas

  • Management Science and Operations Research
  • Industrial and Manufacturing Engineering
  • Management of Technology and Innovation

Fingerprint Dive into the research topics of 'Scoring vs. Ranking: An Experimental Study of Idea Evaluation Processes'. Together they form a unique fingerprint.

Cite this