Research assessment: guidelines when using quantitative indicators (for evaluation organisers)

In general

There are many quantitative indicators, and new ones are still being developed. Examples of quantitative indicators include: the number of (scholarly) publications, number of citations, citation impact (normalized), number of downloaded (open access) papers, number of users of deposited datasets, number of mentions of a study/publication on social media and/or academic networking sites, number of policy recommendations issued, number of supervised doctorates, size of one's own research group, number of public lectures, number of patents, amount of research funding acquired.

Quantitative indicators can be used in an evaluation, provided that this is done in a responsible and thoughtful manner, in line with Ghent University's policy on responsible evaluation and the responsible use of quantitative indicators. In particular, Ghent University does not the Journal Impact Factor to be used when assessing the quality of an individual researcher's work (cf. DORA).

More on Responsible Research Assessment and the responsible use of quantitative indicators

How to use?

Whether or not quantitative indicators are used in an evaluation (and if so, which ones) depends on the context, the objectives of the evaluation and often the level of aggregation. The higher the level of aggregation (e.g., department, faculty, university), the less feasible comprehensive qualitative evaluation methods, such as peer review, become. At the same time, the (undesirable) side effects of (certain) quantitative indicators have less significant implications at higher levels of aggregation. In practice, especially at lower levels of aggregation (e.g., when evaluating a researcher), a combination of (several) quantitative and qualitative indicators and methods is used. After all, research is too complex to be captured in a single indicator or descriptor.

If, in the context of an evaluation, it is considered appropriate to use one or more quantitative indicators, then each quantitative indicator should meet the following criteria:

  1. The quantitative indicator is relevant to the evaluation, and its use is necessary to conduct the evaluation in a high-quality manner.
  2. The quantitative indicator reflects (measures) what the organizers of the evaluation want to know (measure).
  3. The quantitative indicator is used for its intended purpose.
  4. The quantitative indicator is sufficiently reliable, robust, verifiable and reproducible.
  5. The quantitative indicator is composed in a transparent manner.
  6. The quantitative indicator is suitable to be used at the aggregation level of the evaluation.
  7. The quantitative indicator does not discriminate or can be corrected (normalized) to prevent bias (against disciplines, career stages, etc.).
  8. The quantitative indicator is as resistant as possible to 'gaming'.
  9. The quantitative indicator has no other unwanted side effects.
  10. All stakeholders (organizers, evaluators and those being evaluated) can understand and verify the quantitative indicator: the calculation method (formula) is known, and the data on which the quantitative indicator is based are accessible (or can be made available upon request from a stakeholder).
  11. Ghent University has sufficient internal expertise to use the quantitative indicator in an evaluation in a high-quality manner.
  12. The quantitative indicator is used in combination with other quantitative and qualitative indicators relevant to the evaluation.

 

More tips

Translated tip


Last modified May 2, 2024, 8:34 a.m.