CURE recognizes that use of the term “reproducibility” varies across contexts, the definition of which can become confounding when other words such as “replicability” and “repeatability” are used to express the same or similar concepts.  CURE does not seek to establish a universal standard for use of any of these terms; rather, it believes that it is necessary for data curation practitioners to understand their various meanings and nuances.  The following are examples of ways in which these terms have been defined.

Goodman, Fanelli, & Ioannidis (p. 2)

  • Methods reproducibility: The ability to implement, as exactly as possible, the experimental procedures, with the same data and tools, to obtain the same results.
  • Results reproducibility: The production of corroborating results in a new study, having followed the same experimental methods.
  • Inferential reproducibility: The making of knowledge claims of similar strength from a study replication or reanalysis.

Open Science Collaboration (p. 300-301)

  • Reproducibility: (Related to repeatability and replicability) Refers to whether research findings occur. Broadly, reproducibility refers to direct replication, an attempt to replicate the original observation using the same methods of a previous investigation but collecting unique observations.Most broadly, reproducibility refers to conceptual replication, an attempt to validate the interpretation of the original observation by manipulating or measuring the same conceptual variables using different techniques.

NSF Workshop (p. 32)

  • Reproducibility: The ability of an experiment or calculation to be duplicated by other researchers working independently.
  • Repeatability: The ability of an experiment or calculation to be duplicated by using the same method.
  • Reliability: The extent to which a research method produces the same results each time it is applied to the same system. A scientific result is said to have a high reliability if the same result is obtained within stated uncertainty under consistent conditions.

Patel, Peng, & Leek (p. 3)

  • Reproducible: Given a population, hypothesis, experimental design, experimenter, data, analysis plan, and code you get the same parameter estimates in a new analysis.
  • Replicable: Given a population, hypothesis, experimental design, and analysis plan you get consistent estimates when you recollect data and redo the analysis.

Pröll & Rauber (Introduction)

  • Reproducible: If and only if consistent, scientific results can be obtained, by processing the same data with the same algorithms using the same tools. For an experiment to be reproducible, we need to have knowledge of at least the following information: research data and metadata used; methods applied in the experiment; and ools, software and execution environment used in the experiment.

Stodden (Terminology Section)

  • Repeatability: (Same team, same experimental setup) The measurement can be obtained with stated precision by the same team using the same measurement procedure, the same measuring system, under the same operating conditions, in the same location on multiple trials. For computational experiments, this means that a researcher can reliably repeat her own computation.
  • Replicability: (Different team, same experimental setup) The measurement can be obtained with stated precision by a different team using the same measurement procedure, the same measuring system, under the same operating conditions, in the same or a different location on multiple trials. For computational experiments, this means that an independent group can obtain the same result using the author’s own artifacts.
  • Reproducibility: (Different team, different experimental setup) The measurement can be obtained with stated precision by a different team, a different measuring system, in a different location on multiple trials. For computational experiments, this means that an independent group can obtain the same result using artifacts which they develop completely independently.

Goodman, S. N., Fanelli, D., & Ioannidis, J. P. A. (2016). What does research reproducibility mean? Science Translational Medicine, 8(341), 341ps12. https://doi.org/10.1126/scitranslmed.aaf5027

National Science Foundation. (2017). Workshop on robustness, reliability, and reproducibility in scientific research. Retrieved from http://www.mrsec.harvard.edu/2017NSFReliability/include/NSF_Workshop_Robustness.Reliability.Reproducibility.Report.pdf

Open Science Collaboration. (2014). The reproducibility project. In V. Stodden, F. Leisch, & R. D. Peng (Eds.), Implementing Reproducible Research (pp. 299-323). Chapman and Hall/CRC.

Patil, P., Peng, R. D., & Leek, J. (2016). A statistical definition for reproducibility and replicability. BioRxiv, 066803. https://doi.org/10.1101/066803

Pröll, S., & Rauber, A. (2017). Enabling reproducibility for small and large scale research data sets. D-Lib Magazine, 23(1/2), https://doi.org/10.1045/january2017-proell

Stodden, V. (2016). Artifact review and badging. Retrieved from https://www.acm.org/publications/policies/artifact-review-badging

Comments are closed.