Abstract
- Repeatability is the cornerstone of science, and it is particularly important for systematic reviews. However, little is known on how researchers’ choice of database, and search platform influence the repeatability of systematic reviews. Here, we aim to unveil how the computer environment and the location where the search was initiated from influence hit results.
- We present a comparative analysis of time-synchronized searches at different institutional locations in the world and evaluate the consistency of hits obtained within each of the search terms using different search platforms.
- We revealed a large variation among search platforms and showed that PubMed and Scopus returned consistent results to identical search strings from different locations. Google Scholar and Web of Science's Core Collection varied substantially both in the number of returned hits and in the list of individual articles depending on the search location and computing environment. Inconsistency in Web of Science results has most likely emerged from the different licensing packages at different institutions.
- To maintain scientific integrity and consistency, especially in systematic reviews, action is needed from both the scientific community and scientific search platforms to increase search consistency. Researchers are encouraged to report the search location and the databases used for systematic reviews, and database providers should make search algorithms transparent and revise access rules to titles behind paywalls. Additional options for increasing the repeatability and transparency of systematic reviews are storing both search metadata and hit results in open repositories and using Application Programming Interfaces (APIs) to retrieve standardized, machine-readable search metadata.
Document Type
Article
Publication Date
10-14-2021
Digital Object Identifier (DOI)
https://doi.org/10.1002/ece3.8154
Funding Information
This work is supported by a grant of "111 project" in China. G.P. is supported by a postdoctoral fellowship by the State Key Laboratory of Ecological Pest Control for Fujian and Taiwan Crops, and A.M. by the grants #20765-3/2018/FEKUTSTRAT and #TUDFO/47138/2019-ITM.
Related Content
All data are deposited in the Dryad Digital Repository (https://doi.org/10.5061/dryad.djh9w0w17). The computer script associated with this project is hosted on GitHub code repository (https://github.com/pozsgaig/search_location).
Repository Citation
Pozsgai, Gábor; Lövei, Gábor L.; Vasseur, Liette; Gurr, Geoff; Batáry, Péter; Korponai, János; Littlewood, Nick A.; Liu, Jian; Móra, Arnold; Obrycki, John J.; Reynolds, Olivia; Stockan, Jenni A.; VanVolkenburg, Heather; Zhang, Jie; Zhou, Wenwu; and You, Minsheng, "Irreproducibility in Searches of Scientific Literature: A Comparative Analysis" (2021). Entomology Faculty Publications. 227.
https://uknowledge.uky.edu/entomology_facpub/227
Appendix S1
ece38154-sup-0002-appendixs2.xlsx (13 kB)
Appendix S2
ece38154-sup-0003-appendixs3.docx (3057 kB)
Appendix S3
ece38154-sup-0004-appendixs4.pdf (6 kB)
Appendix S4
ece38154-sup-0005-appendixs5.pdf (7 kB)
Appendix S5
Notes/Citation Information
Published in Ecology and Evolution, v. 11, issue 21.
© 2021 The Authors
This is an open access article under the terms of the Creative Commons Attribution License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited.