Abstract
The mapping of metabolite-specific data to pathways within cellular metabolism is a major data analysis step needed for biochemical interpretation. A variety of machine learning approaches, particularly deep learning approaches, have been used to predict these metabolite-to-pathway mappings, utilizing a training dataset of known metabolite-to-pathway mappings. A few such training datasets have been derived from the Kyoto Encyclopedia of Genes and Genomes (KEGG). However, several prior published machine learning approaches utilized an erroneous KEGG-derived training dataset that used SMILES molecular representations strings (KEGG-SMILES dataset) and contained a sizable proportion (~26%) duplicate entries. The presence of so many duplicates taint the training and testing sets generated from k-fold cross-validation of the KEGG-SMILES dataset. Therefore, the k-fold cross-validation performance of the resulting machine learning models was grossly inflated by the erroneous presence of these duplicate entries. Here we describe and evaluate the KEGG-SMILES dataset so that others may avoid using it. We also identify the prior publications that utilized this erroneous KEGG-SMILES dataset so their machine learning results can be properly and critically evaluated. In addition, we demonstrate the reduction of model k-fold cross-validation (CV) performance after de-duplicating the KEGG-SMILES dataset. This is a cautionary tale about properly vetting prior published benchmark datasets before using them in machine learning approaches. We hope others will avoid similar mistakes.
Document Type
Article
Publication Date
5-2024
Funding Information
This work has been supported by the National Science Foundation [NSF 2020026 to HNBM] and the National Institute of Environmental Health Sciences [P42ES007380].
Related Content
The dataset, all code, and all results are provided in the following Figshare item: https://doi.org/10.6084/m9.figshare.22661185.
Repository Citation
Huckvale, Erik D. and Moseley, Hunter N. B., "A cautionary tale about properly vetting datasets used in supervised learning predicting metabolic pathway involvement" (2024). Markey Cancer Center Faculty Publications. 179.
https://uknowledge.uky.edu/markey_facpub/179
Notes/Citation Information
Copyright: © 2024 Huckvale, Moseley. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Citation: Huckvale ED, Moseley HNB (2024) A cautionary tale about properly vetting datasets used in supervised learning predicting metabolic pathway involvement. PLoS ONE 19(5): e0299583. https://doi.org/10.1371/journal.pone.0299583