(LB03-24-24) The Application of AI on Nutritional Systematic Reviews: A Validation Study

Objectives: To develop an innovative, artificial intelligence (AI) application designed specifically for accelerating systematic reviews by screening articles automatically using GPT-4.

Methods: Review Copilot requires researchers to first define their research goals and upload reference lists. Subsequently, Review Copilot can extract relevant information from articles and make independent decisions of inclusion or exclusion by comparing the extracted PICOS with the goals. 

To validate Review Copilot, we used previously published systematic reviews and meta-analyses as validation datasets and used human researcher-extracted data as the gold standard. The first validation dataset was related to the safety of using enteral formula with dietary fiber in hospitalized critical care patients. 463 studies were excluded and 19 articles were finally included in the review. The second validation dataset was about the associations between 100% orange juice and biomarkers of inflammation and oxidation in generally healthy populations. 1161 studies were excluded and 22 articles were finally included.

Results: Overall, for title/abstract screening, the average sensitivity for Review Copilot’s was 100%. In terms of the average specificity in the same screening stage, Review Copilot achieved an average of 91.75%. Regarding the balanced accuracy, Review Copilot had an overall average balanced accuracy of 95.9%. For full-text screening, Review Copilot had an overall average sensitivity of 97.6% and specificity of 47.4%. The balanced accuracy was 72.5%. Regarding the reliability of Review Copilot in two separate runs in the title/abstract screening stage, the average percent agreement between two Review Copilot’s runs was 97.0%. The average kappa statistic is 0.84. For full-text screening stage, the average percent agreement was 78.7% and the average kappa statistic was 0.55. Regarding efficiency, the average number of person hour spent by Review Copilot to complete title and abstract screening is 4 person hours per 1000 articles. For full-text screening, Review Copilot spent about 2 person hours to complete the screening of 100 articles in the PDF format.

Conclusions: Review Copilot could be a helpful and useful tool to streamline the screening processes for systematic reviews and help researchers save time for other tasks.

Funding Sources: Microsoft for Startups Program