- Subscribe
- Log In More
Log in via Institution
Log in via OpenAthens
Log in using your username and password
- Basket
- Search More
Advanced search
- Latest content
- Current issue
- Archive
- Authors
- About
Advanced search
- CloseMore
Main menu
- Latest content
- Current issue
- Archive
- Authors
- About
- Subscribe
- Log in More
Log in via Institution
Log in via OpenAthens
Log in using your username and password
- BMJ Journals
You are here
- Home
- Online First
- Integrating large language models in systematic reviews: a framework and case study using ROBINS-I for risk of bias assessment
Email alerts
Article Text
Article menu
- Article Text
- Article info
- Citation Tools
- Share
- Rapid Responses
- Article metrics
- Alerts
Research methods and reporting
Integrating large language models in systematic reviews: a framework and case study using ROBINS-I for risk of bias assessment
- http://orcid.org/0000-0001-9531-4990Bashar Hasan1,2,
- http://orcid.org/0000-0001-9225-1197Samer Saadi1,2,
- Noora S Rajjoub1,
- Moustafa Hegazi1,2,
- Mohammad Al-Kordi1,2,
- Farah Fleti1,2,
- Magdoleen Farah1,2,
- Irbaz B Riaz3,
- Imon Banerjee4,5,
- http://orcid.org/0000-0002-9368-6149Zhen Wang1,6,
- http://orcid.org/0000-0001-5502-5975Mohammad Hassan Murad1,2
- 1Kern Center for the Science of Healthcare Delivery, Mayo Clinic, Rochester, Minnesota, USA
- 2Public Health, Infectious Diseases and Occupational Medicine, Mayo Clinic, Rochester, Minnesota, USA
- 3Division of Hematology-Oncology Department of Medicine, Mayo Clinic, Rochester, Minnesota, USA
- 4Department of Radiology, Mayo Clinic Arizona, Scottsdale, Arizona, USA
- 5School of Computing and Augmented Intelligence, Arizona State University, Tempe, Arizona, USA
- 6Health Care Policy and Research, Mayo Clinic Minnesota, Rochester, Minnesota, USA
- Correspondence to Dr Bashar Hasan, Mayo Clinic, Rochester, MN 55905, USA; Hasan.Bashar{at}mayo.edu
Abstract
Large language models (LLMs) may facilitate and expedite systematic reviews, although the approach to integrate LLMs in the review process is unclear. This study evaluates GPT-4 agreement with human reviewers in assessing the risk of bias using the Risk Of Bias In Non-randomised Studies of Interventions (ROBINS-I) tool and proposes a framework for integrating LLMs into systematic reviews. The case study demonstrated that raw per cent agreement was the highest for the ROBINS-I domain of ‘Classification of Intervention’. Kendall agreement coefficient was highest for the domains of ‘Participant Selection’, ‘Missing Data’ and ‘Measurement of Outcomes’, suggesting moderate agreement in these domains. Raw agreement about the overall risk of bias across domains was 61% (Kendall coefficient=0.35). The proposed framework for integrating LLMs into systematic reviews consists of four domains: rationale for LLM use, protocol (task definition, model selection, prompt engineering, data entry methods, human role and success metrics), execution (iterative revisions to the protocol) and reporting. We identify five basic task types relevant to systematic reviews: selection, extraction, judgement, analysis and narration. Considering the agreement level with a human reviewer in the case study, pairing artificial intelligence with an independent human reviewer remains required.
- Evidence-Based Practice
- Methods
- Systematic Reviews as Topic
Data availability statement
Data are available upon reasonable request. Search strategy, selection process flowchart, prompts and boxes containing included SRs and studies are available in the appendix. Analysed datasheet is available upon request.
Statistics from Altmetric.com
Request Permissions
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.
- Evidence-Based Practice
- Methods
- Systematic Reviews as Topic
Data availability statement
Data are available upon reasonable request. Search strategy, selection process flowchart, prompts and boxes containing included SRs and studies are available in the appendix. Analysed datasheet is available upon request.
View Full Text
Footnotes
Twitter @BasharHasanMD, @M_Hassan_Murad
Contributors MHM and BH conceived this study. BH, SS, MH, MA-K, FF, MF, ZW, IBR, IB and NSR participated in data identification, extraction and analysis. MHM, SS, IBR and IB wrote the first draft. All authors critically revised the manuscript and approved the final version. BH is the guarantor.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests None declared.
Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.
Read the full text or download the PDF:
Subscribe
Log in via Institution
Log in via OpenAthens
Log in using your username and password
Read the full text or download the PDF:
Subscribe
Log in via Institution
Log in via OpenAthens
