P0020 - Detection of Ultra-High Risk Pancreatic Cancer Diagnoses Using a Custom NLP Pipeline Incorporating a Large Language Model: A Pilot Safety Net for Missed Diagnoses

Award: ACG Presidential Poster Award

Aman Mohapatra, MD¹, Arvind Ravi, MD², Heather Israel Deaconess. Hardway, BA², Joseph Feuerstein, MD³
1Beth Israel Deaconess Medical Center, Harvard Medical School, Folsom, CA; 2Halo Solutions, Boston, MA; 3Beth Israel Deaconess Medical Center, Boston, MA

Introduction: Radiologic findings suggestive of malignancy may be missed or inadequately acted upon in large health systems, leading to delayed diagnoses. Natural language processing (NLP) offers the potential to surface high-risk cases buried in unstructured data. We developed a custom NLP pipeline that incorporates a commercial Large Language Model (LLM) along with additional filtering components to flag radiology reports indicative of new, unexpected, or high-risk cancer findings. This pilot uses pancreatic cancer to evaluate whether such a system can uncover clinically significant yet overlooked diagnoses.

Methods: We used NLP to create a custom pipeline using 589,513 radiology reports from a large health system (Figure 1). The model flagged 2,897 reports related to pancreatic cancer; 43 were identified as suspicious for new, unexpected, or particularly worrisome findings. We then conducted detailed manual review of 10 consecutively flagged cases to assess follow-up, diagnostic closure, and timeliness of response.

Results: Among the 10 patients, 9 had findings concerning for new or progressive pancreatic cancer, and 1 had stable known disease. Gaps in follow-up were common: two patients lacked biopsy or additional imaging months after initial scan. Three had care refusal or communication breakdowns without documented contingency planning. Two presented emergently with symptoms but lacked outpatient coordination. One had cystic disease with malignant transformation and no interim evaluation. These ultra-high-risk scenarios occurred in fewer than 1 in 10,000 reports—illustrating the rarity and value of LLM-enabled detection (Figure 2).

Discussion: This pilot shows that an LLM-based pipeline, combined with targeted filtering, can detect rare but urgent radiologic findings that may otherwise be missed. AI-driven safety nets could help health systems identify overlooked malignancies and improve early cancer detection. These cases highlight both the potential for harm and the opportunity for intervention when actionable signals are surfaced early.


Figure: Figure 1. Radiology-to-Action Workflow in LLM-Enhanced Pipeline. A conceptual diagram outlining the process used in the study: radiology reports are analyzed through a custom NLP pipeline incorporating a Large Language Model (LLM) and supporting filters. Outputs flagged as potentially high-risk are manually reviewed to assess diagnostic closure and timeliness of clinical response.


Figure: Figure 2. Examples of LLM-Flagged High-Risk Pancreatic Cancer Cases with Delayed or Incomplete Follow-Up. Two patient case timelines showing radiologic findings highly concerning for pancreatic malignancy, identified by the custom NLP/LLM pipeline. Despite clear documentation of high-risk language and provider intent to follow up, care gaps—including delays, incomplete referrals, and lack of biopsy confirmation—were evident months after initial imaging.

Disclosures:
Aman Mohapatra indicated no relevant financial relationships.
Arvind Ravi indicated no relevant financial relationships.
Heather Hardway indicated no relevant financial relationships.
Joseph Feuerstein indicated no relevant financial relationships.

Aman Mohapatra, MD¹, Arvind Ravi, MD², Heather Israel Deaconess. Hardway, BA², Joseph Feuerstein, MD³. P0020 - Detection of Ultra-High Risk Pancreatic Cancer Diagnoses Using a Custom NLP Pipeline Incorporating a Large Language Model: A Pilot Safety Net for Missed Diagnoses, ACG 2025 Annual Scientific Meeting Abstracts. Phoenix, AZ: American College of Gastroenterology.