P4575 - A Multimodal Vision Transformer Model for Real-Time Histopathological and Morphological Classification of Colonic Polyps

Sri Harsha Boppana, MBBS, MD¹, Manaswitha Thota, MD², Gautam Maddineni, MD³, Sachin Sravan Kumar Komati, ⁴, Ashujot K. Dang, MD⁵, Fnu Aakash, MD³, Sarath Chandra Ponnada, ⁶, Sai Lakshmi Prasanna Komati, MBBS⁷, C. David Mintz, MD, PhD⁸
1Nassau University Medical Center, East Meadow, NY; 2Virginia Commonwealth University, Richmond, VA; 3Florida State University, Cape Coral, FL; 4Florida International University, Florida, FL; 5University of California Riverside School of Medicine, Riverside, CA; 6Great Eastern Medical School and Hospital, Srikakulam, Srikakulam, Andhra Pradesh, India; 7Government Medical College, Ongole, Ongole, Andhra Pradesh, India; 8Johns Hopkins University School of Medicine, Baltimore, MD

Introduction: Colorectal cancer is the second leading cause of cancer-related mortality worldwide, claiming over 900 000 lives annually. Automated deep learning for polyp detection has improved accuracy, but most systems rely solely on visual features and omit key clinical metadata.

Methods: We extracted video frames at 2 frames per second from the ERC PMP-v5 colonoscopy dataset and aligned each frame with corresponding high-resolution still images. We linked every image or frame to structured clinical metadata—patient age, polyp size, dysplasia grade, and differentiation status—via a unique Patient ID. Continuous variables underwent scaling and categorical fields received one-hot encoding. Parallelly, we resized images to 224×224 pixels, normalized pixel intensities, and passed them through a pre-trained Vision Transformer (ViT) to generate 768-dimensional feature vectors. We concatenated these visual embeddings with the processed metadata to form a unified input, then trained two Random Forest classifiers—one for histopathological prediction and another for morphological classification. All 217 patients were partitioned at the patient level into training (60%), validation (20%), and testing (20%) cohorts to ensure strict separation and to prevent data leakage .

Results: On the independent test cohort, the pathology classifier achieved 99.05% overall accuracy in distinguishing histological types (Tubular, Villous, Serrated, Adenocarcinoma), and the morphology classifier reached 94.94% accuracy for Paris classification subtypes (e.g., 0-IIa, 0-Ip). Both models demonstrated stable performance across patient subgroups, with no evidence of overfitting on the validation data. The fusion of image features and clinical metadata yielded robust, real-time predictive capability suitable for integration into endoscopic workflows .

Discussion: By fusing Vision Transformer–derived image embeddings with encoded patient metadata, our model achieved 99 % accuracy for histopathological prediction and 95 % accuracy for Paris-based morphology classification on an independent test set. This multimodal approach can standardize polyp assessment, reduce inter-observer variability, and deliver real-time decision support during colonoscopy. Prospective validation and seamless integration into endoscopy platforms are the next critical steps.

Disclosures:
Sri Harsha Boppana indicated no relevant financial relationships.
Manaswitha Thota indicated no relevant financial relationships.
Gautam Maddineni indicated no relevant financial relationships.
Sachin Sravan Kumar Komati indicated no relevant financial relationships.
Ashujot Dang indicated no relevant financial relationships.
Fnu Aakash indicated no relevant financial relationships.
Sarath Chandra Ponnada indicated no relevant financial relationships.
Sai Lakshmi Prasanna Komati indicated no relevant financial relationships.
C. David Mintz indicated no relevant financial relationships.

Sri Harsha Boppana, MBBS, MD¹, Manaswitha Thota, MD², Gautam Maddineni, MD³, Sachin Sravan Kumar Komati, ⁴, Ashujot K. Dang, MD⁵, Fnu Aakash, MD³, Sarath Chandra Ponnada, ⁶, Sai Lakshmi Prasanna Komati, MBBS⁷, C. David Mintz, MD, PhD⁸. P4575 - A Multimodal Vision Transformer Model for Real-Time Histopathological and Morphological Classification of Colonic Polyps, ACG 2025 Annual Scientific Meeting Abstracts. Phoenix, AZ: American College of Gastroenterology.