ConV-ViT: Feature Fusion-based Detection of Gastrointestinal Abnormalities using CNN and ViT in WCE Images

Yassine Oukdach, Zakaria Kerkaou, Mohamed El Ansari, Lahcen Koutti, Ahmed Fouad El Ouafdi

2023 · 11 citations · 26 references

Concepts

Abstract

Vision Transformer (ViT) and its variants have gained significant prominence in computer vision due to their exceptional performance across various tasks. However, ViTs are data-hungry models that require vast amounts of data for training. Given the scarcity of medical data, this paper presents a fine-tuned vision transformer specifically designed for small-size datasets. We fine-tuned the original model using convolutional neural networks (CNNs) to extract both high and low-level features from wireless capsule endoscopy (WCE) images. In this work, we integrate a CNN module into the original ViT to extract features from WCE patches, which are then fused with the original ViT features. The classification is accomplished using a multilayer perceptron (MLP) to categorize images into normal and abnormal categories on the Kvasir Capsule Endoscopy dataset, as well as bleeding or non-bleeding categories on the Red Lesion Endoscopy dataset. The experimental findings substantiate the efficacy of the suggested approach, yielding favorable outcomes in comparison to other state-of-the-art methods.

References

26