Enhanced Multimodal Swin Transformer for Fine-Grained Plant Species Classification Using Contextual Metadata

Abstract

Fine-grained plant species classification is difficult because closely related species can exhibit highly similar morphological characteristics, limiting the effectiveness of image-only recognition. This study developed an Enhanced Multimodal Swin Transformer that combines visual features from PlantCLEF images with contextual metadata describing taxonomic hierarchy, geographical distribution, seasonal occurrence, and ecological characteristics. A total of 91,673 images representing 1,000 plant species were partitioned into training (63,366), validation (14,081), and test (14,226) sets. Global Biodiversity Information Facility (GBIF) data were retrieved and linked to species using scientific names. ResNet50 and EfficientNet-B0 were trained as image-only baselines under the same dataset partitions, while the proposed framework used a pertained Swin Transformer visual branch, metadata encoder, metadata reliability module, and attention-based multimodal fusion. The proposed model achieved 97.12% best validation accuracy and 97.15% test accuracy, with weighted precision, recall, and F1- score of 97.48%, 97.15%, and 97.07%, respectively. These results exceeded ResNet50 and EfficientNet-B0 test accuracy by 24.23 and 27.22 percentage points, respectively. Confusion matrices showed fewer off-diagonal errors for the proposed model, while Grad-CAM visualizations indicated attention to biologically meaningful structures such as flowers, leaves, and reproductive organs. The findings support multimodal transformer-based learning as an effective approach for fine-grained plant identification and biodiversity monitoring.