Stabilized Progressive Fine-Tuning (SPFT)

Abstract

Foundation models have demonstrated strong performance in medical imaging; however, their ability to generalize across heterogeneous datasets, imaging modalities, and downstream tasks remains limited. Existing approaches often rely on modality-specific architectures, task-specific training pipelines, or large-scale retraining, which hinder scalability and practical deployment in real-world clinical settings.
In this work, we present a unified and reproducible optimization-driven framework for improving cross-dataset, cross-modality, and cross-task generalization of foundation models in medical imaging. Rather than introducing new architectures, we investigate how principled optimization strategies can enhance performance, stability, and transferability across diverse vision tasks. Building on the observation that pretrained encoders capture modality-agnostic representations, we propose a lightweight adaptation strategy, termed Stabilized Progressive Fine-Tuning (SPFT), which combines staged fine-tuning, progressive layer unfreezing, class-aware loss weighting, and Exponential Moving Average (EMA) stabilization.
We evaluate our approach across multiple datasets and tasks, including chest radiograph classification on ChestX-ray14 (Wang et al. 2017) and CheXpert (Irvin et al. 2019), dermoscopic image classification on HAM10000 (Tschandl et al. 2018), object detection on VinDr-CXR, and medical image segmentation using SAM-based models. Importantly, the same SPFT strategy is applied consistently across all tasks and architectures, including transformer-based and CNN-based models.
Experimental results demonstrate that our approach achieves strong and stable performance across classification (AUC up to 0.95), detection (mAP@50 up to 0.76), and segmentation (Dice up to 0.96), while maintaining low variance across runs. We further show that optimization plays a critical role in improving generalization and stability, even when architectural choices vary significantly.
These findings highlight that carefully designed optimization strategies can serve as a scalable and effective alternative to task-specific architectural modifications, enabling unified and robust medical imaging systems across diverse datasets and problem settings.