Multi-modal fusion and transferable deep learning for rare disease detection: a CNN-Transformer framework with cross-domain adaptation on limited CT and MRI data

Jingyu Tang

doi:10.54254/2977-3903/2025.24722

Abstract

Medical imaging diagnosis of rare diseases faces the dual challenge of scarce labeled data and significant differences in equipment. This study proposes a hybrid framework integrating CT and MRI modalities, combining CNN and Transformer architectures and introducing an adversarial domain adaptation mechanism. The dedicated CNN encoder extracts fine-grained local features, while the Transformer module captures long-range cross-modal dependencies. The gradient inversion domain discriminator aligns the feature distribution of different scanning devices to ensure the device independence of the model. On the two public datasets of neurological diseases, intracranial hemorrhage and demyelinating lesions, the average accuracy rate of this model reached 91.3%, and the F1 score was 0.89, which is 5 to 10 percentage points higher than the single-mode and pure Transformer baseline. Ablation experiments confirmed that the domain-based adversarial transformer component and training contributed to significant performance gains. In the cross-domain (CT to MRI) experiment, the domain adaptation technique increased the F1 score from 0.74 to 0.84. These results highlight the effectiveness of local feature extraction, global context modeling, and collaborative adversarial alignment schemes in multi-agency scenarios with sparse data. Future research will be extended to three-dimensional volume data, integrate semi-supervised learning of unlabeled images, and optimize the reasoning process of real-time clinical decision support.

Keywords: rare disease detection , multi-modal fusion , cnn-transformer , domain adaptation , transfer learning

Article Information

Received: 29 May 2025

Published: 1 July 2025

DOI: https://doi.org/10.54254/2977-3903/2025.24722

Article Type: Research Article

Cite this article

Tang,J. (2025). Multi-modal fusion and transferable deep learning for rare disease detection: a CNN-Transformer framework with cross-domain adaptation on limited CT and MRI data. Advances in Engineering Innovation,16(6),144-148.

Data availability

The datasets used and/or analyzed during the current study will be available from the authors upon reasonable request.

Disclaimer/Publisher's Note

The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of EWA Publishing and/or the editor(s). EWA Publishing and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

About this Journal

Journal: Advances in Engineering Innovation

Volume: Vol.16, Issue 6

ISSN: 2977-3903 (Print) / 2977-3911 (Online)

Copyright & License

[1]Lee, J., Liu, C., Kim, J., Chen, Z., Sun, Y., Rogers, J. R., Chung, W. K., & Weng, C. (2022). Deep learning for rare disease: A scoping review.Journal of Biomedical Informatics, 135, 104227. https: //doi.org/10.1016/j.jbi.2022.104227

[2]Salehi, A. W., Khan, S., Gupta, G., Alabduallah, B. I., Almjally, A., Alsolai, H., Siddiqui, T., & Mellit, A. (2023). A study of CNN and transfer learning in medical imaging: Advantages, challenges, future scope.Sustainability, 15(7), 5930. https: //doi.org/10.3390/su15075930

[3]Salem, M., Valverde, S., Rovira, À., Salvi, J., Oliver, A., Lladó, X., & Kushibar, K. (2021). Transductive transfer learning for domain adaptation in brain magnetic resonance image segmentation.Frontiers in Neuroscience, 15, 608808. https: //doi.org/10.3389/fnins.2021.608808

[4]Hong, J., Yu, S. C.-H., & Chen, W. (2021). Unsupervised domain adaptation for cross-modality liver segmentation via joint adversarial learning and self-learning.arXiv preprint arXiv: 2109.05664.

[5]Liu, X., Qiu, H., Li, M., Yu, Z., Yang, Y., & Yan, Y. (2024). Application of multimodal fusion deep learning model in disease recognition.arXiv preprint arXiv: 2406.18546.

[6]Segura-Bedmar, I., Camino-Perdonas, D., & Guerrero-Aspizua, S. (2021). Exploring deep learning methods for recognizing rare diseases and their clinical manifestations from texts.arXiv preprint arXiv: 2109.00343.

[7]Zhao, Z., Yeoh, P. S. Q., Zuo, X., Chuah, J. H., Chow, C.-O., Wu, X., & Lai, K. W. (2024). Vision transformer-equipped convolutional neural networks for automated Alzheimer’s disease diagnosis using 3D MRI scans.Frontiers in Neurology, 15, 1490829. https: //doi.org/10.3389/fneur.2024.1490829

[8]Mutnuri, M. K., Stelfox, H. T., Forkert, N. D., & Lee, J. (2024). Using domain adaptation and inductive transfer learning to improve patient outcome prediction in the intensive care unit: Retrospective observational study.Journal of Medical Internet Research, 26, e52730. https: //doi.org/10.2196/52730

[9]Ghafoorian, M., Mehrtash, A., Kapur, T., Karssemeijer, N., Marchiori, E., Pesteie, M., … & Fedorov, A. (2021). Transfer learning for domain adaptation in brain lesion segmentation.Frontiers in Neuroscience,15, 608808. https: //doi.org/10.3389/fnins.2021.608808

[10]Segura-Bedmar, I., Camino-Perdonas, D., & Guerrero-Aspizua, S. (2021). Exploring deep learning methods for recognizing rare diseases and their clinical manifestations from texts.arXiv preprint arXiv: 2109.00343.