Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
Multicenter Study
. 2022 Jun;32(6):4101-4115.
doi: 10.1007/s00330-021-08519-z. Epub 2022 Feb 17.

The importance of multi-modal imaging and clinical information for humans and AI-based algorithms to classify breast masses (INSPiRED 003): an international, multicenter analysis

André Pfob  1   2 , Chris Sidey-Gibbons  2   3 , Richard G Barr  4 , Volker Duda  5 , Zaher Alwafai  6 , Corinne Balleyguier  7 , Dirk-André Clevert  8 , Sarah Fastner  1 , Christina Gomez  1 , Manuela Goncalo  9 , Ines Gruber  10 , Markus Hahn  10 , André Hennigs  1 , Panagiotis Kapetas  11 , Sheng-Chieh Lu  2   3 , Juliane Nees  1 , Ralf Ohlinger  6 , Fabian Riedel  1 , Matthieu Rutten  12   13 , Benedikt Schaefgen  1 , Maximilian Schuessler  14 , Anne Stieber  1 , Riku Togawa  1 , Mitsuhiro Tozaki  15 , Sebastian Wojcinski  16 , Cai Xu  2   3 , Geraldine Rauch  17 , Joerg Heil  1 , Michael Golatta  18
Affiliations
Multicenter Study

The importance of multi-modal imaging and clinical information for humans and AI-based algorithms to classify breast masses (INSPiRED 003): an international, multicenter analysis

André Pfob et al. Eur Radiol. 2022 Jun.

Abstract

Objectives: AI-based algorithms for medical image analysis showed comparable performance to human image readers. However, in practice, diagnoses are made using multiple imaging modalities alongside other data sources. We determined the importance of this multi-modal information and compared the diagnostic performance of routine breast cancer diagnosis to breast ultrasound interpretations by humans or AI-based algorithms.

Methods: Patients were recruited as part of a multicenter trial (NCT02638935). The trial enrolled 1288 women undergoing routine breast cancer diagnosis (multi-modal imaging, demographic, and clinical information). Three physicians specialized in ultrasound diagnosis performed a second read of all ultrasound images. We used data from 11 of 12 study sites to develop two machine learning (ML) algorithms using unimodal information (ultrasound features generated by the ultrasound experts) to classify breast masses which were validated on the remaining study site. The same ML algorithms were subsequently developed and validated on multi-modal information (clinical and demographic information plus ultrasound features). We assessed performance using area under the curve (AUC).

Results: Of 1288 breast masses, 368 (28.6%) were histopathologically malignant. In the external validation set (n = 373), the performance of the two unimodal ultrasound ML algorithms (AUC 0.83 and 0.82) was commensurate with performance of the human ultrasound experts (AUC 0.82 to 0.84; p for all comparisons > 0.05). The multi-modal ultrasound ML algorithms performed significantly better (AUC 0.90 and 0.89) but were statistically inferior to routine breast cancer diagnosis (AUC 0.95, p for all comparisons ≤ 0.05).

Conclusions: The performance of humans and AI-based algorithms improves with multi-modal information.

Key points: • The performance of humans and AI-based algorithms improves with multi-modal information. • Multimodal AI-based algorithms do not necessarily outperform expert humans. • Unimodal AI-based algorithms do not represent optimal performance to classify breast masses.

Keywords: Artificial intelligence; Breast cancer; Machine learning; Ultrasonography.

PubMed Disclaimer

Conflict of interest statement

The authors of this manuscript declare no relationships with any companies whose products or services may be related to the subject matter of the article.

Figures

Fig. 1
Fig. 1
Performance comparison between the clinical routine, the ultrasound experts, the unimodal machine learning algorithms, and the multi-modal machine learning algorithms
Fig. 2
Fig. 2
Receiver operating characteristic curves of the clinical routine, the ultrasound experts, the unimodal machine learning algorithms, and the multi-modal machine learning algorithms
Fig. 3
Fig. 3
Ultrasound Images. a This patient’s ultrasound images were evaluated to show a benign breast mass by the three ultrasound experts but to show a malignant breast mass by full clinical breast evaluation. This patient was 41 years old with a positive family history for breast cancer and a clinically suspicious palpable tumor. Histopathology showed a luminal B, NST, G3 carcinoma. b This patient’s ultrasound images were evaluated to show a benign breast mass by the three physician experts and by full clinical breast evaluation. This patient was 25 years old without any clinically suspicious signs. Histopathology showed a fibroadenoma
Fig. 4
Fig. 4
Shapley Additive Explanations (SHAP) Value Summary Plot of the Extreme Gradient Boosting (XGBoost) Tree Model. a XGboost – unimodal Algorithm. SHAP values on the left side of the x-axis indicate that the variable was important for predicting malignancy; values on the right side indicate that the variable was important for predicting a benign breast mass. Purple indicates a high variable value (e.g. margin – non-circumscribed: yes); yellow indicates a low variable value (e.g age: lower patient age). The values on the y-axis represent the overall global variable importance. b XGboost – multi-modal Algorithm
See this image and copyright information in PMC

Similar articles

See all similar articles

Cited by

See all "Cited by" articles

References

    1. Hosny A, Parmar C, Quackenbush J, Schwartz LH, Aerts HJWL. Artificial intelligence in radiology. Nat Rev Cancer. 2018;18(8):500–510. doi: 10.1038/s41568-018-0016-5. - DOI - PMC - PubMed
    1. McDonald RJ, Schwartz KM, Eckel LJ, et al. The effects of changes in utilization and technological advancements ofcross-sectional imaging onradiologist workload. Acad Radiol. 2015;22(9):1191–1198. doi: 10.1016/j.acra.2015.05.007. - DOI - PubMed
    1. Liu X, Faes L, Kale AU, et al. A comparison of deep learning performance against health-care professionals in detecting diseases from medical imaging: a systematic review and meta-analysis. Lancet Digit Heal. 2019;1(6):e271–e297. doi: 10.1016/S2589-7500(19)30123-2. - DOI - PubMed
    1. American College of Radiology. Subject: (Docket No. FDA-2019-N-5592) “Public Workshop - Evolving Role of Artificial Intelligence in Radiological Imaging;” Comments of the American College of Radiology. https://www.acr.org/-/media/ACR/NOINDEX/Advocacy/acr_rsna_comments_fda-a.... Published 2020. Accessed 3 Apr 2021
    1. National Comprehensive Cancer Network . Breast cancer screening and diagnosis. Harborside Press; 2020.

Publication types