Key words
breast cancer - mammography screening - artificial intelligence - breast calcifications - positive predictive value - ductal carcinoma in situ
Introduction
Mammography screening is considered the only evidence-based method for early detection of breast cancer and has been established nationwide in Germany with scientifically proven effects [1]
[2]
[3].
Artificial intelligence (AI) uses different algorithms to solve various tasks and can simplify tasks or support human activity [4]. Evolution of computer-aided detection (CAD) systems resulting from technical advances and deep learning algorithms can increase the efficacy of mammographic screening. A meta-analysis of retrospective mammographic studies shows that the sole application of AI algorithms can reduce the radiologists’ reading workload by 17 %–91 % with a reduction in breast cancer detection of 0 %–7 % [5].
In the 50–69 years age group, calcifications constitute the second most frequent mammographic abnormality leading to further assessment and is also the second most frequent mammographic abnormality in the detection of breast cancer [6]
[7]. Calcifications constitute a broad spectrum of lesions, ranging from mastopathic breast lesions to high-risk lesions and precursor lesions of invasive breast cancer to invasive cancer with varying biological significance and differing positive predictive value in invasive work-up procedures (PPV3) [8]
[9].
From a user perspective, evaluating the diagnostic value of an individual AI algorithm towards calcifications is essential in order to safely integrate the abstract AI information on a defined mammographic lesion into the final human decision-making process. Retrospective integration of an available AI system [10] into the consensus conference decision-making process had the potential to mitigate false-positive recalls, but sensitivity was lower for calcifications than for masses [11].
The aim of the present retrospective study was to assess the diagnostic value of an individual AI algorithm in assessing the probability of malignancy of screen-detected, histologically-clarified calcifications with respect to human findings.
Materials and Methods
The study included 634 women with calcification-related invasive work-up after participation in a mammography-based screening program between July 2012 and June 2018. All mammographic examinations were carried oud in one screening unit. Histologically-clarified calcifications were retrospectively evaluated by an AI system at the lesion level. The evaluation of the system was compared to human evaluation and categorized based on histology, which was used as gold standard.
The study was carried out in the EU Project INTERREG V A, InMediValue 122207. Approval was obtained from the Ethics Committee of the Medical Association of Westphalia-Lippe and the Medical Faculty of the University of Münster, which had no reservations of ethical or legal nature regarding the performance of the research project.
Screening Process
As part of the German Mammography Screening Program, women between the ages of 50 and 69 are invited by letter for a two-view digital mammography screening examination. Screening mammograms are evaluated by two independent certified readers. In case of at least one abnormality, both readers discuss the case in a consensus conference together with the responsible physician, who finally decides whether a recall for further assessment is indicated and eventually performs subsequent diagnostics including invasive assessment [1].
Clinical Study Data
Grading of suspicion of malignancy in the consensus conference (4a, 4b, 5) was conducted in the screening software MaSc (KVWL, Dortmund, Germany). Categories were based on the Breast Imaging Reporting and Data System (BI-RADS) version 4 [8].
The two-view mammography was obtained at two locations (Sectra MDM L30, Linköping, Sweden; Philips MDM L50, Philips Healthcare, Eindhoven, Netherlands; Hologic 3Dimensions, Marlborough, MA, USA; Mammomat Inspiration, Mammomat Revelation, Siemens Healthcare, Erlangen, Germany). Independent double reading was performed by five readers, including two physicians supervising the program. Standardized assessment of calcification-associated lesions included sonography to exclude associated masses (Acuson S2000, Siemens Healthcare, Erlangen, Germany) and magnification views in cranio-caudal and lateral projection (Hologic Selenia Dimensions, Marlborough, MA, USA). For suspicious calcifications without any other associated findings, radiographic vacuum-assisted biopsy (Hologic Multicare Platinum, Marlborough, MA, US) was indicated as first-line method.
Data Collection
The CE- and FDA-certified AI-based software Transpara (version 1.7.0) from ScreenPoint Medical, Netherlands is a deep learning algorithm based on a deep convolutional neural network. The algorithm was trained using image data from over two million histologically-confirmed lesions and underwent external clinical validations [10]. The AI-software assigned a lesion score betweeen 1 and 100 to each suspicious region and indicated whether a calcification or mass was detected. All calcifications had been confirmed by the responsible physician based on the screening mammogram and clinical documentation. The morphology and distribution of calcifications were determined [8]. If not displayed automatically, the lesion-specific AI score was collected by user selection with a mouse click. A value of 100 represented the highest level of malignancy [12]. No score was displayed to the user (analyzed as score = 0) for lesions evaluated by the system with a score ≤ 28. For lesions rated 98–100, a score of 98 was displayed. In the case of varying scores of a lesion in the different mammographic views, the higher score was used.
Inclusion/Exclusion Criteria
Only screen-detected, calcification-associated lesions with biopsy-confirmed histological pathology were selected for the study. In case further invasive diagnostics had been recommended, all diagnostic procedures had to be completed before study inclusion. Cases with negative outcome and no indication of breast cancer were referred to biennial mammography follow-up. Exclusion criteria are shown in [Fig. 1].
Fig. 1 Presentation of the study cohort. * Lack of radiological-pathological correlation (n = 6), lack of implementation of advised diagnostic excisions (n = 11), lack of implementation of recommended follow-ups after biopsy (n = 24), summarized reasons such as invasive clarification of calcifications in association with masses or architectural distortions, biopsy indication did not correspond to the recall lesion and resulted from magnification images during assessment (n = 14). # Women without breast cancer or women with breast cancer not resulting from a calcification-related lesion. DCIS: ductal carcinoma in situ. High-risk lesions: In the case of atypical epithelial proliferations of the ductal type, the final histology was based on surgical histology such as atypical ductal hyperplasia. For lesions such as flat epithelial atypia, papilloma and radial scars, an individual indication was made regarding residues of lesions and atypia.
Screening-positive Calcifications
Screening-positive lesions included ductal carcinoma in situ (DCIS) and invasive breast cancer. The final surgical result was used for classification. For neoadjuvant therapy, the result of minimally invasive assessment was used. Breast cancer cases were differentiated by grade.
Screening-negative Calcifications
Histologically-benign lesions were considered screening-negative. In line with the screening evaluation, lesions of uncertain malignant potential (high-risk lesions) were also scored screening-negative. The post-surgical final histological result was used in case additional diagnostic excision was indicated. Indications for diagnostic excision were present in any case of atypical epithelial proliferation of the ductal type, as well as in residual lesion portions of flat epithelial atypia (FEA), papilloma and radial scars [13]. Screening-negative assessments were confirmed only if a two-year negative follow-up examination was present.
AI-negative and positive Calcifications
Lesions for which the AI system did not display a score (score = 0) were considered Al-negative. All lesions with a region-specific score (score ≥ 29) were considered AI-positive.
Statistical Evaluation
Analyses were performed using the statistics software R (version 4.0.2). Categorical parameters were presented as absolute and relative frequencies, continuous parameters were displayed as median and interquartile range. We determined the lesion-specific positive predictive value of performed invasive assessment (PPV3) towards calcifications. The performance of the AI system was evaluated based on the lesion-specific rate of false-negative evaluations, i. e., the proportion of AI-negative lesions among screening-positive calcifications (1 – sensitivity), and the rate of AI-true-negative assessments, i. e. the proportion of AI-negative lesions among screening-negative calcifications (specificity). A 95 % confidence interval was calculated for each performance indicator using non-parametric bootstrapping.
Results
Screening results
Histological results from 634 women with 644 calcifications-related histologic lesions were included ([Fig. 1]). Two women received invasive assessment of calcifications in different screening rounds.
Columnar cell metaplasia (n = 104), cystic adenotic changes (n = 64), fibroadenomas (n = 54), and scleradenoses (n = 26) occurred most frequently among the calcifications with benign outcome (390 of 644 lesions, 60.6 %). Screening-negative high-risk lesions (59 of 644 lesions, 9.2 %) included atypical ductal hyperplasia (n = 26), lobular neoplasms (n = 13), and papilloma (n = 12). Screening-positive breast cancers (189 of 634 women, 29.8 %) resulted from DCIS diagnoses (151 of 644 lesions, 23.4 %) and invasive breast cancers (44 of 644 lesions, 6.8 %).
Human reading revealed a lesion-specific PPV3 of 30.3 % (195/644), increasing from 21.2 % to 57.7 % to 100 % across categories 4a, 4b, and 5 ([Table 1]). Among calcification-indicated biopsies, category 4a dominated with 76.1 % (490/644). The proportion of DCIS of high nuclear grade and invasive cancers increased across reporting categories 4a, 4b, and 5 with 5.9 % (29/490), 22.1 % (33/149), 60 % (3/5), and 3.9 % (19/490), 16.1 % (24/149), 20 % (1/5), respectively.
Table 1
Lesion-specific positive predictive value of the invasive assessment of screen-detected calcifications.
|
Screen-detected calcifications[*]
|
Category 4a
n = 490 (100 %)
|
Category 4b
n = 149 (100 %)
|
Category 5
n = 5 (100 %)
|
Total
n = 644 (100 %)
|
|
No breast cancer
|
386 (78.8)
|
63 (42.3)
|
0 (0)
|
449 (69.7)
|
|
Benign lesions
|
335 (68.4)
|
55 (36.9)
|
0 (0)
|
390 (60.6)
|
|
Lesions with uncertain malignant potential[**]
|
51 (10.4)
|
8 (5.4)
|
0 (0)
|
59 (9.2)
|
|
Breast cancer (DCIS + invasive breast cancer)
|
104 (21.2)
|
86 (57.7)
|
5 (100)
|
195 (30.3)
|
|
DCIS G1
|
17 (3.5)
|
8 (5.4)
|
0 (0)
|
25 (3.9)
|
|
DCIS G2
|
39 (8.0)
|
21 (14.1)
|
1 (20.0)
|
61 (9.5)
|
|
DCIS G3
|
29 (5.9)
|
33 (22.1)
|
3 (60.0)
|
65 (10.1)
|
|
Invasive breast cancer
|
19 (3.9)
|
24 (16.1)
|
1 (20.0)
|
44 (6.8)
|
|
Lesion-specific PPV3 of human reading (%)
|
21.2 (104/490)
|
57.7 (86/149)
|
100.0 (5/5)
|
30.3 (195/644)
|
Unless otherwise stated, data represent absolute frequencies (percentages).
DCIS: ductal carcinoma in situ, G1: low nuclear grade, G2: intermediate nuclear grade, G3: high nuclear grade. PPV3: positive predictive value of performed invasive assessment.
* All calcification-related lesions were confirmed by vacuum-assisted biopsy. In case of a surgical intervention, the final histology was used for evaluation. For benign lesions, a biennial negative follow-up was present.
** In the case of atypical epithelial proliferation of the ductal type, final histology was based on post-surgical histology. For other lesion types, such as flat epithelial atypia, papillomas and radial scars, an individual decision was made regarding surgery depending on lesion residues and atypia.
AI Performance
Of 195 screen-detected, calcification-associated malignancies, 14 were not detected as lesions in lesion-specific AI assessment including manual selection (score 0).
AI-positive lesions (score > 0), had a lesion-specific PPV3 of 20.8 % (91/437) in category 4a, 57.8 % (85/147) in category 4b, and 100 % (5/5) in category 5. The lesion-specific PPV3 of AI across all categories was 30.7 % (181/589) ([Table 2]).
Table 2
Lesion-specific positive predictive value of biopsy-confirmed screen-detected calcifications based on a retrospective AI evaluation.
|
AI assessment of screen-detected calcifications
|
Category 4a
n = 490
|
Category 4b
n = 149
|
Category 5
n = 5
|
Total
n = 644
|
|
No breast cancer
|
386 (100)
|
63 (100)
|
0 (0)
|
449 (100)
|
|
Benign lesions with region score = 0 (true-negative)
|
40 (10.4)
|
1 (1.6)
|
0 (0)
|
41 (9.1)
|
|
Benign lesions with region score > 0 (false-positive)
|
346 (89.6)
|
62 (98.4)
|
0 (0)
|
408 (90.9)
|
|
Breast cancer (DCIS + invasive breast cancer)
|
104 (100)
|
86 (100)
|
5 (100)
|
195 (100)
|
|
Malignant lesions with region score > 0 (true-positive)
|
91 (87.5)
|
85 (98.8)
|
5 (100)
|
181 (92.8)
|
|
Malignant lesions with region score = 0 (false-negative)
|
13 (12.5)
|
1 (1.2)
|
0 (0)
|
14 (7.2)
|
|
Lesion-specific PPV3 AI (%)
|
20.8 (91/437)
|
57.8 (85/147)
|
100 (5/5)
|
30.7 (181/589)
|
Unless otherwise stated, data represent absolute frequencies (percentages).
AI: artificial intelligence; DCIS: ductal carcinoma in situ; PPV3: positive predictive value of performed invasive assessment.
The lesion-specific false-negative AI rate was 7.2 % (95 % CI: 4.3 %, 11.4 %), corresponding to a sensitivity of 92.8 %. The non-AI-detected breast cancer cases included 13 cases of DCIS (low grade n = 3, intermediate grade n = 6, high grade n = 4) and one case of invasive breast cancer. Thirteen out of 14 were category 4a cases; the morphology ([Table 2]) amorphous (amorphous n = 12 (85.7 %), granular n = 1 (7.1 %), linear n = 1 (7.1 %)) and the distribution calcification clusters (clustered n = 8 (57.1 %), segmental n = 3 (21.4 %), regional n = 2 (14.3 %), linear n = 1 (7.1 %)) were most frequent.
No score was shown (score = 0) for 41 of 449 calcification-associated screening-negative lesions. The rate of true-negative AI ratings was 9.1 % (95 % CI: 6.6 %, 11.9 %).
The AI-system assigned the lowest median score (61, interquartile range: 45–74) to benign lesions and the highest median score (81, 64–86) to invasive cancers. DCIS led to median scores of 74 (63–84), 70 (52–79), and 74 (66–83) with increasing nuclear grade. The distribution of the AI scores showed a strong overlap between the different histological lesions ([Fig. 2]).
Fig. 2 Region-based AI scores of biopsy-confirmed calcifications based on digital screening mammograms in relation to the final histology. Lesions with uncertain malignant potential: In the case atypical epithelial proliferation of the ductal type, the final histology was based on post-surgical histology. For other lesions such as flat epithelial atypia, papillomas and radial scars, an individual recommendation was made with regard to surgery depending on lesion residues and atypia. DCIS: ductal carcinoma in situ, G1: low nuclear grade, G2: intermediate nuclear grade, G3: high nuclear grade.
Only malignant lesions were recorded in the score group 96–100, which included 1.1 % (n = 7) of all 644 lesions. In the adjacent 91–95 and 86–90 score groups, the malignancy percentage decreased to 77.8 % (14 of 18) and 53.5 % (23 of 43), respectively. In the subsequent descending score groups, the malignancy percentage decreased steadily to 31.3 % (score group 65–70 (21 of 67)). In score groups 61–65 to 26–30, the malignancy percentage varied from 0 % to 25 %. There were 25.5 % (14 of 55) malignant lesions in the score 0 group. Calcification-associated invasive cancers were distributed with varying proportions across 13 of 16 score groups ([Fig. 3]).
Fig. 3 Relative frequencies of lesions within available score group of region-based AI evaluation of calcifications. Lesions with uncertain malignant potential: In the case of atypical epithelial proliferations of the ductal type, the final histology was based on post-surgical histology. For other lesions, such as flat epithelial atypia, papillomas and radial scars, an individual recommendation regarding surgical procedures was made depending on lesion residues and atypia. DCIS: ductal carcinoma in situ, G1: low nuclear grade, G2: intermediate nuclear grade, G3: high nuclear grade.
Discussion
Clinical studies have demonstrated that AI increases radiologistsʼ sensitivity and potentially specificity when evaluating mammograms [5]. Validation studies are needed in order to assess AI performance in different diagnostic processes [4]. The present study tested the diagnostic value of an AI application towards histologically assessed calcifications. In contrast to other validation studies, the assessment of the AI system was performed at the lesion level rather than at the mammogram level [5]
[10]
[12]. Therefore, the performance of the AI-system was evaluated based on preselected, specific regions. The present work on the positive predictive value of performed invasive assessment of calcifications (PPV3) complements AI validation regarding the positive predictive value of recall (PPV1) [11]. With lower malignancy rates in calcifications than in masses, there is a need for AI-applications that enable performance increase and allow for reduction of unnecessary invasive assessments of benign lesions [14].
In the present study, the PPV3 for calcifications based on human assessment was 30 % and was comparable to the AI system (31 %) when set to the lowest threshold with a false-negative rate of 7 % and a true-negative rate of 9 %. Detailed examination of the AI assessment showed increasing PPV3 values with increasing categories (4a: 21 %, 4b: 58 %, 5: 100 %), consistent with the human assessment and the current literature [8].
There are only a few validation studies evaluating AI regarding its positive prediction towards calcification based on histological assessment. Using another AI system, the probability of malignancy was retrospectively visually categorized by radiologists and lesion-specifically compared with the AI system at a 10 % threshold. There was no significant difference between the area under the receiver-operator-characteristic-curve (AUC) regarding malignancy scores and categorizations between readers and AI [15]. The results are in line with previous work, stating that neural networks can achieve over 98 % accuracy in categorizing suspicious calcifications [16].
Our study showed that highly suspicious calcifications were consistantly rated with a high degree of reliability by the AI system with a score > 0 (category 4b: false-negative 1/86, category 5: 0/5). In contrast, there was a higher absolute and relative number of AI false-negative calcification assessments (13/104) in cases of lower suspicion of malignancy (4a). Even though the detection of highly suspicious lesions is essential, also for medico-legal reasons, screen-detected calcifications associated with malignant lesions show a higher absolute frequency in category 4a (104 of 195 calcification-associated malignant histologies). Screening efficiency and ultimately patients would strongly benefit from an AI-system that yields an increase in the PPV3 and thus allows to reduce the number of invasive assessments of calcifications depicting benign lesions [17].
Score ranges between 96–100 and 91–95 indicated a high malignancy rate of 100 % and 77.8 %, respectively. In contrast, our validation indicated that a single score value ≤ 90 was less specific with varying malignancy percentages from 0 % to 54 %. A specific score threshold indicating the probability of malignancy could not be derived in the dominant proportion of all suspicious calcifications. The histological complexity of calcification-associated lesions might be the reason [7]
[17]. Clustered amorphous calcifications of category 4a are most common among invasive assessments of calcifications [15].
A differentiation of calcifications with regard to grading of DCIS or invasive breast cancers based on the AI score was not possible with the used version of the AI system. It would be beneficial if AI could reliably distinguish delete detect in particular DCIS of intermediate and high degree as well as invasive breast cancer [18]. Among the AI false-negative lesions DCIS were most prominent (92.9 %, 13/14), regardless of grade. Further, there was one invasive breast cancer among the AI false-negatives (7.1 %, 1/14). Prospective studies of AI use in mammography-based screening are needed in order to evaluate the potential of AI to optimize the rate of unnecessary assessments while increasing the rate of biologically relevant diagnoses with regard to breast cancer mortality [9]
[19].
The particular strength of the present work is the lesion-based AI assessment on a high number of cases. The diagnostic assessment as well as histological findings were subject to a high degree of standardization with follow-up. The study data were not part of the data set used to train the AI system.
A limitation is that the study design was not structured to test AI-related detection of additional, calcification-associated, malignant lesions in addition to the biopsied lesions, or to increase the sensitivity for calcification-associated malignancies, since there was no comparison with interval cancers available. The use of AI to reduce invasive assessments of benign calcifications requires further studies including interval cancers [20]. One retrospective study demonstrated that up to 50.9 % of interval cancers can be detected by an AI-system at the time of screening [21]. Yet, neither the amount of additional AI false-positive calcifications nor its prospective use with automated lesion display was tested. These results could only be related to our findings if initial diagnostic requirements, such as exclusion of associated masses, would be aligned.
In summary, the AI-system’s PPV3 for calcifications in total and per biopsy-indicating category was comparable to the reader-dependent evaluation. The implementation of AI did not lead to a reduction in screening-negative calcifications leading to invasive assessment without breast cancer detection. The category-based evaluation of the AI performance revealed false-negative evaluations in the group with less suspicious lesions, particularly in category 4a, yet with a reduction of false-positive biopsies. An AI-score based histological differentiation could not be derived based on the present results.
Compared to human assessment, the implementation of AI does not lead to increases in positive predictive values for invasive assessment of calcification-related lesions across BIRADS-based categories.
Especially at the lowest radiological suspicion level, a dedicated human evaluation seems reasonable due to a potentially higher risk of an AI false-negative evaluation than in the more suspicious categories.
