Positive Predictive Value for the Malignancy of Mammographic Abnormalities Based on the Presence of an Ultrasound Correlate

Key words

breast - AREAS - STRUCTURES & SYSTEMS - mammography - METHODS & TECHNIQUES - ultrasound - METHODS & TECHNIQUES

Introduction

Targeted breast ultrasound (US) is frequently used as an adjunct to diagnostic mammography because of its ability to characterize lesions [1]. Moreover, the use of US in addition to mammography increases the sensitivity and specificity of breast cancer diagnosis from 63% to 95 and 89% to 92%, respectively [2]. Moreover, it increases the cancer detection rate from 3% to 5% [3].

If abnormal findings are observed on screening mammography, the patients are recalled for diagnostic evaluation. Supplemental mammographic views are traditionally obtained with targeted breast US (if required). The goal of targeted US during the evaluation of a mammographic lesion is to achieve a more specific diagnosis of the cause of a mammographic abnormality, to prevent unnecessary biopsies, and to detect more carcinomas.

Despite a meticulous mammographic-sonographic correlation and skilled personnel, targeted US may not identify mammographic findings. Sampling must be performed by either stereotactic, tomosynthesis-guided biopsy or primary surgical incisional biopsy for lesions without a US correlate. Institutions that lack special equipment to perform stereotactic or tomosynthesis-guided biopsy may hesitate to perform a surgical intervention in the absence of a US correlate [4]. We aimed to compare the outcome of different mammographic lesions based on the presence of a US correlate and to assess how often targeted US can identify these lesions.

Patients and Methods

This retrospective study was approved by our Institutional Review Board and was in compliance with the Health Insurance Portability and Accountability Act. A waiver of consent was granted based on its retrospective design.

Patient selection

We included all attending consecutive cases over 7 years that had been assigned Breast Imaging Reporting and Database System (BI-RADS) categories 4 and 5 on diagnostic mammography because of abnormal screening mammography (BI-RADS 0) and underwent a US examination as part of the diagnostic workup. We excluded lesions that were detected only on screening US or magnetic resonance imaging (MRI).

We reviewed patient medical records for the type of mammographic abnormalities, as stated in the diagnostic report, the presence of a US correlate, biopsy guidance, and biopsy results. Mammographic abnormalities were classified according to the BI-RADS lexicon [5] as masses, asymmetries, architectural distortions, and calcifications. The lesion was classified according to the dominant component in the diagnostic study when it included two or more findings, such as a mass with associated calcifications ([Fig. 1]). Moreover, the term calcification was only used for pure calcifications without associated findings.

Fig. 1 A 60-year-old woman was recalled following screening for the evaluation of calcifications. a A tiny group of calcifications (arrows) observed in the left upper outer quadrant at 2 o’clock. b Magnification Lt. craniocaudal depicting an additional irregular mass with spiculated margins (white arrow), associated with segmental fine pleomorphic calcifications (black arrow). c Targeted ultrasound (US) depicting a US correlate for the mass (black arrows) and segmental calcifications (between asterisks). US-guided core needle biopsy displaying invasive ductal carcinoma.

We calculated and compared the positive predictive value (PPV) for malignancy between patients with and without a US correlate. In general, patients with a US correlate underwent US-guided core needle biopsy with post-biopsy marker deployment upon completion of the procedure. In contrast, those without a correlate underwent stereotactic biopsy. All US-guided core needle biopsy procedures were done using a 14-gauge automatic needle, while all stereotactic biopsy procedures were performed using 11-guage vacuum-assisted needles. Typically, 5 samples were enough for US-guided biopsy, whereas 6–12 samples were taken in stereotactic procedures. In cases including a subtle US correlate, stereotactic biopsy may have been chosen over US guidance at the discretion of the radiologist. Upon the completion of all US- and stereotactic-guided biopsy procedures, a clip tissue marker was placed to mark the biopsy site. A post-biopsy mammogram with two basic views (CC and MLO) was also obtained.

Imaging Technique and Interpretation

All patients underwent screening mammography (2D only or both 2D and 3D). The mammograms were interpreted using the BI-RADS lexicon by dedicated breast radiologists with one to 23 years of experience [5]. Patients with abnormal findings identified on screening mammography were assigned BI-RADS 0 and recalled for diagnostic mammography, which included additional mammographic views and/or targeted US (if required). US examinations were performed by four fellowship-trained breast imaging radiologists with 5 years to 20 years of experience in breast imaging and performing breast US. Linear multi-frequency transducers of US machines dedicated to breast imaging were used. During this period, we used Philips IU 22 and Siemens Acuson S2000

Patients who did not undergo breast US, those assigned final categories BI-RADS 1, 2, or 3 following the diagnostic workup, and patients with incomplete data including declined or failed biopsy and unavailable results were excluded from the study.

Statistical analyses

Categorical data are presented as frequencies and relative frequencies (i. e., percentages). We used the confidence interval method of Agresti and Couli (1998) to construct 95% confidence intervals for PPVs of the US-guided CNB and stereotactic CNB malignance classifications [6]. Furthermore, we conducted inter-imaging-modality comparisons of diagnostic relative frequencies (e. g., PPV) based on conventional chi-square frequency tests. A p≤0.05 decision rule was established a priori as the null hypothesis rejection rule for inter-imaging-modality diagnostic comparisons of relative frequencies. All statistical analyses were conducted using the Spotfire Splus version 8.2 statistical package (TIBCO Inc., Palo Alto, CA).

Results

Following abnormal mammography screening, 2,092 lesions underwent diagnostic workup during the study period. We excluded 1,259 lesions because of a lack of US examination in the diagnostic workup, lack of a biopsy, or incomplete data. 833 lesions in 811 patients met our inclusion criteria. The mean patient age was 59.1 years (range: 31 to 86 years; SD: 11.9 years). Mammographic lesions included masses (64.3%, n=536), asymmetries (4.6%, n=38), architectural distortions (19%, n=158), and calcifications (12.1%, n=101).

A US correlate was identified for 552 lesions (66.3%), which varied significantly according to the original mammography finding (p<0.001) ([Table 1]). The PPV of a US correlate was significantly greater for a mass (77.8%) than for architectural distortions (53.8%) (p<0.001) or calcifications (24.8%) (p<0.001). Similarly, the likelihood of finding a US correlate was significantly greater for an asymmetry (65.8%) or architectural distortion (53.8%) than for calcifications (24.8%) (p<0.001 for both) ([Table 1]).

Table 1 Presence of a US correlate based on mammographic findings.

	US correlate	No US correlate	Total	PPV [95% CI]
Masses	417	119	536	77.8% [74.0, 81.2%]
Asymmetries	25	13	38	65.8% [48.6, 80.4%]
Architectural distortions	85	73	158	53.8% [45.7, 61.8%]
Calcifications	25	76	101	24.8% [16.7, 34.3%]
Total	552	281	833

p<0.001; PPV: positive predictive value; US: ultrasound.

While lesions with a US correlate (66.3%, n=552) underwent US-guided CNB, those without a correlate (33.7%, n=281) underwent stereotactic biopsy.

Comparing the pathological outcomes of mammographic lesions based on biopsy guidance

The overall malignancy rate of lesions that underwent US-guided CNB was significantly higher than that of those that underwent stereotactic biopsy (40.2 vs. 18.9%, respectively) (p<0.001) ([Table 2]).

Table 2 Biopsy outcomes by imaging guidance method.

	Malignant	High-risk lesions	Benign	Total	PPV [95% CI]	P-value
US-guided CNB	222 (40.2%)	53 (9.6%)	277 (50.2%)	552	40.2% [36.1, 44.4%]	<0.001
Stereotactic CNB	53 (18.9%)	30 (10.7%)	198 (70.5%)	281	18.9% [14.5, 23.9%]
Total	275 (33.0)	83 (10.0)	475 (57.0)	833

CNB: core needle biopsy; PPV: positive predictive value; and US: ultrasound.

Moreover, we estimated the pathological outcomes for each mammographic finding. Masses that underwent US-guided CNB demonstrated a significantly higher PPV for malignancy than those that underwent stereotactic CNB (39.3 vs. 19.3%, respectively) (p<0.001) ([Table 3]).

Table 3 Pathological outcomes of different mammographic findings by biopsy guidance method.

	Malignant	High-risk lesion	Benign	Total	PPV [95% CI]	P-value
	Masses
US-guided CNB	164 (39.3%)	33 (7.9%)	220 (52.8%)	417	39.3% [34.6, 44.2%]	<0.001
Stereotactic CNB	23 (19.3%)	8 (6.7%)	88 (73.9%)	119	19.3% [12.7, 27.6%]
Total	187 (34.9%)	41 (7.6%)	308 (57.5%)	536
	Architectural Distortions
US-guided CNB	35 (41.2%)	13 (15.3%)	37 (43.5%)	85	41.2% [30.6, 52.4%]	<0.001
Stereotactic CNB	9 (12.3%)	9 (12.3%)	55 (75.3%)	73	12.3% [5.8, 22.1%]
Total	44 (27.8%)	22 (13.9%)	92 (58.2%)	158
	Asymmetries
US-guided CNB	11 (44.0%)	3 (12.0%)	11 (44.0%)	25	44.0% [24.4, 65.1%]	0.019
Stereotactic CNB	2 (15.4%)	2 (15.4%)	9 (69.2%)	13	15.4% [1.9, 45.4%]
Total	13 (34.2%)	5 (13.1%)	20 (52.6%)	38
	Calcifications
US-guided CNB	12 (48.0%)	4 (16.0%)	9 (36.0%)	25	48.0% [27.8, 68.7%]	0.034
Stereotactic CNB	19 (25.0%)	11 (14.5%)	46 (60.5%)	76	25.0% [15.8, 36.3%]
Total	31 (30.7%)	15 (14.9%)	55 (54.4%)	101

CNB: core needle biopsy; PPV: positive predictive value; and US: ultrasound.

Similarly, architectural distortions that underwent US-guided CNB were more likely to have higher malignancy rates than lesions that underwent stereotactic biopsy (PPV, 41.2 vs. 112.3%, respectively) (p<0.001) ([Table 3]).

Asymmetries that underwent US-guided CNB revealed a significantly higher PPV for malignancy than those that underwent stereotactic CNB (44.0 vs. 15.4%, respectively) (p=0.019) ([Table 3]).

Likewise, calcifications that underwent US-guided CNB had a significantly greater likelihood of malignancy than those that underwent stereotactic biopsy (PPV, 48 vs. 25%, respectively) (p=0.034) ([Table 3]).

There were no significant differences in the incidence of high-risk lesions between lesions that underwent US- or stereotactic-guided CNB (p=0.713).

Benign discordant lesions identified following targeted US-guided CNB

We identified benign discordant results in 54 cases that underwent targeted US-guided CNB (9.8%, n=54 of 552). 19 (35.2%, 19 of 54) cases were upgraded to malignancy based on surgical excision or repeated biopsy under stereotactic guidance ([Table 4]). Invasive ductal carcinoma (IDC), tubular carcinoma, invasive lobular breast cancer, and ductal carcinoma in situ were observed in 12, 1, and 3 cases, respectively.

Table 4 Association between upgrading to malignant and original mammographic findings.

Mammographic finding	Upgraded	Not upgraded	Total
Masses	7	11	18
Architectural distortions	5	12	17
Asymmetries	5	12	17
Calcifications	2	0	2
Total	19	35	54

p=0.232.

On reviewing the clip site in the aforementioned cases, the clip was found to be misplaced (off) in 20 cases (37%, 20 of 54). In other words, the presumed US correlate was wrong, and the exact lesion was not observed on mammography ([Fig. 2]). In contrast, the clip was found to be in a good place in 34 cases (63%).

Fig. 2 a Screening mammography; left mediolateral oblique (MLO) view of a 54-year-old woman depicting a developing asymmetry that was not detected on her prior mammogram b. The finding cannot be clearly identified on the craniocaudal view (not shown). Diagnostic workup to localize the lesion reveals its location in the upper outer quadrant at 2 o’clock. b A prior screening mammography; left mediolateral oblique (MLO) view of a 54-year-old woman that did not detect the developing asymmetry. Diagnostic workup to localize the lesion reveals its location in the upper outer quadrant at 2 o’clock. c Targeted US was performed as part of the diagnostic workup and displayed a potential correlate; a suspicious mass in the upper outer quadrant at 2 o’clock, 8 cm from the nipple. US-guided core needle biopsy was performed, and a post-biopsy clip marker has been deployed. d Post-biopsy ML view shows that the clip marker is not present in the mammographic lesion and is located high up in the axilla (red arrow), thus indicating that the mammographic lesion has not been correctly sampled. The circled clip was obtained from a prior biopsy. Pathology indicates a discordant normal breast parenchyma. Repeated biopsy under stereotactic guidance was performed, and the final pathology was invasive lobular carcinoma.

27 (50%) patients underwent surgical excision for benign discordant biopsy. While 16 (29.5%) patients underwent repeated biopsy under stereotactic guidance, 3 (5.5%) underwent MRI to verify if the original mammographic lesions were benign. Moreover, 8 (15%) patients refused surgery and chose follow-up. We performed follow-up by mammography for at least 2 years (2–5 years).

Misplacement of a post-biopsy clip is more likely to result in an upgrade in the subsequent surgical excision (65%, 13 of 20) than in the case of discordant lesions with the clip in place (21.4%, 6 of 28) (p=0.001)

Discussion

Our findings demonstrate that, despite being common for screening-detected masses, asymmetries, and architectural distortions, the presence of a US correlate is less common for calcifications. Moreover, the PPV for biopsy is much higher for all lesion types when a US correlate is detected. The lack of a US correlate does not indicate consideration of follow-up over biopsy due to a>2% rate of malignancy.

Bahl et al. [7] investigated factors that influence the outcome of architectural distortions. Architectural distortions detected by screening mammography are less likely to represent malignancy than those detected by diagnostic mammography (67.0 vs. 83.1%, respectively, p<0.001). Targeted US could identify 304 of 347 (87.6%) architectural distortions. Moreover, distortions without a sonographic correlate were less likely to represent a malignant lesion than those with a correlate (27.9 vs. 82.9%, respectively, p<0.001).

Targeted US could identify a correlate in 53.8% of architectural distortions. Similarly, the malignancy rate was significantly lower for distortions without a US correlate compared to those with a correlate (41.2 vs. 12.3%, respectively, p<0.001). However, the rate was not low enough to forgo biopsy ([Fig. 3]).

Fig. 3 a Screening mammography with tomosynthesis in a 52-year-old woman depicting an area of architecture distortion at 8 o’clock in the right breast (arrows). b Spot compression views with tomosynthesis clearly show the architectural distortion (arrows). Targeted US (not shown) was performed as part of the diagnostic workup. However, the results were negative. Stereotactic-guided core needle biopsy depicts an invasive ductal carcinoma.

Chesebro et al. [8] evaluated the outcomes of developing asymmetries and found that US could characterize 30 out of 201 lesions with an accuracy of 15%. Moreover, they established an association between developing asymmetries with US correlates and an increased risk of malignancy, with 57% malignancy versus 37% for asymmetries without a correlate.

Shetty et al. [9] investigated the role of US in the evaluation of focal asymmetries among 36 women. They observed a solid mass, complicated cyst, echogenic tissue, and no US correlate in 41.7% (15 of 36), 5.6% (2 of 36), 25.0% (9 of 36), and 27.8% (10 of 36) of cases, respectively. Excisional biopsy of focal asymmetries showed IDC in seven patients (19.4%, 7 of 36). Two of these patients (28.6%) revealed no abnormality during US, thereby supporting the idea that negative US does not exclude malignancy and should not prevent biopsy.

A series by Soo et al. [10] mentioned that 23% of the calcifications observed on mammography were detected by US. In addition, US-detected calcifications were three times more likely to be malignant and invasive than those detected by mammography alone.

Bae et al. [11] reviewed 336 patients with suspicious microcalcifications who underwent biopsy under image guidance. Only 17.5% of the calcifications could be identified on targeted US. In contrast, 74% were mammography only findings. The remaining 8.5% of cases demonstrated an association between a mass and calcifications during US. In addition, the lesions visible on US were more likely to represent malignancy (66.2% vs. 23.2%, respectively; p<0.001) and depicted higher BI-RADS categories than those not detected by US (61.0 vs. 22.2%, respectively; p<0.001).

Our findings correspond with the findings of previous studies that mammographic lesions with no US correlates were associated with significantly lower malignancy rates than those with correlates. However, the rates were high enough to recommend biopsy.

The rate of upgrades to malignancy among patients who underwent surgical excision for benign discordant results demonstrated a significant association with post-biopsy clip position (p=0.001). Misplaced clips following biopsy were associated with an upgrading rate of 65% (13 cases of 20) at the time of surgical excision. Post-biopsy, misplaced clip markers should alert radiologists to the fact that the finding sampled under targeted US guidance is not the same as the lesion primarily detected by mammography.

However, surgeons should still excise discordant lesions with a good clip position. This can be attributed to the upgrading rate of 21.4%, which is not negligible. Good position of the clip should not prevent the surgical excision of a benign discordant biopsy result. Nonetheless, a misplaced clip should definitely prompt action, either re-biopsy under stereotactic guidance or surgical excision.

Misplaced clips should be documented and followed with placement of another clip. Cases of migrated clips are challenging if the lesion is no longer palpable or detectable by imaging. It is nearly impossible to perform breast-conserving surgery with no intraoperative guidance. Some surgeons perform blind wide lumpectomy with the aid of intraoperative fluoroscopy. Tomosynthesis does not usually improve the accuracy of marker placement [12].

Previous studies mentioned that all benign discordant results should be surgically excised, owing to the substantial cancer detection rate in subsequent surgical excision 15–50% [13] [14] [15] [16] [17] [18] [19] [20] [21] [22].

Previous studies showed an acceptable interobserver agreement regarding BIRADS lexicon since its introduction in 1993 for mammography and its redesign for mammography in 2003 [23] [24] [25] [26] [27]. The current study included a relatively wide range of radiologist experience in breast imaging reflecting the real practice of radiology in most institutions. The effect of inter- and intraobserver variability may have influenced the results. Palazuelos et al. [28] criticized the Choi et al. study [24] for the evaluation of interobserver agreement emphasizing the importance of using a well-designed and dedicated study design to evaluate this point.

However in the current study, the mammography and US results were homogeneously distributed among the radiologists for various BIRADS categories with no predilection for any category to be assigned to a specific level of experience. Correlating the efficacy of clip placement with the level of radiologist experience would have also enriched the study and influenced its results. This issue requires a different study design.

Our study had several limitations. The study had a retrospective design and was performed at a single institution. Moreover, targeted US was not performed in all cases. We only included lesions that underwent US examination. Therefore, the incidence of a US correlate did not predict the overall finding on diagnostic imaging. Moreover, we only included suspicious and malignant lesions, so that the data did not predict the incidence of overall malignancy in screening. In addition, we could not accurately evaluate lesions with subtle correlates on US as they underwent stereotactic biopsy and were considered with no US correlates. Evaluation of inter-reader variability and the effect of the “number of years of experience of the performing radiologist” on the accuracy of targeted US and on targeted US-guided biopsy would have enriched the study. This, however, could not be achieved, because in our institution the radiologist who performed the diagnostic workup is not necessarily the same one who performed the procedure.

The strengths of this study include that all targeted US examinations were performed by dedicated breast imaging radiologists. Moreover, this is the first study to compare the PPV of all mammographic findings based on the presence of a US correlate.

The study findings are clinically relevant as targeted US could identify a correlate in only 66% of cases. The incidence of malignancy is significantly lower for lesions with no US correlate, although it is still high enough to recommend biopsy. The PPV was not low enough to prevent biopsy.

• References

• 1 Stavros AT, Thickman D, Rapp CL, Dennis MA, Parker SH, Sisney GA. Solid breast nodules: use of sonography to distinguish between benign and malignant lesions. Radiology 1995; 196: 123-134
• 2 Leconte I, Feger C, Galant C, Berliere M, Berg BV, D’Hoore W. et al. Mammography and subsequent whole-breast sonography of nonpalpable breast cancers: the importance of radiologic breast density. AJR Am J Roentgenol 2003; 180: 1675-1679
• 3 Corsetti V, Ferrari A, Ghirardi M, Bergonzini R, Bellarosa S, Angelini O. et al. Role of ultrasonography in detecting mammographically occult breast carcinoma in women with dense breasts. Radiol Med 2006; 111: 440-448
• 4 Durand MA, Wang S, Hooley RJ, Raghu M, Philpotts LE. Tomosynthesis-detected Architectural Distortion: Management Algorithm with Radiologic-Pathologic Correlation. Radiographics. 2016; 36: 311-321
• 5 D’Orsi CSE, Mendelson EB, Morris EA. Breast Imaging Reporting and Data System: ACR BI-RADS breast imaging atlas. 5th ed. Reston, Va: American College of Radiology,; 2013
• 6 Agresti A, Coull BA. Approximate is better than “exact” for interval estimation of binomial proportions. American Statistician 1998; 52: 119-126
• 7 Bahl M, Baker JA, Kinsey EN, Ghate SV. Architectural Distortion on Mammography: Correlation With Pathologic Outcomes and Predictors of Malignancy. AJR Am J Roentgenol 2015; 205: 1339-1345
• 8 Chesebro AL, Winkler NS, Birdwell RL, Giess CS. Developing Asymmetry at Mammography: Correlation with US and MR Imaging and Histopathologic Findings. Radiology 2016; 279: 385-394
• 9 Shetty MK, Watson AB. Sonographic evaluation of focal asymmetric density of the breast. Ultrasound Q 2002; 18: 115-121
• 10 Soo MS, Baker JA, Rosen EL. Sonographic detection and sonographically guided biopsy of breast microcalcifications. AJR Am J Roentgenol 2003; 180: 941-948
• 11 Bae S, Yoon JH, Moon HJ, Kim MJ, Kim EK. Breast Microcalcifications: Diagnostic Outcomes According to Image-Guided Biopsy Method. Korean J Radiol 2015; 16: 996-1005
• 12 Madeley C, Kessell M, Madeley C, Taylor D. A comparison of stereotactic and tomosynthesisguided localisation of impalpable breast lesions. J Med Radiat Sci 2019; 66: 170-176
• 13 Poole BB, Wecsler JS, Sheth P, Sener SF, Wang L, Larsen L. et al. Malignancy rates after surgical excision of discordant breast biopsies. J Surg Res 2015; 195: 152-157
• 14 Liberman L, Drotman M, Morris EA, LaTrenta LR, Abramson AF, Zakowski MF. et al. Imaging-histologic discordance at percutaneous breast biopsy. Cancer 2000; 89: 2538-2546
• 15 Son EJ, Kim EK, Youk JH, Kim MJ, Kwak JY, Choi SH. Imaging-histologic discordance after sonographically guided percutaneous breast biopsy: a prospective observational study. Ultrasound Med Biol 2011; 37: 1771-1778
• 16 Shah VI, Raju U, Chitale D, Deshpande V, Gregory N, Strand V. False-negative core needle biopsies of the breast: an analysis of clinical, radiologic, and pathologic findings in 27 concecutive cases of missed breast cancer. Cancer 2003; 97: 1824-1831
• 17 Soyder A, Taskin F, Ozbas S. Imaging-histological discordance after sonographically guided percutaneous breast core biopsy. Breast Care 2015; 10: 33-37
• 18 Schueller G, Jaromi S, Ponhold L, Fuchsjaeger M, Memarsadeghi M, Rudas M. et al. US-guided 14-gauge core-needle breast biopsy: results of a validation study in 1352 cases. Radiology 2008; 248: 406-413
• 19 Mainiero M, Koellike S, Lazarus E, Schepps B, Lee C. Ultrasound-guided Large-Core Needle Biopsy of the Breast: Frequency and Results of Repeat Biopsy. Journal of Women’s Imaging. 2002; 4: 52-57
• 20 Berg W, Berg A, Loffe O. Initial success and frequency of rebiopsy after ultrasound-guided 14-gauge core breast biopsy. American Journal of Roentgenology 2003; 180: 10
• 21 Schoonjans JM, Brem RF. Fourteen-gauge ultrasonographically guided large-core needle biopsy of breast masses. J Ultrasound Med 2001; 20: 967-972
• 22 Crystal P, Koretz M, Shcharynsky S, Makarov V, Strano S. Accuracy of sonographically guided 14-gauge core-needle biopsy: results of 715 consecutive breast biopsies with at least two-year follow-up of benign lesions. J Clin Ultrasound 2005; 33: 47-52
• 23 Abdullah N, Mesurolle B, El-Khoury M, Kao E. Breast imaging reporting and data system lexicon for US: interobserver agreement for assessment of breast masses. Radiology 2009; 252: 665-672
• 24 Choi EJ, Lee EH, Kim YM. et al. Interobserver agreement in breast ultrasound categorization in the Mammography and Ultrasonography Study for Breast Cancer Screening Effectiveness (MUST-BE) trial: results of a preliminary study. Ultrasonography 2019; 38: 172-180
• 25 Elverici E, Zengin B, Nurdan Barca A, Didem Yilmaz P, Alimli A, Araz L. Interobserver and intraobserver agreement of sonographicBIRADS lexicon in the assessment of breast masses. Iran J Radiol 2013; 10: 122-127
• 26 Calas MJ, Almeida RM, Gutifilen B, Pereira WC. Interobserver concordance in the BI-RADS classification of breast ultrasound exams. Clinics (Sao Paulo) 2012; 67: 185-189
• 27 Lee HJ, Kim EK, Kim MJ, Youk JH, Lee JY, Kang DR. et al. Observer variability of Breast Imaging Reporting and Data System (BI-RADS) for breast ultrasound. Eur J Radiol 2008; 65: 293-298
• 28 Palazuelos G, Valencia SA, Romero JA. More than interobserver agreement is required for comparisons of categorization systems. Ultrasonography 2019; 38: 374-376