Key words thorax - CT - metastases - neoplasms - observer performance
Introduction
Pulmonary metastasectomy is considered a useful treatment option in particular situations during the course of some metastatic malignant tumors of extrapulmonary origin. Surgical resection of lung metastases is aimed to completely remove all metastases. Long-term follow-up data after pulmonary metastasectomy demonstrated that overall survival was improved with complete resection, compared to incomplete resection [1 ]
[2 ]
[3 ]. These data emphasize the necessity to detect all pulmonary metastases during surgery. Manual palpation of the deflated lung during open thoracotomy is regarded as the most sensitive method. However, even a thorough examination cannot preclude occasionally missing of a lung nodule, especially in central lung areas or in coexisting chronic obstructive lung disease.
Prior to surgery, knowledge of the expected number and location of lung metastases may therefore be helpful to assure their complete resection. Computed tomography (CT) is widely used for this purpose. Unfortunately, CT findings were found to be in imperfect agreement with surgical findings in published studies [4 ]
[5 ]. More metastases were surgically detected than previously found on CT images. Lung nodules as small as 2 mm are clearly visible on thin-slice CT scans. The underlying problem is probably caused by the non-detection of small nodules on hundreds of CT images by the radiologist rather than by the non-depiction of nodules by the CT scanner.
The moderate performance of radiologists with respect to the detection of small lung nodules on multi-detector row CT images is well known [6 ]
[7 ]
[8 ]
[9 ]. To overcome these deficits, computer-aided detection (CAD) systems have been developed to automatically detect lung nodules on CT scans. Used as a second or concurrent reader to the radiologist, these systems are able to increase the radiologist’s sensitivity [6 ]
[8 ]
[10 ]
[11 ].
Video-assisted thoracoscopy (VATS) is a less invasive surgical approach for the resection of small pulmonary lesions compared to open thoracotomy [12 ]. However, manual palpation is impossible during VATS, and therefore its suitability for the resection of a solitary pulmonary metastasis is still under discussion due to the current lack of other methods for the reliable exclusion of multiple lung metastases [13 ]
[14 ].
This prospective study was designed to separately quantify the contribution of the radiologist and of CAD used as a second reader to the preoperative detection of lung metastases. The results are compared to the surgical and histopathological findings as the standard of reference.
Material and methods
The responsible IRB approved the study design (reference number EA1/310/10). Written informed consent was obtained from all patients.
This prospective single-center trial was designed to separately evaluate the radiologist’s and CAD performance with respect to the detection of pulmonary metastases in comparison to the surgical and histopathological findings as the standard of reference. The contribution of CAD as a second reader was assessed. For radiologically detected lesions missed during surgery, an attempt was made to obtain follow-up data to determine whether these lesions were metastatic or benign.
Patient population
Inclusion criteria are shown in [Table 1 ]. Patients were consecutively enrolled from January 2011 to July 2014. Especially for lesions missed during surgery, follow-up data was collected until June 2016.
Table 1
Surgical and radiological inclusion criteria.Tab. 1 Chirurgische und radiologische Einschlusskriterien.
surgical inclusion criteria
radiological inclusion criteria
complete resection of malignant primary tumor
exclusion of tumor relapse
exclusion of synchronous metastases in other regions
limited number of lung metastases
technical and functional operability
current chest CT scan not older than 8 weeks at the time of surgery
thin-slice CT with slice thickness ≤ 2 mm and reconstruction increment ≤ 1 mm
CAD analysis performed by an experienced radiologist
During the study period, 95 patients (49 male, 46 female) met the inclusion criteria and were enrolled in the study. A total of 8 surgical procedures were excluded from evaluation because of deviations of the expected histopathology: 4 of these were proven to be primary lung cancer, 2 neuroendocrine tumors, 1 primary lung cancer and metastases simultaneously and 1 tuberculoma.
Patient age ranged from 17 to 83 years (mean: 62.8 ± 12.9 years, median: 64.6 years). A total of 115 surgical procedures were performed in these patients: all pulmonary nodules which were either palpable by the surgeon or detected on CT and marked on the CT image by the radiologist were resected via unilateral open thoracotomy. Primary tumors were colorectal cancer (34 cases, 29.6 %), renal cell cancer (33, 28.7 %), sarcoma (24, 20.9 %), head and neck cancer (6, 5.2 %), prostate cancer (3, 2.6 %), malignant melanoma (3, 2.6 %), malignant neurogenic tumor (2, 1.7 %), bladder cancer (2, 1.7 %) and one case of uterine cancer, cervical cancer, breast cancer, tracheal cancer, lung cancer, carcinoid, fibrohistiocytic skin tumor and pancreatic cancer, respectively.
Scan parameter and CAD performance
A CT scan with a slice thickness of ≤ 2 mm and a reconstruction increment of ≤ 1 mm not older than 8 weeks was required. Standard-dose CT scans from various institutions were accepted if the slice thickness requirements were fulfilled. Time between CT scan and surgery ranged from 0 to 56 days (mean: 17.9 ± 16.6 days, median: 12 days).
CT images were transferred to the CAD system (LMS 6.0, Median Technologies, Valbonne, France), the CAD analysis was sent to the PACS system.
CT scans were reported by one of four board-certified radiologists with specific experience in reporting chest CT scans (6 to 20 years) using CAD as a second reader:
The radiologist read the CT using the LMS user interface which provides original images and sliding thin-slab maximum intensity projection (MIP) and documented all detected nodules without knowledge of the CAD results. Lesions were labeled (RE01, RE02 and so on for the right lung, LI01, LI02 and so on for the left lung), and a screenshot was obtained for each lesion.
After completion of unassisted reading, the second reader CAD findings were given. The results were reviewed by the radiologist. True-positive findings were confirmed and labeled (RE51, RE52 and so on or LI51, LI52 and so on), false-positive CAD markers were rejected and the total number per CT scan was recorded in a comment field of the CAD analysis.
Finally, the radiologist filled out a study documentation sheet containing information on lesion label, lung segment, lesion size, and a lung scheme in three views on which the radiologist marked the approximate position of all lesions ([Fig. 1 ]).
Fig. 1 Study documentation sheet with notes by radiologist and surgeon.Abb. 1 Studiendokumentationsbogen mit Eintragungen durch Radiologen und Chirurgen.
The CAD results were sent to the PACS and together with the CT images and the study documentation sheet, were available to the surgeon in the operating room.
The reporting radiologist was unaware of the surgical and histopathological findings that were obtained after completion of the CAD analysis. During the evaluation phase of the study, the CAD system was set to recalculate the CAD analysis on the same CT dataset, and an experienced chest radiologist (DW, 20 years professional experience) reviewed the CAD results to decide if CAD also detected the lesions that had been marked by the radiologist during the unassisted first part of the reading process. The CAD algorithm remained unchanged for the duration of the study. Moreover, in the same reading session the radiologist attempted to identify lesions additionally resected by the surgeon without having been detected preoperatively on the CT scans. If a lesion could retrospectively be identified on the CT images, the maximum lesion diameter was measured on the CT scan and was recorded. If a lesion could not be identified even retrospectively, the lesion size was recorded as missing.
The information as to whether each nodule was attached to the pleura was gathered at the time of the compilation of the study. The surgical free-text comments on the study documentation sheets were assessed, and if necessary, the CT images and the operation reports were reviewed.
Surgical and histopathological procedure
Pulmonary metastasectomy was conducted by one of four board-certified staff thoracic surgeons who regularly perform at least 300 thoracic surgical procedures per year.
Open thoracotomy was performed to explore the deflated lung manually. Consequently, the lung was palpated in a standardized fashion segment by segment, starting with the upper lobe down to the lower lobe segments before laser resection started with the first nodule in the craniocaudal direction.
During metastasectomy, the surgeon matched the preoperatively detected nodules to the intraoperatively palpated and resected lesions using the study documentation sheet. If a lesion was unidentifiable, the surgeon made a remark on the study documentation sheet. All additionally resected lesions were documented as well.
The necessity of this documentation sheet was a result of a prior retrospective study conducted by our department [15 ]. It bridged the gap between the radiologist, the surgeon and the pathologist and ensured that the radiologically detected lesion was equal to the one being resected.
All palpable intrapulmonary lesions were resected by laser. Each specimen was labeled and separately submitted to histopathological analysis. The label was recorded on the study documentation sheet. The pathologist reported the histopathological findings for each specimen separately with reference to the given label.
Follow-up
An attempt was made to obtain follow-up information for radiologically detected lesions missed during surgery. An attempt was also made to retrieve subsequent CT scans acquired either in-house or at external facilities. A lesion was considered benign if no lesion growth was documented for at least two years [16 ]
[17 ]. Lesions demonstrating growth at follow-up of any duration were regarded as metastases. Otherwise, the lesion was counted as lost to follow-up.
Statistics
Statistical analysis was performed with Microsoft Excel 2010 (Microsoft Corp., Redmond, Washington, USA) and SPSS 15.0 (SPSS Inc., Chicago, Illinois, USA). One-way analysis of variance was used to test for sensitivity differences between radiologists. One-sided McNemar-Test was used to test for sensitivity differences between radiologists without and with CAD. P-values < 0.05 were considered to indicate statistical significance. The sensitivity and 95 % confidence intervals were calculated. In the given study design, the specificity could not be calculated on a per-nodule or per-patient basis. Both values were calculated on a per-lobe basis instead. The number of false-positive CAD findings per CT scan was given as documented in the comment section of the CAD report.
The main unit for all analyses dealing with metastases consisted of all resected nodules in which metastasis was confirmed at histopathology. For all analyses dealing with all lesions, the main unit consisted of all resected lesions independent of the histopathological result. In analyses dealing with radiologically detected lesions missed during surgery, the main unit consisted of all lesions detected on CT either by the radiologist or by CAD that were not resected during surgery. A metastatic nature of the lesion was assumed when growth consistent with malignancy was detected on follow-up CT scans.
From a retrospective pilot study [15 ], estimates of mean sensitivity and standard deviation were known. Assuming a sensitivity of the radiologist alone of 0.7 with a standard deviation of 0.2, and having the null hypothesis mean at 0.7 with a superiority margin of 0.05, the required study size calculates to 99 cases to test the superiority of radiologist with CAD over radiologist alone with a statistical power of 0.8 with a type I error rate of 5 %.
Results
Number and size of detected lesions
The number of lesions per case ranged from 1 to 18 (mean: 6.0 ± 3.8 lesions, median: 5 lesions). A total of 693 lesions were detected either with CT or surgically. 47 lesions were detected on CT but were not found during surgery. These lesions have not been included in the main unit and are described separately at the end of the results section. The remaining 646 lesions were resected, and consequently histopathology was available. Of these, 262 (37.8 %) were found to be metastases at histopathology. 550 (79.4 %) lesions were detected on CT.
The size distribution of the nodules is given in [Fig. 2 ]. The largest diameter was measured in millimeters. The maximum lesion diameter ranged from 1 to 70 mm (mean: 8.0 ± 9.4 mm, median: 4.0 mm). Only 20.0 % of all lesions were larger than 10 mm. Most lesions were very small, and the majority of lesions with a size smaller than 7 mm were non-metastatic as shown in [Fig. 3 ].
Fig. 2 Size distribution of pulmonary nodules (longest axial diameter in mm).Abb. 2 Größenverteilung der pulmonalen Herde (längster axialer Durchmesser in mm).
Fig. 3 Nodule size and percentage of metastases.Abb. 3 Herdgröße und Prozentsatz von Metastasen.
Radiological detection of lung lesions
Radiologists without CAD detected a total of 436 lesions out of 646 resected lesions, thus resulting in an overall sensitivity of 67.5 %. The sensitivity rose to 87.4 % when considering only metastases (229 of 262 metastases). All radiologists involved in the study performed equally; one-way analysis of variance found no difference in sensitivity (all lesions: p = 0.878; metastases: p = 0.970).
The sensitivity of CAD as a stand-alone system was 53.6 % for all lesions and 61.8 % for metastases. The number of false positive findings per CT study ranged from 0 to 99 with an average of 13.6 ± 30.9. CAD added 10.4 % sensitivity for all lesions (sensitivity of radiologist with CAD is 77.9 %) and 5.3 % sensitivity for metastases (sensitivity of radiologist with CAD is 92.7 %). These differences were highly significant with p < 0.001 for all nodules and for metastases.
More details regarding sensitivity are presented in [Table 2 ]. Moreover, it demonstrates that there are only negligible differences in the performance of radiologists and CAD when evaluating only non-pleural lesions compared to all lesions. Hence, the information as to whether a lesion was pleural was no longer taken into account in subsequent analyses.
Table 2
Sensitivities for the detection of all lesions, metastases and non-pleural metastases by radiologist and CAD combined, radiologist alone, CAD alone and palpation, (numbers in parentheses are 95 % confidence intervals).Tab. 2 Sensitivität der Detektion aller Läsionen, aller Metastasen und aller nicht-pleuraler Metastasen durch Radiologen und CAD kombiniert, Radiologen allein, CAD allein, und chirurgische Palpation (Angaben in Klammern sind 95 %-Konfidenzintervalle).
all lesions (%)
all metastases (%)
non-pleural metastases (%)
radiologist + CAD
77.9 (74.7–81.1)
92.7 (89.6–95.9)
93.5 (90.4–96.6)
radiologist alone
67.5 (63.9–71.1)
87.4 (83.4–91.4)
88.2 (84.1–92.2)
CAD alone
53.6 (49.7–57.4)
61.8 (55.9–67.8)
63.3 (57.2–69.3)
palpation
99.5 (99.0–100.1)
99.2 (98.2–100.3)
99.2 (98.0–100.3)
The sensitivity was highly dependent on lesion size. Both radiologists and CAD performed poorly in the case of very small lesions and were nearly perfect in lesions with a diameter of at least 6 mm, as demonstrated in [Table 3 ].
Table 3
Sensitivity of radiologist, CAD, and both combined for the detection of all lesions and metastases (in parentheses: 95 % confidence interval).Tab. 3 Sensitivität von Radiologen, CAD, und beidem kombiniert für die Detektion von allen Läsionen und Metastasen (Angaben in Klammern sind 95 %-Konfidenzintervalle).
lesion size (mm)
sensitivity
radiologist
CAD
radiologist with CAD
all lesions
1–2
3–4
5–6
7–8
9–10
> 10
0.30 (0.23–0.37)
0.65 (0.58–0.72)
0.78 (0.69–0.87)
0.93 (0.86–1.00)
1.00
0.97 (0.94–1.00)
0.20 (0.14–0.26)
0.65 (0.58–0.72)
0.79 (0.70–0.88)
0.77 (0.66–0.88)
0.82 (0.69–0.96)
0.50 (0.42–0.59)
0.35 (0.28–0.42)
0.88 (0.83–0.93)
0.94 (0.89–0.99)
1.00
1.00
0.98 (0.96–1.01)
metastases
1–2
3–4
5–6
7–8
9–10
> 10
0.47 (0.23–0.72)
0.68 (0.52–0.83)
0.81 (0.66–0.97)
0.94 (0.87–1.02)
1.00
0.97 (0.93–1.00)
0.37 (0.13–0.61)
0.51 (0.34–0.68)
0.85 (0.71–1.00)
0.83 (0.71–0.96)
0.82 (0.67–0.97)
0.52 (0.43–0.61)
0.53 (0.28–0.77)
0.81 (0.68–0.94)
0.96 (0.89–1.04)
1.00
1.00
0.98 (0.96–1.01)
The specificity on a per-lobe basis was 98.2 % for the radiologist alone, 88.7 % for CAD alone, and 99.5 % for the radiologist with CAD.
Of the 143 resected lesions missed by the radiologist and CAD, 75 could retrospectively be detected with the information provided by the surgeon. 68 lesions could not be identified on the CT scans even in retrospect, among them 3 (5 %) histologically proven metastases; 9 of these lesions were reported by the surgeon to be pleural. The average size of the missed lesions that could be retrospectively identified on CT was 2.6 mm (range: 1–12 mm, median: 2.0 mm).
Surgical detection of lung lesions
644 (99.7 %) of 646 surgically resected lesions were palpable during surgery. Two more lesions (0.3 %) were resected based on their location on CT although they were not palpable. Both were found to be metastases at histopathology. Of all lesions, 503 (77.9 %) were detected by the radiologist with CAD, including 243 metastases. An additional 143 lesions were palpated and resected during surgery. Of these additional lesions, 19 (13.3 %) were metastatic. Details regarding the sensitivity of palpation during surgery for detecting lesions and metastases are presented in [Table 2 ].
Compared to the radiological findings, all 12 (100 %) patients with a solitary lesion on CT were found to have only one nodule during surgery. 9 of these lesions were metastatic (75 %). If CT demonstrated more than one lesion, more lesions than expected on CT were found in 59 of the 103 surgical procedures (57.3 %), but only in two cases (1.9 %) were there more histologically proven metastases.
A total of 49 lesions could not be palpated. Two non-palpable lesions were resected based on their location on the CT scan. Both were found to be metastatic at histopathology. The other 47 lesions remained in situ.
Lesions missed during surgery
Radiologists with CAD detected 47 (6.8 %) lesions that were not found during surgery and were not resected. For 20 (42.6 %) lesions no follow-up information could be obtained. They were regarded as lost to follow-up. For the remaining lesions, the available follow-up ranged from 69 to 1843 days (mean: 509 ± 483 days, median: 258 days). Using the criteria described in the methods section, sufficient follow-up was available in 7 (14.9 %) lesions. The remaining 40 (85.1 %) lesions were also regarded as lost to follow-up. Of the 7 lesions with sufficient follow-up, 2 (29 %; 95 % confidence interval: –8 % to 65 %) demonstrated growth at subsequent CT scans and were therefore regarded as metastatic. The other 5 lesions were considered benign with a follow-up of 1032 to 1843 days. Of the two metastatic lesions, one was detected only by CAD and was missed by the radiologist ([Fig. 4 ]), while the other was detected by the radiologist but missed by CAD.
Fig. 4 Metastasis in left upper lobe, overlooked by the radiologist but detected by CAD, missed during surgery and found to be metastatic on subsequent CT scans. Axial maximum-intensity projection with 8-mm slice thickness.Abb. 4 Metastase im linken Oberlappen, vom Radiologen übersehen, von CAD detektiert, intraoperativ nicht gefunden, war progredient in späteren CT-Verlaufskontrollen. Axiale Maximum-Intensitäts-Projektion mit 8 mm Schichtdicke.
If the proportion of metastases found in the cases with sufficient follow-up (29 %) is projected to the population of all 47 lesions missed by the surgeon, it can be roughly estimated that 13 of the 47 nodules might be metastases missed by the surgeon.
Discussion
Previous studies on the detection of lung metastases yielded a moderate sensitivity of CT. In an older study using single-slice spiral CT with a 3 and 5-mm slice thickness, Diederich reported 64 % sensitivity for all pulmonary lesions and 77 % for metastases [4 ]. In 2008, Ludwig [5 ] compared the number of lung metastases detected on CT and during surgery. Details of the CT technique were not given, but it can be supposed that they were not consistently thin-slice as the data was from 1998 to 2003. This study found more metastases during surgery than on CT in 27.2 % of the cases, a situation that was rare in our study. In our study the sensitivity of CT was higher. This may be explained by the subtler CT technique with CAD. None of the previously published studies used CAD.
It could be expected that CAD increases the sensitivity of the detection of pulmonary metastases. Our study found a significant increase in the sensitivity for the detection of lung metastases when using CAD as a second reader. The detection of unexpected metastases during surgery was rare and occurred in less than 2 % of all cases. Moreover, a relevant proportion of lesions detected on CT but missed during surgery represents metastases. However, the absolute number of these lesions in which sufficient follow-up information was available was small. In surgical studies not using CAD, CT performance was markedly inferior compared to our data, with lung metastases missed on CT in up to 36 % of cases [5 ]
[18 ]
[19 ]
[20 ]
[21 ]. This is an indirect argument for the use of CAD in addition to the review of the CT scan by the radiologist.
The specificity of the CAD system with more than 13 false-positives per CT scan was only moderate compared to more recent systems with less than 5 false-positives and the tendency to higher sensitivities [22 ]
[23 ]
[24 ]
[25 ]
[26 ]. However, it proved to be useful since it significantly added sensitivity to the performance of the radiologist alone.
Not surprisingly, manual palpation was found to be the most sensitive method in our study which is consistent with other studies [18 ]
[19 ]
[20 ]
[21 ]. However, a higher proportion of non-palpable lesions was metastatic in our study, thus emphasizing the potential benefit of CT with CAD as guidance for the surgeon.
Radiologists tend to detect typical metastases while ignoring atypical findings. 50 % of all lesions detected by radiologists were metastatic. Lesions not detected by the radiologist but detected by CAD were less likely to be metastatic (15.2 %). Lesions additionally detected by CAD tend to have CT-morphological features less typical for metastases. Nevertheless, CAD was useful even in these atypical lesions as illustrated in [Fig. 5 ].
Fig. 5 Largest metastasis missed by the radiologist but detected by CAD. Interpreted as scar at first read, detected by CAD and considered a positive second reader CAD finding by the radiologist: 31-mm subpleural lesion in the middle lobe. Adjacent scars due to prior surgery. Primary tumor was colorectal cancer. Screenshot of the CAD user interface is shown.Abb. 5 Größte vom Radiologen übersehene Metastase, die vom CAD detektiert wurde. Initial als Narbe interpretiert, wurde es als vom Radiologen als positiver Befund von CAD als second reader gewertet: 31 mm großer subpleuraler Herd im Mittellappen. Benachbart narbige Veränderungen nach früherer Metastasenchirurgie. Primärtumor war ein kolorektales Karzinom. Dargestellt ist die CAD-Nutzeroberfläche.
The detection of pleural metastases is more difficult than the detection of intrapulmonary metastases for both radiologists and CAD. This limitation is compensated by the easy detection of pleural metastases during surgery. CT can be regarded as complementary to intraoperative inspection and palpation since CT is able to easily detect small metastases deep inside the lung parenchyma which may be more challenging to detect during surgery.
Use of CAD increases the number of preoperatively known lesions and therefore has the potential to harm the patient by increasing the number of unnecessarily resected benign lesions. On the other hand, palpation during surgery results in even more additional lesions ([Fig. 6 ]). The percentage of additional metastases found by CAD was similar to the percentage of additional metastases found by manual palpation during surgery. Therefore, resection of lesions additionally detected by CAD is as beneficial to the patient as resection of additionally palpated lesions.
Fig. 6 Number of findings and number of metastases detected by radiologist alone (upper box), additionally by CAD (middle box) and by manual palpation (lower box). First number: number of detected lesions. Number in parentheses: number of detected metastases. Percentage: percentage of metastases in all detected lesions. § estimated values (see text for details).Abb. 6 Anzahl der Befunde und Anzahl der Metastasen, die durch den Radiologen allein (oberer Kasten), zusätzlich durch CAD (mittlerer Kasten) und intraoperativ durch Palpation (unterer Kasten) detektiert wurden. Erste Zahl: Anzahl der detektierten Läsionen. Zahl in Klammern: Anzahl der detektierten Metastasen. Prozentzahlen: Anteil der Metastasen an allen detektierten Läsionen. § Schätzwert (zu Details siehe Text).
Our study has certain limitations:
Since the presented study is a single-center study, the results may not be applicable to other centers with different levels of expertise resulting in higher or lower sensitivities for radiologists and surgeons. The study was performed in a “real world” setting, thus involving different radiologists and surgeons which may result in an inhomogeneous performance which is not quantifiable in our study. Moreover, the radiologist’s results with CAD are susceptible to the performance of the CAD system. In our study a single CAD system with its specific strengths and weaknesses regarding the detection of pulmonary nodules was used.
Even if a resected lesion was found to be non-metastatic at histopathology, it may well be possible that the lesion represented a metastasis that was sterilized by prior systemic treatment. Therefore, the number of metastases in our dataset may be underestimated. However, the study results taking into account all lesions did not substantially differ from the results for metastases.
No histopathological proof was available for lesions missed by surgery, and the percentage of lesions in which sufficient follow-up could be obtained was small.
Furthermore, a methodical limitation is a certain heterogeneity of imaging data in terms of CT vendor, slice thickness, reconstruction increment and reconstruction kernel because external CT scans from different institutions were accepted as long as they fulfilled the defined minimum requirements.
We found that CAD used as the second reader increases the sensitivity for the detection of lung metastases on CT scans.
The combination of CT with CAD and intraoperative palpation of the deflated lung may increase the chance for complete resection of lung metastases.
CT can therefore be used as guidance during surgery to achieve complete resection of lung metastases.
