Clinical evaluation of keyword-based, semi-automated created surgical reports using the example of FESS

Background

The postoperative creation of surgery reports as a time-consuming work step can promote errors in content. Aim of this study is to use an AI-recognition tool to create and evaluate partially automated, keyword-based operating reports intraoperatively, using the example of FESS.

Methods

In a pilot study, a compatible vocabulary for keyword-based documentation was created. For this purpose, n=48 surgical reports of experienced FESS surgeons were used. The text modules were subsequently implemented in a language model and validated by objective metrics (BLEU, ROUGE, VTECH, METEOR). In a follow-up study with n=18 ENT physicians, n=3 computer-generated OR reports were evaluated and compared with conventional reports. A questionnaire with the categories: "subjectively assessed added value", "time saving" and "general evaluation of the reports" was used for evaluation. Furthermore, parameters such as "grammar/content" and "number of corrections" were collected.

Results

There was an improvement in the objective parameters through optimisation of the language model (p<.05). 83% of the test persons stated an "added value" and 100% a "time saving" through this software tool (89% to 30min/d). In direct comparison, however, the conventional reports performed slightly better. Objectively, the number of corrections (M=23.25) of the artificially generated reports gave indications of grammatical and content optimisations.

Conclusions

The results show that this software tool can be of clinical benefit to surgeons by saving time and reducing workload. The neural model will be further trained to improve the quality of reports in terms of content and grammar.

HNO Uniklinik, ICCAS

Publication History

Article published online:
24 May 2022