• KSII Transactions on Internet and Information Systems
    Monthly Online Journal (eISSN: 1976-7277)

Generating Radiology Reports via Multi-feature Optimization Transformer

Vol. 17, No. 10, October 31, 2023
10.3837/tiis.2023.10.010, Download Paper (Free):


As an important research direction of the application of computer science in the medical field, the automatic generation technology of radiology report has attracted wide attention in the academic community. Because the proportion of normal regions in radiology images is much larger than that of abnormal regions, words describing diseases are often masked by other words, resulting in significant feature loss during the calculation process, which affects the quality of generated reports. In addition, the huge difference between visual features and semantic features causes traditional multi-modal fusion method to fail to generate long narrative structures consisting of multiple sentences, which are required for medical reports. To address these challenges, we propose a multi-feature optimization Transformer (MFOT) for generating radiology reports. In detail, a multi-dimensional mapping attention (MDMA) module is designed to encode the visual grid features from different dimensions to reduce the loss of primary features in the encoding process; a feature pre-fusion (FP) module is constructed to enhance the interaction ability between multi-modal features, so as to generate a reasonably structured radiology report; a detail enhanced attention (DEA) module is proposed to enhance the extraction and utilization of key features and reduce the loss of key features. In conclusion, we evaluate the performance of our proposed model against prevailing mainstream models by utilizing widely-recognized radiology report datasets, namely IU X-Ray and MIMIC-CXR. The experimental outcomes demonstrate that our model achieves SOTA performance on both datasets, compared with the base model, the average improvement of six key indicators is 19.9% and 18.0% respectively. These findings substantiate the efficacy of our model in the domain of automated radiology report generation.


Show / Hide Statistics

Statistics (Cumulative Counts from December 1st, 2015)
Multiple requests among the same browser session are counted as one view.
If you mouse over a chart, the values of data points will be shown.

Cite this article

[IEEE Style]
R. Wang and R. Hua, "Generating Radiology Reports via Multi-feature Optimization Transformer," KSII Transactions on Internet and Information Systems, vol. 17, no. 10, pp. 2768-2787, 2023. DOI: 10.3837/tiis.2023.10.010.

[ACM Style]
Rui Wang and Rong Hua. 2023. Generating Radiology Reports via Multi-feature Optimization Transformer. KSII Transactions on Internet and Information Systems, 17, 10, (2023), 2768-2787. DOI: 10.3837/tiis.2023.10.010.

[BibTeX Style]
@article{tiis:56209, title="Generating Radiology Reports via Multi-feature Optimization Transformer", author="Rui Wang and Rong Hua and ", journal="KSII Transactions on Internet and Information Systems", DOI={10.3837/tiis.2023.10.010}, volume={17}, number={10}, year="2023", month={October}, pages={2768-2787}}