深度学习在医学影像诊断中取得显著成果,基于深度神经网络的模型可以有效辅助医生进行决策。然而,随着模型参数规模逐渐增大,且高质量医学影像数据的标签需要专业医师手工完成,因此大规模参数模型在医疗领域愈发面临数据稀缺的挑战。一种解决方案是引入与医学影像成对的医学报告指导训练,这涉及2种模态的交互,而通用领域的跨模态对齐方法缺乏对细节信息的捕捉,不能完全适用于医疗领域。为解决此问题,提出一种注意力指导的双粒度跨模态医学特征学习框架ADCRL,实现了医学影像和报告在粗粒度和细粒度上的对齐。ADCRL能够提取出医学影像和医学报告2种粒度上的特征,使用注意力指导的模块选择医学任务可能感兴趣的影像区域,并去除噪声区域。通过对比学习式的代理任务实现2个粒度上模态的对齐。ADCRL在无监督范式下训练模型理解2种模态的全局语义和细节语义,并在下游任务中仅使用有限标注数据,即可表现出优秀的性能。主要工作包括提出细粒度特征选择方法和双粒度跨模态特征学习框架,并在公开医疗数据集上预训练并验证了框架的有效性。


Deep learning has achieved significant results in medical imaging diagnosis, and models based on deep neural networks can effectively assist doctors in making decisions. However, as the scale of model parameters gradually increases, large-scale parameter models in the medical domain are increasingly facing the challenge of data scarcity, as the labeling of high-quality medical image data requires professional physicians to manually complete. One solution is to introduce medical report guidance training paired with medical images, which involves the interaction of two modalities. However, cross-modal alignment methods in the general field lack capture of detailed information and cannot be fully applicable to the medical domain. To address this issue, an attention-guided dual-granularity cross-modal medical representation learning framework ADCRL is proposed to achieve alignment of medical images and reports at both coarse-grained and fine-grained levels. ADCRL can extract features from medical images and medical reports at two granularities, use an attention-guided module to select image regions of interest for medical tasks and remove noisy regions, and align two modalities at different granularities through contrastive learning based proxy tasks. ADCRL trains models under unsupervised paradigms to understand the global and detailed  semantics of two modalities, and demonstrates excellent performance in downstream tasks using only limited annotated data. The main work include proposing fine-grained feature selection methods and a dual-granularity cross-modal feature learning framework, and pretraining and validating the effectiveness of the framework on publicly available medical datasets.


相关文章