Automatic quantitative analysis of atherosclerotic aortic plaques in patients with embolic cerebral infarction using deep learning
Article information
Abstract
Background/Aims
Transesophageal echocardiography (TEE) is a commonly used imaging modality for assessing embolic stroke of undetermined source (ESUS) in clinical practice. We aimed to develop an automatic plaque segmentation model based on U-net and evaluate its clinical usefulness in patients with ESUS.
Methods
We used two aorta image sets. TEE aortic images of 711 patients visiting two cardiovascular centers for various causes were randomly divided into training, validation, and test sets to automatically segment plaques and estimate the aortic plaque area (APA) and aortic plaque ratio (APR) using U-net. The model was tested in a clinical data set of patients with ESUS who attended three cardiovascular centers to determine whether it could predict a composite cardiovascular event in those patients.
Results
The mean intersection of over union to assess the accuracy of the U-net model was 0.997 ± 0.002 and 0.997 ± 0.001 for the model development and clinical application data sets, respectively. When using the U-net–based model, the APA and APR significantly differed between complex and simple aortic plaques (p < 0.001). However, unlike complex aortic plaques measured in clinical practice, APA or APR estimated by U-net models or manual segmentation did not show additional value in predicting major adverse cardiovascular and cerebrovascular events.
Conclusions
The estimation of APA and APR by the U-net model could be helpful in predicting complex aortic plaques. Additional comprehensive quantitative image analysis of plaque characteristics using artificial intelligence, such as movability and morphology, may be needed to predict prognosis.
INTRODUCTION
Stroke is imposing a growing burden on the global healthcare system, and the annual number of deaths due to stroke has increased substantially in recent years [1]. Ischemic stroke, accounting for approximately 80% of strokes, can be divided into three main subtypes: 1) thrombosis, which generally refers to local in situ obstruction of an artery; 2) embolism, which refers to particles of debris originating from heart or aortic atherosclerotic plaques; and 3) systemic hypoperfusion [2].
Aortic atherosclerotic plaques manifest as systemic atherosclerosis and are an important cause of cerebral embolization [3–6]. Transesophageal echocardiography (TEE) is the procedure of choice for detecting and measuring thoracic aortic plaques, especially when they are present in the ascending or proximal descending thoracic aorta and for cardiac sources. TEE can distinguish minimal intimal thickening from severe plaques and has 93% diagnostic accuracy compared with pathologic evaluation [7–9]. Therefore, TEE is a commonly used image modality in the clinic for assessing embolic stroke.
An automated pipeline for the segmentation of transthoracic echocardiography (TTE) using artificial intelligence (AI) was previously reported [10]. AI offers the potential to quickly and clinically measure plaques by automatically analyzing not only TTE but also TEE images. As a deep learning (DL) technique, U-net is a four-layer DL architecture with a stack of four encoder/decoder stages on both sides of the U shape [11]. Studies on the use of U-net for image processing have reported its excellent performance in carotid ultrasound image segmentation [12,13]. In this study, we aimed to evaluate the ability of the U-net DL model to predict complex aortic plaques and assess the prognosis of embolic stroke of undetermined source (ESUS) patients.
METHODS
Study population
In this study, we evaluated the performance of the U-net algorithm on two datasets. The first dataset, referred to as the model development set, was derived from TEE images of patients admitted for various clinical conditions, such as valvular heart disease, stroke, including ESUS, atrial fibrillation, patent foramen ovale, and atrial septal defects, from March 2016–January 2021, and the other dataset was derived from patients diagnosed with ESUS at two cardiovascular centers between May 2009 and January 2016 (Fig. 1).
Flow diagram indicating patient selection for the training, validation, and test sets for developing the U-net algorithm. The developed U-net algorithm was then tested with a clinical dataset including transthoracic echocardiographic images from patients with embolic cerebral ischemia. CHUH, Chungnam National University Hospital; CNUH, Chonnam National University Hospital; DJCMC, Daejeon Catholic Medical Center.
The second dataset, known as the clinical application set, comprised patients diagnosed with ESUS at three cardiovascular centers between May 2009 and January 2016 (Fig. 1). The clinical application set included clinical, laboratory, electrocardiogram, and brain magnetic resonance imaging (MRI) data, as well as TTE and TEE examination data. Subjects were excluded from this set if they had (1) missing values or information from the examinations, (2) a patent foramen ovale, (3) persistent or paroxysmal atrial fibrillation, (4) carotid artery atherosclerosis, or (5) small vessel stroke. The subjects in the clinical application set were divided into two groups according to the presence of complex aortic plaques, as confirmed by a cardiology expert, considering plaque thickening (≥ 4 mm) or the presence of ulcerations or mobile components [7,8]. This retrospective observational study was approved by the institutional review board (IRB) of Daejeon St. Mary Hospital, Chungnam National University Hospital, and Chonnam National University Hospital (DC15RISI0086, DC21DEDI0057, 2020-09-032, CNUH-2017-008). Due to the retrospective design, the IRB waived the requirement for informed consent. The subjects in the clinical application set underwent diagnostic examinations, including brain MRI, ultrasound imaging, MR angiography of the internal carotid and vertebral arteries (according to the standard protocol), 12-lead electrocardiographic studies, and TTE and TEE studies assessing the thoracic aorta. All hospitalized patients with embolic infarcts were eligible for enrollment in the study. Risk factors for embolic stroke were noted at the time of admission and included the following: diabetes mellitus, hypertension, hyperlipidemia, previous myocardial infarction, peripheral vascular disease, previous stroke, carotid artery stenosis, cigarette smoking, alcohol, and known atrial flutter or atrial fibrillation recorded within eight days after the detection of the cerebral infarct.
Transesophageal echocardiographic assessment of the aorta
All the subjects underwent TEE by trained cardiologists to evaluate possible causes of brain infarction in the clinical application set, if needed, along with the agitated saline bubble test. The TEE results were stored in the digital database in DICOM format. We used commercially available imaging systems with a 5 MHz multiplane probe according to standard techniques [14] (Supplementary Material). Two-dimensional B-mode ultrasound images of plaques in the descending aorta, distal arch, proximal arch, and ascending aorta were saved to the ultrasound machine and transferred to a personal computer for analysis. In cases of multiple plaques, the most advanced lesion was selected. Each B mode image was cropped manually by a circular sector to remove annotations from TEE vendors.
Ground truth
Raw aortic TEE images from a representative frame were manually labeled by two observers with more than ten years of clinical experience and used as the ground truth. To evaluate interobserver variability, 30 randomly selected images of aortic plaques were segmented by each observer, and the differences and correlation coefficients of the plaque area measurements were calculated.
Model development
U-net was used as the image segmentation method. It is a pixel-based image segmentation DL system first presented by Ronneberger et al. [11] in 2015. The base structure is symmetrical, taking on the shape of a “U”. One side is classified into an encoding area in which images are reduced, and the other side is classified into a decoding area in which images are enlarged. In addition, it is possible to minimize the loss and obtain a segmentation map with more accurate location information by associating the reduced information of the encoder in the decoder process. U-net training and testing were performed using Python PyTorch. A learning process system with Windows 10, 11th Generation Core i7-11700K CPUSs, 16 GB of RAM, and an NVIDIA GeForce RTX 3070 GPU was used for the GPU-based system.
Testing of the AI model
To evaluate the accuracy of the U-net model, we used the intersection over union (IoU) in the first dataset for model formation and the clinical application set. During training, weighted binary cross-entropy with logits, including a sigmoid function, was utilized as the loss function. Finally, we choose the Jaccard index (IoU) as an evaluation metric, which can be interpreted as a similarity measure between a finite number of sets. The IoU for the similarity measure between two sets A and B can be defined as follows:
Furthermore, Bland–Altman analysis was used to compare the U-net model and manual segmentation in the clinical application set. The U-net model was timed from when the DICOM data were input until the plaque segmentation results were produced.
Clinical outcomes
The electronic medical records of patients were examined for follow-up. The primary outcome was major adverse cardiovascular and cerebrovascular events (MACCEs), including acute coronary syndrome, recurrent hemorrhagic or ischemic stroke, and cardiovascular death.
Statistical analyses
To evaluate the clinical usefulness of aortic plaque quantification using U-net, we analyzed the clinical information from the clinical application set. Continuous variables are expressed as the means and standard deviations, and categorical variables are presented as frequencies and percentages. Student’s t-test and the chi-square test (or Fisher’s exact test) were used to compare the means and proportions of baseline characteristics between the two groups. We compared embolic stroke patients with complex aortic plaques and embolic stroke patients without complex aortic plaques. Receiver operating characteristic (ROC) analyses were performed to estimate the predictive value of complex aortic plaques by the area under the curve (AUC). The cumulative incidence was estimated by the Kaplan–Meier method, and differences were assessed using the log-rank test. A Cox proportional hazards model was used to identify independent predictors of the primary composite endpoints. The multivariate, stepwise Cox regression models included covariates with a p value smaller than 0.05 on univariate analysis and clinically relevant variables.
The presence of complex aortic plaques according to cardiology expert insight; plaque output; and confounding factors, including age, sex, E/e’, and left ventricular ejection fraction (LVEF), were included in the risk models. The ability to classify risk was assessed with the use of the C statistic in each Cox regression model. In addition, category-free net reclassification improvement and integrated discrimination improvement were calculated to assess the predictive improvement of Model 1 [15,16]. All the statistical analyses were performed using SAS 9.4 (SAS Institute, Cary, NC, USA), SPSS version 21.0 (IBM Corp., Armonk, NY, USA) and R version 3.3.2 (R Foundation for Statistical Computing, Vienna, Austria). All p values were two-sided, and values less than 0.05 were considered statistically significant.
RESULTS
Study population for model development
Aortic plaque images from 711 patients who underwent TEE for various clinical indications were randomly divided into training, validation, and test sets at a 6:2:2 ratio (Fig. 1). We developed the U-net model to automatically measure the aortic plaque area (APA) and plaque-to-aorta ratio (APR) (Fig. 2).
(A) Ground-truth image (upper panel) and an image automatically segmented with U-net (lower panel) provide the plaque area based on the pixels in the image. (B) The plaque aorta ratio was assessed as a quantitative plaque measurement.
Study population for the clinical application set
The second dataset consisted of 484 patients with ESUS who were divided into two groups according to the presence of complex aortic plaques. The developed U-net model was applied to 485 aortic plaques from 484 patients with ESUS from the first dataset. Table 1 and Table 2 present both groups' baseline clinical, laboratory, and echocardiographic characteristics. Patients with complex aortic plaques were older and had higher frequencies of hypertension and previous stroke. The complex aortic plaque group also had lower eGFRs, total cholesterol levels, and fibrinogen levels; a lower LVEF; a higher E/e’; and a higher LV mass index.
Baseline characteristics of the clinical application set
Transthoracic echocardiographic findings and assessment of aortic plaques in the clinical application set by clinicians and automatic assessments with U-net
Individual manual plaque segmentation variability
Two observers manually delineated all of the aortic images as ground-truth reference images. Thirty randomly selected images were measured by the two observers. The Pearson’s r value of the plaque area between the manual measurements by the two observers was 0.932 (95% confidence interval [CI] = 0.861 to −0.967, p < 0.001), and the coefficient of variation was 0.24, indicating good agreement between the manual segmentations by the two observers.
Accuracy and time savings of the U-net model
The development of the model took 15 hours for 711 images, including training, validation, and testing. The analysis by U-net took an average of 0.123 sec/image, compared with 155 ± 35 sec/image for manual measurements.
To evaluate the model's accuracy, the IoU results of the model were 0.997 ± 0.002 for the first dataset and 0.997 ± 0.001 for the clinical application dataset. The APAs and APRs determined by the U-net model and manual segmentation in the clinical application set were well correlated (r = 0.747, p < 0.001; r = 0.55, p < 0.001; Fig. 3A, B). The Bland–Altman plot showed good agreement between manual segmentation and U-net in the assessment of APA and APR (APA: 11.5 [95% CI = 90.6 to −67.6]; APR: −1.418 [95% CI = 9.59 to −12.42], Fig. 3C, D) (Fig. 4).
(A, B) Correlation of quantitative plaque characteristics between the ground-truth images and U-net for complex and simple aortic plaques. (C, D) Bland–Altman plots showing the average plotted against the difference in aortic plaque area and plaque aorta ratio between the ground truth and U-net measurements. (E, F) Model performance in the clinical application set for predicting complex aortic plaques. (G) Kaplan–Meier curve showing the clinical significance of complex aortic plaques using the traditional method to assess the clinical data set in terms of long-term clinical follow-up. AUC, area under the curve.
Among the cases, there were subtle differences in the plaque aortic ratio between the U-net model and manual model as ground truth. Humans tended to create a smooth line around the adventitia, but U-net used a more irregular margin around the adventitia in its segmentation.
U-net model performance in the prediction of complex plaques
The U-net model revealed significant differences between the sizes of complex and simple aortic plaques (98.4 ± 52.5 mm2 vs. 47.2 ± 31.3 mm2, p < 0001; Table 2). ROC analysis revealed that the APR of the U-net model had better performance in predicting complex aortic plaques than the APR of the manual model (manual model, AUC = 0.772; U-net model, AUC = 0.846, p = 0.009) (Fig. 3E). There were no significant differences in the ability of the APAs calculated from the manual model and the U-net model to predict complex plaques (manual model, AUC = 0.805; U-net model, AUC = 0.848, p = 0.072) (Fig. 3F).
Prognostic value of the U-net model
Age, E/e’, LVEF, and the presence of complex aortic plaques, as determined by the observers, were independent predictors of MACCEs (Table 3, Supplementary Table 1). However, complex aortic plaques defined by human experts predicted clinical events; however, neither APA nor APR, as determined by the U-net model or manual measurement, provided additional predictive value for MACCEs beyond the classic risk factor model (Table 3, Fig. 3G).
C-statistics, net reclassification improvement (NRI) and integrated discrimination improvement (IDI) for the composite clinical outcome
DISCUSSION
In this study, we developed a U-net-based model for the automatic measurement of APA using TEE images and evaluated its clinical predictive value in patients with ESUS. While APA and APR effectively differentiated complex plaques, plaque size alone was not predictive of clinical prognosis in patients with ESUS.
Compared with other AI algorithms, the U-net algorithm has the ability to segment complex structures effectively and possesses precise boundary delineation capabilities, making it well suited for aortic plaque segmentation [12,13]. Additionally, many studies have demonstrated the potential clinical advantages of AI in medical imaging, with active progress in multicenter cardiovascular imaging studies and the prediction of clinical events or clinical utility [17–19]. Despite these advancements, research related to TEE remains limited, with the majority focusing on differentiating cardiac masses or assessing valve pathology. To the best of the authors' knowledge, this study is the first to investigate the clinical utility of AI using TEE imaging of the aorta [20,21]. This study used real-world clinical TEE images from three medical centers and demonstrated that the APA and APR automatically measured by the U-net model correlated well with manual segmentation. Furthermore, both APA and APR measured by U-net were significantly different between complex and simple aortic plaques.
We hypothesized that either APA or APR could better predict clinical outcome events. However, only complex plaques defined by cardiology expert could predict clinical events, whereas neither the manually nor automatically measured APAs and APRs could. These results suggest that plaque size alone may be less predictive of prognosis than plaque instability is.
Complex plaques are not simply large, even though large plaques have been considered a risk factor for stroke [8]. Other plaque characteristics should be considered in the definition of a complex aortic plaque, such as an ulcerative or protruding morphology or movability as evidence of superimposed thrombi, which have been shown to contribute to increased risk [8,22,23]. In contrast, the presence of calcification appears to reduce the risk of stroke [23]. If features related to plaque instability were incorporated into the machine learning data, the U-net model might have achieved better predictive accuracy for event occurrence. Another possible explanation is that ESUS tends to have a more favorable prognosis than other stroke subtypes [24–26]. Moreover, the 12% MACCE rate observed in this study may not be sufficient for robust cardiovascular event prediction.
We did not expect either the APA or APR measured by U-net to more precisely predict complex plaques than the manually measured APA or APR; however, the U-net–measured APR showed better performance in identifying complex plaques than manual APR. The reason for this could be that humans tend to label plaques with smooth lines, but U-net segmented plaques with more irregular lines based on echogenicity. Consequently, simple plaques may be smaller and complex plaques are larger with U-net than with manual measurements.
AI models offer remarkable advantages. Notably, the U-net model requires no user interaction for plaque identification and region-of-interest segmentation [27]. In this study, manual segmentation was performed by two experienced observers (≥ 10 years of experience) and demonstrated a strong correlation with AI-derived measurements. The model can efficiently assist physicians in plaque analysis during the postacquisition phase, processing images faster than manual interpretation and reducing workload.
Plaques in the thoracic aorta are common findings on routine TEE. The likelihood of embolization from aortic plaques, such as plaque morphology and location, has been identified with TEE and computed tomographic angiography. However, TEE provides more information about plaques; for example, the mobile component is usually a thrombus superimposed on the plaque, which is presumably ruptured and may be more characteristic of a high lipid content [28,29]. Further studies are needed to determine whether an automated measurement algorithm incorporating those additional plaque characteristics can improve clinical event prediction.
Clinical implications
This study utilized AI to automatically analyze aortic atherosclerotic plaques in ESUS patients with embolic events using TEE, which provides real-time, high-resolution imaging of aortic atherosclerosis. This study demonstrated that AI-based APA measurements significantly differed between complex and simple aortic plaques and correlated well with manual measurements. However, unlike complex plaques defined by cardiology experts, AI-derived APA did not add predictive value for MACCEs within the classic risk factor model. These findings suggest that plaque size alone is less predictive of cardiovascular events than plaque instability. If plaque instability features were incorporated into the machine learning data, the U-net model might have better predictive accuracy for event occurrence.
Study limitations
This study was retrospective and may have inherent limitations related to data collection, such as selection bias. All examinations were conducted according to ESUS criteria [6]. However, several limitations of ultrasound-based diagnosis were noted. TEE does not visualize certain portions of the ascending aorta due to tracheal interference, which may result in up to 2% of plaques being missed [30,31]. Although some regions could be partially evaluated, most of the proximal and distal arch and descending thoracic aorta were accessible in most patients.
The accuracy of AI algorithms depends on the amount of data used in their creation, validation, and testing. This study had a relatively small sample size. Although previous studies have assessed such algorithms using carotid artery data, the correlation between the automatically measured APA and the ground-truth APA in this study was lower than that in previous studies [13,27]. This result is presumed to have been caused by several factors. First, this study used real-world clinical images from several commercially available vendors, and there may be subtle differences in the ultrasound images obtained therein. Furthermore, unlike previous studies, we did not employ image manipulation, such as cropping the region of interest around the plaque, which could improve plaque recognition [12]. It is also worth noting that to our knowledge, there are no previously published studies about the U-net algorithm using TEE images for assessing aortic plaques in patients with stroke. Finally, this study segmented plaques from a single representative frame. Future studies should integrate video-based AI analysis to enhance its clinical utility.
The U-net model’s estimation of APAs could enhance the identification of complex aortic plaques by improving both time efficiency and accuracy. To predict the clinical prognosis of ESUS patients, a more comprehensive quantitative image analysis that reflects plaque instability, including not only aortic plaque size but also plaque mobility and tissue composition, is required.
KEY MESSAGE
1. A U-net-based automated segmentation model was used to estimate the aortic plaque area in patients with ESUS.
2. This model may aid in identifying complex aortic plaques with both speed and accuracy.
3. However, APA/APR from U-net could not independently predict MACCEs in ESUS patients. Further AI advancements are needed to assess plaque tissue characteristics, similar to human evaluation, for better MACCE prediction.
Notes
CRedit authorship contributions
Hye Jin Bang: methodology, investigation, data curation, validation, software, writing - original draft, visualization; Jae-Hyeong Park: methodology, writing - original draft, writing - review & editing, supervision; Sun Geu Chae: methodology, validation, software; Suk Joo Bae: supervision; Ji-Hoon Jung: formal analysis, validation; You Hee Cho: data curation, validation, supervision; Jong Won Park: software, supervision; Dae-Won Kim: supervision; Jung Sun Cho: conceptualization, methodology, investigation, validation, software, writing - original draft, writing - review & editing, visualization
Conflicts of interest
The authors disclose no conflicts.
Funding
This work was supported by a National Research Foundation of Korea (NRF) grant funded by the Korean government (MSIT) (No. 5-20225A0154-00126).