Unit performance evaluation test report
The number of positive (with fibrotic ILD findings) and negative (no fibrotic ILD findings) images (number of cases) and ratio based on the correct data of 1,307 images to be tested in this test are 114 positive (8.7%) and negative 1,193 sheets (91.3%).
Interpretation test report
There were 120 randomly selected images (24 positive images, 96 negative images) for the reading test image data set.
/
匿名加工情報を収集する研究のため、疾病等の発生状況についての評価は実施していない。
Since this study collects anonymously processed information, we did not evaluate the occurrence of adverse events.
Unit performance evaluation test report
5.3 Conclusion
Based on the results of this test alone, with a sensitivity of 88.6% and a specificity of 87.2%, this program is considered to be able to accurately detect the imaging findings seen in fibrotic ILD from the intended images. The stratified analysis data obtained in this study are as follows.
1) Among fibrotic ILD negative images, this program falsely identified negative findings, especially infiltrative shadows-increased density areas, linear shadows-strand shadows, and pleural thickening-calcifications. It was confirmed that it is easy to lead to positivity. With regard to this, it was considered necessary to warn users through package inserts, etc. that this program is likely to lead to false positives.
In addition to the information on the false positive rate of this program for each negative finding, data on the degree of confidence is also provided to the user in the instruction manual, etc., so that the risk of leading doctors to false positives by using this program together It was considered possible to reduce
2) In the evaluation data set of this study, the number of images with variations in the position of reticular shadows or ground-glass opacities in fibrotic ILD positive images, which are assumed to be rare even in clinical practice, was very small. We should continue to collect information about the impact on the performance of On the other hand, regarding the ratio of reticular shadows or ground-glass opacities in fibrotic ILD positive images, a certain number of images were obtained in each layer, and it is unlikely that this difference would affect the ability of this program to detect fibrotic ILD. It was considered. Confidence trends in fibrotic ILD 'positive' images based on the proportion of reticular or ground-glass opacities in both lungs were considered useful as ancillary data for physicians to interpret the program-calculated confidence levels. .
3) The specificity of this program is higher for correct negative images judged negative by all three correct judgment doctors than for correct negative images judged negative by two correct judgment doctors. gave a higher result. There was a tendency for the false positive rate of this program to be high for the correct negative images judged as negative by two correct judgment doctors.
4) As for the effect of X-ray diagnostic equipment vendors on the performance of this program, the performance of this program deteriorated for Hitachi's X-ray generator. Information on vendor differences obtained in this study should be provided to users through package inserts, etc.
5) As for the effects on the performance of this program due to differences in imaging method, patient age, and image quality, the performance of this program deteriorated significantly when the imaging direction was AP. From these results, it was considered appropriate to exclude the standing AP from the input image conditions of this program.
6) If the confidence level calculated by this program is as high as 0.7 or higher, there is a high possibility that the patient is fibrotic ILD positive, suggesting that the confidence level calculated by this program is useful for determining fibrotic ILD.
Interpretation test report
6. Discussion and Conclusion
6.1 Primary endpoint
Using an interpretation test image data set created assuming the use of this program in actual clinical practice, based on the results of interpretation by a single doctor (without CAD) by 20 non-specialized interpretation doctors and the results of interpretation by using this program (with CAD) As a result of comparing the AUC values calculated by each ROC analysis, the AUC with CAD improved statistically significantly. Therefore, the combined use of this program is thought to improve the ability to detect fibrotic ILD in non-specialist chest X-ray images, demonstrating the usefulness of this program. By using this program, we believe that we can help prevent oversight of fibrotic ILD by primary care physicians.
6.2 Secondary endpoints
A significant increase in sensitivity and a significant reduction in the false-negative rate were found with CAD versus without CAD in 20 non-specialized reading physicians. Therefore, it is thought that the combined use of this program will reduce the oversight of fibrotic ILD by non-specialists, and in addition to the results of the primary endpoint, the results reinforce the usefulness of preventing oversight of fibrotic ILD by primary care physicians. Got. On the other hand, there was no significant change in the specificity and false-positive rate depending on the presence or absence of concomitant use of CAD among the 20 non-specialist reading physicians. The harm (overdiagnosis) from concomitant use of this program is considered to be at an acceptable level. The user of this program is assumed to be a non-specialist doctor (family doctor), but considering the effect on fibrosis ILD detection ability when a specialist uses this program together, The presence or absence of the effect was secondarily evaluated by calculating the positive rate, false negative rate, and accuracy. As a result, no statistically significant changes were observed in any index of sensitivity, specificity, false-positive rate, false-negative rate, and accuracy among the five specialist reading physicians. Therefore, the impact of this program on the ability of specialists to detect fibrotic ILD is considered to be small. For the purpose of evaluating the risk of misdiagnosis by using this program together, we calculated the darkening rate and improvement rate of non-specialist reading doctors and specialist reading doctors. As a result, for both non-specialist and specialist radiologists, when this program was used in combination with the correct positive images, the darkening rate of the interpretation results was smaller than the improvement rate, and the median value was 0. Therefore, more than half of the reading physicians did not see any darkening of the interpretation results due to the combined use of this program. In some interpretation doctors, it has been confirmed that the reading result of the correct positive image darkens when this program is used in combination. The improvement rate of the result is higher than the darkening rate, and it is considered that the benefit of improving the reading result by using this program together is greater.
6.3 Exploratory evaluation
6.3.1 Stratified analysis of the position and ratio of reticular opacities or ground-glass opacities in the left and right lungs in the positive images
In a stratified analysis of the position and ratio of reticular opacities or ground-glass opacities in the left and right lungs in the correct positive images, there were no features of the "positive" images that caused the darkening rate to worsen.
6.3.2 Stratified analysis of the percentage of reticular opacities or ground-glass opacities in both lungs in the positive images
In the interpretation results of non-specialist radiologists, when the proportion of reticular opacities or ground-glass opacities in both lungs was 3 points or more, there was no change in sensitivity with or without the combined use of this program. This is probably due to the high sensitivity of 96% for non-specialist radiologists without CAD for this proportion of images. On the other hand, when the percentage of reticular opacities or ground-glass opacities in both lungs was less than 3 points, combined use of this program was effective in improving sensitivity. Therefore, this program is considered to have the effect of increasing the sensitivity of non-specialists to simple chest X-ray images of fibrotic ILD, in which the ratio of reticular opacities or ground-glass opacities in both lungs is small. The usefulness of the program was suggested.
This study showed that this program is useful as a fibrotic ILD detection support program for non-specialists. In particular, this program was thought to be useful for preventing oversight of fibrotic ILD by primary care physicians. Based on the results of the darkening rate and improvement rate, we believe that the benefit of improving interpretation results by using this program in combination is greater than the risk of misdiagnosis.
2022年11月30日
3 IPDシェアリング
無
No
管理的事項
研究の種別
観察研究
登録日
令和4年11月14日
jRCT番号
jRCT1032220090
1 臨床研究の実施体制に関する事項及び臨床研究を行う施設の構造設備に関する事項
(1)研究の名称
胸部単純 X 線画像による線維化を伴う間質性肺疾患検出を支援する CAD ソフトウェアの性能評価
Performance evaluation of CAD software to support the detection of chronic fibrosing interstitial lung diseases by chest simple X-ray imaging
胸部単純 X 線画像による線維化を伴う間質性肺疾患検出を支援する CAD ソフトウェアの性能評価
Performance evaluation of CAD software to support the detection of chronic fibrosing interstitial lung diseases by chest simple X-ray imaging
(2)研究責任医師(多施設共同研究の場合は、研究代表医師)に関する事項等
本田 聖和
Honda Seiwa
/
エムスリー株式会社
M3, Inc.
AIラボ
107-0052
/
東京都港区赤坂1-11-44
1-11-44, Akasaka, Minato-ku, Tokyo
03-6229-8900
s-honda@m3.com
本田 聖和
Honda Seiwa
エムスリー株式会社
M3, Inc.
AIラボ
107-0052
東京都港区赤坂1-11-44
1-11-44, Akasaka, Minato-ku, Tokyo
03-6229-8900
s-honda@m3.com
令和4年5月2日
(3)研究責任医師以外の臨床研究に従事する者に関する事項
株式会社メディサイエンスプラニング
畑中 究
臨床開発部
(4)多施設共同研究における研究責任医師に関する事項等
多施設共同研究の該当の有無
なし
2 臨床研究の目的及び内容並びにこれに用いる医薬品等の概要
(1)臨床研究の目的及び内容
胸部単純 X 線画像による線維化を伴う間質性肺疾患(以下「線維化 ILD」と称す)を支援する CAD ソフトウェアの、非専門医による線維化 ILD診断における診断支援に対する有用性を、読影試験により検証する。
N/A
実施計画の公表日
2022年06月08日
実施計画の公表日
2022年09月30日
実施予定被験者数 / Sample Size
1000
試験の種類 /
Study Type
観察研究
Observational
試験デザイン
Study Design
無作為化 / allocation
盲検化 /masking
対照 / control
割付け / assignment
研究目的 / purpose
プラセボの有無
盲検の有無
無作為化の有無
保険外併用療養の有無
臨床研究を実施する国(日本以外) /
Countries of Recruitment(Except Japan)
1. Images of patients who have been continuously followed up based on imaging findings, subjective symptoms, objective findings, and other test results since the first visit.
2. Images of patients who obtained written or oral informed consent by May 2022 after the approval of this study by the Institutional Review Board at each image collection medical institution.Or images other than the patient who expressed refusal about the image specifications in this study during the same period.
3. Image of a patient over 20 years old at the time of imaging the target image
4. Simple chest X-ray images taken at an imaging medical institution for 6 consecutive months between January 2000 and May 2022
5. Images taken from 100 cm to 200 cm from the X-ray source to the film
6. An image with 1750 x 1750 pixels or more
7. Images that do not deviate from the correction process used in normal medical care
1. Image with incomplete DICOM information
2. Images not taken in a standing position
3. Images that may be difficult to read due to artifacts, etc.
4. Other images that the principal investigator of the image collection medical institution deems inappropriate
A paired t-test is performed on the AUC in the ROC curve with and without CAD based on the "reading judgment result" of a non-specialist, and a significant improvement with CAD is examined.