Heart failure & cardiomyopathies
Natural Language Processing for Adjudication of Heart Failure in a Multicenter Clinical Trial: A Secondary Analysis of a Randomized Clinical Trial
Authors' conclusion
The C3PO NLP model developed within 1 health care system identified HF events with good agreement relative to the gold-standard CEC in an external multicenter clinical trial. Fine-tuning the model improved agreement and approximated human reproducibility.
Study context
- Population
- Hospitalizations of participants with CVD in the INVESTED influenza vaccine trial, 157 sites in the US and Canada, 2016–2019
- Participants
- 1,973 participants4,060 hospitalizations
- Comparison
- Cohort Project (C3PO) natural language processing (NLP) model vs gold-standard clinical events committee (CEC) adjudication of heart failure (HF) hospitalization
- Outcome
- Concordance of NLP HF adjudication with CEC adjudication (raw agreement, κ, sensitivity, specificity)
Methodological detailsretrospective analysis of the INVESTED trial, external validation, multicentre, Analysis performed November 2022 to October 2023
- Retrospective analysis of the INVESTED trial
- External validation
- Multicentre
- Analysis performed November 2022 to October 2023
Performance
As reported in the abstract
HF hospitalization adjudication vs CEC
| Test or model | Raw agreement | κ | Sensitivity | Specificity |
|---|---|---|---|---|
| C3PO NLP | 87% (95% CI, 86-88) | 0.69 (95% CI, 0.66-0.72) | 94% (95% CI, 92-95) | 84% (95% CI, 83-85) |
C3PO NLP
Raw agreement87% (95% CI, 86-88)
κ0.69 (95% CI, 0.66-0.72)
Sensitivity94% (95% CI, 92-95)
Specificity84% (95% CI, 83-85)
Further findings
| Outcome | C3PO + INVESTED | De novo NLP |
|---|---|---|
| Fine-tuned and de novo NLP models vs CEC (internal validation cohort)Internal validation cohort not used for training | ||
| Agreement | 93% (95% CI, 92-94) | 93% (95% CI, 92-94) |
| κ | 0.82 (95% CI, 0.77-0.86) | 0.83 (95% CI, 0.79-0.87) |
- CEC reviewer interrater reproducibility
- 94% (95% CI, 93-95) · Reproducibility · κ 0.85 (95% CI, 0.80-0.89)
Original publication
Natural Language Processing for Adjudication of Heart Failure in a Multicenter Clinical Trial: A Secondary Analysis of a Randomized Clinical Trial. JAMA Cardiol. Published 1 February 2024. doi:10.
Composed by AI from the published abstract and checked against it. How Briefs are made →