Diagnostic Assessment of Assumptions for External Validity: An Example Using Data in Metastatic Colorectal Cancer

Michael A. Webster-Clark, Hanna Sanoff, Til Hans Robert Stuermer, Sharon Peacock Hinton, Jennifer LEIGH Lund

Research output: Contribution to journalArticle

Abstract

BACKGROUND: Methods developed to estimate intervention effects in external target populations assume that all important effect measure modifiers have been identified and appropriately modeled. Propensity score-based diagnostics can be used to assess the plausibility of these assumptions for weighting methods. METHODS: We demonstrate the use of these diagnostics when assessing the transportability of treatment effects from the standard of care for metastatic colorectal cancer control arm in a phase III trial (HORIZON III) to a target population of 1,942 Medicare beneficiaries age 65+ years. RESULTS: In an unadjusted comparison, control arm participants had lower mortality compared with target population patients treated with the standard of care therapy (trial vs. target hazard ratio [HR] = 0.72, 95% confidence interval [CI], 0.58, 0.89). Applying inverse odds of sampling weights attenuated the trial versus target HR (weighted HR = 0.96, 95% CI = 0.73, 1.26). However, whether unadjusted or weighted, hazards did not appear proportional. At 6 months of follow-up, mortality was lower in the weighted trial population than the target population (weighted trial vs. target risk difference [RD] = -0.07, 95% CI = -0.13, -0.01), but not at 12 months (weighted RD = 0.00, 95% CI = -0.09, 0.09). CONCLUSION: These diagnostics suggest that direct transport of treatment effects from HORIZON III to the Medicare population is not valid. However, the proposed sampling model might allow valid transport of the treatment effects on longer-term mortality from HORIZON III to the Medicare population treated in clinical practice. See video abstract at, http://links.lww.com/EDE/B435.

LanguageEnglish (US)
Pages103-111
Number of pages9
JournalEpidemiology (Cambridge, Mass.)
Volume30
Issue number1
DOIs
StatePublished - Jan 1 2019

Fingerprint

Health Services Needs and Demand
Colorectal Neoplasms
Medicare
Confidence Intervals
Standard of Care
Mortality
Population
Propensity Score
Therapeutics
Weights and Measures

ASJC Scopus subject areas

  • Epidemiology

Cite this

Diagnostic Assessment of Assumptions for External Validity : An Example Using Data in Metastatic Colorectal Cancer. / Webster-Clark, Michael A.; Sanoff, Hanna; Stuermer, Til Hans Robert; Peacock Hinton, Sharon; Lund, Jennifer LEIGH.

In: Epidemiology (Cambridge, Mass.), Vol. 30, No. 1, 01.01.2019, p. 103-111.

Research output: Contribution to journalArticle

@article{571f0dded38e4a64996549649c7d6fda,
title = "Diagnostic Assessment of Assumptions for External Validity: An Example Using Data in Metastatic Colorectal Cancer",
abstract = "BACKGROUND: Methods developed to estimate intervention effects in external target populations assume that all important effect measure modifiers have been identified and appropriately modeled. Propensity score-based diagnostics can be used to assess the plausibility of these assumptions for weighting methods. METHODS: We demonstrate the use of these diagnostics when assessing the transportability of treatment effects from the standard of care for metastatic colorectal cancer control arm in a phase III trial (HORIZON III) to a target population of 1,942 Medicare beneficiaries age 65+ years. RESULTS: In an unadjusted comparison, control arm participants had lower mortality compared with target population patients treated with the standard of care therapy (trial vs. target hazard ratio [HR] = 0.72, 95{\%} confidence interval [CI], 0.58, 0.89). Applying inverse odds of sampling weights attenuated the trial versus target HR (weighted HR = 0.96, 95{\%} CI = 0.73, 1.26). However, whether unadjusted or weighted, hazards did not appear proportional. At 6 months of follow-up, mortality was lower in the weighted trial population than the target population (weighted trial vs. target risk difference [RD] = -0.07, 95{\%} CI = -0.13, -0.01), but not at 12 months (weighted RD = 0.00, 95{\%} CI = -0.09, 0.09). CONCLUSION: These diagnostics suggest that direct transport of treatment effects from HORIZON III to the Medicare population is not valid. However, the proposed sampling model might allow valid transport of the treatment effects on longer-term mortality from HORIZON III to the Medicare population treated in clinical practice. See video abstract at, http://links.lww.com/EDE/B435.",
author = "Webster-Clark, {Michael A.} and Hanna Sanoff and Stuermer, {Til Hans Robert} and {Peacock Hinton}, Sharon and Lund, {Jennifer LEIGH}",
year = "2019",
month = "1",
day = "1",
doi = "10.1097/EDE.0000000000000926",
language = "English (US)",
volume = "30",
pages = "103--111",
journal = "Epidemiology",
issn = "1044-3983",
publisher = "Lippincott Williams and Wilkins",
number = "1",

}

TY - JOUR

T1 - Diagnostic Assessment of Assumptions for External Validity

T2 - Epidemiology

AU - Webster-Clark, Michael A.

AU - Sanoff, Hanna

AU - Stuermer, Til Hans Robert

AU - Peacock Hinton, Sharon

AU - Lund, Jennifer LEIGH

PY - 2019/1/1

Y1 - 2019/1/1

N2 - BACKGROUND: Methods developed to estimate intervention effects in external target populations assume that all important effect measure modifiers have been identified and appropriately modeled. Propensity score-based diagnostics can be used to assess the plausibility of these assumptions for weighting methods. METHODS: We demonstrate the use of these diagnostics when assessing the transportability of treatment effects from the standard of care for metastatic colorectal cancer control arm in a phase III trial (HORIZON III) to a target population of 1,942 Medicare beneficiaries age 65+ years. RESULTS: In an unadjusted comparison, control arm participants had lower mortality compared with target population patients treated with the standard of care therapy (trial vs. target hazard ratio [HR] = 0.72, 95% confidence interval [CI], 0.58, 0.89). Applying inverse odds of sampling weights attenuated the trial versus target HR (weighted HR = 0.96, 95% CI = 0.73, 1.26). However, whether unadjusted or weighted, hazards did not appear proportional. At 6 months of follow-up, mortality was lower in the weighted trial population than the target population (weighted trial vs. target risk difference [RD] = -0.07, 95% CI = -0.13, -0.01), but not at 12 months (weighted RD = 0.00, 95% CI = -0.09, 0.09). CONCLUSION: These diagnostics suggest that direct transport of treatment effects from HORIZON III to the Medicare population is not valid. However, the proposed sampling model might allow valid transport of the treatment effects on longer-term mortality from HORIZON III to the Medicare population treated in clinical practice. See video abstract at, http://links.lww.com/EDE/B435.

AB - BACKGROUND: Methods developed to estimate intervention effects in external target populations assume that all important effect measure modifiers have been identified and appropriately modeled. Propensity score-based diagnostics can be used to assess the plausibility of these assumptions for weighting methods. METHODS: We demonstrate the use of these diagnostics when assessing the transportability of treatment effects from the standard of care for metastatic colorectal cancer control arm in a phase III trial (HORIZON III) to a target population of 1,942 Medicare beneficiaries age 65+ years. RESULTS: In an unadjusted comparison, control arm participants had lower mortality compared with target population patients treated with the standard of care therapy (trial vs. target hazard ratio [HR] = 0.72, 95% confidence interval [CI], 0.58, 0.89). Applying inverse odds of sampling weights attenuated the trial versus target HR (weighted HR = 0.96, 95% CI = 0.73, 1.26). However, whether unadjusted or weighted, hazards did not appear proportional. At 6 months of follow-up, mortality was lower in the weighted trial population than the target population (weighted trial vs. target risk difference [RD] = -0.07, 95% CI = -0.13, -0.01), but not at 12 months (weighted RD = 0.00, 95% CI = -0.09, 0.09). CONCLUSION: These diagnostics suggest that direct transport of treatment effects from HORIZON III to the Medicare population is not valid. However, the proposed sampling model might allow valid transport of the treatment effects on longer-term mortality from HORIZON III to the Medicare population treated in clinical practice. See video abstract at, http://links.lww.com/EDE/B435.

UR - http://www.scopus.com/inward/record.url?scp=85057538492&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85057538492&partnerID=8YFLogxK

U2 - 10.1097/EDE.0000000000000926

DO - 10.1097/EDE.0000000000000926

M3 - Article

VL - 30

SP - 103

EP - 111

JO - Epidemiology

JF - Epidemiology

SN - 1044-3983

IS - 1

ER -