Developing, Testing, and Reviewing Measures in Implementation Science

Project: Research project


UNC will be responsible for carrying out all research activities associated with developing and testing the
psychometric properties of three measures of implementation outcomes. These research activities include:
? Item Generation: We will use deductive methods to generate 10 items for each construct whereby we use the construct definitions and nomological network to ascertain whether items from existing measures, obtained through SIRC and updated through literature searches, adequately capture the theoretical content of the construct. While searching the literature for items, we will also search for evidence-based strategies for increasing acceptability, appropriateness, and feasibility.
? Substantive Validity: We will recruit 30 Ph.D.-trained implementation researchers and 30 implementation-experienced mental health professionals to participate in a web-based survey. Study
participants will complete a web-based survey in which the 3 constructs are defined at the top and the 30 items (10 per construct) are listed below in random order. Participants will assign each item to the
construct(s) that they perceive the item measures. They will rate their confidence in each assignment from 0% for “not at all confident” to 100% for “extremely confident.” We will use one-sample, one-tailed
t-tests and Light’s Kappa to assess substantive validity, inter-rater agreement, and discriminant content validity. We will formally assess the validity of the implementation outcome conceptual model using
confirmatory factor analysis (CFA).
? Structural Validity: We will use a 3 x 2 factorial between-subjects design in which we manipulate the three constructs at two levels (high versus low) in vignettes depicting a mental health clinic about to adopt a new assessment tool and the reactions of mental health professionals working there. We will
recruit a convenience sample of 240 mental health counselors belonging to the American Mental Health Counselors Association (AMHCA). In a web-based survey, participants will read one of six randomly
assigned vignettes and, using the items that survive the substantive validity test and a 5-point ordinal response scale that ranges from “disagree” to “agree,” participants will rate the acceptability, appropriateness, and feasibility of using the new assessment tool as they believe the mental health
professionals at the clinic would rate it. We will use CFA to test structural validity and ANOVA to test known-groups validity.
? Reliability: We will use a variant of a 3 x 2 factorial within-subjects design in which half of study participants in the structural validity test will be randomly assigned to receive the same vignette they received earlier; the other half will be assigned to receive the exact opposite vignette they received earlier. We will use the same data collection procedures and measures that we used in the structural validity test. We will assess test-retest reliability using paired t-tests and Pearson correlation
coefficients. Scales demonstrating non-significant paired t-test and correlations greater than .70 will be considered reliable. We will assess sensitivity to change using repeated measures ANOVA with one
between-group factor at two levels (same vignette versus opposite vignette) and one within-group factor at two levels (first versus second survey administration).
? Predictive Validity: Using a list of Community Clinical Oncology Program (CCOP) physicians provided by NCI/DCP and data from the American Medical Association (AMA) Physician Masterfile, we will draw a random sample of 600 CCOP physicians, stratified by CCOP organization and oncology specialty
(hematology, medical, and radiation). We will survey physicians to obtain data on their perceptions of acceptability, appropriateness, and feasibility of an innovative clinical practice embedded in a Phase II/III clinical trial. We will obtain data on physicians’ adoption and use of the clinical trial from the CCOP
Effective start/end date4/1/153/31/18


  • Trustees of Indiana University


Medical Oncology
Mental Health
Analysis of Variance
Mental Health Associations
Phase III Clinical Trials
Phase II Clinical Trials
American Medical Association
Reproducibility of Results
Statistical Factor Analysis
Research Personnel
Outcome Assessment (Health Care)
Surveys and Questionnaires
Clinical Trials