Candidate Evaluation Using Targeted Construct Assessment in the Multiple Mini-Interview: A Multifaceted Rasch Model Analysis

Research output: Contribution to journalArticle

  • 1 Citations

Abstract

Construct: A 7-station multiple mini-interview (MMI) circuit was implemented and assessed for 214 candidates rated by 37 interviewers (N = 1,498 ratings). The MMI stations were designed to assess 6 specific constructs (adaptability, empathy, integrity, critical thinking, teamwork [receiving instruction], teamwork [giving instruction]) and one open station about the candidate's interest in the school. Background: Despite the apparent benefits of the MMI, construct-irrelevant variance continues to be a topic of study. Refining the MMI to more effectively measure candidate ability is critical to improving our ability to identify and select candidates that are equipped for success within health professions education and the workforce. Approach: Each station assessed a single construct and was rated by a single interviewer who was provided only the name of the candidate and no additional information about the candidate's background, application, or prior academic performance. All interviewers received online and in-person training in the fall prior to the MMI and the morning of the MMI. A 3-facet multifaceted Rasch measurement analysis was completed to determine interviewer severity, candidate ability, and MMI station difficulty and examine how the model performed overall (e.g., rating scale). Results: Altogether, the Rasch measures explained 62.84% of the variance in the ratings. Differences in candidate ability explained 45.28% of the variance in the data, whereas differences in interviewer severity explained 16.09% of the variance in the data. None of the interviewers had Infit or Outfit mean-square scores greater than 1.7, and only 2 (5.4%) had mean-square scores less than 0.5. Conclusions: The data demonstrated acceptable fit to the multifaceted Rasch measurement model. This work is the first of its kind in pharmacy and provides insight into the development of an MMI that provides useful and meaningful candidate assessment ratings for institutional decision making.

LanguageEnglish (US)
Pages68-74
Number of pages7
JournalTeaching and Learning in Medicine
Volume29
Issue number1
DOIs
StatePublished - Jan 2 2017

Fingerprint

model analysis
candidacy
Interviews
interview
evaluation
Aptitude
rating
ability
teamwork
instruction
Health Occupations
rating scale
Health Education
empathy
Names
integrity
Decision Making

Keywords

  • admissions
  • MFRM
  • Multiple mini-interview
  • Rasch modeling
  • student selection

ASJC Scopus subject areas

  • Education

Cite this

@article{2164201ffd1144f6ace08866f84f2956,
title = "Candidate Evaluation Using Targeted Construct Assessment in the Multiple Mini-Interview: A Multifaceted Rasch Model Analysis",
abstract = "Construct: A 7-station multiple mini-interview (MMI) circuit was implemented and assessed for 214 candidates rated by 37 interviewers (N = 1,498 ratings). The MMI stations were designed to assess 6 specific constructs (adaptability, empathy, integrity, critical thinking, teamwork [receiving instruction], teamwork [giving instruction]) and one open station about the candidate\{textquoteleft}s interest in the school. Background: Despite the apparent benefits of the MMI, construct-irrelevant variance continues to be a topic of study. Refining the MMI to more effectively measure candidate ability is critical to improving our ability to identify and select candidates that are equipped for success within health professions education and the workforce. Approach: Each station assessed a single construct and was rated by a single interviewer who was provided only the name of the candidate and no additional information about the candidate\{textquoteleft}s background, application, or prior academic performance. All interviewers received online and in-person training in the fall prior to the MMI and the morning of the MMI. A 3-facet multifaceted Rasch measurement analysis was completed to determine interviewer severity, candidate ability, and MMI station difficulty and examine how the model performed overall (e.g., rating scale). Results: Altogether, the Rasch measures explained 62.84\{%} of the variance in the ratings. Differences in candidate ability explained 45.28\{%} of the variance in the data, whereas differences in interviewer severity explained 16.09\{%} of the variance in the data. None of the interviewers had Infit or Outfit mean-square scores greater than 1.7, and only 2 (5.4\{%}) had mean-square scores less than 0.5. Conclusions: The data demonstrated acceptable fit to the multifaceted Rasch measurement model. This work is the first of its kind in pharmacy and provides insight into the development of an MMI that provides useful and meaningful candidate assessment ratings for institutional decision making.",
keywords = "admissions, MFRM, Multiple mini-interview, Rasch modeling, student selection",
author = "McLaughlin, {Jacqueline E.} and David Singer and Cox, {Wendy C.}",
year = "2017",
month = "1",
day = "2",
doi = "10.1080/10401334.2016.1205997",
language = "English (US)",
volume = "29",
pages = "68--74",
journal = "Teaching and Learning in Medicine",
issn = "1040-1334",
publisher = "Routledge",
number = "1",

}

TY - JOUR

T1 - Candidate Evaluation Using Targeted Construct Assessment in the Multiple Mini-Interview

T2 - Teaching and Learning in Medicine

AU - McLaughlin,Jacqueline E.

AU - Singer,David

AU - Cox,Wendy C.

PY - 2017/1/2

Y1 - 2017/1/2

N2 - Construct: A 7-station multiple mini-interview (MMI) circuit was implemented and assessed for 214 candidates rated by 37 interviewers (N = 1,498 ratings). The MMI stations were designed to assess 6 specific constructs (adaptability, empathy, integrity, critical thinking, teamwork [receiving instruction], teamwork [giving instruction]) and one open station about the candidate's interest in the school. Background: Despite the apparent benefits of the MMI, construct-irrelevant variance continues to be a topic of study. Refining the MMI to more effectively measure candidate ability is critical to improving our ability to identify and select candidates that are equipped for success within health professions education and the workforce. Approach: Each station assessed a single construct and was rated by a single interviewer who was provided only the name of the candidate and no additional information about the candidate's background, application, or prior academic performance. All interviewers received online and in-person training in the fall prior to the MMI and the morning of the MMI. A 3-facet multifaceted Rasch measurement analysis was completed to determine interviewer severity, candidate ability, and MMI station difficulty and examine how the model performed overall (e.g., rating scale). Results: Altogether, the Rasch measures explained 62.84% of the variance in the ratings. Differences in candidate ability explained 45.28% of the variance in the data, whereas differences in interviewer severity explained 16.09% of the variance in the data. None of the interviewers had Infit or Outfit mean-square scores greater than 1.7, and only 2 (5.4%) had mean-square scores less than 0.5. Conclusions: The data demonstrated acceptable fit to the multifaceted Rasch measurement model. This work is the first of its kind in pharmacy and provides insight into the development of an MMI that provides useful and meaningful candidate assessment ratings for institutional decision making.

AB - Construct: A 7-station multiple mini-interview (MMI) circuit was implemented and assessed for 214 candidates rated by 37 interviewers (N = 1,498 ratings). The MMI stations were designed to assess 6 specific constructs (adaptability, empathy, integrity, critical thinking, teamwork [receiving instruction], teamwork [giving instruction]) and one open station about the candidate's interest in the school. Background: Despite the apparent benefits of the MMI, construct-irrelevant variance continues to be a topic of study. Refining the MMI to more effectively measure candidate ability is critical to improving our ability to identify and select candidates that are equipped for success within health professions education and the workforce. Approach: Each station assessed a single construct and was rated by a single interviewer who was provided only the name of the candidate and no additional information about the candidate's background, application, or prior academic performance. All interviewers received online and in-person training in the fall prior to the MMI and the morning of the MMI. A 3-facet multifaceted Rasch measurement analysis was completed to determine interviewer severity, candidate ability, and MMI station difficulty and examine how the model performed overall (e.g., rating scale). Results: Altogether, the Rasch measures explained 62.84% of the variance in the ratings. Differences in candidate ability explained 45.28% of the variance in the data, whereas differences in interviewer severity explained 16.09% of the variance in the data. None of the interviewers had Infit or Outfit mean-square scores greater than 1.7, and only 2 (5.4%) had mean-square scores less than 0.5. Conclusions: The data demonstrated acceptable fit to the multifaceted Rasch measurement model. This work is the first of its kind in pharmacy and provides insight into the development of an MMI that provides useful and meaningful candidate assessment ratings for institutional decision making.

KW - admissions

KW - MFRM

KW - Multiple mini-interview

KW - Rasch modeling

KW - student selection

UR - http://www.scopus.com/inward/record.url?scp=84980006569&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84980006569&partnerID=8YFLogxK

U2 - 10.1080/10401334.2016.1205997

DO - 10.1080/10401334.2016.1205997

M3 - Article

VL - 29

SP - 68

EP - 74

JO - Teaching and Learning in Medicine

JF - Teaching and Learning in Medicine

SN - 1040-1334

IS - 1

ER -