The following Twitter post explains the science and statistical analysis behind the USMLE compromise by Nepali candidates.

“If low-scoring examinees are more likely to answer a question correctly than high scorers, it likely means they’re responding to something in the question stem other than what the construct is intended to assess.

So what you do is look at each examinee’s performance on the compromised items, and compare it to their performance on not-known-to-be-compromised items. It should be the same. But if certain examinees systematically perform better on compromised items – you’ve found a cheater.

This procedure was first described after the Medical Council of Canada had an exam breach on the MCCQE Part 1 exam in 2004.”

I’ve gotten DMs asking “How could the USMLE prove they cheated? High scores alone don’t prove that!”

-and-

“How do I know my high score won’t be flagged as evidence of cheating just because of where I took the test?”

Lemme explain how I think these cheaters were caught.

(🧵) https://t.co/W1iGCzF1bY

— Bryan Carmody (@jbcarmody) February 1, 2024