Casper and PREview Insights: 2023 SJT Experience Survey

Last Updated on February 21, 2024 by Laura Turner

The Health Professional Student Association recently completed its 2023 HPSA/SDN Situational Judgment Test Experience survey. An executive summary is available, and the full report can be separately requested.

Here is a summary of the findings.

Who Took the Survey?

147 aspiring health professionals gave input about their experiences preparing for situational judgment assessments and their early application cycle results. The cohort has higher metrics than cohorts shown in post-test surveys of PREview examinees or expected of Casper test-takers.

• More females than males or gender-non-conforming completed the survey, but males self-reported higher PREview scores (on average). There was no significant difference in the Casper quartile averages between males and females.
• Respondents could self-identify as more than one race or ethnicity. Around 15% of respondents identified themselves as historically underrepresented races/ethnicities (Hispanic/Latinx, Black/African-American/Caribbean, Indigenous/Native American). Their Casper and PREview self-reported scores were generally lower than the entire cohort.
• While over half (58.2%) identified as White/Caucasian, 37% identified as “overrepresented minorities“ (Middle Eastern/North Africans, East Asians, and South Asians). 
• The respondents’ age range was 20 to over 50, with 57% between 22 and 25 years. Casper and PREview performance differed by age range among survey respondents. Younger respondents (up to age 25) self-reported a higher average Casper quartile, while older respondents (26 and older) have a relatively higher average PREview score.
• Survey respondents self-identified as first-generation students (43.1%), Pell Grant recipients (37.3%), socioeconomically disadvantaged (36.3%), non-Native English households (33.3%), and rural backgrounds (31.4%). 17.6% had a diagnosis of being disabled. 12.7% self-identified as being an immigrant, refugee, or asylee. 5.9% served in the U.S. military. Sixty-four respondents described significant impactful experiences that guide their career pursuits.
• Most submitted primary applications from May to July (in the first three months of the 2023 application cycle).

The survey responses focused on prehealth applicants to graduate programs, but we received a few who had applied to residency programs. While most (92.8%) were pursuing medicine, and the remainder were pursuing other careers in dentistry, physical therapy, physician assistant, and veterinary medicine.

The GPA ranges were skewed towards 4.00 for undergraduate and graduate coursework. Among those who disclosed their undergraduate GPAs, 35.9% had 3.90 to 4.00, and 12.0% had 3.80 to 3.89. Reported graduate GPAs (when reported) were similar: 45.1% had 3.90 to 4.00, and 27.5% had 3.80 to 3.89. Furthermore, 121 premed respondents reported their MCAT CARS score; their average was 128 (90th percentile).

Preparing for SJTs

2023 SJT Experience Survey respondents have prepared for Casper and PREview using available online resources. Our respondents further reinforced using online resources like reddit (50%) and Student Doctor Network (37%) for insight into taking these exams. Further resources include professional organization websites (27%), admissions consultant websites (18%), and the testing company’s website (17%). Prehealth advising and mentoring organizations such as the prehealth advising offices (24%), other peers or mentors (13%), or AMSA/SNMA (5%) were consulted less. Most respondents put in some effort to prepare for SJTs. Over 90% took up to 10 hours to prepare for their first SJT, spending no longer than four weeks.

The Acuity Insights Casper Testing Experience

Almost all of our respondents took Casper, and some also took the Duet (63.6%) and Snapshot (11.6%) assessments if required by their programs. (Snapshot will not be offered for the 2024-2025 cycle.) 1.5% of the respondents took the Assessments with accommodations, and 16.0% qualified for a testing fee waiver. 80.2% took the Acuity Insights Suite for the first time, and 87.8% took the 2023-2024 application cycle exams.

Respondents’ scores were skewed towards higher scores, shown by the larger proportion of 4th quartile reported results (Figure 1). 41.2% of respondents self-disclosed a 4th quartile result on their first Casper attempt, 14.5% 3rd quartile, 16.0% 2nd quartile, 21.4% 1st quartile, and 6.9% did not disclose a result. The average quartile was 2.83, higher than the expected average for all test-takers of 2.50. Repeat test-takers had a more normal distribution of their second-attempt scores (Figure 1), but they improved upon their Casper scores or replicated 4th-quartile results (discussed in the full report).

68.7% were invited to an interview at a program that required Casper. Asked if their results accurately represented their competency development, 43.5% of respondents said yes, while 56.5% said no. Test-takers were generally neutral (37.4%) to negative (42.7%) about any program that required the Acuity Insights Assessments. At the time of the survey, 12.2% received an offer or were enrolled in a program requiring the Acuity Insights Assessments.

Some express surprise or confusion about scoring differences between similar tests but generally maintain confidence in their abilities and the alignment of their responses with their true character. Their criticisms collectively highlight concerns about the test’s format, fairness, subjectivity, and capacity to authentically assess the multifaceted qualities and competencies necessary for medical school or similar professional programs.

Figure 1. Survey respondents’ self-reported Casper results

Distribution of Casper quartiles between first and second attempts (proportion out of 100%).

Our Casper test-takers reported getting more third (50th percentile to 75th) or fourth-quartile (75%ile or better) scores on their first (1st) attempt. 41.2% of respondents self-disclosed a 4th quartile result. Those who retook their Casper exam (2nd) had roughly equal representation by all four quartiles.

The AAMC PREview Testing Experience

Most of the 73 respondents took the AAMC PREview test in 2023 (91.8%), with others taking the exam between 2020 and 2022 during pilot testing. 2.8% of the respondents took the exam with accommodations, and 20.8% qualified for a testing fee waiver. Following similar trends as Casper, most took their exam in their bedrooms (53.4%) or a different public room at home (28.8%). Interactions with the testing proctor were reported in 78.1% of this cohort.

Among 67 examinees reporting scores, most applicants received a 6 or higher (59.7%), representing the 4th quartile of all examinees (Figure 2). The overall average score for this cohort was 5.90, which is higher than the average among 2020-2021 test-takers (4.84, Using AAMC PREview Data in Medical Student Selection 2023, Figure 6).

49.3% were invited to an interview that requested a PREview result (either required or recommended). Applicants felt neutral (31.5%) or negative (49.3%) about programs that required PREview; similarly, applicants felt neutral (43.8%) or negative (42.5%) about programs that “recommended” PREview. At the time of the survey, 16.4% of respondents were accepted to a program that required or recommended PREview.

Figure 2. Survey respondents’ self-reported PREview results

Distribution of PREview scores (2023 attempts)

59.7% of respondents had a PREview score of 6 or higher, corresponding to the 4th quartile of PREview test-takers.

Test-Taker Impressions of SJTs

Overall, only 21.1% of 2023 SJT Experience Survey respondents felt SJTs offer useful insights about a future student or resident for admissions or residency decisions. Comments expressed various opinions on SJTs in admissions or residency selection. Some argue that SJTs offer valuable insights into real-world decision-making, ethical judgment, and interpersonal abilities, providing a fair and predictive measure of academic and professional success. Others criticize SJTs for various reasons. Most respondents from historically underserved races and ethnicities were strongly skeptical of the value of SJTs, even if they were selected for interviews.

Applicants felt negatively about schools that run SJTs, though not enough to avoid applying or decline interviews or offers at those programs. As long as becoming a doctor is desirable despite rising testing or application costs, SJTs probably will not deter applicants from applying or affect their decision to enroll. 

Interestingly, respondents with more favorable impressions about SJTs have higher success of getting accepted to their professional programs than those who did not feel SJTs had value in the admissions process. Comments from those who felt their Casper results represented their competency development reflect a positive outlook on the test and confidence in their abilities to handle ethical situations. They mention drawing from personal experiences, ethical reasoning, and the ability to consider multiple perspectives in their responses. Many believe their scores accurately represent their ethical judgment, compassion, and professionalism (discussed in the full report).

Future Directions and Advice to 2024 Premed Applicants

Applicants should anticipate taking an SJT during their application process. These assessments should be scheduled, taking into account deadlines for school-specific applications and any pending admissions exams. Our results suggest that submitting a flawless, desirable primary application (completed by July) is the most important factor in getting an interview and a pre-December offer, so take any required SJTs early to present your completed application for screening. 

In 2024, unlike Casper, PREview examinees can take the exam twice within an application cycle. As we design the 2024 SJT Experience Survey, we recommend treating PREview and Casper like the MCAT: plan to take it once (when you are fully ready) just before the start of the application cycle to give your file a chance at an offer by December.