What are students encountering as they take situational judgment tests? This is the first of two articles summarizing the findings from the 2022 Health Professional Student Association (HPSA) Situational Judgment Test Experience survey. The questions focused on students’ experiences with situational judgment tests such as Acuity Insights Casper and AAMC PREview during their health professional school admissions cycles.
Background: What Are Situational Judgment Tests?
Situational Judgment Tests (SJTs) describe various assessments that gauge a candidate’s behavior when making common decisions in the desired job. Rather than asking a candidate questions about situations they have dealt with (such as “think about a time when you had to manage a conflict”), SJTs present situations and ask the candidate to think about possible approaches and solutions methodically.
Historically, post-test evaluations show that most perceive SJTs to be very fair. Results from well-designed SJTs also are much less affected by racial, demographic, or socioeconomic factors compared to performance on content-based exams. Whether administered through a computer or in person, well-designed SJTs are received positively as candidates feel they address plausible situations they will encounter. Hiring managers appreciate the insight into how well-aligned candidates are to how the company handles personnel management and conflicts. Many businesses have included SJTs in their hiring process, especially for executive and managerial positions. Furthermore, SJTs are commonly part of the selection process in graduate and medical programs in other parts of the world.(1)
Health professional schools and organizations have been working for some time to develop a way to test an applicant’s soft skills and professionalism. The flexibility of delivering MMI-like assessments online has accelerated interest since the COVID-19 pandemic. The Acuity Insights (formerly Altus) Casper assessment has been around for at least a decade and is used widely among Canadian health professions programs. Since its Admissions Initiative, the AAMC has been developing the PREview exam, which moved out of pilot testing in 2022.
However, many have expressed concerns about how such assessments would factor in admissions decisions and whether such assessments could hurt disadvantaged applicants. This summer, the National Association of Advisors for the Health Professions formed a Testing Task Force (per an association communication, 2022) to further advocate for equitable access, data, and justification of the usage of these exams in the admissions process.
The Health Professional Student Association (HPSA) hosts a Situational Judgment Test forum with our partner PrepMatch. In addition to posting sample scenarios and questions to prepare for Casper, many have posted their questions and experiences with taking the SJTs. Students have posted their concerns about how their scores are considered in the application process; little insight has been shared by the schools that use the SJTs. Some schools claim it will be a “plus factor” when considering a candidate for a waitlist or offer decision, while others have not confirmed using Casper/PREview scores/SJTs as a means of weeding out a large number of applicants during screening before an interview. Small study groups are being formed to support peer-to-peer SJT preparation.
The 2022 HPSA Situational Judgment Test Experience Survey
To better understand these experiences, HPSA sent an anonymous poll to users of its resources, such as the Student Doctor Network (SDN) forums, from September to October 2022. The goal of the survey was to get quick insights into the experience of SJT test-takers over the past few cycles, including the 2020 and 2021 cycles that were affected by the COVID-19 pandemic. The insights from the respondents will hopefully help applicants and advisors better understand the potential effects of the inclusion of SJTs into the process and seek strategies to better prepare future applicants for these assessments. Our survey was conducted independently, without support from Acuity Insights, AAMC, or other organizations, to mitigate any potential conflict of interest.
How was the survey conducted?
Calls to complete this survey (via Google forms) were made throughout September and October 2022 through newsletters, SDN forum posts, and posts in other internet forums. No compensation was offered as an incentive for completing the surveys. The anonymous survey was designed to quickly compile information about the examinees’ experience to inform content for the HPSA Situational Judgment Skills Workshop, conducted at the 2022 University of California Davis Prehealth Conference.
Who responded to the survey?
We received responses from 86 participants, and the demographics are shown in Table 1. Almost all had taken the Acuity Insights Casper exam, but many also had taken the AAMC PREview exam. While most participants were pursuing admission to medical school (MD/DO), we also received responses from students seeking entry to veterinary medicine, dental medicine, and other graduate health programs such as physician assistant or physical therapy. Respondents were mostly post-baccalaureates, either not currently in school or taking postbac or graduate courses; others were professional students or trainees. A relatively small number of current undergraduate students were captured in this survey.
Table 1-1. Survey response demographics (n=86)
Gender
| Female | 61.2% |
| Male | 36.5% |
| Gender non-conforming | 3.3% |
Race/Ethnicity
| White/Caucasian | 67.9% |
| Asian/Pacific Islander/Hawaiian | 20.2% |
| Black/African-American/Caribbean | 14.3% |
| Latino/Hispanic | 4.8% |
| Middle Eastern/North African | 4.8% |
| Native American/Indigenous | 0.0% |
Special Groups
| Socioeconomically disadvantaged | 31.4% |
| Pell grant received | 29.1% |
| Disabled | 14.0% |
| Military or veteran | 5.8% |
Current Status
| Undergraduate student | 11.6% |
| Postbac not a student | 32.6% |
| Postbac/Graduate student | 30.2% |
| Professional student or trainee | 25.6% |
Tests Taken
| Altus Casper | 97.7% |
| AAMC PREview | 41.9% |
| AAMC VITA | 9.3% |
| Kira Talent | 9.3% |
Why Take the Exam?
| MD/DO (medicine) admissions | 83.7% |
| VMD/DVM (veterinary) admissions | 8.1% |
| DMD/DDS (dental) admissions | 4.7% |
| Postbac/Master’s Program admissions | 4.7% |
| PA/PT admissions | 2.3% |
Application Cycles
| 2023 entry | 67.4% |
| 2022 entry | 23.3% |
| 2021 entry | 14.0% |
| 2020 or before entry | 8.1% |
| Plan to take | 3.5% |
Most respondents self-identified as racial-majority applicants (White or Asian); minority races or ethnicities were represented in relatively smaller proportions (no Native American, 4.8% Middle Eastern/North African, 4.8% Latino/Hispanic, 14.3% Black/African/Caribbean). However, 31.4% self-described as socioeconomically disadvantaged, and 29.1% received Pell Grants during their education. 14.0% self-identified as disabled applicants, and 5.8% were military/veterans.
Respondents were primarily made up of current applicants in the 2023 cycle (67.4%), but some also were applicants in previous years. A small number (3.5%) had not begun the application process and planned to take an SJT as part of their process.
How did examinees prepare for SJTs?
Most respondents followed the advice on the Acuity Insights Casper and AAMC PREview websites for logistical planning to take the exam: they completed their exams at home on personal computers with a single monitor (no larger than 24 inches). A few took their exams at a university facility, like a library or available classroom space, and a very small number took the exam at work. Most used wired or wireless headphones, though PREview apparently discouraged the use of headphones. Others used the internal computer speaker, while others used an external speaker connected to the computer.
Most received their orientation to the testing format using resources from the test-makers (86.4%). Others also relied on internet videos produced by influencers on YouTube, Instagram, TikTok, or other social media (22.2%). These influencers also gave examinees strategies to excel on Casper or similar SJTs (48.1%). Some used free resources from the SDN/PrepMatch SJT forum or other discussion chats or websites (30.9%) to practice or gain feedback on simulated questions. Only 8.6% attended free online workshops or webinars hosted by Acuity Insights, AAMC, other organizations, or prehealth advising offices on SJTs. The range of hours (between the 25th and 75th percentiles) spent preparing for the exam was between 2 and 8 hours (median of 4 hours).
Want to learn more about preparing for Situational Judgment Tests? Here are some additional resources from the SDN article archives:
What challenges did examinees face taking the SJTs?
Respondents were asked to identify the challenges they had. 73.3% said they had difficulty with managing the time constraints, especially with Casper’s five-minute timing to answer three questions for each given situation. Other challenges that were mentioned include the availability of free practice resources (36.7%), connection and proctoring challenges, especially with PREview (28.3%), challenges with displaying or reading situations (15.0%), and lack of clarity in selecting an answer notably on PREview (11.7%).
When prompted for more detail, many respondents felt that Casper’s timing protocols reward individuals who could completely answer the three situational questions and that a successful result was more related to typing speed than attribute assessment. Despite assurances by Acuity Insights, no evidence has convincingly contradicted this widely-held view among applicants. Many who had retaken Acuity Insights Casper over multiple application cycles were frustrated as their self-reported results changed after each cycle or between different test forms (e.g., “second quartile first attempt, first quartile second attempt”), further undermining any applicant confidence in the true reliability of the result. Since programs that accept Casper results fail to disclose how they consider these scores from re-takes, many test-takers expressed anxiety that any poor performance would undermine an otherwise successful application.
PREview test-takers expressed dissatisfaction with the “invasiveness” of proctoring, especially when yielding control of one’s personal computer to the remote proctor; one respondent even wanted to take the exam at a secured center (like Prometric) instead of at home if this eliminated the need for a proctor and a scan of one’s bedroom before taking the exam. The multiple-choice test format made some examinees feel unable to demonstrate their personality or ethical decision-making competencies.
Overall, applicants were frustrated that their complex pre-professional competency development was reduced to a normalized score or quartile range. Many disclosed they would avoid considering any schools that required Acuity Insights Casper or AAMC PREview as they feel taking these exams was more a liability than a positive. Regardless, as SJTs become more widespread among medical schools and graduate/residency programs, it seems inevitable that applicants will need to adapt.
In the next part of this article, we will look at the self-disclosed results from the respondents to see if there are patterns that show how SJT results have been used in admissions decisions.
NOTE: Since the original writing of this article, Altus Assessments (which runs the Suite of Casper, Duet, and Snapshot assessments) has rebranded to become Acuity Insights. This article has been updated, but many resources still reference Altus.
For inquiries on the Situational Judgment Test Experience survey, contact Dr. Emil Chuck (emilchuck@hpsa.org).
About HPSA
HPSA is a non-profit 501(c)3 organization that has supported individuals striving for health professional careers with donor-supported tools, resources, and community for over 20 years. HPSA is the publisher of Student Doctor Network articles, courses, and webinars. We welcome partnerships and collaborations with other non-profit organizations that share our mission.
References
Emil Chuck, Ph.D., is Director of Advising Services for the Health Professional Student Association. He brings over 15 years of experience as a health professions advisor and an admissions professional for medical, dental, and other health professions programs. In this role for HPSA, he looks forward to continuing to play a role for the next generation of diverse healthcare providers to gain confidence in themselves and to be successful members of the interprofessional healthcare community.
Previously, he served as Director of Admissions and Recruitment at Rosalind Franklin University of Medicine and Science, Director of Admissions at the School of Dental Medicine at Case Western Reserve University, and as a Pre-Health Professions Advisor at George Mason University.
Dr. Chuck is an expert on admissions, has been quoted by the Association of American Medical Colleges (AAMC), and has volunteered as a workshop facilitator on holistic admissions for the American Dental Education Association (ADEA). He has also contributed to the essay collection The Perfect Doctor by Pager Publications and has developed competency-based rubrics supporting holistic review.