In Part 1, I introduced you to some of the results from the 2022 SJT Experience Survey, sponsored by the Health Professional Student Association. This anonymous survey captured input from 86 respondents who took a situational judgment test (SJT) and were willing to reflect on their experience. For this article, we focus on their self-disclosed results to see if there are any inferences on how the SJTs have been used with admissions decisions in the past.
How well did the examinees do?
Respondents self-disclosed their exam results for Acuity Insights (formerly Altus) Casper and AAMC PREview. The breakdown of results from the SJT Experience survey is summarized in Table 2-1.
Table 2-1: Self-disclosed SJT results
Altus (Acuity Insights) Casper
| Quartile 1 | Quartile 2 | Quartile 3 | Quartile 4 | Quartile Average | |
|---|---|---|---|---|---|
| All (n=68) | 14 | 15 | 18 | 21 | 2.6 |
| Postbac non-student (n=25) | 7 | 4 | 9 | 5 | 2.5 |
| Postbac/ Graduate student (n=23) | 3 | 6 | 5 | 9 | 2.9 |
| Professional students (n=12) | 0 | 4 | 4 | 4 | 3.0 |
| Professional MD/DO (n=9) | 0 | 4 | 3 | 2 | 2.8 |
| Professional not MD/DO (n=3) | 0 | 0 | 1 | 2 | 3.7 |
| URM (n=13) | 3 | 4 | 4 | 2 | 2.4 |
AAMC PREview
| Quartile 1 | Quartile 2 | Quartile 3 | Quartile 4 | Quartile Average | |
|---|---|---|---|---|---|
| All (n=29) | 3 | 5 | 7 | 14 | 3.1 |
| Postbac non-student (n=13) | 3 | 2 | 3 | 5 | 2.8 |
| Postbac/ Graduate student (n=12) | 1 | 1 | 3 | 7 | 3.3 |
| Professional students MD (n=4) | 0 | 2 | 1 | 1 | 2.8 |
| URM (n=5) | 1 | 1 | 3 | 0 | 2.4 |
68 respondents disclosed their Casper quartile. The distribution of Casper scores was fairly even across all quartiles (the average quartile is 2.6). While postbac non-student applicants also showed an even distribution of results (average quartile 2.5), postbac/graduate student applicants showed a slight skew towards higher quartiles (2.9). Among the professional students who disclosed their Altus Casper results, no professional students reported a first-quartile result. Interestingly non-medical professional students only reported scores in the upper two quartiles. A larger number of results from current professional students would be needed to see if this observation is just coincidental, especially as Acuity Insights does not recommend that schools use Casper results as a cutoff.
20 respondents disclosed AAMC PREview quartile results. Unlike the Acuity Insights Casper results, we observed a skew towards the higher quartiles (average quartile 3.1). Although postbac non-students reported a slight skew (2.8), postbac/graduate students showed a higher likelihood for 4th-quartile scores (3.3). Results from four professional students also showed a similar skew (2.8). These results contrast with AAMC pilot data that show a normal performance curve among test-takers, consistent with well-designed psychometric exams.
How well do Casper and PREview assess your competencies?
27 respondents disclosed both Casper and PREview results in the survey (Table 2-2). While Casper scores were evenly distributed (2.4), PREview scores were skewed (3.1). These data further confirm that most MD/DO applicants and matriculants should be able to perform well on PREview. In contrast, applicants with low Casper scores may be relegated for additional screening or placed on hold.
| AAMC PREview (Average Quartile 3.1) | |||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Altus Casper (Average Quartile 2.4) |
|
How do underrepresented minority applicants perform?
Most SJTs show attenuated differences in scoring due to racial differences, so we were interested in any differences in Altus Casper and AAMC PREview performance among underrepresented minority respondents (Black/African/Caribbean and Latinx). While 13 URM applicants’ Casper results showed an even distribution comparable to the entire cohort (average quartile 2.4), five URM applicants’ PREview results showed a skew towards lower quartiles (average quartile 2.4) compared to the overall cohort (3.1). Four took both Casper and PREview: two had Q1 Casper results but Q3 PREview results; one had a Q3 Casper and a Q2 PREview, and one had a Q4 Casper and a Q3 PREview. We hope a subsequent survey can provide more insights.
What are some takeaways?
While a larger survey that encompasses multiple health professions (including residency selection) and more diverse applicants on the SJT experience needs to be developed, there are a few insights the SJT Experience survey provided about the strategies applicants have taken to prepare for these SJTs and possible connections with admissions decisions.
- No professional students disclosed a first-quartile result in this survey. A larger sample is needed to determine if this is sampling bias or a bonafide result of a consensus of admissions policies. This observation is particularly noticeable as non-medical professional students are only represented by third- and fourth-quartile Altus Casper respondents. In contrast, anecdotally on the forums, a few students have claimed they received an acceptance despite a first-quartile score. Transparency of the distribution of SJT results among matriculants must be routinely disclosed by all health professions admissions committees and offices along with GPA and standardized exam metrics.
- Premedical applicants tend to have high scores on PREview. Interestingly, applicants currently completing coursework in postbac or graduate programs report doing very well on PREview compared to postbac non-students. Presuming that formal postbac programs do not have specific resources that help applicants with the PREview exam, does the PREview exam favor those exposed and immersed in academic/medical educational culture by being advised and taught by current healthcare professionals? The influence of immersion in a high-stakes but supportive academic environment on PREview results should be further investigated.
- Applicants and advisors need more insight into how SJT results affect admissions decisions. As shown by the Casper and PREview score distributions in our sample, applicants and advisors need to know how a PREview score can be a “plus factor” if the testing population already scores rather highly. Additionally, admissions professionals should better describe how lower-quartile applicants are reviewed as part of holistic review. Data showing that matriculants have a similar even distribution of Casper scores would affirm statements that Casper is used as one consideration in a holistic process and not used as a cutoff.
- Based on anecdotes from those who took multiple test forms throughout an application cycle, applicants are skeptical that these SJTs measure intrinsic competency development with any precision. While SJTs can be designed to assess specific competencies in multiple scenarios, the lack of a bonafide breakdown of scores linked to explicit competencies remains problematic. Acuity Insight Casper scores seem arbitrary and dependent on the population of candidates taking the specific test form at a specific time. In contrast, AAMC PREview may reward those who can navigate the rules of academic and medical culture.
- More resource materials must be made available to all prospective test-takers, including those who are socioeconomically disadvantaged. Respondents cite the lack of freely-available resources and the cost of taking these exams that discourage them from applying to schools that use SJTs in their selection process. More transparency in scoring rubrics should dispel rumors that the exam outcomes measure typing ability or oral English fluency without a noticeable accent or stutter. Additionally, more research should be conducted that is absent of potential conflicts of interest to show these tests are an accurate gauge of competency development and to study the impact these exams have on subsequent issues of trainee professionalism. With the possibility that these exams may be used for resident selection in the future, a longitudinal study of performance on these exams would provide evidence that these exams truly measure competency development.
How can you prepare for the “unpreparable”?
At the time of this article’s publication, programs are evaluating the impact of Casper and PREview exams on the admission of future classes of students. More programs are expected to use these exams on a “research” basis to gain confidence that fewer students with academic or professional issues are admitted while still meeting goals for a diverse and inclusive learning community. We have compiled some tips to help you strengthen your situational awareness and knowledge of professional expectations of conduct.
- Through your life experiences, develop your empathy and advocacy with social justice issues affecting healthcare.
As one of our posters, acstylin, points out: “The best way to prepare is to have some life experiences with people that disagree with you, debate a heated/sensitive topic and defend the side you don’t agree with, immerse yourself in cultures that make you uncomfortable, etc.” As more programs seek applicants with strong advocacy/leadership skills and a track record as a change agent toward better health equity or social justice, gaining a better understanding of ethical and empathetic behavior in the context of diversity and belonging will help all aspiring professionals. Students should commit themselves to such activities to hone one’s awareness of bigger-picture issues and external influences on social determinants of health.
Lior Krimus of PrepMatch agrees: “One thing I always recommend is to write down a list of scenarios in your life involving the following: conflicts (professional/personal), scenarios reflecting your strengths, scenarios reflecting your weaknesses, list of failures, times you showed good communication, times you showed good collaboration, and times your collaboration didn’t go as planned. Having these scenarios written beforehand really helps with Casper and the MMI.”
For example, as more restrictions to provide healthcare access to LGBTQ+ community members have been implemented, applicants should pursue opportunities to become stronger allies. HPSA has developed a free Google Classroom curated resource on Treating Transgender Patients that includes over nine hours of audiovisual materials from transgender patients and health professionals on making the healthcare system more inclusive, as well as quality journalistic articles that address issues of concern. - Reflect on how you approach – or avoid – conflicts.
By understanding your own tendencies to react to situations, you can better understand how to evaluate effective or ineffective outcomes to your responses. This insight will help you as you further hone your preparation for the question formats of each SJT or MMI question you could face. HPSA has developed a situational awareness workshop that debuted at the 2022 University of California Davis Prehealth Conference, and more is available in the HPSA Becoming a Student Doctor course.
- Ask schools about how they factor SJT and online MMI results into admissions decisions.
Find out if the schools you are considering will require an SJT or rely on an online platform such as Kira Talent or SortSmart to interview candidates. By the spring, programs should finalize their participation with Casper, PREview, or another system for the upcoming cycle. Frequent contact with admissions counselors during spring recruitment will help determine whether you need to prepare for these exams during the summer or fall.
- Schedule extra time to prepare for different SJT formats.
Most test-takers will take a few hours to get oriented to the exam format, review practice questions, and secure appropriate technology and support to schedule and take the exam. Seek out information about processes to request accommodations or fee assistance before scheduling your exam. Note that most respondents did not go out of their way to take the exams and most likely prepared for no more than a month before the exam date.
Scheduling the exam will depend on the applicant’s school list and any other test preparation. Many only think about taking this exam after they receive results from high-stakes exams like the MCAT, DAT, or GRE. For many premedical students, while Casper dates may be available as early as April, most will likely take the exam between July and September as they discover a school of high interest requires Casper or PREview. Hundreds of students passively withdraw from applying to schools requiring these tests by not taking these exams, especially after they receive sub-optimal MCAT/DAT score results. The only advantage to having Casper scores in hand (taking the exam before July) is to have a complete file for early screening and consideration for an interview. Still, one must also submit all supplemental information (secondary application, high-stakes exam result, possibly letters of recommendation) early to take full advantage. There is little information on whether taking any of these exams near the end of the testing window affects your chances for an interview or admission.
For those pursuing residency, you should prepare to take the Altus Suite if you are interested in anesthesiology, internal medicine, OB/GYN, ophthalmology, and surgery (more may be added after this article has been published, so check the Take Altus website). One should follow up with specific programs requiring you to take the Acuity Insights assessments and submit your responses concurrent with your ERAS application so that all your materials are received by the program directors promptly. - Use free practice resources.
PrepMatch, a student/trainee-run organization that provides peer-assisted Casper preparation, has provided free resources for the last few years. Both written and video scenarios are available through the PrepMatch website, and some sample written scenarios can be accessed on the SDN forums. We also have an extensive article archive on Casper with strategic tips from other influencers or admissions consultants.
- Stop overthinking and be yourself.
While you can view dozens of influencer videos on how to “crush” these SJTs, remember that the best answer is often the answer you come up with. Aside from suggestions to improve your typing speed on Casper or to practice responding to behavioral/situational questions in mock interviews, the best performance you can have is the one that reflects your authenticity when handling the conflict. Based on what the survey suggests, PREview may include more situations you have experienced within an academic setting. In comparison, Casper may present more “real world” situations when it comes to working with others; this is shown by the more diverse representation of its raters [https://altusassessments.com/rating-casper-infographic/p/1/]. As long as you are familiar with your own timing and the testing formats, answer the questions like a supplemental interview, and hope for the best.
For residency programs that require completion of the Acuity Insights Duet and Snapshot assessments, a thorough self-reflection of your purpose in pursuing the specific specialty and expectations of what you need to be successful should guide your preparation.
Finally, the utilization of SJTs with health professional education needs to become more transparent. Specifically: - Admissions, student affairs administrators, and residency administrators must publish objective studies on the reliability of SJT assessments throughout professional training to confirm competency mastery.
Because all health professions programs assess professionalism as a condition of student promotion or graduation from residency, each program should be able to map specific professional competencies with results from the SJTs, including tracking any gradual improvement on attributes that contribute to EI. Many published studies co-authored by Acuity Insights (formerly Altus) suggest that there is a relationship between Casper results and student success outcomes [https://altusassessments.com/resources/case-studies/]. Similarly, residency directors should be able to measure satisfactory completion of entrustable professional activities to show improvement of EI from the “baseline” SJT. There should ultimately be a determined threshold for SJT score with a candidate’s ability to complete residency training and manifest the ACGME (or similar professional) competencies successfully.
To date, no known long-term objective studies have been conducted that show that the SJTs truly measure competency mastery. Many faults with the aforementioned studies have been pointed out. True reproducibility and validity have not been demonstrated on an individual basis, which ultimately is how the scores are presented for candidate selection. A well-designed meta-analysis would also inform residency program directors and professional school student advisors of the usefulness of these SJTs in their selection process, especially if their own programs have used SJTs to admit and enroll students and trainees.
What is more remarkable is that there is little known about how Casper has been used for student selection at non-medical health professions programs after so many years of using it. How Casper results influence student selection in Canadian programs that have had years of using Acuity Insights assessments remains a carefully guarded secret. Furthermore, no independent peer-reviewed articles have published insights into how Casper is used in US non-medical health professions programs; this includes over 50 physician assistant programs, 26 physical therapy programs, 12 dental programs, 11 veterinary medical programs, 8 speech-language pathology programs, and 3 anesthesiology assistant programs. Most successful examples are digested in old Altus blog posts or archived presentations.
Furthermore, claims that one cannot be “coached” to perform better on these SJTs contradict the consensus that emotional intelligence (EI) – consisting of self-awareness, self-regulation, social awareness, and social skills – can be improved. SJTs could be used as an assessment tool that rewards those who benefit from effective interventions or remediations to improve these qualities throughout professional education. Designing studies that correlate SJT performance with validated instruments that assess EI throughout the educational and training continuum should give greater insight into the effectiveness of the SJTs in measuring important professional competencies.
With the impending ban on race-factored admissions anticipated and the onset of more “pass/fail” scoring of board exams, the adoption of these SJTs will likely become widespread as a measure of non-cognitive attributes that are desired in holistic review. Future applicants – including those seeking graduate professional training – should see these assessments as a regular part of their future. Removing any doubt that these exams truly measure competency mastery will give confidence to admissions committees, residency program directors, employers, and thousands of future health professionals that an unfortunate low score will not derail one’s career trajectory.
NOTE: Since the original writing of this article, Altus Assessments (which runs the Suite of Casper, Duet, and Snapshot assessments) has rebranded to become Acuity Insights. This article has been updated, but many resources still reference Altus.
For inquiries on the SJT Experience survey, contact Dr. Emil Chuck (emilchuck@hpsa.org).
About HPSA
HPSA is a non-profit 501(c)3 organization that has supported individuals striving for health professional careers with donor-supported tools, resources, and community for over 20 years. HPSA is the publisher of Student Doctor Network articles, courses, and webinars. We welcome partnerships and collaborations with other non-profit organizations that share our mission.
Emil Chuck, Ph.D., is Director of Advising Services for the Health Professional Student Association. He brings over 15 years of experience as a health professions advisor and an admissions professional for medical, dental, and other health professions programs. In this role for HPSA, he looks forward to continuing to play a role for the next generation of diverse healthcare providers to gain confidence in themselves and to be successful members of the interprofessional healthcare community.
Previously, he served as Director of Admissions and Recruitment at Rosalind Franklin University of Medicine and Science, Director of Admissions at the School of Dental Medicine at Case Western Reserve University, and as a Pre-Health Professions Advisor at George Mason University.
Dr. Chuck is an expert on admissions, has been quoted by the Association of American Medical Colleges (AAMC), and has volunteered as a workshop facilitator on holistic admissions for the American Dental Education Association (ADEA). He has also contributed to the essay collection The Perfect Doctor by Pager Publications and has developed competency-based rubrics supporting holistic review.