On day one of a typical Stats 101 course, your subject’s introduction likely revolves around these foundational concepts: the pursuit of statistical significance and generalizability. We’re taught, fifteen minutes in, that good data must meet specific criteria, including: an n > 30, p-value < 0.05, a sample that is representative of its population. In essence, we first learn that in statistical analysis, if data does not check these boxes, it may be “bad data.”
We know this well as assessment professionals in higher education. As we report out findings to inform decisions, this criteria becomes especially pertinent. Faculty and colleagues ask, “Is this data statistically significant? Is it generalizable?” These are important questions, but at times, may overlook experiences and voices of students whose identities are not easily captured by traditional statistical methods.
Intersectionality, a term coined by Kimberlé Crenshaw (1989), has been a critical lens through which we examine the intersecting and compounding aspects of identity and disadvantage. It recognizes that the experience of many individuals are shaped by the complex interplay of factors such as race, gender, sexuality, and more. That said, while intersectionality has been studied for decades, there lies a significant gap in how this lens is employed in research, especially when statistical significance is the gold standard.
Navigating the Nuances of Intersectional Respondents: An Example
To better understand potential harm on intersectional students within our work, let’s dive into a hypothetical, borrowing data from my institution of work, the University of Colorado, Boulder.
As CU Boulder looks to learn more about first-year students’ co-curricular experience and wellbeing, my office – the Student Affairs’ Office of Planning, Assessment, and Data Analytics – releases an annual, campus-wide survey to all first-year students. In Fall 2022, about 19.7% of students at CU Boulder were first-year students, making up approximately 7,116 students (“Overall Enrollment Profile,” 2022). Assuming a high response rate of 20%, our sample size dwindles down to an expected 1,423 respondents, still a robust number for statistical analysis and generalizability.
Within the Fall 2022 data, Indigenous students accounted for 1.4% of the total CU Boulder population (2022). Here’s where the challenge emerges. Presuming our sample is representative of the campus population, our 1,423 respondents would yield a mere 19 Indigenous students. Suddenly, our sample size seems inadequate in drawing meaningful conclusions about this specific group.
Now, let’s factor in another dimension of identity. According to data from a 2020 Association of American Universities (AAU) study, approximately 17% of undergraduate and graduate students identified as gay, lesbian, bisexual, asexual, queer, or questioning (“Report on the AAU Campus Climate Survey”). When we apply this statistic to our Indigenous first-year population (n⋍19), we are left with less than approximately 4 Indigenous, LGBTQ+ students.
At this moment, a question arises: Why should CU Boulder invest resources in studying the experiences of just 4 students at a university of ~36,000 students? Moreover, protecting student anonymity is paramount, and on the off-chance of identifiability, public reports cannot produce visuals that regard only four individuals. Additionally, referring students to resources via survey data may be complex. We now come face to face with the heart of this issue.
Despite these challenges, Indigenous, LGBTQ+ students deserve our attention. Mental health outcomes, alarming suicide rates, and higher levels of LGBTQ+ hate experienced by these students are all serious and real concerns (Anderson, 2021; Cowing, 2021; Zadvirna, 2023). It’s a complicated and delicate journey to advocate for a small number of students within a vast and diverse university population, though necessary. It’s a reminder that good data must also consider the unique experiences and identities that may not fit neatly into traditional frameworks.
Intersectional Stigma
One critical aspect of this conversation, and certainly within our prior example, is the concept of intersectional stigma (Turan et al., 2019). Intersectional stigma refers to discriminatory outcomes that intersectional identities face, often through negligence (2019). In the case of statistical analysis, intersectional stigma infiltrates and informs our assumptions and decision-making, potentially harming students in the university system.
Actionable Strategies for Inclusive Assessment
Despite clear barriers, there are actionable ways that assessment can help us bridge the gap between supporting our intersectional students, as well as our data standards.
In Planning: 1) Find ways to gather intersectional data, such as conducting focus groups, longitudinal analyses, or benchmarking with similar schools or nationwide data 2) Collaborate and brainstorm with offices aligned with these identities to better support and market toward these students.
In Data Collection: Question your methods of data collection. Critically analyze what types of data your office prioritizes, and by what means they typically collect data. While quantitative data typically receives greater acclaim in research and assessment, collecting qualitative, or mixed-methods data is often key towards understanding our smaller populations of interest (Hernández, 2015).
In Survey Building: 1) Add links to resources for support 2) Include a confidentiality disclaimer that enables outreach if necessary 3) Include (few) strategic, intentional open-response questions that allow for students to share unique experiences 4) Ensure that your survey is culturally competent, free of jargon, and accessible to all respondents.
In Data Analysis: 1) Utilize your confidentiality disclaimer as a tool for outreach 2) Pay attention to experiences and themes of intersectional students during qualitative coding.
In Statistical Testing: Explore and use alternative tests to see a bigger picture from your data, such as latent profile and latent class analysis (Bauer, 2022).
In Data Visualization: Disaggregate and filter data by identity. Tools like Power BI and Tableau can help with this process. Create visuals that are accessible in color and font size.
In Reporting: If you can't publish visuals in order to protect intersectional students (ex. If disaggregation poses harm to anonymity), you can still use your results as a means of outreach. Share data that highlights specific issues affecting intersectional groups with relevant campus offices to support these students. And, a reminder: just because data lacks statistical significance does not mean it should be hidden.
Conclusion
The above list of strategies is not nearly exhaustive, but it is a step in creating a more equitable and inclusive approach within assessment. In our commitment to statistical rigor, we must acknowledge that some students may be inadvertently overlooked at our institutions. All of our student experiences matter and deserve to be heard– representative of its population, or not. It is our responsibility to ensure that this is reflected in our data practice. In doing so, we can collect good data on campus while representing and supporting the diverse voices that we so deeply value.
Sydney Kayne
Assessment Specialist, University of Colorado at Boulder and SAAL Blog Team Writer
Resources/Citations:
Aguilar, D. D. (2012). Tracing the roots of intersectionality. Monthly Review, 12.
Anderson, G. (2021). Indigenous and LGBTQ students’ mental health most hurt by pandemic. Inside Higher Ed | Higher Education News, Events and Jobs. https://www.insidehighered.com/quicktakes/2021/02/17/indigenous-and-lgbtq-students-mental-health-most-hurt-pandemic
Cowing, J. L. (2021). 6 things white educators can do to support native LGBTQ+ and two-spirit students. GLSEN. https://www.glsen.org/blog/6-things-white-educators-support-native-lgbtq-two-spirit-students
Crenshaw, Kimberle. (1989). "Demarginalizing the Intersection of Race and Sex: A Black Feminist Critique of Antidiscrimination Doctrine, Feminist Theory and Antiracist Politics," University of Chicago Legal Forum: Vol. 1989, Article 8. Available at: https://chicagounbound.uchicago.edu/uclf/vol1989/iss1/8
Hernández, E. (2015). What Is “Good” Research? Revealing the Paradigmatic Tensions in Quantitative Criticalist Work. New Directions for Institutional Research, 2014(163), 93-101. https://doi.org/10.1002/ir.20088
Overall Enrollment Profile [PDF]. (2022). Office of Data Analytics, CU Boulder. https://www.colorado.edu/oda/sites/default/files/attached-files/overallprofilefall22.pdf
Report on the AAU Campus Climate Survey on Sexual Assault and Misconduct [PDF]. (2020). Association of American Universities. Retrieved from https://www.aau.edu/sites/default/files/AAU-Files/Key-Issues/CampusSafety/Revised%20Aggregate%20report%20%20and%20appendices%201-7_(01-16-2020_FINAL).pdf
Spurk, D., Hirschi, A., Wang, M., Valero, D., & Kauffeld, S. (2020). Latent profile analysis: A review and “how to” guide of its application within vocational behavior research. Journal of vocational behavior, 120, 103445.
Turan, J. M., Elafros, M. A., Logie, C. H., Banik, S., Turan, B., Crockett, K. B., ... & Murray, S. M. (2019). Challenges and opportunities in examining and addressing intersectional stigma and health. BMC medicine, 17, 1-15.
Zadvirna, D. (2023, July 26). New report shines spotlight on “shocking” rates of self-harm among young indigenous Lgbtiqa+ people. ABC News. https://www.abc.net.au//news/2023-07-26/indigenous-lgbtqia-suicide-walkern-katatdjin-rainbow-knowledge/102646976