Bringing Culturally Responsive Approaches to Your Residence Life Assessment (Part 2)

Other Posts in This Series: Part 1 | Part 2 | Part 3 | Part 4

Within Student Affairs, assessment is often an expectation of everyone’s work. Whether you are someone in an entry level role or have been around the block for 20 academic years, assessment likely comes up. I believe it is critical within our assessment efforts that we center humanizing values and perspectives. I believe that applying the Culturally Responsive Evaluation (CRE) framework to Student Affairs assessment is the best next step in using humanizing values and perspectives to reimagine assessment. Through this series, we will explore what the CRE framework is and how Student Affairs practitioners can apply the CRE framework to their assessment through a case example. In Part 1, an introduction to culturally responsive evaluation and working through step 1 and step 2 of the CRE framework was shared. In this second part, we will go through step 3, step 4, and step 5 of the CRE framework with our case study. As a reminder, below is the most often conceptualization of CRE which showcases that cultural competence is centered in every step of the assessment process:

Culturally Responsive Evaluation - Part 2 — Culturally Responsive Evaluation by Hood, Hopson, and Kirkhart (2015)

How To Apply CRE to Student Affairs Assessment

For this section, we will use the following case study created by ChatGPT. For the sake of this case example, let’s assume that you are a member of the Harmony University Housing team and have been involved for the entire history of the STEM Living and Learning Community. I am coming in as your CRE consultant and will be sharing information and posing important questions to consider within each step based on what has already been shared in the case study.

In the heart of a bustling urban campus, Harmony University prides itself on fostering a vibrant and inclusive community. The Residence Life department plays a pivotal role in shaping this community by providing students with a supportive living environment. However, recent changes in the university’s programming prompted the Residence Life team to initiate a comprehensive assessment of the Living Learning Communities (LLCs) within the residential halls.

As part of an innovative initiative, the university introduced specialized LLCs tailored to students pursuing STEM disciplines seven years ago. Recognizing the unique needs and challenges faced by students in these fields, the Residence Life team sought to evaluate the effectiveness of the STEM-focused LLCs. The decision to conduct an assessment arose from the desire to ensure that these communities not only met the academic needs of residents but also contributed positively to their overall college experience.

Feedback from students offered valuable insights into the impact of the STEM LLCs. One student remarked, “Being part of the STEM LLC has been a game-changer for me. The collaborative atmosphere and access to academic resources have greatly enhanced my learning.” Conversely, another student expressed concerns, stating, “While the focus on academics is commendable, the social aspect feels a bit lacking. It would be great to have more events that foster a sense of community outside the classroom.” These perspectives underscored the importance of a nuanced assessment to refine and tailor the STEM LLC program to better meet the holistic needs of its residents.

Step 3 – Prepare for the Evaluation

Now that you have your stakeholders identified and have been drowning in context-setting from preparing for the assessment, you can collaboratively identify the evaluation purpose with your stakeholders. This involves answering the following questions to help set up your assessment questions in the right context:

Is this assessment required by the institution? If so, who are you being asked to demonstrate accountability to?
Is it called for by a student group or alumni organization?
Is it part of the routine assessment process or is it intended to clarify and troubleshoot an apparent problem?
Is continuation, expansion, or reduction of the program contingent upon conducting this assessment?
Do the results of this assessment have a direct impact on the future of this program?
Is the assessment intended to stimulate change and promote social justice?

Based on what we know about the STEM LLC, we know that this assessment is to clarify the role of the STEM LLC within a changed approach to university programming and to refine the offerings of the STEM LLC to meet student needs. An assessment serving more than one purpose (whether those multiple purposes are explicitly shared or not) is not uncommon. It is critical in this phase that while applying the CRE framework you are aware of how this assessment and program may “maintain or challenge (im)balances of power” and also how equity is supported through the assessment you build.

Step 4 – Frame The Right Questions

After the purpose of the assessment has been decided, an agreement can be made on what overarching assessment questions will be answered through this process and which questions are the most important. The overarching assessment questions are not the actual questions you will ask during data collection. For example, we might ask an evaluation question such as “In what ways, if any, does the STEM LLC experience differ for students based on gender?”. To answer this question we would ask more pointed prompts in our data collection, like a survey or interview, about the STEM LLC experience and then analyze the data based on the gender of respondents to see if any differences exist.

It is likely that in this question framing stage more assessment questions will be created than you have the capacity and time to address, which is why prioritizing questions within this phase is also necessary to maintain trust with your stakeholders. This stage will include clearly defining the focus and scope of your assessment questions and considering the cultural nuance of specific words and language. For example, given that the STEM LLC has a focus on STEM, what does each letter of STEM mean on your campus and what programs are included in each? Being specific and clear in your language will allow for the scope of the assessment to be easily defined. Within this phase, a few questions to consider include:

Will the evaluation focus on community needs and strengths, on the daily operation of the program, on appropriate and equitable use of resources, on progress toward intended outcomes, or on overall effectiveness?
How will the perspectives of STEM LLC participants and overall community be included in framing the questions?
How well is the STEM LLC connecting with the values, lifestyles, and worldviews of the current and prospective students?
How are the burdens and benefits of the STEM LLC distributed?
What is considered “good” evidence that the STEM LLC is working? Who has decided this and how will we explore it?

While this process may feel daunting, it is crucial for ensuring that your assessment will actually meet the concerns of your stakeholder group. Part of doing this in an authentic way is to have deep discussions with your stakeholders to decide what will be considered credible and trustworthy evidence. That is to say, how will we know that the STEM LLC is working? For the sake of clarity in the following sections, let’s say below are our overarching assessment questions. In a true assessment of the STEM LLC, there would likely be more than two overarching questions with subquestions:

What is the STEM LLC experience like for students?
- What similarities and differences exist in experiences based on demographics?
What are unmet needs for STEM LLC students?
- What similarities and differences exist in these unmet needs based on demographics?

Step 5 – Design The Evaluation

After setting the right questions for your evaluation, it is now time to design the assessment itself. The best practice within CRE is to utilize a mixed methods approach, however a qualitative approach can be used instead depending on the context of the assessment and the overarching questions being asked. The important part of designing the assessment is being responsive to the specific context in which you are working. This means that to perform an assessment in alignment with the CRE framework, you cannot verbatim replicate what other campuses are doing for assessing a similar program.

I know, this is a very hot take when considering we see requests for resources daily in Facebook groups and listservs. However, your assessment design needs to be in alignment with the questions being asked in your assessment and measure the evidence standards and cultural values set in collaboration with the stakeholders. All of these are seeped in cultural context that are relevant in your unique context. This design phase also includes selecting the sources of information to answer your assessment questions, the time frame of data collection, and the format of data collection. As these decisions are being made, questions to consider include:

Who is best situated to provide us the answers we are seeking? What is the best way for us to collect data from them?
What types of understandings are we seeking? Are these holistic understandings?
How long will it take for us to create our data collection instruments? To collect the data? To clean and analyze the data? To report it back?
Are comparisons required among persons receiving STEM LLC services and those not in the STEM LLC?
In order to answer the priority questions, will it be important to disaggregate the data by culturally relevant categories?

Within our example, utilizing a mixed methods approach seems to be the best method. For our first question, I would advocate that we have interviews or focus groups to understand what the experience of the STEM LLC is like from the STEM LLC students. I would also encourage a survey to supplement those interviews or focus groups depending on the size of the STEM LLC, to allow all students to provide insight about their experiences. Then for the second question, I would recommend similar methods: a survey with supplementary interviews or focus groups of STEM LLC participants to surface unmet needs.

It would be critical to also collect demographic information in all data collection methods to explore patterns of experiences or challenges across demographic groups. The demographics we choose to collect will be directly connected to the cultural context of our institution. For example, I would imagine that including more “expected” demographics such as race, ethnicity, and gender would be necessary to be in alignment with your division’s best practices. As your CRE consultant, I would also advocate for you to consider adding disability status and STEM program to your demographics. This is because the disabled communities have specialized needs when it comes to community participation, which are often heightened as an “othering” experience within STEM programs. Also being able to see if specific STEM programs are having differing experiences will allow the housing department to be better situated to share the student experience that is relevant for specific campus partners during the results sharing phase. In this stage, it is important to ask yourself what populations may have questions or concerns which may be lost if we don’t break out their data within our analysis. Those populations are those that you then need to create demographic questions around.

Wrapping Up

Whew, there we are and we are over halfway there! Make sure to stop back next month where we will continue building out this case example by working through our data collection instruments and the data collection process through the Culturally Responsive Evaluation framework.