ideas42

The Student Success Toolkit: From "Test” to "Scale” at CUNY and Beyond

Mon, 10 Aug 2020 18:36:56 GMT

Testing is one of the most exciting parts of implementing behavioral science interventions for us at ideas42. After months of research and interviews, and countless design iterations, we are finally able to gain insight into the extent of our impact. But what happens once we’ve run the regressions, printed the tables, and synthesized our key takeaways? Once we have evidence about what works—and what doesn’t work—how do we apply our findings in new contexts?

Among the many takeaways from the Student Persistence Initiative, ideas42’s five-year collaboration with The City University of New York (CUNY) and the New York City Mayor’s Office for Economic Opportunity (NYC Opportunity), is this: behavioral science interventions in post-secondary education, even those that are email and text based, are rarely plug-and-play. In order to scale successful interventions within a school network, or indeed to scale to other school systems, we need to put effective interventions in the hands of education experts and equip them with the tools they need to adapt the interventions for the realities of their own school contexts.

Making the leap from “test” to “scale” is the motivation behind the Student Success Toolkit, an actionable guide for education experts looking to scale and adapt the interventions that our rigorous research suggests can help students succeed. The toolkit distills insights and lessons learned from 13 randomized controlled trials (RCTs) conducted over five years across six CUNY community colleges, touching on challenges ranging from increasing the number of students who annually renew the Free Application for Federal Student Aid (FAFSA), to encouraging students to take enough course credits to graduate within two to three years.

As we set out to create this toolkit, we reflected on our own implementation experience. Between 2015 and 2020, as we iterated on behavioral interventions, we had to adapt our designs for new contexts. We learned that the individual colleges within the CUNY network, the nation’s largest urban university system, differ in many unexpected ways—some purely semantic (different monikers for their respective advising offices) and some more meaningful (different transfer office structures that support students planning to transfer to a bachelor’s program). Additionally, each academic year is different—sometimes in predictable ways (school events fall on different dates each year), and sometimes in less predictable ways (as instruction moved online this year due to the COVID-19 pandemic, we adjusted the timing of our intervention messages to make sure they’d still feel relevant).

With these experiences in mind, we created a resource that not only shares open-source versions of our intervention materials (like text messages and emails), but also describes the methods and tools we used to consistently adapt and improve our interventions over a five-year period.

We divided the toolkit into two sections in order to systematically guide readers through our behavioral diagnosis and design methodology. The first section describes the behavioral barriers—situational, psychological, or contextual features that prevent students from making optimal decisions or acting on their good intentions on their journey to academic success. These insights, generated from hundreds of conversations with students and staff across the CUNY system, as well as our own user testing (including as enrolled students), are the foundation for all of our intervention designs.

In the second section, we provide case studies on our five most successful interventions to demonstrate how we used behavioral design at CUNY to help students overcome the diagnosed barriers. Specifically, we outline strategies for effective communication, and illustrate how we applied those strategies through our designs. For example, we found that simple language and an urgent tone works well in emails and texts reminding students to renew FAFSA. Along with sample messages, we highlight why we used particular language and images. Students at CUNY who received our FAFSA messages were between 4 and 9 percentage points more likely to renew FAFSA compared to a control group; if scaled, the low-cost designs presented in the toolkit could help educators at CUNY and elsewhere bring millions of additional dollars in financial aid to students with minimal additional cost to the schools.

At ideas42, we are already using the Student Success Toolkit’s insights and designs to further promote student persistence and success beyond CUNY. For example, with support from the Michael & Susan Dell Foundation, we’re currently working to deliver the content tested in the Student Success Toolkit to hundreds of new schools by integrating our behaviorally informed campaigns into a leading student communications platform. Through this multi-year project, we hope to facilitate a seamless experience for schools across the country to select, adjust, and implement messages that are relevant to their students’ needs.

As behavioral scientists, we constantly return to the question at the heart of our work: what drives human behavior? Testing is indeed exciting, but in order for evidence-based interventions to help students succeed across the country, we must continue to share foundational insights about human behavior along with best practices for behavioral design, including practical ways to translate research evidence to practice. With this in mind, we will continue to seek ways to make it as easy as possible for educators—the experts in their own contexts—to make the leap from knowing what works to making it work for their students.

The post The Student Success Toolkit: From “Test” to “Scale” at CUNY and Beyond appeared first on ideas42.

Street Smarts in the Field: Insights for Designing Public Housing Infrastructure

Tue, 04 Aug 2020 16:07:08 GMT

For several years, city governments across the world have been incorporating insights from behavioral science to help design new policies and programs to solve important problems. This entails behavioral experts designing interventions to support groups or communities they themselves may not belong to. In fact, a strength behavioral scientist bring to pervasive problems is an outside, fresh perspective, combined with evidence of what has been effective in other contexts. This outside perspective makes it crucial to listen to people with lived experience in communities when designing solutions. At ideas42, that is always our aim—we conduct interviews with people who will be impacted by our interventions in order to diagnose barriers and inform new designs. But that doesn’t mean we do it perfectly, or that we don’t have our own biases that cause us to overlook important details.

In a recent project my team worked on to improve public housing, we saw the benefits of listening to residents—and the limitations of not listening enough. With support from our partners at New York City Housing Authority (NYCHA), John Jay College, and the Mayor’s Office of Criminal Justice (MOCJ), our team came together to address resident quality of life concerns arising from improper disposal of household trash, litter, and dog waste on NYCHA grounds. We created a robust set of interventions: we installed a set of large, moveable trash containers (tilt trucks) and trash cans in convenient locations and provided a package of communications posters to inform people of the new policy and encourage them to use the new infrastructure.

As a native NYCHA resident working on this project, I wanted to leverage my perspective to help the team better design and communicate our project to the NYCHA community. My experiences growing up in NYCHA and my position on our team gave me a special combination of formal expertise and tools from the behavioral sciences as well as a natural understanding of NYCHA community norms and values.

For anyone interested in project design in familiar and unfamiliar communities, these three insights showcase how highlighting and neglecting lived experience in design matters for the success of the overall project, and its impact on people’s lives:

Contextualize language choices

Our team’s package of communications posters was intended to inform NYCHA residents of new infrastructure installations for trash, litter, and dog waste. These posters were designed to speak to resident identities as NYCHA community members to create new social norms that would boost compliance with the new infrastructure installments. However, activating resident intentions to take care of their environment only works if we’re all on the same page about what that environment is.

When our team began drafting communications poster designs, we tailored each poster to the specific community where it would land. However, the signage included the formal, technical names of the NYCHA complexes that were used by NYCHA administration. What we needed instead were the informal, everyday names that residents used to refer to their specific NYCHA community.

Using the formal, administrative name for a NYCHA community on a poster that aims to introduce new social norms to a longstanding community only showcases the extent to which we are outsiders to these communities. As a NYCHA resident of what is formally known as Gouverneur Morris Houses I, I would have been totally unfamiliar with that term if I saw it on a poster; what I do know is that I live in the Morris community absent of any distinction between Gouverneur Morris Houses I and Gouverneur Morris Houses II. The same goes for the residents of what are formally known as Lower East Side Rehab (Group 5) and Seward Park Extension, better known as The Lower East Side and Seward Park. Because of my insight into the context of how naming works at NYCHA, I pushed our team to include the informal names of complexes on our posters in order to establish trust between the residents and our messaging, and ultimately the informal names are the ones we used.

Be intentional about product placement

As part of our strategy to get the intervention materials to residents, our team gave NYCHA property managers guidelines for where to best install new posters, trash cans, and tilt trucks. Although property managers were given the final say about where to place all materials, our team wanted to suggest a consistent strategy for placement. Because of my lived experiences with moving through NYCHA buildings and interacting with their many nooks and crannies, I was able to suggest a variety of locations to place our materials in order to capture resident attention. More importantly, my lived experience in NYCHA communities gave me an understanding of where not to place those materials. For example, I know that elevators, staircase doors, and mailbox areas are important spaces for residents and that the lobby door area, the parking lot, and resident hallways are not given much attention. The difference between putting a poster in the elevator vs. lobby door may sound minor—but regardless of our intentions to make an impact in the NYCHA community and no matter how good the design or messaging was, if we put our hundreds of intervention materials someplace no one looks, they wouldn’t be effective.

Value lived experience as legitimate knowledge

During the product design phase of this project, I raised concerns about the tilt truck handle to my team. When using the tilt trucks, residents had to first lift the lid to throw their trash inside. I mentioned that, from my own experience, NYCHA residents don’t like touching the handle of the small trash compactor located in their hallways and that because of this, the handle would limit our impact. However, my team remained skeptical about whether such a tiny detail would make a significant difference. As the most junior member on the team at the time, I doubted my own critique because I couldn’t prove my claim with anything other than my intuition from my lived experiences. Although my team was not antagonistic in response to my critique, their doubt was enough to dissuade me from trusting my gut. Ultimately, the burden of proof from my lived experiences in this instance felt too complicated for me to navigate and overcome.

During the study, one main piece of feedback that our research assistants, who interviewed residents after we rolled out the intervention, received was that the presence of the handle on the tilt trucks we purchased were a barrier to properly throw out trash. This was an issue because residents did not want to touch the (often dirty) tilt truck handle because they thought it was gross and were often on their way to work or school. For elderly NYCHA residents, they reported the handle to be a barrier because it was a heavy lift. This is a behavioral barrier: The disgust from touching the handle was enough to prevent people from using the tilt truck. Although our team is made up of people who use behavioral science and design thinking daily, we collectively overlooked this element. Notably, even though I did foresee the problem, it wasn’t because of my behavioral science knowledge, but my lived experience. This underscores that behavioral insights and lived experience are both valuable when developing new designs for any community. While our team still saw statistically significant improvement in how residents disposed of trash using the tilt trucks, this may not be an accurate measurement of the full potential of our intervention because of the tilt truck handle.

For NYCHA sites that received the new infrastructure, the average number of household trash bags decreased by 25%, the average amount of litter observed decreased by 16%, and the average instances of dog waste spotted decreased by 11%. Providing easier access to trash disposal infrastructure, complemented by community-oriented and instructional communications, reduced the burden of trash disposal for residents and significantly reduced the amount of trash visible on NYCHA grounds, demonstrating that new structures and resources can effectively change behavior.

A key aspect of our team approach to problem solving is our emphasis on the insights we can gain from listening to stakeholder perspectives on their own community. Over the course of this project, we were able to leverage resident perspectives, including my own, to inform project design and implementation. Nevertheless, our team’s missed opportunities to improve key design elements illustrates our own biases. To put it clearly, community is an invaluable resource in designing new infrastructure. As we continue to design and tailor interventions to the communities we wish to serve, we must avoid privileging formal, scientific knowledge over informal, everyday know-how from community members.

The post Street Smarts in the Field: Insights for Designing Public Housing Infrastructure appeared first on ideas42.