The Need for More Formative Research in Edtech

By: Katie Boody Adorno

Declines in NAEP scores and recent reports of low uptake of newly implemented education technology programs don’t mean that folks are no longer looking to education technology with hope. While there is no denying that the education sector will continue to be digitized, there is a growing call to action for edtech to live up to the promise of expanding access to educational opportunities that can accelerate learning.

However, while the call for more tech efficacy research is warranted and important, we found an overlooked and perhaps more urgent need for product research that focuses on streamlining user experiences and enhancing product implementation. After all, a larger threat looms in the potentially overinflated edtech space–lack of engagement.  

According to LearnPlatform, on average, school districts access over 1,400 edtech tools every month. Additionally, in a report by SETDA, 57% of respondents said that they have a lot of edtech programs/products but they aren’t always used effectively.

And this is consistent with our own findings. At Leanlab, when we conduct research, we work meticulously to match edtech companies with the right school partners. But, even when priorities are aligned between the researchers, the edtech company, and the school, if an edtech tool is too hard to use or teachers don’t understand how to fit it into their lesson plan, we know we’ll see some level of attrition throughout the study.

For products to be successful in creating impact, they need to be easy to use within the chaotic realities of K12 education in its current moment. If nothing else, as school districts continue to become more savvy consumers, they will jettison edtech products that are underleveraged, no matter how evidence-based.

We recently worked with a third-party researcher to analyze 14 research studies we conducted with seven edtech products in schools across the country.

What we found is that while edtech products do demonstrate potential in aiding student learning, more weight needs to be given to formative education technology research that focuses on improving product features and implementation protocols, which are currently impeding student progress on many education technology solutions.

We studied products focused on social-emotional learning, parent engagement, elementary math, Spanish reading comprehension, and teacher and administrator efficiency and administrative tools.  

The types of studies ranged from usability studies (measuring if the product is easy to use), feasibility studies (measuring if the product is feasible to use as intended within the realities of a school day), implementation studies (measuring under what conditions is the product best used, and at scale), and correlational studies (what relationship might exist between usage and student outcomes).

What We Found

Gamification

Gamification is a growing trend featured in edtech products but companies may need additional support to implement gamification effectively.

Onboarding

Ineffective training for product use was a concern; most companies overloaded content in their training sessions rather than offering it in a bite-sized/need-to-know basis.

Usability

Poor user design, in some instances, led to lower uptake of the product highlighting the need for robust usability studies.

Implementation

Logistical difficulties such as students being unable to log in, navigating support in a virtual classroom, student absences, or companies not having a usable implementation guide was a relatively consistent finding across four of seven products investigated limiting both product use and related research.

Attrition

Correlational studies saw attrition and statistically insignificant results; this may be due to a variety of factors including, insufficient preparation, potential lack of a clear theory of change, and school closures due to the COVID-19 pandemic.

These findings suggest that more companies can benefit from research that focuses on user experience and the contextual realities of school communities that can improve the uptake of products.  

Since the 2015 passing of the Every Student Succeeds Act (ESSA), schools and districts have been equipped with the ESSA Tiers of Evidence framework, which supports schools in evaluating interventions and technologies based on the rigor of independent research interventions they have engaged in to evaluate the performance of the tool.

The Tiers—ranging from Tier I (strong evidence from an experimental study) to Tier IV (demonstrates a rationale)—help companies boost credibility, however, they can, and should, be leveraged as important guide posts to inform product development that best serves learning outcomes for students.

This poses a more significant question to the field; how might we reframe and expand the ESSA Tiers of evidence so that they support the realities of iterative product development cycles while also focusing on the realities of classroom implementation?

Katie Boody Adorno is the founder & CEO, LEANLAB Education.