Helping Teachers and Schools Run Experiments

Teachers are scientists, they’ve always experimented. Most of the time it’s informal, “Let’s try a new behavior management routine.” Or, “Watch this video tonight and we’ll discuss it in the morning.” Or, “Let’s try a really hard problem.”

It’s been hard to run formal experiments because measurement systems in education are so bad–a year end test won’t help with a reading experiment this week.

What if we gave teachers, and whole schools, the chance to run experiments of their own or to join larger trials? What if they had access to better measures and powerful analytics?

This sort of experimentation is common in business where a number of big vendors (SAS, SAP, Microsoft, IBM) provide business analytics to support quick experiments. For example, APT, acquired earlier this year by MasterCard has, “cloud-based software that determines cause-and-effect relationships between tested programs and key performance metrics. Beyond determining the overall impact of each initiative, by rapidly sifting through huge data-sets, the software helps companies target their actions to the customers predicted to respond most profitably.”

In software development it’s common to run A/B tests that quickly test user experience (UX) features. Rapid feedback is critical to iterative development.

The combination of social media and quantified self is ushering an age of health experimentation. There are advocates for self-experimenting, trials with an n-of-one.

Ian Eslick, who went back to MIT for a mid-career PhD, has studied self-experimentation. He’s interested in efficacy not validity. He’s willing to be wrong for a while but wants to make quick practical progress. He wants to help people find treatments that work for them, fast. “As individuals we make the best progress when our hypotheses are well founded and the conclusions drawn from our experiments are accurate. As a community we make the best progress when we can share our ideas in ways that accurately predict how someone else will fare.”

We need more of this in education, and it’s getting easier to manage. Powerful and easy to use formative assessment systems (from Edmodo, MasteryConnect, and OpenEd, for example) make it easy for a school, district, or network to deploy common assessments around an quick A/B test.

Intelligence platforms like BrightBytes improve hypothesis generation, as well as measurement of results over time. In business, this is typically thought of as Return on Investment, or ROI. BrightBytes thinks of it as ROL, Return on Learning.

Test what? If teachers and schools had the ability to run short trials, what experiments would they run? Some examples include comparing:

  • Reading strategies and materials
  • Blended learning models (e.g., lab rotation vs station rotation)
  • Math software
  • Behavior management systems

Like projects, experiments can be assessed by a mixture of quantitative and qualitative factors. A variety of measurement systems can be used including formative assessment, teacher and student surveys, and observations.

Why not just rely on national studies? Results in education are highly context specific– particularly implementation efficacy. As a result, national studies may be suggestive but they won’t tell you how something will work at your school. And they’re not available for a new app or strategy.  

The first head of the Institute of Education Sciences, Russ Whitehurst set a gold standard for research particularly around the use of randomized controlled trials (RCT). There is obvious benefit to setting a high bar for evidence but we often need better evidence faster and cheaper than is possible through RCT.

I’ve complained before that many randomized trials (like this 2013 small school study) providing a blinding flash of the obvious a decade late. They sacrificed efficacy for validity.

We need well designed large scale controlled studies but there is a growing need to support an iterative change strategy in education.  

Path forward. I’m optimistic about the shift to personalized learning not only for the customization and motivation students will experience, but for the tools that it will provide to teachers. Big Data will allow interested teachers to become data scientists and participate in large scale experiments in real time.

With support from the Gates Foundation, organization like LEAP Innovations are creating test beds of groups of schools hosting short cycle trials. This is a big step in the right direction.

The next step in helping teachers as scientists is an education intelligence platform that combines information about best practices, easy to use formative assessment, templates for setting up experiments, and social features for sharing.   

For more, check out:

Tom Vander Ark

Tom Vander Ark is the CEO of Getting Smart. He has written or co-authored more than 50 books and papers including Getting Smart, Smart Cities, Smart Parents, Better Together, The Power of Place and Difference Making. He served as a public school superintendent and the first Executive Director of Education for the Bill & Melinda Gates Foundation.

Discover the latest in learning innovations

Sign up for our weekly newsletter.


Leave a Comment

Your email address will not be published. All fields are required.