Short Cycle Efficacy Trials Key to Personalized Learning

This fall the Bill & Melinda Gates Foundation made grants of $1.3 million for short cycle efficacy trials.  Three “test bed networks” each received $100,000: LEAP InnovationsNYC iZone, and the Bay Area Innovation Hub. Five platforms shared the balance: BrightBytes, CFY (PowerMyLearning), eSpark, Motion Math, and Common Sense Media.
The grants are the result of an August RFP seeking “new approaches to facilitating short-cycle feedback on product efficacy so that teachers, school decision-makers, and parents have better information about the effectiveness of digital courseware, and developers of such products receive more rapid input to improve their offerings.”
The explosion of education applications and blended learning models has rendered traditional methods inadequate. As noted earlier this year, randomized controlled trials–the gold standard for education research–aren’t very useful for providing design and implementation guidance. John Bailey, Executive Director of Digital Learning Now!, said, “The technology and models are new and quickly evolving, which makes long-term, experimental design studies difficult.”
Short cycle trials are common operating procedure in lean startup strategies adopted by most EdTech companies and a few school networks like Summit Public Schools.
The test bed sites will, according to the RFP, “Lower risks and barriers to all parties of piloting new, potentially-transformative products,” which will, “Encourage the rapid development of a healthy, transparent market for highly effective learning technologies.”
Chicago is one of the three text bed sites.  Phyllis Lockett, CEO of Chicago-based LEAP Innovations, said:

We’re launching an EdTech Innovation Hub in Chicago, and a key element is our Pilot Network, comprised of a diverse range of learning environments that will pilot and research the best existing and emerging tech learning solutions.

Our initial focus is on K-12, and we received a Gates grant to support the first cohort of schools in our Pilot Network.  The grant allows us to begin multi-phased evaluations of tech solutions and their effect on student outcomes using nationally normed assessments, as well as qualitative reviews.

Our goal is to build an industry standard for quality products, helping ensure that effective tech tools solving critical learning gaps become available — and scalable.  The pilots also expose educators to promising solutions, open companies’ eyes to real-world challenges facing teachers, provide a low-risk environment for trying innovation, and build a pipeline of innovative educators and schools to impact the greatest number of students.

Andrea Coleman described New York City’s iZone project:

The iZone’s Innovate NYC Schools initiative received a development grant to explore the incorporation various kinds of efficacy metrics into the test bed model we validated for the GapApp prototyping work. Our emphasis was on building capacity through collaboration–the capacity of developers to understand better the realities of the urban classrooms that are the market for their work, and the capacity of educators to incorporate aspects of design thinking, prototyping, and agile/lean methodology into their pedagogy.

The Gates grant will allow us to incorporate an efficacy component into our lessons-learned. Both of these elements–capacity-building and an understanding of what works in what contexts–are crucial to improving the ecosystem.

Real-world, organic deployments of new technology and new pedagogy require more real-world organic indicators of impact: Not just the logistics but the premises of classic RCT work against real ROI for promising new approaches and ignore the realities of real-world implementation. It’s very helpful that this new portfolio is paving the way for better-faster-cheaper evaluation and we’re looking forward to contributing the benefits of our experience.

CFY was one of five platform grantees. CEO Elizabeth Stock, described the project:

CFY received the largest grant from the Gates Foundation’s RFP on evaluating the efficacy of digital courseware.  CFY’s proposal focused on short-cycle “A/B testing” of granular pieces of digital content – such as games, interactives, video, short simulations – to help software developers improve their products and know “what works.” To conduct short-cycle A/B tests, CFY has developed a win-win process that plugs in to what teachers are already teaching (rather than being an “add on” that wastes classroom time), and provides valuable mastery feedback so students and teachers can see where help is needed. CFY will enlist teachers and students from its network of PowerMyLearning® users to participate in this process. PowerMyLearning is CFY’s free K-12 platform, which has received major support from national foundations and is currently in use in more than 14,000 school communities nationwide.

Maya Lopuch, Data Scientist at eSpark Learning, summarized their project:

eSpark Learning is embarking on an ambitious project to measure the impact of specific educational apps on student achievement and engagement. With the support of the Gates Foundation, we will evaluate hundreds of apps using real-time feedback from thousands of students all over the country.

eSpark has been collaborating with school districts to increase student achievement for over three years. No other organization has more expertise in helping teachers and school leaders leverage iPads for student learning, and no other organization has a comparable infrastructure to analyze data on the impact of educational apps. eSpark’s data on student achievement and engagement provides a previously unattainable view of the educational quality of iPad apps. This data will help us rigorously identify which content is truly best of breed, as revealed to us by thousands of students. We believe that student voice is central to building the leading digital curriculum.

The aim of this project is to continually improve the digital curriculum. Specifically, we will link standard-level data on student achievement to apps and learning pathways, then use A/B testing techniques to determine how to restructure our curriculum to produce bigger learning gains. We already do much of this app-level optimization with student engagement, so the major step here is incorporating detailed student learning outcomes. We will report back to teachers, administrators, and app developers about what works in the modern classroom.

BrightBytes is an education data analytics organization.  CEO Rob Mancabelli summarized their project:

The three goals of the work are to (1) empower students to provide learning-experience data to teachers and school administrators; (2) enable teachers to access outcome-based application feedback from students and other teachers, in order to make better choices when selecting learning applications; and (3) establish a feedback channel between the teachers/students and vendors enabling, quicker and constant improvement throughout the system and its constituents.

The application will be an extension of the widely popular Clarity platform, a SAAS-based tool used by thousands of schools globally. The tool collects feedback in a fun and engaging manner using gamification, while focusing data collection on learning outcomes and factors that influence them.

Seeta Pai, VP of Research at Common Sense Media, described their work:

The grant funds concept testing and piloting of a process via Common Sense Media’s Graphite ™ that links developers with expert educators for rapid, iterative testing and feedback on early stage EdTech products. We will consider both immediate (like teacher satisfaction, student learning and engagement) as well as more downstream (such as integration into technological and pedagogical infrastructures) measures of efficacy at schools/district, teacher/classroom, and student levels.

Last week the Broad Foundation invested $3 million in the Graphite rating system.
More short cycle trials will improve the development and adoption of personalized learning products.  Access to test bed networks and rapid feedback could lower the risks and cost of developing and piloting potentially-transformative products.
BrightBytes and eSpark are portfolio companies of Learn Capital

Tom Vander Ark

Tom Vander Ark is the CEO of Getting Smart. He has written or co-authored more than 50 books and papers including Getting Smart, Smart Cities, Smart Parents, Better Together, The Power of Place and Difference Making. He served as a public school superintendent and the first Executive Director of Education for the Bill & Melinda Gates Foundation.

Discover the latest in learning innovations

Sign up for our weekly newsletter.


Leave a Comment

Your email address will not be published. All fields are required.