Resourcefulness: Balefire Labs’ Evaluation Helps Teachers Find Quality Apps

By: Steve Peha

I’ve written in the past about an interesting problem I’ve noted amid the abundance of digital education resources now available. There’s so much stuff out there, it’s becoming harder and harder to separate what works from what doesn’t.

Nowhere is this more of a challenge than in the educational app market. Just about any of us can take a quick look at someone else’s lesson or video and know right away if it will be useful. But try taking this kind of “skim and scan” approach to apps and you’ll probably discover as I did recently that making thoughtful assessments isn’t as easy—and not nearly as quick.

One night, I sat down with my iPad and the specific intent of finding 3-5 early reading apps for my niece. All I wanted were things that were consistent with one or two of the most basic scientific criteria for instructional credibility in reading—things like the Alphabetic Principle, for example, or repeated reading to improve fluency. Three hours and dozens of apps later, I didn’t have a single winner—not even a few worthy runner-ups.

The problem I realized was two-fold: (1) It takes at least 5-10 minutes to find, download, and play long enough with any single app to learn much about it; and (2) There are so many apps available for things like early literacy that I don’t think I browsed across even 1% of them in what ended up being a multi-hour exercise in futility.

I’m an education consultant and technologist. I have way more free time than most classroom teachers, 25 years of experience making technology products, and a thorough handle on the research base, especially in my main practice area of literacy. And yet the simple job of finding even one marginally worthy early reading app was not something I could complete in several hours.

I raised the issue in one of my newsletters, and a helpful reader sent me a link to Balefire Labs. Balefire has created a research-based evaluative model for reviewing educational apps. The company’s founder, Karen Mahon, seems to have started Balefire out of the same frustration I experienced.

“I started this company because I was really frustrated by the process of finding high quality educational apps,” says Mahon. “I’d spend all this time in the app store, find something that looked and sounded good, only to download it and find myself feeling very disappointed. And when I went to other app curation sites online, their evaluations were pretty subjective. As an Educational Psychologist and Instructional Designer myself, I knew we could do better than that.”

I liken Balefire to a kind of Consumer Reports for education apps. It has an easily browsable table-and-checkbox view of hundreds of apps which allows users to see at a glance which ones match up best with Balefire’s model.

“Our philosophy at Balefire Labs is that there’s a body of educational research that tells us what works well in instruction and that we should be using what science tells us!” Mahon exclaims. “Our review criteria are all scientifically-based. We even have a list of references on our website to tell you more about why our criteria matter. We use these criteria to evaluate all instructional apps in the same, systematic way. That allows us to compare and contrast apps directly with one another to tell you what’s worth your time and money and what’s not.”

Needless to say, my experience so far of app hunting on Balefire seems faster, easier, and more rewarding than previous experiences at app stores or other vehicles that aggregate educational resources.

In addition to ratings across their scientific model, Balefire also publishes reviews. I like two things here: (1) Good ratings are hard to get; very few apps get top grades; and (2) Balefire is very frank in their tone and directness of their negative reviews. They’re not mean-spirited, but they are clear about tools that may be a waste of time. I think this is particularly appropriate in an educational context where learning time is our single most valuable resource.

Mahon explains Balefire’s rationale this way: “We often get asked why it is that we publish negative reviews. The reason is twofold: first, we want teachers and parents to know what apps we think they should avoid. That one’s obvious. But the other reason we publish negative reviews is to try to help the developers of those apps know exactly what they can do to improve upon their work. Our purpose isn’t to ambush anyone. It’s to try to help move the needle toward having better educational apps across the marketplace.”

This last reason is something that interests me a lot. We have thousands of apps in the learning app space and more coming out all the time. But unless the overall quality of that market improves, learning apps are unlikely to have a positive effect on student learning. An explosive proliferation of ill-considered apps could bury us all in mediocrity—or worse, just become a treasure trove of time-wasters.

Right now, app development for education is an immature market. It tends to be filled with “first movers”. That’s normal. But as we might expect, the creators of these apps—who seem like intelligent, talented, and well-intended entrepreneurs to me—have a lot more experience creating software than they do creating learning.

Many learning app developers contact me for advice. The most common issue I find is a hesitation to modify existing apps to match research-based findings about educational effectiveness. While that’s completely understandable, it’s also unfortunate.

But not all app developers look at it this way. Some, like Monkey in the Middle, a developer of apps targeted at middle schoolers, have used what they’ve learned from Balefire’s rigorous rating system to improve the educational quality of their work.

“We want kids to learn using our apps, so we listened carefully to the Balefire feedback,” says John Meier of Monkey in the Middle. “There were two specific ways we improved our Monkey in the Middle apps using the Balefire criteria. The main change we made was to switch to Mastery-Based Instruction. We changed our apps so the student had to completely master a level before moving onto a harder level. Another change we made was to add Error Remediation. We changed the way we ask questions to give better feedback when the student made a mistake and provided a second chance to answer the question.

Mastery-Based-Instruction? Error Remediation? These aren’t the kinds of phrases I usually hear learning app developers using when they call me for advice on improving their apps. So Balefire represents, I think, a kind of double feedback loop—one for educators and another for developers—that could get more educators and parents making better choices and more developers making better apps.

Unlike most digital education resource curation systems, Balefire is not free. It charges a modest subscription fee that is easily affordable for a single teacher or parent. While most curation services these days are free, there a good reason why Balefire isn’t.

“We want customers to know that we have no vested interest in apps doing well or poorly in review,” says Mahon. “We feel that accepting advertising or, perhaps even worse, fees for reviews from app developers introduces potential bias into our process. When customers come to our site we want them to know that when we say an app is good it is only because it is good, not because we were paid by the developer. So we ask for financial support from the people we work for: teachers.”

Teachers aren’t wealthy by any means. But the cost of an annual Balefire subscription is less than the cost of buying half a dozen bad apps. So teachers who are inclined to thoughtful selection regarding the apps their students use should find the price point attractive. The fact that Balefire’s work may help conscientious developers make better apps, makes the whole idea of what they do even more attractive.

 Steve Peha is a learning strategist and education technologist with more than 25 years of experience. He is the founder of Teaching That Makes Sense, an education consultancy focused on literacy, leadership, and school-wide change. He has written extensively about teaching, technology, and education policy for sites like The Washington PostThe National JournalEdutopia, and others. 

Guest Author

Getting Smart loves its varied and ranging staff of guest contributors. From edleaders, educators and students to business leaders, tech experts and researchers we are committed to finding diverse voices that highlight the cutting edge of learning.

Discover the latest in learning innovations

Sign up for our weekly newsletter.

1 Comment

Karen Mahon
2/24/2014

Thanks so much for your kind words, Steve. Two notes that may be of further interest to your readers: first, our app reviews are now searchable by Common Core State Standard (we are the only app review service that we know of that is offering that); second, we will be at the Learn Launch conference later this week...looking forward to meeting any of your readers who may be there.
Best,
Karen

Leave a Comment

Your email address will not be published. All fields are required.