An MIT EE and a Harvard MD, Bror Saxberg is the chief academic officer of Kaplan. He appreciates scientific inquiry—and he’d like to see more of it in education.
“The lack of an efficacy framework will hold us back,” said Saxberg, “It will result in frustration and wasted resources.”
At every conference he attends Bror mentions the lack of basic research in education. Instead, the typical EdTech story is that somebody’s sister is a teacher so he developed an app to solve a problem she faced.
“While the biotech community has embrace approaches that are both evidence-based and entrepreneurial, similar approaches are nearly absent in education,” said Saxberg. “But we’re hard work inside Kaplan trying to change that—we’re doing randomized controlled trials to test the impact of laboratory results from learning research at scale in a wide array of courses, to help define the “rules of the road” for better instructional design –and more student success.”
Kaplan is also “looking closely at research on motivation, to see if we can make alterations to our learning environments to support learners “head of steam” around the sometimes difficult practice and feedback needed,” according to Saxberg.
What is efficacy? Saxberg defines efficacy as “How well a learning environment does in getting a wide array of learners to measurably master outcomes that are truly valuable to performance in the real world.”
On Pearson’s new efficacy framework, Bror thinks it is “a terrific contribution,” and is “incredibly pleased by structure and tone.” He appreciated how clearly the documents (including the forwards by past and present CEOs) indicate they’re not meaning “getting paper degrees” or “getting high test scores.” Rather, “the claimed focus for Pearson is on achieving results for learners that make them more successful in the world – which can be measured in a variety of ways.”
In November, Pearson introduced an efficacy framework. It asks reviewers “the tough questions” meant to assess a product or strategy along four areas and a dozen points considered essential for determining if a tool can achieve it’s intended learning results:
- Outcomes and Impact: intended outcomes, overall design, value for money;
- Strength of Evidence Base: comprehensiveness of evidence, quality of evidence, and application of evidence;
- Quality of Planning and Implementation: action plan, governance, and monitoring and reporting;
- Capacity to Deliver: internal capacity and culture, user capacity and culture, and stakeholder relationships.
Pearson not only shared the efficacy framework with the sector, but they built an web app where you can get a report on the likely impact of your educational product or serve–kind of like a prospective FreeCreditReport.com for EdTech.
More significantly, Pearson committed to reporting efficacy results as well as financial results by 2018. CEO John Fallon said, “Our aim is to ensure that every action, every decision, every process, and every investment we make will be driven by a clear sense and understanding of how it will make a measurable impact on learning outcomes.”
Good plan? Does it make sense for Pearson to attempt to report results against an efficacy framework? “In the long term, mediocre education will be free” said Saxberg, “the only sustainable value for a learning organization is helping people develop valuable skills. He added, “Companies will need to be good to charge a fee.”
Saxberg sees the change as inevitable, “The way we laugh at early medicine, people 100 years from now will laugh at early 21st century learning when we didn’t have a common efficacy framework in place.” About organizational learning, Saxberg wonders, “Won’t it be interesting when value creation can be tracked by improved performance of valuable employees?”
In the short term, customers and investors often seem more focused on whether a companies content will run on the next device than knowing if it’s any good. “It will be easy for bean counters to say why bother,” said Saxberg. “It may take time for the markets to realize what’s actually possible — that there really are measurably better ways to combine media, outcomes, interactives, data, to more reliably build better expertise.”
Is it possible for a company to embrace efficacy and return? “Yes, that is what Kaplan thinks,” said Saxberg. “What is clearly valuable is to have more people with better skills for things that matter in the real world,” and “long-term economics will have its day — if you truly are making people better at deciding and doing what measurably matters at scale, organizations can then pay for that performance difference–and create return.”
While it will be challenge to implement a review and reporting process, Bror sees big internal benefits of signaling that efficacy matters.
Audited? Some observers have suggested that the efficacy framework should sit in a nonprofit and that some kind of external review would boost confidence in reported efficacy. Saxberg wants to see Pearson go far fast but thinks eventually join efforts of large players—foundations, governments, industry—would help. “Schools and companies will suffer until market recognizes it could demand better outcomes,” said Saxberg.
Auditing results will be a challenge because dipsticks are different by domain. A process oriented review may be possible—it could verify, with spot checks, that the framework has been uniformly applied.
Saxberg thinks there could be a voluntary standard for ‘efficacy certified’ results—that would ensure that product teams are considering guidelines to make stuff more efficacious. A second stage could include a voluntary standard for ‘learning engineered’ including a deeper dive into research claims and product design.
Damper? In December I asked Michael Barber, author of the efficacy framework, if it would dampen innovation (the way a focus on charter quality dampened innovation in school models). Barber thought new framework would help clarify objectives and focus innovation on demonstrable outcomes.
Bror’s response was, “Can you imagine asking that question to a group of health care folks? ” It is so obviously useful to biotech and pharma companies to have products that are useful. “What’s remarkable about education, is that opinions DO vary about this–you can find venture folks, investors, researchers, even educators, who will say that no one is currently buying learning materials/environments based on whether they really work or not or the data on whether they work or not, and so, therefore, there’s no point investing in making products and services line up with the data.”
But that is changing—and Pearson’s efficacy framework is a big step in the right direction.