Prizes are an efficient mechanism for focusing and accelerating innovation.
Automated scoring of student essays is fast, accurate, and affordable.
Those are two conclusions drawn from prize competitions sponsored by the William and Flora Hewlett Foundation in 2012.
The case study, “Automated Student Assessment Prize Phase One and Phase Two: A Case Study to Promote Focused Innovation in Student Writing Assessment,” was released today. It takes a close look at the ways the two phases of The Automated Student Assessment Prize (ASAP) demonstrated current capabilities and mobilized global talent to accelerate innovation in writing assessment.
ASAP began in February of 2012 with a demonstration of capabilities of the eight largest testing vendors. The “bake off” was hosted on the Kaggle platform and, as Mark Shermis and Ben Hamner reported, demonstrated that current scoring engines could match expert graders across eight sets of essays.
Next was an invitation to data scientists worldwide to try to beat the best and win part of a $100,000 purse. The level and diversity of talent mobilized was extraordinary — an actuary in Singapore, a weather scientist in Washington, a teaching assistant in Slovenia, and geologist in Canada, just to name a few. In 60 days, more than 20 teams had topped the best performance of the vendor demonstration.
Jason Tigg, the British particle physicist turned high frequency trader who was a member of the first place team said, “I enjoyed working on a real-life problem that has the potential to revolutionize the way education is delivered.”
ASAP Phase Two took on the more difficult challenge of scoring short-form constructed responses (short answers). Luis Tandalla, a college student from Ecuador, won the $50,000 first prize a year after taking Andrew Ng’s machine scoring MOOCs.
The diversity of entrants in both phases is a reminder that most innovation is translational: Something that worked in one field may work in another. The question is how to expose cross-discipline and cross-sector innovation? Prizes are a super-efficient means of promoting translational innovation. Some of the winners from ASAP Phase One are already helping some of the big testing companies improve their services.
The two phases of ASAP established standards for the utilization of assessment technologies; advanced the field of machine scoring in the application of student assessment; and introduced new players with different and disruptive approaches to the field.
ASAP is now expanding to other challenges in machine-scoring applications, including the development of systems to support writing instruction in a classroom setting. ASAP remains committed to its role as an open, fair and impartial arbiter of machine scoring and writing assessment capabilities through a series of scientifically rigorous studies and field trials.