ARIS in the City: Using Data Effectively

: Image via Wikipedia

Bill Tucker, Managing Director at Education Sector, has put out today a pdf on data usage in the New York City Public Schools. He focuses a bit on ARIS and points out some of the problems that have been experienced in getting it to work well.

But ARIS has been fraught with problems, as well. Developers have confronted a tangle of antiquated systems that can’t talk to each other—information silos that prevented any one person from getting a complete picture of a student. And they continue to struggle with making the data timely and accurate and giving educators the time and training they need to use it well. In the process, they have learned that technology holds little value unless it is flexible, relevant, and provides the fine-grained information that teachers really need. Above all, the district has realized that building a data system is only the first step—what educators do with the data is the critical second. Building the conditions and demand for data- based analysis is often more difficult than collecting the data itself.

It’s hard to use data well, especially when there are silos involved.. We learned this lesson with the CIO from the Arizona Public Schools, Donald Houde. In a little interview we did with him that has so far gotten a lot of clicks, Houde says that data is a very sensitive subject in school districts, because it is often protected politically and used to make decisions with lasting impact, some of them painful.
In his report, Tucker throws in a bit of a commentary, which I think is right, about how teachers work today. He is right to point out that giving teachers the reins here is a testament to the effectiveness of data implementation, especially if you can get teachers to use it widely and with success (I am leaving off all mention of the current New York City debate over the release of teacher added value data for the moment):

Likewise, the best teachers have always used information about their students to help them improve instruction—and they know that more and better information can lead to even better results. Yet, unlike for almost all other professionals who perform complex, demanding work, the information tools available to teachers have been remarkably limited. Most teachers still work isolated in their classrooms, with only their own eyes and rudimentary assessment tools to guide them. For the most part, they aren’t benefiting from sophisticated information-gathering tools, from their colleagues’ knowledge, or from analyses of thousands of similar situations—the very kind of information that physicians, police officers, and even sports executives use on a daily basis. Concludes a recent article from SRI International’s Center for Technology and Learning: “Teachers do not have the data-rich, performance-support, and information-feedback work environment that virtually all other high-performance professionals … have at their disposal.”

So, how does New York City work with data, and can we move away from isolated islands of teacher effectiveness, or even isolated pockets of success with students? Tucker says that with ARIS, there’s a focus on collaborative inquiry, and using teacher teams to sort through the assessment data. He also says that there have been problems. Here’s a screenshot of the diagram provided by the NYC Department of Education, and a brief explanation from Tucker about how it works.

A team of teachers and administrators looks at schoolwide data to identify a focus group—say, sixth-graders with scores of two or less on last year’s state English Language Arts test, which is graded on a scale of one to four. The team then studies both the students’ work and the data, and reviews instructional approaches: “How are these students being taught?” they ask each other. “How are they being grouped?” After answering these questions, the team develops a theory about why the students are not succeeding and begins to implement changes.20 Periodic or interim assessments, such as those from Columbia University’s Teachers College Reading and Writing Project, which measure recognition of letters and their corresponding sounds, are meant to help teachers gauge progress along the way. (See Figure 1.)
ARIS provides these teams of educators with a common set of data, including state and interim assessment results, attendance records, and course grades. (See Figure 2.) The teams use this information to collaborate on instructional interventions, not just for single students but for whole groups of students. The data helps teachers identify students’ strengths and learning gaps and inform parents of their children’s progress.21 Teachers can also use the data to customize progress reports. Educators looking for patterns of performance are no longer limited to what they see in their own classrooms. They can easily group and track students receiving specific help or identify students with a particular learning challenge.
It looks like you can find the information here, but I have not seen it listed this morning. The report is called “Putting Data Into Practice.”
I’ll leave you with a couple of the problems, as outlined in the ES report:

Each school has a different perspective on which data is important. And there is a torrent of potential information to be gathered and analyzed at the classroom level, from homework scores to behavioral incidents, which every school collects in its own way. As for ARIS data, teachers say that it can become quickly outdated, preventing them from acting in time to help students who are struggling. Teachers tell Anne LaTarte, director of instructional and data tools for the city’s Education Department, that although ARIS gives them good information, “it doesn’t move frequently enough.” Since much of the data in ARIS is generated at the end of the academic year, it becomes less relevant as the following year goes on. And even though ARIS contains data from periodic assessments, the information doesn’t flow instantaneously across databases. If a test is given in
January, it may not load onto ARIS until February. So teachers have to go into separate databases to get the latest scores.
Teachers also complain that ARIS is not as detailed as they would like it be. Properly designed assessments, for instance, can provide teachers with remarkably specific information about student performance; they can show how well a student seemed to grasp a math problem, for instance, or how fluently he was able to read certain letter pairings. Recently, the system was updated to allow educators to see not only overall student assessment scores, but also individual student performance on specific components of the state assessments.