Testing Expert Responds to NECAP Controversy

Mar 25, 2013

Freelance policy analyst and liberal blogger Tom Sgouros is one of a number of critics raising questions about Rhode Island's use of the New England Common Assessment Program, or NECAP, as a graduation requirement. Specifically, Sgouros has argued that by its very nature, NECAP was designed to magnify differences in student achievement, thereby ensuring that a certain number of students will score poorly every year.

I'm no expert in standardized test construction, so I asked the Department of Education to respond to these claims. Here are the answers they provided, courtesy of Charles Depascale, a consultant at the Center for Assessment, a group that has worked with Rhode Island education officials from the early days of NECAP on issues related to assessment design and the use of test results.

On the assertion that NECAP was designed to rank schools and students and that a certain number of students are destined to fail each year, Depascale says:

  • The NECAP tests were designed specifically to evaluate student proficiency.
  • The NECAP tests were designed to meet the assessment and accountability requirements of No Child Left Behind.  Although a primary use of assessment results under NCLB was school and district accountability, the accountability model has shifted from ranking schools and students.  In the standards-based era of NCLB, contrary to “ensuring that a certain number of students will flunk” the measure of school accountability was the percentage of its students demonstrating performance at the Proficient level or higher and the goal was 100% of students Proficient.
  • The results of the Grade 11 NECAP Reading test bear this out.  On the most recent Fall 2012 test, 79% of students performed at the Proficient level or higher (up from 76% the previous two years) and 92% of students met the student graduation requirement of Partially Proficient.
    • One-third of grade 11 students (33 percent) scored at the highest achievement level on the Grade 11 Reading test.

Sgouros argues:

"In other words, very few of the questions are correctly answered by all students. In Appendix F of the 2011-12 manual, you can see some item-level analyses. There, one can read that, of the 22 test questions analyzed, there are no questions on the 11th grade math test correctly answered by more than 80% of students, and only nine out of 22 were correctly answered by more than half the students.

Put another way, if all the students in a grade answered all the questions properly, the NECAP designers would consider that test to be flawed and redesign it so that doesn’t happen. Much of the technical manual, especially chapters 5 and 6 (and most of the appendices), are devoted to demonstrating that the NECAP test is not flawed in this way. Again, the NECAP test is specifically designed to flunk a substantial proportion of students who take it, though this is admittedly a crude way to put it."

Depascale responds:

  • The item statistics cited in Appendix F of the 2011-2012 Technical Report apply to only the 22 1-point or 2-point short-answer and 4-point constructed-response items included on the test.  These items account for 40 of the 64 points on the Grade 11 Mathematics test.  Historically, these items which require students to produce a response are more difficult than the multiple-choice items which require students to select a correct response.
  • Item statistics for the multiple-choice items on the Grade 11 Mathematics test are presented in Appendix E of the same Technical Report.  Across those 24 items, 10 were answered correctly by more than half the students and two were answered correctly by at least 80% of the students.
  • Once again, however, results from the Grade 11 Reading test demonstrate that the item statistics cited by the author are more a reflection of student performance in mathematics than intentional test design.  On the reading test, there are 28 multiple-choice items.  Across those items, 27 of 28 were answered correctly by more than half the students, with 49% answering the remaining item correctly.  Additionally, at least 80% of students answered eight of the reading items correctly with 90% of students answering one of the items correctly.

Depascale goes on to address several other arguments and provides sample questions from the math portion of NECAP along with the percentages of students correctly answering those questions. Here's the full text of his response.