Proficiency Index calculations for JC McKenna and EHS

As promised, I ran calculations of the Proficiency Index used to determine AYP compliance for JC McKenna and EHS, both for students with disabilities and students without disabilities. I have to put in a disclaimer on the EHS calculations: there are only 21 students with disabilities in Grade 10 so the government does not consider this subgroup separately in determining EHS AYP compliance. According to the government, there are accountability indices in place for small schools, but they don't really give much more information than that.

JC McKenna flat out wins the Evansville Schools award for achieving AYP without invoking any Confidence Intervals or "safe harbor" requirements whatsoever. There are 53 students with disabilities enrolled and they attained a 78.4% Proficiency in Reading (compare to the target of 74%) and 64.2% Proficiency in Math (compare to the target of 58%). Students without disabilities (305 students) scored 97% and 95.6% respectively in Reading and Math this year. Good show to JC McKenna! Keep up the good work Mr. Flaherty and crew!

EHS students without disabilities achieved Proficiency Indices of 92.6% in Reading (vs. 74%) and 89.2% in Math (vs. 58%). AYP is achieved at EHS. To gauge the progress of EHS students with disabilities, I also calculated the Proficiency Indices for this subgroup. If this data were included for AYP, Evansville would be in trouble with scores of 52.3% in Reading and 42.9% in Math in 2009-2010. In a not too distant future, I envision the district enrollment increasing to the point that these data will be included in our AYP designation. Or there may be changes in the law that hold small schools more accountable to this group of students. So I decided to run a "safe harbor" calculation on this data to see if EHS students with disabilities would qualify under those circumstances. Only the Math achievement qualified with a 22.6% decrease in non-proficient students from a year ago. That was reassuring. The Reading achievement for these students, conversely, showed a 9.8% increase in non-proficient students in one year. One has to keep a perspective that these are pretty small sample sizes, but a district wide trend of gradual increases in non-proficient students seems to be developing in Evansville. I'm sure this data is one reason Mr. Everson asked the Board to fund a Reading Strategies Class beginning next year at EHS. Good proactive approach, Mr. Everson. I'm anxious to see the impact this class has on EHS scores in the next few years.

The bottom line is that TRIS isn't alone in their struggles with AYP in our district. Their Students with Disabilities enrollment meets the minimum to require TRIS to include these data in their AYP reports and the High School grade 10 enrollment is not sufficient in this subgroup to require inclusion in AYP reports. As the targets increase to 100% Proficiency Indices over the next 4 years, more and more subgroups will begin to challenge the system. I predict that the next at-risk group in our district will be the students with economic disadvantage. These scores are typically 10% less than those of our students without economic disadvantage. What we have here is a bad news, good news sort of scenario. We have a gap between the achievement of these two groups, but it is smaller than the statewide gap for two reasons: 1) our disadvantaged kids score better than those in the state as a whole and 2) our non-disadvantaged kids score lower than those in the state as a whole. The trick is to increase both groups at the same time, and my next post will address this issue as well as further delve into the possibility of gradual declines in our achievement data over the last 5 years, as measured by WKCE data. Stay tuned for "Eagle Eye View: ECSD 5 year Achievement Index"

How did TRIS meet Adequate Yearly Progress?

Explaining the Standards

There are three criteria that must be met to satisfy AYP at both the school and district level to continue to receive Federal Title I funding. The first standard to meet is a Test Participation Rate of 95% or greater for each subgroup for which there is a sufficient number of students (40 or more). The second benchmark is a Graduation Rate of 85% or greater, or a 2% improvement in graduation rate over the previous year if less than 85%. For schools or districts that do not graduate students, an Attendence Rate of 85% or more must be met. The final hurdle in meeting AYP is meeting the ever increasing Math and Reading Proficiency Indices for each school in every subgroup for which there is a sufficient population (40 or more). This is where TRIS "failed to meet AYP" in Reading in the Students with Disabilities subgroup in 2008-2009.

If a school misses AYP two years in a row, they are placed on a School Identified For Improvement (SIFI) list and Title I sanctions will begin. Meeting the failed objective (and I presume maintaining all the rest of the standards) in year 3 places that school on the "Improved" list and only after meeting the failed objective two years in a row does the school meet "satisfactory" AYP. Districts have a little more leeway to be determined as failing to meet AYP. The district must fail to meet the same objective in all schools for which the standards are measured before they fail to meet AYP. In our case, the district would fail to meet AYP only if, just for example, the Students with Economic Disadvantage failed to meet the Reading standard at all three schools (TRIS, JCMcKenna and EHS). It follows that if each school fails a different standard, each schools will fail to meet AYP, but the district remains in compliance. It's kind of a shell game, but the system is designed to distinguish true district wide issues from annual grade fluctuations, especially in smaller districts like Evansville.

Evansville's Data

I have delved further into this year's data and calculated TRIS's Proficiency Index for students with disabilities. Click on the post for a description of the formula used for this calculation on page 4. The Reading result of 54% for this subgroup still does not meet the Reading Target of 74% dictated by the feds. It's better than the 38.7% I originally thought is was prior to giving half-credit for students achieving "basic" this year. However, it is still far below 74% and likely not statistically within the 99% confidence interval, so I had to ask how did TRIS regain AYP for this group of students?

Further scrutiny of the "explanation of AYP" document reveals a "Safe Harbor" clause in AYP designations (pp 4-6) that gives failing schools credit for decreasing the number of non-proficient students. Schools that show a 10% or greater decrease in non-proficient students will achieve AYP. TRIS has achieved a 13.8% decrease in non-proficient students with disabilities for the Reading objective. It's my best guess that this is how TRIS has recovered AYP status.


The 99% Confidence Interval I refer to above is associated with all statistical data. This CI is applied to schools that have relatively small populations of subgroups, but still large enough (>40) to be included in the AYP calculations. It measures whether or not a certain value is statistically distinguishable from the target value. I'm still trying to work with the DPI to understand how they set 99% confidence intervals for these types of data sets. It's possible but improbable that the 54% Proficiency index TRIS earned this year falls in the confidence interval of being statistically the same as 74% for such a small data set. The number of students at TRIS in this subgroup is only 62, just 50% above the cut-off for subgroup size. More on this topic if the DPI gets back to me. They are "reluctant to assign cause" when a school goes off the failure list because they don't want to start a speculation game. Too late for me.

After I did the TRIS calculations, I became curious about the district as a whole. I will post district wide results of these calculations next. Since AYP for a district is assigned through individual school achievements, it's necessary to run each school individually, which will take me a while.

I also ran TRIS's math data. They only met the Math target last year by applying the Confidence Interval. Somewhat alarming is the continuing high percentage (38.7%) of students with disabilities scoring only "Minimum achievement" on the Math test. A "safe harbor" calculation on the math data shows a 26.8% decrease in non-proficient students since last year, meaning significant progress has been made. This is great news. Hopefully, the techniques they used to reduce reduce the numbers of non-proficient students will begin to move even more out of the "minimum" category.

My big concern is for next year when the Reading target moves up to 80.5% and the Math target moves up to 68.5%. My discussions with the DPI indicate that WKCE will not be replaced for at least another three years. MAPS testing, while a useful tool for teachers, is not a sufficient tool for the Federal Government to measure student achievement for the purposes of awarding Title I funds. So whatever replaces WKCE, it won't be the MAPS test as we know it. It also won't save the district before the huge increases in the NCLB targets start to be a serious issue for all students in the district. Our Students with Disabilities subgroup is simply the canary in the coal mine, a harbinger of things to come as targets approach the 100% proficient plus standards. And it won't just be Evansville that feels the pain. There is change coming to the law, according to the DPI. It remains to be seen what changes we will see, but I hope common sense prevails.