University of Arkansas Office for Education Policy

Arkansas’ Struggling Readers

In The View from the OEP on October 10, 2018 at 3:12 pm

Today we are excited to release new research about Arkansas’ struggling readers. We thought since school performance reports are being released this Friday, it is a good time to remember that actual kids are behind the test scores used to generate the reports.  We hope you take a moment to reflect on who Arkansas’ struggling readers are, and how their reading skills develop through early high school.

We think this research is particularly important in light of all the effort that Arkansas educators are putting into improving early reading ability. By better understanding the historical improvement patterns of students who demonstrate low reading ability in third grade, we can better evaluate the effectiveness of the efforts to improve outcomes for struggling readers.

We examined the reading achievement of nearly 77,000 Arkansas students who were continuously enrolled in Arkansas public schools from 3rd grade through early high school. We hope you read the policy brief and more in-depth Arkansas Education Report, but we briefly summarize our findings here:


Who isn’t reading ‘on grade level’ in 3rd grade?

  • Students who qualify for free or reduced price lunches were twice as likely to be low-achieving readers in 3rd grade, compared to their more economically advantaged peers.
  • Students who are Black or Hispanic were twice as likely to be low-achieving readers in 3rd grade, compared to their White peers.

We know- you’re like “Duh” any teacher could have told you that, but it is important to have the data, the facts, about out struggling readers.

  • Males are somewhat more likely than females to be identified as low-achieving readers but the difference is not as large as it is between economic and racial groups.
  • ELLs are somewhat more likely than non-ELLs to be identified as low-achieving readers but the difference is not as large as it is between economic and racial groups.

Do the students who demonstrate low reading achievement in 3rd grade ‘catch up’ to their peers over time and what are the characteristics of students who do?  Note- we use standardized scores (z-scores) to examine student achievement over time due to changes in assessment.  You can read more about the methodology in the full report.

  • Of students who were initially low-achieving in 3rd grade, 12% ‘caught up’ to average state reading performance by early high school.
  • Students who were economically advantaged, White, Hispanic and/or female students were more likely to reach average reading achievement by early high school than their Black, male, and economically disadvantaged peers.
  • Among over 6,000 Black students who were identified as low-achieving in 3rd grade, only 6% demonstrated average reading achievement by early high school.
  • All types of low-achieving students demonstrated large improvements between 3rd and 4th grades, although rates of improvement after 4th grade is very different for different types of students.

 

Presented below are the standardized scores of initially low-achieving students from 3rd through 10th grade.  Results are presented by FRL participation and by race.

Figure 1: Average Reading Scores of Initially Low-Achieving Students: Grade 3 through 10 by Economic Disadvantage (FRL) Status g3 reading frl

Figure 2: Average Reading Scores of Initially Low-Achieving Students: Grade 3 through 10 by Race

g3 reading race

None of these initially low-performing student groups, even White or economically advantaged students, caught back up to the state average as a group by early high school.

  • Hispanic and economically advantaged students are achieving almost a half standard deviation increase in achievement as a group, and White students are making approximately 0.4 standard deviation increase, while Black and low-income students are making closer to a quarter of a standard deviation increase in achievement.
  • Even though low-achieving Hispanic students initially have very low average scores, they make advancements comparable to those of White students, the most advantaged group. This is an exciting trend to observe because it indicates potential for a narrowing achievement gap between White and Hispanic students.

You might be thinking that there are differences in reading score improvement between Hispanic students who are identified as ELL and Hispanic students who are not. We were pleased to find that both ELL and Non-ELL students made large gains in reading achievement over time as presented in Figure 3.

Figure 3: Average Reading Scores of Initially Low-Achieving Students: Grade 3 through 10 by English Language Learner Status

g3 reading ELL

 


In summary, Arkansas students face large and persistent racial and socioeconomic disparities in third grade reading scores. Moreover, few of our students who are struggling readers in third grade ever catch up to the state average. And these are for relatively stable students, those who are continuously enrolled in our schools from grades 3-10.

Our hope is that Arkansas’ average reading scores will continue to increase and all students will grow to read proficiently, but it is evident that special attention needs to be given to low income and racial minority students and students who are struggling with basic reading skills in third grade.

Although some schools saw double-digit reading proficiency gains after RISE trainings in 2017, similar improvement was not reflected on 2018 assessments. Programs must be carefully monitored to determine what, if any, impact they are having on changing the long-terms outcomes for students who, as demonstrated in this research, are likely to continue to struggle to read proficiently throughout their educational experience.

Schools and districts should carefully examine the progress of their struggling readers and consider the effectiveness of any interventions or programs that are being implemented.  Although this analysis uses state assessments as the measure of student achievement, schools and districts should examine multiple measures, including high quality formative assessments, to evaluate progress in student’s reading.

We must continue to strive to ensure that all students are leaving elementary school as competent readers, equipped with the literacy foundation necessary for future academic success.

Advertisements

Unpacking School Performance Ratings

In The View from the OEP on October 3, 2018 at 1:52 pm

Arkansas school performance ratings and A-F letter grades will be released to the public on October 12th.

Here’s what we think you can expect:

1) Most schools will get the same Letter Grade as they did last year.

2) Schools serving a smaller percentage of students eligible for free and reduced lunch will be more likely to get an “A” or a “B” than schools serving a population where a greater percentage of students experience economic hardship.

3) Arkansas’ growth measure- a powerful indicator of students’ academic improvement over time- will still be over shadowed by single-year achievement and will still be challenging for educators and parents to understand.

____________________

In advance of the release, we wanted to review the purpose of a school performance report.

This is the true baseline year for Arkansas’ new accountability system.  The state has worked diligently to develop an accountability system that will support student learning, as presented in the theory of action presented below.

Theoryofaction

The idea is that if schools get good information about what is really happening in their schools, then they can make improvements that will improve outcomes for students.

The continuous cycles of inquiry will take time to develop and build, and will require some new feedback systems to support schools and districts as they work to identify needs within their systems.

But does a school performance report really give schools the tools that they need to improve?  It’s complicated, and not just for Arkansas! A report released yesterday provides some insight into how stakeholders feel about school accountability across the nation.

One of the big benefits of school accountability identified in the report has been the increased transparency and quality of information about what is going on in schools. The interpretation of the data and the ability of schools to interpret the data and develop a plan to improve learning and resource allocation is key to Arkansas’ plan, as presented below.

Cycle

Arkansas’ school personnel have been able to preview the school performance report since September 21, and we can tell that they have because ADE has made some updates to the reports and extended the private review deadline.

We appreciate the state letting school personnel review the information, and fully support the pursuit of high quality data in the system.  But we worry that school personnel are spending a lot of time trying to check the scores, as opposed to interpreting it and developing a plan to support student learning.

One thing that we think will help school personnel re-direct their time from checking a bunch of data points to developing a plan to support student learning, is to really understand what is driving the school performance score.

The Biggie- Achievement

The majority of the school performance score is determined by student achievement on the 2017-18 ACT Aspire English Language Arts and Mathematics performance of students in grades 3-8.  Wealthier schools will generally have higher achievement.  We know that achievement on assessments is negatively correlated with student risk factors such as poverty.  You can check out the relationship in our data visualization.

Schools and the public have had information about achievement since scores were released early in the summer, which was much earlier than in previous years. Schools may have difficultly, however, calculating what their weighted achievement score would be because student ELA scores not reported by the full range of categories used in the school performance calculations.  Additional information about cut scores for the full range of performance categories is included in the Final ESSA Decision Rules, and could be applied to student scores but that would take quite a bit of time!

Note: As we mentioned in our earlier blog– achievement on the ACT Aspire pretty much stayed the same as last year, but the weighted achievement scores will be lower than last year for most schools because ACT Aspire modified the criteria for ELA readiness. Thankfully, because of the forward thinking actions taken by the ADE to equate the scores with prior years and adjust the cut points for the letter grade, the lower achievement scores should not result in lower school letter grades across the state.

The Most Important (to us!)- Academic Growth

Here at OEP we feel that this is the most important piece of information in the school performance report because it reflects how much students at the school increased their academic performance over time compared to how much the average student improved.  We feel growth is a much more informative indicator of how schools are educating students than achievement, and are pleased that it is much less correlated with school poverty rates.

Unfortunately, schools can’t verify this growth information because it is calculated at the state level- using the relationship between current and historical test scores of every student in the state to develop a ‘predicted score’.  This score is then compared to each student’s actual score to determine if the student’s academic achievement as measured by the ACT Aspire assessment was more than expected, as expected, or less than expected.  These student-level scores are averaged at the school-level and reported as a reported to the school as a transformed variable with a mean of 80 and a standard deviation of around 3. We are confident that the calculations are correct, and would advise schools to worry less about re-calcaulting the values themselves (which they can’t due to only having access to their school’s data) and more about understanding what this indicator means.

Many schools throughout the state are familiar with NWEA’s MAP Growth, where there is a target score required for students to meet annual growth.  This makes it easy for schools to identify if students met or exceed growth.  NWEA has information with 370 million test event records spanning more than 15 years, so they have a really good idea of how a typical student will increase their score over time.  ACT Aspire is a relatively new assessment however, so we want to make sure that we aren’t ‘guessing’ how much a typical student ‘should’ increase.   Instead, the state uses real data to inform how much a typical student DID increase from one year to the next and then compares that to the performance of students with similar test score histories.

Growth is really the most meaningful at the student level.  If students in program X are not meeting growth expectations, while students in program Y are, then careful consideration should be given to re-allocating resources so more students can benefit from program Y.  In discussions with the ESSA advisory team yesterday, we were thrilled to hear that the state may be able to provide student-level growth information in the future which would be super valuable to school leaders as they develop a plan to enhance student learning and resource allocation.

The Most Distracting- SQSS

The School Quality and Success (SQSS) indicator is a mouthful, but is really the smallest contributor to the overall school performance score. Since parts of SQSS reflects achievement, it is not surprising that SQSS scores are also negatively correlated with school poverty rates (r= -0.48).

As we have said before there are a lot of indicators included in this measure and school personnel may be spending a lot of time focusing on each indicator and wondering if the data are accurate. For some indicators, schools could verify the data through their own systems by applying the business rules, but for other indicators they cannot.  The first time schools saw the SQSS indicators was last spring, and the data included in the current school performance reports was pulled soon after.  Because SQSS indicators represent systems in place at schools, such as attendance reporting practices and course enrollment, and because these systems may require some time to adjust, we don’t expect to see large (or meaningful) changes in these scores yet.

We like how SQSS indicators can help schools get more accurate information about what is happening at their school, but are looking forward to when they are presented in a way that schools can really use them in their strategic planning to support student outcomes.

The continuous cycles of inquiry will take time to develop and build, and will require some new feedback systems to support schools and districts as they work to identify needs within their systems.



What’s in a Grade? 

The school performance report also supports Arkansas’ legislation that every school must receive an A-F letter grade. The letter grade was designed to create a method for parents to easily understand the quality of a school, but does an A-F letter grade really give parents the information that they need about how a school is doing? One stakeholder in yesterday’s report captured the challenge of assigning schools letter grades:

“How do you make something that is simple enough to be understood, like an A through F rating system, but also incorporate a number of different factors that are complex enough to capture all of the things we want schools to do? Everything from math and reading to also discipline data or enrollment data or attendance data or all these other sort of facets of that system. So how do you make something that is usable and understandable, but also nuanced?”

Another stakeholder pointed out how the A-F grade represents what matters to the developers of the metrics:

“I sort of feel like the single rating of either A through F or on a number is sort of the worst impulses of accountability. Because not only are you saying what matters by its inclusion in that, but how much it matters, by how it’s weighted. So man, that takes a lot of faith in yourself that you can specify how much you should care about academics relative to attendance, relative to these other things.”

We agree- and have addressed before how although the intention was to make growth scores weight more heavily in the school performance reports and associated letter grade determination.  When schools where students make the largest improvements in achievement can still be saddled with a low grade due to the characteristics of the population they serve, we are sending the message that growth doesn’t really matter.   And as long as the performance index results in schools serving more advantaged students getting higher letter grades, we also send the message to parents that what makes a good school is not the learning that happens inside the building, but how large the houses are that surround the school.

We urge educators and parents to focus on the academic growth indicator, and view the purpose of a school performance report as the beginning of an ongoing conversation about how to continually increase student learning. 

Arkansas Discipline Update

In The View from the OEP on September 19, 2018 at 11:36 am

Last week, the Student Discipline Task Force submitted their report to the Arkansas State Board of Education, and OEP was pleased to present the annual report on student discipline.

OEP’s report examines student discipline in Arkansas public schools. We identify trends and a number of key student outcomes related to student discipline in the Arkansas public schools. While the data are only limited to what schools report, there are several meaningful findings from this work. While we recommend the full report and this introductory policy brief, today we wanted to share the highlights of what we found.


What are trends in reported student infractions and associated consequences?

  • There has been an 87% increase in reported discipline infractions since 2012-13, with over 270,000 discipline referrals in 2016-17. We believe the increase in referrals likely reflects greater focus on reporting discipline infractions as opposed to an increase in misbehavior in Arkansas schools.

discipline1

  • Over 80% of discipline referrals are for insubordination, disorderly conduct, or “other” infractions.
  • The majority of the increase in infraction referrals has been for “other” infractions. In 2016-17, additional reporting categories were included, but over a third of infractions remained identified only as “other”.

discipline2

  • Over 93% of discipline consequences are for out-of-school suspension (OSS), in-school suspension (ISS), or “other” action. There has been a decline in reported reliance on OSS, ISS, and corporal punishment over time.
  • The majority of the increase in consequences has been for “other” actions. In 2016-17, additional reporting categories were included, but about 19% of consequences remained identified only as “other”. While trends away from exclusionary discipline might indicate benefits for students, knowing more about what the “other” consequences are is important for understanding whether this represents a meaningful change for students.

discipline3


Are schools complying with Act 1329, which bans the use of OSS as a consequence for truancy?

  • The use of OSS for truancy declined from about 14% of all truancy cases in 2012-13 to about 7% of cases in 2016-17.
  • In 2016-17, 76 schools reported at least five or more truancy infractions and reported using OSS in at least 10% of those cases. Many of these were concentrated in a few districts.

discipline4


Are there racial or programmatic disproportionalities in school discipline?

  • Disproportionalities by race, free- and reduced- price lunch eligibility, and special education status exist both in terms of the number of referrals for infractions of various types, as well as in the likelihood of receiving exclusionary discipline, conditional on referral for a particular type of infraction. For example, black students receive 117.6 referrals per 100 students, relative to only about 37-40 for white students, Hispanic students, or students of other races. Then, conditional on being written up for any infraction, Black students receive OSS, expulsions, or referrals to ALE in about 25% of these cases, relative to only about 15% for students of other races.

discipline5


Which types of schools are High-Exclusion schools?

  • Certain types of schools in the state are more likely to administer lengthy exclusionary punishments: schools with greater proportions of black students, high schools and middle schools (relative to elementary schools).
  • There also appears to have been a decline in the severity used, on average, between 2014-15 and 2016-17.

What is the relationship between student absenteeism and exclusionary discipline?

  • There is a moderate correlation between student absenteeism and OSS days received, with the strongest correlations between grades 7 and 10.
  • Students marked as chronically absent in those grades received 0.5 to 0.64 more days of OSS on average, compared to those not chronically absent.
  • This suggests that schools seeking to tackle absenteeism may consider discipline reforms as one possible solution.

discipline6


What is the relationship between educational attainment and exclusionary discipline?

  • Exclusionary discipline in high school (and particularly ninth grade) is associated with lower likelihood of high school graduation and lower likelihood of enrolling in college, conditional on a variety of student characteristics as well as baseline achievement in eighth grade.
  • The magnitude of these relationships decline after controlling for the behaviors (types of infractions) reported, although there is still a small relationship detected in some cases.

discipline7

 


Act 1329 has provided an opportunity to examine student discipline the the state.  The issue is complex, but one of the Student Discipline Task Force recommendations was that schools consider alternatives to exclusionary discipline practices when addressing student behaviors. The Board expressed concerns that OSS continues to be used as a consequence for truancy, and discussed the importance of continued effort to support schools and communities in reducing the use of exclusionary discipline for students. We look forward to continued discussion about improving learning environments for Arkansas students!