University of Arkansas Office for Education Policy

Growth, Poverty, and the Recognition Blues

In The View from the OEP on April 17, 2018 at 4:17 pm

school moneLast week, the ADE released a bunch of information about Arkansas schools, including A-F letter grades, state report cards, and ESSA reports.  Here at OEP, we feel that growth scores are the most important piece of information that was released, and today we want to share some more details about why growth is so important, and why some deserving schools may have missed out on the recognition and reward money.

Growth and Poverty

Growth is so important because it gives a different perspective on how well students are learning in a school, and is not as correlated with student demographics as achievement is. In the graph below, we present the weighted achievement scores and the % of FRL students enrolled at the school (a proxy for poverty).  Weighted achievement scores range from 2 to 105, and FRL rates range from 100% of students eligible to fewer than 5% (note: Haas Hall does not report FRL %ages and so are excluded from the graph).

The values are related in the way that we would expect (a lower percentage of FRL students= higher achievement), and are correlated at R=0.52.  There are some schools that have much higher than typical achievement given the % FRL in their student population, which is awesome, but in general, schools serving more FRL-eligibile students have lower achievement scores.

WA.png

By contrast, below we present the content growth scores and the % of FRL students enrolled at the school.  You can see that the values are not as related  (fewer FRL students doesn’t always mean higher growth), and have a lower correlation at R=0.21.  This is a good thing- because we want kids in all schools to be making growth in learning from one year to the next!

CG

You will also notice that the content growth values are all clustered around 80, making it is hard to tell a difference between ‘high growth’ and ‘low growth’ when the axis is scaled from 0 to 125 like the weighted achievement graph.  This is exactly what we mentioned in the OpEd last week– the growth values have a relatively small range (very small standard deviations) compared to the achievement scores.  Below we share a version of the content and a FRL graph with an ‘adjusted axis’ that runs from 70 to 90, so you can see differences in the growth scores.

CG_adj

With the adjusted axis, you can see differences in content growth scores! There are some schools with low-FRL percentages with high content growth in the upper right corner of the graph.  One example is Willowbrook Elementary in Bentonville, with 15% of students eligible for FRL and content growth score of 89.35.  There are also some high-FRL schools with high content growth scores which are in the upper left corner of the graph. One example is Jones Elementary in Springdale where 98% of students are eligible for FRL, 84% are identified as limited English, and a content growth score of 88.96.  Despite the differences in the student populations served by these two schools, students at both schools demonstrated high growth scores. This is something to celebrate!


The Recognition Blues

Arkansas’ School Recognition Program provides funds for “outstanding schools”.  Schools are rewarded for being in the top 5% (or the 6th to 10th %) in achievement and/or growth.

Given what we know about the relationship between achievement and FRL rates, it should not be surprising that Willowbrook Elementary (with 15% FRL) received reward money for being in the top 5% for achievement, and that Jones Elementary (with 97% FRL) did not. However, both Willowbrook and Jones Elementary received a reward and recognition money for being in the top 5% of content growth among Arkansas schools.

When we were examining who else was rewarded, we noticed that most of the money went to elementary schools. In fact, 59% of the performance rewards, and 65% of the growth rewards went to elementary schools. This made us scratch our heads.

We know that elementary schools are different from schools serving middle and high schools in many ways, but school level also matters when it comes to achievement and growth. In the powerpoint that summarizes the ESSA Indicators, descriptive statistics for each indicator is provided by school level: Elementary, Middle, or High.  You can find the rules for how schools were assigned a level here.  There are substantial differences between the school level groups on achievement scores. For example, let’s examine the achievement score received by schools in the top 5% of each school level.

  • Elementary level = 93.79,
  • Middle level = 91.85, and
  • High school level = 76.53.

The top 5% of elementary schools have higher achievement scores than middle schools and much higher achievement scores (+17 points  or greater than 1 standard deviation) than high schools. It makes sense, then, that about 7% of elementary level and middle level schools were rewarded for highest 5% achievement, but only 1% of high schools received reward money for achievement.

The differences for growth scores between the groups are not as glaring as achievement differences, but remember that the standard deviation is only about 3, so the top 5% of elementary schools have growth scores again about 1 standard deviation higher than middle and high schools.

  • Elementary level = 87.09,
  • Middle level = 84.71, and
  • High school level = 83.94.

We expected to find the top 5% rewards again dominated by  elementary schools, but were surprised to find that 7% of elementary level schools were rewarded for growth along with 5% of high schools. Interestingly, NO middle level schools were rewarded for growth.

We saw that the top 5% of middle schools and high schools have similar growth scores, so why are middle schools not getting recognized?  The recognition program for high schools includes graduation rate, (70% for growth and 30% graduation rate), which generally increases the growth score because graduation rates are typically higher than growth rates. With the deck stacked against them, not even J.O. Kelly, the highest growth middle level school in the state ( J.O. Kelly from Springdale) could crack the top 5% for growth.

The legislation for the reward program clearly states that schools will be rewarded for being in the top tier “of all public schools”, but here at OEP, we would love to see schools  awarded recognition and reward money based on their ranking WITHIN their school level.  Making this change would be more equitable for all schools, and would align more closely with the state’s ESSA plan. If we want to incentivize schools to achieve and show growth, we have to make sure schools in all levels have a chance for rewards and recognition.

Hey! If you want to see how your school ranks within schools serving similar grade levels , check our database. We ranked schools within their school levels, making it easy to identify the elementary schools with the highest achievement scores as well as the high schools with the highest growth scores.

Advertisements

NAEP Nuggets!

In The View from the OEP on April 10, 2018 at 3:31 pm

NAEP results were released today, and Arkansas’ results look about the same as they did in 2015. NAEP is administered nationally to a representative sample of students from all 50 states, so acts as a standard measure of student performance across states and time.

This trend of ‘meh’ was widespread across the country (although Florida had some strong gains!).  Here at OEP, we dug into the new results and are pleased to share six NAEP nuggets with you. You can learn more details in today’s policy brief!

 

NAEP Nugget #1:  Arkansas’ 2017 NAEP scores were essentially unchanged from the 2015 results BUT 2015 was a decline from 2013, so this is not great news because we were all hoping 2015 was a one-year-blip that we would bounce back from. In fact, as the figure below highlights, Arkansas scores were the highest in 2011 and 2013, and have trailed off since. Fingers crossed for 2019!!

NAEP 2017

Arkansas NAEP Scale Scores, 2003-2017

 

NAEP Nugget #2: 4th and 8th grade Math scores are lower than those of Arkansas’ border states (this group includes Louisiana, Mississippi, Missouri, Oklahoma, Tennessee, and Texas).  This is particularly a bummer in 4th grade because we outperformed them from 2005 to 2013!

4thMath

4th Grade NAEP Math Scores, 2003-2017

 

NAEP Nugget #3: 4th and 8th grade Reading scores are also lower than those of Arkansas’ border states. Again, this is particularly a bummer in 4th grade because we outperformed them from 2003 to 2013.

g4 reading

4th Grade NAEP Reading Scores 2003-2017

 

NAEP Nugget #4: Math score gaps between student groups widened in 2017 due to decreased performance of at-risk groups and increased performance of other students.

frlg4math

FRL NAEP Math Score Gap: 4th Grade 2003-2017

 

NAEP Nugget #5: 8th grade Reading score gaps between student groups decreased slightly in 2017, due to an increase in the scores for at-risk student groups.  Although the scores for black and FRL-Eligible students increased, they remain below 2013 levels.

bwg8reading

Black/White NAEP Reading Score Gap: 8th Grade 2003-2017

 

NAEP Nugget #6: ACT Aspire ELA performance is similar to NAEP Reading, but Math proficiency rates are higher for ACT Aspire than for NAEP. We need to pay careful attention to the difference between the NAEP and ACT Aspire math scores.  When we send and receive conflicting messages about how well our students are performing in math, it can make it difficult to determine how well our students are doing and which sorts of educational interventions are making a difference for our students.

ACTASpireNAEP

NAEP Proficiency and ACT Aspire Performance, 2017

We have our fingers crossed that the changes laid out in ESSA will make a big difference to student learning in Arkansas, and look forward to seeing NAEP results again in 2019.

Meanwhile- there is a lot more data coming out this week about Arkansas’ schools- follow OEP to get insight about what all the numbers really mean!

 

Growth Scores Matter

In The View from the OEP on April 10, 2018 at 11:00 am

Screen Shot 2018-04-12 at 11.28.10 AM

OEP wanted to share our OpEd that was published in today’s Democrat-Gazette.  We wrote it to highlight why we think a school’s growth score is critical to understanding how well a school is serving all students.  Although growth scores were supposed to be weighted more heavily in the ESSA index (on which the letter grades are calculated), in reality the schools with high growth received lower grades than schools with high achievement.

We created an interactive data visualization to help you see what we think is the most important measure of schools for parents, students, educators, and policymakers to understand.  You can also download the data behind the viz from our website. Check it out and let us know what you think!


 

School ratings miss opportunity

Posted: April 12, 2018 at 2:47 a.m.

Arkansas’ public schools are being assigned A-F letter grades, and we at the Office for Education Policy are always supportive of providing information on school improvement. But for a thorough understanding of how well our schools are doing, we must look beyond the school grades.

Letter grades are familiar to parents and students because teachers use them to communicate how well the student is performing in their class. Teachers can choose what counts most in their class; one teacher makes the final a huge part of the grade but doesn’t count homework for much, while another teacher counts every homework assignment and allows students who are doing well to skip the final altogether. Different approaches to grading send a signal about what is important.

Arkansas’ new school grading system was developed to send the signal that increasing students’ learning over time is more important than how many students at the school pass the annual test. We agree, but were disappointed to find that, in practice, schools with higher passing rates receive higher grades than those where students are growing more. Because the letter grade doesn’t reflect what we think is the real measure of school quality, we urge you to look beyond the grade.

The intention of the A-F school grades is to help parents and the public better understand how well a school is performing, but the current system still paints an incomplete picture and thus sends the wrong message about what matters. For years, since the No Child Left Behind legislation was signed in 2002, the measure of how well a school was performing was current achievement, measured by the percentage of the schools’ students who passed the state’s annual exams. Schools serving more advantaged students typically received “good” scores because a high percentage of their students passed, while schools serving a larger percentage of students who lived in poverty, participated in special education, or were learning English often were labeled “not good” because too few of their students were able to pass the test.

The clear connection between passage rates and student demographics suggests that point-in-time test scores were not a good measure of how well a school was educating students, but rather a reflection of the wealth of the community being served by the school. Critics (like us at OEP) suggested a better measure of school success would be based on student learning growth. Growth measures how much individual students at the school increased their scores from year to year. Using growth as a measure of school success levels the playing field because all students are evaluated by the extent to which they grow from their own starting point; thus, students facing socioeconomic barriers to achievement have the same opportunity for growth as their peers from advantaged backgrounds. All students can grow their understanding, and we should expect all schools to foster student growth, regardless of family income, first language, or learning needs.

It is true that Arkansas’ new grading system includes a category for student academic growth, alongside the category for current test passage rates. In fact, for elementary schools, growth counts as 50 percent of the grade, achievement counts for 35 percent, while “other” school quality indicators count 15 percent. Based on these numbers, it would seem that schools with high growth would get a better grade than schools with high passage rates, but it doesn’t work out that way.

In the current system, the overall school grades are influenced very little by student growth. For example, an elementary school with high growth and low current passage rates gets a “C,” while one with low growth but high passage rates gets a “B.” Even when a school is growing student learning better than 97 percent of the schools in the state (+2 standard deviations), if the school boasts only average passage rates, that school will earn a “B.” On the other hand, a school with average growth and very high passage rates will receive an “A.” Simply put, schools with high passage rates still earn better grades than schools with high growth.

The mismatch between what the letter grades were supposed to reward and what the grades actually reward is due to a mathematical problem of big differences in the variability of the measures for passage rate and for growth. Without wading too deep into the technical details, we can tell you that this issue won’t be difficult to fix, and we hope the Department of Education will adjust this for future school grades. In the meantime, however, we recommend you look past the overall grade and check your school’s growth score.

What is a really good growth score? It depends on the grade levels served by the school. Elementary schools with a growth score of 83 or higher (82 or higher for middle/ junior high, and high schools) are growing students’ understanding more than 75 percent of schools in the state.

If your school has a really good growth score, you should celebrate in a big way! Even though the overall letter grade may not reflect it, your school is doing what’s really important: helping all students learn.