University of Arkansas Office for Education Policy

ACT Aspire: The Summer Sequel with a Bummer Ending

In The View from the OEP on July 12, 2018 at 4:47 pm

This summer it seems like all the movies are sequels: and the preliminary scores from the spring administration of the ACT Aspire for the 2017-18 school year seems like a sequel too!

Unfortunately, even though student performance hasn’t changed, ESSA Weighted Achievement scores will decline for most schools!! This bums us out because we hate it when systems send inconsistent messages about how well schools are doing. 

In today’s blog we share new data visualizations, review ACT Aspire performance patterns by subject overall and by grade, and explain why your school’s ESSA achievement scores will likely be lower this year than last year.


Show me the Data!

We are excited about some new interactive data visualizations of performance patterns across the state. Maps are available for both district and school-level, and you can select specific districts/schools and see how they scored. Unlike other data currently available, we have combined the grade level results to provide an OVERALL score for each district/ school.

We use the OEP GPA as this overall indicator of performance. The OEP’s GPA is a weighted measure of student performance that gives the most credit to students who have exceeded expectations and the least credit to those that are in need of support. In this GPA measure, we treat the ACT Aspire test scores similar to the familiar grade point average for individual students: 1.0 is the lowest score, indicating that all students in a districts were In Need of Support, while 4.0 is the highest score, indicating that all students in a districts were Exceeding Expectations on the ACT Aspire.

ACT Aspire_map

As you can see in the maps above, districts in the upper left hand half of the state are more likely to be blue, indicating higher performance. This is not surprising since we are showing performance on the ACT Aspire, which is highly correlated with the percentage of students in the district participating in the Free/ Reduced Lunch program.

If you want to see more detailed district- and school- level information, both by grade level and OVERALL, you can find it on the OEP website here! Also, check out our interactive data visualizations of district and school performance. You can select specific districts/schools (no limit) and see how they scored, or limit the view to particular regions or levels of FRL participation.


ACT Aspire Performance:

The figure below presents the percentage of Arkansas students who met or exceeded expectations in each content area by year since we began administering the ACT Aspire in 2016. Overall, the percentage of students meeting or exceeding benchmarks stayed exactly the same in mathematics, science, English, and reading. This year, unlike prior years, writing scores were not provided. Note: From here on, we use the Benchmark percent meeting or exceeding benchmarks so we can compare Arkansas performance with national results.

Figure 1. Percentage of Arkansas students meeting or exceeding expectations on the ACT Aspire, by content area, 2016 to 2018.

ACT_Aspire_18_Text

While overall ACT Aspire results did not change at all, there were some changes by grade level within each content area. We examine each content area by grade level and consider the national average performance as well.


Math: Performance in math declined somewhat in the middle grades, with 6th grade evidencing the greatest decline from the prior year (-6 percentage points). Since 6th grade experienced an increase of 7 percentage points in 2016-17, this year’s performance is similar to that of 6th graders in 2015-16. There were increases in the percentage of 8th, 9th, and 10th graders meeting or exceeding expectations. This continues a pattern of improvement in these grades, which is good news, since these grades have lower math performance than earlier grades. Math performance in 7th and 8th grades was above the national average (indicated by the red diamonds).

Figure 2. Percentage of Arkansas students meeting or exceeding expectations on the ACT Aspire Math, by grade, 2016 to 2018.

ACt_Math_Grade_2018


Science: Performance in science was generally consistent with prior performance, but declined somewhat in the middle grades, with 6th grade again evidencing the greatest decline from the prior year (-3 percentage points). Even with the decline, however, 6th grade maintained the highest performance across the grades. Science performance in 4th, 7th, and 8th grades was above the national average (indicated by the red diamonds).

Figure 3. Percentage of Arkansas students meeting or exceeding expectations on the ACT Aspire Science, by grade, 2016 to 2018.

ACT_Science_2018


English: Performance in English was generally consistent with prior performance. Scores for 8th grade students increased by 2 percentage points. English performance in grades 3-8 was above the national average (indicated by the red diamonds).

Figure 4. Percentage of Arkansas students meeting or exceeding expectations on the ACT Aspire English, by grade, 2016 to 2018.

ACT_English_Grade_2018


Reading: Performance was generally consistent with prior performance. The greatest increase was for 5th graders, with a 4 percentage point increase. Reading performance in grades 3-8 was at or near the national average (indicated by the red diamonds).

Figure 5. Percentage of Arkansas students meeting or exceeding expectations on the ACT Aspire Reading, by grade, 2016 to 2018.

ACT_Reading_Grade_2018


Plot Twist:

Every good sequel has a plot twist, and this year’s ACT Aspire results are no different. The one that has everyone scratching their head (okay- maybe just us here at OEP) is the updated cut-points for ELA and STEM readiness benchmarks. As you can see in the figure below, these new cut points (which are, according to ACT Aspire, more aligned with the performance expectations of the ACT) decreased the percentage of students meeting readiness benchmarks. In ELA, the change was a decrease of 8 percentage points from last year, but in STEM the new cut points resulted in a 25 percentage point decrease.

Figure 6. Percentage of Arkansas students Meeting Readiness Benchmarks on the ACT Aspire.

ACT_readiness_2018

The decrease in STEM readiness is startling, especially considering that we saw NO CHANGE in statewide math or science performance, but won’t have an impact on schools because STEM readiness is not used in accountability.

The decrease in ELA readiness, however, will impact schools because ELA scores are used to calculate the Weighted Achievement score in the ESSA school index. As we’ve discussed before, although stakeholders intended for growth to ‘count more’, Achievement continues to be the primary factor driving school letter grades. On a positive note, however, these changes to the ELA cut points should not impact growth scores in any way!

Decreases in ELA readiness are apparent in grades 4-10, with declines in the double digits for 6th, 7th, and 8th grades. This means that schools serving students in these grades are going to receive LOWER achievement scores on the ESSA index even though performance in English and reading was unchanged. It is unfortunate that this change was made, as Arkansas schools were just gaining familiarity with the ACT Aspire and the new ESSA metrics.

ACT_ELA_2018_arrow


So- what are the big takeaways from the preliminary ACT Aspire results? 

  • Performance on the ACT Aspire is related to school/ district poverty rates
  • Performance is generally the same as last year
  • Arkansas is scoring at or near the national average in most subjects/ grades
  • ELA cut points for meeting readiness benchmarks were changed this year by ACT Aspire, so schools should expect to have lower Weighted Achievement scores than last year on the 2018 ESSA reports
  • Growth scores should be unrelated to changes in the ELA and STEM cut points.
  • You can use the data resources from OEP to see overall district and school values
  • Data visualizations can help us see statewide patterns in performance and compare performance to other schools/ districts of interest.

We are happy to be able to share these resources with you and looking forward to seeing the Growth Scores on the ESSA reports in September. Stay tuned to OEP for more info.

Update 7/23: The cut scores for 2018 A-F school letter grade assignment have been lowered to compensate for the new ACT Aspire ELA cut points. You can find the new cut points here. 

 

 

Advertisements

School Report Cards for NWA and Pulaski County (new!)

In The View from the OEP on June 27, 2018 at 2:48 pm

NWARC        PCRC

Today we are pleased to release the 2017 Northwest Arkansas and Pulaski County Education Report Card.  These report cards provide an easy to understand overview of how students in the area schools are performing. OEP has produced a regional review of Benton and Washington county schools for several years, but this year is the inaugural release of a report card that focuses on the school systems in Pulaski County.

The report cards are in a ‘dashboard’ format that makes it easy for educators, school administrators, parents, and policymakers to see how school districts and charter schools are performing. Performance on key measures is broken down by Elementary, Middle, and High School levels and compared to regional and state scores.  For large districts, the report cards also include individual school data, where percentile ranks make the achievement and growth scores easy to interpret.

Springdale

The Report Cards put district-level information about academic growth, academic achievement, school quality, and A-F letter grades into a one-page context for quick interpretation. The performance data used in the report card are from the 2016-17 school year, the most recent data available at this time.

These key metrics of school performance are reported by the ADE at the school level in ESSA reports, but we feel they are important to consider from a district level to examine how effectively the school system as a whole is educating students. Additional information on ACT scores and high school graduation rates, which are important outcomes for students at the end of the K-12 journey, are also included.  To help make the connection between district resources and student success, we also include the district’s student to teacher ratio and amount of money that each district spent per-pupil.


That’s a lot of information!  What is the most important?

We believe that the growth scores are the best indicator that districts are doing what’s really important: helping all students learn. Growth scores are less related to student characteristics than achievement scores, as districts serving fewer At Risk students don’t always have higher growth scores.  The Growth Score indicates how much the district’s students in grades 3-10 improved over time on state assessments in English language arts and mathematics. This score also includes how well non-native speakers are progressing toward English language proficiency.

An average district, where students are growing academically just as predicted, will have a growth score of 80. In some districts, however, students are demonstrating greater increases in their academic performance from one year to the next than we would have predicted. To have a ‘good’ growth score, to be in the top 25% of schools in the state, Elementary schools need a growth score of 83 or higher, Middle level schools need a growth score of 82 or higher, and High Schools need a growth score of 81 or higher.

legendgrowth_sdale


How Are NWA Schools Doing?

Overall: Northwest Arkansas students are demonstrating greater growth in achievement and earning higher scores on the ACT Aspire than are the students in the state overall. Schools in NWA also have higher School Quality and Student Success scores, higher graduation rates, and are more likely to receive an “A” or “B” rating than are other schools across the state.

Academic Growth: Springdale School District had the highest overall growth score among the traditional districts, and was the only one where students demonstrated high academic growth at all levels: Elementary, Middle, and High School. Springdale students at the Elementary and Middle levels demonstrated the greatest academic growth in NWA, and nearly 70% of Springdale schools are in the top 10% of schools in the state for academic growth. Haas Hall Academy and Haas Hall Bentonville had the highest growth scores at the High School level. Many NWA districts had high growth at one or two levels, and we recommend they focus on identifying how they are supporting student learning at the schools where students are not demonstrating high growth overall.

Academic Achievement: Haas Hall Academy and Haas Hall Bentonville have the highest point-in-time achievement in the NWA area, with both NWA Classical and Arkansas Arts Academy joining in outperforming traditional districts in achievement. Bentonville School District had the highest achievement score among the traditional districts, and students at the Elementary levels demonstrated the greatest academic achievement in NWA. Since point-in-time achievement is so reflective of student demographics, we want to point out that among NWA districts where more than half of the students are eligible for the free/ reduced lunch program, Rogers, Gentry, and Siloam Springs reported the highest achievement.

School Quality: Lincoln School District had the highest School Quality/ Student Success Indicator Score at the Elementary Level. Bentonville School District had the highest Indicator Score at the Middle Level and Haas Hall Academy and Haas Hall Bentonville had the highest School Quality/ Student Success Indicator Score at the High School level.


How Are Schools in Pulaski County Doing?

Overall: Pulaski County students are demonstrating greater growth in achievement at the Elementary and High School levels than are the students in the state overall, and academic growth for students in Middle level schools is similar to the state average. This is particularly noteworthy since schools in Pulaski County serve a higher percentage of students who are likely to be at risk for not achieving in school than are served by the state overall. Pulaski County schools have lower academic achievement, School Quality and Student Success scores, and graduation rates than students in the state overall. Due to lower scores in these areas, Pulaski County schools are less likely to receive an “A” or “B” rating than are other schools across the state.

Academic Growth: eStem students demonstrated the greatest academic growth overall, with students at the Elementary, Middle, and High School levels receiving high growth scores. North Little Rock School District had the highest overall growth score among the traditional districts, reflecting high growth at the High School level. Little Rock School District had the highest growth at the Middle level and Pulaski County Special School District had the highest growth at the Elementary level.

Academic Achievement: eStem also had the highest point-in-time achievement of the Pulaski County area schools, followed closely by Academics Plus. Pulaski County Special School District had the highest achievement score among the traditional districts, overall and across all school levels. Since point-in-time achievement is highly correlated with student demographics, we want to point out that among districts where more than half of the students are eligible for the free/reduced lunch program, LISA Academy and Jacksonville Lighthouse reported the highest achievement.

School Quality: eStem had the highest School Quality/ Student Success Indicator Score at the Elementary level, Exalt Academy had the highest score at the Middle level, and Quest Academy was the highest scoring in School Quality at the High School level. Among the traditional public schools, Jacksonville had the highest School Quality/ Student Success Indicator Score at the Elementary level, Little Rock School District and Pulaski County Special School District tied at the Middle level, and North Little Rock was the highest scoring in School Quality at the High School level.



What’s the Takeaway?

In both the NWA and Pulaski County region, there are educational settings where students are demonstrating high growth by making larger academic gains than predicted based on their past performance. We want to point out that high academic growth can be found at all different types of schools:

  • schools like Haas Hall Bentonville that serve few “at risk” students,
  • schools like Covenant Keepers where 95% of students participate in the free/reduced lunch program and 30% are non-native English speakers,
  • schools like Janie Darr Elementary in Rogers which has high academic achievement,
  • schools where academic achievement is low,
  • open-enrollment charter schools, like eStem High and
  • traditional public school districts, like Springdale.

Growth scores for schools in NWA and Pulaski County also appear unrelated to resources like per-pupil expenditure or student:teacher ratio. Here at the OEP, we think growth scores are a meaningful reflection of increased student learning, and that high growth scores can be achieved by any type of school.

  • To have a ‘good’ growth score, to be in the top 25% of schools in the state, Elementary schools need a growth score of 83 or higher, Middle level schools need a growth score of 82 or higher, and High Schools need a growth score of 81 or higher.
  • If your school or district received a growth score of 80, students are demonstrating average growth in their academic performance on the state assessments in English language arts and mathematics.
  • If your school or district received a growth score below 78, students in your school or district are less likely to demonstrate academic growth than in the majority of schools in the state, and you should look for the reason.   Remember that unlike achievement, student characteristics like poverty are not highly related to growth.

On a side note: this is the first year for School Quality and Student Success scores to be reported. The School Quality score reflects a variety of indicators, and there may be a lack of consistency in how different districts report them, so here at OEP we are not freaking out about those scores.

If you want to know more about your school’s performance, check out myschoolinfo and type in your school name.  Under the “Reports” tab you can find the “ESSA report” for your school.

We hope that these report cards stimulate meaningful discussion about the educational settings within the communities, and look forward to hearing your thoughts. We invite you to share these report cards with those who are curious about the state of education in Northwest Arkansas or Pulaski County.

For more information about current education issues, check out OEP’s Policy Briefs and Blog.  If you are interested in digging into data, head on over to our website, where you can dive into all of the publicly available data on demographicstest scores, and finances.  The more we can be informed, share the good news, and look for ways to improve, the better Arkansas education will be.

If you would like a printed copy of a report card, please send us an email at oep@uark.edu and let us know which one and where you want it sent!

Summer is fun, but the Fall holds Promise

In The View from the OEP on June 20, 2018 at 2:10 pm

El Dorado Promis logo

We’re reaching the middle of summer—the time when college dorm room furniture starts popping up in prominent store displays and kids run past shelves of notebooks on sale because it’s too soon to be thinking about going back to school. While we’d all love an endless summer, classes, homework, and late night studying will again be the norm for college students across the state. For many students and families, the start of college brings anxiety not just about schoolwork, but about the rising cost of tuition and fees. According to the College Board, 60% of Bachelor’s recipients in the 2015-16 school year graduated with some debt. Concern about college debt may deter students from attending college, or it may lead students to work instead of study while in college. Thanks to the El Dorado Promise, hundreds of students coming to campus won’t have this anxiety impacting their postsecondary decisions.

The El Dorado Promise, announced in 2007, guarantees a full scholarship to all students who attend El Dorado School District from kindergarten through 12th grade, and a partial scholarship to students who attend at least from 9th-12th grade. The scholarship is capped at the maximum cost of tuition fees for an in-state resident at any public university in Arkansas—in the 2017-18 school year, this was a little over $9,000/semester. Students can combine this scholarship with other forms of financial aid, like the Arkansas Lottery Scholarship or Pell Grants, up to the full cost of attendance at any accredited university in the country. The Promise has impacted thousands of students from El Dorado since its establishment by Murphy Oil in 2007, and has now been around long enough for us to be able to analyze its impact.

In a recent paper and policy brief, we at the OEP asked what impact the El Dorado Promise has had on students’ postsecondary outcomes. Specifically, we wanted to know whether the Promise increased the rate at which El Dorado graduates enrolled in college and completed a Bachelor’s degree. To do this, we worked with folks at the El Dorado Promise office and the National Student Clearinghouse to gather data on whether students had received a Promise scholarship, if they went to college, and if they earned a BA within 6 years of graduating high school.

We can’t know the impact of a program like the Promise just by looking at how postsecondary enrollment and graduation rates have changed over time, because there are multiple factors that impact students’ college decisions—for example, during the Great Recession more people went to college to delay entering the workforce, and as the economy gradually recovered more people went straight to work after high school. So, we conducted what’s called a difference-in-differences analysis. We compared students who were eligible for the Promise (e.g. attended the district from at least 9th-12th grade) with students who weren’t eligible for the Promise (e.g. transferred to El Dorado in 10th grade or later). We examined the difference in college enrollment and completion rates between these two groups before the Promise was introduced (students who graduated between 2004 and 2006) and the difference in enrollment and completion rates between these two groups after the Promise was introduced (between 2007 and 2016 for enrollment, and 2007 and 2011 for completion). The descriptive results are shown in Figure 1. From this, it looks like the Promise was associated with a 16.5 percentage point increase in enrollment. That’s great news for El Dorado students!

pre and post promise chart

But, we wanted to be confident in our results, since students who were and were not eligible for the Promise could have been different in other ways besides mobility. We put this basic analysis into a regression framework, so we could control for things like high school GPA and student demographics. When we did this, we found that the Promise led to an 11.4 percentage point increase in enrollment, and a 10.7 percentage point increase in 6-year BA completion! 

In past work by folks at the OEP, we’ve found that the El Dorado Promise attracted families to the area, increasing enrollment in El Dorado Public Schools, and that the Promise led to a change in culture throughout the district—students, teachers, and parents are more focused on ensuring all students are ready for college. These efforts are paying off—the Promise is helping students get to and through college!

There’s a lot to celebrate about summer vacation—getting to spend time with your kids and/or friends, a chance to sleep in, and a reason to get outside. But there’s also a lot to look forward to in the Fall—including, for many students, a Promising postsecondary experience.