University of Arkansas Office for Education Policy

Are Arkansas’ High School Graduates Prepared to Succeed in College?

In The View from the OEP on April 17, 2019 at 1:50 pm

Earlier this week, the Arkansas Department of Education released 2018 Report Cards for schools, districts, and the state (press release). Included in these data was information on the percentage of Arkansas students who graduate from high school, go to college, and earn college credits.

While college is not the right path for every Arkansas student, college degrees are increasingly valued by employers (Harvard Business School Report). Although there has been some recent pushback on employers’ degree requirements, to compete in tomorrow’s job market, Arkansas schools will need to prepare a large proportion of their students to succeed in college. Unfortunately, this is an area where Arkansas has some ground to make up. Only 22 percent of Arkansans age 25 and older have a bachelor’s degree, which is 9 percentage points below the national average (31%) with only Mississippi (21.3%) and West Virginia (19.9%) having  lower degree attainment (Census Bureau map).

Given our degree deficit and the priority the state has put on college readiness, we were eager to see if more of Arkansas’ students were hitting important milestones on the way to college graduation. We have written about this before (here and here), but at the time only three years of data were available. With yesterday’s release, we now have 5 years of data, but unfortunately, the story is not great.

There were, however, some encouraging signs in the data. For example, Arkansas students are graduating high school in greater numbers (see Figure 1). The state’s 4-year graduation rates inched up slightly, climbing to 89 percent in 2018. Minority students also saw increases over this period, with black students experiencing the biggest gains, increasing by 5 percentage points from 81 to 86 percent.

Figure 1: Arkansas 4-Year High School Graduation Rates 2014-2018

Screen Shot 2019-04-17 at 9.32.18 AM

Even though Arkansas’ students are graduating from high school at higher rates, they are not more likely to attend college (see Figure 2). The college going rate remained relatively flat over most of the 5-year period ending in 2018. However, if the recent data is to be trusted, college going rates declined by 8 percentage points in 2018. Such a large drop makes us question the validity of the 2018 data, but if accurate, this would be a huge deal that demands more discussion and monitoring.

***IMPORTANT NOTE:  ADHE confirmed to OEP that the college-going data are not correct and they are re-calculating the values. The initial re-calculation reflects a statewide college going rate consistent with prior years. *** We will update the analyses below with the new college-going data when released, but the general information still applies.  

Figure 2: Arkansas College Going Rate 2014-2018

Screen Shot 2019-04-17 at 9.35.43 AM

One important caveat about the state’s college going data is that the values only reflect students who attend a college in Arkansas. These rates miss Arkansas high school graduates who leave the state to pursue their post-secondary education, and so understate the state’s college going rate. The National Center for Educational Statistics produced a report in 2017 that can help us estimate by how much the rates are understated (see report here). In the fall of 2016, there were 3,318 Arkansas residents attending a college outside of the state. Assuming this number includes students who graduated at some point over the previous four years and dividing by the roughly 30,000 Arkansas high school graduates per year yields an estimate of approximately 3 percent for the percentage of Arkansas high school graduates in any given year attending college out of state. So our best guess as to how much Arkansas’ college going rate is underestimated is somewhere between 2.5 and 5 percentage points. The upshot is that even accounting for underestimation, Arkansas’ high school graduates enroll in college at far lower rates than the national average of 67 percent (https://nces.ed.gov/programs/coe/indicator_cpa.asp).

As we have done in previous blog posts, we can use the state’s data on graduation and college going rates along with the data on college credit completion to estimate how many of today’s 9th graders we can expect to get to college and earn at least one year of credits (figure 3). The picture is not encouraging. Of a group of 100 hypothetical 9th graders, we would expect 89 to graduate high school, 36 to enroll in college within the next year, and only 19 to complete one year’s worth of credits in the subsequent 2 years. So only 1 in 5 of today’s 9th graders would be expected to get to college and start down the path toward degree completion. And the story is worse for the state’s minority students.

Figure 3: Expected Education Attainment for Hypothetical 9th Grade Cohort

Screen Shot 2019-04-17 at 9.40.58 AM

Part of the challenge is that even once kids get to college, 63 percent of Arkansas’ high school graduates require remediation in one or more subjects. Remedial courses do not count toward degree completion, and can make persistence and degree completion seem out of reach.

College admission exams provide another measure of college readiness. All Arkansas high school students now take the ACT, and only 17 percent of 2018’s graduating class met the ACT’s college readiness benchmarks in all four tested subjects (i.e., English, math, reading, and science; Arkansas ACT report). Missouri, which also requires students to take the ACT and is similar to Arkansas in terms of its rural/urban mix, performed significantly better, with 22 percent of 2018 graduates meeting the college readiness benchmarks in all four subjects (Missouri ACT report). You can find a table summarizing states’ 2018 ACT performance here.

Although college readiness has been a priority for many years and the state’s institutions of higher education have been working to improve remediation and college persistence, it’s clear that we are not making fast enough progress. Arkansas is still far behind in college enrollment and degree completion, and we will need to redouble our efforts and try new strategies if we expect to compete on a global stage.

Note: If you want to dig into Arkansas’ graduation rate and college going rate data, the OEP website provides handy spreadsheets that stratify the data by district and school. You can find graduation rate here and college going rate here (the 2018 rate data table will be released after re-calculated data are released by ADE).

Are We Closing the Achievement Gap?

In The View from the OEP on April 10, 2019 at 1:23 pm

ednext_XIX_3_hanushek_img01Yesterday, EducationNext discussed new research demonstrating a persistent 50 year achievement gap between “Haves and Have Nots”, and we wanted to chime in about what the gap looks like for students in Arkansas.

While the national and Arkansas-specific research use different data sources and methodologies, the conclusions are the same: the gap in academic performance between students with fewer and greater economic resources is large and isn’t getting smaller.

The national research used an index of student socioeconomic status (SES) developed from student-reported information about the education level of their parents and possessions that they have in their home. The achievement data come from four testing programs: National Assessment of Educational Progress (NAEP) Long-Term Trend;  Main NAEP; the Trends in International Mathematics and Science Study (TIMSS); and the Program for International Student Assessment (PISA). The researchers found that the gap in achievement between the highest 10% SES and the lowest 10% SES students has remained consistent over the last 50 years.  The conclusion holds when the ranges are expanded to the highest and lowest SES quartiles as well.

What about Arkansas’ Gap?

Here at OEP, we looked into the achievement gap for Arkansas students and found similarly disheartening results.

For the Arkansas analysis, we use the student-level indicator of Free-Reduced Lunch (FRL) program participation as the indicator of student socioeconomic status because we don’t have information on parental education or home possessions. FRL eligibility is based on family income, and is often used as an imperfect proxy for a student’s socioeconomic status. We group students into two groups: Not participating in FRL, or participating in FRL. In 2008-09, the first year of our analysis, 56% of Arkansas students participated in FRL.  The percentage increased to 59% in 2009-10, and has remained fairly consistent since then, ranging as high as 61% before declining slightly to 60% in 2017-18 .

For the achievement data, we use ten years of the annual Arkansas state assessments, which have changed over time. For the first six years of our analysis we use data from the Benchmark and End-of-Course exams. In the 2014-15 school year, students in Arkansas completed the PARCC exam, before switching to the ACT Aspire exam for 2016-17 and beyond.

All of these test changes have made it essentially impossible for Arkansas schools to examine achievement gaps over time, because each test scores and reports performance in different ways. Although the assessments have varied over time, each provides a measure of how students statewide perform in literacy and mathematics. We think it is very important for education stakeholders to be able to examine performance over time, so we used a common standardizing procedure to track the relative performance of different groups.  We transform these scores into percentiles to aid in interpretation. The statewide average percentile in literacy or mathematics at each grade level is the 50th percentile each year. Note that the percentiles are standardized within year and state, meaning that they are not indicators of how ‘true’ achievement has changed over time or how performance compares to students in other states. Percentiles are used to compare the relative annual performance of Arkansas student groups over time and examine if the gap in achievement has changed (details about the methodology can be found at the end of this blog).

We found that in Arkansas, the difference in achievement between “Haves and Have Nots” has remained essentially unchanged over the past decade.

As seen in Figure 1, Arkansas’ literacy achievement gap closed slightly from 25 percentage points to 23 percentage points over the past 10 years. In 2008-09, students participating in the FRL program scored at the 39th percentile in literacy, while students who were not participating scored at the 65th percentile on average.  Over time, the performance on non-participating students remained consistent, while FRL-participating student performance increased slowly over time to the 42nd percentile by 2017-18. The literacy achievement gap closed slightly from 25 percentage points to 23 percentage points over the past 10 years.

FRL LIT

As seen in Figure 2, Arkansas’ mathematics achievement gap increased slightly from 20 percentage points to 23 percentage points over the past 10 years. In 2008-09, students participating in the FRL program scored at the 42nd percentile in mathematics, while students who were not participating scored, on average, at the 62nd percentile.  Student performance fluctuated over time for students in both groups, but the performance of non-FRL participating student increased slightly over time, while the performance of their FRL participating peers remained steady, leading to the slight widening of the achievement gap in mathematics.

FRL Math

It is interesting to note that the achievement gaps in both literacy and mathematics were smallest in the 2014-15 school year, the first (and only) year that Arkansas students completed the PARCC assessment. The gaps closed due to an increase in the average performance of FRL-participating students AND to a decline in the average performance of non-FRL participating students. It is tempting to hypothesize causes behind this decreased gap, but futile because we have only a single data point, and the gap then returns to established levels.

Well Shoot!

As with the national research, we feel these are important things to consider as education/ the level of education attained is key to upward mobility. Both the national and Arkansas results show that certain subgroups are limited in their ability to achieve success academically in comparison to their peers, and that the achievement gaps have persisted over time.

Therefore, it is seen that schools in Arkansas and across the country are not truly meeting the challenge of reducing those disparities among these subgroups. How can that be addressed?

We need to keep digging in order to get to the root of this achievement gap issue. OEP is digging into these gaps more in depth, and will share more information about trends in grade-level achievement gaps, as well as gaps between racial/ethnic and gender groups. We look forward to highlighting the Arkansas schools and districts that HAVE closed the achievement gaps for some student populations.

 


About the methodology:

Students in grades 3-8 were consistently assessed in both content areas, but there was variation in when high school students were assessed. For our analysis we used all grades assessed in literacy and/ or math in a given year, and limited our analyses to students completing the general assessment.

Scale scores were standardized for each year within grade level and content area, creating a Z-score (mean of 0, standard deviation of 1).  These z-scores were then averaged across student groups and transformed into percentiles to ease interpretation. The statewide average percentile in literacy or mathematics at each grade level is the 50th percentile each year. Percentiles are standardized within year and state, meaning that they are not indicators of how actual achievement has changed over time or how performance compares to students in other states. Percentiles are used to compare the relative annual performance of Arkansas student groups over time and examine if the gap in achievement has changed. 

What’s the deal with vouchers?

In The View from the OEP on April 3, 2019 at 2:06 pm

The Arkansas legislature is considering two bills, SB 539 and SB 620, that would make it easier for low-income students to attend private schools. These bills highlight two different approaches for expanding school choice by offering families money to cover a portion of their children’s private school tuition – often referred to as vouchers. Discussions about private school vouchers tend to elicit strong reactions from supporters and opponents alike, making it difficult to wade through the noise and understand what they are, why they exist, what we know about their impact. In this post we hope to help our readers understand the answer to these three questions.

What are vouchers?

 Vouchers are publicly funded grants or scholarships that are used use pay for private school tuition. Vouchers are not new in education policy. As the National Council of State Legislatures’ (NCSL) notes in their summary of vouchers’ history, “state support for private school education has existed in Maine and Vermont for nearly 140 years.” Vouchers have been used in higher education since 1965, offered by the federal government in the form of Pell Grants. However, there has been renewed interest in vouchers since the early 1990s when the first modern K-12 voucher program was established in Milwaukee.

Today twenty-three states offer some form of vouchers for private K-12 education, and they come in three different flavors: 1) tax-credit scholarships, 2) state-funded scholarships, and 3) education savings accounts. Eighteen states offer a tax-credit scholarship, 15 offer a state-funded scholarships, and 6 offer education savings accounts. While program design differs somewhat across states, most of these state-based voucher programs restrict eligibility to either low-income students or students with disabilities (i.e., who have an individual education plan (IEP)). Arkansas currently offers a voucher program called Succeed Scholarship Program to students with disabilities. For more information on state-based voucher programs check out the NCSL Interactive Guide to School Choice Laws and EdChoice’s summary of state choice programs.

Why do vouchers exist?

Vouchers are a public policy tool that can be used to increase families’ schooling options. This is especially true for low-income families who, in the absence of a school choice program, often lack the resources to choose a school different from the one to which they are currently assigned based on their address. Wealthier families, on the other hand, have many more schooling options because they have the resources not only to choose amongst a wider set of neighborhoods but also to send their children to private schools. School choice programs, like vouchers, can help ameliorate the inequity in schooling options, providing low-income parents with the ability to choose the school that best fits their children’ needs.

Some social scientists, most famously Milton Freedman, have also argued that providing more schooling choices in an environment where families have few if any options will increase overall school quality because schools will need to get better to attract and keep students.

What do we know about the impact of vouchers?

Over the years there have been many studies of K-12 education vouchers. Several of these studies are high-quality randomized controlled trials (RCTs) which provide researchers with the best chance of establishing cause and effect. Researchers have studied vouchers in many different locations/contexts and with varying designs (e.g., targeted to low-income families vs not). Despite the amount of high-quality research around vouchers, the story about their impact is equivocal. You can find good summaries of the research in Chalkbeat, Journal of Economic Literature (JEL), and School Choice at a Crossroads ch.4.

The research has studied vouchers’ impact on three main categories of outcomes: 1) voucher students’ achievement on standardized tests, 2) voucher students’ educational attainment (e.g., high school completion, college enrollment, etc.) and longer-term outcomes (e.g., criminal behavior), and 3) the effect on traditional public schools (e.g., achievement, finances, segregation, etc.).

Student Achievement

The results for the impact of vouchers on student achievement are mixed. While several older studies consistently found either no effect or a slight positive effect on achievement, four recent studies have found negative effects and one has found positive effects (see Figure 1 below, omits North Carolina results). Research design may have played a role in the studies’ findings – many of the older studies are RCTs, giving us significant confidence in their results; however, two of the recent studies are as well. The differences in the context and design of the voucher programs are likely more explanatory. The more recent studies evaluated programs that are less tightly targeted to low-income families and students with disabilities and, in some cases, included significant regulation that limited the supply of private options. Another explanation for the mixed results could be that the local public schools used for comparison in the recent research have gotten pretty good at delivering results on state tests. Regardless, while the overarching impact of vouchers on student achievement is not completely clear, it’s likely that, even when well designed, their impact is small to neutral and that the design and context of the voucher program matters a lot.

Figure 1: Math and Reading Test Score Impacts from 12 Voucher Studies

Reproduced from the Brookings Institution Evidence Speaks Reports, Vol 2, #18.

The x-axis is the estimated impact in standard deviation units. For each study, the dot represents the estimated mean impact and the bars represent the 95 percent confidence interval around the mean.

Screen Shot 2019-04-03 at 9.57.10 AM

Attainment and Longer-Term Outcomes

The research story for educational attainment and longer-term outcomes is more positive, although the evidence is not very strong. Several studies have found that vouchers increase high-school graduation and college enrollment. There is, however, mixed evidence on college completion with one study finding a positive impact and another finding none. A recent working paper produced by EDRE’s own Cory DeAngelis and Pat Wolf found that the Milwaukee voucher program reduced criminal activity and paternity suits among participants in adulthood. These results on longer-term outcomes are promising, but more evidence is needed to verify these initial findings.

Effect on Traditional Public Schools

Opponents of vouchers often claim that they harm traditional public schools by lowering the achievement of those left behind, decreasing resources, and increasing segregation. However, we are not aware of any rigorous evidence documenting these harms occurring in real-world programs. In fact, there is significant evidence that voucher programs improve traditional public school performance. As the JEL research summary linked above put it, “Evidence on both small-scale and large- scale programs suggests that competition induced by vouchers leads public schools to improve.” Likewise, there is significant evidence that voucher programs are either cost neutral or save the state and districts money. The evidence on vouchers segregation is not as robust, but does suggest that programs that target low-income families do not have a racially segregative effect (e.g., Louisiana, Milwaukee). However, just because these harms have not been documented in existing programs does not mean that they can be ignored. Policymakers must be aware of the potential harms and do their best to mitigate these concerns when they design voucher programs.

What does that mean for Arkansas?

As noted in the intro, the Arkansas legislature is considering two voucher bills.

SB 620 would create a pilot, state-funded scholarship program for low-income Pulaski County students. This bill is currently in the Senate education committee.

SB 539 would offer tax credits to individuals and corporations who make donations to a fund that will provide scholarships for low-income students to attend private schools. This bill passed out of the Senate a few days ago and is now set to be considered in the House. Seventeen other states already offer similar tax-credit scholarship programs (visit EdChoice for more info on these programs). Our colleague Julie Trivett has produced an analysis of the potential fiscal impact of SB 539, and found that under conservative assumptions the proposed tax-credit scholarship program will be effectively cost neutral.

The bills strictly limit eligibility to low-income families and cap the resources available so that the programs start small, which are both features of programs that have demonstrated positive impacts in past research. The bills also require participants to take a norm-referenced standardized test and schools to report results. Given the research evidence on private school vouchers, we see no compelling reason for policymakers not to consider piloting the proposed voucher programs. While the proposed programs are unlikely to yield large positive results or cost savings, they will increase low-income families’ schooling options and may lead to small improvements for participants and traditional public schools alike. If either of the bills are passed, the legislature should include a requirement that the programs be rigorously evaluated so that we learn from the experience.